Debugging ASGI requests

Hello,

I’m trying to get a new kubernetes cluster spun up with Baserow. I’ve got everything working as intended except for the websocket connection.

I’ve followed the k8s install guide with pretty good success. I’ve got everything routed behind a Traefik ingress with a little help from the traefik install guide and the nginx guide. I’ve gotten stumped on a backend-asgi connection issue that I cannot seem to track down.

I’ve looked through a number of Issues in the main repo and inside of this forum, but have not found anything like what I am seeing. The Debugging Connection Issues post has been helpful but there is still something missing.

Here are the debug logs from what I am seeing wihtin backend-asgi:

...
127.0.0.1:47792 - "GET /api/_health/ HTTP/1.1" 200
[2022-10-20 19:53:05 +0000] [43] [DEBUG] = connection is CONNECTING
[2022-10-20 19:53:05 +0000] [43] [DEBUG] ! invalid handshake
Traceback (most recent call last):
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 275, in read_http_request
    path, headers = await read_request(self.reader)
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/http.py", line 89, in read_request
    headers = await read_headers(stream)
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/http.py", line 160, in read_headers
    line = await read_line(stream)
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/http.py", line 197, in read_line
    raise SecurityError("line too long")
websockets.exceptions.SecurityError: line too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 167, in handler
    extra_headers=self.extra_headers,
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 583, in handshake
    path, request_headers = await self.read_http_request()
  File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 279, in read_http_request
    raise InvalidMessage("did not receive a valid HTTP request") from exc
websockets.exceptions.InvalidMessage: did not receive a valid HTTP request
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > HTTP/1.1 400 Bad Request
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Date: Thu, 20 Oct 2022 19:53:05 GMT
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Server: Python/3.7 websockets/10.3
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Content-Length: 77
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Content-Type: text/plain
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Connection: close
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > [body] (77 bytes)
[2022-10-20 19:53:05 +0000] [43] [INFO] connection failed (400 Bad Request)
[2022-10-20 19:53:05 +0000] [43] [DEBUG] x closing TCP connection
[2022-10-20 19:53:05 +0000] [43] [DEBUG] = connection is CLOSED
[2022-10-20 19:53:05 +0000] [43] [INFO] connection closed
127.0.0.1:47804 - "GET /api/_health/ HTTP/1.1" 200
...

I have tried a number of ways to either modify headers, change headers, use traefik’s TCP ingress vs the standard http one, but I have not been able to get it to work.

For further configuration details, my environment variables look like so:

  PRIVATE_BACKEND_URL: "http://backend-wsgi.baserow.svc.cluster.local"
  PUBLIC_BACKEND_URL: "http://localhost:8081/backend"
  PUBLIC_WEB_FRONTEND_URL: "http://localhost:8081"

And I am simply port forwarding into traefik to access the baserow UI and backend

kubectl port-forward deployment/traefik-deployment -n kube-system 8081:80

Where port 80 is a basic traefik entrypoint, with the following definitions:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: baserow-ingress
  namespace: baserow
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: PathPrefix(`/`)
    priority: 1
    services:
    - name: web-frontend
      namespace: baserow
      port: 80
  - kind: Rule
    match: PathPrefix(`/backend/`)
    priority: 10
    middlewares:
      - name: mwbackend
        namespace: baserow
    services:
      - name: backend-wsgi
        namespace: baserow
        port: 80
  - kind: Rule
    match: PathPrefix(`/ws/`)
    priority: 5
    services:
      - name: backend-asgi
        namespace: baserow
        port: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: mwbackend
  namespace: baserow
spec:
  stripPrefix:
    forceSlash: false
    prefixes:
      - /backend

Notice here that /backend/ is stripped and rerouted to the backed-wsgi container, and when the websocket connection comes through with its prefix it is routed to the backend-asgi container.

I am fairly confident kubectl port-forward can handle websocket connections, and can see at this line the websocket connection url is swapped with the Public Backend URL correctly, as the GET request from my browser looks like

ws://localhost:8081/ws/core/?jwt_token=eyJ0eXAiOiJKV...Gqzix2-hXQos

And immediately the debug log from the asgi container pops up with the log above.

A couple options that I have not figured out…

  • Use Traefik’s TCP Ingress Route, but only handle the websocket connection. When I apply that configuration, all my http traffic goes through the TCP route
  • Make the websocket url an environment variable to allow a different traefik entrypoint through, perhaps using the same ingress or tcp ingress. This would allow us to better debug websocket connections
  • Dig deeper into the websocket library and why my line is too long. What line?

I have not modified the containers at this point so I suspect there is some kind of header manipulation or request manipulation that is happening to malform the request.

If we can figure out this configuration, I would be happy to let it be a contribution to the community to run baserow behing traefik inside of kubernetes.

Cheers.

EDIT: Apologies if this was not clear - the websocket connection issue is impacting the frontend user experience with the modal popup on in the upper right corner. I simply never re-establish the connection and it fails after ~10 tries.

Connection to the server has failed. Please refresh the page. 

Just to follow up on this - I never actually was able to debug this error and the calls that were being made through the port-forward. I decided to actually get a real local domain going that the traefik service would route through. This seems to have solved the problem and admittedly the configuration above is pretty much exactly the same, but instead I am hitting the service through something like

http://baserow.kube-system.svc.cluster.local:8080

The above Traefik IngressRoute is valid, and the env vars look like:

  PRIVATE_BACKEND_URL: "http://backend-wsgi.baserow.svc.cluster.local"
  PUBLIC_BACKEND_URL: "http://baserow.kube-system.svc.cluster.local:8080/backend"
  PUBLIC_WEB_FRONTEND_URL: "http://baserow.kube-system.svc.cluster.local:8080"

So, in conclusion it appears like using kubectl port-forward is not the right approach for initial testing of baserow.

Thanks for sharing the solution @madisonb, this might useful for others as well! Glad to hear that you made it all work in the end.