Hello,
I’m trying to get a new kubernetes cluster spun up with Baserow. I’ve got everything working as intended except for the websocket connection.
I’ve followed the k8s install guide with pretty good success. I’ve got everything routed behind a Traefik ingress with a little help from the traefik install guide and the nginx guide. I’ve gotten stumped on a backend-asgi
connection issue that I cannot seem to track down.
I’ve looked through a number of Issues in the main repo and inside of this forum, but have not found anything like what I am seeing. The Debugging Connection Issues post has been helpful but there is still something missing.
Here are the debug logs from what I am seeing wihtin backend-asgi
:
...
127.0.0.1:47792 - "GET /api/_health/ HTTP/1.1" 200
[2022-10-20 19:53:05 +0000] [43] [DEBUG] = connection is CONNECTING
[2022-10-20 19:53:05 +0000] [43] [DEBUG] ! invalid handshake
Traceback (most recent call last):
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 275, in read_http_request
path, headers = await read_request(self.reader)
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/http.py", line 89, in read_request
headers = await read_headers(stream)
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/http.py", line 160, in read_headers
line = await read_line(stream)
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/http.py", line 197, in read_line
raise SecurityError("line too long")
websockets.exceptions.SecurityError: line too long
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 167, in handler
extra_headers=self.extra_headers,
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 583, in handshake
path, request_headers = await self.read_http_request()
File "/baserow/venv/lib/python3.7/site-packages/websockets/legacy/server.py", line 279, in read_http_request
raise InvalidMessage("did not receive a valid HTTP request") from exc
websockets.exceptions.InvalidMessage: did not receive a valid HTTP request
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > HTTP/1.1 400 Bad Request
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Date: Thu, 20 Oct 2022 19:53:05 GMT
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Server: Python/3.7 websockets/10.3
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Content-Length: 77
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Content-Type: text/plain
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > Connection: close
[2022-10-20 19:53:05 +0000] [43] [DEBUG] > [body] (77 bytes)
[2022-10-20 19:53:05 +0000] [43] [INFO] connection failed (400 Bad Request)
[2022-10-20 19:53:05 +0000] [43] [DEBUG] x closing TCP connection
[2022-10-20 19:53:05 +0000] [43] [DEBUG] = connection is CLOSED
[2022-10-20 19:53:05 +0000] [43] [INFO] connection closed
127.0.0.1:47804 - "GET /api/_health/ HTTP/1.1" 200
...
I have tried a number of ways to either modify headers, change headers, use traefik’s TCP ingress vs the standard http one, but I have not been able to get it to work.
For further configuration details, my environment variables look like so:
PRIVATE_BACKEND_URL: "http://backend-wsgi.baserow.svc.cluster.local"
PUBLIC_BACKEND_URL: "http://localhost:8081/backend"
PUBLIC_WEB_FRONTEND_URL: "http://localhost:8081"
And I am simply port forwarding into traefik to access the baserow UI and backend
kubectl port-forward deployment/traefik-deployment -n kube-system 8081:80
Where port 80 is a basic traefik entrypoint, with the following definitions:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: baserow-ingress
namespace: baserow
spec:
entryPoints:
- web
routes:
- kind: Rule
match: PathPrefix(`/`)
priority: 1
services:
- name: web-frontend
namespace: baserow
port: 80
- kind: Rule
match: PathPrefix(`/backend/`)
priority: 10
middlewares:
- name: mwbackend
namespace: baserow
services:
- name: backend-wsgi
namespace: baserow
port: 80
- kind: Rule
match: PathPrefix(`/ws/`)
priority: 5
services:
- name: backend-asgi
namespace: baserow
port: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: mwbackend
namespace: baserow
spec:
stripPrefix:
forceSlash: false
prefixes:
- /backend
Notice here that /backend/
is stripped and rerouted to the backed-wsgi
container, and when the websocket connection comes through with its prefix it is routed to the backend-asgi
container.
I am fairly confident kubectl port-forward
can handle websocket connections, and can see at this line the websocket connection url is swapped with the Public Backend URL correctly, as the GET request from my browser looks like
ws://localhost:8081/ws/core/?jwt_token=eyJ0eXAiOiJKV...Gqzix2-hXQos
And immediately the debug log from the asgi
container pops up with the log above.
A couple options that I have not figured out…
- Use Traefik’s TCP Ingress Route, but only handle the websocket connection. When I apply that configuration, all my http traffic goes through the TCP route
- Make the websocket url an environment variable to allow a different traefik entrypoint through, perhaps using the same ingress or tcp ingress. This would allow us to better debug websocket connections
- Dig deeper into the websocket library and why my line is too long. What line?
I have not modified the containers at this point so I suspect there is some kind of header manipulation or request manipulation that is happening to malform the request.
If we can figure out this configuration, I would be happy to let it be a contribution to the community to run baserow behing traefik inside of kubernetes.
Cheers.
EDIT: Apologies if this was not clear - the websocket connection issue is impacting the frontend user experience with the modal popup on in the upper right corner. I simply never re-establish the connection and it fails after ~10 tries.
Connection to the server has failed. Please refresh the page.