More Matrix Nonsense

Once again, I’m forced to install some bullshit that only exists because of an obscure service’s insistence to have experimental features. What does that mean? In my humble opinion, there’s no reason that synapse, the matrix server I use, shouldn’t just have livekit built in, but because the protocol is trying its best to look like it has some kind of governmental structure, it has to come under the experimental branch. I’ll just explain

Texting but using your voice

All major calling platforms (think discord, whatsapp, whatever) use VOIP, or Voice Over Internet Protocol, in order to send whatever you’re screaming down the microphone to the speaker of the person on the other end. This, as usual, is really easy when you control all parts of the exchange. Take the whatsapp example: They make and control the apps that both parties are using, they probably control all the infrastructure behind the DNS record, and they control all the servers that the clients connect to. They can hard code as many values as they want, and they know that the system will work in every possible scenario because all the versions of the whatsapp client are derived from the same webassembly project.

Now flip to matrix, where almost every part of the equation isn’t controlled by the admin. Other homeservers could be using different versions of the server, different servers altogether, and have different features enabled, not to mention that the hundreds of clients out there differ massively in how they interact with the homeserver beyond just fetching messages. Some clients might have full support for the protocol, while others won’t even know what a voice channel is, or will just crash if the webrtc version is wrong.

This leads to a “jack of all trades, master of none” system, as is typical of federated networks. It also, as usual, leads to a load of avoidable and badly documented trade offs and unforseen issues.

Unforseen issues…

The backbone of what’s known as “Element call”, element as in the element client, which i hate for various other reasons, is a completely unrelated project called livekit. Livekit is what would be called a VOIP relay, if it wasn’t trying to be some weird ai bridge thing with weird api based room management. This room management will be a problem in approximately 2 paragraphs, so keep that in mind. I won’t pretend to know how livekit works at all, but apparently it’s the only good opensource VOIP client that doesn’t come packaged with its own calling infrastructure. It uses a port range in order to sustain various parallel stream connections, for voice and video from like 20 people at the same time. Of course, this is still limited by bandwidth, but raises a few more issues, caused by the fact I’m a cheapskate.

The only “big tech” company I’ll willingly use is cloudflare, because I trust them a hell of a lot more with my IP than some other random nameserver host. However, unless I pay a large sum of money to Israel, they’ll only forward port 443 (or port 80 if you have ssl disabled) through their proxy servers. Their servers also block raw tcp connections, meaning that for most slightly advanced streaming apps, you need to either pay up or deal with exposing your public address. Guess which one I picked. I’d tried getting ssh to work over their proxy servers before, using various methods of trickery and many nginx configs, but whatever they’re doing inside the DNS validation layer is pretty geared against giving away paid features for free.

The other more important piece of the puzzle is the “element” portion of element call, the authentication service. What this amazing (awful) service does is first convert a matrix key into an internal oauth key, before then using that oauth key to generate a room key that can be used in livekit. Why you can’t just use the matrix keys for this (that use the same key length and everything as far as I can tell), I have no idea, other than to add more complexity for the fun of it. Point matrix at the registration config for the auth service, then point the auth service at the livekit endpoint, and matrix calling should just about work. For my reference and yours, here’s all the config I can think of that you would need to make this setup work:

But how???

The first thing you’re gonna want to do is add the actual containers, like this

services:
    livekit:
    image: livekit/livekit-server:latest
    restart: unless-stopped
    ports:
      - "7880:7880/tcp"
      - "7881:7881/tcp"
      - "3478:3478/udp"
      - "50000-50100:50000-50100/udp"
      - "60000-60500:60000-60500/udp"
    command: ["--config", "/etc/livekit.yaml"]
    extra_hosts:
          - "host.docker.internal:host-gateway"
    volumes:
      - ./synapse-data/livekit.yaml:/etc/livekit.yaml:ro

  lk-jwt-service:
    image: ghcr.io/element-hq/lk-jwt-service:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      LIVEKIT_URL: "wss://livekit.example.com/"
      LIVEKIT_KEY: "${LIVEKIT_KEY}"
      LIVEKIT_SECRET: "${LIVEKIT_SECRET}"
      LIVEKIT_AS_REGISTRATION_FILE: "/etc/lk-jwt/registration.yaml"
      LIVEKIT_HS_SERVER_NAME: "example.com"
      LIVEKIT_FULL_ACCESS_HOMESERVERS: "example.com"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./synapse-data/registration.yaml:/etc/lk-jwt/registration.yaml:ro

This implies a few things, mostly the locations and names of the config files. You’re gonna want to replace the urls and server names to match your domain, and also generate the keys, with these commands:

openssl rand -hex 16 # key
openssl rand -hex 32 # secret

These keys should be consistent across the rest of the config, and you’ll need to use the latter command to generate the HS and AS keys later on.

You can then set up registration.yaml like so:

id: "matrix-livekit"
as_token: "(your as token)"
hs_token: "(your hs token)"
sender_localpart: "_livekit"
url: null

namespaces: # Allow all users to enter voice channels
  users:
    - exclusive: false
      regex: ".*"

scopes: # Actual implemented standard
  - "urn:matrix:client:rooms:is_joined"

io.element.msc4502.scopes: # legacy standard compat
  - "urn:matrix:client:io.element.msc4502:rooms:is_joined"

# Make sure synapse can advertise the server properly
io.element.msc4512.proxy_prefix: "rtc/livekit"
io.element.msc4512.proxy_url: "http://(local ip of lk-jwt-service):8080"

As mentioned, use the command openssl rand -hex 32 to generate the as and hs keys. They’re only in this file, so don’t worry about remembering them too hard. This is a good point to set up homeserver.yaml to use livekit as well. Assuming that both these config files are in the same directory, add all of this to your homeserver.yaml:

# Not necessary but nice to have for calls
presence:
  enabled: true

# Enable features that should be enabled by default
experimental_features:
  # Room summaries (see who is in a vc)
  msc3266_enabled: true
  # Enable matrixRTC (advertise livekit and jw auth)
  msc4143_enabled: true
  # State-after sync (fixes delayed event timeline issues)
  msc4222_enabled: true
  # Unrestricted room member summaries (see who is in a vc without auth)
  msc4502_enabled: true
  # Api proxying mechanism (allow other apps like auth to bypass synapse)
  msc4512_enabled: true

# Required for video and audio streams
rc_delayed_event_mgmt:
  per_second: 1
  burst_count: 20

# Required to account for video delay etc
max_event_delay_duration: 24h

# LiveKit MatrixRTC transport
matrix_rtc:
  transports:
    - type: livekit
      livekit_service_url: "https://livekit.example.com" # legacy compat
      url: "wss://livekit.example.com" # new standard endpoint

# Allow synapse to advertise the auth server
app_service_config_files:
  - /data/registration.yaml

You can find more information on the msc (matrix specification changes) options here. I’m sure you can piece together what all of the stuff does thanks to my amazing commenting skill. It’s important to note that livekit_service_url should point to the auth service, and that url should point directly to livekit’s websocket.

The final piece of the puzzle is configuring livekit itself. In livekit.yaml, again in the same directory:

# Https endpoint
port: 7880

# Configure websocket
rtc:
  use_external_ip: true
  # Stops crashes due to ip discovery
  skip_external_ip_validation: true
  
  # Disable ipv6
  require_ipv4: true
  ips:
    excludes:
      - "::/0"
      
  # Fallback for websocket over tcp
  tcp_port: 7881
  
  # Define data port range
  port_range_start: 50000
  port_range_end: 50100

# Enable turn, required for some element clients to connect properly
turn:
  enabled: true
  udp_port: 3478
  
  # Can overlap with previous port range but better to keep seperate
  relay_range_start: 60000
  relay_range_end: 60500

keys:
  (LIVEKIT_KEY): "(LIVEKIT_SECRET)"

# Stop spammy room creation (can take up lots of ram if you aren't using redis)
room:
  auto_create: false

# Point to the auth server
webhook:
  api_key: "(LIVEKIT_KEY)"
  urls:
    - "https://livekit.example.com"

And voila! You’re done with backend configuration. There are some other settings that you can change, but as far as functionality, this setup will allow 99% of modern homeservers to connect. It’s important to note that in order for other homeservers to connect to your voice channels, they also have to have this set up, or an equivalent functional matrixRTC system. For mainstream servers like matrix.org, this won’t be an issue.

DNS? My favorite!

First of all, you’ll need to decide on a domain to host your livekit instance on. You could use subdomains of your matrix domain, but this is much more fiddly than just having a seperate subdomain, mostly because to forward ports and slugs simultaneously, cloudflare requires another blood sacrifice. I use livekit.voidarc.co.uk so keep that in mind with the next few code blocks.

Once you have an endpoint, you’re gonna want to make a dns record for it, and disable cloudflare proxying. The record should point to your public ip, if it doesn’t then none of the rest of this will work. If you don’t use cloudflare, then go nuts and figure it out yourself. In order to allow connections, you’ll need to forward all of the ports above except from 7880. That’s all of these:

ports:
      - "7881:7881/tcp"
      - "3478:3478/udp"
      - "50000-50100:50000-50100/udp"
      - "60000-60500:60000-60500/udp"

The specific protocol (tcp/udp) does matter here, but there’s not really any harm in just allowing both for all of the connections. Port 8080 from the other container remains internal, and we’ll handle it in the next step.

Then, presuming you have a proxy that goes to port 443, you’ll want to proxy_pass port 7880 to whatever domain you chose. As well as this, you’ll need to add this configuration, in order to make the auth service actually able to authorise:

location ~ ^/(sfu/get|healthz|get_token) {
        proxy_pass http://(ip of server hosting auth):8080$request_uri;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_buffering off;
    }

These 3 endpoints are all you need to essentially multiplex both services over one address. Neat, huh?

Finally, you’ll need to advertise your rtc endpoint in your .well-known file. If you don’t have one, then you can skip this step and hop straight into a vc. Use this nginx config to serve the endpoint:

location = /.well-known/matrix/client {
    default_type application/json;
    add_header Access-Control-Allow-Origin "*" always;
    return 200 '{
    "m.homeserver": {"base_url":"https://matrix.example.com"},
    "org.matrix.msc4143.rtc_foci":[
    	{"type": "livekit",
    	"livekit_service_url": "https://livekit.example.com"}
    ]}';
}

Once that change is live, you should be able to join whatever vc you want. If it doesn’t work, go back through all the steps and make sure that you replaced example.com with your domain (that had me stuck for literally like half an hour). Also make sure that your livekit instance doesn’t immediately crash because the port is already being used. As long as the range is consistent across all of the config files, you can change it to whatever you want, but livekit’s docs reccommend at least 100 ports for both rtc and turn.

Morals are for nerds

I still don’t understand why synapse doesn’t just deal with all of this automatically. RTC over TCP has been a solved problem for like 10 years at this point. I know it’s all experimental and whatever, but this has been in experimental for long enough that every major homeserver has it enabled. I would say it’s about as experimental as flakes are, that to say not very.

Voice calls are funny asl tho