zLAN: Worldwide LAN via Zerotier

It might sound counterintuitive, but it’s actually possible. The reason I started looking into this was a news story about a malicious contributor to the xz package who attempted to include a backdoor.

I began wondering if there was a way to SSH into a server while blocking all incoming connections:

SSH port blocked
Server is connected to a relay (outbound)
It gets a message from the relay that someone is “knocking”
Opens the SSH port
Lives happily ever after

I then came across ZeroTier, which can create a virtual Ethernet device, effectively creating a virtual LAN. This virtual device behaves like a physical one on your local network. I became aware of Tailscale later, but its perceived complexity and my satisfaction with my existing zLAN setup have kept me from switching.

Setup¶

To get started, head over to https://www.zerotier.com and create a free account. Once logged in, create a new network from the web dashboard; this will act as your virtual LAN. On each device you want to connect, install the ZeroTier client.

After installation, join your network using the network ID shown in the dashboard:

sudo zerotier-cli join ${NETWORK_ID}

(On macOS, you can also use the tray icon).

Each new device will appear in your web dashboard under the network’s “Members” list. Approve (authorize) each device manually to allow it to join the network. Once approved, every node will have its own ZeroTier-assigned IP address. Personally, I prefer choosing an easy-to-remember subnet (avoiding public ones) and manually setting each member’s IP.

Isolating our zLAN via firewalld¶

Let’s assume we now have ZeroTier up and running.

At home, this setup is usually not an issue. The problem arises on my lab server, where multiple people are connected to the same subnet, some of whom are network security researchers...

The easiest way around this is to set up a firewall that blocks incoming connections from the physical Ethernet port using firewalld. Another alternative is to use a router to isolate the computer from the shared network (TODO).

I’ve just installed firewalld and want to configure it. The first thing to do is not to enable it until it’s properly set up; otherwise, you’ll lock yourself out of your server if you’re configuring it over SSH.

Since firewalld isn’t running yet, we use the offline command: firewalld-offline-cmd.

Run:
```
sudo firewalld-offline-cmd --list-all-zones
```
This will show that there is a public (default) zone with the ssh service, but no interface associated with it. At this point, all interfaces are associated with the default public zone.
Let’s change the default zone to drop:
```
sudo firewalld-offline-cmd --set-default-zone=drop
```
If firewalld is started now, all incoming connections will be dropped.
Using ip addr or ifconfig, we can get the device name for our zLAN, e.g., zeroeth.
```
sudo firewalld-offline-cmd --zone=public --add-interface=zeroeth
```
Multiple interfaces can be added to the same zone. --list-all should now show the new interface in the public zone.
Start the firewalld service.

ZeroTier doesn’t require any specific firewall rules, but we can make things easier by adding its service to the public zone. The default service definitions are located in /usr/lib/firewalld/services/, and ZeroTier uses port 9993/udp.

sudo firewall-cmd --zone=public --add-service=zerotier --permanent
sudo firewall-cmd --reload

You can now test SSH access via the LAN and via the zLAN.

ZeroTier Hacks¶

I’ll add here any hack worth noting to leverage ZeroTier. The obvious one is the ability to SSH into remote servers without having to set up a VPN or expose routers/computers to the internet. If you think this is overkill and that a public key and SSH encryption are enough, think again.

Remote Jupyter Server¶

Sometimes I run a Jupyter server on my lab computer and want to access it remotely. The server runs on ports 8080–8089 by default. I want to make that range accessible on my zLAN:

sudo firewall-cmd --permanent --new-service=jupyter
sudo firewall-cmd --permanent --service=jupyter --add-port=8080-8089/tcp
sudo firewall-cmd --permanent --service=jupyter --add-port=8080-8089/udp
sudo firewall-cmd --permanent --zone=public --add-service=jupyter
sudo firewall-cmd --reload

Let’s test this now by running:

jupyter notebook --no-browser --port=8080 --ip=0.0.0.0 --NotebookApp.token=''

Then open $ZLAN_IP_ADDR:8080 on another computer that’s part of the zLAN. It will usually be port 8080 unless it’s already in use; check the stdout for the correct one. This makes the Jupyter server accessible to other computers.

To avoid typing this long command each time, run:

jupyter notebook --generate-config

This creates a config file at ~/.jupyter/jupyter_notebook_config.py. Add the following settings:

c = get_config()  # noqa
c.ServerApp.open_browser = False
c.ServerApp.password = ''
c.ServerApp.port = 8080
c.ServerApp.token = ''
c.ExtensionApp.open_browser = False
c.LabServerApp.open_browser = False
c.ServerApp.ip = '0.0.0.0'

This is basically my workflow for a quick Jupyter server instance:

Quick SSH to my server: ssh my-server (via pubkey to avoid typing passwords)
Start a tmux session (so the Jupyter server will remain open if we disconnect)
Run the server jupyter notebook from within my Python environment
Open $ZLAN_IP_ADDR:$PORT on my laptop
Voila!

Test

Try running:

jupyter notebook --port=8090

Open $ZLAN_IP_ADDR:8090

SSH Hopping¶

The goal here is to leverage our fixed IPs on the zLAN to do some SSH “hopping”: on the VPS/server, we keep an SSH connection open to a host “login.domain.com” which only accepts passwords and other verification tools. Then, we connect to “login.domain.com” from any device on the zLAN via the VPS’s open connection to bypass the verification process.

On the VPS ~/.ssh/config:

Host remote-node
    HostName login.domain.com
    User ayghri
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist yes        # keep master daemon around
    ServerAliveInterval 60
    ServerAliveCountMax 3

On the laptop ~/.ssh/config:

Host vps
 	Hostname 10.1.0.2
 	User ayghri
 	IdentityFile ~/.ssh/keys/vps-key


Host remote-via-vps
    Hostname 10.1.0.2         # VPS' ZeroTier fixed IP
    User ayghri               # username
    RequestTTY force
    RemoteCommand ssh remote-node  # run on VPS
    IdentityFile ~/.ssh/keys/vps-key
    IdentitiesOnly yes

Now we have to SSH into the VPS and open a persistent SSH connection in the background, which will require going through the login.domain.com verification process:

ssh -fN remote-node

Now we can either ssh vps from the laptop, then ssh remote-node, which should use the existing connection, or simply run ssh remote-via-vps directly from the laptop.

Merging zLAN with LAN¶

Check OpenWrt Router (Bpi-R3).