I’ve been hosting few apps on my LAN network with Avahi DNS for a while and that was mostly it. Using things like Nextcloud with WebDav/CalDav/CardDav is convenient because most interactions are infrequent, e.g. you add a contact / sync a file for backup. iOS handles this surprisingly well even with native Contacts and Reminders apps that sync on next available connection. Having a local-only server is easy, don’t have to patch to latest version, monitor for CVEs, etc. etc. Only respect for developers making and maintaining Nextcloud but that’s some random community software written in PHP. There’s a company behind it but it’s not something I would like to be open on the internet with access to my data.

So I let it run for a while and used mostly as backup store and a way to sync calendar/contacts. But recently I have grown fond of a few more utilities, some OSS, some written by myself and I find it useful to get access to them when I’m on the go. Mainly this is about usememos note hosting service that has a neat companion iOS app and also provides “masonry” view that’s very important to me but it deserves another post. Memos has its own auth implemented but again I have no idea how reliable that is and it would be much easier to have a single access control layer in front of it all. Some other utilities I use are plain Streamlit apps and implementing auth for them is additional hurdle.

After doing a bit of research, a VPN seems like a perfect fit for me to expose those resources on the private network with small attack surface. So with this in mind my first draft architecture came to live. Essentially running it on a small box with public IP on some non-mainstream cloud provider with good data governance laws, like GDPR, e.g. OVH, IONOS to name a few. Another positive of this is most of those clouds provide sign up credits so my first year running this would likely be free and since the box is very small I am unlikely to pay more than 7-10 usd / mo to keep it runnning.

To implement my VPN I ended up with WireGuard, which is highly praised, lightweight tool with good iOS support (only need to scan a QR to get the client config). I’ve also had to add nginx to the mix to make iOS memos app happy, otherwise it would refuse to connect to bare HTTP endpoint due to iOS restrictions. That setup was already done for local mode anyway so I just moved my existing nginx installation to remote box.

I’ve set up a basic docker compose for this, as I want it to be movable. It should be easy to switch providers or just re-deploy on a different IP address as part of instance migration. WireGuard container is the only one that is publishing ports to host network, rest run in bridge mode to communicate with each other. Could have possibly tightened it even more with layered networking but considering there was nothing else running on that box to begin with it seemed not worth the time.

# compose.yaml
services:
  wireguard_server:
    ports:
      - "51820:51820/udp" # only port published on remote box
    ...
  nginx:
    expose:
      - 80
      - 443
    ...
  usememos:
    expose:
      - 5230
    ...

It took a bit of figuring out the IP ranges I need to setup for client.conf and server.conf in AllowedIPs fields respectively but I ended up with WireGuard configuration roughly like this.

# server.conf
[Interface]
Address = 10.0.0.1/24
PrivateKey = // <Generated>
DNS = 10.0.0.1

[Peer]
PublicKey = // <Generated>
Endpoint = 132.32.89.4:51820 # public ip of the box, port needs to match compose file
AllowedIPs = 10.0.0.0/24 # must match your tunnel network

I’ve also added makefile and commands to re-generate public/private keys if I need to re-deploy the setup. Since I need to write those IP ranges in mulitple places including compose.yaml file I ended up creating a few templates for those using Jinja so I don’t have to repeat myself in case I add new containers to the mix or decide to change those ranges.

Nginx has a neat feature it can understand container names from docker compose in its config file (I’ve no idea how it does it, did not look into the details), so to implement routing one can reference those by name. Here nginx is doing a proxy pass to memos container serving a SSL termination endpoint to make iOS app happy.

# nginx.conf under your http {} clause

upstream {
  server usememos:5230;
}

server {
  # ...

  location / {
    proxy_pass http://usememos;
    # ...
  }
}

Final piece of the puzzle is to make those resources reachable from our client. We need a custom DNS. Don’t worry, WireGuard image is often bundled with Coredns so we just need to enable it by setting corresponding environment variable.

environment:
  - USE_COREDNS=true

Then all we need is to define a hosts file that would point client to correct resources.

# hosts.local
10.0.0.1 memos.vpn
10.0.0.1 another-service.vpn # all ips need to point to the same nginx container 

We can use any cool arbitrary domain names here as resolution is done through VPN configuration on the network layer, no additional configuration is required by the client.

Working with this setup a major drawback that became painful very soon is - I care about some data on that instance. Memos uses .sqlite DB that I need to persist and keep backups of now it is running in a cloud environment. I decided to move my containers, except for wireguard back to local and use wireguard as a “gateway” with public IP. After pondering a bit I for some reason decided to tunnel ports of running containers on my home PC (original hosting location with local-only mode) to remote server to “emulate” distributed Docker setup. That was dumb but it was fun excercise in iptables trickery making it possible to bind on non-priveledged ports while still serving traffic on 80 and 433 for app to work.

ssh -R -N <local_host>:<local_port>:<remote_host>:<remote_port> <desination>

It did work but something was still wrong with hostnames during routing so when nginx received the request iOS could not confirm cerificate even though it was working flawlessly with the same certificate on full remote or full local setup. So I scrapped this idea and talked to Gemini for a bit. Surprising amount of configuration and ideation was done through testing ideas with LLMs for this project, so there’s one good thing about all the AI craze. Gemini righfully suggested since I run a VPN network already I can just use my remote box with public IP as a central connection point which both my local setup and my client (phone) would talk to.

To do this we need to add one important modification to place nginx together with wireguard client container on local on the same network interface.

# compose-local.yaml

wireguard_client:
  ...

nginx:
  ...
  network_mode: "service:wireguard_client"

Then it took some time to rewrite docker compose.yaml template to have 2 “kinds” of wireguard containers, one acting as a client running on local and another acting as a “gaeway”-server. As a bonus after switching to this setup all my certificate issues went away too - good riddance! So am I done yet? Sort of, there are few more things I would like to add to the setup, also make adding stuff a bit more compositional, currently it still requires editing both nginx config and compose file to add a new service.

To conclude, it is not ideal by any means, if you do this - do your own research on what kinds of attacks you need to be careful about including your “gateway” getting compromised. But I quite like the fact that I can tear whole thing down, including instance running in the cloud and the only thing I need to setup again on my phone is to scan the QR to get the WireGuard VPN config in place. All secrets, like wireguard keys and nginx certificates are also ephemeral and I set them to re-generate on re-deploy for maximum paranoia.

SERVER_IP=132.32.89.4 make fullstack

The whole setup deploys in few seconds and all I need is to have initial SSH connection to remote box. I ended up writing a Makefile with few targets that would:

• generate wireguard secrets
• generate hosts, compose and wireguard configurations
• scp one compose file with wireguard gateway to the remote host
• spin up local containers with all my services using local compose
• qrencode client config so I can scan from my phone once deployment is done

And all I have to do is to provide an IP address of instance I want to get gateaway running on, this IP will be reflected in updated client configuration. I will cleanup my scripts a bit and share, likely in another post with a bit more commentary on how the templating works so people can extend it.