One Tuesday morning, every site we self-host went dark at the same time. Our client dashboards, our internal SaaS tools, our staging environments — all unreachable. Nothing had crashed on our end. The server was fine. The application logs were quiet. From the outside the whole network just looked offline.
It took an embarrassingly long time to figure out what had happened: our ISP had changed our public IP address without notice. Every DNS A-record we had pointing at the old IP was now pointing at someone else's modem.
The first-aid fix
The short-term fix was tedious but straightforward: update every DNS record across every domain we host to point at the new IP. Then wait for propagation. By lunch, services were trickling back online. By dinner, everything was green again.
What stuck with us wasn't the recovery time. It was the realization that the architecture was inherently fragile. Every service we hosted depended on a fixed public IP that we didn't actually own. The ISP could change it again next week, and we'd be doing the same scramble.
The architectural fix
We migrated every public-facing self-hosted service to Cloudflare Tunnel. The model is simple: instead of opening ports on our network and exposing our IP to the public DNS, the server makes an outbound connection to Cloudflare's edge. Cloudflare receives the public traffic, terminates the TLS, and proxies it down the tunnel to the application.
The upshots:
- No public-facing IP. The server is invisible from the open internet. Port scans return nothing.
- The ISP can change our IP whenever they want. The tunnel reconnects automatically and everything keeps working. Zero DNS updates needed.
- Cloudflare's edge protection comes free. DDoS mitigation, bot blocking, geographic restrictions — all available without standing up our own infrastructure for it.
- One config file, many hostnames. Adding a new service is a YAML edit, not a firewall rule and a DNS record.
What we'd do differently
In hindsight, the original architecture had a few warning signs we ignored:
"It hasn't changed in years" is not a guarantee. Our previous public IP had been stable for so long that we'd internalized it as fixed. Cable ISPs have always reserved the right to rotate addresses, and they sometimes exercise it without warning.
Centralized dependencies are easy to forget. We had no single document listing every DNS record that pointed at our IP. Building one was the first thing we did after the recovery. If we'd had it on day one, we could have updated everything in 30 minutes instead of three hours.
Monitoring "is the server up" isn't enough. Our uptime monitor was checking each service from the outside. When the IP changed, everything alerted at once and the dashboard turned red. What we actually needed was a monitor that distinguished between "the server is down" (rare) and "the server is unreachable from the public internet" (a different problem with a different fix).
Should you self-host?
This experience didn't sour us on self-hosting — we still run a lot of services in-house and recommend it for the right use cases. We have a separate post on when self-hosting makes sense. But it did sharpen our view that self-hosting without proper edge infrastructure is fragile in ways that aren't obvious until something goes sideways.
If you self-host anything client-facing today and don't have a tunnel or a proper edge layer in front of it, take a Tuesday morning to set one up. The cost is essentially zero. The peace of mind is real.
If you want help planning the right hosting setup for what you're building, we're happy to talk it through.
Want to work with us?
If something here resonated, let's talk about what you're building.