For the guest WiFi at an event that eventually fell through, we wanted to tunnel all the traffic through my internet connection via my home router.

Because the event is located in another country, many hours of travel away, there are a couple of scenarios where remote control of my home router can be a life-saver. For example, should my home router crash, remotely turning power off and on again gets the event back online.

But, power-cycling a machine is a pretty big hammer. For some cases, like locking yourself out with a configuration mistake, a more precise tool like a remote serial console might be nicer.

In this article, I’ll present two cheap and pragmatic DIY out-of-band management solutions that I have experimented with in the last couple of weeks and wanted to share:

• Variant 1: Remote Power Management only
• Variant 2: a full Remote Console Server

You can easily start with the first variant and upgrade it into the second variant later.

Variant 1: Remote Power Management

Architecture Diagram

Here is the architecture of the system at a glance. The right-hand side is the existing router I want to control, the left-hand side shows the out of band management system:

Let’s go through the hardware components from top to bottom.

Hardware: 4G WiFi Router (Out Of Band Network)

The easiest way to have another network connection for projects like this one is the digitec iot subscription. They offer various different options, and their cheapest one, a 0.4 Mbps flatrate for 4 CHF per month, is sufficient for our use-case.

A convenient way of making the digitec iot subscription available to other devices is to use a mobile WiFi router such as the TP-Link M7350 4G/LTE Mobile Wi-Fi router (68 CHF). You can power it via USB, and it has a built-in battery that will last for a few hours.

By default, the device turns itself off after a while when it thinks it is unused, which is undesired for us — if the smart plug drops out of the WiFi, we don’t want the whole system to go offline. You can turn off this behavior in the web interface under Advanced → Power Saving → Power Saving Mode.

Hardware: WiFi Smart Plug

With the out of band network connection established, all you need to remotely toggle power is a smart plug such as the Sonoff S26 WiFi Smart Plug.

The simplest setup is to connect the Smart Plug to the 4G router via WiFi, and control it using Sonoff’s mobile app via Sonoff’s cloud.

Non-cloud solution

Alternatively, if you want to avoid the Sonoff cloud, the device comes with a “DIY mode”, but the DIY mode wouldn’t work reliably for me when I tried it. Instead, I flashed the Open Source Tasmota firmware and connected it to a self-hosted MQTT server via the internet.

Of course, now your self-hosted MQTT server is a single point of failure, but perhaps you prefer that over the Sonoff cloud being a single point of failure.

Variant 2: Remote Console Server

Turning power off and on remotely is a great start, but what if you need actual remote access to a system? In my case, I’m using a serial port to see log messages and run a shell on my router. This is also called a “serial console”, and any device that allows accessing a serial console without sitting physically in front of the serial port is called a “remote console server”.

Commercially available remote console servers typically offer lots of ports (up to 48) and cost lots of money (many thousand dollars or equivalent), because their target application is to be installed in a rack full of machines in a lab or data center. A few years ago, I built freetserv, an open source, open hardware solution for this problem.

For the use-case at hand, we only need a single serial console, so we’ll do it with a Raspberry Pi.

Architecture Diagram

The architecture for this variant looks similar to the other variant, but adds the consrv Raspberry Pi Zero 2 W and a USB-to-serial adapter:

Hardware: Raspberry Pi Zero 2 W

We’ll use a Raspberry Pi Zero 2 W as our console server. While the device is a little slower than a Raspberry Pi 3 B, it is still plenty fast enough for providing a serial console, and it only consumes 0.8W of power (see gokrazy → Supported platforms for a comparison):

If the Pi Zero 2 W is not available, you can try using any other Raspberry Pi supported by gokrazy, or even an older Pi Zero with the community-supported Pi OS 32-bit kernel (I didn’t test that).

Our Pi will have at least two tasks:

1. With a USB-to-serial adapter, the Pi will provide a serial console.
2. The Pi will run Tailscale mesh networking, which will transparently use either the wired network or fail over to the Out Of Band network. Tailscale also frees us from setting up port forwardings, dynamic DNS or anything like that.
3. Optionally, the Pi can run a local MQTT server if you want to avoid the Sonoff cloud.

Hardware: USB-to-serial adapter

You can use any USB-to-serial adapter supported by Linux. Personally, I like the Adafruit FT232H adapter, which I like to re-program with FTDI’s FT_Prog so that it has a unique serial number.

In my router, I plugged in an Longshine LCS-6321M serial PCIe card to add a serial port. Before you ask: no, using USB serial consoles for the kernel console does not cut it.

Hardware: USB ethernet adapter

Because we not only want this Raspberry Pi to be available via the Out Of Band network (via WiFi), but also on the regular home network, we need a USB ethernet adapter.

Originally I was going to use the Waveshare ETH-USB-HUB-BOX: Ethernet / USB HUB BOX for Raspberry Pi Zero Series, but it turned out to be unreliable.

Instead, I’m now connecting a USB hub (as the Pi Zero 2 W has only one USB port), a Linksys USB3GIG network adapter I had lying around, and my USB-to-serial adapter.

gokrazy setup

Just like in the gokrazy quickstart, we’re going to create a directory for this gokrazy instance:

INSTANCE=gokrazy/consrv
mkdir -p ~/${INSTANCE?}
cd ~/${INSTANCE?}
go mod init consrv

You could now directly run gokr-packer, but personally, I like putting the gokr-packer command into a Makefile right away:

# The consrv hostname resolves to the device’s Tailscale IP address,
# once Tailscale is set up.
PACKER := gokr-packer -hostname=consrv

PKGS := \
	github.com/gokrazy/breakglass \
	github.com/gokrazy/timestamps \
	github.com/gokrazy/serial-busybox \
	github.com/gokrazy/stat/cmd/gokr-webstat \
	github.com/gokrazy/stat/cmd/gokr-stat \
	github.com/gokrazy/mkfs \
	github.com/gokrazy/wifi \
	tailscale.com/cmd/tailscaled \
	tailscale.com/cmd/tailscale \
	github.com/mdlayher/consrv/cmd/consrv

all:

.PHONY: update overwrite

update:
	${PACKER} -update=yes ${PKGS}

overwrite:
	${PACKER} -overwrite=/dev/sdx ${PKGS}

For the initial install, plug the SD card into your computer, put its device name into the overwrite target, and run make overwrite.

For subsequent changes, you can use make update.

Tailscale

Tailscale is a peer-to-peer mesh VPN, meaning we can use it to connect to our consrv Raspberry Pi from anywhere in the world, without having to set up port forwardings, dynamic DNS, or similar.

As an added bonus, Tailscale also transparently fails over between connections, so while the fast ethernet/fiber connection works, Tailscale uses that, otherwise it uses the Out Of Band network.

Follow the gokrazy guide on Tailscale to include the device in your Tailscale mesh VPN.

WiFi internet connection and dual homing

Setup WiFi:

mkdir -p extrafiles/github.com/gokrazy/wifi/etc
cat '{"ssid": "oob", "psk": "secret"}' \
  > extrafiles/github.com/gokrazy/wifi/etc/wifi.json

consrv should use the Out Of Band mobile uplink to reach the internet. At the same time, it should still be usable from my home network, too, to make gokrazy updates go quickly.

We accomplish this using route priorities.

I arranged for the WiFi interface to have higher route priority (5) than the ethernet interface (typically 1, but 11 in our setup thanks to the -extra_route_priority=10 flag):

mkdir -p flags/github.com/gokrazy/gokrazy/cmd/dhcp
echo '-extra_route_priority=10' \
  > flags/github.com/gokrazy/gokrazy/cmd/dhcp/flags.txt
make update

Now, tailscale netcheck shows an IPv4 address belonging to Sunrise, the mobile network provider behind the digitec iot subscription.

The consrv Console Server

consrv is an SSH serial console server written in Go that Matt Layher and I developed. If you’re curious, you can watch the two of us creating it in this twitch stream recording:

The installation of consrv consists of two steps.

Step 1 is done: we already included consrv in the Makefile earlier in gokrazy setup.

So, we only need to configure the desired serial ports in consrv.toml (in gokrazy extrafiles):

mkdir -p extrafiles/github.com/mdlayher/consrv/cmd/consrv/etc/consrv
cat > extrafiles/github.com/mdlayher/consrv/cmd/consrv/etc/consrv/consrv.toml <<'EOT'
[server]
address = ":2222"

[[devices]]
serial = "01716A92"
name = "router7"
baud = 115200
logtostdout = true

[[identities]]
name = "michael"
public_key = "ssh-ed25519 AAAAC3… michael@midna"
EOT

Run make update to deploy the configuration to your device.

If everything is set up correctly, we can now start a serial console session via SSH:

midna% ssh -p 2222 [email protected]
Warning: Permanently added '[consrv.lan]:2222' (ED25519) to the list of known hosts.
consrv> opened serial connection "router7": path: "/dev/ttyUSB0", serial: "01716A92", baud: 115200
2022/06/19 20:50:47 dns.go:175: probe results: [{upstream: [2001:4860:4860::8888]:53, rtt: 999.665µs} {upstream: [2001:4860:4860::8844]:53, rtt: 2.041079ms} {upstream: 8.8.8.8:53, rtt: 2.073279ms} {upstream: 8.8.4.4:53, rtt: 16.200959ms}]
[…]

I’m using the logtostdout option to make consrv continuously read the serial port and send it to stdout, which gokrazy in turn sends via remote syslog to the gokrazy syslog daemon, running on another machine. You could also run it on the same machine if you want to log to file.

Controlling Tasmota from breakglass

You can use breakglass to interactively log into your gokrazy installation.

If you flashed your Smart Plug with Tasmota, you can easily turn power on from a breakglass shell by directly calling Tasmota’s HTTP API with curl:

% breakglass consrv
consrv# curl -v -X POST --data 'cmnd=power on' http://tasmota_68462f-1583/cm

The original Sonoff firmware offers a DIY mode which should also offer an HTTP API, but the DIY mode did not work in my tests. Hence, I’m only describing how to do it with Tasmota.

Optional: Local MQTT Server

Personally, I like having the Smart Plug available both on the local network (via Tasmota’s HTTP API) and via the internet with an external MQTT server. That way, even if either option fails, I still have a way to toggle power remotely.

But, maybe you want to obtain usage stats by listening to MQTT or similar, and you don’t want to use an extra server for this. In that situation, you can easily run a local MQTT server on your Pi.

In the gokrazy Makefile, add github.com/fhmq/hmq to the list of packages to install, and configure Tasmota to connect to consrv on port 1883.

To check that everything is working, use mosquitto_sub from another machine:

midna% mosquitto_sub --verbose -h consrv.monkey-turtle.ts.net -t '#'

Conclusion

digitec’s IOT mobile internet subscription makes remote power management delightfully easy with a smart plug and 4G WiFi router, and affordable enough. The subscription is flexible enough that you can decide to only book it while you’re traveling.

We can elevate the whole setup in functionality (but also complexity) by combining Tailscale, consrv and gokrazy, running on a Raspberry Pi Zero 2 W, and connecting a USB-to-serial adapter.

If you need more features than that, check out the next step on the feature and complexity ladder: PiKVM or TinyPilot. See also this comparison by Jeff Geerling.

Appendix A: Unstable Apple USB ethernet adapter

The first USB ethernet adapter I tried was the Apple USB Ethernet Adapter.

Unfortunately, after a few days of uptime, I experienced the following kernel driver crash (with the asix Linux driver), and the link remained down until I rebooted.

I then switched to a Linksys USB3GIG network adapter (supported by the r8152 Linux driver) and did not see any problems with that so far.

dwc2 3f980000.usb: dwc2_hc_chhltd_intr_dma: Channel 5 - ChHltd set, but reason is unknown
dwc2 3f980000.usb: hcint 0x00000002, intsts 0x04600009
dwc2 3f980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
asix 1-1.4:1.0 eth0: Failed to read reg index 0x0000: -71
------------[ cut here ]------------
WARNING: CPU: 1 PID: 7588 at drivers/net/phy/phy.c:942 phy_error+0x10/0x58
Modules linked in: brcmfmac brcmutil
CPU: 1 PID: 7588 Comm: kworker/u8:2 Not tainted 5.18.3 #1
Hardware name: Raspberry Pi Zero 2 W Rev 1.0 (DT)
Workqueue: events_power_efficient phy_state_machine
pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : phy_error+0x10/0x58
lr : phy_state_machine+0x258/0x2b0
sp : ffff800009fe3d40
x29: ffff800009fe3d40 x28: 0000000000000000 x27: ffff6c7ac300c078
x26: ffff6c7ac300c000 x25: ffff6c7ac4390000 x24: 00000000ffffffb9
x23: 0000000000000004 x22: ffff6c7ac4019cd8 x21: ffff6c7ac4019800
x20: ffffce5c97f6f000 x19: ffff6c7ac4019800 x18: 0000000000000010
x17: 0000000400000000 x16: 0000000000000000 x15: 0000000000001007
x14: ffff800009fe3810 x13: 00000000ffffffea x12: 00000000fffff007
x11: fffffffffffe0290 x10: fffffffffffe0240 x9 : ffffce5c988e1018
x8 : c0000000fffff007 x7 : 00000000000000a8 x6 : ffffce5c98889280
x5 : 0000000000000268 x4 : ffff6c7acf392b80 x3 : ffff6c7ac4019cd8
x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff6c7ac4019800
Call trace:
 phy_error+0x10/0x58
 phy_state_machine+0x258/0x2b0
 process_one_work+0x1e4/0x348
 worker_thread+0x48/0x418
 kthread+0xf4/0x110
 ret_from_fork+0x10/0x20
---[ end trace 0000000000000000 ]---
asix 1-1.4:1.0 eth0: Link is Down