A while ago, I got myself a bunch of HomeMatic home automation gear (valve drives, temperature and humidity sensors, power switches). The gear itself works reasonably well, but I found the management software painfully lacking. Hence, I re-implemented my own management software. In this article, I’ll describe my method, in the hope that others will pick up a few nifty tricks to make future re-implementation projects easier.

Motivation

When buying my HomeMatic devices, I decided to use the wide-spread HomeMatic Central Control Unit (CCU2). This embedded device runs the proprietary rfd wireless daemon, which offers an XML-RPC interface, used by the web interface.

I find the CCU2’s web interface really unpleasant. It doesn’t look modern, it takes ages to load, and it doesn’t indicate progress. I frequently find myself clicking on a button, only to realize that my previous click was still not processed entirely, and then my current click ends up on a different element that I intended to click. Ugh.

More importantly, even if you avoid the CCU2’s web interface altogether and only want to extract sensor values, you’ll come to realize that the device crashes every few weeks. Due to memory pressure, the rfd is killed and doesn’t come back. As a band-aid, I wrote a watchdog cronjob which would just reboot the device. I also reported the bug to the vendor, but never got a reply.

When I tried to update the software to a more recent version, things went so wrong that I decided to downgrade and not touch the device anymore. This is not a good state to be in, so eventually I started my project to replace the device entirely. The replacement is hmgo, a central control unit implemented in Go, deployed to a Raspberry Pi running gokrazy. The radio module I’m using is HomeMatic’s HM-MOD-RPI-PCB, which is connected to a serial port, much like in the CCU2 itself.

Preparation: gather and visualize traces

In order to compare the behavior of the CCU2 stock firmware against my software, I wanted to capture some traces. Looking at what goes on over the air (or on the wire) is also a good learning opportunity to understand the protocol.

1. I wrote a Wireshark dissector (see contrib/homematic.lua). It is a quick & dirty hack, does not properly dissect everything, but it works for the majority of packets. This step alone will make the upcoming work so much easier, because you won’t need to decode packets in your head (and make mistakes!) so often.
2. I captured traffic from the working system. Conveniently, the CCU2 allows SSH'ing in as root after setting a password. Once logged in, I used lsof and ls /proc/$(pidof rfd)/fd to identify the file descriptors which rfd uses to talk to the serial port. Then, I used strace -e read=7,write=7 -f -p $(pidof rfd) to get hex dumps of each read/write. These hex dumps can directly be fed into text2pcap and can be analyzed with Wireshark.
3. I also wrote a little Perl script to extract and convert packet hex dumps from homegear debug logs to text2pcap-compatible format. More on that in a bit.

Preparation: research

Then, I gathered as much material as possible. I found and ended up using the following resources (in order of frequency):

1. homegear source
2. FHEM source
3. homegear presentation
4. hmcfgusb source
5. FHEM wiki

Preparation: lab setup

Next, I got the hardware to work with a known-good software. I set up homegear on a Raspberry Pi, which took a few hours of compilation time because there were no pre-built Debian stretch arm64 binaries. This step established that the hardware itself was working fine.

Also, I got myself another set of traces from homegear, which is always useful.

Implementation

Now the actual implementation can begin. Note that up until this point, I hadn’t written a single line of actual program code. I defined a few milestones which I wanted to reach:

1. Talk to the serial port.
2. Successfully initialize the HM-MOD-RPI-PCB
3. Receive any BidCoS broadcast packet
4. Decode any BidCoS broadcast packet (can largely be done in a unit test)
5. Talk to an already-paired device (re-using the address/key from my homegear setup)
6. Configure an already-paired device
7. Pair a device

To make the implementation process more convenient, I changed the compilation command of my editor to cross-compile the program, scp it to the Raspberry Pi and run it there. This allowed me to test my code with one keyboard shortcut, and I love quick feedback.

Retrospective

The entire project took a few weeks of my spare time. If I had taken some time off of work, I’m confident I could have implemented it in about a week of full-time work.

Consciously doing research, preparation and milestone planning was helpful. It gave me good sense of my progress and achievable goals.

As I’ve learnt previously, investing in tools pays off quickly, even for one-off projects like this one. I’d recommend everyone who’s doing protocol-related work to invest some time in learning to use Wireshark and writing custom Wireshark dissectors.