After launching Debian Code Search, sometimes its index-backend processes would crash when presented with some class of queries. The queries itself did not show an interesting pattern, and in fact, it wasn’t their fault.

Looking at the system’s journal, I noticed that the processes were crashing with SIGILL, the signal when an illegal instruction for the CPU is encountered:

Nov 07 00:11:33 codesearch index-backend[10517]: SIGILL: illegal instruction
Nov 07 00:11:33 codesearch index-backend[10517]: PC=0x42558d

Interestingly, on my workstation, I could not reproduce this issue.

Therefore, I fired up gdb and reproduced the problem. After gdb stopped due to SIGILL, I examined the current instruction:

gdb $ x/4i $pc
0x42558d :	vzeroupper 
0x425590 :	retq   
0x425591 :	nopl   0x0(%rax)
0x425598 :	cmp    %ebx,%r10d

Some quick googling revealed that vzeroupper is an instruction which is pretty new, but supported by Intel’s i7 and AMD’s Bulldozer CPUs. I am using an AMD CPU in the server on which Debian Code Search is hosted, but why would the Go compiler add such an instruction to the code in the first place?

Then it struck me: It’s GCC, invoked because I use cgo for a small part of the code! To get the most performance out of my code when benchmarking, I had setup the cflags like this:

#cgo CFLAGS: -std=gnu99 -O3 -march=native

…leading to GCC putting in instructions which are only available on my workstation, but not on my server. The fix was to simply remove -march=native.

Therefore: Be careful with optimizations (doh), especially when you are compiling code on a different machine than you intend to run it on.