After launching Debian Code Search, sometimes its index-backend processes would crash when presented with some class of queries. The queries itself did not show an interesting pattern, and in fact, it wasn’t their fault.
Looking at the system’s journal, I noticed that the processes were crashing with SIGILL, the signal when an illegal instruction for the CPU is encountered:
Nov 07 00:11:33 codesearch index-backend: SIGILL: illegal instruction Nov 07 00:11:33 codesearch index-backend: PC=0x42558d
Interestingly, on my workstation, I could not reproduce this issue.
Therefore, I fired up gdb and reproduced the problem. After gdb stopped due to SIGILL, I examined the current instruction:
gdb $ x/4i $pc 0x42558d
: vzeroupper 0x425590 : retq 0x425591 : nopl 0x0(%rax) 0x425598 : cmp %ebx,%r10d
Some quick googling revealed that
vzeroupper is an instruction
which is pretty new, but supported by Intel’s i7 and AMD’s Bulldozer CPUs. I am
using an AMD CPU in the server on which Debian Code Search is hosted, but why
would the Go compiler add such an instruction to the code in the first place?
Then it struck me: It’s GCC, invoked because I use cgo for a small part of the code! To get the most performance out of my code when benchmarking, I had setup the cflags like this:
#cgo CFLAGS: -std=gnu99 -O3 -march=native
…leading to GCC putting in instructions which are only available on my
workstation, but not on my server. The fix was to simply remove
Therefore: Be careful with optimizations (doh), especially when you are compiling code on a different machine than you intend to run it on.