<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Michael Stapelbergs Website: posts tagged rsync</title>
  <link href="https://michael.stapelberg.ch/posts/tags/rsync/feed.xml" rel="self"/>
  <link href="https://michael.stapelberg.ch/posts/tags/rsync/"/>


  <id>https://michael.stapelberg.ch/posts/tags/rsync/</id>
  <author>
    <name>Michael Stapelberg</name>
  </author>
  <generator>Hugo -- gohugo.io</generator>
  <entry>
    <title type="html"><![CDATA[How my minimal, memory-safe Go rsync steers clear of vulnerabilities]]></title>
    <link href="https://michael.stapelberg.ch/posts/2026-05-24-minimal-memory-safe-go-rsync-vulns/"/>
    <id>https://michael.stapelberg.ch/posts/2026-05-24-minimal-memory-safe-go-rsync-vulns/</id>
    <published>2026-05-24T16:20:22+02:00</published>
    <content type="html"><![CDATA[<p>Back in January 2025, multiple different security researchers published a total
of <a href="https://www.openwall.com/lists/oss-security/2025/01/14/3">6 security vulnerabilities in
rsync</a>, some of which
allow arbitrary code execution and file leaks, so naturally I was wondering
whether/how my <a href="https://github.com/gokrazy/rsync">gokrazy/rsync</a> implementation
was affected. Did implementing my own (compatible, but minimal) rsync in Go, a
modern and memory-safe programming language, really rule out entire classes of
security vulnerabilities?</p>
<p>This deep dive article was in the making since January 2025, but was delayed
because we uncovered more unpublished vulnerabilities in the process! The
“Security Vulnerabilities” section now covers all 12 vulnerabilities from the
January 2025 batch and the May 2026 batch.</p>
<p><strong>If you are running (upstream, samba)
<a href="https://github.com/RsyncProject/rsync">rsync</a> in production, upgrade to version
3.4.3 or newer.</strong></p>
<p>If you are running <a href="https://github.com/gokrazy/rsync">gokrazy/rsync</a> in
production, upgrade to version v0.3.3 or newer.</p>
<p>Feel free to skip over the nitty-gritty security issue details and jump directly to:</p>
<ul>
<li>The <a href="#go-verdict">verdict on whether using Go has helped</a>.</li>
<li>The <a href="#gokrazy-rsync-verdict">verdict on whether a minimal re-implementation like gokrazy/rsync helps</a>.</li>
<li>My <a href="#openrsync">comparison with OpenBSD’s <code>openrsync</code></a> (written in C).</li>
<li><a href="#defenseindepth">Defense in depth mechanisms</a> one can use on Linux.</li>
<li>The <a href="#conclusion">conclusion</a>.</li>
</ul>
<h2 id="context-my-own-rsync">Context: My own rsync</h2>
<p>For context, I blogged about <a href="/posts/2022-06-18-rsync-overview/">rsync, how I use it, and how it
works</a> back in June 2022. See also <a href="/posts/tags/rsync/">all posts
tagged “rsync”</a>.</p>
<p>The original motivation for writing my own rsync (back then only a server, today
all directions are supported) was to provide the software packages of <a href="https://distr1.org/">distri,
my Linux distribution research project for fast package
management</a>, which I wanted to host on
<a href="https://router7.org/">router7</a>, my small home Linux+Go internet router, which
in turn is built on <a href="https://gokrazy.org/">gokrazy</a>, my Go appliance platform.</p>
<p>I am still running multiple gokrazy/rsync servers for this original purpose, and
also many others! Having rsync available as a primitive (that you can link into
your Go programs!) is really nice.</p>
<h2 id="vulns">Security Vulnerabilities</h2>
<p>This article covers the following security vulnerabilities:</p>
<ul>
<li>CVE-2024-12084 to 12088 <a href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">(original report)</a></li>
<li>CVE-2024-12747 (discovered separately by Aleksei Gorban &ldquo;loqpa&rdquo;)</li>
<li>CVE-2026-29518 (discovered by Damien Neil and myself! and independently by <a href="https://nullx3d.com/">Nullx3D</a>)</li>
<li>CVE-2026-43617 to 43620</li>
<li>CVE-2026-45232</li>
</ul>
<p>The first batch of the vulnerabilities above was announced on the <a href="https://www.openwall.com/lists/oss-security/2025/01/14/3">oss-security
mailing list</a>, but
note that the original report has more detail compared to the oss-security
summaries!</p>
<p>The later vulnerabilities were announced via <a href="https://github.com/RsyncProject/rsync/security/advisories">GitHub Security Advisories on the
rsync project</a>.</p>
<h3 id="jan2025">January 2025 batch</h3>
<h4 id="cve-2024-12084">CVE-2024-12084: Heap Buffer Overflow (9.8)</h4>
<p><strong>Summary:</strong></p>
<ul>
<li>rsync performed insufficient validation: It read the (attacker-controlled)
checksum length from the network and compared the length against
<code>MAX_DIGEST_LEN</code>.</li>
<li>However, rsync’s data structures always declared a 16 byte buffer: <code>char sum2[SUM_LENGTH]</code>
<ul>
<li><code>SUM_LENGTH</code> is always 16 (bytes), which is sufficient to hold an
<a href="https://en.wikipedia.org/wiki/MD4">MD4</a> or
<a href="https://en.wikipedia.org/wiki/MD5">MD5</a> checksum.</li>
<li><code>MAX_DIGEST_LEN</code> used to be 16 (bytes), but can be larger when rsync is
compiled with SHA256 or SHA512 checksum support.</li>
</ul>
</li>
<li>Hence, the bounds check was ineffective! An attacker could write out of bounds.</li>
<li>This issue was introduced with <a href="https://github.com/RsyncProject/rsync/commit/ae16850dc58e884eb9f5cb7f772342b2db28f471">commit <code>ae16850</code> in September
2022</a>,
which added SHA256/SHA512 checksum support.</li>
</ul>
<details>
<summary>Click to expand the <strong>full description of the improper checksum length validation</strong> (quoting the <a href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">Google Security
report</a>)</summary>
<blockquote>
<p>When the checksums are read by the daemon, two different checksums are read:</p>
<ol>
<li>A 32-bit Adler-CRC32 Checksum</li>
<li>A digest of the file chunk. The digest algorithm is determined at the beginning of the protocol negotiation.
The corresponding code can be seen below:
<a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/sender.c#L96-L100">sender.c</a>:</li>
</ol>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>s<span style="color:#666">-&gt;</span>sums <span style="color:#666">=</span> <span style="color:#06287e">new_array</span>(<span style="color:#007020;font-weight:bold">struct</span> sum_buf, s<span style="color:#666">-&gt;</span>count);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">for</span> (i <span style="color:#666">=</span> <span style="color:#40a070">0</span>; i <span style="color:#666">&lt;</span> s<span style="color:#666">-&gt;</span>count; i<span style="color:#666">++</span>) {
</span></span><span style="display:flex;"><span>    s<span style="color:#666">-&gt;</span>sums[i].sum1 <span style="color:#666">=</span> <span style="color:#06287e">read_int</span>(f);
</span></span><span style="display:flex;"><span>    <span style="color:#06287e">read_buf</span>(f, s<span style="color:#666">-&gt;</span>sums[i].sum2, s<span style="color:#666">-&gt;</span>s2length);
</span></span></code></pre></div><p>Most importantly, note that <code>sum2</code> field is filled with <code>s-&gt;s2length</code> bytes. <code>sum2</code> always has a size of 16:
<a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/rsync.h#L955-L962">rsync.h</a></p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020">#define SUM_LENGTH 16
</span></span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic">// …
</span></span></span><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">struct</span> sum_buf {
</span></span><span style="display:flex;"><span>    OFF_T offset;           <span style="color:#60a0b0;font-style:italic">/**&lt; offset in file of this chunk */</span>
</span></span><span style="display:flex;"><span>    int32 len;              <span style="color:#60a0b0;font-style:italic">/**&lt; length of chunk of file */</span>
</span></span><span style="display:flex;"><span>    uint32 sum1;            <span style="color:#60a0b0;font-style:italic">/**&lt; simple checksum */</span>
</span></span><span style="display:flex;"><span>    int32 chain;            <span style="color:#60a0b0;font-style:italic">/**&lt; next hash-table collision */</span>
</span></span><span style="display:flex;"><span>    <span style="color:#902000">short</span> flags;            <span style="color:#60a0b0;font-style:italic">/**&lt; flag bits */</span>
</span></span><span style="display:flex;"><span>    <span style="color:#902000">char</span> sum2[SUM_LENGTH];  <span style="color:#60a0b0;font-style:italic">/**&lt; checksum  */</span>
</span></span><span style="display:flex;"><span>};
</span></span></code></pre></div><p><code>s2length</code> is an attacker-controlled value and can have a value up to <code>MAX_DIGEST_LEN</code> bytes, as the next snipper shows:</p>
<p><a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/io.c#L1979-L1984">io.c</a></p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>sum<span style="color:#666">-&gt;</span>s2length <span style="color:#666">=</span> protocol_version <span style="color:#666">&lt;</span> <span style="color:#40a070">27</span> <span style="color:#666">?</span> <span style="color:#002070;font-weight:bold">csum_length</span> : (<span style="color:#902000">int</span>)<span style="color:#06287e">read_int</span>(f);
</span></span><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">if</span> (sum<span style="color:#666">-&gt;</span>s2length <span style="color:#666">&lt;</span> <span style="color:#40a070">0</span> <span style="color:#666">||</span> sum<span style="color:#666">-&gt;</span>s2length <span style="color:#666">&gt;</span> MAX_DIGEST_LEN) {
</span></span><span style="display:flex;"><span>    <span style="color:#06287e">rprintf</span>(FERROR, <span style="color:#4070a0">&#34;Invalid checksum length %d [%s]</span><span style="color:#4070a0;font-weight:bold">\n</span><span style="color:#4070a0">&#34;</span>,
</span></span><span style="display:flex;"><span>            sum<span style="color:#666">-&gt;</span>s2length, <span style="color:#06287e">who_am_i</span>());
</span></span><span style="display:flex;"><span>    <span style="color:#06287e">exit_cleanup</span>(RERR_PROTOCOL);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The problem here is that <code>MAX_DIGEST_LEN</code> can be larger than 16 bytes, depending on the digest support the binary was compiled with:</p>
<p><a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/lib/md-defines.h#L11-L21">md-defines.h</a></p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020">#define MD4_DIGEST_LEN 16
</span></span></span><span style="display:flex;"><span><span style="color:#007020">#define MD5_DIGEST_LEN 16
</span></span></span><span style="display:flex;"><span><span style="color:#007020">#if defined SHA512_DIGEST_LENGTH
</span></span></span><span style="display:flex;"><span>  <span style="color:#007020">#define MAX_DIGEST_LEN SHA512_DIGEST_LENGTH
</span></span></span><span style="display:flex;"><span><span style="color:#007020">#elif defined SHA256_DIGEST_LENGTH
</span></span></span><span style="display:flex;"><span>  <span style="color:#007020">#define MAX_DIGEST_LEN SHA256_DIGEST_LENGTH
</span></span></span><span style="display:flex;"><span><span style="color:#007020">#elif defined SHA_DIGEST_LENGTH
</span></span></span><span style="display:flex;"><span>  <span style="color:#007020">#define MAX_DIGEST_LEN SHA_DIGEST_LENGTH
</span></span></span><span style="display:flex;"><span><span style="color:#007020">#else
</span></span></span><span style="display:flex;"><span>  <span style="color:#007020">#define MAX_DIGEST_LEN MD5_DIGEST_LEN </span><span style="color:#60a0b0;font-style:italic">/* 16 bytes */</span><span style="color:#007020">
</span></span></span><span style="display:flex;"><span><span style="color:#007020">#endif
</span></span></span></code></pre></div><p><code>SHA256</code> support is common and sets the <code>MAX_DIGEST_LENGTH</code> value to 64. As a
result, an attacker can write up to 48 bytes past the <code>sum2</code> buffer limit.</p>
</blockquote>
</details>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/0902b52f6687b1f7952422080d50b93108742e53">upstream fix for
CVE-2024-12084</a>
changes the <code>sum2</code> field to a dynamically-allocated <code>sum2_array</code> field, which is
allocated with <code>xfer_sum_len</code> length, and fixes the bounds check to check
against the <code>xfer_sum_len</code> (checksum length for this transfer’s algorithm).</p>
<p><strong>Can Go help prevent this?</strong></p>
<p>Yes: Missing or incorrect bounds checks will not result in a heap buffer
overflow in Go! Instead, attempting to write out of bounds will result in a
panic because the Go runtime performs bounds checks.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync also had insufficient validation! Our issue was different, though:
It wasn’t size confusion, we just were not doing any validation of the sum
header at all — oops!</p>
<p>We can confirm that the Go runtime’s bounds check triggers
on an attempt to <a href="https://github.com/gokrazy/rsync/blob/3a9d1ec136e049b2bffc9db15230eb70308ac90d/rsyncd/sender.go#L136">write out of
bounds</a>
by changing the code like so and running the tests:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span><span style="color:#000080;font-weight:bold">diff --git i/types.go w/types.go
</span></span></span><span style="display:flex;"><span><span style="color:#000080;font-weight:bold">index 5601697..899fcb8 100644
</span></span></span><span style="display:flex;"><span><span style="color:#a00000">--- i/types.go
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+++ w/types.go
</span></span></span><span style="display:flex;"><span><span style="color:#800080;font-weight:bold">@@ -59,7 +59,7 @@ func (sh *SumHead) WriteTo(c *rsyncwire.Conn) error {
</span></span></span><span style="display:flex;"><span> 	var buf rsyncwire.Buffer
</span></span><span style="display:flex;"><span> 	buf.WriteInt32(sh.ChecksumCount)
</span></span><span style="display:flex;"><span> 	buf.WriteInt32(sh.BlockLength)
</span></span><span style="display:flex;"><span><span style="color:#a00000">-	buf.WriteInt32(sh.ChecksumLength)
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	buf.WriteInt32(512 /*sh.ChecksumLength*/)
</span></span></span><span style="display:flex;"><span> 	buf.WriteInt32(sh.RemainderLength)
</span></span><span style="display:flex;"><span> 	return c.WriteString(buf.String())
</span></span><span style="display:flex;"><span> }
</span></span></code></pre></div><p>As expected, the Go runtime panics with the following message:</p>
<div style="font-size: 80%">
<pre tabindex="0"><code>panic: runtime error: slice bounds out of range [:512] with length 16

goroutine 277 [running]:
github.com/gokrazy/rsync/rsyncd.(*sendTransfer).receiveSums(0xc0000d7b68)
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/sender.go:136 +0x339
github.com/gokrazy/rsync/rsyncd.(*sendTransfer).sendFiles(0xc0000d7b68, 0xc000120820)
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/sender.go:46 +0x134
github.com/gokrazy/rsync/rsyncd.(*Server).handleConnSender(0xc000476090, {{0x95ed9b, 0x7}, {0xc000426810, 0x2a}, {0x0, 0x0, 0x0}}, {0xa2a120, 0xc0000b2ba0}, ...)
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/rsyncd.go:397 +0x26a
github.com/gokrazy/rsync/rsyncd.(*Server).HandleConn(0xc000476090, {{0x95ed9b, 0x7}, {0xc000426810, 0x2a}, {0x0, 0x0, 0x0}}, {0xa2a120, 0xc0000b2ba0}, ...)
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/rsyncd.go:351 +0x37a
github.com/gokrazy/rsync/rsyncd.(*Server).HandleDaemonConn(0xc000476090, {0x94db80?, 0xc00018a040?}, {0x7fd15838b118, 0xc000428028}, {0xa2bd90, 0xc0002303c0})
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/rsyncd.go:307 +0xdbb
github.com/gokrazy/rsync/rsyncd.(*Server).Serve.func2()
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/rsyncd.go:450 +0xaf
created by github.com/gokrazy/rsync/rsyncd.(*Server).Serve in goroutine 260
	/home/michael/go/src/github.com/gokrazy/rsync/rsyncd/rsyncd.go:448 +0xd2
</code></pre></div>
<p>Of course, crashing the entire server is not the best failure mode, so I <a href="https://github.com/gokrazy/rsync/commit/178216f10f2e05fd74bf865f2d0725fce4f907cd">added
the missing bounds checking to turn the panic into an
error</a>.</p>
<h4 id="cve-2024-12085">CVE-2024-12085: Stack Info Leak Defeats ASLR (7.5)</h4>
<p><strong>Summary:</strong></p>
<p>Because of the same lack of validation as in the previous CVE-2024-12084
vulnerability, an attacker could select a checksum algorithm with short
checksums (e.g. <code>xxhash64</code> with 8 byte checksums), but then claim they were
sending longer checksums (e.g. 9 bytes), making the victim leak one byte of
uninitialized stack content in the response.</p>
<p>Leaking one byte of stack content may seem benign, but as the <a
href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">Google
Security report</a> puts it:</p>
<blockquote>
<p>The first pair of vulnerabilities are a Heap Buffer Overflow and an Info Leak.
When combined, they allow a client to execute arbitrary code on the machine a
Rsync server is running on. The client only requires anonymous read-access to
the server.</p>
</blockquote>
<details>
<summary>Click to expand the <strong>full description of the info leak</strong>
(quoting the <a
href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">Google
Security report</a>)</summary>
<blockquote>
<p>The daemon matches checksums of chunks the client sent to the server against
the local file contents in
<a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/match.c#L140-L145"><code>hash_search()</code></a>. Part
of the function prologue is to allocate a buffer on the stack of
<code>MAX_DIGEST_LEN</code> bytes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">static</span> <span style="color:#902000">void</span> <span style="color:#06287e">hash_search</span>(<span style="color:#902000">int</span> f, <span style="color:#007020;font-weight:bold">struct</span> sum_struct <span style="color:#666">*</span>s,
</span></span><span style="display:flex;"><span>                        <span style="color:#007020;font-weight:bold">struct</span> map_struct <span style="color:#666">*</span>buf, OFF_T len)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    OFF_T offset, aligned_offset, end;
</span></span><span style="display:flex;"><span>    int32 k, want_i, aligned_i, backup;
</span></span><span style="display:flex;"><span>    <span style="color:#902000">char</span> sum2[MAX_DIGEST_LEN];
</span></span></code></pre></div><p>The daemon then iterates over the checksums the client sent and generates a
digest for each of the chunks and compares them to the remote digest:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">if</span> (<span style="color:#666">!</span>done_csum2) {
</span></span><span style="display:flex;"><span>    map <span style="color:#666">=</span> (schar <span style="color:#666">*</span>)<span style="color:#06287e">map_ptr</span>(buf, offset, l);
</span></span><span style="display:flex;"><span>    <span style="color:#06287e">get_checksum2</span>((<span style="color:#902000">char</span> <span style="color:#666">*</span>)map, l, sum2);
</span></span><span style="display:flex;"><span>    done_csum2 <span style="color:#666">=</span> <span style="color:#40a070">1</span>;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">if</span> (<span style="color:#06287e">memcmp</span>(sum2, s<span style="color:#666">-&gt;</span>sums[i].sum2, s<span style="color:#666">-&gt;</span>s2length) <span style="color:#666">!=</span> <span style="color:#40a070">0</span>) {
</span></span><span style="display:flex;"><span>    false_alarms<span style="color:#666">++</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">continue</span>;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Notably, the number of bytes that are compared again are <code>s-&gt;s2length</code>
bytes. In this case, the comparison does not go out of bounds since
<code>s-&gt;s2length</code> can be a maximum of <code>MAX_DIGEST_LEN</code>.</p>
<p>However, the local <code>sum2</code> buffer, not to be confused with the
attacker-controlled <code>s-&gt;sums[i].sum2</code>, is a buffer on the stack that is not
cleared and thus contains uninitialized stack contents.</p>
<p>A malicious client can send a (known) <code>xxhash64</code> checksum for a given chunk of
a file, which leads to the daemon writing 8 bytes to the stack buffer
<code>sum2</code>. The attacker can then set <code>s-&gt;s2length</code> to 9 bytes. The result of such
a setup would be that the first 8 bytes match and an attacker-controlled 9th
byte is compared with an unknown value of uninitialized stack data.</p>
<p>An attacker can divide a file into 255 chunks and as a result leak one byte
per file download. An attacker can incrementally repeat the process, either in
the same connection or by resetting the connection.</p>
<p>As a result, they can leak <code>MAX_DIGEST_LEN - 8</code> bytes of uninitialized stack
data, which can contain pointers to Heap objects, Stack cookies, local
variables and pointers to global variables and return pointers. With those
pointers they can defeat ASLR.</p>
</blockquote>
</details>
<p><strong>Upstream fix:</strong></p>
<p>There are two relevant upstream fixes:</p>
<ul>
<li>The <a href="https://github.com/RsyncProject/rsync/commit/0902b52f6687b1f7952422080d50b93108742e53">“Some checksum buffer
fixes”</a>
commit prevents this attack because the attacker-controlled <code>s-&gt;s2length</code> can
no longer be larger than the transfer’s checksum length.</li>
<li>The <a href="https://github.com/RsyncProject/rsync/commit/589b0691e59f761ccb05ddb8e1124991440db2c7">“prevent information leak off the
stack”</a>
commit initializes the <code>sum2</code> memory to zero, thereby making any stack leak
through <code>sum2</code> impossible.</li>
</ul>
<p><strong>Can Go help prevent this?</strong></p>
<p>Yes: By design, Go initializes all variables to the zero value. Go programmers
do not need to remember to explicitly initialize variables.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not affected by this vulnerability: Variables are always
initialized in Go.</p>
<p>Additionally, selecting checksums other than MD4 was only introduced in protocol
version 30 (gokrazy/rsync implements protocol version 27).</p>
<!--
Go does not do ASLR (https://github.com/golang/go/issues/27583#issuecomment-1002725264).
Not sure if it makes sense to mention that here.
-->
<h4 id="cve-2024-12087">CVE-2024-12087: Path Traversal using Symlinks (7.5)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">Google Security
report</a>)</p>
<blockquote>
<p>When the syncing of symbolic links is enabled, either through the <code>-l</code> or <code>-a</code>
(<code>--archive</code>) flags, a malicious server can make the client write arbitrary
files outside of the destination directory.  A malicious server can send the
client a file list such as:</p>
<pre tabindex="0"><code>symlink -&gt; /arbitrary/directory
symlink/poc.txt
</code></pre><p>Symbolic links, by default, can be absolute or contain characters such as <code>../../</code>.</p>
<p>In practice, the client validates the file list and when it sees the
<code>symlink/poc.txt</code> entry, it will look for a directory called <code>symlink</code>,
otherwise it will error out. If the server sends <code>symlink</code> as [both, a
directory and a symbolic link], [the client] will only keep the directory
entry, thus the attack requires some more details to work.</p>
<p>In <code>inc_recurse</code> mode, which the server can enable for the client, the server
sends the client multiple file lists. The deduplication of the entries happens
on a per-file-list basis. As a result, a malicious server can send a client
multiple file lists, where:</p>
<pre tabindex="0"><code># file list 1:
.
./symlink (directory)
./symlink/poc.txt (regular file)

# file list 2:
./symlink -&gt; /arbitrary/path (symlink)
</code></pre><p>As a result, the <code>symlink</code> directory is created first and <code>symlink/poc.txt</code> is
considered a valid entry in the file list. Then, the attacker changes the type
of <code>symlink</code> to a symbolic link.</p>
<p>When the server then instructs the client to create the <code>symlink/poc.txt</code>
file, it will follow the symbolic link and thus files can be created outside
of the destination directory.</p>
</blockquote>
<p><strong>Can Go help prevent this?</strong></p>
<p>No. This vulnerability is caused by a logic error: when multiple file lists are
used, the merged file list needs to be re-verified.</p>
<p>But see <a href="#goosroot">Defense in depth: Go’s <code>os.Root</code></a></p>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/688f5c379a433038bde36897a156d589be373a98">upstream fix for
CVE-2024-12087</a>
adds the missing validation.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not affected by this vulnerability: gokrazy/rsync does not
implement the incremental recursion mode (<code>--inc-recursive</code>).</p>
<p>The trade-off here is implementation complexity vs. resource usage: the
incremental recursion mode allows working with the file set in a “windowed” way,
as opposed to having to scan the entire file set before any transfer can
begin. See also my <a href="/posts/2022-07-02-rsync-how-does-it-work/">How does rsync
work?</a> blog post.</p>
<h4 id="cve-2024-12088">CVE-2024-12088: Bypass for <code>--safe-links</code> (7.5)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">Google Security
report</a>)</p>
<blockquote>
<p>The <code>--safe-links</code> CLI flag makes the client validate any symbolic links it
receives from the server. The desired behavior is that symbolic links target
can only be 1) relative to the destination directory and 2) never point
outside of the destination directory.</p>
<p>The <code>unsafe_symlink()</code> function is responsible for validating these symbolic
links. The function calculates the traversal depth of a symbolic link target,
relative to its position within the destination directory.</p>
<p>As an example, the following symbolic link is considered unsafe:</p>
<p><code>{DESTINATION}/foo -&gt; ../../</code></p>
<p>As it points outside the destination directory. On the other hand, the
following symbolic link is considered safe as it still points within the
destination directory:</p>
<p><code>{DESTINATION}/foo -&gt; a/b/c/d/e/f/../../</code></p>
<p>This function can be bypassed as it does not consider if the destination of a
symbolic link contains other symbolic links in the path. For example, take the
following two symbolic links:</p>
<p><code>{DESTINATION}/a -&gt; .</code>
<code>{DESTINATION}/foo -&gt; a/a/a/a/a/a/../../</code></p>
<p>In this case, foo would actually point outside the destination
directory. However, the <code>unsafe_symlink()</code> function assumes that <code>a/</code> is a
directory and that the symbolic link is safe.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/407c71c7ce562137230e8ba19149c81ccc47c387">upstream fix for
CVE-2024-12088</a>
makes <code>unsafe_symlink()</code> stricter by not allowing <code>../</code> anywhere within the
path, except at the very beginning.</p>
<p><strong>Can Go help prevent this?</strong></p>
<p>No. This vulnerability is caused by a logic error: the validation function was
incorrect. We could have implemented that same bug.</p>
<p>But see <a href="#goosroot">Defense in depth: Go’s <code>os.Root</code></a></p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not vulnerable: The <code>--safe-links</code> feature is not yet
implemented in gokrazy/rsync.</p>
<h4 id="cve-2024-12086">CVE-2024-12086: Arbitrary File Leak (6.8)</h4>
<p><strong>Summary:</strong></p>
<p>The rsync receiver (in client mode) did not sanitize file names provided by the
rsync sender, or otherwise prevent opening files outside the destination tree. A
malicious sender could instruct a receiver to compare checksums of arbitrary
files outside the destination tree. By observing the receiver’s reaction to a
provided one-byte checksum, a malicious sender can leak arbitrary files.</p>
<details>
<summary>Click to expand the <strong>full description of the file leak</strong>
(quoting the <a
href="https://github.com/google/security-research/security/advisories/GHSA-p5pg-x43v-mvqj">Google
Security report</a>)</summary>
<blockquote>
<p>When a client connects to a malicious server the server is able to leak the
contents of an arbitrary file on the client’s machine.  In
<a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/rsync.c#L322C5-L322C23"><code>read_ndx_and_attrs()</code></a>
the client will read <code>fnamecmp</code> type as well as the <code>xname</code> from the server if
the server sets the appropriate flags. The flag <code>sanitize_paths</code> will not be
set for the client.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">if</span> (iflags <span style="color:#666">&amp;</span> ITEM_BASIS_TYPE_FOLLOWS)
</span></span><span style="display:flex;"><span>    fnamecmp_type <span style="color:#666">=</span> <span style="color:#06287e">read_byte</span>(f_in);
</span></span><span style="display:flex;"><span><span style="color:#666">*</span>type_ptr <span style="color:#666">=</span> fnamecmp_type;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">if</span> (iflags <span style="color:#666">&amp;</span> ITEM_XNAME_FOLLOWS) {
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">if</span> ((len <span style="color:#666">=</span> <span style="color:#06287e">read_vstring</span>(f_in, xname, MAXPATHLEN)) <span style="color:#666">&lt;</span> <span style="color:#40a070">0</span>)
</span></span><span style="display:flex;"><span>        <span style="color:#06287e">exit_cleanup</span>(RERR_PROTOCOL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">if</span> (sanitize_paths) { <span style="color:#60a0b0;font-style:italic">/* not enabled when client receives */</span>
</span></span><span style="display:flex;"><span>        <span style="color:#06287e">sanitize_path</span>(xname, xname, <span style="color:#4070a0">&#34;&#34;</span>, <span style="color:#40a070">0</span>, SP_DEFAULT);
</span></span><span style="display:flex;"><span>        len <span style="color:#666">=</span> <span style="color:#06287e">strlen</span>(buf);
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>} <span style="color:#007020;font-weight:bold">else</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#666">*</span>buf <span style="color:#666">=</span> <span style="color:#4070a0">&#39;\0&#39;</span>;
</span></span><span style="display:flex;"><span>    len <span style="color:#666">=</span> <span style="color:#666">-</span><span style="color:#40a070">1</span>;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span><span style="color:#666">*</span>len_ptr <span style="color:#666">=</span> len;
</span></span></code></pre></div><p>The caller
(<a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/receiver.c#L521"><code>recv_files()</code></a>)
then uses the server provided values to determine a file to compare the
incoming data with.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">case</span> <span style="color:#002070;font-weight:bold">FNAMECMP_FUZZY</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">if</span> (file<span style="color:#666">-&gt;</span>dirname) {
</span></span><span style="display:flex;"><span>        <span style="color:#06287e">pathjoin</span>(fnamecmpbuf, <span style="color:#007020;font-weight:bold">sizeof</span> fnamecmpbuf, file<span style="color:#666">-&gt;</span>dirname, xname);
</span></span><span style="display:flex;"><span>        fnamecmp <span style="color:#666">=</span> fnamecmpbuf;
</span></span><span style="display:flex;"><span>    } <span style="color:#007020;font-weight:bold">else</span>
</span></span><span style="display:flex;"><span>        fnamecmp <span style="color:#666">=</span> xname;
</span></span><span style="display:flex;"><span>        <span style="color:#007020;font-weight:bold">break</span>;
</span></span><span style="display:flex;"><span><span style="">…</span>
</span></span><span style="display:flex;"><span>fd1 <span style="color:#666">=</span> <span style="color:#06287e">do_open</span>(fnamecmp, O_RDONLY, <span style="color:#40a070">0</span>);
</span></span></code></pre></div><p>In
<a href="https://github.com/RsyncProject/rsync/blob/9615a2492bbf96bc145e738ebff55bbb91e0bbee/receiver.c#L368"><code>receive_data()</code></a>
the contents of the file specified by <code>xname</code> are copied into the destination
file. This can be achieved by the server sending a negative token.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">while</span> ((i <span style="color:#666">=</span> <span style="color:#06287e">recv_token</span>(f_in, <span style="color:#666">&amp;</span>data)) <span style="color:#666">!=</span> <span style="color:#40a070">0</span>) {
</span></span><span style="display:flex;"><span>..snip..
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">if</span> (i <span style="color:#666">&gt;</span> <span style="color:#40a070">0</span>) {
</span></span><span style="display:flex;"><span>..snip..
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>..snip..
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">if</span> (fd <span style="color:#666">!=</span> <span style="color:#666">-</span><span style="color:#40a070">1</span> <span style="color:#666">&amp;&amp;</span> map <span style="color:#666">&amp;&amp;</span> <span style="color:#06287e">write_file</span>(fd, <span style="color:#40a070">0</span>, offset, map, len) <span style="color:#666">!=</span> (<span style="color:#902000">int</span>)len)
</span></span></code></pre></div><p>The server sends a checksum to compare. If they don&rsquo;t match, a 0 is returned.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#007020;font-weight:bold">if</span> (fd <span style="color:#666">!=</span> <span style="color:#666">-</span><span style="color:#40a070">1</span> <span style="color:#666">&amp;&amp;</span> <span style="color:#06287e">memcmp</span>(file_sum1, sender_file_sum, xfer_sum_len) <span style="color:#666">!=</span> <span style="color:#40a070">0</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#007020;font-weight:bold">return</span> <span style="color:#40a070">0</span>;
</span></span></code></pre></div><p>When the return value is 0 the receiver will then send a <code>MSG_REDO</code> to the
generator. The generator will then write a message to the server.</p>
<p>The server can use this as a signal to determine if the checksum they sent was
correct. By starting off with a <code>blength</code> of 1 a malicious server is able to
determine the contents of the target file byte by byte.</p>
</blockquote>
</details>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/c35e28331f10ba6eba370611abd78bde32d54da7">upstream fix for
CVE-2024-12086</a>
prevents opening files outside the destination tree by verifying the
sender-provided path.</p>
<p><strong>Can Go help prevent this?</strong></p>
<p>Yes, Go offers an API to prevent this, see <a href="#goosroot">Defense in depth: Go’s
<code>os.Root</code></a>.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not vulnerable: the fuzzy matching feature was introduced with
rsync protocol version 29, but gokrazy/rsync implements protocol version 27.</p>
<h4 id="cve-2024-12747">CVE-2024-12747: Symlink Race Condition (5.6)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://access.redhat.com/security/cve/CVE-2024-12747">Red Hat Security
Advisory</a>)</p>
<blockquote>
<p>A flaw was found in rsync. This vulnerability arises from a race condition
during rsync&rsquo;s handling of symbolic links. Rsync&rsquo;s default behavior when
encountering symbolic links is to skip them. If an attacker replaced a regular
file with a symbolic link at the right time, it was possible to bypass the
default behavior and traverse symbolic links. Depending on the privileges of
the rsync process, an attacker could leak sensitive information, potentially
leading to privilege escalation.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/0590b09d9a34ae72741b91ec0708a820650198b0">upstream fix for
CVE-2024-12747</a>
changes <code>open()</code> calls in the rsync sender to use the <code>O_NOFOLLOW</code> option. The
paths are not expected to be symlinks at that point in the algorithm (symlinks
would be handled with <a href="https://manpages.debian.org/readlink.2"><code>readlink(2)</code></a>
).</p>
<p><strong>Can Go help prevent this?</strong></p>
<p>Yes, Go offers an API to prevent this, see <a href="#goosroot">Defense in depth: Go’s
<code>os.Root</code></a>.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync was vulnerable before <a href="https://github.com/gokrazy/rsync/commit/1b1fbf65b8ed696db464bf53445e8f9b97a37210">commit
<code>1b1fbf6</code></a>,
which introduces the same <code>O_NOFOLLOW</code> mitigation that upstream rsync uses.</p>
<details>
<summary>Click to expand the <strong>reproduction steps</strong> to trigger the
issue in <code>gokrazy/rsync</code></summary>
<p>To reproduce the issue, use the following steps:</p>
<ol>
<li>
<p>Check out gokrazy/rsync v0.2.7:</p>
<pre tabindex="0"><code>git clone https://github.com/gokrazy/rsync
cd rsync
git checkout v0.2.7
</code></pre></li>
<li>
<p>Patch the code as follows to undo the fix and execute the attack:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span><span style="color:#000080;font-weight:bold">diff --git i/internal/nofollow/nofollow_unix.go w/internal/nofollow/nofollow_unix.go
</span></span></span><span style="display:flex;"><span><span style="color:#a00000">--- i/internal/nofollow/nofollow_unix.go
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+++ w/internal/nofollow/nofollow_unix.go
</span></span></span><span style="display:flex;"><span><span style="color:#800080;font-weight:bold">@@ -2,8 +2,6 @@
</span></span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> package nofollow
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a00000">-import &#34;golang.org/x/sys/unix&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#a00000">-
</span></span></span><span style="display:flex;"><span> // Maybe resolves to unix.O_NOFOLLOW on unix systems,
</span></span><span style="display:flex;"><span> // 0 on other platforms.
</span></span><span style="display:flex;"><span><span style="color:#a00000">-const Maybe = unix.O_NOFOLLOW
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+const Maybe = 0 // unix.O_NOFOLLOW
</span></span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#000080;font-weight:bold">diff --git i/internal/sender/do.go w/internal/sender/do.go
</span></span></span><span style="display:flex;"><span><span style="color:#a00000">--- i/internal/sender/do.go
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+++ w/internal/sender/do.go
</span></span></span><span style="display:flex;"><span><span style="color:#800080;font-weight:bold">@@ -2,6 +2,8 @@ package sender
</span></span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> import (
</span></span><span style="display:flex;"><span> 	&#34;fmt&#34;
</span></span><span style="display:flex;"><span><span style="color:#00a000">+	&#34;os&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	&#34;path/filepath&#34;
</span></span></span><span style="display:flex;"><span> 	&#34;sort&#34;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> 	&#34;github.com/gokrazy/rsync/internal/log&#34;
</span></span><span style="display:flex;"><span><span style="color:#800080;font-weight:bold">@@ -55,6 +57,15 @@ func (st *Transfer) Do(crd *rsyncwire.CountingReader, cwr *rsyncwire.CountingWri
</span></span></span><span style="display:flex;"><span> 		st.Logger.Printf(&#34;file list sent&#34;)
</span></span><span style="display:flex;"><span> 	}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00a000">+	// HACK: swap out the passwd file with a symlink to /etc/passwd
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	if err := os.Remove(filepath.Join(modPath, &#34;passwd&#34;)); err != nil {
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+		return nil, err
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	}
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	if err := os.Symlink(&#34;../passwd&#34;, filepath.Join(modPath, &#34;passwd&#34;)); err != nil {
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+		return nil, err
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	}
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	st.Logger.Printf(&#34;HACK: swapped passwd file for symlink&#34;)
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+
</span></span></span><span style="display:flex;"><span> 	// Sort the file list. The client sorts, so we need to sort, too (in the
</span></span><span style="display:flex;"><span> 	// same way!), otherwise our indices do not match what the client will
</span></span><span style="display:flex;"><span> 	// request.
</span></span></code></pre></div></li>
</ol>
<p>Running the <code>TestReceiverSymlinkTraversal</code> test now shows that the server
traversed the symlink:</p>
<pre tabindex="0"><code>    receiver_test.go:371: unexpected file contents: diff (-want +got):
          bytes.Join({
        - 	&#34;benign&#34;,
        + 	&#34;secret&#34;,
          }, &#34;&#34;)
</code></pre></details>
<p><strong>A surprising discovery</strong></p>
<p>When I shared a draft of this article with Damien Neil, member of the Go
Security Team and the author of <a href="#goosroot">the traversal-resistant <code>os.Root</code>
API</a>, he pointed out:</p>
<blockquote>
<p>I believe the <a href="https://github.com/gokrazy/rsync/commit/1b1fbf65b8ed696db464bf53445e8f9b97a37210">gokrazy fix for
CVE-2024-12747</a>
is insufficient. You&rsquo;re calling <code>os.Open</code> with <code>O_NOFOLLOW</code>, but <code>O_NOFOLLOW</code>
only prevents symlink traversal in the last path component.</p>
<p>This is probably still vulnerable to replacing an earlier path component so
<code>os.Open(&quot;dir/passwd&quot;)</code> can be redirected by symlinking <code>dir</code> to <code>/etc</code>.</p>
</blockquote>
<p>We reported this to the rsync security contact address in April 2025. In
December 2025 I learned that someone else had also independently discovered and
reported this issue.</p>
<p>Ultimately, this resulted in CVE-2026-29518, published on 2026-05-20.</p>
<h3 id="may2026">May 2026 batch</h3>
<h4 id="cve-2026-29518">CVE-2026-29518: Symlink Race Condition (7.0)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/RsyncProject/rsync/pull/895/changes/8471fdd1561049ef5f58df44a1811a50bd9a531d">rsync 3.4.3 NEWS entry</a>)</p>
<blockquote>
<p>TOCTOU symlink race condition allowing local privilege escalation in
daemon mode without chroot.</p>
<p>An rsync daemon configured with <code>use chroot = no</code> is exposed
to a time-of-check / time-of-use race on parent path components. A local
attacker with write access to a module can replace a parent directory
component with a symlink between the receiver&rsquo;s check and its open(),
redirecting reads (basis-file disclosure) and writes (file overwrite)
outside the module. Under elevated daemon privilege this allows privilege
escalation.</p>
<p>Default <code>use chroot = yes</code> is not exposed.</p>
<p>Reach: local attacker on the daemon host, write access to a module path,
daemon configured with <code>use chroot = no</code>.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/859d44fa4f1420775e4ba050337ef32092f2894c">upstream fix for
CVE-2026-29518</a>
uses <code>secure_relative_open()</code>, which is similar to Go’s <code>os.Root</code> API.</p>
<p><strong>Can Go help prevent this?</strong></p>
<p>Yes, Go offers an API to prevent this, see <a href="#goosroot">Defense in depth: Go’s
<code>os.Root</code></a>.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync was vulnerable until I switched <a href="https://github.com/gokrazy/rsync/commit/b6d50e5e9fd1fdf196f09947a6bc796b1ac48f46">the
sender</a>
and <a href="https://github.com/gokrazy/rsync/commit/cc42e03cad132ba19e0905f149c7c34dab45cd3a">the
receiver</a>
to <a href="#goosroot">the traversal-resistant <code>os.Root</code> API</a>.</p>
<h4 id="cve-2026-43618">CVE-2026-43618: Integer overflow leaks remote memory (8.1)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/RsyncProject/rsync/security/advisories/GHSA-g37v-g3gj-pmwq">GitHub Security Advisory</a>)</p>
<blockquote>
<p>Description: The receiver&rsquo;s compressed-token decoder accumulated a 32-bit
signed counter without overflow checking. A malicious sender can trigger an
overflow that, with careful manipulation, leaks process memory contents to
the attacker &ndash; environment variables, passwords, heap and library pointers
&ndash; significantly weakening ASLR and facilitating further exploitation.</p>
<p>Reach: authenticated daemon connection with compression enabled (the
default for protocols &gt;= 30 when both peers advertise it). Disabling
compression on the daemon (&ldquo;refuse options = compress&rdquo; in rsyncd.conf) is
the available workaround.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/c44c90e9460c666c965446a8c0957f0b9fa4c66a">upstream fix for
CVE-2026-43618</a>
introduces the missing checks.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not vulnerable because it does not implement compression. See
<a href="https://github.com/gokrazy/rsync/issues/35#issuecomment-2988582190">gokrazy/rsync issue
#35</a> for
details on why compression support sounds simple, but is non-trivial.</p>
<h4 id="cve-2026-43620">CVE-2026-43620: DOS after Out-of-bounds read (6.5)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/RsyncProject/rsync/security/advisories/GHSA-28pw-r563-rxvm">GitHub Security Advisory</a>)</p>
<blockquote>
<p>The 2025 fix that added a <code>parent_ndx&lt;0</code> guard in <code>send_files()</code>
was not applied to the visually-identical block in <code>recv_files()</code>. A
malicious rsync server can drive any connecting client into a deterministic
<code>SIGSEGV</code> by setting <code>CF_INC_RECURSE</code> in the compatibility flags, sending a
flist whose first sorted entry is not a leading &ldquo;.&rdquo; directory (which causes
<code>recv_file_list()</code> to set <code>parent_ndx = -1</code>), then sending a transfer record
with <code>ndx=0</code> and a non-<code>ITEM_TRANSFER</code> iflag word. The receiver reads
<code>dir_flist-&gt;files[-1]</code> and dereferences the result. On glibc x86-64 the
dereferenced pointer is mmap chunk metadata that lands at an unmapped
address, hence a clean <code>SEGV_MAPERR</code>; non-glibc allocators have not been
audited.</p>
<p>Reach: any rsync client doing a normal pull from an attacker-controlled
URL. Works for both rsync:// URLs and remote-shell pulls. <code>inc_recurse</code> is the
protocol-30+ default; no special options are required on the victim.</p>
<p>Workaround: <code>--no-inc-recursive</code> on the client.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/0cf200ecbb8baaf58070d825c4fbd892b4a63b69">upstream fix for
CVE-2026-43620</a>
adds the <code>parent_ndx&lt;0</code> guard to <code>recv_files()</code> as well.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>Just like for <a href="#cve-2024-12087">CVE-2024-12087</a>, gokrazy/rsync is not affected
by this vulnerability: gokrazy/rsync does not implement the incremental
recursion mode (<code>--inc-recursive</code>).</p>
<h4 id="cve-2026-43619">CVE-2026-43619: More symlink races (6.3)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/RsyncProject/rsync/security/advisories/GHSA-4h9m-w5ff-j735">GitHub Security
Advisory</a>)</p>
<blockquote>
<p>Description: Earlier fixes for symlink races on the receiver&rsquo;s open() call
(CVE-2026-29518) missed the same race class on every other path-based
system call: chmod, lchown, utimes, rename, unlink, mkdir, symlink, mknod,
link, rmdir, lstat. On rsync daemons with &ldquo;use chroot = no&rdquo; a local
attacker with filesystem access on the daemon host can swap a symlink into
a parent directory component between the receiver&rsquo;s check and one of these
syscalls, redirecting it outside the exported module. The fix routes each
affected path-based syscall through a parent dirfd opened under
RESOLVE_BENEATH-equivalent kernel-enforced confinement (openat2 on Linux
5.6+, O_RESOLVE_BENEATH on FreeBSD 13+ and macOS 15+, per-component
O_NOFOLLOW walk elsewhere). Default &ldquo;use chroot = yes&rdquo; is not exposed.</p>
<p>Reach: local attacker on the daemon host, write access to a module path,
daemon configured with use chroot = no.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/30656c5e358b1c6033f4caf24f3e11b29d25c9f4">upstream fix for
CVE-2026-43619</a>
uses the <code>*at</code> family of syscalls, just like Go’s <code>os.Root</code>.</p>
<p><strong>Can Go help prevent this?</strong></p>
<p>Yes, Go offers an API to prevent this, see <a href="#goosroot">Defense in depth: Go’s
<code>os.Root</code></a>.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not affected, because it uses Go’s <code>os.Root</code> API throughout.</p>
<h4 id="cve-2026-43617">CVE-2026-43617: Hostname/ACL bypass (4.8)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/RsyncProject/rsync/security/advisories/GHSA-rjfm-3w2m-jf4f">GitHub Security
Advisory</a>)</p>
<blockquote>
<p>On an rsync daemon configured with the global <code>daemon chroot = /X</code> rsyncd.conf
setting, the reverse-DNS lookup of the connecting client was performed <em>after</em>
the daemon had chrooted into <code>/X</code>. If <code>/X</code> did not contain the files glibc
needs for resolution (<code>/etc/resolv.conf</code>, <code>/etc/nsswitch.conf</code>, <code>/etc/hosts</code>,
NSS service modules), the lookup failed and the connecting hostname was set to
&ldquo;UNKNOWN&rdquo;. Hostname-based deny rules (&ldquo;hosts deny = *.evil.example&rdquo;) therefore
could not match, and an attacker controlling their PTR record could connect
from a hostname the administrator had intended to deny. IP-based ACLs are
unaffected. The per-module <code>use chroot</code> setting is unrelated to this issue.</p>
<p>Reach: rsync daemon configured with <code>daemon chroot = /X</code> AND hostname-based
ACLs AND <code>/X</code> does not include the libc resolver fixtures.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/c38f20c5ffabacd0c0c483786ab224a36e14bf43">upstream fix for
CVE-2026-43617</a>
moves the DNS lookup to an earlier point in the protocol.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync is not vulnerable because we only implement IP-based allow/deny
lists, not hostname-based allow/deny lists.</p>
<h4 id="cve-2026-45232">CVE-2026-45232: stack out-of-bounds write (3.1)</h4>
<p><strong>Description:</strong> (quoting the <a href="https://github.com/RsyncProject/rsync/security/advisories/GHSA-8f85-j2cv-59m8">GitHub Security Advisory</a>)</p>
<blockquote>
<p>The rsync client&rsquo;s HTTP <code>CONNECT</code> proxy support contains an off-by-one
out-of-bounds stack write in <code>establish_proxy_connection()</code> (<code>socket.c</code>). After
issuing the <code>CONNECT</code> request, rsync reads the proxy&rsquo;s first response line one
byte at a time into a 1024-byte stack buffer with the bound <code>cp &lt; &amp;buffer[sizeof buffer - 1]</code>, so the loop only ever writes <code>buffer[0..sizeof-2]</code>. If the proxy
(or a man-in-the-middle in front of it) returns 1023+ bytes on the first
response line without a <code>'\n'</code> terminator, the loop exits with <code>cp == &amp;buffer[sizeof buffer - 1]</code> — a slot the loop never wrote, so <code>*cp</code> holds stale
stack bytes left there by the earlier <code>snprintf()</code> that formatted the outgoing
<code>CONNECT</code> request. The post-loop code then does:</p>
<pre tabindex="0"><code>if (*cp != &#39;\n&#39;)   /* (*cp is uninitialised stack data) */
    cp++;          /* cp now &amp;buffer[sizeof]: one past end */
*cp-- = &#39;\0&#39;;      /* one-byte OOB write on the stack */
</code></pre><p>The <code>'\0'</code> lands one byte past the end of the on-stack <code>buffer[1024]</code>,
corrupting whatever lives in the adjacent stack slot. AddressSanitizer reports
<code>stack-buffer-overflow</code> at <code>socket.c:95</code> in the <code>establish_proxy_connection</code>
frame.</p>
</blockquote>
<p><strong>Upstream fix:</strong></p>
<p>The <a href="https://github.com/RsyncProject/rsync/commit/a5fc5ebe7a8ef1aa72f6e344599f97fd4427ecba">upstream fix for
CVE-2026-45232</a>
validates the attacker-supplied data.</p>
<p><strong>How does gokrazy/rsync fare?</strong></p>
<p>gokrazy/rsync does not implement such proxy support, so it is not vulnerable.</p>
<h3 id="go-verdict">Go verdict</h3>
<p>Let’s summarize how Go fares:</p>
<ul>
<li>The Go runtime’s bounds checks turn more serious security issues into a panic.
<ul>
<li>A panic is still a denial-of-service risk, but that’s much preferable.</li>
</ul>
</li>
<li>Go initializes memory to zero, making info leaks like CVE-2024-12085 impossible.</li>
<li>Go’s <code>os.Root</code> API prevents most of the remaining vulnerabilities.</li>
<li>Only one out of twelve vulnerabilities (CVE-2026-43617) is a proper bug in
the application logic that using Go could not have prevented.</li>
</ul>
<table>
	<thead>
			<tr>
					<th>CVE number</th>
					<th>Cause</th>
					<th>Risk (C)</th>
					<th>Does Go help?</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>2024-12084</td>
					<td>insufficient validation</td>
					<td>Heap Buffer Overflow</td>
					<td>✅ bounds check (panics!)</td>
			</tr>
			<tr>
					<td>2024-12085</td>
					<td>insufficient validation</td>
					<td>Info Leak</td>
					<td>✅ zero init</td>
			</tr>
			<tr>
					<td>2024-12086</td>
					<td>missing validation</td>
					<td>Arbitrary File Leak</td>
					<td>✅ <code>os.Root</code></td>
			</tr>
			<tr>
					<td>2024-12087</td>
					<td>insufficient validation</td>
					<td>Write Arbitrary Files</td>
					<td>✅ <code>os.Root</code></td>
			</tr>
			<tr>
					<td>2024-12088</td>
					<td>insufficient validation</td>
					<td>Create Arbitrary Symlinks</td>
					<td>✅ <code>os.Root</code></td>
			</tr>
			<tr>
					<td>2024-12747</td>
					<td>TOCTOU</td>
					<td>Leak Privileged Files</td>
					<td>✅ <code>os.Root</code></td>
			</tr>
			<tr>
					<td>2026-29518</td>
					<td>TOCTOU</td>
					<td>Leak Privileged Files</td>
					<td>✅ <code>os.Root</code></td>
			</tr>
			<tr>
					<td>2026-43617</td>
					<td>missing validation</td>
					<td>Deny List Ineffective</td>
					<td>❌ logic bug</td>
			</tr>
			<tr>
					<td>2026-43618</td>
					<td>insufficient validation</td>
					<td>Info Leak</td>
					<td>✅ bounds check (panics!)</td>
			</tr>
			<tr>
					<td>2026-43619</td>
					<td>TOCTOU</td>
					<td>Leak Privileged Files</td>
					<td>✅ <code>os.Root</code></td>
			</tr>
			<tr>
					<td>2026-43620</td>
					<td>insufficient validation</td>
					<td>crash (DOS)</td>
					<td>✅ bounds check (panics!)</td>
			</tr>
			<tr>
					<td>2026-45232</td>
					<td>missing validation</td>
					<td>Memory Write</td>
					<td>✅ bounds check (panics!)</td>
			</tr>
	</tbody>
</table>
<h3 id="gokrazy-rsync-verdict">gokrazy/rsync verdict</h3>
<p>Aside from being written in Go, another key difference between gokrazy/rsync and
the official upstream rsync is that the gokrazy implementation is <strong>minimal</strong>:</p>
<ul>
<li>gokrazy/rsync is unaffected by many vulnerabilities because it does not
implement the feature in question, for example <code>--inc-recursive</code>.</li>
<li>Like all other wire protocol-compatible rsync implementations, gokrazy/rsync
targets protocol version 27, because later protocol versions introduce
significant complexity.</li>
<li>In some cases, features that would be good to implement come with significant
blockers, e.g. compression is tricky, see <a href="https://github.com/gokrazy/rsync/issues/35#issuecomment-2988582190">gokrazy/rsync issue
#35</a> for
details.</li>
</ul>
<p>Let’s have a look at whether gokrazy/rsync was affected by each CVE at the time
of publishing:</p>
<table>
	<thead>
			<tr>
					<th>CVE number</th>
					<th>Cause</th>
					<th>gokrazy/rsync impl?</th>
					<th>gokrazy/rsync affected?</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>2024-12084</td>
					<td>insufficient validation</td>
					<td>yes</td>
					<td>⚠️ panic</td>
			</tr>
			<tr>
					<td>2024-12085</td>
					<td>insufficient validation</td>
					<td>no (proto 30)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2024-12086</td>
					<td>missing validation</td>
					<td>no (proto 29)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2024-12087</td>
					<td>insufficient validation</td>
					<td>no (<code>inc-rec</code>)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2024-12088</td>
					<td>insufficient validation</td>
					<td>no (<code>safe-links</code>)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2024-12747</td>
					<td>TOCTOU</td>
					<td>yes</td>
					<td>❌ vuln</td>
			</tr>
			<tr>
					<td>2026-29518</td>
					<td>TOCTOU</td>
					<td>yes</td>
					<td>⚠️ patched</td>
			</tr>
			<tr>
					<td>2026-43617</td>
					<td>missing validation</td>
					<td>no (host deny lists)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2026-43618</td>
					<td>insufficient validation</td>
					<td>no (compression)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2026-43619</td>
					<td>TOCTOU</td>
					<td>yes</td>
					<td>⚠️ patched</td>
			</tr>
			<tr>
					<td>2026-43620</td>
					<td>insufficient validation</td>
					<td>no (<code>inc-rec</code>)</td>
					<td>✅ not vuln</td>
			</tr>
			<tr>
					<td>2026-45232</td>
					<td>missing validation</td>
					<td>no (proxy)</td>
					<td>✅ not vuln</td>
			</tr>
	</tbody>
</table>
<p>To be clear: all known vulnerabilities are fixed in gokrazy/rsync! The table
above documents what the state was at the time when each CVE was published. In
other words:</p>
<p>When the January 2025 vulnerabilities were published, gokrazy/rsync panicked
(CVE-2024-12084) and was vulnerable to a TOCTOU race (CVE-2024-12747). In the
process of fixing the TOCTOU issue, we discovered CVE-2026-29518, which was
fixed in gokrazy/rsync before the CVE was published. CVE-2026-43619 was
discovered even later, but was also already fixed in gokrazy/rsync with the same
fix: using Go’s <code>os.Root</code> everywhere.</p>
<h3 id="terminology">Imprecise terminology</h3>
<p>As I was reading the vulnerability reports, I noticed that the reports were
slightly misleading by their choice of words: most reports just spoke of
“server” and “client”. However, in an rsync transfer, both sides, the rsync
client and the rsync server can assume either role: sender (upload files) or
receiver (download files)!</p>
<p>Some setups come with further restrictions that make certain attacks harder or
impossible to pull off. For example, when running in daemon mode, file system
access can be restricted to the pre-configured module paths (but not in command
mode!).</p>
<p>Here is a diagram to give you an overview of the 4 different setups and
role/protocol layering:</p>















<a href="https://michael.stapelberg.ch/posts/2026-05-24-minimal-memory-safe-go-rsync-vulns/rsync-combinations.png"><img
  srcset="https://michael.stapelberg.ch/posts/2026-05-24-minimal-memory-safe-go-rsync-vulns/rsync-combinations_hu_fe6c47e615c529cc.png 2x,https://michael.stapelberg.ch/posts/2026-05-24-minimal-memory-safe-go-rsync-vulns/rsync-combinations_hu_2af07d046b10779e.png 3x"
  src="https://michael.stapelberg.ch/posts/2026-05-24-minimal-memory-safe-go-rsync-vulns/rsync-combinations_hu_676301528b69c19f.png"
  alt="diagram showing 4 different rsync setups: (1) rsync --daemon (2) rsync -e ssh; rsync://server/module/dir (3) rsync server:/some/path (4) rsync /src" title="diagram showing 4 different rsync setups: (1) rsync --daemon (2) rsync -e ssh; rsync://server/module/dir (3) rsync server:/some/path (4) rsync /src"
  width="600"
  height="547"
  style="

border: 1px solid #000;

"
  
  loading="lazy"></a>



<p>In the context of our vulnerability reports, I would say that the Arbitrary File
Leak vulnerability (CVE-2024-12086)’s original title “Server leaks arbitrary
client files” can easily be misunderstood.</p>
<p>Instead, I would say: The <strong>rsync receiver</strong> will leak arbitrary files to a
<strong>malicious sender</strong>.</p>
<p>I have verified that a malicious client sender can make an unpatched remote
rsync open files outside the destination tree (e.g. the <code>/etc/shadow</code> system
password database) when running in command mode, for example over SSH. (But,
when running in daemon mode, the server enables additional path sanitization,
which prevents this attack.)</p>
<p>Similarly, the Symlink Path Traversal vulnerability (CVE-2024-12087) speaks
about a “malicious server”, but again, it should be “malicious sender”, which
can be either the client or the server.</p>
<h2 id="openrsync">Comparison with OpenBSD’s openrsync (C)</h2>
<p>The OpenBSD project is known for its security focus, so how does openrsync
compare?</p>
<p>openrsync is not affected by the Heap Buffer Overflow (CVE-2024-12084) and Stack
Info Leak (CVE-2024-12085) vulnerabilities because it validates the checksum
length and only supports one checksum size/algorithm (MD4).</p>
<p>openrsync is not affected by CVE-2024-12086, CVE-2024-12087 and CVE-2024-12088
because it does not implement the relevant features (like gokrazy/rsync). Even
if it was vulnerable, openrsync’s defense-in-depth measures like using
<a href="https://man.openbsd.org/unveil.2">OpenBSD’s <code>unveil(2)</code></a> and
<a href="https://man.openbsd.org/pledge.2"><code>pledge(2)</code></a> to restrict file system access
would have prevented successful exploitation — at least when running on OpenBSD.</p>
<p>openrsync is not affected by CVE-2024-12747 because it <a href="https://github.com/kristapsdz/openrsync/commit/8f1927c35021cd1fbf31cb8c5a6393cc37ba4de8">used <code>O_NOFOLLOW</code> from
the very moment they implemented symlink
support</a>. But,
because <code>O_NOFOLLOW</code> is not a sufficient fix for this issue, openrsync <strong>is</strong>
affected by CVE-2026-29518!</p>
<p>The above covers the January 2025 batch of vulnerabilities; the May 2026 batch
is similar in that most features just are not implemented.</p>
<p>Overall, I say: Well done, Kristaps and contributors! By diligently implementing
validation, restricting the attack surface and employing defense-in-depth
measures, openrsync manages to not be affected by almost all of the reported
vulnerabilities.</p>
<h2 id="defenseindepth">Defense in depth</h2>
<p>Which APIs and environments can we use on Linux for defense-in-depth measures?</p>
<p>I’ll go through the ones <code>gokrazy/rsync</code> supports, ordered by traditional to
modern.</p>
<h3 id="mountnamespaces">Linux mount namespaces</h3>
<p>Within a few weeks after starting the <code>gokrazy/rsync</code> project, I <a href="https://github.com/gokrazy/rsync/commit/d63aaed3be4f69ac98581c4b0ac2646c37f1f8c0">added support
for dropping privileges and using mount/pid namespaces on
Linux</a>
to restrict the file system objects that my rsync server could work with.</p>
<p>This approach works very well to mitigate path traversal attacks, but requires
privileges, meaning we need to run as <code>root</code> or <a href="https://github.com/gokrazy/rsync/commit/da9802e33c89af947d4958ec31e51b14f43ebfb2">in a Linux user
namespace</a>
(if enabled on your distribution / system).</p>
<p>That limitation makes mount namespaces well-suited for server setups, but
usually unavailable for interactive one-off transfers that are typically running
under a human’s user account.</p>
<h3 id="systemdhardening">systemd hardening</h3>
<p>In the same commit that introduced Linux mount/pid namespace support, I also
included a systemd service file that restricted file system access to home
directories and encouraged folks in the README to further restrict file system
access, depending on what their use-case allows.</p>
<p>These file system restrictions, if set up correctly, mitigate the File Leak
(CVE-2024-12086) and Path Traversal (CVE-2024-12087) vulnerabilities.</p>
<p>The Symlink Race Condition (CVE-2024-12747) relies on privilege escalation
through the rsync process, but thanks to the
<a href="https://0pointer.net/blog/dynamic-users-with-systemd.html">DynamicUser</a>
feature, our process has fewer privileges than other users.</p>
<p>Similarly to mount namespaces, these measures are great for server setups, but
too cumbersome to set up for interactive one-off usages.</p>
<h3 id="landlock">Linux Landlock</h3>
<p>I stumbled upon Justine’s blog post <a href="https://justine.lol/pledge/#paths">Porting OpenBSD pledge() to Linux
(2022)</a> and was reminded that Linux offers
the <a href="https://docs.kernel.org/userspace-api/landlock.html">Landlock API</a> for
unprivileged, per-process access control, similar to OpenBSD’s
<a href="https://man.openbsd.org/unveil"><code>unveil(2)</code></a> system call, which openrsync
uses. The basic idea is that once your program knows the directory it works
with, it makes a call like <code>unveil(&quot;/home/michael/backups&quot;, &quot;rw&quot;);</code> and no
longer has access to other file system locations.</p>
<p>I had previously heard of Landlock at a Go Meetup, so I knew there was Go
support for Landlock. Back in 2022, I enabled Landlock support in the gokrazy
kernel images.</p>
<p>So I gave it a shot in March 2025 and <a href="https://github.com/gokrazy/rsync/commit/f84d1e39ec6613599cd9610974e89be5ca183a6d">implemented Landlock support to restrict
file system
access</a>. It
took me a few hours, which seems a little longer than one might expect at
first. Making Landlock work (and/or skipping it) in our test environment ran
into a couple of road blocks: Our tests had defined many functions that get run
in the same process, but when repeatedly adding rulesets, we would exceed the
limit of 16 (!) policy layers per process.</p>
<p>Once I had it set up just right, it is a beautiful solution. Now we can restrict
rsync transfers to their sources (read-only) or destination directories
(read-write), even for unprivileged invocations of <code>gokrazy/rsync</code>! 🎉</p>
<hr>
<p>The downside to Landlock is that Landlock operates at the process level. This
means that Landlock policies must include the files that your program needs,
e.g. <code>gokrazy/rsync</code> needs to be able to read <code>/etc/passwd</code> for user id lookup,
so if the attacker is after the <code>/etc/passwd</code> file, Landlock does not help.</p>
<h3 id="goosroot">Go’s os.Root</h3>
<p>In February 2025, the Go 1.24 release introduced the
<a href="https://pkg.go.dev/os#Root"><code>os.Root</code></a> API, which is resistant against path
traversal, see <a href="https://go.dev/blog/osroot">The Go Blog: Traversal-resistant file
APIs</a> (by Damien Neil, March 2025). This API allows
more fine-grained control (per file system operation) compared to Landlock.</p>
<p>Go 1.25 (released in August 2025) added more methods to <code>os.Root</code>, making it a
convenient choice for most file system usage.</p>
<p>I have converted all of <code>gokrazy/rsync</code>’s file system usage to use <code>os.Root</code>,
which is a great fit: users configure input/output directories, but the
filenames received over the network are untrusted. That’s exactly what <code>os.Root</code>
was designed for!</p>
<hr>
<p>When I first looked into using <code>os.Root</code>, I thought that some system calls could
inherently not be made with this API, like for example <a href="https://manpages.debian.org/mknod.2"><code>mknod(2)</code></a>
 to create device node files. Damien explained:</p>
<blockquote>
<p>It won&rsquo;t support mknod, though.</p>
<p>However, you should be able to use it to enable a safe mknod:</p>
<ul>
<li>os.Root.OpenFile the parent directory of the target,</li>
<li>File.Fd to get the file descriptor for that directory,</li>
<li><a href="https://pkg.go.dev/golang.org/x/sys/unix#Mknodat">https://pkg.go.dev/golang.org/x/sys/unix#Mknodat</a> to create the file.</li>
</ul>
</blockquote>
<p>If you’re curious how that looks in practice, check out <code>gokrazy/rsync</code>’s usage
in <a href="https://github.com/gokrazy/rsync/blob/45444634288e753178af56c61e064ffb08636695/internal/receiver/generatormknod_linux.go#L15-L29"><code>internal/receiver/generatormknod_linux.go</code>, line
15-29</a>.</p>
<p>Another stumbling block was when I realized that unlike with <a href="https://manpages.debian.org/mknodat.2"><code>mknodat(2)</code></a>
, Linux only implements <a href="https://manpages.debian.org/bind.2"><code>bind(2)</code></a>
, but no <code>bindat</code> (as of Linux 7.0)!</p>
<p>Luckily, <a href="https://mastodon.social/@pid_eins/115869484548013992">Lennart Poettering pointed
out</a> that there’s a trick
to skip path resolution without <code>bindat</code>:</p>
<blockquote>
<p>you can probably bind to <code>/proc/self/&lt;fd&gt;/foobar</code> in the meantime&hellip;</p>
</blockquote>
<p>And indeed, this works! Path resolution is skipped because we only specify a
basename (last component of a path) after the known-safe <code>/proc/self/&lt;fd&gt;</code>, not
a path (see <a href="https://github.com/gokrazy/rsync/blob/45444634288e753178af56c61e064ffb08636695/internal/receiver/generatormknod_linux.go#L49-L56">line
49-56</a>).</p>
<p>With these two tips, <code>gokrazy/rsync</code> v0.3.1 and newer are fully using <code>os.Root</code>,
meaning all file system access is traversal-safe! 🥳</p>
<h2 id="conclusion">Conclusion</h2>
<p><strong>Lacking validation causes vulnerabilities</strong></p>
<p>It is interesting to note that aside from the TOCTOU vulnerabilities
(CVE-2024-12747, CVE-2026-29518 and CVE-2026-43619), all other vulnerabilities
were caused by missing or incorrect input validation. In three cases, there was
just no validation to begin with. In another case (CVE-2024-12088), the subject
matter of file system path resolution is tricky enough that the existing
validation did not cover all edge cases.</p>
<p>As the <a href="#go-verdict">Go verdict</a> section explains in more detail, the most
valuable structural fixes are to provide bounds checking (= always-on
validation) and safe-by-default APIs like Go’s <code>os.Root</code>.</p>
<p><strong>Too much complexity</strong></p>
<p>A few of the vulnerabilities came from evolution of the rsync protocol: The
code used to correctly perform sufficient validation, but then new features were
added. For example, when checksum algorithm negotiation was added (protocol
version 30), the validation was not correctly updated. When incremental
recursion was added (also protocol version 30), the validation that made sense
for individual file lists was not updated for the new processing approach of
merging incremental file lists.</p>
<p>Avoiding complexity avoids vulnerabilities! Both gokrazy/rsync and also
openrsync were not vulnerable to 8 out of the 12 security vulnerabilities simply
because they do not implement the feature with the vulnerability.</p>
<p>Of course, these features were added to rsync because they were valuable to
someone at some point, and of course I am not saying that we should just… not
develop software any further, ever.</p>
<p>But, I consider it ideal to <strong>use an implementation whose complexity is
appropriate for and proportional to the complexity of the use-case</strong>. In other
words: for simple use-cases, reach for a simple implementation. Only reach for
the fully-featured implementation where needed.</p>
]]></content>
  </entry>
  <entry>
    <title type="html"><![CDATA[Tips to debug hanging Go programs]]></title>
    <link href="https://michael.stapelberg.ch/posts/2025-02-27-debug-hanging-go-programs/"/>
    <id>https://michael.stapelberg.ch/posts/2025-02-27-debug-hanging-go-programs/</id>
    <published>2025-02-27T17:51:38+01:00</published>
    <content type="html"><![CDATA[<p>I was helping someone get my <a href="https://github.com/gokrazy/rsync">gokrazy/rsync</a>
implementation set up to synchronize <a href="https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure">RPKI
data</a> (used
for securing BGP routing infrastructure), when we discovered that with the right
invocation, my rsync receiver would just hang indefinitely.</p>
<p>This was a quick problem to solve, but in the process, I realized that I should
probably write down a few Go debugging tips I have come to appreciate over the
years!</p>
<h2 id="scenario-hanging-go-program">Scenario: hanging Go program</h2>
<p>If you want to follow along, you can reproduce the issue by building an older
version of gokrazy/rsync, just before the bug fix commit (you’ll need <a href="https://go.dev/dl/">Go 1.22
or newer</a>):</p>
<pre tabindex="0"><code>git clone https://github.com/gokrazy/rsync
cd rsync
git reset --hard 6c89d4dda3be055f19684c0ed56d623da458194e^
go install ./cmd/...
</code></pre><p>Now we can try to sync the repository:</p>
<pre tabindex="0"><code>% gokr-rsync \
  -rtO \
  --delete \
  rsync://rsync.paas.rpki.ripe.net/repository/ \
  /tmp/rpki-repo
[…]
2025/02/08 09:35:10 Opening TCP connection to rsync.paas.rpki.ripe.net:873
2025/02/08 09:35:10 rsync module &#34;repo&#34;, path &#34;repo/&#34;
2025/02/08 09:35:10 (Client) Protocol versions: remote=31, negotiated=27
2025/02/08 09:35:10 Client checksum: md4
2025/02/08 09:35:10 sending daemon args: [--server --sender -tr . repo/]
2025/02/08 09:35:10 exclusion list sent
2025/02/08 09:35:10 receiving file list
2025/02/08 09:35:11 [Receiver] i=0 ? . mode=40755 len=4096 uid=0 gid=0 flags=?
[…]
2025/02/08 09:35:11 [Receiver] i=89 ? clonoth/1/3139332e33322e3130302e302f32342d3234203d3e203537313936.roa mode=100644 len=1747 uid=0 gid=0 flags=?
</code></pre><p>…and then the program just sits there.</p>
<h2 id="sigquit-stack-trace">Tip 1: Press Ctrl+\ (SIGQUIT) to print a stack trace</h2>
<p>The easiest way to look at where a Go program is hanging is to press <code>Ctrl+\</code>
(backslash) to <a href="https://en.wikipedia.org/wiki/Signal_(IPC)#SIGQUIT">make the terminal send it a <code>SIGQUIT</code>
signal</a>. When the Go runtime
receives <code>SIGQUIT</code>, it prints a stack trace to the terminal before exiting the
process. This behavior is enabled by default and can be customized via the
<code>GOTRACEBACK</code> environment variable, see the <a href="https://pkg.go.dev/runtime"><code>runtime</code> package
docs</a>.</p>
<p>Here is what the output looks like in our case. I have made the font small so
that you can recognize the shape of the output (the details are not important,
continue reading below):</p>
<div style="font-size: 60%">
<pre tabindex="0"><code>^\SIGQUIT: quit
PC=0x47664e m=0 sigcode=128

goroutine 0 gp=0x6e6020 m=0 mp=0x6e6ec0 [idle]:
internal/runtime/syscall.Syscall6()
	/home/michael/sdk/go1.23.0/src/internal/runtime/syscall/asm_linux_amd64.s:36 +0xe fp=0x7ffc58665090 sp=0x7ffc58665088 pc=0x47664e
internal/runtime/syscall.EpollWait(0x586651e0?, {0x7ffc5866511c?, 0x3000000018?, 0x7ffc586651f0?}, 0x58665110?, 0x7ffc?)
	/home/michael/sdk/go1.23.0/src/internal/runtime/syscall/syscall_linux.go:32 +0x45 fp=0x7ffc586650e0 sp=0x7ffc58665090 pc=0x4765e5
runtime.netpoll(0xc0000000c0?)
	/home/michael/sdk/go1.23.0/src/runtime/netpoll_epoll.go:116 +0xd2 fp=0x7ffc58665768 sp=0x7ffc586650e0 pc=0x432332
runtime.findRunnable()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:3580 +0x8c5 fp=0x7ffc586658e0 sp=0x7ffc58665768 pc=0x43f045
runtime.schedule()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:3995 +0xb1 fp=0x7ffc58665918 sp=0x7ffc586658e0 pc=0x4405b1
runtime.park_m(0xc0000061c0)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:4102 +0x1eb fp=0x7ffc58665970 sp=0x7ffc58665918 pc=0x4409cb
runtime.mcall()
	/home/michael/sdk/go1.23.0/src/runtime/asm_amd64.s:459 +0x4e fp=0x7ffc58665988 sp=0x7ffc58665970 pc=0x470e2e

goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
runtime.gopark(0x452658?, 0x0?, 0x98?, 0xb3?, 0xb?)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:424 +0xce fp=0xc0000eb358 sp=0xc0000eb338 pc=0x46bc0e
runtime.netpollblock(0x4a01b8?, 0x4058e6?, 0x0?)
	/home/michael/sdk/go1.23.0/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000eb390 sp=0xc0000eb358 pc=0x4318f7
internal/poll.runtime_pollWait(0x7ef586628808, 0x72)
	/home/michael/sdk/go1.23.0/src/runtime/netpoll.go:351 +0x85 fp=0xc0000eb3b0 sp=0xc0000eb390 pc=0x46af05
internal/poll.(*pollDesc).wait(0xc0000ce180?, 0xc00020e99c?, 0x0)
	/home/michael/sdk/go1.23.0/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000eb3d8 sp=0xc0000eb3b0 pc=0x4b0ce7
internal/poll.(*pollDesc).waitRead(...)
	/home/michael/sdk/go1.23.0/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0000ce180, {0xc00020e99c, 0x4, 0x4})
	/home/michael/sdk/go1.23.0/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc0000eb470 sp=0xc0000eb3d8 pc=0x4b17da
net.(*netFD).Read(0xc0000ce180, {0xc00020e99c?, 0x6eeea0?, 0x1?})
	/home/michael/sdk/go1.23.0/src/net/fd_posix.go:55 +0x25 fp=0xc0000eb4b8 sp=0xc0000eb470 pc=0x4f7e85
net.(*conn).Read(0xc000206000, {0xc00020e99c?, 0xc000212000?, 0x6e6ec0?})
	/home/michael/sdk/go1.23.0/src/net/net.go:189 +0x45 fp=0xc0000eb500 sp=0xc0000eb4b8 pc=0x5001a5
net.(*TCPConn).Read(0x0?, {0xc00020e99c?, 0xc0000eb568?, 0x46d449?})
	&lt;autogenerated&gt;:1 +0x25 fp=0xc0000eb530 sp=0xc0000eb500 pc=0x50bb25
io.ReadAtLeast({0x5d9640, 0xc000206000}, {0xc00020e99c, 0x4, 0x4}, 0x4)
	/home/michael/sdk/go1.23.0/src/io/io.go:335 +0x90 fp=0xc0000eb578 sp=0xc0000eb530 pc=0x4957d0
io.ReadFull(...)
	/home/michael/sdk/go1.23.0/src/io/io.go:354
encoding/binary.Read({0x5d9640, 0xc000206000}, {0x5da8b0, 0x7059a0}, {0x55e7c0, 0xc0000eb6a0})
	/home/michael/sdk/go1.23.0/src/encoding/binary/binary.go:244 +0xa5 fp=0xc0000eb670 sp=0xc0000eb578 pc=0x5102a5
github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).ReadMsg(0xc00020a100)
	/home/michael/kr/rsync/internal/rsyncwire/wire.go:50 +0x48 fp=0xc0000eb6e8 sp=0xc0000eb670 pc=0x514428
github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).Read(0x7ef5869b9a68?, {0xc000280000, 0x40000, 0x4dd4fb?})
	/home/michael/kr/rsync/internal/rsyncwire/wire.go:72 +0x2f fp=0xc0000eb788 sp=0xc0000eb6e8 pc=0x5145af
bufio.(*Reader).Read(0xc0002020c0, {0xc00020e998, 0x4, 0x40ece5?})
	/home/michael/sdk/go1.23.0/src/bufio/bufio.go:241 +0x197 fp=0xc0000eb7c0 sp=0xc0000eb788 pc=0x4d5a57
io.ReadAtLeast({0x5d93e0, 0xc0002020c0}, {0xc00020e998, 0x4, 0x4}, 0x4)
	/home/michael/sdk/go1.23.0/src/io/io.go:335 +0x90 fp=0xc0000eb808 sp=0xc0000eb7c0 pc=0x4957d0
io.ReadFull(...)
	/home/michael/sdk/go1.23.0/src/io/io.go:354
github.com/gokrazy/rsync/internal/rsyncwire.(*Conn).ReadInt32(0xc000208060)
	/home/michael/kr/rsync/internal/rsyncwire/wire.go:163 +0x4a fp=0xc0000eb850 sp=0xc0000eb808 pc=0x51490a
github.com/gokrazy/rsync/internal/receiver.(*Transfer).recvIdMapping1(0xc000202120, 0x5a9b58)
	/home/michael/kr/rsync/internal/receiver/uidlist.go:16 +0x3d fp=0xc0000eb8c0 sp=0xc0000eb850 pc=0x51fc7d
github.com/gokrazy/rsync/internal/receiver.(*Transfer).RecvIdList(0xc000202120)
	/home/michael/kr/rsync/internal/receiver/uidlist.go:52 +0x1dd fp=0xc0000eba08 sp=0xc0000eb8c0 pc=0x51ffbd
github.com/gokrazy/rsync/internal/receiver.(*Transfer).ReceiveFileList(0xc000202120)
	/home/michael/kr/rsync/internal/receiver/flist.go:229 +0x378 fp=0xc0000ebb10 sp=0xc0000eba08 pc=0x51c5b8
github.com/gokrazy/rsync/internal/receivermaincmd.clientRun({{0x5d9280, 0xc000078058}, {0x5d92a0, 0xc000078060}, {0x5d92a0, 0xc000078068}}, 0xc0000d0d90, {0x7ef53d47efc8, 0xc000206000}, {0x7ffc5866600e, ...}, ...)
	/home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:341 +0x5cd fp=0xc0000ebc10 sp=0xc0000ebb10 pc=0x550c2d
github.com/gokrazy/rsync/internal/receivermaincmd.socketClient({{0x5d9280, 0xc000078058}, {0x5d92a0, 0xc000078060}, {0x5d92a0, 0xc000078068}}, 0xc0000d0d90, {0x7ffc58665ff4?, 0x1?}, {0x7ffc5866600e, ...})
	/home/michael/kr/rsync/internal/receivermaincmd/clientserver.go:44 +0x425 fp=0xc0000ebcd0 sp=0xc0000ebc10 pc=0x54c205
github.com/gokrazy/rsync/internal/receivermaincmd.rsyncMain({{0x5d9280, 0xc000078058}, {0x5d92a0, 0xc000078060}, {0x5d92a0, 0xc000078068}}, 0xc0000d0d90, {0xc00007e440, 0x1, 0x2}, ...)
	/home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:160 +0x5d7 fp=0xc0000ebdf0 sp=0xc0000ebcd0 pc=0x54f697
github.com/gokrazy/rsync/internal/receivermaincmd.Main({0xc0000160a0, 0x5, 0x5}, {0x5d9280?, 0xc000078058?}, {0x5d92a0?, 0xc000078060?}, {0x5d92a0?, 0xc000078068?})
	/home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:394 +0x272 fp=0xc0000ebee8 sp=0xc0000ebdf0 pc=0x5510d2
main.main()
	/home/michael/kr/rsync/cmd/gokr-rsync/rsync.go:12 +0x4e fp=0xc0000ebf50 sp=0xc0000ebee8 pc=0x5515ae
runtime.main()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:272 +0x28b fp=0xc0000ebfe0 sp=0xc0000ebf50 pc=0x438d4b
runtime.goexit({})
	/home/michael/sdk/go1.23.0/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000ebfe8 sp=0xc0000ebfe0 pc=0x472e61

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:424 +0xce fp=0xc000074fa8 sp=0xc000074f88 pc=0x46bc0e
runtime.goparkunlock(...)
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:430
runtime.forcegchelper()
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:337 +0xb3 fp=0xc000074fe0 sp=0xc000074fa8 pc=0x439093
runtime.goexit({})
	/home/michael/sdk/go1.23.0/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x472e61
created by runtime.init.7 in goroutine 1
	/home/michael/sdk/go1.23.0/src/runtime/proc.go:325 +0x1a
</code></pre></div>
<p>Phew! This output is pretty dense.</p>
<p>We can use the <a href="https://github.com/maruel/panicparse">https://github.com/maruel/panicparse</a> program to present this
stack trace in a more colorful and much shorter version:</p>















<a href="https://michael.stapelberg.ch/posts/2025-02-27-debug-hanging-go-programs/2025-02-08-panicparse.jpg"><img
  srcset="https://michael.stapelberg.ch/posts/2025-02-27-debug-hanging-go-programs/2025-02-08-panicparse_hu_4c5f7b3d7df4ad22.jpg 2x,https://michael.stapelberg.ch/posts/2025-02-27-debug-hanging-go-programs/2025-02-08-panicparse_hu_ad1e8c2d9905e0.jpg 3x"
  src="https://michael.stapelberg.ch/posts/2025-02-27-debug-hanging-go-programs/2025-02-08-panicparse_hu_a426e16585061786.jpg"
  
  width="600"
  height="394"
  style="

border: 1px solid #000;

"
  
  loading="lazy"></a>



<p>The functions helpfully highlighted in red are where the problem lies: My rsync
receiver implementation was incorrectly expecting the server to send a uid/gid
list, despite the PreserveUid and PreserveGid options not being enabled. <a href="https://github.com/gokrazy/rsync/commit/6c89d4dda3be055f19684c0ed56d623da458194e">Commit
<code>6c89d4d</code></a>
fixes the issue.</p>
<h2 id="attach-dlv">Tip 2: Attach the delve debugger to the process</h2>
<p>If dumping the stack trace in the moment is not sufficient to diagnose the
problem, you can go one step further and reach for an interactive debugger.</p>
<p>The most well-known Linux debugger is probably GDB, but when working with Go, I
recommend using <a href="https://github.com/go-delve/delve">the delve debugger</a> instead
as it typically works better. Install delve if you haven’t already:</p>
<pre tabindex="0"><code>% go install github.com/go-delve/delve/cmd/dlv@latest
</code></pre><p>In this article, I am using delve v1.24.0.</p>
<aside class="admonition note">
  <div class="note-container">
    <div class="note-icon" style="width: 20px; height: 20px">
      <svg id="exclamation-icon" width="100%" height="100%" viewBox="0 0 24 24" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:1.41421;">
    <path d="M0,0L24,0L24,24L0,24L0,0Z" style="fill:none;"/>
    <g transform="matrix(1.2,0,0,1.2,-2.4,-2.4)">
        <path d="M12,2C6.48,2 2,6.48 2,12C2,17.52 6.48,22 12,22C17.52,22 22,17.52 22,12C22,6.48 17.52,2 12,2ZM13,17L11,17L11,15L13,15L13,17ZM13,13L11,13L11,7L13,7L13,13Z" style="fill-rule:nonzero;"/>
    </g>
</svg>

    </div>
    <div class="admonition-content"><p><strong>Note:</strong> If you want to explore local variables, you should rebuild your
program without optimizations and inlining (see the <a href="https://github.com/go-delve/delve/blob/master/Documentation/usage/dlv_exec.md"><code>dlv exec</code>
docs</a>):</p>
<pre tabindex="0"><code>% go install -gcflags=all=&#34;-N -l&#34; ./cmd/...
</code></pre></div>
  </div>
</aside>

<p>While you can run a new child process in a debugger (use <code>dlv exec</code>) without any
special permissions, attaching existing processes in a debugger is <a href="https://www.kernel.org/doc/Documentation/security/Yama.txt">disabled by
default in Linux</a>
for security reasons. We can allow this feature (remember to turn it off later!)
using:</p>
<pre tabindex="0"><code>% sudo sysctl -w kernel.yama.ptrace_scope=0
kernel.yama.ptrace_scope = 0
</code></pre><p>…and then we can just <code>dlv attach</code> to the hanging <code>gokr-rsync</code> process:</p>
<pre tabindex="0"><code>% dlv attach $(pidof gokr-rsync)
Type &#39;help&#39; for list of commands.
(dlv)
</code></pre><p>Great. But if we just print a stack trace, we only see functions from the
<code>runtime</code> package:</p>
<pre tabindex="0"><code>(dlv) bt
0  0x000000000047bb83 in runtime.futex
   at /home/michael/sdk/go1.23.6/src/runtime/sys_linux_amd64.s:558
1  0x00000000004374d0 in runtime.futexsleep
   at /home/michael/sdk/go1.23.6/src/runtime/os_linux.go:69
2  0x000000000040d89d in runtime.notesleep
   at /home/michael/sdk/go1.23.6/src/runtime/lock_futex.go:170
3  0x000000000044123e in runtime.mPark
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:1866
4  0x000000000044290d in runtime.stopm
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:2886
5  0x00000000004433d0 in runtime.findRunnable
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:3623
6  0x0000000000444e1d in runtime.schedule
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:3996
7  0x00000000004451cb in runtime.park_m
   at /home/michael/sdk/go1.23.6/src/runtime/proc.go:4103
8  0x0000000000477eee in runtime.mcall
   at /home/michael/sdk/go1.23.6/src/runtime/asm_amd64.s:459
</code></pre><p>The reason is that no goroutine is running (the program is waiting indefinitely
to receive data from the server), so we see one of the OS threads waiting in the
Go scheduler.</p>
<p>We first need to switch to the goroutine we are interested in (<code>grs</code> prints all
goroutines), and then the stack trace looks like what we expect:</p>
<pre tabindex="0"><code>(dlv) gr 1
Switched from 0 to 1 (thread 414327)
(dlv) bt
 0  0x0000000000474ebc in runtime.gopark
    at /home/michael/sdk/go1.23.6/src/runtime/proc.go:425
 1  0x000000000043819e in runtime.netpollblock
    at /home/michael/sdk/go1.23.6/src/runtime/netpoll.go:575
 2  0x000000000047435c in internal/poll.runtime_pollWait
    at /home/michael/sdk/go1.23.6/src/runtime/netpoll.go:351
 3  0x00000000004ed15a in internal/poll.(*pollDesc).wait
    at /home/michael/sdk/go1.23.6/src/internal/poll/fd_poll_runtime.go:84
 4  0x00000000004ed1f1 in internal/poll.(*pollDesc).waitRead
    at /home/michael/sdk/go1.23.6/src/internal/poll/fd_poll_runtime.go:89
 5  0x00000000004ee351 in internal/poll.(*FD).Read
    at /home/michael/sdk/go1.23.6/src/internal/poll/fd_unix.go:165
 6  0x0000000000569bb3 in net.(*netFD).Read
    at /home/michael/sdk/go1.23.6/src/net/fd_posix.go:55
 7  0x000000000057a025 in net.(*conn).Read
    at /home/michael/sdk/go1.23.6/src/net/net.go:189
 8  0x000000000058fcc5 in net.(*TCPConn).Read
    at &lt;autogenerated&gt;:1
 9  0x00000000004b72e8 in io.ReadAtLeast
    at /home/michael/sdk/go1.23.6/src/io/io.go:335
10  0x00000000004b74d3 in io.ReadFull
    at /home/michael/sdk/go1.23.6/src/io/io.go:354
11  0x0000000000598d5f in encoding/binary.Read
    at /home/michael/sdk/go1.23.6/src/encoding/binary/binary.go:244
12  0x00000000005a0b7a in github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).ReadMsg
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:50
13  0x00000000005a0f17 in github.com/gokrazy/rsync/internal/rsyncwire.(*MultiplexReader).Read
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:72
14  0x0000000000528de8 in bufio.(*Reader).Read
    at /home/michael/sdk/go1.23.6/src/bufio/bufio.go:241
15  0x00000000004b72e8 in io.ReadAtLeast
    at /home/michael/sdk/go1.23.6/src/io/io.go:335
16  0x00000000004b74d3 in io.ReadFull
    at /home/michael/sdk/go1.23.6/src/io/io.go:354
17  0x00000000005a19ef in github.com/gokrazy/rsync/internal/rsyncwire.(*Conn).ReadInt32
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:163
18  0x00000000005b77d2 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).recvIdMapping1
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:16
19  0x00000000005b7ea8 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).RecvIdList
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:52
20  0x00000000005b18db in github.com/gokrazy/rsync/internal/receiver.(*Transfer).ReceiveFileList
    at /home/michael/kr/rsync/internal/receiver/flist.go:229
21  0x0000000000605390 in github.com/gokrazy/rsync/internal/receivermaincmd.clientRun
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:341
22  0x00000000005fe572 in github.com/gokrazy/rsync/internal/receivermaincmd.socketClient
    at /home/michael/kr/rsync/internal/receivermaincmd/clientserver.go:44
23  0x0000000000602f10 in github.com/gokrazy/rsync/internal/receivermaincmd.rsyncMain
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:160
24  0x0000000000605e7e in github.com/gokrazy/rsync/internal/receivermaincmd.Main
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:394
25  0x0000000000606653 in main.main
    at /home/michael/kr/rsync/cmd/gokr-rsync/rsync.go:12
26  0x000000000043fa47 in runtime.main
    at /home/michael/sdk/go1.23.6/src/runtime/proc.go:272
27  0x000000000047bd01 in runtime.goexit
    at /home/michael/sdk/go1.23.6/src/runtime/asm_amd64.s:1700
</code></pre><h2 id="save-core-dump">Tip 3: Save a core dump for later</h2>
<p>If you don’t have time to poke around in the debugger now, you can save a core
dump for later.</p>
<aside class="admonition note">
  <div class="note-container">
    <div class="note-icon" style="width: 20px; height: 20px">
      <svg id="exclamation-icon" width="100%" height="100%" viewBox="0 0 24 24" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:1.41421;">
    <path d="M0,0L24,0L24,24L0,24L0,0Z" style="fill:none;"/>
    <g transform="matrix(1.2,0,0,1.2,-2.4,-2.4)">
        <path d="M12,2C6.48,2 2,6.48 2,12C2,17.52 6.48,22 12,22C17.52,22 22,17.52 22,12C22,6.48 17.52,2 12,2ZM13,17L11,17L11,15L13,15L13,17ZM13,13L11,13L11,7L13,7L13,13Z" style="fill-rule:nonzero;"/>
    </g>
</svg>

    </div>
    <div class="admonition-content"><strong>Tip:</strong> Check out my <a href="/posts/2024-10-22-debug-go-core-dumps-delve-export-bytes/">debugging Go core dumps with
delve</a> blog post from
2024 for more details! This section just explains how to collect core dumps.</div>
  </div>
</aside>

<p>In addition to printing the stack trace on <code>SIGQUIT</code>, we can make the Go runtime
crash the program, which in turn makes the Linux kernel write a core dump, by
running our program with the environment variable
<a href="https://pkg.go.dev/runtime"><code>GOTRACEBACK=crash</code></a>.</p>
<p>Modern Linux systems typically include <a href="https://manpages.debian.org/systemd-coredump.8"><code>systemd-coredump(8)</code></a>
 (but you might need to explicitly install it, for example on
Ubuntu) to collect core dumps (and remove old ones). You can use <a href="https://manpages.debian.org/coredumpctl.1"><code>coredumpctl(1)</code></a>
 to list and work with them. On macOS,
<a href="https://developer.apple.com/forums/thread/694233#695943022">collecting cores is more
involved</a>. I don’t
know about Windows.</p>
<p>In case your Linux system does not use <code>systemd-coredump</code>, you can use <code>ulimit -c unlimited</code> and set the kernel’s <code>kernel.core_pattern</code> sysctl setting. You can
find more details and options in the <a href="https://go.dev/wiki/CoreDumpDebugging">CoreDumpDebugging page of the Go
wiki</a>. For this article, we will stick to
<code>coredumpctl</code>:</p>
<pre tabindex="0"><code>% GOTRACEBACK=crash gokr-rsync -rtO --delete rsync://rsync.paas.rpki.ripe.net/repo/ /tmp/rpki-repo
[…]
^\SIGQUIT: quit
[…]
zsh: IOT instruction (core dumped)  GOTRACEBACK=crash gokr-rsync -rtO […]
</code></pre><p>The last line is what we want to see: it should say “core dumped”.</p>
<p>This core should now show up in <a href="https://manpages.debian.org/coredumpctl.1"><code>coredumpctl(1)</code></a>
:</p>
<pre tabindex="0"><code>% coredumpctl info
           PID: 414607 (gokr-rsync)
           UID: 1000 (michael)
           GID: 1000 (michael)
        Signal: 6 (ABRT)
     Timestamp: Sat 2025-02-08 10:18:27 CET (12s ago)
  Command Line: gokr-rsync -rtO --delete rsync://rsync.paas.rpki.ripe.net/repo/ /tmp/rpki-repo
    Executable: /bin/gokr-rsync
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (michael)
       Boot ID: 6158dd3b52af4b8384c103a8a336fc02
    Machine ID: ecb5a44f1a5846ad871566e113bf8937
      Hostname: midna
       Storage: /var/lib/systemd/coredump/core.gokr-rsync.1000.6158dd3b52af4b8384c103a8a336fc02.414607.1739006307000000.zst (present)
  Size on Disk: 158.3K
       Message: Process 414607 (gokr-rsync) of user 1000 dumped core.
                
    Module [dso] without build-id.
    Module [dso]
    Stack trace of thread 1604447:
    #0  0x0000000000475a41 runtime.raise.abi0 (/bin/gokr-rsync + 0x75a41)
    #1  0x0000000000451d85 runtime.dieFromSignal (/bin/gokr-rsync + 0x51d85)
    #2  0x00000000004522e6 runtime.sigfwdgo (/bin/gokr-rsync + 0x522e6)
    #3  0x0000000000450c45 runtime.sigtrampgo (/bin/gokr-rsync + 0x50c45)
    #4  0x0000000000475d26 runtime.sigtramp.abi0 (/bin/gokr-rsync + 0x75d26)
    #5  0x0000000000475e20 n/a (/bin/gokr-rsync + 0x75e20)
    ELF object binary architecture: AMD x86-64
</code></pre><p>If you see only hexadecimal addresses followed by <code>n/a (n/a + 0x0)</code>, that means
<code>systemd-coredump</code> could not symbolize (= resolve addresses to function names)
your core dump. Here are a few possible reasons for missing symbolization:</p>
<ul>
<li>Linux 6.12 and 6.13 <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=32713">produced core dumps that elfutils cannot
symbolize</a>. <code>systemd-coredump</code>
uses elfutils for symbolization, so avoid 6.12/6.13 in favor of using 6.14 or
newer.</li>
<li>With systemd v234-v256, <code>systemd-coredump</code> did not have permission to look
into programs living in the <code>/home</code> directory (fixed with <a href="https://github.com/systemd/systemd/commit/4ac1755be2d6c141fae7e57c42936e507c5b54e3">commit
<code>4ac1755</code></a>
in systemd v257+).
<ul>
<li>Similarly, <code>systemd-coredump</code> runs with
<a href="http://manpages.debian.org/systemd.exec"><code>PrivateTmp=yes</code></a>, meaning it
won’t be able to access programs you place in <code>/tmp</code>.</li>
</ul>
</li>
<li>Go builds with debug symbols by default, but maybe you are explicitly
stripping debug symbols in your build, by building with <code>-ldflags=-w</code>?</li>
</ul>
<p>We can now use <a href="https://manpages.debian.org/coredumpctl.1"><code>coredumpctl(1)</code></a>
 to launch delve for
this program + core dump:</p>
<pre tabindex="0"><code>% coredumpctl debug --debugger=dlv --debugger-arguments=core
[…]
Type &#39;help&#39; for list of commands.
(dlv) gr 1
Switched from 0 to 1 (thread 414607)
(dlv) bt
[…]
16  0x00000000004b74d3 in io.ReadFull
    at /home/michael/sdk/go1.23.6/src/io/io.go:354
17  0x00000000005a19ef in github.com/gokrazy/rsync/internal/rsyncwire.(*Conn).ReadInt32
    at /home/michael/kr/rsync/internal/rsyncwire/wire.go:163
18  0x00000000005b77d2 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).recvIdMapping1
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:16
19  0x00000000005b7ea8 in github.com/gokrazy/rsync/internal/receiver.(*Transfer).RecvIdList
    at /home/michael/kr/rsync/internal/receiver/uidlist.go:52
20  0x00000000005b18db in github.com/gokrazy/rsync/internal/receiver.(*Transfer).ReceiveFileList
    at /home/michael/kr/rsync/internal/receiver/flist.go:229
21  0x0000000000605390 in github.com/gokrazy/rsync/internal/receivermaincmd.clientRun
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:341
22  0x00000000005fe572 in github.com/gokrazy/rsync/internal/receivermaincmd.socketClient
    at /home/michael/kr/rsync/internal/receivermaincmd/clientserver.go:44
23  0x0000000000602f10 in github.com/gokrazy/rsync/internal/receivermaincmd.rsyncMain
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:160
24  0x0000000000605e7e in github.com/gokrazy/rsync/internal/receivermaincmd.Main
    at /home/michael/kr/rsync/internal/receivermaincmd/receivermaincmd.go:394
25  0x0000000000606653 in main.main
    at /home/michael/kr/rsync/cmd/gokr-rsync/rsync.go:12
26  0x000000000043fa47 in runtime.main
    at /home/michael/sdk/go1.23.6/src/runtime/proc.go:272
27  0x000000000047bd01 in runtime.goexit
    at /home/michael/sdk/go1.23.6/src/runtime/asm_amd64.s:1700
</code></pre><h2 id="conclusion">Conclusion</h2>
<p>In my experience, in the medium to long term, it always pays off to set up your
environment such that you can debug your programs conveniently. I strongly
encourage every programmer (and even users!) to invest time into your
development and debugging setup.</p>
<p>Luckily, Go comes with stack printing functionality by default (just press
<code>Ctrl+\</code>) and we can easily get a core dump out of our Go programs by running
them with <code>GOTRACEBACK=crash</code> — provided the system is set up to collect core
dumps.</p>
<p>Together with the delve debugger, this gives us all we need to effectively and
efficiently diagnose problems in Go programs.</p>
]]></content>
  </entry>
  <entry>
    <title type="html"><![CDATA[rsync, article 3: How does rsync work?]]></title>
    <link href="https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/"/>
    <id>https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/</id>
    <published>2022-07-02T14:30:10+02:00</published>
    <content type="html"><![CDATA[<p>This post is the third article in a series of blog posts about rsync, <a href="../2022-06-18-rsync-overview/">see the
Series Overview</a>.</p>
<p>With rsync up and running, it’s time to take a peek under the hood of rsync to
better understand how it works.</p>
<h2 id="howdoesrsyncwork">How does rsync work?</h2>
<p>When talking about the rsync protocol, we need to distinguish between:</p>
<ul>
<li>protocol-level roles: “sender” and “receiver”</li>
<li>TCP roles: “client” and “server”</li>
</ul>
<p>All roles can be mixed and matched: both rsync clients (or servers!) can
either send or receive.</p>
<p>Now that you know the terminology, let’s take a high-level look at the rsync
protocol. We’ll look at protocol version 27, which is older but simpler, and
which is the most widely supported protocol version, implemented by <code>openrsync</code>
and other third-party implementations:</p>
<p><img src="2022-05-29-rsync-logical-view.svg" alt="rsync architecture"></p>
<p>The rsync protocol can be divided into two phases:</p>
<ol>
<li>
<p>In the first phase, the sender walks the local file tree to generate and send
the <em>file list</em> to the receiver. The file list must be transferred in full,
because both sides sort it by filename (later rsync protocol versions
eliminate this synchronous sorting step).</p>
</li>
<li>
<p>In the second phase, concurrently:</p>
<ul>
<li>The receiver compares and requests each file in the file list. The
receiver requests the full file when it didn’t exist on disk yet, or it
will send checksums for the rsync <em>hash search</em> algorithm when the file
already existed.</li>
<li>The receiver receives file data from the sender. The sender answers the
requests with just enough data to reconstruct the current file contents
based on what’s already on the receiver.</li>
</ul>
</li>
</ol>
<p>The architecture makes it easy to implement the second phase in 3 separate
processes, each of which sending to the network as fast as possible using heavy
pipelining. This results in utilizing the available hardware resources (I/O,
CPU, network) on sender and receiver to the fullest.</p>
<h3 id="observing-rsyncs-transfer-phases">Observing rsync’s transfer phases</h3>
<p>When starting an rsync transfer, looking at the resource usage of both
machines allows us to confirm our understanding of the rsync architecture, and
to pin-point any bottlenecks:</p>















<a href="https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/2022-05-29-rsync-exo1-backup.jpg"><img
  srcset="https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/2022-05-29-rsync-exo1-backup_hu_a9d9f70fb5fe8625.jpg 2x,https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/2022-05-29-rsync-exo1-backup_hu_28af2a145a98ddb.jpg 3x"
  src="https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/2022-05-29-rsync-exo1-backup_hu_d49b3d3aab60f55f.jpg"
  
  width="600"
  height="331"
  style="

border: 1px solid #000;

"
  
  loading="lazy"></a>



<ol>
<li>phase: The rsync sender needs 17 seconds to walk the file system and send
the file list. The rsync receiver reads from the network and writes into RAM
during that time.
<ul>
<li>This phase is random I/O (querying file system metadata) for the sender.</li>
</ul>
</li>
<li>phase: Afterwards, the rsync sender reads from disk and sends to the
network. The rsync receiver receives from the network and writes to disk.
<ul>
<li>The receiver does roughly the same amount of random I/O as the sender did
in phase 1, as it needs to create directories and request missing files.</li>
<li>The sender does sequential disk reads and possibly checksum calculation, if
the file(s) existed on the receiver side.</li>
</ul>
</li>
</ol>
<p>(Again, the above was captured using rsync protocol version 27, later rsync
protocol versions don’t synchronize after completing phase 1, but instead
interleave the phases more.)</p>
<h3 id="rsync-hash-search">rsync hash search</h3>
<p>Up until now, we have described the rsync protocol at a high level. Let’s zoom
into the hash search step, which is what many people might associate with the
term “rsync algorithm”.</p>
<p>When a file exists on both sides, rsync sender and receiver, the receiver first
divides the file into blocks. The block size is a rounded square root of the
file’s length. The receiver then sends the checksums of all blocks to the
sender. In response, the sender finds matching blocks in the file and sends only
the data needed to reconstruct the file on the receiver side.</p>
<p>Specifically, the sender goes through <strong>each byte</strong> of the file and tries to
match existing receiver content. To make this less computationally expensive,
rsync combines two checksums.</p>
<p>rsync first calculates what it calls the
<a href="https://github.com/WayneD/rsync/blob/0ac7ebceef70417355f25daf9e2fd94e84c49749/checksum.c#L182-L203">“sum1”</a>,
or “fast signature”. This is a small checksum (two <code>uint16</code>) that can be
calculated with minimal effort for a rolling window over the file data. tridge
rsync comes with <a href="https://github.com/WayneD/rsync/blob/v3.2.4/simd-checksum-x86_64.cpp">SIMD
implementations</a>
to further speed this up where possible.</p>
<p>Only if the sum1 matches will
<a href="https://github.com/WayneD/rsync/blob/0ac7ebceef70417355f25daf9e2fd94e84c49749/checksum.c#L205-L300">“sum2”</a>
(or “strong signature”) be calculated, a 16-byte MD4 hash. Newer protocol
versions allow negotiating the hash algorithm and support the much faster xxhash
algorithms.</p>
<p>If sum2 matches, the block is considered equal on both sides.</p>
<p>Hence, the best case for rsync is when a file has either not changed at all, or
shares as many full blocks of content as possible with the old contents.</p>
<h2 id="changing-data-sets">Changing data sets</h2>
<p>Now that we know how rsync works on the file level, let’s take a step back to
the data set level.</p>
<p>The easiest situation is when you transfer a data set that is not currently
changing. But what happens when the data set changes while your rsync transfer
is running? Here are two examples.</p>
<p><a href="https://github.com/Debian/debiman">debiman</a>, the manpage generator powering
<a href="https://manpages.debian.org/">manpages.debian.org</a> is running on a Debian VM on
which an rsync job periodically transfers the static manpage archive to
different static web servers across the world. The rsync job and debiman are
not sequenced in any way. Instead, debiman is careful to only ever <a href="https://github.com/Debian/debiman/blob/79b8abe8573daaaf15e4cf40e85daf309e00d454/internal/write/atomically.go#L23">atomically
swap out
files</a>
in its output directory, or add new files before it swaps out an updated index.</p>
<p>The second example, the <a href="https://en.wikipedia.org/wiki/PostgreSQL">PostgreSQL</a>
database management system, is the opposite situation: instead of having full
control over how files are laid out, here I don’t have control over how files
are written (this generalizes to any situation where the model of only ever
replacing files is not feasible). The data files which my Postgres installation
keeps on disk are not great to synchronize using rsync: they are large and
frequently change. Instead, I now exempt them from my rsync transfer and use <a href="https://manpages.debian.org/pg_dump.1"><code>pg_dump(1)</code></a>
 to create a snapshot of my databases instead.</p>
<p>To confirm rsync’s behavior regarding changing data sets in detail, I <a href="#confirmation-hack">modified
rsync to ask for confirmation</a> between generating the file
list and transferring the files. Here’s what I found:</p>
<ul>
<li>If files are added after rsync has transferred the file list, the new files
will just not be part of the transfer.</li>
<li>If a file vanishes between generating the file list and transfering the file,
rsync exits with status code 24, which <a href="https://manpages.debian.org/bullseye/rsync/rsync.1.en.html#EXIT_VALUES">its manpage documents as “Partial
transfer due to vanished source
files”</a>. My
<code>rsyncprom</code> monitoring wrapper <a href="https://github.com/stapelberg/rsyncprom/commit/bbf82934093b938d054d460a7f9764234d30bc17">offers a flag to treat exit code 24 like exit
code
0</a>,
because depending on the data set, vanishing files are expected.</li>
<li>If a file’s contents change (no matter whether the file grows, shrinks, or is
modified in-place) between generating the file list and the actual file
transfer, that’s not a problem — rsync will transfer the file contents as it
reads them once the transfer starts. Note that this might be an inconsistent
view of the data, depending on the application.
<ul>
<li>Ideally, don’t ever modify files within a data set that is rsynced. Instead,
atomically move complete files into the data set.</li>
</ul>
</li>
</ul>
<p>Another way of phrasing the above is that data consistency is not something that
rsync can in any way guarantee. It’s up to you to either live with the
inconsistency (often a good-enough strategy!), or to add an extra step that
ensures the data set you feed to rsync is consistent.</p>
<h2 id="next-up">Next up</h2>
<p>The fourth article in this series is rsync, article 4: My own rsync
implementation (To be published.)</p>
<h2 id="confirmation-hack">Appendix A: rsync confirmation hack</h2>
<p>For verifying rsync’s behavior with regards to changing data sets, I checked
out the following version:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>% git clone https://github.com/WayneD/rsync/ rsync-changing-data-sets
</span></span><span style="display:flex;"><span>% <span style="color:#007020">cd</span> rsync-changing-data-sets
</span></span><span style="display:flex;"><span>% git checkout v3.2.4
</span></span><span style="display:flex;"><span>% ./configure
</span></span><span style="display:flex;"><span>% make
</span></span></code></pre></div><p>Then, I modified <code>flist.c</code> to add a confirmation step between sending the file
list and doing the actual file transfers:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span><span style="color:#000080;font-weight:bold">diff --git i/flist.c w/flist.c
</span></span></span><span style="display:flex;"><span><span style="color:#000080;font-weight:bold">index 1ba306bc..98981f34 100644
</span></span></span><span style="display:flex;"><span><span style="color:#a00000">--- i/flist.c
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+++ w/flist.c
</span></span></span><span style="display:flex;"><span><span style="color:#800080;font-weight:bold">@@ -20,6 +20,8 @@
</span></span></span><span style="display:flex;"><span>  * with this program; if not, visit the http://fsf.org website.
</span></span><span style="display:flex;"><span>  */
</span></span><span style="display:flex;"><span> 
</span></span><span style="display:flex;"><span><span style="color:#00a000">+#include &lt;stdio.h&gt;
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+
</span></span></span><span style="display:flex;"><span> #include &#34;rsync.h&#34;
</span></span><span style="display:flex;"><span> #include &#34;ifuncs.h&#34;
</span></span><span style="display:flex;"><span> #include &#34;rounding.h&#34;
</span></span><span style="display:flex;"><span><span style="color:#800080;font-weight:bold">@@ -2516,6 +2518,17 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
</span></span></span><span style="display:flex;"><span> 	if (DEBUG_GTE(FLIST, 2))
</span></span><span style="display:flex;"><span> 		rprintf(FINFO, &#34;send_file_list done\n&#34;);
</span></span><span style="display:flex;"><span> 
</span></span><span style="display:flex;"><span><span style="color:#00a000">+	char *line = NULL;
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	size_t llen = 0;
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	ssize_t nread;
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	printf(&#34;file list sent. enter &#39;yes&#39; to continue: &#34;);
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	while ((nread = getline(&amp;line, &amp;llen, stdin)) != -1) {
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	  if (nread == strlen(&#34;yes\n&#34;) &amp;&amp; strcasecmp(line, &#34;yes\n&#34;) == 0) {
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	    break;
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	  }
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	  printf(&#34;enter &#39;yes&#39; to continue: &#34;);
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+	}
</span></span></span><span style="display:flex;"><span><span style="color:#00a000">+
</span></span></span><span style="display:flex;"><span> 	if (inc_recurse) {
</span></span><span style="display:flex;"><span> 		send_dir_depth = 1;
</span></span><span style="display:flex;"><span> 		add_dirs_to_tree(-1, flist, stats.num_dirs);
</span></span></code></pre></div><p>My rsync invocation is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>./rsync -av --debug<span style="color:#666">=</span>all4 --protocol<span style="color:#666">=</span><span style="color:#40a070">27</span> ~/i3/src /tmp/DEST/
</span></span></code></pre></div><p>It’s necessary to use an older protocol version to make rsync generate a full
file list before starting the transfer. Later protocol versions interleave these
parts of the protocol.</p>
]]></content>
  </entry>
  <entry>
    <title type="html"><![CDATA[rsync, article 2: Surroundings]]></title>
    <link href="https://michael.stapelberg.ch/posts/2022-07-02-rsync-surroundings/"/>
    <id>https://michael.stapelberg.ch/posts/2022-07-02-rsync-surroundings/</id>
    <published>2022-07-02T14:30:00+02:00</published>
    <content type="html"><![CDATA[<p>This post is the second article in a series of blog posts about rsync, <a href="../2022-06-18-rsync-overview/">see the
Series Overview</a>.</p>
<p>Now that we know what to use rsync for, how can we best integrate rsync into
monitoring and alerting, and on which operating systems does it work?</p>
<h2 id="monitoring-and-alerting-for-rsync-jobs-using-prometheus">Monitoring and alerting for rsync jobs using Prometheus</h2>
<p>Once you have one or two important <code>rsync</code> jobs, it might make sense to alert
when your job has not completed as expected.</p>
<p>I’m using <a href="https://prometheus.io/">Prometheus</a> for all my monitoring and alerting.</p>
<p>Because Prometheus <em>pulls</em> metrics from its (typically always-running) targets,
we need an extra component: the <a href="https://prometheus.io/docs/practices/pushing/">Prometheus
Pushgateway</a>. The Pushgateway
stores metrics pushed by short-lived jobs like <code>rsync</code> transfers and makes them
available to subsequent Prometheus pulls.</p>
<p>To integrate <code>rsync</code> with the Prometheus Pushgateway, I wrote
<a href="https://github.com/stapelberg/rsyncprom"><code>rsyncprom</code></a>, a small tool that wraps
<code>rsync</code>, or parses rsync output supplied by you. Once <code>rsync</code> completes,
<code>rsyncprom</code> pushes the rsync exit code and parsed statistics about the transfer
to your Pushgateway.</p>
<h3 id="prometheus-server-side-setup">Prometheus server-side setup</h3>
<p>First, I set up the Prometheus Pushgateway (via Docker and systemd) on my
server.</p>
<p>Then, in my <code>prometheus.conf</code> file, I instruct Prometheus to pull data from my
Pushgateway:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># prometheus.conf</span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#062873;font-weight:bold">rule_files</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span>- backups.rules.yml<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#062873;font-weight:bold">scrape_configs</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># […]</span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span>- <span style="color:#062873;font-weight:bold">job_name</span>:<span style="color:#bbb"> </span>pushgateway<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">  </span><span style="color:#062873;font-weight:bold">honor_labels</span>:<span style="color:#bbb"> </span><span style="color:#007020;font-weight:bold">true</span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">  </span><span style="color:#062873;font-weight:bold">static_configs</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">  </span>- <span style="color:#062873;font-weight:bold">targets</span>:<span style="color:#bbb"> </span>[<span style="color:#4070a0">&#39;pushgateway:9091&#39;</span>]<span style="color:#bbb">
</span></span></span></code></pre></div><p>Finally, in <code>backups.rules.yml</code>, I configure an alert on the time series <code>rsync_exit_code</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># backups.rules.yml</span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#062873;font-weight:bold">groups</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span>- <span style="color:#062873;font-weight:bold">name</span>:<span style="color:#bbb"> </span>backups.rules<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">  </span><span style="color:#062873;font-weight:bold">rules</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">  </span>- <span style="color:#062873;font-weight:bold">alert</span>:<span style="color:#bbb"> </span>RsyncFailing<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">    </span><span style="color:#062873;font-weight:bold">expr</span>:<span style="color:#bbb"> </span>rsync_exit_code{job=&#34;rsync&#34;} &gt; 0<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">    </span><span style="color:#062873;font-weight:bold">for</span>:<span style="color:#bbb"> </span>1m<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">    </span><span style="color:#062873;font-weight:bold">labels</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">      </span><span style="color:#062873;font-weight:bold">job</span>:<span style="color:#bbb"> </span>rsync<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">    </span><span style="color:#062873;font-weight:bold">annotations</span>:<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">      </span><span style="color:#062873;font-weight:bold">description</span>:<span style="color:#bbb"> </span>rsync {{ $labels.instance }} is failing<span style="color:#bbb">
</span></span></span><span style="display:flex;"><span><span style="color:#bbb">      </span><span style="color:#062873;font-weight:bold">summary</span>:<span style="color:#bbb"> </span>rsync {{ $labels.instance }} is failing<span style="color:#bbb">
</span></span></span></code></pre></div><p>This alert will fire any time an rsync job monitored via <code>rsyncprom</code> exits with
a non-zero exit code.</p>
<h3 id="rsync-client-side-setup">rsync client-side setup</h3>
<p>On each machine that runs <code>rsync</code> jobs I want to monitor, I first install
<code>rsyncprom</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>go install github.com/stapelberg/rsyncprom/cmd/rsync-prom@latest
</span></span></code></pre></div><p>Then, I just wrap <code>rsync</code> transfers where it’s most convenient, for example in
my <a href="https://manpages.debian.org/crontab.5"><code>crontab(5)</code></a>
:</p>
<pre tabindex="0"><code class="language-crontab" data-lang="crontab"># crontab -e
9 9 * * * /home/michael/go/bin/rsync-prom --job=&#34;cron&#34; --instance=&#34;gphotos-sync@midna&#34; -- /home/michael/gphotos-sync/sync.sh
</code></pre><p>The same wrapper technique works in <a href="https://github.com/stapelberg/rsyncprom#setup-example-systemd">shell scripts or systemd service
files</a>.</p>
<p>You can also provide <code>rsync</code> output <a href="https://github.com/stapelberg/zkj-nas-tools/blob/02d46d718df60c413844d9218f6dd702ad94e5f1/dornroeschen/sshutil.go#L134-L139">from Go
code</a>
(this example runs <code>rsync</code> via SSH).</p>
<h3 id="monitoring-architecture">Monitoring architecture</h3>
<p>Here’s how the whole setup looks like architecturally:</p>
<p><img src="2022-05-29-rsync-prom-architecture.svg" alt="rsync-prom architecture"></p>
<p>The rsync scheduler runs on a Raspberry Pi running
<a href="https://gokrazy.org/">gokrazy</a>. The scheduler invokes the <code>rsync</code> job to back
up <em>websrv.zekjur.net</em> via SSH and sends the output to Prometheus, which is
running on a (different) server at an ISP.</p>
<h3 id="monitoring-dashboard">Monitoring dashboard</h3>
<p>The grafana dashboard looks like this in action:</p>















<a href="https://michael.stapelberg.ch/posts/2022-07-02-rsync-surroundings/2022-06-12-rsync-grafana-featured.jpg"><img
  srcset="https://michael.stapelberg.ch/posts/2022-07-02-rsync-surroundings/2022-06-12-rsync-grafana-featured_hu_2db3563c2595b8dc.jpg 2x,https://michael.stapelberg.ch/posts/2022-07-02-rsync-surroundings/2022-06-12-rsync-grafana-featured_hu_109d709c8651f78a.jpg 3x"
  src="https://michael.stapelberg.ch/posts/2022-07-02-rsync-surroundings/2022-06-12-rsync-grafana-featured_hu_8dfdde166eaab1ad.jpg"
  alt="rsync grafana dashboard" title="rsync grafana dashboard"
  width="600"
  height="469"
  style="

border: 1px solid #000;

"
  
  loading="lazy"></a>



<ul>
<li>The top left table shows the most recent rsync exit code, green means 0 (success).</li>
<li>The top right graph shows rsync runtime (wall-clock time) over time. Long
runtime can have any number of bottlenecks as the reason: network connections,
storage devices, slow CPUs.</li>
<li>The bottom left graph shows rsync dataset size over time. This allows you to
quickly pinpoint transfers that are filling your disk up.</li>
<li>The bottom right graph shows transferred bytes per rsync over time. The higher
the value, the higher the amount of change in your data set between
synchronization runs.</li>
</ul>
<h2 id="rsync-operating-system-availability">rsync operating system availability</h2>
<p>Now that we have learnt about a couple of typical use-cases, where can you use
<code>rsync</code> to implement these use-cases? The answer is: in most environments, as
<code>rsync</code> is widely available on different Linux and BSD versions.</p>
<p>Macs come with <code>rsync</code> available by default (but it’s an old, patched version),
and OpenBSD comes with a BSD-licensed implementation called
<a href="https://www.openrsync.org/">openrsync</a> by default.</p>
<p>On Windows, you can use the <a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux">Windows Subsystem for
Linux</a>.</p>
<table>
	<thead>
			<tr>
					<th>Operating System</th>
					<th>Implementation</th>
					<th>Version</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>FreeBSD 13.1 (ports)</td>
					<td>tridge</td>
					<td><a href="https://www.freebsd.org/cgi/man.cgi?query=rsync&amp;manpath=FreeBSD+13.1-RELEASE+and+Ports">3.2.3</a></td>
			</tr>
			<tr>
					<td>OpenBSD 7.1</td>
					<td>openrsync</td>
					<td>(7.1)</td>
			</tr>
			<tr>
					<td>OpenBSD 7.1 (ports)</td>
					<td>tridge</td>
					<td><a href="https://openports.se/net/rsync">3.2.4</a></td>
			</tr>
			<tr>
					<td>NetBSD 9.2 (pkgsrc)</td>
					<td>tridge</td>
					<td><a href="https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/net/rsync/index.html">3.2.4</a></td>
			</tr>
			<tr>
					<td>Linux</td>
					<td>tridge</td>
					<td><a href="https://repology.org/project/rsync/versions">repology</a></td>
			</tr>
			<tr>
					<td>macOS</td>
					<td>tridge</td>
					<td><a href="https://www.unix.com/man-page/osx/1/rsync/">2.6.9</a></td>
			</tr>
	</tbody>
</table>
<h2 id="next-up">Next Up</h2>
<p>The third article in this series is <a href="../2022-07-02-rsync-how-does-it-work/">rsync, article 3: How does rsync
work?</a>. With rsync up and running, it’s
time to take a peek under the hood of rsync to better understand how it works.</p>
]]></content>
  </entry>
  <entry>
    <title type="html"><![CDATA[rsync, article 1: Scenarios]]></title>
    <link href="https://michael.stapelberg.ch/posts/2022-06-18-rsync-article-1-scenarios/"/>
    <id>https://michael.stapelberg.ch/posts/2022-06-18-rsync-article-1-scenarios/</id>
    <published>2022-06-18T15:04:00+02:00</published>
    <content type="html"><![CDATA[<p>This post is the first article in a series of blog posts about rsync, <a href="../2022-06-18-rsync-overview/">see the
Series Overview</a>.</p>
<p>To motivate why it makes sense to look at rsync, I present three scenarios for
which I have come to appreciate rsync: <a href="#dokuwiki-transfers-using-rsync">DokuWiki
transfers</a>, <a href="#software-deployment-using-rsync">Software
deployment</a> and
<a href="#backups-using-rsync">Backups</a>.</p>
<h2 id="dokuwiki-transfers-using-rsync">Scenario: DokuWiki transfers using rsync</h2>
<p>Recently, I set up a couple of tools for a website that is built on DokuWiki,
such as a dead link checker and a statistics program. To avoid overloading the
live website (and possibly causing spurious requests that interfere with
statistics), I decided it would be best to run a separate copy of the DokuWiki
installation locally. This requires synchronizing:</p>
<ol>
<li>The PHP source code files of DokuWiki itself (including plugins and configuration)</li>
<li>One text file per wiki page, and all uploaded media files</li>
</ol>
<p>A DokuWiki installation is exactly the kind of file tree that <a href="https://manpages.debian.org/scp.1"><code>scp(1)</code></a>
 cannot efficiently transfer (too many small files),
but <a href="https://manpages.debian.org/rsync.1"><code>rsync(1)</code></a>
 can! The <code>rsync</code> transfer only takes a few seconds, no matter if
it’s a full download (can be simpler for batch jobs) or an incremental
synchronization (more efficient for regular synchronizations like backups).</p>
<h2 id="software-deployment-using-rsync">Scenario: Software deployment using rsync</h2>
<p>For smaller projects where I don’t publish new versions through Docker, I
instead use a shell script to transfer and run my software on the server.</p>
<p><code>rsync</code> is a great fit here, as it transfers many small files (static assets and
templates) efficiently, only transfers the binaries that actually changed, and
doesn’t mind if the binary file it’s uploading is currently running (contrary to
<a href="https://manpages.debian.org/scp.1"><code>scp(1)</code></a>
, for example).</p>
<p>To illustrate how such a script could look like, here’s my push script for
<a href="https://codesearch.debian.net/">Debian Code Search</a>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f0f0f0;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#007020">#!/bin/zsh
</span></span></span><span style="display:flex;"><span><span style="color:#007020">set</span> -ex
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># Asynchronously transfer assets while compiling:</span>
</span></span><span style="display:flex;"><span><span style="color:#666">(</span>
</span></span><span style="display:flex;"><span>    ssh root@dcs <span style="color:#4070a0">&#39;for i in $(seq 0 5); do mkdir -p /srv/dcs/shard${i}/{src,idx}; done&#39;</span>
</span></span><span style="display:flex;"><span>    ssh root@dcs <span style="color:#4070a0">&#34;adduser --disabled-password --gecos &#39;Debian Code Search&#39; dcs || true&#34;</span>
</span></span><span style="display:flex;"><span>    rsync -r systemd/ root@dcs:/etc/systemd/system/ &amp;
</span></span><span style="display:flex;"><span>    rsync -r cmd/dcs-web/templates/ root@dcs:/srv/dcs/templates/ &amp;
</span></span><span style="display:flex;"><span>    rsync -r static/ root@dcs:/srv/dcs/static/ &amp;
</span></span><span style="display:flex;"><span>    <span style="color:#007020">wait</span>
</span></span><span style="display:flex;"><span><span style="color:#666">)</span> &amp;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># Compile a new Debian Code Search version:</span>
</span></span><span style="display:flex;"><span><span style="color:#bb60d5">tmp</span><span style="color:#666">=</span><span style="color:#007020;font-weight:bold">$(</span>mktemp -d<span style="color:#007020;font-weight:bold">)</span>
</span></span><span style="display:flex;"><span>mkdir <span style="color:#bb60d5">$tmp</span>/bin
</span></span><span style="display:flex;"><span><span style="color:#bb60d5">GOBIN</span><span style="color:#666">=</span><span style="color:#bb60d5">$tmp</span>/bin <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span><span style="color:#bb60d5">GOAMD64</span><span style="color:#666">=</span>v3 <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  go install <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  -ldflags <span style="color:#4070a0">&#39;-X github.com/Debian/dcs/cmd/dcs-web/common.Version=$version&#39;</span> <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  github.com/Debian/dcs/cmd/...
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># Transfer the Debian Code Search binaries:</span>
</span></span><span style="display:flex;"><span>rsync <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  <span style="color:#bb60d5">$tmp</span>/bin/dcs-<span style="color:#666">{</span>web,source-backend,package-importer,compute-ranking,feeder<span style="color:#666">}</span> <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  <span style="color:#bb60d5">$tmp</span>/bin/dcs <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  root@dcs:/srv/dcs/bin/
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># Wait for the asynchronous asset transfer to complete:</span>
</span></span><span style="display:flex;"><span><span style="color:#007020">wait</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#60a0b0;font-style:italic"># Restart Debian Code Search on the server:</span>
</span></span><span style="display:flex;"><span><span style="color:#bb60d5">UNITS</span><span style="color:#666">=(</span>dcs-package-importer.service dcs-source-backend.service dcs-compute-ranking.timer dcs-web.service<span style="color:#666">)</span>
</span></span><span style="display:flex;"><span>ssh root@dcs systemctl daemon-reload <span style="color:#4070a0;font-weight:bold">\&amp;\&amp;</span> <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  systemctl <span style="color:#007020">enable</span> <span style="color:#70a0d0">${</span><span style="color:#bb60d5">UNITS</span><span style="color:#70a0d0">}</span> <span style="color:#4070a0;font-weight:bold">\;</span> <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  systemctl reset-failed <span style="color:#70a0d0">${</span><span style="color:#bb60d5">UNITS</span><span style="color:#70a0d0">}</span> <span style="color:#4070a0;font-weight:bold">\;</span> <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  systemctl restart <span style="color:#70a0d0">${</span><span style="color:#bb60d5">UNITS</span><span style="color:#70a0d0">}</span> <span style="color:#4070a0;font-weight:bold">\;</span> <span style="color:#4070a0;font-weight:bold">\
</span></span></span><span style="display:flex;"><span>  systemctl reload nginx
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>rm -rf <span style="color:#4070a0">&#34;</span><span style="color:#70a0d0">${</span><span style="color:#bb60d5">tmp</span>?<span style="color:#70a0d0">}</span><span style="color:#4070a0">&#34;</span>
</span></span></code></pre></div><h2 id="backups-using-rsync">Scenario: Backups using rsync</h2>
<p>The first backup system I used was
<a href="https://en.wikipedia.org/wiki/Bacula">bacula</a>, which Wikipedia describes as an
enterprise-level backup system. That certainly matches my impression, both in
positive and negative ways: while bacula is very powerful, some seemingly common
operations turn out quite complicated in bacula. Restoring a single file or
directory tree from a backup was always more effort than I thought
reasonable. For some reason, I often had to restore backup catalogs before I was
able to access the backup contents (I don’t remember the exact details).</p>
<p>When moving apartment last time, I used the opportunity to change my backup
strategy. Instead of using complicated custom software with its own volume file
format (like bacula), I wanted backed-up files to be usable on the file system
level with standard tools like <code>rm</code>, <code>ls</code>, <code>cp</code>, etc.</p>
<p>Working with files in a regular file system makes day-to-day usage easier, and
also ensures that when my network storage hardware dies, I can just plug the
hard disk into any PC, boot a Linux live system, and recover my data.</p>
<p>To back up machines onto my <a href="/posts/2019-10-23-nas/">network storage PC</a>’s file
system, I ended up with a <a href="https://github.com/stapelberg/zkj-nas-tools/blob/85e445a284c89590d595a52e16cb6dd652b1388e/dornroeschen/backup-remote.pl">hand-written rsync wrapper
script</a>
that copies the full file system of each machine into dated directory trees:</p>
<pre tabindex="0"><code>storage2# ls -l backup/midna/2022-05-27
bin   boot  etc  home  lib  lib64  media  opt
proc  root  run  sbin  sys  tmp    usr    var

storage2# ls -l backup/midna/2022-05-27/home/michael/configfiles/zshrc
-rw-r--r--. 7 1000 1000 14554 May  9 19:37 backup/midna/2022-05-27/home/michael/configfiles/zshrc
</code></pre><p>To revert my <code>~/.zshrc</code> to an older version, I can <a href="https://manpages.debian.org/scp.1"><code>scp(1)</code></a>
 the file:</p>
<pre tabindex="0"><code>midna% scp storage2:/srv/backup/midna/2022-05-27/home/michael/configfiles/zshrc ~/configfiles/zshrc
</code></pre><p>To compare a whole older source tree, I can mount it using <a href="https://manpages.debian.org/sshfs.1"><code>sshfs(1)</code></a>
:</p>
<pre tabindex="0"><code>midna% mkdir /tmp/2022-05-27-i3
midna% sshfs storage2:/srv/backup/midna/2022-05-27/$HOME/i3 /tmp/2022-05-27-i3
midna% diff -ur /tmp/2022-05-27-i3 ~/i3/
</code></pre><h3 id="incremental-backups">Incremental backups</h3>
<p>Of course, the idea is not to transfer the full machine contents every day, as
that would quickly fill up my network storage’s 16 TB disk! Instead, we can use
rsync’s <code>--link-dest</code> option to elegantly deduplicate files using file system
hard links:</p>
<pre tabindex="0"><code>backup/midna/2022-05-26
backup/midna/2022-05-27 # rsync --link-dest=2022-05-26
</code></pre><p>To check the de-duplication level, we can use <a href="https://manpages.debian.org/du.1"><code>du(1)</code></a>
,
first on a single directory:</p>
<pre tabindex="0"><code>storage2# du -hs 2022-05-27 
113G	2022-05-27
</code></pre><p>…and then on two subsequent directories:</p>
<pre tabindex="0"><code>storage2# du -hs 2022-05-25 2022-05-27
112G	2022-05-25
7.3G	2022-05-27
</code></pre><p>As you can see, the 2022-05-27 backup took 7.3 GB of disk space, and 104.7 GB
were re-used from the previous backup(s).</p>
<p>To print all files which have changed since the last backup, we can use:</p>
<pre tabindex="0"><code>storage2# find 2022-05-27 -type f -links 1 -print
</code></pre><h3 id="limitation-file-system-compatibility">Limitation: file system compatibility</h3>
<p>A significant limitation of backups at the file level is that the destination
file system (network storage) needs to support all the file system features used
on the machines you are backing up.</p>
<p>For example, if you use <a href="https://help.ubuntu.com/community/FilePermissionsACLs">POSIX
ACLs</a> or <a href="https://wiki.archlinux.org/title/File_permissions_and_attributes#Extended_attributes">Extended
attributes</a>
(possibly for <a href="https://wiki.archlinux.org/title/Capabilities">Capabilities</a> or
<a href="https://wiki.archlinux.org/title/SELinux">SELinux</a>), you need to ensure that
your backup file system has these features enabled, and that you are using <a href="https://manpages.debian.org/rsync.1"><code>rsync(1)</code></a>
’s <code>--xattrs</code> (or <code>-X</code> for short) option.</p>
<p>This can turn from a pitfall into a dealbreaker as soon as multiple operating
systems are involved. For example, the <code>rsync</code> version on macOS has
<a href="https://github.com/apple-oss-distributions/rsync/blob/aa4e500aa53b9417014c718a5ff0e29215f08e48/rsync/generator.c#L1447">Apple-specific
code</a>
to work with Apple <a href="https://en.wikipedia.org/wiki/Resource_fork">resource forks</a>
and other extended attributes. It’s not clear to me whether macOS <code>rsync</code> can
send files to Linux <code>rsync</code>, restore them, and end up with the same system state.</p>
<p>Luckily, I am only interested in backing up Linux systems, or merely home
directories of non-Linux systems, where no extended attributes are used.</p>
<h3 id="downside-slow-bulk-operations-disk-usage-deletion">Downside: slow bulk operations (disk usage, deletion)</h3>
<p>The biggest downside of this architecture is that working with the directory
trees in bulk can be very slow, especially when using a hard disk instead of an
SSD. For example, deleting old backups can easily take many hours to multiple
days (!). Sure, you can just let the <code>rm</code> command run in the background, but
it’s annoying nevertheless.</p>
<p>Even merely calculating the disk space usage of each directory tree is a
painfully slow operation. I tried using stateful disk usage tools like
<a href="http://duc.zevv.nl/">duc</a>, but it <a href="https://github.com/zevv/duc/issues/240">didn’t work
reliably</a> on my backups.</p>
<p>In practice, I found that for tracking down large files, using <a href="https://manpages.debian.org/ncdu.1"><code>ncdu(1)</code></a>
 on any recent backup typically quickly shows the
large file. In one case, I found <code>var/lib/postgresql</code> to consume many
gigabytes. I excluded it in favor of using <a href="https://manpages.debian.org/pg_dump.1"><code>pg_dump(1)</code></a>
, which resulted in much smaller backups!</p>
<p>Unfortunately, even when using an SSD, determining which files take up most
space of a full backup takes a few minutes:</p>
<pre tabindex="0"><code>storage2# time du -hs backup/midna/2022-06-09
742G	backup/midna/2022-06-09

real	8m0.202s
user	0m11.651s
sys	2m0.731s
</code></pre><h3 id="backup-transport-ssh-and-scheduling">Backup transport (SSH) and scheduling</h3>
<p>To transfer data via <code>rsync</code> from the backup host to my network storage, I’m
using SSH.</p>
<p>Each machine’s SSH access is restricted in my network storage’s SSH <a href="https://manpages.debian.org/authorized_keys.5"><code>authorized_keys(5)</code></a>
 config file to not allow arbitrary
commands, but to perform just a specific operation. The only allowed operation
in my case is running <code>rrsync</code> (“restricted rsync”) in a container whose file
system only contains the backup host’s sub directory, e.g. .<code>websrv.zekjur.net</code>:</p>
<pre tabindex="0"><code>command=&#34;/bin/docker run --log-driver none -i -e SSH_ORIGINAL_COMMAND -v /srv/backup/websrv.zekjur.net:/srv/backup/websrv.zekjur.net stapelberg/docker-rsync /srv/backup/websrv.zekjur.net&#34;,no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3…
</code></pre><p>(The <a href="/posts/2016-11-21-gigabit-nas-coreos/#dockerfiles-rrsync-and-samba">corresponding <code>Dockerfile</code> can be found in my Gigabit NAS
article</a>.)</p>
<p>To trigger such an SSH-protected <code>rsync</code> transfer remotely, I’m using a small
custom scheduling program called
<a href="https://github.com/stapelberg/zkj-nas-tools/tree/master/dornroeschen">dornröschen</a>. The
program arranges for all involved machines to be powered on (using
<a href="https://en.wikipedia.org/wiki/Wake-on-LAN">Wake-on-LAN</a>) and then starts
<code>rsync</code> via <em>another operation-restricted SSH connection</em>.</p>
<p>You could easily replace this with a cron job if you don’t care about WOL.</p>
<p>The architecture looks like this:</p>
<p><img src="2022-05-29-backup-architecture.svg" alt="backup architecture"></p>
<p>The operation-restricted SSH connection on each backup host is configured in
SSH’s <a href="https://manpages.debian.org/authorized_keys.5"><code>authorized_keys(5)</code></a>
 config file:</p>
<pre tabindex="0"><code>command=&#34;/root/backup-remote.pl&#34;,no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3…
</code></pre><h2 id="next-up">Next up</h2>
<p>The second article in this series is <a href="../2022-07-02-rsync-surroundings/">rsync, article 2:
Surroundings</a>. Now that we know what to use
rsync for, how can we best integrate rsync into monitoring and alerting, and on
which operating systems does it work?</p>
]]></content>
  </entry>
  <entry>
    <title type="html"><![CDATA[rsync: Series Overview]]></title>
    <link href="https://michael.stapelberg.ch/posts/2022-06-18-rsync-overview/"/>
    <id>https://michael.stapelberg.ch/posts/2022-06-18-rsync-overview/</id>
    <published>2022-06-18T15:00:00+02:00</published>
    <content type="html"><![CDATA[<p>For many years, I was only a casual user of
<a href="https://en.wikipedia.org/wiki/Rsync">rsync</a> and used it mostly for one-off file
transfers.</p>
<p>Over time, I found rsync useful in more and more cases, and would recommend
every computer user put this great tool into their toolbox 🛠 🧰 !</p>
<p>I’m publishing a series of blog posts about rsync:</p>
<ul>
<li><a href="../2022-06-18-rsync-article-1-scenarios/">rsync, article 1: Scenarios</a>. To
motivate why it makes sense to look at rsync, I present three scenarios for
which I have come to appreciate rsync: DokuWiki transfers, Software deployment
and Backups.</li>
<li><a href="../2022-07-02-rsync-surroundings/">rsync, article 2: Surroundings</a>. Now that
we know what to use rsync for, how can we best integrate rsync into monitoring
and alerting, and on which operating systems does it work?</li>
<li><a href="../2022-07-02-rsync-how-does-it-work">rsync, article 3: How does rsync
work?</a>. With rsync up and running, it’s
time to take a peek under the hood of rsync to better understand how it works.</li>
<li>rsync, article 4: My own rsync implementation (To be published.)</li>
</ul>
]]></content>
  </entry>
</feed>
