Using_strace_for_SDP_Services.html #1

<h1>Using strace with SDP-Managed Services</h1>

<h2>Overview</h2>

<p>The SDP init scripts <code>p4d_base</code>, <code>p4broker_base</code>, and <code>p4p_base</code> include
built-in support for running a service under <code>strace</code> at startup. This
provides a repeatable, low-friction way to capture syscall traces for
diagnostic purposes without modifying any init script or configuration file.</p>

<blockquote>
  <p><strong>Note:</strong> <code>strace -f</code> on a busy server generates large output volumes and
adds measurable overhead. This feature is intended for short-term diagnostic
use only — not for permanent production deployment.</p>
</blockquote>

<hr />

<h2>Two strace Modes</h2>

<p>The init scripts automatically select one of two modes based on whether the
service is managed by <strong>systemd</strong> or started <strong>directly</strong> (non-systemd).</p>

<h3>Wrap Mode (systemd — complete capture)</h3>

<p>When a service is managed by systemd, <code>strace</code> is interposed between the init
script and the service binary:</p>

<p><code>
systemd → p4d_N_init → strace ... p4d $P4D_FLAGS
</code></p>

<p><code>strace</code> runs the service binary as its own child process. Every syscall is
captured from the very first instruction — nothing from startup is missed.
This is ideal for diagnosing issues that occur during early startup (e.g.
replication thread initialization on a replica).</p>

<p><strong>Trade-off:</strong> Because <code>strace</code> is the parent process, the service cannot be
stopped independently from <code>strace</code>. Stop the service normally
(<code>systemctl stop p4d_N</code>) to terminate both.</p>

<h3>Attach Mode (non-systemd — best effort)</h3>

<p>When a service is started directly (without systemd), the init script starts
the service normally and then attaches <code>strace</code> using <code>strace -p &lt;pid&gt;</code> once
the service process ID is known.</p>

<p><strong>Trade-off:</strong> Syscalls that occur between process start and the moment the
PID is discovered (typically a second or less) are <strong>not captured</strong>. For
services where startup-phase syscalls are important, use systemd wrap mode.</p>

<p><code>strace</code> runs as a completely independent background process. It can be
stopped at any time without affecting the running service.</p>

<hr />

<h2>File Naming Convention</h2>

<p>All strace-related files live in <code>$LOGS</code> and incorporate the <strong>service name</strong>
(e.g. <code>p4d_1</code>, <code>p4broker_1</code>, <code>p4p_1</code>) so that multiple instances on the same
host never collide.</p>

<p>The strace output log also includes a <strong>timestamp</strong> (generated at the moment
strace starts) so successive runs do not overwrite each other.</p>

<p>| File | Purpose |
|------|---------|
| <code>$LOGS/.strace.&lt;ServiceName&gt;</code> | Marker file — create this to enable strace |
| <code>$LOGS/.strace.&lt;ServiceName&gt;.pid</code> | PID of the running strace process (attach mode only) |
| <code>$LOGS/strace.&lt;ServiceName&gt;.&lt;YYYYMMDD_HHMMSS&gt;.log</code> | strace output |</p>

<p>For p4d instance <code>1</code> on a host where <code>$LOGS</code> is <code>/p4/1/logs</code>:</p>

<p><code>
/p4/1/logs/.strace.p4d_1                       ← marker file (you create this)
/p4/1/logs/.strace.p4d_1.pid                   ← strace PID (attach mode only)
/p4/1/logs/strace.p4d_1.20260627_143022.log    ← strace output
</code></p>

<p>For a p4broker on the same instance:</p>

<p><code>
/p4/1/logs/.strace.p4broker_1
/p4/1/logs/.strace.p4broker_1.pid
/p4/1/logs/strace.p4broker_1.20260627_143055.log
</code></p>

<p>A p4broker started with a non-default config tag (e.g. <code>mytag</code>) uses
<code>p4broker_1_mytag</code> as the service name.</p>

<hr />

<h2>Enabling strace — Step by Step</h2>

<h3>p4d (Helix Core Server)</h3>

<p><strong>With systemd (wrap mode — recommended for startup diagnosis):</strong></p>

<p>```bash</p>

<h1>1. Create the marker file (empty = use default flags: -f -tt)</h1>

<p>touch /p4/N/logs/.strace.p4d_N</p>

<h1>2. (Re)start the service</h1>

<p>sudo systemctl restart p4d_N</p>

<h1>3. Confirm strace is running (the service log will show the strace log path)</h1>

<p>grep strace /p4/N/logs/p4d_init.log | tail -5
```</p>

<p><strong>Without systemd (attach mode):</strong></p>

<p>```bash
touch /p4/N/logs/.strace.p4d<em>N
/p4/N/bin/p4d</em>N_init start</p>

<h1>Confirm strace is running</h1>

<p>cat /p4/N/logs/.strace.p4d<em>N.pid
ps -p $(cat /p4/N/logs/.strace.p4d</em>N.pid)
```</p>

<h3>p4broker (Helix Broker)</h3>

<p>```bash
touch /p4/N/logs/.strace.p4broker_N</p>

<p>sudo systemctl restart p4broker_N   # systemd (wrap mode)</p>

<h1>or</h1>

<p>/p4/N/bin/p4broker<em>N</em>init start     # non-systemd (attach mode)
```</p>

<h3>p4p (Helix Proxy)</h3>

<p>```bash
touch /p4/N/logs/.strace.p4p_N</p>

<p>sudo systemctl restart p4p_N        # systemd (wrap mode)</p>

<h1>or</h1>

<p>/p4/N/bin/p4p<em>N</em>init start          # non-systemd (attach mode)
```</p>

<hr />

<h2>Stopping strace Without Affecting the Service</h2>

<p><strong>Attach mode only.</strong> Because <code>strace</code> runs as a separate background process
in attach mode, it can be killed independently at any time. The service
continues running normally.</p>

<p>```bash</p>

<h1>Generic form:</h1>

<p>kill $(cat /p4/N/logs/.strace.<ServiceName>.pid)</p>

<h1>Examples:</h1>

<p>kill $(cat /p4/1/logs/.strace.p4d<em>1.pid)
kill $(cat /p4/1/logs/.strace.p4broker</em>1.pid)
kill $(cat /p4/1/logs/.strace.p4p_1.pid)
```</p>

<p>The init script also calls <code>stop_strace_if_running</code> automatically when the
service is stopped via the init script, cleaning up the PID file.</p>

<blockquote>
  <p><strong>Note:</strong> In attach mode, <code>strace</code> also exits automatically when its target
process dies, so no explicit cleanup is needed if the service itself stops.</p>
</blockquote>

<p><strong>Wrap mode:</strong> <code>strace</code> is the parent process. The only way to stop it is to
stop the service normally (e.g. <code>systemctl stop p4d_N</code>). Killing strace
directly will also stop the service.</p>

<hr />

<h2>Preventing strace from Running on Next Restart</h2>

<p>Remove the marker file before restarting the service:</p>

<p><code>bash
rm /p4/N/logs/.strace.p4d_N
sudo systemctl restart p4d_N
</code></p>

<hr />

<h2>Custom strace Flags</h2>

<p>By default, strace is invoked with <code>-f -tt</code> (follow forks, include timestamps
with microsecond resolution). To use different flags, write them to the marker
file on a single line before starting the service:</p>

<p>```bash</p>

<h1>Trace only network-related syscalls:</h1>

<p>echo '-f -e trace=network' > /p4/1/logs/.strace.p4d_1</p>

<h1>Trace file-related syscalls with string lengths up to 256 bytes:</h1>

<p>echo '-f -s 256 -e trace=file' > /p4/1/logs/.strace.p4d_1</p>

<h1>Default (empty file):</h1>

<p>touch /p4/1/logs/.strace.p4d_1
```</p>

<p>The flags in the marker file <strong>replace</strong> the defaults entirely, so include
<code>-f</code> explicitly if you want fork-following.</p>

<hr />

<h2>Viewing the Output</h2>

<p>The exact log file path (including the timestamp) is recorded in the service
init log (<code>p4d_init.log</code>, <code>p4broker_init.log</code>, or <code>p4p_init.log</code>) at the
moment strace starts.</p>

<p>```bash</p>

<h1>Find the strace log path for the most recent run:</h1>

<p>grep 'strace' /p4/1/logs/p4d_init.log | grep 'output:'</p>

<h1>Follow the trace in real time (strace writes line-buffered with -tt):</h1>

<p>tail -f /p4/1/logs/strace.p4d<em>1.20260627</em>143022.log</p>

<h1>Count syscalls by name:</h1>

<p>sort /p4/1/logs/strace.p4d<em>1.20260627</em>143022.log | uniq -c | sort -rn | head -20
```</p>

<hr />

<h2>Implementation Notes</h2>

<h3>Shared Library</h3>

<p>The shared library <code>/p4/common/lib/strace.lib</code> is sourced by all three init
scripts. It provides three functions:</p>

<p>| Function | Purpose |
|----------|---------|
| <code>run_service_under_strace &lt;binary&gt; [args...]</code> | Wrap mode: run service as strace child |
| <code>attach_strace_if_requested &lt;pid_cmd&gt;</code> | Attach mode: attach strace to running service |
| <code>stop_strace_if_running</code> | Stop background strace and remove PID file |</p>

<h3>Mode Selection</h3>

<p>Mode is selected automatically based on <code>$UseSystemd</code>:</p>

<p><code>
$UseSystemd == 1  →  wrap mode   (run_service_under_strace)
$UseSystemd == 0  →  attach mode (attach_strace_if_requested ... &amp; disown)
</code></p>

<p><code>UseSystemd</code> is set to <code>1</code> when a systemd unit file for the service exists and
is active (detected at script startup via <code>systemctl</code>).</p>

<h3>ServiceName</h3>

<p>The <code>ServiceName</code> variable (e.g. <code>p4d_1</code>) is derived from the service binary
name (e.g. <code>${P4DBIN##*/}</code>) and is set unconditionally before any systemd
detection, so strace file naming works correctly on both systemd and
non-systemd hosts.</p>

<h3>Why Not Always Use Wrap Mode?</h3>

<p>In non-systemd operation, the service binary forks immediately and daemonizes.
Using <code>strace binary args</code> (wrap mode) in this case would cause strace to
block indefinitely tracking the daemon, holding the init script open. The
attach approach is better suited here: the init script exits normally, and
strace attaches in a separate background process.</p>

<p>With systemd, the init script is already meant to block (the service runs in
the foreground as <code>ExecStart</code>). Replacing <code>binary args</code> with
<code>strace ... binary args</code> is a natural fit — no additional complexity is
introduced, and systemd manages the <code>strace</code> process as the service.</p>