---
name: sdp-jenkins-build-check
description: Check the status of (or manually trigger) the SDP Docker/Podman Jenkins build for a given stream. Use whenever asked to run, re-run, or check the Docker/Podman/Jenkins test suite.
---

# SDP Jenkins Build Check / Trigger

Checks (and, if needed, manually triggers) the Jenkins build that runs the SDP
Docker/Podman test suite. See `SDPTestSuite-DockerPodman.md` (in `//test-install_sdp`'s
`ai_dev_support`) for background; this skill codifies the operational steps.

## Prerequisites

- SSH access to `workshop-jenkins.das.perforce.com` -- requires being on the Perforce VPN.
  If SSH times out, that's almost always a VPN-connectivity issue, not a Jenkins problem --
  confirm VPN before troubleshooting further.

## Relevant jobs

- `p4-sdp-build_dev` -- Streams-based, builds `//p4-sdp/dev`. As of 2026-09-02, this is the
  one to check/trigger for current mainline (Streams-era) SDP work.
- `p4-sdp-build_dev_rebrand` -- builds `//p4-sdp/dev_rebrand`, the now-retired predecessor
  to `dev` (Copy Up completed 2026-09-02, change 33409). Kept around for now but no longer
  the primary target; don't assume it still matters without checking `SessionLog-*.md`.
- `SDP-Dev-Build` -- Classic depot path (`//guest/perforce_software/sdp/dev/...`). Frozen/
  deprecated as of the Classic-to-Streams migration, but was briefly re-fixed and kept
  live in parallel during 2026-08. Check `SessionLog-*.md` for the current status before
  assuming it's still relevant.

Both jobs live under `/var/lib/jenkins/jobs/<job-name>/` on the Jenkins host.

## Each job builds a *virtual stream*, not the dev stream directly

The job's workspace `streamName` (in `config.xml`) is `//p4-sdp/build_<stream>` -- a
`virtual` stream whose parent is `//p4-sdp/<stream>` -- not `//p4-sdp/<stream>` itself. This
indirection means retargeting to a brand-new dev-cycle stream doesn't require inventing a
Jenkins config from scratch:

1. Create the virtual stream (parented off the new target):
   ```bash
   p4 stream -o //p4-sdp/build_<old-stream> | \
      sed -e 's@^Stream:.*@Stream:\t//p4-sdp/build_<new-stream>@' \
          -e 's@^Parent:.*@Parent:\t//p4-sdp/<new-stream>@' \
          -e 's@^Name:.*@Name:\tbuild_<new-stream>@' \
      | grep -v '^Update:\|^Access:' | p4 stream -i
   ```
2. Clone the job directory and swap the stream name/description in `config.xml`:
   ```bash
   ssh workshop-jenkins.das.perforce.com "
   sudo mkdir -p /var/lib/jenkins/jobs/p4-sdp-build_<new-stream>
   sudo cp /var/lib/jenkins/jobs/p4-sdp-build_<old-stream>/config.xml /var/lib/jenkins/jobs/p4-sdp-build_<new-stream>/config.xml
   sudo sed -i -e 's@//p4-sdp/build_<old-stream>@//p4-sdp/build_<new-stream>@g' /var/lib/jenkins/jobs/p4-sdp-build_<new-stream>/config.xml
   sudo chown -R jenkins:jenkins /var/lib/jenkins/jobs/p4-sdp-build_<new-stream>
   "
   ```
3. Make Jenkins pick up the new job directory (no restart needed, just a config reload):
   ```bash
   ssh workshop-jenkins.das.perforce.com "
   curl -s -c /tmp/jcookie.txt 'http://localhost:8080/crumbIssuer/api/json' -o /tmp/jcrumb.json
   CRUMB=\$(grep -o '\"crumb\":\"[^\"]*\"' /tmp/jcrumb.json | cut -d'\"' -f4)
   curl -s -b /tmp/jcookie.txt -X POST -H \"Jenkins-Crumb: \$CRUMB\" 'http://localhost:8080/reload' -o /dev/null -w '%{http_code}\n'
   "
   ```
   **Gotcha found 2026-09-02**: fetching the crumb and POSTing `/reload` as two separate
   `curl` calls without a shared cookie jar can 403 with "No valid crumb was included" even
   though the crumb value itself is correct -- Jenkins' CSRF protection here is
   session-tied, not just value-tied. Use `-c`/`-b` with the same cookie file across both
   calls (as above). A `200`/`302` means it worked; verify with
   `curl -s 'http://localhost:8080/job/p4-sdp-build_<new-stream>/api/json?tree=name,url'`.

## RAM headroom -- check before adding a new concurrently-pollable build

Confirmed 2026-09-02: this host has only ~3.6GB total RAM, with ~1GB genuinely free
(~2.3GB "available" counting reclaimable cache) -- `free -h` to recheck. Disk is not a
concern (174G free of 195G as of this writing). Each job polls SCM independently
(`H/10 * * * *`); nothing prevents two different jobs' Docker/Podman container builds from
running at the same time even though `concurrentBuild=false` only blocks a *single* job
from overlapping with itself. Before triggering a newly-added job for the first time (or
relying on its independent polling schedule long-term), confirm with the user whether to
request more RAM, rather than silently risking an OOM if both jobs' container fleets end up
running concurrently.

## Container crashes (`container state improper` / exit 137)

This host has repeatedly hit a rootless-podman/systemd container-startup crash --
first seen 2026-08-24, recurred 2026-09-02/03. Symptoms: `podman exec <container>
/p4/docker_entry.sh` either errors immediately with `Error: can only create exec
sessions on running containers: container state improper`, or runs and exits with
`137` (SIGKILL) with no useful output. Both are the same underlying container
dying right around startup; which exact error you see seems to depend on timing.

**Things that have NOT reliably fixed it** (tried 2026-09-02, all ineffective that
time even though some had fixed it before):
- A full podman wipe for the `jenkins` user (`podman rm -f $(podman ps -aq)`;
  `podman rmi -a -f`; `podman system prune -a -f --volumes`).
- A full host reboot (`sudo reboot`) -- freed RAM from ~1GB to ~2.5GB free but
  the crash reproduced identically right after.
- Checking for a cgroup-delegation regression -- confirmed still correctly in
  place (`/etc/systemd/system/user@.service.d/delegate.conf`,
  `Delegate=cpu cpuset io memory pids`); not the cause this time.
- Checking dmesg/journalctl for OOM kills -- found none, so despite the exit-137
  signature this isn't necessarily a real OOM event.

**What actually fixed it, per Robert Cowham 2026-09-03**: don't wipe everything --
find and remove the *specific* stale/corrupted image IDs, then clear containers,
then let images rebuild fresh:
```bash
podman images                              # find the suspect image ID(s)
podman rmi <bad-image-id> [<bad-image-id2>...]
podman kill $(podman ps -aq)
podman rm $(podman ps -aq)
podman rmi <bad-image-id> [<bad-image-id2>...]   # again, in case a kill/rm regenerated them
podman images                              # confirm clean
```
Robert then verified by running the test script **directly** (bypassing Jenkins
entirely) as a faster diagnostic loop:
```bash
sudo -u jenkins -H bash -c 'cd /var/lib/jenkins/workspace/<job-name> && sdp/test/run_docker_tests.sh'
```
Only after that came back clean did he consider it fixed -- worth doing this
direct-script check yourself before/instead of burning a full Jenkins build cycle
when iterating on a fix.

**Known-harmless noise to expect in an otherwise-passing run**: the "Testing the
SDP Upgrade Procedure" step prints `gzip: stdin: not in gzip format` /
`tar: Child returned status 1` -- this is the pre-existing, already-known
`sdp.Unix.tgz` 404/gzip-format quirk from 2026-08-24, not a new failure. The real
signal is the final `SDP Upgrade OK` / `SDP Post-Upgrade Verify Completed OK` /
`Post-upgrade Test Message is as expected: Life is Good` lines after it.

**Not fully understood**: why this recurs, why the fixes that worked once don't
reliably work the next time, and why the specific bad image IDs vary. Treat any
"fix" here as provisional, not a permanent resolution -- re-check with
`podman images`/`podman ps -a` if it recurs again rather than assuming a new root
cause.

## Checking the latest build

```bash
ssh workshop-jenkins.das.perforce.com "
LAST=\$(ls -t /var/lib/jenkins/jobs/<job-name>/builds/ | grep -E '^[0-9]+\$' | head -1)
echo LAST_BUILD=\$LAST
grep -m1 '<result>' /var/lib/jenkins/jobs/<job-name>/builds/\$LAST/build.xml
grep -m1 '<timestamp>' /var/lib/jenkins/jobs/<job-name>/builds/\$LAST/build.xml
grep -o '3[0-9][0-9][0-9][0-9]' /var/lib/jenkins/jobs/<job-name>/builds/\$LAST/build.xml | sort -u
"
```

The last `grep` pulls any 5-digit changelist numbers referenced in the build record, so you
can confirm whether a specific changelist you care about was actually included in that
build (Perforce changelist numbers in this depot are currently in the 33xxx range -- adjust
the pattern if that range has moved on).

`<timestamp>` is milliseconds since the Unix epoch; convert with e.g.
`python3 -c "import datetime; print(datetime.datetime.fromtimestamp(<ms>/1000))"`.

## Is it already going to build on its own?

Check the polling schedule before manually triggering anything:

```bash
ssh workshop-jenkins.das.perforce.com "grep -A2 SCMTrigger /var/lib/jenkins/jobs/<job-name>/config.xml"
```

`p4-sdp-build_dev_rebrand` polls SCM every 10 minutes (`H/10 * * * *` as of this writing).
If you just submitted a change, it will likely pick it up on its own within that window --
check the latest build's included changelists (above) before triggering a redundant manual
build.

## Manually triggering a build (no auth needed from localhost)

```bash
ssh workshop-jenkins.das.perforce.com "
CRUMB=\$(curl -s 'http://localhost:8080/crumbIssuer/api/json' | grep -o '\"crumb\":\"[^\"]*\"' | cut -d'\"' -f4)
curl -s -X POST -H \"Jenkins-Crumb: \$CRUMB\" 'http://localhost:8080/job/<job-name>/build'
"
```

This returns immediately (build queued, not run synchronously) -- poll the "Checking the
latest build" query above until a new build number with a `<result>` appears (an in-progress
build has no `<result>` tag yet).

## Pulling build logs for failure analysis

```bash
ssh workshop-jenkins.das.perforce.com "cat /var/lib/jenkins/jobs/<job-name>/builds/<N>/log" > <scratchpad>/build_<N>.log
```

Grep for the actual Docker/Podman test suite's pass/fail markers within that log (the
underlying mechanism is the same `run_docker_tests.sh` / container-based suite described
in `test/README.md` in the SDP tree) rather than just checking Jenkins' own overall
`<result>`, since a container build failure can sometimes fall through to a stale cached
image silently passing -- see prior session logs for a real example of this happening on
the Classic-path job.