--- name: sdp-jenkins-build-check description: Check the status of (or manually trigger) the SDP Docker/Podman Jenkins build for a given stream. Use whenever asked to run, re-run, or check the Docker/Podman/Jenkins test suite. --- # SDP Jenkins Build Check / Trigger Checks (and, if needed, manually triggers) the Jenkins build that runs the SDP Docker/Podman test suite. See `SDPTestSuite-DockerPodman.md` (in `//test-install_sdp`'s `ai_dev_support`) for background; this skill codifies the operational steps. ## Prerequisites - SSH access to `workshop-jenkins.das.perforce.com` -- requires being on the Perforce VPN. If SSH times out, that's almost always a VPN-connectivity issue, not a Jenkins problem -- confirm VPN before troubleshooting further. ## Relevant jobs - `p4-sdp-build_dev` -- Streams-based, builds `//p4-sdp/dev`. As of 2026-09-02, this is the one to check/trigger for current mainline (Streams-era) SDP work. - `p4-sdp-build_dev_rebrand` -- builds `//p4-sdp/dev_rebrand`, the now-retired predecessor to `dev` (Copy Up completed 2026-09-02, change 33409). Kept around for now but no longer the primary target; don't assume it still matters without checking `SessionLog-*.md`. - `SDP-Dev-Build` -- Classic depot path (`//guest/perforce_software/sdp/dev/...`). Frozen/ deprecated as of the Classic-to-Streams migration, but was briefly re-fixed and kept live in parallel during 2026-08. Check `SessionLog-*.md` for the current status before assuming it's still relevant. Both jobs live under `/var/lib/jenkins/jobs//` on the Jenkins host. ## Each job builds a *virtual stream*, not the dev stream directly The job's workspace `streamName` (in `config.xml`) is `//p4-sdp/build_` -- a `virtual` stream whose parent is `//p4-sdp/` -- not `//p4-sdp/` itself. This indirection means retargeting to a brand-new dev-cycle stream doesn't require inventing a Jenkins config from scratch: 1. Create the virtual stream (parented off the new target): ```bash p4 stream -o //p4-sdp/build_ | \ sed -e 's@^Stream:.*@Stream:\t//p4-sdp/build_@' \ -e 's@^Parent:.*@Parent:\t//p4-sdp/@' \ -e 's@^Name:.*@Name:\tbuild_@' \ | grep -v '^Update:\|^Access:' | p4 stream -i ``` 2. Clone the job directory and swap the stream name/description in `config.xml`: ```bash ssh workshop-jenkins.das.perforce.com " sudo mkdir -p /var/lib/jenkins/jobs/p4-sdp-build_ sudo cp /var/lib/jenkins/jobs/p4-sdp-build_/config.xml /var/lib/jenkins/jobs/p4-sdp-build_/config.xml sudo sed -i -e 's@//p4-sdp/build_@//p4-sdp/build_@g' /var/lib/jenkins/jobs/p4-sdp-build_/config.xml sudo chown -R jenkins:jenkins /var/lib/jenkins/jobs/p4-sdp-build_ " ``` 3. Make Jenkins pick up the new job directory (no restart needed, just a config reload): ```bash ssh workshop-jenkins.das.perforce.com " curl -s -c /tmp/jcookie.txt 'http://localhost:8080/crumbIssuer/api/json' -o /tmp/jcrumb.json CRUMB=\$(grep -o '\"crumb\":\"[^\"]*\"' /tmp/jcrumb.json | cut -d'\"' -f4) curl -s -b /tmp/jcookie.txt -X POST -H \"Jenkins-Crumb: \$CRUMB\" 'http://localhost:8080/reload' -o /dev/null -w '%{http_code}\n' " ``` **Gotcha found 2026-09-02**: fetching the crumb and POSTing `/reload` as two separate `curl` calls without a shared cookie jar can 403 with "No valid crumb was included" even though the crumb value itself is correct -- Jenkins' CSRF protection here is session-tied, not just value-tied. Use `-c`/`-b` with the same cookie file across both calls (as above). A `200`/`302` means it worked; verify with `curl -s 'http://localhost:8080/job/p4-sdp-build_/api/json?tree=name,url'`. ## RAM headroom -- check before adding a new concurrently-pollable build Confirmed 2026-09-02: this host has only ~3.6GB total RAM, with ~1GB genuinely free (~2.3GB "available" counting reclaimable cache) -- `free -h` to recheck. Disk is not a concern (174G free of 195G as of this writing). Each job polls SCM independently (`H/10 * * * *`); nothing prevents two different jobs' Docker/Podman container builds from running at the same time even though `concurrentBuild=false` only blocks a *single* job from overlapping with itself. Before triggering a newly-added job for the first time (or relying on its independent polling schedule long-term), confirm with the user whether to request more RAM, rather than silently risking an OOM if both jobs' container fleets end up running concurrently. ## Container crashes (`container state improper` / exit 137) This host has repeatedly hit a rootless-podman/systemd container-startup crash -- first seen 2026-08-24, recurred 2026-09-02/03. Symptoms: `podman exec /p4/docker_entry.sh` either errors immediately with `Error: can only create exec sessions on running containers: container state improper`, or runs and exits with `137` (SIGKILL) with no useful output. Both are the same underlying container dying right around startup; which exact error you see seems to depend on timing. **Things that have NOT reliably fixed it** (tried 2026-09-02, all ineffective that time even though some had fixed it before): - A full podman wipe for the `jenkins` user (`podman rm -f $(podman ps -aq)`; `podman rmi -a -f`; `podman system prune -a -f --volumes`). - A full host reboot (`sudo reboot`) -- freed RAM from ~1GB to ~2.5GB free but the crash reproduced identically right after. - Checking for a cgroup-delegation regression -- confirmed still correctly in place (`/etc/systemd/system/user@.service.d/delegate.conf`, `Delegate=cpu cpuset io memory pids`); not the cause this time. - Checking dmesg/journalctl for OOM kills -- found none, so despite the exit-137 signature this isn't necessarily a real OOM event. **What actually fixed it, per Robert Cowham 2026-09-03**: don't wipe everything -- find and remove the *specific* stale/corrupted image IDs, then clear containers, then let images rebuild fresh: ```bash podman images # find the suspect image ID(s) podman rmi [...] podman kill $(podman ps -aq) podman rm $(podman ps -aq) podman rmi [...] # again, in case a kill/rm regenerated them podman images # confirm clean ``` Robert then verified by running the test script **directly** (bypassing Jenkins entirely) as a faster diagnostic loop: ```bash sudo -u jenkins -H bash -c 'cd /var/lib/jenkins/workspace/ && sdp/test/run_docker_tests.sh' ``` Only after that came back clean did he consider it fixed -- worth doing this direct-script check yourself before/instead of burning a full Jenkins build cycle when iterating on a fix. **Known-harmless noise to expect in an otherwise-passing run**: the "Testing the SDP Upgrade Procedure" step prints `gzip: stdin: not in gzip format` / `tar: Child returned status 1` -- this is the pre-existing, already-known `sdp.Unix.tgz` 404/gzip-format quirk from 2026-08-24, not a new failure. The real signal is the final `SDP Upgrade OK` / `SDP Post-Upgrade Verify Completed OK` / `Post-upgrade Test Message is as expected: Life is Good` lines after it. **Not fully understood**: why this recurs, why the fixes that worked once don't reliably work the next time, and why the specific bad image IDs vary. Treat any "fix" here as provisional, not a permanent resolution -- re-check with `podman images`/`podman ps -a` if it recurs again rather than assuming a new root cause. ## Checking the latest build ```bash ssh workshop-jenkins.das.perforce.com " LAST=\$(ls -t /var/lib/jenkins/jobs//builds/ | grep -E '^[0-9]+\$' | head -1) echo LAST_BUILD=\$LAST grep -m1 '' /var/lib/jenkins/jobs//builds/\$LAST/build.xml grep -m1 '' /var/lib/jenkins/jobs//builds/\$LAST/build.xml grep -o '3[0-9][0-9][0-9][0-9]' /var/lib/jenkins/jobs//builds/\$LAST/build.xml | sort -u " ``` The last `grep` pulls any 5-digit changelist numbers referenced in the build record, so you can confirm whether a specific changelist you care about was actually included in that build (Perforce changelist numbers in this depot are currently in the 33xxx range -- adjust the pattern if that range has moved on). `` is milliseconds since the Unix epoch; convert with e.g. `python3 -c "import datetime; print(datetime.datetime.fromtimestamp(/1000))"`. ## Is it already going to build on its own? Check the polling schedule before manually triggering anything: ```bash ssh workshop-jenkins.das.perforce.com "grep -A2 SCMTrigger /var/lib/jenkins/jobs//config.xml" ``` `p4-sdp-build_dev_rebrand` polls SCM every 10 minutes (`H/10 * * * *` as of this writing). If you just submitted a change, it will likely pick it up on its own within that window -- check the latest build's included changelists (above) before triggering a redundant manual build. ## Manually triggering a build (no auth needed from localhost) ```bash ssh workshop-jenkins.das.perforce.com " CRUMB=\$(curl -s 'http://localhost:8080/crumbIssuer/api/json' | grep -o '\"crumb\":\"[^\"]*\"' | cut -d'\"' -f4) curl -s -X POST -H \"Jenkins-Crumb: \$CRUMB\" 'http://localhost:8080/job//build' " ``` This returns immediately (build queued, not run synchronously) -- poll the "Checking the latest build" query above until a new build number with a `` appears (an in-progress build has no `` tag yet). ## Pulling build logs for failure analysis ```bash ssh workshop-jenkins.das.perforce.com "cat /var/lib/jenkins/jobs//builds//log" > /build_.log ``` Grep for the actual Docker/Podman test suite's pass/fail markers within that log (the underlying mechanism is the same `run_docker_tests.sh` / container-based suite described in `test/README.md` in the SDP tree) rather than just checking Jenkins' own overall ``, since a container build failure can sometimes fall through to a stale cached image silently passing -- see prior session logs for a real example of this happening on the Classic-path job.