# Session Log -- 2026-09-03

## Summary

Morning: confirmed Robert Cowham's fix for the Jenkins Docker/Podman crash held
through a real Jenkins-triggered build. Then a substantial detour into
formalizing how Claude Code Skills are organized/versioned/synced across
this whole multi-depot effort, plus a real rename cleanup that came out of
that discussion.

## Jenkins verification

Robert's fix (targeted removal of two specific stale image IDs, not a full
wipe -- see his Slack reply) verified through an actual Jenkins-triggered
build of `p4-sdp-build_dev` (build 7): `tests_failed=0`, `Ran 2 tests...
OK`, `SDP Upgrade OK`, `Life is Good`, `Finished: SUCCESS`. Documented his
fix procedure in the `sdp-jenkins-build-check` skill, including what did
NOT work (full wipe, reboot) so we don't repeat those next time.

## Skills inventory, organization, and versioning

User asked for a list of all skills built so far, and wanted to figure out
where skills should live in version control and how new machines get set
up. Researched (via claude-code-guide subagent) Claude Code's actual skill
discovery/plugin mechanisms rather than guess:
- Discovery: `~/.claude/skills/`, `.claude/skills/` (project + parent dirs),
  `--add-dir`-added dirs' `.claude/skills/`, cloud-synced, plugin-sourced.
- Claude Code has a first-class git-based plugin/marketplace system
  (semver, `{plugin}--v{version}` git tags) -- the "established practice"
  answer -- but it's git-native, no P4 source type, so adopting it wholesale
  would mean either a git mirror or a `command`-source adapter. Decided
  NOT to adopt it: the existing symlink-into-P4 approach (discovered to
  already be in active, working use for all 5 pre-existing skills) is
  simpler and already proven for this P4-based, small-team context.
- Confirmed (separately, second research pass) that GitHub Copilot does
  NOT natively read `SKILL.md`/`.claude/skills/` -- it has its own,
  different mechanisms (`.github/copilot-instructions.md`, newer
  per-repo custom agent definitions). So colleagues on Copilot-only (e.g.
  Robert) can reuse the underlying skill *content* but need a translated
  companion file, not just a different symlink path. Some other claims
  from that research (a cross-vendor "Agent Skills" open standard;
  Gemini CLI/Cursor native SKILL.md support) came from lower-confidence
  sources (blog-like sites, not primary docs) -- flagged as unverified
  rather than asserted as fact.

Decided against a dedicated `~/.claude/skills`-rooted P4 workspace --
symlinks into the existing per-depot working clients are the right fit
(user: "symlinks are a better fit").

## The `ai/` vs `ai_dev_support/` naming cleanup

User realized they'd unintentionally created a naming collision risk: some
older Public Depot projects used a plain `ai/` folder for what this project
now calls `ai_dev_support/` (internal dev-process content, isolated, never
released) -- and now `p4-sdp` is about to get a *new*, differently-scoped
`ai/` (SDP-1386: non-isolated, shipped product content for SDP
customers/contributors, still a separate open design question). Surveyed
every depot on the Public Depot server for this collision; found exactly
three affected, none related to SDP itself: `p4lf` (isolate on `main`),
`p4mona` (isolate on `dev`), `p4sudo` (isolate on `dev`). `p4-sdp` and
`test-install_sdp` themselves never had a plain `ai/` -- not part of this.

Discussed Deep Rename (`p4 duplicate`+`p4 snap`+`p4 obliterate`, which the
user described using and recommending against for this exact kind of
cosmetic cleanup for customers over the years) vs. front-door `p4 move`.
Recommended `p4 move`: `obliterate` is irreversible and meant for truly
exceptional cases (leaked secrets, huge accidental binaries), not naming
hygiene; nothing here has ever propagated elsewhere (isolate blocked it),
so `move` has zero downside `obliterate` would avoid. User agreed
("we are presently in that situation where all the changes would only
affect pre-release things... I'm convinced by what sounds like my own
words from years ago sent back at me").

Executed for all three (using `~/pub/p4lf`, `~/pub/p4sudo` -- existing
workspaces -- and a newly-created `tom_tyler.P4MBPro5.p4mona` client,
since no local client for `p4mona` existed on this machine yet):
- `p4lf`: `p4 move ai/... ai_dev_support/...` on `dev` (change 33415);
  updated `main`'s isolate rule (already correctly on `main`).
- `p4sudo`: hit a real conflict -- `ai/AGENTS.md` was also open for plain
  edit on a different machine (`tom_tyler.cbb1.p4sudo`), no numbered
  changelist. Flagged to the user rather than guessing; user submitted it
  from `cbb1` (a since-relevant edit: softened `AGENTS.md`'s greeting from
  "Greetings, Claude!" to "Greetings, Agent!", tying directly into the
  cross-agent-compatibility question below). Resolved (`-am`, keeping that
  content), moved (change 33418), and relocated the isolate rule from
  `dev` to `main` to match the now-standard convention.
- `p4mona`: created a local client, moved (change 33421), relocated the
  isolate rule from `dev` to `main` same as `p4sudo`.

All three verified clean afterward (no `ai/` anywhere, `ai_dev_support/`
in place, `isolate ai_dev_support/...` consistently on `main`).

## New `//p4-sdp/dev/ai_dev_support/` scaffolding (change 33424)

- `SKILLS_INDEX.md`: master manifest of all 6 skills (the pre-existing 5
  plus the new `skills-sync` below) -- name, P4 server, depot path,
  purpose, as a markdown table that doubles as `skills-sync`'s parseable
  manifest source (deliberately one file, not two, so they can't drift
  apart). Also documents the `ai_dev_support/` vs. future `ai/dev`+`ai/ops`
  (SDP-1386) distinction inline, and notes the `p4lf`/`p4mona`/`p4sudo`
  rename for context.
- `skills/skills-sync/`: new skill covering both halves of "get a machine
  set up" -- `sync_skills.sh` automates re-syncing already-symlinked
  skills to head (reads the manifest, resolves each symlink, finds the
  right `.p4config*` file per target since different clients in this
  project use different config filenames, runs `p4 sync ./...`); the
  `SKILL.md` documents the guided (intentionally not scripted) bootstrap
  procedure for a brand-new machine, since choosing a new P4 client's Root
  is a real judgment call. Two real bugs found and fixed while testing:
  a naive table-row parser that choked on the word "|-delimited" in the
  index's own prose, and `p4 sync .` (which P4 treats as a literal
  filename) needing to be `p4 sync ./...` instead.

Symlinked all 6 skills (including the new `skills-sync` itself) into
`~/.claude/skills/` on this machine; ran `sync_skills.sh` clean at the end.

## Open / carried forward

- SDP-1158 go/no-go for 2026.1 -- user said they'd decide today, not yet
  revisited as of this log entry.
- The unverified cross-agent-portability claims (open "Agent Skills"
  standard, Gemini CLI/Cursor native support) should be independently
  confirmed before relying on them, if it ever matters for real (e.g. if
  a colleague on Gemini or Cursor actually tries pointing their agent at
  these same symlinks).
- Copilot-only colleagues (e.g. Robert) will need a translated companion
  file for any skill content we want them to have -- not yet started, not
  urgent unless someone asks.

## Two skill files found unsubmitted (caught by a user question)

User asked for the changelist reference behind yesterday's Jenkins-skill
update -- turned up that both `sdp-jenkins-build-check/SKILL.md` and
`install-sdp-test-suite-run/SKILL.md` had been sitting as local-only,
never-`p4 edit`'d disk changes since 2026-09-02 (the Edit-tool-vs-P4-
readonly gotcha again). Swept every other skill file across both servers
to check for the same problem and found one more: `bsw-qa-build-run`'s
retargeting section on PPN. All three fixed and submitted (changes 33425,
2763933). The other 3 skills (`bsw-lab-ssh`, `bsw-ppn-jobs`, and the new
`skills-sync`) were already clean.

User also had me strip a paragraph from change 33425's description via
`p4 change -u` -- it explained the unsubmitted-edit mistake itself, which
is process noise, not something useful a year from now. New standing rule
saved to memory (`feedback_changelist_description_no_tmi`): changelist
descriptions should only contain what a future reader of the history
actually needs.

## SDP-1158 decision: staying at 2026.2

Reviewed the JIRA issue directly. Surprise: it was already Fix Version
2026.2, not 2026.1 -- so the real question was whether to pull it *forward*,
not whether to defer it. Recommended against: the feature is essentially
0% implemented (still just a placeholder flag), has an unresolved design
question sitting in the ticket itself (clean up the old structure after
migration, or move it aside?), and the "soft launch" idea (implement,
leave undocumented + non-default) reduces customer exposure but not the
actual engineering risk/cost. User confirmed there's no external forcing
function -- just a known, low-priority, not-very-common point of customer
confusion (two documented upgrade procedures depending on original SDP
structure) that this would eventually fix, once promoted/documented later
anyway. Decided: SDP-1158 stays a 2026.2 deliverable, no `-pkg`
implementation work in 2026.1. Full reasoning in the
`project_sdp_structure_and_1158` memory.

## Fixed opt_perforce_sdp_backup.sh false-error report (change 33426)

Real user-reported issue: `install_sdp.sh` was reporting an *error* when
`opt_perforce_sdp_backup.service` failed to start, even though the service
actually works fine afterward and the install is fully usable -- this
happens near the very end of the process, so users got a scary-looking
error for something that wasn't one. Downgraded to `warnmsg` (leaves
`ErrorCount` at 0, so the run now correctly reports SUCCESS with a warning
noted, exit code 0) while leaving the timer enable/start steps as real
errors, since those are more consequential.

## First real rehearsal of the Streams-native release process

Started walking `doc/ReleaseProcessOverview.md` step by step for real, per
the user's request to exercise as much of the actual release flow as
possible (stopping short of anything that makes it live) and acknowledge
each step, including the intentional no-ops.

- Steps 1-2 (Plan/Develop): no-ops, all real work already done this cycle.
- Step 3 (target new P4D in test suite, major releases only): bumped
  `test_Upgrade.py`'s `--to-version` 25.2 -> 26.1 (change 33427).
- Step 4 (verify regression suites): re-ran all three test suites fresh
  against `dev` after the above changes -- AWS fleet all 7 machines clean
  (0 fails; the `show_versions` race from 2026-09-02 did not reproduce,
  reinforcing that it really was transient/timing, not a real bug), BSW
  `qa_build` SUCCESS, Jenkins `p4-sdp-build_dev` build 10 SUCCESS
  (confirmed genuinely clean: `tests_failed=0`, `Life is Good`).
- Step 5: determined Major release, "SDP 2026.1 GA."
- Step 6 (DefaultP4Version, major only): already `r26.1` -- no-op, done
  earlier this cycle.
- Step 7 (P4*.json files, major only): refreshed from
  updates.perforce.com, newer build available (2797379 -> 2873834) across
  P4/P4D/P4Broker/P4Proxy (change 33428).
- Step 8 (load release tools): done; incidentally did the *first-ever*
  full `p4 sync` of the `bot_Claude_Anthropic.p4-sdp_dev` client (it had
  only ever been synced path-by-path until now) -- large but harmless.
- Step 9 (Merge Down): dry run clean, nothing to merge -- expected, no
  hotfixes have touched `main` directly since the last Copy Up.
- **Paused before Step 10 (Copy Up)** for a real design discussion (below)
  before resuming -- Copy Up itself has not yet been executed as of this
  log entry.

### Old-URL safety think-through, and a real gap it surfaced

User asked for a "think through" of what happens when a customer executes
a stale, bookmarked pre-2026.1 procedure against/after the 2026.1 release.
Conclusion: safe today by construction, since Classic stays frozen-but-
alive (not deleted or redirected) and the actual upgrade mechanics aren't
drastically changing. Full writeup in the new
`project_sdp1170_stale_url_safety` memory (SDP-1170 design, not yet
implemented).

That analysis went through two rounds on one specific question -- whether
to replace `install_sdp.sh` in Classic with a "We've Moved" stub. Round 1:
yes for `install_sdp.sh` (any fresh install should land on the new
location), no for `sdp_upgrade.sh` (legitimate for customers not moving to
Streams/2026.1 yet) or the tarball itself (fallback for stale local script
copies). **Round 2 (final, user's change of heart)**: dropped the stub
idea entirely -- SDP's N-2 support model means 2025.2 stays supported
until 2028.1 ships, so a fresh install landing on 2025.2 via a stale
bookmark is a fully legitimate, supported outcome, not just a tolerable
fallback. Final scope: zero edits to any Classic functional content; the
only proactive "We've Moved" mechanism is the already-planned Swarm
project home page retarget (a discovery aid for browsers, not something
touching anyone running scripts), pointing at `main`'s browsable URL.

This surfaced a real, separate problem while thinking through today's
actual mechanics: **Copy Up (Step 10) happens well before the release
stream exists or is packaged (Steps 17-19)**, so if `main`'s
`install_sdp.sh` referenced the upcoming release stream's own URL (the
original plan -- baking in `r26.1.0.BETA` now, for real `r26.1.0` later),
that URL would 404 for the entire window in between. User's fix, which is
clearly better than the original plan: `isolate downloads/...` on
`//p4-sdp/main` (added to the stream spec), so Copy Up never disturbs
`main`'s tarball; point `install_sdp.sh`'s `SDPURL`/`ThisScriptURL`/the
`get_p4_binaries.sh` example/the two `ConfigDoc` doc-links at a **fixed,
permanent `main` path** instead of any release-stream name (change 33430,
reverting the just-added `r26.1.0.BETA` references); and add a new
Step 20 ("Publish the Tarball to `main`") that explicitly duplicates the
freshly-packaged tarball into `main`'s own `downloads/` once it exists
(change 33432, along with rewriting the doc to explain the design and
adding Step 22 for the manual post-release smoke tests -- fresh-VM
install-from-scratch plus checkpoint script exercise, and dogfooded
real-environment upgrade -- that have always been done by hand but were
never written down).

Net effect: `install_sdp.sh` never needs a URL edit again, for any future
release, and there is no dead-link window anywhere in the process --
`main`'s fixed download URL always resolves to a real tarball, old until
published and new immediately after, exactly mirroring how Classic's
single fixed download location always behaved.

Decided (per the user, "we won't be deleting the Classic area nor making
it so folks using old bookmarks get 404's") that this whole investigation
was itself a good instance of the paranoia driving SDP-1170's design --
worth keeping around as the reference case if the stub idea, or anything
like it, ever gets reconsidered.

User flagged wanting new Claude Code Skills for various release-process
steps eventually -- noted, not started yet.

## Full release process rehearsal executed for real (Steps 3-20)

Continued the step-by-step rehearsal all the way through packaging and
publishing, for the first time ever under the new Streams structure.
Executed, not just documented: Copy Up (change 33433, main's first-ever
population, 463 files), doc revnumber/revdate bumps (33434), Update
Release Notes (33442, Version added fresh since main never had one),
cut `//p4-sdp/r26.1.0.BETA` (populated via change 33444), final regen in
the release stream (33446/33447), packaging (33451), and publishing the
tarball to `main`'s permanent download URL (33452, filetype bug fixed in
33453).

**Six real, previously-latent bugs found and fixed** purely by executing
the process instead of just reading it -- all now fixed in both code and
`doc/ReleaseProcessOverview.md`:
1. `gen_file_format_check.sh` silently checked stale frozen Classic
   content instead of the current stream (change 33435).
2. `opt_perforce_sdp_backup.sh` was missing from
   `gen_script_man_pages.sh`'s known-scripts list entirely, breaking
   `SDP_Guide.Unix.adoc`'s build (33438/33439).
3. `preview_changes_and_update_version.sh` couldn't handle `Version` not
   existing yet -- the first-release case (33441).
4. `p4 populate -S` without `-r` populates backwards, parent from child
   instead of child from parent (33445).
5. Nothing seeded the release stream's own isolated `Version` file after
   populate -- silently broke tarball version-stamping into
   `sdp.Unix...tgz` with empty gaps (33449/33450).
6. A path mistake in the new Step 20 (`cd $WSRoot/tools` instead of
   `$WSRoot`) led `p4 add` to silently mistype two real tarball/zip files
   as `text` instead of `binary` -- caught via md5 verification before it
   could cause real corruption, fixed in both the depot (33453) and the
   doc, which now passes `-t binary+F` explicitly (33454).

Result: `//p4-sdp/r26.1.0.BETA` exists, fully packaged, and its tarball is
live at `main`'s permanent, version-independent download URL -- but the
Swarm project home page for `p4-sdp` still points at Classic, so nothing
customer-facing has actually changed yet. That's deliberate.

## Real timeline set for the actual release

**The Swarm project home page retarget (not yet done) is the actual point
of no return**, not anything done so far. User's plan: use today
(2026-09-03) and tomorrow (2026-09-04, Friday) to test thoroughly and be
100% confident, but the *actual* release is deferred to **Tuesday
2026-09-08** -- Support Management asked to avoid a Friday release
(despite the user's own preference for Friday releases), and this coming
Monday is Labor Day.

## Swarm HTML rendering bug found and fixed (README.md)

User found a real, customer-facing bug while testing: clicking `README.md`'s
plain relative HTML links (e.g. `doc/SDP_Guide.Unix.html`) in Swarm lands
on `/files/...` (Swarm's raw browser -- ugly raw HTML + P4 metadata), not
the rendered `/view/...` page. Researched Swarm's own docs (couldn't get
conclusive answers from indirect doc-page fetches; Swarm's live file-browse
UI requires auth WebFetch doesn't have) -- found Swarm's own convention for
relative *image* references uses a site-root-relative path explicitly
starting with `/view/<depot-path>/...`. Ran a real A/B test directly on
`r26.1.0.BETA` (user's explicit permission to edit the release stream
directly and reconcile after): added a second link using that form
alongside the existing one. **Confirmed by the user**: the `/view/`-prefixed
site-relative link renders correctly; the plain relative one does not,
regardless of which page it's clicked from.

Fixed properly:
- First did a **Merge Down (main -> dev, change 33456)** to reconcile 19
  files that had drifted -- all of today's Steps 11-20 hotfixes had been
  made directly in `main`, per the doc's own process, and needed pulling
  back into `dev` before the real fix could land in the right place.
  Verified safe to bulk-accept-theirs since every file was exactly this
  session's own known edits, nothing concurrent.
- Fixed all 7 HTML cross-reference links in `README.md` to use
  `/view/p4-sdp/main/...` (change 33457 in `dev`) -- pointed at `main`
  permanently, matching the same design philosophy as `install_sdp.sh`'s
  `SDPURL`/`ThisScriptURL`, so these never need per-release updates either.
  PDF links left as plain relative paths (not reported broken, and PDFs
  aren't Swarm-rendered the same way).
- Found and fixed a second, unrelated real bug along the way:
  `SDP_Developer_Guide.{pdf,html}` didn't match the actual generated
  filenames (`SDP_DeveloperGuide.{pdf,html}`, no underscore) -- a
  genuinely broken link, not a rendering issue. Also fixed two lines
  missing their opening `(` before `[PDF]`.
- Applied the identical fix directly to `r26.1.0.BETA` too (change 33458,
  replacing the temporary A/B test line), since release streams don't
  receive `dev`/`main` fixes through the normal flow.

## Doc cleanup pass (Change History, stale examples, naming notation)

User asked for a focused cleanup pass, four items:

1. **Release Notes Change History**: filled in the real SDP 2026.1 GA
   "Jobs Fixed" list from JIRA (`fixVersion=2026.1`, closed issues only --
   excluded SDP-1170 (deferred to a later release), SDP-1385 (the
   not-yet-executed Swarm retarget itself), and SDP-1386 (stretch, not
   done)). Found **7** closed issues, not the 8 the user recalled --
   flagged the discrepancy rather than silently trusting either number.
   Also removed a duplicate/typo'd "What's New" bullet about rebranding
   (change 33461 in `dev` for the Jobs list + DeveloperGuide fixes below;
   change 33463 in `main` for the duplicate-bullet fix).
2. **SDP_DeveloperGuide.adoc**: fixed stale `r25.1` version examples to
   `r26.1.0`. Along the way found the release-stream/`main` table row
   described the OLD, explicitly-rejected model (hotfixes made directly
   in release streams, merged down into `main`; regression suites
   targeting `main`) -- corrected to match the actual process (release
   streams never patched in place; regression targets `dev`). **Flagged,
   not fixed**: the "Hot Fix" glossary entry describes the same outdated
   concept (changes made directly in a release stream without a new
   tarball) -- left for a deliberate decision rather than silently
   rewriting a named term.
3. **Confirmed and fixed the release-stream naming notation**: the doc
   said `r<YYYY>.<N>[.<P>]` (patch optional) but every real example this
   session used (`r26.1.0`, `r26.1.0.BETA`) has the patch digit present --
   fixed to `r<YYYY>.<N>.<P>` with `<P>` always present, `0` for GA
   (change 33462 in `main`).
4. **Reviewed README.md and the top of ReleaseNotes.adoc**: fixed a clear
   stale/typo'd line ("The P4 SDP 2025.2 support the folllowing..." ->
   2026.1, grammar fixed). **Flagged, not fixed**: a literal
   `EDITME-UpdateThisURL` placeholder for the Helm Management System link
   (no real URL available to fill in); the "What's Coming Next in SDP
   2026.2?" section is still a bare `EDITME` (forward-looking content, not
   a correction); the "home for the SDP... has changed from Classic to
   Streams" bullet is not yet literally true (Swarm hasn't been
   retargeted) but will likely be true by the real 2026-09-08 release, so
   left alone rather than "fixed" into something that's about to become
   correct anyway.

All of today's `main`-side fixes were Merged Down into `dev` again
afterward (changes 33456, 33464) -- including one genuine two-way
divergence on `README.md`/`ReleaseNotes.adoc` (each stream had picked up
different, non-overlapping fixes), auto-merged cleanly with 0 conflicting
chunks, verified before submitting.

## Real-world smoke test: two disposable AWS VMs, real documented URL

Per the user's request, launched two genuinely new (not part of the
permanent fleet), disposable EC2 instances in us-east-2 -- Ubuntu 24
(`disposable-test-u24-1`, `i-0dac4107629a1c421`, `18.191.178.172`) and
Rocky 9 (`disposable-test-r9-1`, `i-0160e0ab72d868a67`,
`18.189.145.197`) -- matching the existing fleet's AMI/instance-type/
security-group/subnet config. Ran the actual documented bootstrap
procedure from `README.md`'s own EXAMPLES text, using the real, live,
public `https://workshop.perforce.com/download/p4-sdp/main/Server/Unix/setup/install_sdp.sh`
URL (not the internal DVCS fleet mechanism the regular test suite uses).

Confirmed the real download pulled genuine current content (`install_sdp.sh
version MAIN.33433`), and per the user's note that `-demo` is required
without pre-configured storage mounts, used `-sampledepot -demo` matching
the documented example exactly. First run (no `-y`) correctly did a dry
run with an instructive message; second run (`-y`) did a real install on
both: Ubuntu 24 completed with **no errors or warnings**; Rocky 9
completed with **4 warnings**, all benign and matching already-known
fleet patterns (missing optional OS packages, no firewall detected on a
cloud instance) -- nothing new. This is real, end-to-end confirmation
that a brand-new customer following the documented procedure today would
succeed.

Left both VMs running per the user's request ("let me have some fun
playing around in there").

## Open / carried forward (updated)

- Real release now targeted for **Tuesday 2026-09-08**. Today and
  tomorrow (2026-09-04) are for continued testing before the Swarm
  project home page retarget -- the actual point of no return.
- New skills for release-process steps -- flagged by the user, not
  started.
- Worth checking, not yet done: whether PDF links in README.md have any
  analogous Swarm rendering quirk (not reported broken, but untested).
- Post-Release Smoke Tests (doc Step 22: fresh-VM install test, dogfooded
  production upgrade) -- the fresh-VM half is now done for real (see
  above, both disposable VMs); the dogfooded-production-upgrade half is
  still not done -- real production scope, paused pending direction.
- **TODO: terminate the two disposable EC2 instances** (`i-0dac4107629a1c421`,
  `i-0160e0ab72d868a67`, region `us-east-2`) once the user's done playing
  with them -- left running intentionally, not an oversight.
- Decisions still needed from the user: the "Hot Fix" glossary entry in
  SDP_DeveloperGuide.adoc (item 2 above), the real HMS URL, and whether/
  when to write real "What's Coming Next in SDP 2026.2" content.
- Worth double-checking: the user recalled "8" JIRA issues for 2026.1;
  JQL found only 7 closed ones (see Change History fix above).