# Session Log: First Real Dry Run -- Found and Fixed a Silent Data-Loss Bug

**Date:** 2026-08-31
**Author:** Claude (Claude Code session)
**Requested by:** Tom Tyler

---

## Context

Continuation of the SDP-coding-standard alignment work (change 33383).
Tonight's ask: actually run `sync_jira_to_p4jobs.sh` end-to-end for the first
time -- dry run (no `-y`), `-max 5` -- against real JIRA and P4, and report
where the log lands.

## What it took to get a real run going

1. **P4 ticket file.** The script needs `P4PasswdFile` (a flat file with just
   the ticket, from `p4 login -p > file`). Writing that file was blocked
   twice by the Claude Code auto-mode safety classifier as credential
   extraction. Tom ran the command himself:
   `p4 login -p > /Users/ttyler/pub/j2j/secure/p4_ticket.value && chmod 600 ...`
   Tom noted a dedicated service account should replace his personal
   ticket/token before any live (`-y`) run -- personal credentials are fine
   for tonight's dry run only. **Follow-up, not done yet.**
2. **Test config.** Wrote `/Users/ttyler/pub/j2j/secure/dryrun_test.cfg`
   (inside the already-`.p4ignore`d `secure/` tree, `chmod 600`): real
   `JiraUrl`/`JiraUser` (`ttyler@perforce.com`, since JIRA Cloud Basic auth
   needs the account email, not the `ttyler` value in `token.user`), the
   copied `token.value`, `JiraProjects=SDP`, and `P4PORT=public.perforce.com:1666`
   / `P4USER=tom_tyler` (the same server/account already used to version this
   very script -- confirmed working via `p4 info` in an earlier session,
   rather than the untested `ssl:workshop.perforce.com:1666` from the README
   example).
3. **`flock` missing on macOS.** First run failed immediately:
   `flock: command not found`. `flock` is a Linux `util-linux` tool, already
   listed as a prerequisite in the README, but not present on macOS by
   default. Installed a compatible reimplementation via
   `brew install flock` (discoteq/flock 0.4.0) -- unblocked local testing
   without needing to move to a Linux box, which Tom had offered as a
   fallback if needed.

## First run result: looked clean, was actually broken

With those two things in place, `-C secure/dryrun_test.cfg -max 5 -v 5` ran
to completion with `Errors=0 Warnings=0` and reported **0 issues fetched**.
That looked like a legitimate "nothing changed in this window" result, but
it wasn't -- a direct `curl` call using the exact same JQL and URL the script
had just printed returned 50 real issues.

**Root cause:** `pageJson=$(jira_get "$url")` captures `jira_get()`'s stdout
as its return value. But `jira_get()` also calls `detail()` (verbosity 4+)
and `warnmsg()` (always, on any retry) -- and both of those wrote to
**stdout**, the same stream being captured. At `-v 5`, the
`detail "  JIRA GET (attempt 1/3): ..."` line landed in front of the actual
JSON inside `$pageJson`, so `json.loads()` choked on it. That failure was
silently swallowed by the script's own `2>/dev/null || echo 0` fallback
(used throughout to avoid a jq dependency), which is indistinguishable from
a legitimately empty page -- so the run reported a clean 0-issue success and
the watermark advanced right past 50 real, unsynced issues. Confirmed by
inspecting the raw cached page file
(`secure/state_test/jira_page_cache/SDP_page1.json`), which had the debug
line prepended to the real 170KB JSON payload.

**Why this matters beyond `-v 5`:** `warnmsg()` isn't gated by verbosity --
it always prints. So this wasn't just a debug-mode artifact: any transient
JIRA rate-limit (429) or 5xx retry, at any verbosity, including the default
production setting, would corrupt that page's captured JSON the same way,
get treated as "0 issues, no error," and silently advance the watermark past
real issues -- permanent, silent data loss with no error logged.
`map_jira_project_to_p4()` had the identical exposure via its own `warnmsg`
call, corrupting the returned P4 `Project` value instead.

**Fix (change 33384):** all diagnostic output (`msg`/`msgn`/`dbg`/`detail`,
and transitively `warnmsg`/`errmsg`) now writes to stderr, never stdout.
Stdout is reserved exclusively for actual function return values. No
visible-output change, since `init_logging()` already merges stderr into the
same tee'd/logged stream via `2>&1`.

**Re-ran after the fix:** fetched 50 issues, synced 5 (respecting `-max 5`),
correctly stopped early and logged "Watermark NOT advanced (-max 5
reached)," fields mapped as expected
(e.g. `SDP-1172 status=open type=Problem project=perforce-software-sdp`).

## Log location

Confirmed the log path convention works as designed:

- `/tmp/sync_jira_to_p4jobs.<YYYY-MM-DD-HHMMSS>.log` (this run:
  `/tmp/sync_jira_to_p4jobs.2026-08-31-231130.log`)
- `/tmp/sync_jira_to_p4jobs.log` -- symlink to the above, byte-identical to
  the timestamped file (diffed to confirm).
- Lands in `/tmp` because `$LOGS` isn't set in this ad hoc local environment
  (no SDP `p4_vars` sourced); on a real SDP host it'll be
  `$LOGS` (`/p4/<instance>/logs`) per the documented convention.

## Minor observation, not fixed tonight

`jira_type_to_p4()` mapped all 5 fetched issues to `Problem` -- they were all
JIRA type "Feature Request", which doesn't match the `"new feature"*`
pattern in the case statement (JIRA's actual type name doesn't start with
"new"). Falls through to the `Problem` default, which is a safe fallback,
not a crash, but likely not the intended mapping for a fairly common SDP
issue type. Worth adding an explicit `("feature request")` arm alongside
`story`/`epic`/`task` next time this file is touched.

## Follow-ups (not done tonight)

1. **Dedicated JIRA + P4 service accounts** before any live (`-y`) run --
   Tom flagged this himself. Currently using his personal API token and P4
   ticket, fine for dry runs only.
2. **`jira_type_to_p4()` "Feature Request" mapping** (above).
3. Everything already listed in `SESSION_2026-08-30_dry_run_and_next_steps.md`
   remains open: the `curl --user` argv exposure (same class of issue as the
   P4 `-P` fix, not yet addressed) and the `p4 fix` / `P4Blog` idea.