run_if_replica.sh #11

  • //
  • guest/
  • russell_jackson/
  • sdp/
  • Server/
  • Unix/
  • p4/
  • common/
  • bin/
  • run_if_replica.sh
  • View
  • Commits
  • Open Download .zip Download (1 KB)
#!/usr/bin/env bash
#==============================================================================
# Copyright and license info is available in the LICENSE file included with
# the Server Deployment Package (SDP), and also available online:
# https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/LICENSE
#------------------------------------------------------------------------------

instance=${1}
if [ ! -d /p4/${instance} ]; then
  exit 0 
fi
shift

# Load SDP environment functions
source /p4/common/bin/p4_vars "$instance"
source /p4/common/bin/backup_functions.sh

# List of variables that must be set before continuing
required_vars=(SERVER_TYPE P4REPLICA EDGESERVER)

check_vars() {
    local missing=0
    for v in "${required_vars[@]}"; do
        if [[ -z "${!v}" ]]; then
            echo "Missing required variable: $v"
            missing=1
        fi
    done
    return $missing
}

# --- First check ---
check_vars
if [[ $? -ne 0 ]]; then
    echo "Calling set_vars to initialize missing variables..."
    set_vars
fi

# --- Check again after set_vars ---
check_vars
if [[ $? -ne 0 ]]; then
    echo "ERROR: Required variables are still missing after calling set_vars."
    exit 1
fi

# --- Script logic now safe to execute ---

if [[ "${SERVER_TYPE}" == "p4proxy" ]]; then
   exit 0
fi

if [[ "${SERVER_TYPE}" == "p4d_standby" || "${SERVER_TYPE}" == "p4d_edgerep" ]]; then
   exec "$@"
fi
# Change User Description Committed
#14 33340 Russell C. Jackson (Rusty) Add structured logging (JSONL/Prometheus) to SDP maintenance scripts

Failure has historically gone silently unlogged in these scripts while
success was well-logged: verify_shelves.sh had no error detection at
all, most other scripts only log a human-readable narrative with
nothing machine-readable, and nothing here is consumable by
monitoring tools (telegraf, Datadog, Prometheus) without a bespoke
parser per script.

Adds one shared bash helper (sdp_structured_log.sh, sourced
automatically by backup_functions.sh) and one shared stdlib-only
Python module (sdp_structured_log.py, importable by both cron scripts
and trigger scripts). One mode at a time, selected by the new
SDP_LOG_FORMAT p4_vars setting (jsonl by default, or prom for sites
running Prometheus instead, or off) - never simultaneous dual-writing.
jsonl mode appends one JSON object per run alongside each script's
existing .log file; prom mode writes Prometheus textfile-collector
gauges to $SDP_METRICS_DIR (default /p4/metrics, matching the existing
p4prometheus_metrics_dir convention in the perforce-sdp-monitoring
role) using the same atomic write-then-mv convention that role's own
health-check scripts already use.

Success/failure capture is trap-based (bash: EXIT/ERR/HUP/INT/TERM,
consolidated so exactly one terminal record is emitted regardless of
how a script exits) / atexit-based (Python), specifically so a script
dying unexpectedly - an unbound variable, a set -e failure, a signal -
still gets recorded rather than silently vanishing. die() is annotated
(2 lines) rather than made to emit directly, so every one of the ~40
scripts that source backup_functions.sh gets failure-path coverage
automatically once they call the new sdp_log_init, with no per-script
failure plumbing needed.

Also fixes verify_shelves.sh, which had no error detection whatsoever:
`p4 changes`'s exit code was discarded and consumed by an unquoted
`for` loop, so a failed listing produced an empty loop and exit 0 - a
total failure was indistinguishable from "no shelves to verify". `p4
verify`'s own exit code was likewise never checked, and the script
rm -f'd its own logfile every run (after log_init had already started
tailing it interactively, which broke that too). Added the missing
check_vars/set_vars/check_uid/p4login preflight that every other
script here already has.

Restructures the run_if_{master,edge,replica,broker,proxy}.sh cron
wrapper family: each previously ended with `exec "$@"`, which replaces
the wrapper's own process image, so no trap could ever fire and this
wrapper's own success/failure was structurally impossible to record.
Now a normal call + $? capture, still propagating the wrapped command's
exit code unchanged. This alone gives structured coverage for every
cron job routed through these wrappers, before any individual wrapped
script is itself instrumented. Also fixes a related silent gap in the
master/edge/replica variants: SERVER_TYPE matching neither the proxy
skip nor the expected role fell through to an unexplained implicit
exit 1 - now recorded as an explicit failure naming the mismatch,
instead of a bare unexplained nonzero exit cron would mail with no
context.

triggers/keep_group_unset.py: fixed its stale `#!/usr/bin/env python`
shebang to python3 (the body already requires it via os.replace(),
3.3+ only), and added a guarded structured-logging record. Behavior
preservation is deliberate here: an unexpected exception still rejects
the p4 group form edit exactly as before (fail-closed) - the change
only adds a logged record explaining why, never stdout (p4d relays a
trigger's stdout to the connected client), and degrades to a silent
no-op if sdp_structured_log.py hasn't been deployed yet (it ships in
the same tarball as this trigger but via a different deploy path than
the Ansible-templated copy of this same file - see the update-tgz.sh
changelist that follows this one).

Extends rotate_last_run_logs/remove_old_logs to cover the new .jsonl
siblings, including adding verify_shelves.log/.jsonl to the KEEPLOGS
cleanup list - it was absent from that list entirely before (masked
by the rm -f bug this same change removes), so without this addition
it would have grown unbounded.
#13 32426 Russell C. Jackson (Rusty) Claude ai updates.
#12 32388 Russell C. Jackson (Rusty) Updates using Claude.ai to clean up the code, reduce duplication, enhanace security, and use current standards.
#11 32364 Russell C. Jackson (Rusty) Forgot about p4d_edgerep type
#10 32349 Russell C. Jackson (Rusty) Changed to use SERVERTYPE for checks.
#9 32346 Russell C. Jackson (Rusty) Changed logic to only call set_vars if the variables are missing.
#8 31923 Russell C. Jackson (Rusty) Removing set -x
#7 24116 Russell C. Jackson (Rusty) Added check for proxy server.
#6 24078 Russell C. Jackson (Rusty) Added check to see if directory exists.
Exit silently if not.
#5 22981 Russell C. Jackson (Rusty) Made files writeable so they are easier to update once on the server.
#4 22799 Russell C. Jackson (Rusty) Removed () from set_vars call.
Cut and paste mistake.
#3 22719 Russell C. Jackson (Rusty) Added call to source backup_functions.sh and run set_vars.
#2 22718 Russell C. Jackson (Rusty) Added check to make sure we are not on an edge server.
#1 22693 Russell C. Jackson (Rusty) Branched a Unix only version of the SDP.
Removed extra items to create a cleaner tree.
Moved a few items around to make more sense without Windows in the mix.
//guest/perforce_software/sdp/dev/Server/Unix/p4/common/bin/run_if_replica.sh
#1 22686 Russell C. Jackson (Rusty) Two small scripts to allow using a single crontab for all servers by putting all the entries into one file and only running based on server type.