#!/usr/bin/env bash
#==============================================================================
# Copyright and license info is available in the LICENSE file included with
# the Server Deployment Package (SDP), and also available online:
# https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/LICENSE
#------------------------------------------------------------------------------
instance=${1}
if [ ! -d /p4/${instance} ]; then
exit 0
fi
shift
# Load SDP environment functions
source /p4/common/bin/p4_vars "$instance"
source /p4/common/bin/backup_functions.sh
# List of variables that must be set before continuing
required_vars=(SERVER_TYPE P4REPLICA EDGESERVER)
check_vars() {
local missing=0
for v in "${required_vars[@]}"; do
if [[ -z "${!v}" ]]; then
echo "Missing required variable: $v"
missing=1
fi
done
return $missing
}
# --- First check ---
check_vars
if [[ $? -ne 0 ]]; then
echo "Calling set_vars to initialize missing variables..."
set_vars
fi
# --- Check again after set_vars ---
check_vars
if [[ $? -ne 0 ]]; then
echo "ERROR: Required variables are still missing after calling set_vars."
exit 1
fi
# --- Script logic now safe to execute ---
if [[ "${SERVER_TYPE}" == "p4proxy" ]]; then
exit 0
fi
if [[ "${SERVER_TYPE}" == "p4d_standby" || "${SERVER_TYPE}" == "p4d_edgerep" ]]; then
exec "$@"
fi
| # | Change | User | Description | Committed | |
|---|---|---|---|---|---|
| #14 | 33340 | Russell C. Jackson (Rusty) |
Add structured logging (JSONL/Prometheus) to SDP maintenance scripts Failure has historically gone silently unlogged in these scripts while success was well-logged: verify_shelves.sh had no error detection at all, most other scripts only log a human-readable narrative with nothing machine-readable, and nothing here is consumable by monitoring tools (telegraf, Datadog, Prometheus) without a bespoke parser per script. Adds one shared bash helper (sdp_structured_log.sh, sourced automatically by backup_functions.sh) and one shared stdlib-only Python module (sdp_structured_log.py, importable by both cron scripts and trigger scripts). One mode at a time, selected by the new SDP_LOG_FORMAT p4_vars setting (jsonl by default, or prom for sites running Prometheus instead, or off) - never simultaneous dual-writing. jsonl mode appends one JSON object per run alongside each script's existing .log file; prom mode writes Prometheus textfile-collector gauges to $SDP_METRICS_DIR (default /p4/metrics, matching the existing p4prometheus_metrics_dir convention in the perforce-sdp-monitoring role) using the same atomic write-then-mv convention that role's own health-check scripts already use. Success/failure capture is trap-based (bash: EXIT/ERR/HUP/INT/TERM, consolidated so exactly one terminal record is emitted regardless of how a script exits) / atexit-based (Python), specifically so a script dying unexpectedly - an unbound variable, a set -e failure, a signal - still gets recorded rather than silently vanishing. die() is annotated (2 lines) rather than made to emit directly, so every one of the ~40 scripts that source backup_functions.sh gets failure-path coverage automatically once they call the new sdp_log_init, with no per-script failure plumbing needed. Also fixes verify_shelves.sh, which had no error detection whatsoever: `p4 changes`'s exit code was discarded and consumed by an unquoted `for` loop, so a failed listing produced an empty loop and exit 0 - a total failure was indistinguishable from "no shelves to verify". `p4 verify`'s own exit code was likewise never checked, and the script rm -f'd its own logfile every run (after log_init had already started tailing it interactively, which broke that too). Added the missing check_vars/set_vars/check_uid/p4login preflight that every other script here already has. Restructures the run_if_{master,edge,replica,broker,proxy}.sh cron wrapper family: each previously ended with `exec "$@"`, which replaces the wrapper's own process image, so no trap could ever fire and this wrapper's own success/failure was structurally impossible to record. Now a normal call + $? capture, still propagating the wrapped command's exit code unchanged. This alone gives structured coverage for every cron job routed through these wrappers, before any individual wrapped script is itself instrumented. Also fixes a related silent gap in the master/edge/replica variants: SERVER_TYPE matching neither the proxy skip nor the expected role fell through to an unexplained implicit exit 1 - now recorded as an explicit failure naming the mismatch, instead of a bare unexplained nonzero exit cron would mail with no context. triggers/keep_group_unset.py: fixed its stale `#!/usr/bin/env python` shebang to python3 (the body already requires it via os.replace(), 3.3+ only), and added a guarded structured-logging record. Behavior preservation is deliberate here: an unexpected exception still rejects the p4 group form edit exactly as before (fail-closed) - the change only adds a logged record explaining why, never stdout (p4d relays a trigger's stdout to the connected client), and degrades to a silent no-op if sdp_structured_log.py hasn't been deployed yet (it ships in the same tarball as this trigger but via a different deploy path than the Ansible-templated copy of this same file - see the update-tgz.sh changelist that follows this one). Extends rotate_last_run_logs/remove_old_logs to cover the new .jsonl siblings, including adding verify_shelves.log/.jsonl to the KEEPLOGS cleanup list - it was absent from that list entirely before (masked by the rm -f bug this same change removes), so without this addition it would have grown unbounded. |
||
| #13 | 32426 | Russell C. Jackson (Rusty) | Claude ai updates. | ||
| #12 | 32388 | Russell C. Jackson (Rusty) | Updates using Claude.ai to clean up the code, reduce duplication, enhanace security, and use current standards. | ||
| #11 | 32364 | Russell C. Jackson (Rusty) | Forgot about p4d_edgerep type | ||
| #10 | 32349 | Russell C. Jackson (Rusty) | Changed to use SERVERTYPE for checks. | ||
| #9 | 32346 | Russell C. Jackson (Rusty) | Changed logic to only call set_vars if the variables are missing. | ||
| #8 | 31923 | Russell C. Jackson (Rusty) | Removing set -x | ||
| #7 | 24116 | Russell C. Jackson (Rusty) | Added check for proxy server. | ||
| #6 | 24078 | Russell C. Jackson (Rusty) |
Added check to see if directory exists. Exit silently if not. |
||
| #5 | 22981 | Russell C. Jackson (Rusty) | Made files writeable so they are easier to update once on the server. | ||
| #4 | 22799 | Russell C. Jackson (Rusty) |
Removed () from set_vars call. Cut and paste mistake. |
||
| #3 | 22719 | Russell C. Jackson (Rusty) | Added call to source backup_functions.sh and run set_vars. | ||
| #2 | 22718 | Russell C. Jackson (Rusty) | Added check to make sure we are not on an edge server. | ||
| #1 | 22693 | Russell C. Jackson (Rusty) |
Branched a Unix only version of the SDP. Removed extra items to create a cleaner tree. Moved a few items around to make more sense without Windows in the mix. |
||
| //guest/perforce_software/sdp/dev/Server/Unix/p4/common/bin/run_if_replica.sh | |||||
| #1 | 22686 | Russell C. Jackson (Rusty) | Two small scripts to allow using a single crontab for all servers by putting all the entries into one file and only running based on server type. | ||