edge_maintenance #8

  • //
  • guest/
  • russell_jackson/
  • sdp/
  • Maintenance/
  • edge_maintenance
  • View
  • Commits
  • Open Download .zip Download (1 KB)
#!/bin/bash

# This script is for performing weekly maintenance from a Linux client.
export P4INSTANCE=${P4INSTANCE:-Undefined}
export P4INSTANCE=${1:-$P4INSTANCE}
if [[ $P4INSTANCE == Undefined ]]; then
   echo "Instance parameter not supplied."
   echo "You must supply the Perforce instance as a parameter to this script."
   exit 1
fi

. /p4/common/bin/p4_vars $P4INSTANCE

/p4/common/bin/p4login > /dev/null 2>&1

export LOG=/p4/sdp/Maintenance/${HOSTNAME}_${P4SERVER}_log.txt
export P4CLIENT=${HOSTNAME}_unload

echo The time is: `date` >> $LOG
cd "/p4/sdp/Maintenance" >> $LOG

echo . > $LOG
echo ======================================= >> $LOG
echo Running rm -r -f /p4/1/root/server.locks >> $LOG
echo The time is: `date` >> $LOG
echo . >> $LOG

rm -r -f /p4/${P4INSTANCE}/root/server.locks >> $LOG 2>&1

echo ======================================= >> $LOG
echo Running unload_clients.py >> $LOG
echo The time is: `date` >> $LOG
unload_clients.py >> $LOG 2>&1

echo ======================================= >> $LOG
echo Removing old unloaded clients >> $LOG
echo The time is: `date` >> $LOG
delete_unload_clients.py $P4INSTANCE >> $LOG 2>&1

echo ======================================= >> $LOG
echo Removing empty pending changes. >> $LOG
echo The time is: `date` >> $LOG

remove_empty_pending_changes.py $P4INSTANCE > /dev/null 2>&1

echo The time is: `date` >> $LOG

echo ======================================= >> $LOG

mail -s "${HOSTNAME} ${P4SERVER} Weekly edge maintenance log" $MAILTO < $LOG
# Change User Description Committed
#17 33803 Russell C. Jackson (Rusty) Cap maintenance client unloading at 200 workspaces per run without threshold alerts; pass the SDP instance and limit to unload_clients.py.
#16 33799 Russell C. Jackson (Rusty) Include P4PORT in cron diagnostics and SDP alert logging
#15 33760 Russell C. Jackson (Rusty) Commenting out email warnings for now.
#14 33526 Russell C. Jackson (Rusty) Fix unconditional maintenance emails; add jsonl structured logging across Maintenance/

- Maintenance/maintenance and Maintenance/edge_maintenance (run daily via
  cron, despite the "weekly" naming) were emailing the full run log on
  every single run, success or not, because every step was guarded with
  `|| true` and never reported failure anywhere. Added a run_step helper
  that tracks real failures and wires them into sdp_set_status/
  sdp_set_error; the end-of-run email now only fires when something
  actually failed (subject changed to "... FAILED" to make that obvious).

- Every other script in Maintenance/ (39 Python scripts + create_p4_filelist.sh
  + email.sh; sdputils.py excluded as a pure library with no __main__) now
  emits the same jsonl structured-logging record as the cron scripts in
  common/bin, via sdp_structured_log.py's run() context manager (auto-emits
  success/failure on exit, sys.exit(N), or unhandled exception) or, for the
  two shell scripts, sdp_structured_log.sh's sdp_log_init (auto ERR/EXIT
  trap coverage).

- remove_empty_pending_changes.py: the initial pending-changes listing
  failure path did a bare `return` (swallowed by the auto-emit machinery,
  since a normal return maps to exit code 0/success); now returns 1, and
  per-change delete failures inside the loop also flip the overall exit
  code, so a partially-failed run is correctly logged as a failure instead
  of a silent success.

- pymail.py: usage() called a bare sys.exit() (exit code 0) for both the
  -h/help path AND real bad-invocation paths (missing required args,
  getopt errors) - meaning a bad invocation reported success both to the
  OS and to the structured log. usage() now takes an explicit code
  (default 0, preserving -h's exit-0 convention); the two real error call
  sites pass a nonzero code.
#13 33459 Russell C. Jackson (Rusty) Fix cron mail noise from p4login output and unconditional success emails; add sdp_log_init jsonl logging (with matching log rotation) to remaining SDP maintenance scripts; support multiple limits groups in update_limits.py; exempt super users from keep_group_unset.py trigger; remove dead crontab entries
#12 32442 Russell C. Jackson (Rusty) Added the pyenv environment set up to the maintenance scripts.
#11 31928 Russell C. Jackson (Rusty) Removed set -e from scripts.
#10 31836 Russell C. Jackson (Rusty) Added limit on the number of clients to unload
#9 31545 Russell C. Jackson (Rusty) Removed delete of unloaded clients from edge maintenance.
#8 30939 Russell C. Jackson (Rusty) Added delete_unload_clients.py to maintenance
#7 29744 Russell C. Jackson (Rusty) Redirect output to dev null to manage the size of the log file.
#6 29104 Russell C. Jackson (Rusty) Update remove_empty to use ztag and added the script to the edge_maintenance.
#5 28680 Russell C. Jackson (Rusty) Changed sh to bash
#4 28419 Russell C. Jackson (Rusty) Fixed edge maintenance log path.
#3 24713 Russell C. Jackson (Rusty) Removed specific client data.
#2 24675 Russell C. Jackson (Rusty) Fixed bugs in sdputils.py and scripts using it.
Converted to standard 2 space spacing, removed copyright stuff.
#1 24075 Russell C. Jackson (Rusty) Added maintenance and edge_maintenance to crontab, created edge_maintenance and updated maintenance.