maintenance #8

  • //
  • guest/
  • russell_jackson/
  • sdp/
  • Maintenance/
  • maintenance
  • View
  • Commits
  • Open Download .zip Download (2 KB)
#!/bin/bash
# This is an example maintenance script to run the recommended maintenance scripts on a weekly basis.
# You need to make sure you update the hard coded locations to match yours if you are not using the
# default locations.

export P4INSTANCE=${P4INSTANCE:-Undefined}
export P4INSTANCE=${1:-$P4INSTANCE}
if [[ $P4INSTANCE == Undefined ]]; then
   echo "Instance parameter not supplied."
   echo "You must supply the Perforce instance as a parameter to this script."
   exit 1
fi

export PATH=$PATH:/p4/sdp/Maintenance:./
source /p4/common/bin/p4_vars $P4INSTANCE
/p4/common/bin/p4login > /dev/null 2>&1

export LOG=/p4/sdp/Maintenance/${HOSTNAME}_${P4SERVER}_log.txt
export P4CLIENT=${HOSTNAME}_unload

cd "/p4/sdp/Maintenance" >> $LOG

# echo . > $LOG
# echo ======================================= >> $LOG
# echo Running unload_labels.py >> $LOG
# echo The time is: `date` >> $LOG
# unload_labels.py $P4INSTANCE >> $LOG 2>&1

echo ======================================= >> $LOG
echo Running unload_clients.py >> $LOG
echo The time is: `date` >> $LOG
unload_clients.py $P4INSTANCE >> $LOG 2>&1

### Disabled because Perforce doesn't update access times on the commit server!!! ###
# echo ======================================= >> $LOG
# echo Running delusers.py >> $LOG
# echo The time is: `date` >> $LOG
# delusers.py $P4INSTANCE >> $LOG 2>&1

echo ======================================= >> $LOG
echo Running email_pending_client_deletes.py >> $LOG
echo The time is: `date` >> $LOG
email_pending_client_deletes.py $P4INSTANCE >> $LOG 2>&1

# echo ======================================= >> $LOG
# echo Running email_pending_user_deletes.py >> $LOG
# echo The time is: `date` >> $LOG
# email_pending_user_deletes.py $P4INSTANCE >> $LOG 2>&1

echo ======================================= >> $LOG
echo Removing old unloaded clients >> $LOG
echo The time is: `date` >> $LOG
delete_unload_clients.py $P4INSTANCE >> $LOG 2>&1

echo ======================================= >> $LOG
echo Removing empty pending changes. >> $LOG
echo The time is: `date` >> $LOG
remove_empty_pending_changes.py $P4INSTANCE > /dev/null 2>&1

echo The time is: `date` >> $LOG
mail -s "Weekly maintenance log $(hostname) ${P4INSTANCE}" $MAILTO  < $LOG
# Change User Description Committed
#16 33803 Russell C. Jackson (Rusty) Cap maintenance client unloading at 200 workspaces per run without threshold alerts; pass the SDP instance and limit to unload_clients.py.
#15 33799 Russell C. Jackson (Rusty) Include P4PORT in cron diagnostics and SDP alert logging
#14 33759 Russell C. Jackson (Rusty) Commenting out the email tasks for now.
#13 33526 Russell C. Jackson (Rusty) Fix unconditional maintenance emails; add jsonl structured logging across Maintenance/

- Maintenance/maintenance and Maintenance/edge_maintenance (run daily via
  cron, despite the "weekly" naming) were emailing the full run log on
  every single run, success or not, because every step was guarded with
  `|| true` and never reported failure anywhere. Added a run_step helper
  that tracks real failures and wires them into sdp_set_status/
  sdp_set_error; the end-of-run email now only fires when something
  actually failed (subject changed to "... FAILED" to make that obvious).

- Every other script in Maintenance/ (39 Python scripts + create_p4_filelist.sh
  + email.sh; sdputils.py excluded as a pure library with no __main__) now
  emits the same jsonl structured-logging record as the cron scripts in
  common/bin, via sdp_structured_log.py's run() context manager (auto-emits
  success/failure on exit, sys.exit(N), or unhandled exception) or, for the
  two shell scripts, sdp_structured_log.sh's sdp_log_init (auto ERR/EXIT
  trap coverage).

- remove_empty_pending_changes.py: the initial pending-changes listing
  failure path did a bare `return` (swallowed by the auto-emit machinery,
  since a normal return maps to exit code 0/success); now returns 1, and
  per-change delete failures inside the loop also flip the overall exit
  code, so a partially-failed run is correctly logged as a failure instead
  of a silent success.

- pymail.py: usage() called a bare sys.exit() (exit code 0) for both the
  -h/help path AND real bad-invocation paths (missing required args,
  getopt errors) - meaning a bad invocation reported success both to the
  OS and to the structured log. usage() now takes an explicit code
  (default 0, preserving -h's exit-0 convention); the two real error call
  sites pass a nonzero code.
#12 33459 Russell C. Jackson (Rusty) Fix cron mail noise from p4login output and unconditional success emails; add sdp_log_init jsonl logging (with matching log rotation) to remaining SDP maintenance scripts; support multiple limits groups in update_limits.py; exempt super users from keep_group_unset.py trigger; remove dead crontab entries
#11 32442 Russell C. Jackson (Rusty) Added the pyenv environment set up to the maintenance scripts.
#10 31928 Russell C. Jackson (Rusty) Removed set -e from scripts.
#9 31836 Russell C. Jackson (Rusty) Added limit on the number of clients to unload
#8 30939 Russell C. Jackson (Rusty) Added delete_unload_clients.py to maintenance
#7 29737 Russell C. Jackson (Rusty) Redirect empty changes output to null
#6 29392 Russell C. Jackson (Rusty) Disabled delusers since Perforce doesn't update user access time on the commit server.
#5 29342 Russell C. Jackson (Rusty) Turning off unload_labels.py to avoid issues with dates not being updated from edge servers.
#4 26867 Russell C. Jackson (Rusty) Corrected issue with using a instance other than 1.
#3 24675 Russell C. Jackson (Rusty) Fixed bugs in sdputils.py and scripts using it.
Converted to standard 2 space spacing, removed copyright stuff.
#2 24075 Russell C. Jackson (Rusty) Added maintenance and edge_maintenance to crontab, created edge_maintenance and updated maintenance.
#1 22693 Russell C. Jackson (Rusty) Branched a Unix only version of the SDP.
Removed extra items to create a cleaner tree.
Moved a few items around to make more sense without Windows in the mix.
//guest/perforce_software/sdp/dev/Maintenance/maintenance
#6 16638 C. Thomas Tyler Routine merge down to dev from main using:
p4 merge -b perforce_software-sdp-dev
#5 16029 C. Thomas Tyler Routine merge to dev from main using:
p4 merge -b perforce_software-sdp-dev
#4 13906 C. Thomas Tyler Normalized P4INSTANCE to SDP_INSTANCE to get Unix/Windows
implementations in sync.

Reasons:
1. Things that interact with SDP in both Unix and Windows
environments shoudn't have to account for this obscure
SDP difference between Unix and Windows.  (I came across
this doing CBD work).

2. The Windows and Unix scripts have different variable
names for defining the same concept, the SDP instance.
Unix uses P4INSTANCE, while Windows uses SDP_INSTANCE.

3. This instance tag, a data set identifier, is an SDP concept.
I prefer the SDP_INSTANCE name over P4INSTANCE, so I prpose
to normalize to SDP_INSTANCE.

4. The P4INSTANCE name makes it look like a setting that might be
recognized by the p4d itself, which it is not.  (There are other
such things such as P4SERVER that could perhaps be renamed as
a separate task; but I'm not sure we want to totally disallow
the P4 prefix for variable names. It looks too right to be wrong
in same cases, like P4BIN and P4DBIN.  That's a discussion for
another day, outside the scope of this task).

Meanwhile:
* Fixed a bug in the Windows 2013.3 upgrade script that
was referencing undefined P4INSTANCE, as the Windows
environment defined only SDP_INSTANCE.

* Had P4INSTANCE been removed completely, this change would
likely cause trouble for users doing updates for existing
SDP installations.  So, though it involves slight technical debt,
I opted to keep a redundant definition of P4INSTANCE
in p4_vars.template, with comments indicating SDP_INSTANCE should be
used in favor of P4INSTANCE, with a warning that P4INSTANCE
may go away in a future release.  This should avoid unnecessary
upgrade pain.

* In mkdirs.sh, the varialbe name was INSTANCE rather than
SDP_INSTANCE.  I changed that as well.  That required manual
change rather than sub/replace to avoid corrupting other similar
varialbe names (e.g.  MASTERINSTANCE).

This is a trivial change technically (a substitute/replace, plus
tweaks in p4_vars.template), but impacts many files.
#3 11477 Russell C. Jackson (Rusty) Updated to use /usr/bin/env python

Added workshop header.

 Changed cfg to config.
#2 11464 Russell C. Jackson (Rusty) Current maintenance scripts.
#1 10638 C. Thomas Tyler Populate perforce_software-sdp-dev.
//guest/perforce_software/sdp/main/Maintenance/maintenance
#2 10464 Russell C. Jackson (Rusty) Corrected typos and added import sys.
#1 10148 C. Thomas Tyler Promoted the Perforce Server Deployment Package to The Workshop.