#==============================================================================
# Server Deployment Package (SDP) State File
#------------------------------------------------------------------------------
# The state file tracks the usability status of the 3 sets of SDP databases,
# Live, Offline, and Extra.  The SDP requires 3 sets of DBs to maintain
# a consistent RTO in all failure scenarios, even if failures occur at
# untimely moments during execution of certain long-running SDP processes,
# such as the regeneration of offline databases from a checkpoint.
#
# Design Mantra:
#  At no time are there less than 2 usable copies of databases, with
#  the difference between them being no more than the current/active
#  journal file.
#
#   Tag        Description           Location
# --------  -----------------   -----------------
# Live      Live databases      /p4/n/root
# Offline   Offline databases   /p4/n/offline_db
# Extra     Extra copy of DBs   /p4/n/root/save
#
# This file documents States and Transitions possible for the SDP.

# States are normally updated by SDP processing.  However, there are a
# few scenarios where human intervention to manually adjust a state file
# is required.
#
# The remainder of this file documents States, Transitions, and Manual
# State File Adjustment Considerations.
# 
#==============================================================================

#==============================================================================
# STATES
#==============================================================================
# Nominal States:  This represents the state of the SDP in between
# runs of daily, weekly, and manual/live checkpoint operations.
#------------------------------------------------------------------------------
# INITIAL:OK:NR:NR
# INITIAL applies when teh SDP is first installed.  The Offline and 
# Spare
#
# READY:OK:OK:OK
# Normal in-between-long-running-processses state, when all sets of DBs
# are usable to support various recovery procedures.

#------------------------------------------------------------------------------
# Work-In-Progress (WIP) States.  These states apply during the execution of
# certain long-running SDP processes.
#------------------------------------------------------------------------------
# LIVE_START:OK:NR:NR   - Preliminary processing. DBs not touched.
# LIVE_WIP_1:OK:NR:NR   - Live checkpoint in progress.
# LIVE_WIP_2:OK:NR:NR   - Offline DBs being made. Extra copy in 'save'.
# LIVE_WIP_3:OK:OK:NR   - Extra copy of DBs being copied to 'save'
# DAILY_START:OK:OK:OK  - Preliminary processing. DBs not touched.
# DAILY_WIP_1:OK:NR:OK  - Offline DBs being made. Extra in 'save'.
# DAILY_WIP_2:OK:OK:NR  - Extra copy of DBs being copied to 'save'.
# WEEKLY_START:OK:OK:OK - Preliminary processing.  DBs not touched.
# WEEKLY_WIP_1:OK:OK:NR - Live DBs moved to 'save', old save removed.
# WEEKLY_WIP_2:OK:NR:OK - Live DBs ready.  Dumping checkpoint from 'save'.
# WEEKLY_WIP_3:OK:OK:NR - Extra copy of dbs being copied to 'offline_db'.

#------------------------------------------------------------------------------
# Exception States
#------------------------------------------------------------------------------
# RESET:NR:OK:OK
# The RESET state must be set manually.  This state should be set when the
# live databases are suspected of being corrupt, e.g. after a sudden power
# outage, and if a failover to the HA node is not being performed.

#==============================================================================
# TRANSITIONS
#==============================================================================
# Transition - Live Checkpoint (live_checkpoint.sh)
# INITIAL -> LIVE_START -> LIVE_WIP_1 -> LIVE_WIP_2 -> LIVE_WIP_3 -> READY
# READY -> LIVE_START -> LIVE_WIP_1 -> LIVE_WIP_2 -> LIVE_WIP_3 -> READY

# The LIVE_START state applies immediately when live_checkpoint.sh starts.

# The LIVE_WIP_1 state applies when live databases are being checkpointed.

# The LIVE_WIP_2 state applies when the Offline databases are being generated.

# The LIVE_WIP_3 state applies after the Offline databases are usable, and
# while the Extra copies are still being generated.

#------------------------------------------------------------------------------
# Transition - Daily Offline Checkpoint (daily_backup.sh)
# READY -> DAILY_START -> DAILY_WIP_1 -> DAILY_WIP_2 -> READY

# The DAILY_START state applies immediately when daily_backup.sh starts.
#
# The DAILY_WIP_1 state applies only during the portion of processing when
# the Offline databases are not usable, which is the case when databases
# in the offline_db folder are being regenerated from a checkpoint.
# Should a sudden power outage occur when these databases are incomplete,
# the Extra databases remain usable for a quick recovery.

# The DAILY_WIP_2 state applies after checkpoint has been regenerated and
# the Offline databases

#------------------------------------------------------------------------------
# Transition: Weekly Offline Checkpoint (weekly_backup.sh)
# READY -> WEEKLY_START -> WEEKLY_WIP_1 -> WEEKLY_WIP_2 -> WEEKLY_WIP_3 -> READY

# The WEEKLY_START state applies immediately when weekly_backup.sh starts.

# The WEEKLY_WIP_1 state applies for the brief moment in time when the
# live databases are unsuable during initial processing of weekly_backup.sh,
# when p4d is actually offline.

# The WEEKLY_WIP_2 state applies after the Offline databases are usable, and
# while the Extra copies are still being generated.

# The WEEKLY_WIP_3 state applies after the Offline databases are usable, and
# wthe Extra copies are just being copied from offline_db.

#------------------------------------------------------------------------------
# Exception Scenarios:

# When the core SDP scripts (p4d_1_init, live_checkpoint.sh, daily_backup.sh,
# and weekly_backup.sh) start, they check to see that the SDP is in an
# acceptable state, and abort otherwise.
#
# This could happen if, for example, a power outage occurred while checkpoints
# in the offline_db folder were being regenerated from a checkpoint.  The
# state file would indicate that the Offline databases aren't ready and
# cannot be used.
#
# This could also happen if two human administrators attempt to execute
# commands simultaneously, e.g. two live_checkpoint.sh are attempted at the
# same time by two non-coordinating administrators tyring to fight the same
# fire after a failure of some kind.

#==============================================================================
# State File Format
#------------------------------------------------------------------------------
# A state file consists of a series of one-line state entries.  The last
# line/entry in the state file represents the current state.  Previous
# lines are for historical reference only.  The state file lives on the
# /depotdata volume, with the implication that it is on a SAN and writes
# to it are immediately visible on all hosts in the LAN environment.i
# It is automatically rotated out by daily_backup.sh whenever it exceeds
# some number of lines (e.g. 500), to avoid performance impact.
#
# State entries start with a prefix of "SDP:"; other lines are ignored.
# State entries contain a series of colon-delimited fields:
#
# SDP:Host:StateName:LiveDBStatus:OfflineDBStatus:SpareDBStatus:Timestamp
#
# Host is $THIS_HOST in p4_vars, as returned by 'hostname -s'.
#
# The *DBStatus field values are: "OK" and "NR" (Not Ready, i.e. not
# guaranteed to be ready).
#==============================================================================


#==============================================================================
# MANUAL ADJUSTMENT
#==============================================================================
# Manual State File Adjustment Considerations.
#------------------------------------------------------------------------------
#
# Rule #1: First, always confirm with other administrators that only one
# person is executing SDP scripts at any given time.  Even though the
# SDP has some protection against running dangerous commands concurrently,
# human coordination is always wise.
#
# Rule #2: Adjust the state file by adding a new State entry, using the
# set_sdp_state.sh, providing a new state name and colon-separated "OK"/"NR"
# values for the 3 database status.  For example, to confirm that all is
# well manually, do:
#
#    set_sdp_state.sh READY:OK:OK:OK
# 
# This appends a new state entry to the state file that looks like:
#
#    SDP:scm01:READY:OK:OK:OK:2011-12-13-012202
#
# Note that the prefix including the hostname, and the timestamp suffix,
# are generated by set_sdp_state.sh.
#
# The state file is in the 'tmp' folder for the appropriate instance of
# Perforce, e.g:
#
#   /p4/1/tmp/SDP.state
#
# Rule #3: If you are unsure of the state of the live databases, e.g. after
# a sudden power outage, adjust the state file before calling p4_1_init to
# start Perforce.  THIS IS ONLY NECESSARY if you plan to do a local failover;
# that is, if you do not intend to fail over to the HA node.
#
# Adjust the state file by adding a new State entry.  The new entry will be
# similar to the previous, but but change the field representing the status
# of the live databases to "NR" (not ready).  For example, if the last State
# entry is:
#
#    SDP:scm01:DAILY_WIP_2:OK:OK:NR:2011-12-13-042314
#
# Do this to create a new entry:
#
# set_sdp_state.sh DAILY_WIP_2:NR:OK:NR
#                              ^^ Changed!
#
# Special Case: If the State Name was READY, change it to LOCAL_FAILOVER, in
# addition to changein the status value for the live databases to NR.
# That is, if the last State entry is:
# 
#    SDP:scm01:READY:OK:OK:OK:2011-12-13-042314
#
# Do this to create a new entry:
#
#    set_sdp_state.sh LOCAL_FAILOVER:NR:OK:OK
#
# Rule #4:  If (say, after a sudden power outage), you find that SDP state
# file is is beyond the state where offline checkpoints were being generated,
# e.g. LIVE_WIP_2
# That means the Offline databases are OK.  You can regenerate the Extra
# databases from the most recent checkpoint, and then manually reset the
# SDP state by running:
#
#   set_sdp_state.sh READY:OK:OK:OK
#
# If you find that the SDP state file is in one of the *WIP_1 states, that
# indicates that the Offline databases not usable, and the Extra databases
# must be used instead.