pull.sh #1

  • //
  • guest/
  • perforce_software/
  • sdp/
  • dev/
  • Server/
  • Unix/
  • p4/
  • common/
  • bin/
  • triggers/
  • pull.sh
  • View
  • Commits
  • Open Download .zip Download (3 KB)
#!/bin/bash
#------------------------------------------------------------------------------
# Copyright and license info is available in the LICENSE file included with
# the Server Deployment Package (SDP), and also available online:
# https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/LICENSE
#------------------------------------------------------------------------------
# Read filename to get list of files to copy from commit to edge.
# Do the copy using ascp.
#
# configurable pull.trigger.dir should be set to a temp folder like /p4/1/tmp
#
# Startup commands look like:
# startup.2=pull -i 1 -u --trigger --batch=1000
# startup.3=pull -i 1 -u --trigger --batch=1000
#
# The trigger entry for the pull commands looks like this:
#
# pull_archive pull-archive pull "/p4/common/bin/triggers/pull.sh %archiveList%"
#
# There are some pull trigger options, but the are not necessary with Aspera.
# Aspera works best if you give it the max batch size of 1000 and set up 4 or 5 pull
# threads.
#
# pull --trigger options --min-size and --max-size.
#
# These options to 'pull --trigger' set a filesize range for
# a pull thread. Only the files whose data sizes fall in
# the range for that pull thread will be handled.
#
# For example:
# startup.2=pull -u -i 1 --trigger --batch=2 --min-size=240123905 (configure)
# startup.3=pull -u -i 1 --trigger --batch=5 --min-size=1 --max-size=229M (configure)
# 
# The ascp user needs to have ssl public keys set up or export ASPERA_SCP_PASS.
# The ascp user should be set up with the target as / with full write access to the volume where
# the depot files are located. The easiest way to do that is to use the same user that is 
# running the p4d service.
#
# Standard SDP environment is assumed, e.g P4USER, P4PORT, OSUSER, P4BIN, etc.
# are set, PATH is appropriate, and a super user is logged in with a non-expiring
# ticket.

set -u

function msg () { echo -e "$*"; }
function bail () { msg "\nError: ${1:-Unknown Error}\n"; exit ${2:-1}; }

[[ $# -eq 1 ]] || bail "Bad Usage!\n\nUsage:\n\t$0 %archiveList%\n\n"

LOGFILE=$LOGS/pull.log

fileList=$1

export ASPERA_SCP_PASS=xxxx

[[ -f $fileList ]] || { echo "$fileList missing! >> $LOGFILE"; exit 1; }

# Note: the -l parameter should be adjusted to how much bandwidth you want Aspera to use
# in your environment.
ascp -P 33001 -l 500M -d --mode=recv --file-pair-list=$fileList --user=$OSUSER --host=$P4MASTER --partial-file-suffix=asp / >> $LOGFILE 2>&1
if [[ $? -ne 0 ]]; then
  bail "$0: failed to execute the ascp command, contact Perforce admin.\n"
fi

exit 0

# Change User Description Committed
#7 29923 C. Thomas Tyler Updated HTML hyperlinks to use 'portal.perforce.com'.

This replaces currently broken links to 'answers.perforce.com' and
currently redirected links to 'community.perforce.com'.

#review-29924
#6 26851 Robert Cowham Fix typo in tmpfs /etc/fstab entry which stopped it working in the doc.
Mention in pre-requisites for failover and failover guide the need to review
OS Config for your failover server.
Document Ubuntu 2020.04 LTS and CentOS/RHEL 8 support. Note performance
has been observed to be better with CentOS.
Document pull.sh and submit.sh in main SDP guide (remove from Unsupported doc).
Update comments in triggers to reflect that they are reference implementations, not just examples. No code change.
#5 26773 Robert Cowham Move Aspera triggers back to supported section as referred to in KB and Sysadmin Guide.
Clarify description in file headers and link to KB article.
#4 26652 Robert Cowham This is Tom's change:

Introduced new 'Unsupported' directory to clarify that some files
in the SDP are not officially supported. These files are samples for
illustration, to provide examples, or are deprecated but not yet
ready for removal from the package.

The Maintenance and many SDP triggers have been moved under here,
along with other SDP scripts and triggers.

Added comments to p4_vars indicating that it should not be edited
directly. Added reference to an optional site_global_vars file that,
if it exists, will be sourced to provide global user settings
without needing to edit p4_vars.

As an exception to the refactoring, the totalusers.py Maintenance
script will be moved to indicate that it is supported.

Removed settings to support long-sunset P4Web from supported structure.

Structure under new .../Unsupported folder is:
   Samples/bin             Sample scripts.
   Samples/triggers        Sample trigger scripts.
   Samples/triggers/tests  Sample trigger script tests.
   Samples/broker          Sample broker filter scripts.
   Deprecated/triggers     Deprecated triggers.

To Do in a subsequent change: Make corresponding doc changes.
#3 26048 Robert Cowham Added doc notes for pull.sh and links to details.
Added pull_test.sh for easy local testing of the config.
#2 22616 Russell C. Jackson (Rusty) Updated comments about number of threads.
#1 22107 Russell C. Jackson (Rusty) External copy scripts and notes.
//guest/perforce_software/sdp/dev/Unsupported/Samples/triggers/pull.sh
#1 26652 Robert Cowham This is Tom's change:

Introduced new 'Unsupported' directory to clarify that some files
in the SDP are not officially supported. These files are samples for
illustration, to provide examples, or are deprecated but not yet
ready for removal from the package.

The Maintenance and many SDP triggers have been moved under here,
along with other SDP scripts and triggers.

Added comments to p4_vars indicating that it should not be edited
directly. Added reference to an optional site_global_vars file that,
if it exists, will be sourced to provide global user settings
without needing to edit p4_vars.

As an exception to the refactoring, the totalusers.py Maintenance
script will be moved to indicate that it is supported.

Removed settings to support long-sunset P4Web from supported structure.

Structure under new .../Unsupported folder is:
   Samples/bin             Sample scripts.
   Samples/triggers        Sample trigger scripts.
   Samples/triggers/tests  Sample trigger script tests.
   Samples/broker          Sample broker filter scripts.
   Deprecated/triggers     Deprecated triggers.

To Do in a subsequent change: Make corresponding doc changes.
# Change User Description Committed
#3 26048 Robert Cowham Added doc notes for pull.sh and links to details.
Added pull_test.sh for easy local testing of the config.
#2 22616 Russell C. Jackson (Rusty) Updated comments about number of threads.
#1 22107 Russell C. Jackson (Rusty) External copy scripts and notes.