Using alternative file transfer from submit and pull in distributed environment. Requirements: submit from an edge server uses alternative file transfer from edge to commit server using an edge-content trigger. pull from an edge server uses alternative file transfer using pull-archive triggers. Setup for submit. p4 configure set lbr.autocompress=1 p4 configure set rpl.submit.nocopy=1 p4 configure set pull.trigger.dir=<location to write temp files> A user must be set up with the public keys and anything else needed to perform the copy. In the case of using Aspera, this would require ssl public keys or the path to a key, and a license or you set the Aspera password in the environment. edge-content trigger: this will invoke the copy commands using a list of files generated from its 'fstat -Ob' command. For the fstat command, it needs a user with admin privilege. For the copy command, it needs the user, target host, target path, source path. The changelist must be passed as an argument to the trigger, as well as any of the other above-mentioned parameters if they are not hard-coded in the trigger script. The trigger variable %triggerdir% has the value of pull.trigger.dir. How this works: The edge-content trigger invokes 'p4 fstat -Ob @=<shelvedChange>' in order to get a list of files to transfer. In addition, it filters out any lazy copies since those don't require transfer. It will invoke the copy command (in this case ascp) to transfer the files from the shelved change from the edge server to the commit server. If a file transfer failure is detected from the trigger, the error returned will cause the submit to fail. Upon success the submit continues. When rmtSubmitShelf is run, it will Stat() the files on the commit server to verify that the transfer occurred instead of performing the transfer. If any files are not found, an error is returned from submit. Setup for pull. p4 configure set lbr.autocompress=1 p4 configure set pull.trigger.dir=<location to write temp files> p4 configure set lbr.replica.notransfer=1 pull-archive trigger: This will invoke the copy command by reading the list of files from the file whose filename was passed as a parameter to the trigger script. For the copy command, it needs the user, target host, target path, source path. The trigger variable %triggerdir% has the value of pull.trigger.dir. The trigger variable %archiveList% has the name of the file containing the files to be copied. How this works: There may be several startup.* configurables using 'pull --trigger'. When file content should be pulled, each pull will try to grab a batch of work. Then it goes through the list of files to be pulled, creates a temp file and writes the files to be transfered to the temp file. If any min-size or max-size parameters are specified, the pull will use them to filter out files not meeting the constraint. It then invokes the pull-archive trigger with the file containing its file list. The pull-archive trigger invokes the copy command (in this case ascp) to transfer the files from the commit to the edge. Transfer errors are reported to the log file when rpl=4. Files that failed to transfer will be detected and transferred by regular pull threads (not --trigger). Files are checked for transfer error by using Stat() unless the trigger script reported an error. If the trigger reported an error, then the checksums are compared to determine if transfer failed. When lbr.replica.notransfer is configured, sync commands accessing files being pulled by the pull-archive triggers will wait for the pull transfer to finish rather than initiating another fetch mechanism.
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#2 | 27722 | C. Thomas Tyler |
Refinements to @27712: * Resolved one out-of-date file (verify_sdp.sh). * Added missing adoc file for which HTML file had a change (WorkflowEnforcementTriggers.adoc). * Updated revdate/revnumber in *.adoc files. * Additional content updates in Server/Unix/p4/common/etc/cron.d/ReadMe.md. * Bumped version numbers on scripts with Version= def'n. * Generated HTML, PDF, and doc/gen files: - Most HTML and all PDF are generated using Makefiles that call an AsciiDoc utility. - HTML for Perl scripts is generated with pod2html. - doc/gen/*.man.txt files are generated with .../tools/gen_script_man_pages.sh. #review-27712 |
||
#1 | 26652 | Robert Cowham |
This is Tom's change: Introduced new 'Unsupported' directory to clarify that some files in the SDP are not officially supported. These files are samples for illustration, to provide examples, or are deprecated but not yet ready for removal from the package. The Maintenance and many SDP triggers have been moved under here, along with other SDP scripts and triggers. Added comments to p4_vars indicating that it should not be edited directly. Added reference to an optional site_global_vars file that, if it exists, will be sourced to provide global user settings without needing to edit p4_vars. As an exception to the refactoring, the totalusers.py Maintenance script will be moved to indicate that it is supported. Removed settings to support long-sunset P4Web from supported structure. Structure under new .../Unsupported folder is: Samples/bin Sample scripts. Samples/triggers Sample trigger scripts. Samples/triggers/tests Sample trigger script tests. Samples/broker Sample broker filter scripts. Deprecated/triggers Deprecated triggers. To Do in a subsequent change: Make corresponding doc changes. |
||
//guest/perforce_software/sdp/dev/Server/Unix/p4/common/bin/triggers/externalcopy.txt | |||||
#1 | 22107 | Russell C. Jackson (Rusty) | External copy scripts and notes. |