USAGE for bbi.sh v3.5.8: bbi.sh -c -P [-w ] [-R] [-C] [-J] [-L ] [-si] [-v] [-T] [-n] [-D] [-S] or bbi.sh [-h|-man|-V] DESCRIPTION: This is the Perforce Baseline and Branch Import Tool, bbi.sh v3.5.8. This tool implements the Basline & Branch Import (BBI) migration strategy described in the document "Legacy SCM Migration Strategies": https://swarm.workshop.perforce.com/files/p4bbi/main/docs/LegacySCM-MigrationStrategies.pdf This strategy has been used by many companies over the years to quickly migrate from any number of legacy SCM systems to Perforce Helix Core. This strategy has been used by many companies over the years to quickly migrate from any number of legacy SCM systems to Perforce. This tool is savvy with Perforce, but does not undertand your legacy system. You define the interesting history to be imported by defining BBI config files, discussed below. This script is self-logging. That is, all output displayed on the screen (stdout and stderr) is simultaneously captured in a log file. You do not need to run this script with redirection operators like '> log' or '2>&1', and do not need to use 'tee.' The default log file is: /Users/ttyler/pub/old_p4bbi/main/logs/bbi...log BBI CONFIG FILES: The BBI config file defines a distilled, hand-crafted view of the history that you'd like to appear in Perforce. Typically one BBI config file is created for each product/component/module you plan to import. Examine the /Users/ttyler/pub/old_p4bbi/main/BBIConfigFileFormat.txt for detailed information on the BBI config file format. Also see sample BBI config files in the sample_cfg dirctory, FGS.Streams.bbi.cfg and FGS.Classic.bbi.cfg. These sample files illustrate the format. COMMAND LINE OPTIONS: -c Specify the path to your hand-crafted BBI config file. See documentation below on the format of this file. This argument is required (unless you are running the test suite with '-T'). -P Specify P4CONFIG value to use for imports. This must reference the target Perforce server. See additional details below. This argument is required (unless you are running the test suite with '-T'). -w Specify a path with enough space to hold staged copies of your baselines. The default is: /Users/ttyler/pub/old_p4bbi/main/w/.. Where is the import name defined with the NAME tag in the BBI config file, is the current process id, and (combined with ) gives a likely unique folder. -J Specify '-J' to run the 'p4 obliterate' command to remove files exepcted to be garbage. The file /Users/ttyler/pub/old_p4bbi/main/BBIJunkFiles.txt contains a list of file patterns defining junk to be obliterated. If '-J' is not used, it can be done as a separte step later by running: -x /Users/ttyler/pub/old_p4bbi/main/BBIJunkFiles.txt obliterate WARNING: Obliteration can have a large performance impact on a live Production server. See 'p4 help obliterate' and contact Perforce Support for more information if this is a concern. There are options to running 'p4 obliterate' directly on a live server (offline obliterations). -T Execute the BBI Test Suite, as documented in the EXAMPLES below. This bootstraps a local P4D instance on port 8674, and imports the Classic and Streams imports into it. When this option is used, the '-c' and '-P' flags, normally required, are neither necessary or allowed. -v Set verbosity 1-5 (-v1 = quiet, -v5 = highest). -L Specify the path to a log file, or the special value 'off' to disable logging. By default, all output (stdout and stderr) is captured in a log file named: /Users/ttyler/pub/old_p4bbi/main/logs/bbi...log -R Attempt rename detection. By default, this script calls the 'p4 reconcile' command in multiple passes, processing adds, deletes, and edits separately. This has the effect of preventing 'p4 recocnile' from attempting to detect renames, which 'p4 reconcile' would do by default if adds and deletes were processed in a single call, subject to certain server configurables governing reconcile behavior. The configurables dm.status.matchlines and dm.status.matchsize (described in 'p4 help undoc' as of P4D 2016.1) govern rename detection behavior. Rename detection is based on heuristic guesswork, but typically yields accurate results. If it guesses it wrong, i.e. misclassisifes a file action as a rename that wasn't or vice versa, the damage is limited to consuing history -- file lists and contents are still correct. However, with very large data sets and certain data patterns, e.g. deletion of many files, rename detection could take an onerous amount of time. Having a changelist with a big enough number of adds and deletes is rare with normal Perforce usage, but is more likely in the case of a BBI import, where each changelist can represent the sum of months of work and involve hundrededs of thousands of files. The heuristic rename detection has non-linear performance characteristics, which can result in imports taking an excessively long time while performing file diff comparisons on all files opened for add/delete to apply it rename detection logic. In one case while processing 0.75M files, an import of a baseline took 2 hours without attepting rename detection, and over 40 hours before being terminiated with rename detection. So, use '-R' with care -- be aware of potential run away performance if used. That said, the results when it runs are generally wonderful, a more faithful representation of actual history (without the work of using the explicit RENAME tag in the BBI config file, which is the other way to capture renames). -C Check the BBI config file and then stop. This can be used to verify the syntax of the BBI config file without starting an actual import. -si Operate silently. All output (stdout and stderr) is redirected to the log only; no output appears on the terminal. This cannot be used with '-L off'. -n No-Op. Doesn't run commands that affect data. This is for debugging purposes. -D Set extreme debugging verbosity. -S Step mode - if running interactively prompt user to enter return before progressing to the next Config file item HELP OPTIONS: -h Display short help message -man Display man-style help message -V Dispay version info for this script and its libraries. P4CONFIG FILE: In addition to the BBI config file, you must provide a P4CONFIG file that references the Perforce server you want to import into. The value *must* be an absolute path, and the specified file must exist. The P4CONFIG file MUST define: * P4PORT=your_target_server:P4PORT * P4USER=p4import (this use must be a super user) The P4CONFIG file may define: * P4TICKETS (if the target server requires authentication). * P4TRUST (if the target server is SSL enabled.) * P4IGNORE (optional but recommend. A sample P4IGNORE file is provided, p4ignore.bbi. This defines files to avoid adding to Perforce. The P4CONFIG file *MUST NOT* define these settings: * P4CLIENT. An import client workspace is generted dynmically by this script. DEPENDENCIES: The 'p4' and 'p4d' binaries appropriate to this platform must be copied to /Users/ttyler/pub/old_p4bbi/main/bin. You may be able to get them with commands like these samples: cd /Users/ttyler/pub/old_p4bbi/main/bin wget ftp://ftp.perforce.com/perforce/r15.2/bin.linux26x86_64/p4 wget ftp://ftp.perforce.com/perforce/r15.2/bin.linux26x86_64/p4d chmod +x p4 p4d Alternately, you can put symlinks for 'p4' and 'p4d' in /Users/ttyler/pub/old_p4bbi/main/bin. The 'rsync' and 'perl' utilities must also be available in the PATH. This uses modern bash shell featues, mainly an associative array. This is available on modern Unix systems and Linux distros, but not installed by default on Mac OSX systems (at least not as of Yosemite). ENVIRONMENT SETUP: To prepare your shell environment, first cd to the directory where the p4bbi.*.tgz file was extracted (typically /p4/p4bbi). Then source in the env.sh file, like so: cd /install/directory/for/p4bbi source env.sh This works if your shell is /bin/bash. Alternately, if you execute this script from the directory containing env.sh, you do not need to source env.sh. Sourcing the env.sh using bash file is recommended, as it allows you to run from anywhere. In addition to the standard /p4/p4bbi/env.sh file, you can add your own local environment file, /p4/p4bbi/env.local.sh. If this file exists, it will be sourced by the master env.sh file. This can be used to make PATH adjustments or other site-specific environment settings. In bash, the '.' (dot) command is shorthand for the 'source' command. TEST SUITE: To execute the BBI Test Suite: bbi.sh -T or with more details: bbi.sh -T -v5 Test imports appear on a Helix Server on the local host, running on port . To play with a test import on the command line, set the P4CONFIG variable as in this bash-syntax example: export P4CONFIG=/Users/ttyler/pub/old_p4bbi/main/sample_cfg/p4config.test Or view the import with P4V. EXAMPLES: All examples assume the ENVIRONMENT SETUP is complete. To step thru an import, one action at a time, interactively: bbi.sh -c MyProject.bbi.cfg -P /Users/ttyler/pub/old_p4bbi/main/p4config.myproj -S To preview an import: bbi.sh -c MyProject.bbi.cfg -P /Users/ttyler/pub/old_p4bbi/main/p4config.myproj -n Remove the '-n' to execute a live import. CONSIDERATIONS: == Typemap == The 'typemap' should be configured manually, to help Perforce determine file type preferences. While Perforce's built-in type detection is pretty good for base file types, it does not apply file type modifers you may desire, such as exclusive checkout for binary files like *.pdf files. == Case Sensitivty == The case senitivty setting of your target Perforce server should be considered. In particular, if you are targeting a Windows (case-insensitive) server from a Linux machines, there may be issues. == Unicode Mode == The Unicode mode of your server should be considered, especially if you have Unicode characters in pathnames. See www.perforce.com for details on Unicode Support. Files with Unicode in the pathnames may not be handled well, and have not been tested. LIMITATIONS: Files with '...' in the name cannot be added. Rename detection is available as a feature of 'p4 reconcile'. Pratically speaking, the rename detection is very useful and accurate. However, it does use heuristics: the logic that determines if a file was renamed (as opposed to one file deleted and a new one added) is essentially an educated guess, based on the content difference between old and new being close enough. If targeting Winwdow Perforce server, symlinks are supported only from Window Vista and later (when Windows added native OS support for symlinks). Hard links cannot be added to Perforce. (Symlinks are handled normally). If the delta between one baseline and another includes certain complex symlink refactoring, such as replacing a symlink with a file of the same name (or vice versa), you must artificially create additional baselines to 'split' such activity up into two baselines, one basline removing the symlink, and the next baseline addding the file that replaced it. Typically the 'extra' baseline artifically injected between the two real ones will contain only symlink refactoring changes. UNNATURAL DEPENDENCIES: For normal usage, this script does not require super user (i.e. 'root' access). However, in some cases while processing the UPDATE action, when importing tarfiles of Crytpo libraries, permissions are set in such a way that an 'rm -rf' by the regular OS user who extracted the tar file fails, where a 'sudo rm -rf' of the same staging area succeeds. The removal as the regular acccount is always attempted first. If that fails, a second attempt to perform the removal is attempted using 'sudo', which of course can only succeed if the user has sudo access. For general import purposes, sudo access should not be required. TO DO: * Implement support for $N tags in changelist descriptions in the BBI config file. * Enhance P4CONFIG file verifications. These notes describe in detail what should and should not be listed in the P4CONFIG. But the actual check is limited, checking for existence of the file and super user access only. * This script is only tested on Unix/Linux systems. It can be made to run on Windows in a ported bash shell (bash 4.0+), and this has been done. Details of how to do it need to be captured in docs. Out-of-the-box Windows support is a goal, even if it introduces a dependency (e.g. on Git bash). * Document rename detection engine via 'p4 reconcile' and related configurables and potential performance issues for large-scale imports. Add flags to enable or disable. OF SHELLS AND SNAKES: This script is a rewrite of the venerable p4bbi.py, which remains available in th BBI toolkit. * Supports Streams. * Uses 'p4 reconcile' for simpler and faster processing. * Built-in rename detection as a feature of 'p4 reconcile'. * Improved error messages; easier to track down failures. * Faster implementation. Limitations of the bbi.sh vs. p4bbi.py: * This version will have a much tougher time handling Unicode than a Ruby/Python/Perl implementation would. * It will take more work to get this to run on Windows.