bbi.sh.command_summary.txt #2

USAGE for bbi.sh v3.5.8:

bbi.sh -c <bbi_cfg> -P <P4CONFIG> [-w <working_dir>] [-R] [-C] [-J] [-L <log>] [-si] [-v<n>] [-T] [-n] [-D] [-S]

or

bbi.sh [-h|-man|-V]


DESCRIPTION:
	This is the Perforce Baseline and Branch Import Tool,
	bbi.sh v3.5.8.

	This tool implements the Basline & Branch Import (BBI) migration
	strategy described in the document "Legacy SCM Migration Strategies":
	https://swarm.workshop.perforce.com/files/p4bbi/main/docs/LegacySCM-MigrationStrategies.pdf

	This strategy has been used by many companies over the years to quickly
	migrate from any number of legacy SCM systems to Perforce Helix Core.

	This strategy has been used by many companies over the years
	to quickly migrate from any number of legacy SCM systems
	to Perforce.

	This tool is savvy with Perforce, but does not undertand
	your legacy system.  You define the interesting history to
	be imported by defining BBI config files, discussed below.

	This script is self-logging.  That is, all output displayed on the
	screen (stdout and stderr) is simultaneously captured in a log file.
	You do not need to run this script with redirection operators like
	'> log' or '2>&1', and do not need to use 'tee.'  The default log
	file is:

	/Users/ttyler/pub/old_p4bbi/main/logs/bbi.<NAME>.<DATESTAMP>.log


BBI CONFIG FILES:
	The BBI config file defines a distilled, hand-crafted view of the
	history that you'd like to appear in Perforce.  Typically one BBI
	config file is created for each product/component/module you plan
	to import.

  Examine the /Users/ttyler/pub/old_p4bbi/main/BBIConfigFileFormat.txt for
	detailed information on the BBI config file format.

	Also see sample BBI config files in the sample_cfg dirctory,
	FGS.Streams.bbi.cfg and FGS.Classic.bbi.cfg.  These sample files
	illustrate the format.

COMMAND LINE OPTIONS:
 -c <bbi_cfg>
	Specify the path to your hand-crafted BBI config file.  See
	documentation below on the format of this file.

	This argument is required (unless you are running the test
	suite with '-T').

 -P <P4CONFIG>
	Specify P4CONFIG value to use for imports.  This must
	reference the target Perforce server.  See additional
	details below.

	This argument is required (unless you are running the test
	suite with '-T').

 -w <working_dir>
	Specify a path with enough space to hold staged copies of
	your baselines.

	The default is: /Users/ttyler/pub/old_p4bbi/main/w/<name>.<pid>.<random_number>

	Where <name> is the import name defined with the NAME tag
	in the BBI config file, <pid> is the current process id,
	and <random_number> (combined with <pid>) gives a likely
	unique folder.

 -J	Specify '-J' to run the 'p4 obliterate' command to remove 
	files exepcted to be garbage.

	The file /Users/ttyler/pub/old_p4bbi/main/BBIJunkFiles.txt contains a list of
	file patterns defining junk to be obliterated.

	If '-J' is not used, it can be done as a separte step later
	by running:

	    -x /Users/ttyler/pub/old_p4bbi/main/BBIJunkFiles.txt obliterate

	WARNING: Obliteration can have a large performance impact on
	a live Production server.  See 'p4 help obliterate' and contact
	Perforce Support for more information if this is a concern.
	There are options to running 'p4 obliterate' directly on a live
	server (offline obliterations).

 -T	Execute the BBI Test Suite, as documented in the EXAMPLES
	below.  This bootstraps a local P4D instance on port 8674,
	and imports the Classic and Streams imports into it.

	When this option is used, the '-c' and '-P' flags, normally
	required, are neither necessary or allowed.

 -v<n>	Set verbosity 1-5 (-v1 = quiet, -v5 = highest).

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging.  By default, all output (stdout and stderr) is captured
	in a log file named:

	/Users/ttyler/pub/old_p4bbi/main/logs/bbi.<NAME>.<DATESTAMP>.log

 -R	Attempt rename detection.  By default, this script calls the
	'p4 reconcile' command in multiple passes, processing adds,
	deletes, and edits separately.  This has the effect of
	preventing 'p4 recocnile' from attempting to detect renames,
	which 'p4 reconcile' would do by default if adds and deletes
	were processed in a single call, subject to certain server
	configurables governing reconcile behavior.  The configurables
	dm.status.matchlines and dm.status.matchsize (described in
	'p4 help undoc' as of P4D 2016.1) govern rename detection
	behavior.

	Rename detection is based on heuristic guesswork, but typically
	yields accurate results.  If it guesses it wrong, i.e. misclassisifes
	a file action as a rename that wasn't or vice versa, the damage is
	limited to consuing history -- file lists and contents are still
	correct.

	However, with very large data sets and certain data patterns,
	e.g. deletion of many files, rename detection could take an onerous
	amount of time.  Having a changelist with a big enough number of adds
	and deletes is rare with normal Perforce usage, but is more likely in
	the case of a BBI import, where each changelist can represent the sum
	of months of work and involve hundrededs of thousands of files.  The
	heuristic rename detection has non-linear performance
	characteristics, which can result in imports taking an excessively
	long time while performing file diff comparisons on all files opened
	for add/delete to apply it rename detection logic.  In one case
	while processing 0.75M files, an import of a baseline took 2 hours
	without attepting rename detection, and over 40 hours before being
	terminiated with rename detection.

	So, use '-R' with care -- be aware of potential run away performance
	if used.  That said, the results when it runs are generally wonderful,
	a more faithful representation of actual history (without the work
	of using the explicit RENAME tag in the BBI config file, which is the
	other way to capture renames).

 -C	Check the BBI config file and then stop.  This can be used to
	verify the syntax of the BBI config file without starting an actual
	import.

-si	Operate silently.  All output (stdout and stderr) is redirected to
	the log only; no output appears on the terminal.  This cannot be used
	with '-L off'.
      
 -n	No-Op.  Doesn't run commands that affect data.  This is for
	debugging purposes.

 -D     Set extreme debugging verbosity.

 -S     Step mode - if running interactively prompt user to enter return
	before progressing to the next Config file item

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message
 -V	Dispay version info for this script and its libraries.

P4CONFIG FILE:
	In addition to the BBI config file, you must provide a P4CONFIG
	file that references the Perforce server you want to import into.

	The value *must* be an absolute path, and the specified
	file must exist.

	The P4CONFIG file MUST define:
	* P4PORT=your_target_server:P4PORT
	* P4USER=p4import (this use must be a super user)

	The P4CONFIG file may define:
	* P4TICKETS (if the target server requires authentication).
	* P4TRUST (if the target server is SSL enabled.)
	* P4IGNORE (optional but recommend.  A sample P4IGNORE file is
	  provided, p4ignore.bbi.  This defines files to avoid adding
	  to Perforce.

	The P4CONFIG file *MUST NOT* define these settings:
	* P4CLIENT. An import client workspace is generted dynmically by this script.

DEPENDENCIES:
	The 'p4' and 'p4d' binaries appropriate to this platform must be
	copied to /Users/ttyler/pub/old_p4bbi/main/bin.  You may be able to get them with commands like
	these samples:

	cd /Users/ttyler/pub/old_p4bbi/main/bin
	wget ftp://ftp.perforce.com/perforce/r15.2/bin.linux26x86_64/p4
	wget ftp://ftp.perforce.com/perforce/r15.2/bin.linux26x86_64/p4d
	chmod +x p4 p4d

	Alternately, you can put symlinks for 'p4' and 'p4d' in /Users/ttyler/pub/old_p4bbi/main/bin.

	The 'rsync' and 'perl' utilities must also be available in the PATH.

	This uses modern bash shell featues, mainly an associative array.
	This is available on modern Unix systems and Linux distros, but
	not installed by default on Mac OSX systems (at least not as of
	Yosemite).

ENVIRONMENT SETUP:
	To prepare your shell environment, first cd to the directory where
	the p4bbi.*.tgz file was extracted (typically /p4/p4bbi). Then
	source in the env.sh file, like so:

	   cd /install/directory/for/p4bbi
	   source env.sh

        This works if your shell is /bin/bash.  Alternately, if you execute
	this script from the directory containing env.sh, you do not need to
	source env.sh.  Sourcing the env.sh using bash file is recommended,
	as it allows you to run from anywhere.

	In addition to the standard /p4/p4bbi/env.sh file, you can add your
	own local environment file, /p4/p4bbi/env.local.sh.  If this file
	exists, it will be sourced by the master env.sh file.  This can be
	used to make PATH adjustments or other site-specific environment
	settings.
	

        In bash, the '.' (dot) command is shorthand for the 'source' command.

TEST SUITE:
	To execute the BBI Test Suite:
           bbi.sh -T

           or with more details:

           bbi.sh -T -v5

        Test imports appear on a Helix Server on the local host,
        running on port .  To play with a test import
	on the command line, set the P4CONFIG variable as in this
        bash-syntax example:

        export P4CONFIG=/Users/ttyler/pub/old_p4bbi/main/sample_cfg/p4config.test

        Or view the import with P4V.

EXAMPLES:
	All examples assume the ENVIRONMENT SETUP is complete.

	To step thru an import, one action at a time, interactively:
	   bbi.sh -c MyProject.bbi.cfg -P /Users/ttyler/pub/old_p4bbi/main/p4config.myproj -S

	To preview an import:
	   bbi.sh -c MyProject.bbi.cfg -P /Users/ttyler/pub/old_p4bbi/main/p4config.myproj -n

	Remove the '-n' to execute a live import.

CONSIDERATIONS:
	== Typemap ==
	The 'typemap' should be configured manually, to help
	Perforce determine file type preferences.  While
	Perforce's built-in type detection is pretty good for
	base file types, it does not apply file type modifers
	you may desire, such as exclusive checkout for binary
	files like *.pdf files.

	== Case Sensitivty ==
	The case senitivty setting of your target Perforce
	server should be considered.  In particular, if
	you are targeting a Windows (case-insensitive) server
	from a Linux machines, there may be issues.

	== Unicode Mode ==
	The Unicode mode of your server should be considered,
	especially if you have Unicode characters in pathnames.

	See www.perforce.com for details on Unicode Support.

	Files with Unicode in the pathnames may not be
	handled well, and have not been tested.

LIMITATIONS:
	Files with '...' in the name cannot be added.

	Rename detection is available as a feature
	of 'p4 reconcile'.  Pratically speaking, the
	rename detection is very useful and accurate.
	However, it does use heuristics: the logic that
	determines if a file was renamed (as opposed
	to one file deleted and a new one added) is
	essentially an educated guess, based on the
	content difference between old and new being
	close enough.

	If targeting  Winwdow Perforce server, symlinks are
	supported only from  Window Vista and later (when
	Windows added native OS support for symlinks).

	Hard links cannot be added to Perforce.  (Symlinks
	are handled normally).

	If the delta between one baseline and another includes
	certain complex symlink refactoring, such as replacing
	a symlink with a file of the same name (or vice versa),
	you must artificially create additional baselines
	to 'split' such activity up into two baselines, one
	basline removing the symlink, and the next baseline
	addding the file that replaced it.  Typically the
	'extra' baseline artifically injected between the
	two real ones will contain only symlink refactoring
	changes.

UNNATURAL DEPENDENCIES:
	For normal usage, this script does not require
	super user (i.e. 'root' access).  However, in some
	cases while processing the UPDATE action, when
	importing tarfiles of Crytpo libraries, permissions
	are set in such a way that an 'rm -rf' by the regular
	OS user who extracted the tar file fails, where a
	'sudo rm -rf' of the same staging area succeeds.

	The removal as the regular acccount is always attempted
	first.  If that fails, a second attempt to perform the
	removal is attempted using 'sudo', which of course
	can only succeed if the user has sudo access.  For
	general import purposes, sudo access should not
	be required.

TO DO:
	* Implement support for $N tags in changelist
	descriptions in the BBI config file.

	* Enhance P4CONFIG file verifications.  These notes
	describe in detail what should and should not be
	listed in the P4CONFIG.  But the actual check is
	limited, checking for existence of the file and
	super user access only.

	* This script is only tested on Unix/Linux systems.
	It can be made to run on Windows in a ported
	bash shell (bash 4.0+), and this has been done.
	Details of how to do it need to be captured in docs.
	Out-of-the-box Windows support is a goal, even if it
	introduces a dependency (e.g. on Git bash).

	* Document rename detection engine via 'p4 reconcile'
	and related configurables and potential performance
	issues for large-scale imports.  Add flags to enable
	or disable.

OF SHELLS AND SNAKES:
	This script is a rewrite of the venerable p4bbi.py, which
	remains available in th BBI toolkit.

	* Supports Streams.
	* Uses 'p4 reconcile' for simpler and faster processing.
	* Built-in rename detection as a feature of 'p4 reconcile'.
	* Improved error messages; easier to track down failures.
	* Faster implementation.

	Limitations of the bbi.sh vs. p4bbi.py:
	* This version will have a much tougher time handling
	  Unicode than a Ruby/Python/Perl implementation would.

	* It will take more work to get this to run on Windows.