$Id: //public/perforce/utils/pvcstop4/main/README.txt#4 $ PVCS to Perforce converter Latest status - S E Knop 2009/01/23: Complete rewrite of mkdepot.pl. Branching should now work for all cases. Leave branch_level on 2 in config.ini for best results. Needs P4Perl 2008.2 from the Perforce ftp site at ftp://ftp.perforce.com/perforce/r08.2 . This version requires PCLI to be installed - see notes at project home page: http://public.perforce.com/public/perforce/utils/pvcstop4/index.html Summary - Obtain and install a version of Perl 5.8 for your machine. If you are running a version of UNIX and for some reason don't already have Perl available, download http://www.perl.com/CPAN/src/latest.tar.gz and get your sysadmin to compile and install it. If you are running Windows you can download a precompiled binary from http://www.activestate.com/ FOR BEST RESULTS - run on Windows. Tested with version 5.8 only! - Ensure both Perforce and PVCS can execute correctly. It is best if PVCS has a simple or no configuration file. Note that archives with access lists may require you to run as the PVCS super user; either that or disable the access lists before running the converter. Test Perforce by running "p4 info". The Perforce server need not run on the same machine you are running the converter on, however it may be wise to do so to avoid network traffic. Make sure that PVCS commands such as "get" are found early in the PATH before any other commands which might also include a "get". - Be aware that if you are using PVCS through the GUI (or via the web interface, e.g. v6.7) then you can have some issues with moved or deleted files. For example, the web interface allows you to move files. The scripts now handle this, but require PCLI (PVCS Command Line Interface) to be installed as this is the only thing which knows about such info. Note that the PCLI may not be installed by default - it is an option and also requires at least PVCS 6.5 or greater (by 6.7.10 all PCLI commands were introduced). Manual is PCLI.pdf, this is located at: \books\vm\vmbooks\* - Untar or Unzip the converter on a disk large enough to hold all files in your PVCS archives, plus the space required for the "metadata". The directory you put the converter in must not be under the Perforce server root or the PVCS archive root! - Ensure your PVCS archives are organized in a sensible tree structure with a single root. It is better to have directories "c:\archive\projecta" and "c:\archive\projectb" than "c:\projecta\archive" and "c:\projectb\archive"; in the former case your root is "c:\archive", in the latter it is "c:\" and your depot will have useless "archive" directories. As well, it is important that branches are labelled, and files which you wish to be marked as deleted are labelled (see below for more details). - Change directory to the directory the converter was installed into and edit the file "config.ini", then run "perl convert.pl" - Note that it can take a significant amount of time to run. - Deleted files are currently handled by specifying them in a label. It should be possible to use PCLI to discover this information, but this ability has yet to be scripted. - There are also issues with "linked" files - these are not copied in both new locations. Performance =========== The script can take a long time (many hours) to run if you have a large PVCS repository. The script needs to parse the history of PVCS and then get individual versions of each file out and check them in to Perforce. This by its nature takes time. To ensure the fastest possible conversion: - Copy PVCS repository locally (and use a fast machine!) - Install P4Perl from ftp://ftp.perforce.com/perforce/r08.2. This is the Perl interface to Perforce and is significantly faster than executing p4.exe at the command line which is the default. This can reduce conversion times by 25 - 50%. It is highly recommended, however, that you read the following before attempting to do the conversion. Differences between PVCS and Perforce ------------------------------------- PVCS uses the same model as RCS: each "workfile" has associated with it an "archive file" which contains all revisions of the "workfile" as well as all information about those revisions - who checked each one in and when, and so on. We will refer to the latter information as the "metadata". PVCS archives may be located anywhere in the local filesystem; Perforce keeps all archive files in a centralized depot. Local files may be mapped from depot files to any desired location in the local filesystem. Perforce uses RCS format files to store revisions, but all metadata is stored in a single, central database. Perforce stores useful information that PVCS does not. Perforce uses the notion of an atomic change which may consist of many add, edit, and/or delete operations. It is guaranteed that either all of a change is visible, or none of it is. PVCS does not provide atomic change transactions; when checking in a number of files one is given the option of using the same change description for each, but the fact that the revisions were all logically part of one change is lost. PVCS archives contain all branched revisions in one archive file, with branch revisions being numbered based on the branching point (e.g. 1.2.1.0 is a branch from 1.2, 1.4.2.3.1.2 is a branch from 1.4.2.3, which itself is the second branch from 1.4). Perforce handles branches differently. In Perforce when you wish to create a "branch", you simply copy the files, typically to another directory. The recommended convention, and the convention used by the converter, is to make the second component of the depot pathname the branchname. Thus, when it is time for the main line of development to be released as v7, you would typically do a "p4 integrate //depot/main/... //depot/v7/..." which (since there aren't any files in //depot/v7 currently) would create copies of all files in //depot/main in //depot/v7. To refer to the mainline version of foo.c use //depot/main/foo.c; for the v7 branched copy, use //depot/v7/foo.c This works well because Perforce implements a lazy copy. Archive files are never duplicated; only the metadata changes. Merging operations are recorded in Perforce - if, for example, patches were made to v7 code these patches could be merged back into the main line of development with "p4 integrate //depot/v7/... //depot/main/..." Only files which have changes that have not been previously integrated into main will be integrated, and the integration will be recorded so that next time those changes will not have to be revisited. In PVCS there is no way of knowing that a revision was the result of a merge aside from noting that in the change description. Labels are commonly used in PVCS; they are available in Perforce but not usually needed. Perforce uses atomic change transactions, and thus the state of the depot or any part of it is easy to specify. Perforce keeps track of deleted revisions (obviating the need for a special "DELETED" label of some sort) and branches are simply differently named files (labelling branches is optional in PVCS). Possible Missing/Incorrect Information in PVCS metadata - incorrect timestamps due to improperly set clocks or clocks which have drifted relative to each other (time of PVCS timestamp is that of the machine on which it is run, not of a central machine) - no identification of checkins which are logically all part of the same change (identify by similar time + same author and change description) - no identification of branches which are logically all part of the same branch (hopefully identify by label, "floating" label is a branch label) - no identification of merged branches - no consistent method for identifying deleted files (hopefully identify by a label such as "OBSOLETE" or "DELETED") - incorrect file type information ("binary" files with GENERATEDELTA or EXPANDKEYWORDS) Restrictions and limitations Filenames may not contain slashes (/) or backslashes (\) (they would be misinterpreted as pathnames), ASCII unprintable characters, at signs (@) or hashes (#). These characters will be mapped to underscores. Userids and labels may not contain spaces or ASCII unprintable characters. Again, these characters will be mapped to underscores. The date format used by PVCS must be one of the following: dd mon yyyy hh:mm:ss mon dd yyyy hh:mm:ss where mon is the first three letters of the English month name. Change descriptions with the 30-dash separator line in them will be truncated. (you could get around this by running vlog -br for each revision in each archive, but that is a high price to pay..) The converter is not regularly tested against old server versions, and thus it is not guaranteed to always work against all older versions. Unless you have a good reason to use an old server version, you should run the converter against the latest version of Perforce. How does the converter work? The converter works in four phases: Phase I: extract PVCS metadata Phase II: improve the metadata (compact changes using heuristics) Phase III: execute appropriate PVCS and p4 commands to construct Perforce database Phase IV: niceties after the fact PVCS archives are a proprietary binary format, however the metadata is provided in a human readable form by the PVCS vlog command, and individual revisions can be extracted with the PVCS get command. The converter relies on PVCS branches being labelled with floating labels; if they are not labelled then a name is made up for the branch to hold them. If the unlabelled branches are not actually desired then they should be removed from the PVCS archive before running the converter (use "vdel"). The converter marks files as deleted immediately after the checkin of a revision which is marked in PVCS with a label indicating it should be deleted. When the converter runs it reads configuration values from the file "config". Only root must be supplied; the others have defaults. See the comments in "config" for a description of each option. How do I check the end result, and what does it all mean? If you want to be certain that the revisions stored in Perforce are correct, answer yes to the question "Do you wish to verify the result?" (If you say no and then change your mind, just run "perl verify.pl"). The verification step isn't done by default because it takes a long time, and because there is no portable way to determine whether earlier phases of the converter succeeded. NOTE: the verification step does *not* verify that all files were submitted with the correct file type. On Windows/NT submitting a "binary" file as "text" will cause either a verification failure (due to CRLF -> LF translation) or *no error* despite a truncated file. The latter error can occur because files opened as "text" files on NT consider ^Z (ASCII 26) to be an end of file marker. Thus, when the file is submitted, only bytes up to the first occurence of ^Z will be submitted. The solution for this is to force the file to be submitted as type binary by adding the filename extension to the "type_binary:" line in the config file. You can verify the file type the converter chose by examining the file "metadata/files". You can verify that all changes were successfully submitted by running "p4 changes -s pending"; it should produce no output. Look in the directory "metadata" for files containing a mapping of all PVCS archives and revision numbers to the associated Perforce filename and change number. The sorted, more human readable version is in the file "mapping"; the tabular version is in "mapping.ns" (ns for "not sorted"). Change numbers should not be confused with version numbers. To see the files and versions affected by a particular change, type p4 describe -s (Without the -s all differences between files involved in the change are displayed as well). To see the list of all changes (you may want to pipe this through more): p4 changes To see the list of labels: p4 labels To see the list of changes affecting a particular file: p4 filelog filename To list revisions associated with the label "labelname": p4 files @labelname A note about re-running the converter If you want to re-run the converter it would be best to erase the directory tree "data", so that there cannot be any conflicts between runs. If you don't want to keep the results of the previous conversion, delete all files in the server root and directories below it. James Strickland (previously james@perforce.com) Robert Cowham (robert@vaccaperna.co.uk) Sven Erik Knop (sknop@perforce.com)