README #1

cvs2p4 1.2.6	January 18, 2000.

==== INTRODUCTION    

This small set of tools provides a means for importing a CVS module
into Perforce.

It was developed for our use at Network Appliance, to convert our
product source code revision history from CVS history into Perforce.

As such it has sprouted some NetApp-specific features suited to our
special needs, but I have made an attempt to make these unobtrusive to
the general user.

Basically, it is patterned at a high level after the PVCS to
Perforce converter available on the Perforce web site, doing the
following steps during a conversion:

  - Scans the CVS repository to generate a metadata file;

  - Scans the metadata file to identify groups of RCS revisions
    that comprise Perforce changes;

  - Imports the revisions/log history into a Perforce depot, by
    running p4 client commands against a Perforce server (this is
    driven by the output of the previous phase);

  - Finally, (and optionally), generates a map of RCS revisions and
    the Perforce changes they belong to.


==== MANIFEST

After unpacking the distribution archive, use the MANIFEST script to
verify that you have all of the pieces.  The output should go
something like this:

rmg $ MANIFEST
  MANIFEST
  Artistic
  README
  NEWS
  bin/genmetadata
  bin/genchanges
  bin/dochanges
  bin/revmap
  lib/util.pl
  test/file,v
  test/config
  test/runtest
  test/norm
  test/metadata.good
  test/lines.good
  test/changes.good
  test/changes.good
  test/p4_changes_-l.good
  test/p4_describe.good

All ok


==== REQUIREMENTS    

I've tested this all *only* in the following configuration:

  - conversion client machine: Sun UltraSparc running Solaris 2.6
    (I'd expect Solaris 2.5 to be fine)

  - Perl 5.0 patch level 4 subversion 1, with working dbm support
    (i.e., dbmopen()/dbmclose() work). The scripts assume that perl
    will be found via $PATH. It must be a perl5!

  - RCS (actually, only the "co" command is used). Tested with
    RCS version 5.7.

  - Tested with

       p4	P4/SOLARIS/97.3/5047 (01/02/98)
       p4d	P4/SOLARIS/97.3/5047 (01/02/98)

       p4	P4/SOLARIS/98.2/7068 (1998/07/31)
       p4d      P4D/OSF/98.2/8102 (1998/10/16)

    *NOTE*: if you are using 97.3 or earlier, you should rename

       mv test/p4_describe.good test/p4_describe.good-r98.2
       mv test/p4_describe.good-r97.3 test/p4_describe.good

    to avoid spurious reports of failure during the final diff when
    running the test cases provided with this distribution.

    The ability to spoof the actual checkin times and users depends
    on using Perforce version 97.3 or later.

  (Note: I've tended to run with both the server and the converter/client
   sides on the same host, but I am unaware of any requirement to do
   so; it should work if you want to use different hosts for each side).


==== WHAT IT DOES

This converter will import a CVS module into Perforce, preserving the
branching structure seen in the RCS ,v file in the CVS repository, and
translating them into Perforce branches within the depot. As it
stands, it will only import RCS branches up through the highest
numbered revisions on branches that have branch tags referring to
them; thus, it will not necessarily bring *every* revision in the CVS
module into Perforce, but *will* bring in every revision leading up to
the current revision for every branch it imports. I think this is what
most people will want; if not, hack away.

Like the RCS -> Perforce converter available on the Perforce web site,
it applies heuristics to try and identify multiple changes in CVS that
are highly likely to comprise what would be seen as a single change in
Perforce, and does commit them as a single Perforce change. (The
heuristics are: checked in by the same user, proximal in time, and
bearing an identical log message).

It deals OK with files that are dead on the CVS trunk (I.e., where the
RCS ,v files are in the "Attic/".

The "UI" for the converter is not very slick, but for most people it's
a one-time kind of tool anyway. Feel free to improve it if you are so
inclined.

In general, I would call this "hackware"; I'm doing this minimal stab
at documenting it, and then giving it away, in hopes that somebody
will find it useful (Or perhaps only entertaining :-).


==== TESTING

I have included a *very* rudimentary automated test "suite", in the
test/ directory. You can use this to verify that it seems to work in
your environment.

To run it:

  1. Set up a virgin Perforce server;

  2. If you're going to run the client side of the converter as some
     other user than the one you used to set up the server in step
     (1)., use "p4 protect", and grant p4 superuser privileges to the
     user who will be running the client side of the converter.

  3. Edit test/config, and change the lines

       # p4 command location (If other than "/usr/local/bin/p4")
       #
       $P4             = "/u/p4/VERS/bin.solaris/p4";

       # Perforce server we're using.
       #
       $P4PORT = "cranford:1668";

     to reflect the actual location of your "p4" command and the
     server hostname and port that you are using.
     
  4. Run the tests with

       test/runtest

     This should run all of the conversion scripts on the test CVS
     module (well, file - it's a one-file module!), and then verify a
     few things by querying the Perforce server after the conversion
     is complete.

     If everything goes well, the end of the output should be
     something like:

       SUBMITR->:Change 22 updated.
       runtest> /u/p4/VERS/bin.solaris/p4 changes -l | /n/makita/...
       runtest> /u/p4/VERS/bin.solaris/p4 describe 1 2 3 4 5 6 7 ...
       runtest: ok

In this version, the converted CVS "module" consists of a single file,
but it does have a carefully constructed branching structure, intended
to verify that the converter does the right stuff with respect to
branching.


==== USAGE

1. Make a directory to hold all the glop for the importation, and create
   a config file, starting with test/config as a template:

     $ mkdir convdir; cp test/config convdir

   Edit the convdir/config file to reflect your locale and
   intent. (See the comments in the config file).

2. Set up a virgin Perforce server (well, it doesn't have to be a
   virgin, I guess, and if there's no volcano, then why bother?)  Make
   sure P4PORT in the config file is set to the hostname:port for this
   server.

   As described in the prep for running the tests, above, use "p4
   protect" if required, to grant the user who will be running the
   conversion scripts Perforce superuser privileges.

3. Run bin/genmetadata:

   It takes a single argument - the name of the directory where the
   "config" file resides. (It will create all intermediate, temp, and
   working files under this directory. By the end of the entire
   process, this will include a tree with the same topology as what
   you end up with in the imported depot, so, if you're converting a
   big repository, you might need considerable space in this
   directory).

     $ bin/genmetadata convdir
     genmetadata: rm -rf convdir/logmsgs.dir convdir/logmsgs.pag ...
     .
     . (filenames of each file in the CVS module, as they are scanned)
     .
     ===== Lines referenced:
     chupa
     curly
     ha         <- a list of branch tags encountered in the scan;
     larry         also saved to convdir/lines.
     shemp
     xxx

   This reads cvsdir/config to get its marching orders, then scans the
   CVS module for all ,v and Attic/,v files, creating:

     convdir/metadata      <- the extracted RCS/CVS metadata
     convdir/logmsgs.pag   <- An ndbm database
     convdir/logmsgs.dir   <-   of the log messages
     convdir/lines         <- A list of "codelines" (== branch tags)

   At this point, you may want to look at the list of branch tags
   encountered, (which was written to convdir/lines), edit the config
   file, setting WANTLINES to 1, and filling in the "<<LINES" here
   file with the names of the branches you care to import to Perforce;
   then, rerunning bin/genmetadata to rescan and pick up only those
   revisions you care about. (Things should be modified so genchanges
   honors the %WANTLINES hash, and then you wouldn't need to rerun
   genmetadata, sigh).

   
4. Run bin/genchages:

   Again, this takes a single argument - the dir name of the
   "conversion directory":

     rmg $ bin/genchanges convdir
     16354                    <- This counter spins as it's running,
                                 showing the number of changes
                                 processed. It will count up to the
                                 number of lines in the metadata file.

   This reads convdir/config and convdir/metadata, and writes
   convdir/changes.


5. Run bin/dochanges

   Verify that the Perforce server to receive the conversion is up,
   and that the user who will run "bin/dochanges? has superuser
   privileges.

   Also, if you are using the checkpointing feature (I.e., you have
   set a nonzero CHECKPOINT_INTERVAL in the config file), you must
   be able to do "rsh <p4dhost> -l <p4duser> <somecmd>" as the user,
   and from the host, which is going to run "dochanges".

   E.g., Let's say I'm running the p4 server on host "cranford", under
   login name "p4", and running "dochanges" as "rmg" on host "turok".

   Then user "rmg" on turok must be able to run "rsh cranford -l p4
   <cmd>". Typically, this will require an entry in ~p4/.rhosts on
   "cranford", with a line of the form "turok rmg".

   You can avoid all of this rigamarole by running with
   CHECKPOINT_INTERVAL set to 0, or by runnig the converter and the
   p4d on the same host as the same user. [At this time running with
   CHECKPOINT_INTERVAL *other* than 0 is useless anyway, as the
   "recover from checkpoint" stuff hasn't been implemented yet!]

   Finally, you might want to save a copy of the output with "tee".
   The output will look something like:

     $ bin/dochanges T 2>&1 | tee bin/dochanges.out
     dochanges: /bin/rm -rf /n/makita/users/rmg/proj/cvs2p4/convdir/p4
     dochanges: /u/p4/VERS/bin.solaris/p4 client -d dochanges
     Client 'dochanges' doesn't exist.
     dochanges: *** "/u/p4/VERS/bin.solaris/p4 client -d dochanges" ...
     Client dochanges saved.
     
     ========== change group 0
     rmg 971112190556 Tmodule/convdir/1.1 Exp shemp -
     === deletes
     ===adds/edits===
     dochanges: /u/p4/VERS/bin.solaris/p4 files /n/makita/users/rmg/...
     dochanges: /usr/local/bin/co -p1.1 Tmodule/T,v >/n/makita/users...
     Tmodule/T,v  -->  standard output
     revision 1.1
     dochanges: /u/p4/VERS/bin.solaris/p4 add -t ktext /n/makita/use...
     //depoconvdir/ontap/trunk/T#1 - opened for add
     dochanges: ...| /u/p4/VERS/bin.solaris/p4 submit -i >/tmp/p4_su...
     SUBMITR->:Change 1 created with 1 open file(s).
     SUBMITR->:Submitting change 1.
     SUBMITR->:Locking 1 files ...
     SUBMITR->:add //depoconvdir/ontap/trunk/T#1
     SUBMITR->://depoconvdir/ontap/trunk/T#1 - refreshing
     SUBMITR->:Change 1 submitted.
     dochanges: ...| /u/p4/VERS/bin.solaris/p4 change -i -f >/tmp/p4...
     SUBMITR->:Change 1 updated.
     ===branches===
     
     ========== change group 1
        .
        .
        .  (Extremely voluminous output. You may want to tweak the
        .    scripts to recduce it if you're tight on disk space
        .      and tee'ing the output!
        .
     
   Basically, that's it. When this command finishes, your CVS module
   has been imported to Perforce. It make take a while. Converting our
   product source (=~ 4400 ,v files, and 16000 individual Perforce
   changes) takes something like 16 hours. But hey, that represents
   over 5 years of checkins to CVS!...

6. If you want the RCS revision-to-Perforce change map, run:

     $ bin/revmap convdir
     <file> 3.2.2.2.2.1 18
     <file> 1.2.2.1.2.2 15
     <file> 1.2.2.1.2.1 14
     <file> 3.3.4.1 19
     <file> 3.2.2.2 17
     <file> 3.2.2.1 16
     <file> 1.2.2.2 13
     <file> 1.2.2.1 11
     <file> 3.5 20
     <file> 3.4 9
     <file> 3.3 7
     <file> 3.2 5
     <file> 3.1 4
     <file> 1.2 2
     <file> 1.1 1

   I.e., the 1.2.2.1.2.1 rev of <file>,v was imported to Perforce
   as part of Perforce change #14.

   (I don't have it making the RCS -> file#rev map yet, so to
   get that you need to indirect throught what "p4 change -n N"
   tells you).


==== SUPPORT

I can make absolutely no warranty that this will be helpful or even
nontoxic for you, nor any promise of support. I may deny that I
know anything about these tools.

On the other hand, I will probably be happy to act as a collector of
bug reports, and *may* be able to offer some help if you run into
problems. Can't hurt to ask...

  Richard Geiger
  - rmg@netapp.com
  January, 1998
  (revised January 2000, release 1.2.6)