FAQ for integrating Perforce with your build scripts

Contributed by Jeff Bowles

Note - the intended audience for this FAQ is a shell/perl hacker who writes or maintains local build scripts and wants to integrate Perforce into them.

How do I create a build label? What should I include in a label?

Pick a regular format for the label name, e.g. "bld_xxx_1" where 'xxx' is the name of the codeline you're building and '1' is just a counter, i.e. the first such label. Remember this format - you'll need it in the next question!
Make the label itself:

p4 label bld_xxx_1

It'll dump you into an editor, and you can change the default "Views:" lines from

//depot/...

to look solely at the codeline you want, e.g.

//depot/xxx/...

Save/Quit the editor.

Populate the label with the file/revision list you want:

p4 labelsync -l bld_xxx_1 //depot/xxx/...

p4 labelsync -l bld_xxx_1 //depot/...

In this second example, it's okay to include all files in the depot - it'll really only include files described in the label spec you edited when you ran "p4 label".

Always include all files [that are under source management] that are needed to recreate the build . If your build scripts aren't in codeline "xxx" you'll want to have the label specification include the directory where they are, and "p4 labelsync" needs to include them.

If I want to create a new label for each overnight build, how would I do it?

Make sure you're using a regular name for the label, e.g. "bld_xxx_yyy" where "xxx" is the codeline name and "yyy" is a counter.
Create the first label in the sequence, which would be "bld_xxx_1". See the previous question for that.
In the Unix shell, you would do the following (cut and paste):

   CodeLineName=xxx
   LabelSpec=tst_${CodeLineName}_
   toplabelnumber=`p4 labels | grep $LabelSpec | sed "s/^Label $LabelSpec//" | \
        sed "s/ .*//" | sort -r -n | head -1`
   # at this point, we know the most recent label in this codeline and just need
   # to create a new label and populate it.
   TopLabel=$LabelSpec$toplabelnumber
   echo Current Label is $LabelSpec$toplabelnumber
   if [ "$toplabelnumber" != "" ]
   then
        newlabelnumber=`expr $toplabelnumber + 1`
        NewLabel=$LabelSpec$newlabelnumber
        echo "New Label is $NewLabel"

        # the following two commands are from the previous FAQ item.
        p4 label -o -t $TopLabel $NewLabel | p4 label -i
        p4 labelsync -l $NewLabel //depot/...
        # It's best to obsolete labels that are kinda of old, so we
        # do it here.
        if [ "$toplabelnumber" -gt 10 ]
        then
                obsoletelabelnumber=`expr $toplabelnumber - 10`
                ObsoleteLabel=$LabelSpec$obsoletelabelnumber
                p4 label -d $ObsoleteLabel
        fi
   else
        echo "There was no label to match!" 1>&2
        exit 1
   fi

Perl hackers will look at the previous "Bourne shell" example and use it as an example of why you don't want to do string processing in "Bourne shell". One perl treatment of this follows:

$CodeLineName = "xxx";

$LabelSpec="tst_" . $CodeLineName . "_";

   @AllLabels = `p4 labels`;
   @ThisCodeLineLabels = sort(sortrevlabel grep(/$LabelSpec/, @AllLabels));
   $TopLabelLine = $ThisCodeLineLabels[0];
   $TopLabel = $1 if ($TopLabelLine =~ /^Label\s(\S+)\s/);
   $TopLabelNumber = $1 if ($TopLabel =~ /^\S+_(\d+)$/);

   print "The last label in this codeline is $TopLabel\n";
   $NewLabelNumber = $TopLabelNumber + 1;
   $NewLabel = "$LabelSpec$NewLabelNumber";
   print "New Label is $NewLabel\n";
   system("p4 label -o -t $TopLabel $NewLabel | p4 label -i");   # in real life, return code would be checked!
   system("p4 labelsync -l $NewLabel //depot/...");
   if ($TopLabelNumber > 10 )
   {
        $ObsoleteLabelNumber = $TopLabelNumber - 10;
        $ObsoleteLabel = "$LabelSpec$ObsoleteLabelNumber";
        system("p4 label -d $ObsoleteLabel");
   }

   sub sortrevlabel {
        my($labelnum_a, $labelnum_b) = ();
        $labelnum_a = $1 if ($a =~ /.*_(\d+)/);
        $labelnum_b = $1 if ($b =~ /.*_(\d+)/);
        return ($labelnum_b <=> $labelnum_a);
        }

The underlined part in the above fragments probably should have a "sed" command between the two 'p4 label' commands, to change the description for each label.

Should I use labels or change numbers to track the contents of overnight builds?

There's no right answer. The command

p4 changes -m1 //depot/xxx/...

will tell a build script what the top change number for codeline 'xxx' (assuming it's mapped there), so it can parse this output and use that in build logs and build e-mail. The developer would type:

p4 sync @12345

to pull over all files current as of change #12345, but that has a problem: it'll pull over all files [that his/her client maps in] as of that revision, not just the files for codeline 'xxx'. You can tell them to type:

p4 sync //depot/xxx/...@12345

but, by then, you've lost the simplicity that using the change numbers might've given you.

You should probably always use labels for release builds - when you create a release you'll often need to include an additional change to one particular file that you need to track, and you want the label to include exactly the files used to build the release on the build client.

So the release needs to be remade three months after it went out. What do I do?

You'll need a slightly-hacked version of your "get the source and do a build" script. The hacks are:

It shouldn't use or touch any labels except for the release label you've remembered to create when you made the release. Delete the lines in the script that create labels or run "p4 labelsync" on them.
Change the line in the script that does the "p4 sync -f ...#have" so that it looks like:

p4 sync -f //depot/xxx/...@labelname

where "xxx" is the name of the codeline, and "labelname" is the release label.

Bug your system administrators about whether compiler versions, OS releases, etc, have changed since the release was created. Listen carefully to their answers.
Run the script. It should create the original release faithfully, although most compilers embed timestamps within the .exe and .obj files, so you might not have byte-for-byte identical files.
Rerun your regressions to gain confidence in the newly-built product.

I like this overnight build and want to officially designate it as a "release" build. What should I do?

See the previous step - you just need to run "p4 label" (to create a label) and then "p4 labelsync" to the overnight label or change number that created the build:

p4 labelsync -l release_label_name //depot/xxx/...@bld_xxx_23

p4 labelsync -l release_label_name //depot/xxx/...@12345

The first case is for "overnight build #23 for codeline xxx", if you're using that convention for overnight builds; the second case is if you use change numbers to track the overnight build contents.

I need to create a release build - what's different in that process?

This depends on what packages the release for deployment. If you use the same build mechanisms for overnight and release builds, you should only need to do a few steps:

Run the overnight build script to create the build.
Label the build area that created the release, using "p4 label" and then "p4 labelsync". Pick the 'view' for the label carefully, so that it doesn't include parts of the depot not included in this release.
Run "p4 label" a second time on the new label, and change the status of the label to "locked" so that "p4 labelsync" cannot be run against it accidently.

How do I populate the build area for an overnight build? What are the problems that will come up?

If the Perforce client root is C:/p4root, and the codeline is C:/p4root/xxx, then you'll need to do 2-3 things to populate the area prior to building:

Make a copy of the scripts that'll do the populating, if there's a chance that you'll delete them in the next two steps! (Leave a place for your scripts to stand.)
Run the OS command to delete the codeline directory, e.g.

del/s c:\p4root\xxx

it's important to do this if you want a clean build - don't rely on incremental compile for overnight compiles unless the compilation process takes many hours.

Repopulate the source:

p4 sync -f C:/p4root/xxx/...#have

Update the source to the most recent revisions:

p4 sync C:/p4root/xxx/...

Update your build labels (or remember the current change numbers) to reflect this new content.

Most problems that come up in this area have to do with obsolete object/library files. Entirely removing the source area and repopulating saves you from this headache completely.

How do I update the depot to include a new log, or to include a new line in a logfile?

To include a new log or edit a file, you should:

Open the file for either "add" or "edit";
Modify the file in whatever ways make sense;
Use "p4 change -o" and "p4 submit -i" to create the submitted change.

   operation=add
   filename=newfile.txt
   filetype=ltext
   [ -f $filename ] && operation=edit
   description="log of build for `date`"

p4 $operation -t $filetype $filename

   echo New information as of `date` >> $filename
   # this point, we need to construct the change submission with
   # an accurate description in the change text.
   p4 change -o > xx.tmp
   (
        echo "/^Files:/+1,\$v/$filename/d"
        echo "/<enter description here>/s/.*/    $description/"
        echo w
        echo q
   ) | ed - xx.tmp
   p4 submit -i < xx.tmp
   rm xxx.tmp

Perl hackers won't see substantial gains by doing this in perl, and the 'ed' is actually easier from the shell script.

The underlined part above is important, if the file you're working with is a giant log file, because giant text files should include this hint so that the files will be stored in more efficient ways.

Should l check in my log files? What files might need to be modified and checked in as part of a build?

There's no good answer for this - log files are usually huge, and can start eating space quickly. It's probably best to keep them in a disk area that's backed up regularly, but not to check them in if you can avoid it.
If, however, you track overnight builds based on change numbers instead of labels, you might want to have a file somewhere in the depot that contains one-line entries of the form:

change 12345 - overnight build for 07/22/98

change 12366 - overnight build for 07/23/98

It gets really hard to track which change corresponded to which nightly build, otherwise.

What should I back up when I put a release out?

As a minimum, you'll need to back up the source files and build scripts in Perforce that created the product. This makes several assumptions: Usually, however, you'll probably just want to back up the entire Perforce depot so that you have a complete snapshot of everything, and then if you restore from this backup you can refer to current labels such as the product release label you made when you released the product. (You can also refer to intermediate build labels if you choose, since they'll be included in the backup.) The second and third steps are slight overkill for many situations, but if you wanna be incredibly paranoid, that's the approach to take.
That the build environment (machine, OS, compilers, etc) are easily recreateable from media you have in-hand;
That, if you only back up the top revision of those files, you won't need file revision history and the like.
To make the most comprehensive backup, you'll need to do the following:
Checkpoint the Perforce database and save the checkpoint and the subdirectory $P4ROOT/depot with all its contents:

cd $P4ROOT
p4d -r . -jc

Note that this is documented in the "FAQ for System Administrators" and also on-line in the Perforce manuals.

Save a copy of the Perforce executables you're using, server and client versions.
Do a full system backup of the build machine on which the product was created. This isn't so much to get the source onto media as to get compilers, OS, etc.
Put this all onto a medium you trust, that'll be around for a while.
Write the product release label, the one that you created when you built the product, on the media case. Include with the case a hard-copy printout of the exact commands you typed to create the media, e.g. created using "tar rvf".

How do I make an "escrow" copy of the files (that built the release) for the off-site vaults?

Create a label using p4 label, escrowlabelname1.0,that maps in only the directories you want. (You'll edit the label specification to either include as many directories as you choose, or perhaps the exact file list itself.)
Populate the label, using:

p4 labelsync -l escrowlabelname1.0 //buildclientname/...#have

or,

p4 labelsync -l escrowlabelname1.0 @release_source_label

The first example relies on the build client having the exact files, at this moment, that it did when it created the build; the second example relies on having a label of all the source files that were built to create the release.

Create a new Perforce client that maps its client area to an empty part of the disk, and on that new client run the command:

p4 sync @escrowlabelname1.0

Back up this new area.
Run p4 label escrowlabel1.0 and mark the label "locked". This will prevent accidently updates of that label.
When release 2.0 goes out and you need to do this again, follow the same steps. When you create the label, however, use the original (release 1.0) escrowlabel as a template, using p4 label -t escrowlabel1.0 escrowlabel2.0 .

FAQ for integrating Perforce with your build scripts

Note - the intended audience for this FAQ is a shell/perl hacker who writes or maintains local build scripts and wants to integrate Perforce into them.

Labels

Clients

Updating depot files from a script

Putting a release out