<HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <META NAME="GENERATOR" CONTENT="Mozilla/4.05 [en] (X11; U; OSF1 V4.0 alpha) [Netscape]"> <META NAME="Author" CONTENT="Jeff A. Bowles"> <TITLE>FAQ for integrating Perforce with builds</TITLE> <x-html> </HEAD> <BODY> <H1> FAQ for integrating Perforce with your build scripts</H1> <p> Contributed by Jeff Bowles <H3> Note - the intended audience for this FAQ is a shell/perl hacker who writes or maintains local build scripts and wants to integrate Perforce into them.</H3> <OL> <H5> <B>Labels</B></H5> <LI> <A HREF="#How do I create a label? What should I include in">How do I create a label? What should I include in a label?</A></LI> <LI> <A HREF="#If I want to create a new label for each overnight">If I want to create a new label for each overnight build, how would I do it?</A></LI> <LI> <A HREF="#Should I use labels or change numbers to track the">Should I use labels or change numbers to track the contents of overnight builds?</A></LI> <LI> <A HREF="#So the release needs to be remade three months">So the release needs to be remade three months after it went out. What do I do?</A></LI> <LI> <A HREF="#I like this overnight build and want to officially">I like this overnight build and want to officially designate it as a "release" build. What should I do?</A></LI> <LI> <A HREF="#I need to create a release build - what's">I need to create a release build - what's different in that process?</A></LI> <H5> <B>Clients</B></H5> <LI> <A HREF="#How do I populate the build area? What are the">How do I populate the build area from a nightly build script? What are the problems that will come up?</A></LI> <H5> Updating depot files from a script</H5> <LI> <A HREF="#How do I update the depot to include a new log, or">How do I update the depot to include a new log, or to include a new line in a logfile?</A></LI> <LI> <A HREF="#Should I check in my .log files?">Should l check in my log files? What files might need to be modified and checked in as part of a build?</A></LI> <H5> Putting a release out</H5> <LI> <A HREF="#What should I back up when I put a release">What should I back up when I put a release out?</A></LI> <LI> <A HREF="#How do I make an escrow copy of the files (that">How do I make an "escrow" copy of the files (that built the release) for the off-site vaults?</A></LI> </ol> <HR> <ol> <LI> <A NAME="How do I create a label? What should I include in"></A><B>How do I create a build label? What should I include in a label?</B></LI> <BR> <DL> <LI> Pick a regular format for the label name, e.g. "<I>bld_xxx_1</I>" where '<I>xxx</I>' is the name of the codeline you're building and '<I>1</I>' is just a counter, i.e. the first such label. Remember this format - you'll need it in the next question!</LI> <LI> Make the label itself:</LI> <P><TT><FONT COLOR="#990000"> p4 label bld_xxx_1</FONT></TT> <P>It'll dump you into an editor, and you can change the default "Views:" lines from <P><TT><FONT COLOR="#990000"> //depot/...</FONT></TT> <P>to look solely at the codeline you want, e.g. <P><TT><FONT COLOR="#990000"> //depot/xxx/...</FONT></TT> <P>Save/Quit the editor. <BR> <LI> Populate the label with the file/revision list you want:</LI> <P><TT><FONT COLOR="#990000"> p4 labelsync -l bld_xxx_1 //depot/xxx/...</FONT></TT> <P><FONT COLOR="#000000">or</FONT> <P><TT><FONT COLOR="#990000"> p4 labelsync -l bld_xxx_1 //depot/...</FONT></TT> <P>In this second example, it's okay to include all files in the depot - it'll really only include files described in the label spec you edited when you ran "<TT><FONT COLOR="#990000">p4 label</FONT></TT>". <BR> <LI> <B>Always include all files [that are under source management] that are needed to recreate the build</B> . If your build scripts aren't in codeline "xxx" you'll want to have the label specification include the directory where they are, and "<TT><FONT COLOR="#990000">p4 labelsync</FONT></TT>" needs to include them.</LI> </DL> <HR> <LI> <A NAME="If I want to create a new label for each overnight"></A><B>If I want to create a new label for each overnight build, how would I do it?</B></LI> <BR> <DL> <LI> Make sure you're using a regular name for the label, e.g. "bld_xxx_yyy" where "xxx" is the codeline name and "yyy" is a counter.</LI> <LI> Create the first label in the sequence, which would be "bld_xxx_1". See the previous question for that.</LI> <LI> In the Unix shell, you would do the following (cut and paste):</LI> <P><TT><FONT COLOR="#990000"> CodeLineName=xxx</FONT></TT> <BR><TT><FONT COLOR="#990000"> LabelSpec=tst_${CodeLineName}_</FONT></TT> <BR><TT><FONT COLOR="#990000"> toplabelnumber=`p4 labels | grep $LabelSpec | sed "s/^Label $LabelSpec//" | \</FONT></TT> <BR><TT><FONT COLOR="#990000"> sed "s/ .*//" | sort -r -n | head -1`</FONT></TT> <BR><I><TT><FONT COLOR="#990000"> # at this point, we know the most recent label in this codeline and just need</FONT></TT></I> <BR><I><TT><FONT COLOR="#990000"> # to create a new label and populate it.</FONT></TT></I> <BR><TT><FONT COLOR="#990000"> TopLabel=$LabelSpec$toplabelnumber</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo Current Label is $LabelSpec$toplabelnumber</FONT></TT> <BR><TT><FONT COLOR="#990000"> if [ "$toplabelnumber" != "" ]</FONT></TT> <BR><TT><FONT COLOR="#990000"> then</FONT></TT> <BR><TT><FONT COLOR="#990000"> newlabelnumber=`expr $toplabelnumber + 1`</FONT></TT> <BR><TT><FONT COLOR="#990000"> NewLabel=$LabelSpec$newlabelnumber</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo "New Label is $NewLabel"</FONT></TT> <BR> <BR><TT> <I><FONT COLOR="#990000"># the following two commands are from the previous FAQ item.</FONT></I></TT> <BR><TT><FONT COLOR="#990000"> <U>p4 label -o -t $TopLabel $NewLabel | p4 label -i</U></FONT></TT> <BR><TT><FONT COLOR="#990000"> p4 labelsync -l $NewLabel //depot/...</FONT></TT> <BR><TT><FONT COLOR="#990000"> <I># It's best to obsolete labels that are kinda of old, so we</I></FONT></TT> <BR><I><TT><FONT COLOR="#990000"> # do it here.</FONT></TT></I> <BR><TT><FONT COLOR="#990000"> if [ "$toplabelnumber" -gt 10 ]</FONT></TT> <BR><TT><FONT COLOR="#990000"> then</FONT></TT> <BR><TT><FONT COLOR="#990000"> obsoletelabelnumber=`expr $toplabelnumber - 10`</FONT></TT> <BR><TT><FONT COLOR="#990000"> ObsoleteLabel=$LabelSpec$obsoletelabelnumber</FONT></TT> <BR><TT><FONT COLOR="#990000"> p4 label -d $ObsoleteLabel</FONT></TT> <BR><TT><FONT COLOR="#990000"> fi</FONT></TT> <BR><TT><FONT COLOR="#990000"> else</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo "There was no label to match!" 1>&2</FONT></TT> <BR><TT><FONT COLOR="#990000"> exit 1</FONT></TT> <BR><TT><FONT COLOR="#990000"> fi</FONT></TT> <BR> <LI> Perl hackers will look at the previous "Bourne shell" example and use it as an example of why you don't want to do string processing in "Bourne shell". One perl treatment of this follows:</LI> </DL> <DL><TT><FONT COLOR="#990000"> $CodeLineName = "xxx";</FONT></TT> <BR><TT><FONT COLOR="#990000"> $LabelSpec="tst_" . $CodeLineName . "_";</FONT></TT> <P><TT><FONT COLOR="#990000"> @AllLabels = `p4 labels`;</FONT></TT> <BR><TT><FONT COLOR="#990000"> @ThisCodeLineLabels = sort(sortrevlabel grep(/$LabelSpec/, @AllLabels));</FONT></TT> <BR><TT><FONT COLOR="#990000"> $TopLabelLine = $ThisCodeLineLabels[0];</FONT></TT> <BR><TT><FONT COLOR="#990000"> $TopLabel = $1 if ($TopLabelLine =~ /^Label\s(\S+)\s/);</FONT></TT> <BR><TT><FONT COLOR="#990000"> $TopLabelNumber = $1 if ($TopLabel =~ /^\S+_(\d+)$/);</FONT></TT> <P><TT><FONT COLOR="#990000"> print "The last label in this codeline is $TopLabel\n";</FONT></TT> <BR><TT><FONT COLOR="#990000"> $NewLabelNumber = $TopLabelNumber + 1;</FONT></TT> <BR><TT><FONT COLOR="#990000"> $NewLabel = "$LabelSpec$NewLabelNumber";</FONT></TT> <BR><TT><FONT COLOR="#990000"> print "New Label is $NewLabel\n";</FONT></TT> <BR><TT><FONT COLOR="#990000"> system("<U>p4 label -o -t $TopLabel $NewLabel | p4 label -i</U>");</FONT></TT> <i># in real life, return code would be checked!</i> <BR><TT><FONT COLOR="#990000"> system("p4 labelsync -l $NewLabel //depot/...");</FONT></TT> <BR><TT><FONT COLOR="#990000"> if ($TopLabelNumber > 10 )</FONT></TT> <BR><TT><FONT COLOR="#990000"> {</FONT></TT> <BR><TT><FONT COLOR="#990000"> $ObsoleteLabelNumber = $TopLabelNumber - 10;</FONT></TT> <BR><TT><FONT COLOR="#990000"> $ObsoleteLabel = "$LabelSpec$ObsoleteLabelNumber";</FONT></TT> <BR><TT><FONT COLOR="#990000"> system("p4 label -d $ObsoleteLabel");</FONT></TT> <BR><TT><FONT COLOR="#990000"> }</FONT></TT> <P><TT><FONT COLOR="#990000"> sub sortrevlabel {</FONT></TT> <BR><TT><FONT COLOR="#990000"> my($labelnum_a, $labelnum_b) = ();</FONT></TT> <BR><TT><FONT COLOR="#990000"> $labelnum_a = $1 if ($a =~ /.*_(\d+)/);</FONT></TT> <BR><TT><FONT COLOR="#990000"> $labelnum_b = $1 if ($b =~ /.*_(\d+)/);</FONT></TT> <BR><TT><FONT COLOR="#990000"> return ($labelnum_b <=> $labelnum_a);</FONT></TT> <BR><TT><FONT COLOR="#990000"> }</FONT></TT> <LI> <FONT COLOR="#000000">The underlined part in the above fragments probably should have a "sed" command between the two 'p4 label' commands, to change the description for each label.</FONT></LI> </DL> <HR> <LI> <A NAME="Should I use labels or change numbers to track the"></A><B>Should I use labels or change numbers to track the contents of overnight builds?</B></LI> <BR> <DL> <LI> There's no right answer. The command</LI> <P><TT><FONT COLOR="#990000"> p4 changes -m1 //depot/xxx/...</FONT></TT> <P>will tell a build script what the top change number for codeline 'xxx' (assuming it's mapped there), so it can parse this output and use that in build logs and build e-mail. The developer would type: <P><TT><FONT COLOR="#990000"> p4 sync @12345</FONT></TT> <P>to pull over all files current as of change #12345, but that has a problem: it'll pull over all files [that his/her client maps in] as of that revision, not just the files for codeline 'xxx'. You can tell them to type: <P><TT><FONT COLOR="#990000"> p4 sync //depot/xxx/...@12345</FONT></TT> <P>but, by then, you've lost the simplicity that using the change numbers might've given you. <BR> <LI> You should probably always use labels for release builds - when you create a release you'll often need to include an additional change to one particular file that you need to track, and you want the label to include <U>exactly</U> the files used to build the release on the build client.</LI> </DL> <HR> <LI> <A NAME="So the release needs to be remade three months"></A><B>So the release needs to be remade three months after it went out. What do I do?</B></LI> <DL>You'll need a slightly-hacked version of your "get the source and do a build" script. The hacks are: <DL> <LI> It shouldn't use or touch any labels except for the release label you've remembered to create when you made the release. Delete the lines in the script that create labels or run "p4 labelsync" on them.</LI> <LI> Change the line in the script that does the "<TT><FONT COLOR="#990000">p4 sync -f ...#have</FONT></TT>" so that it looks like:</LI> <P><TT><FONT COLOR="#990000"> p4 sync -f //depot/xxx/...@labelname</FONT></TT> <P>where "xxx" is the name of the codeline, and "labelname" is the release label. <LI> Bug your system administrators about whether compiler versions, OS releases, etc, have changed since the release was created. Listen carefully to their answers.</LI> <LI> Run the script. It should create the original release faithfully, although most compilers embed timestamps within the .exe and .obj files, so you might not have byte-for-byte identical files.</LI> <LI> Rerun your regressions to gain confidence in the newly-built product.</LI> </DL> </DL> <HR> <LI> <A NAME="I like this overnight build and want to officially"></A><B>I like this overnight build and want to officially designate it as a "release" build. What should I do?</B></LI> <DL>See the previous step - you just need to run "p4 label" (to create a label) and then "p4 labelsync" to the overnight label or change number that created the build: <P><TT><FONT COLOR="#990000"> p4 labelsync -l release_label_name //depot/xxx/...@bld_xxx_23</FONT></TT> <P>or <P><TT><FONT COLOR="#990000"> p4 labelsync -l release_label_name //depot/xxx/...@12345</FONT></TT> <P>The first case is for "overnight build #23 for codeline xxx", if you're using that convention for overnight builds; the second case is if you use change numbers to track the overnight build contents. <BR> </DL> <HR> <LI> <A NAME="I need to create a release build - what's"></A><B>I need to create a release build - what's different in that process?</B></LI> <BR> <DL>This depends on what packages the release for deployment. If you use the same build mechanisms for overnight and release builds, you should only need to do 1-2 steps: <DL> <LI> Run the overnight build script to create the build.</LI> <LI> Label the build area that created the release, using "p4 label" and then "p4 labelsync". <I>Pick the 'view' for the label carefully, so that it doesn't include parts of the depot not included in this release.</I></LI> <LI> Run "p4 label" a second time on the new label, and change the status of the label to "locked" so that "p4 labelsync" cannot be run against it accidently.</LI> </DL> </DL> <HR> <LI> <A NAME="How do I populate the build area? What are the"></A><B>How do I populate the build area for an overnight build? What are the problems that will come up?</B></LI> <BR> <DL> <LI> If the Perforce client root is C:/p4root, and the codeline is C:/p4root/xxx, then you'll need to do 2-3 things to populate the area prior to building:</LI> <DL> <LI> Make a copy of the scripts that'll do the populating, if there's a chance that you'll delete them in the next two steps! (Leave a place for your scripts to stand.)</LI> <LI> Run the OS command to delete the codeline directory, e.g.</LI> <P><TT><FONT COLOR="#990000"> del/s c:\p4root\xxx</FONT></TT> <P>it's important to do this if you want a clean build - don't rely on incremental compile for overnight compiles unless the compilation process takes many hours. <LI> Repopulate the source:</LI> <P><TT><FONT COLOR="#990000"> p4 sync -f C:/p4root/xxx/...#have</FONT></TT> <LI> Update the source to the most recent revisions:</LI> <P><TT><FONT COLOR="#990000"> p4 sync C:/p4root/xxx/...</FONT></TT> <LI> <FONT COLOR="#000000">Update your build labels (or remember the current change numbers) to reflect this new content.</FONT></LI> </DL> Deleting the entire codeline build tree (and any "obj" or "lib" directories if they're not directly under C:\p4root\xxx) is important because old object files often can bite you. <BR> <LI> Most problems that come up in this area have to do with obsolete object/library files. <U>Entirely removing the source area and repopulating saves you from this headache completely.</U></LI> </DL> <HR> <LI> <A NAME="How do I update the depot to include a new log, or"></A><B>How do I update the depot to include a new log, or to include a new line in a logfile?</B></LI> <BR> <DL> <LI> To include a new log or edit a file, you should:</LI> <DL> <LI> Open the file for either "add" or "edit";</LI> <LI> Modify the file in whatever ways make sense;</LI> <LI> Use "p4 change -o" and "p4 submit -i" to create the submitted change.</LI> </DL> An example of this, in a Unix shell script, follows: <P><TT><FONT COLOR="#990000"> operation=add</FONT></TT> <BR><TT><FONT COLOR="#990000"> filename=newfile.txt</FONT></TT> <BR><TT><FONT COLOR="#990000"> <U>filetype=ltext</U></FONT></TT> <BR><TT><FONT COLOR="#990000"> [ -f $filename ] && operation=edit</FONT></TT> <BR><TT><FONT COLOR="#990000"> description="log of build for `date`"</FONT></TT> <P><TT><FONT COLOR="#990000"> p4 $operation -t $filetype $filename</FONT></TT> <P><TT><FONT COLOR="#990000"> echo New information as of `date` >> $filename</FONT></TT> <BR><TT><FONT COLOR="#990000"> <I># this point, we need to construct the change submission with</I></FONT></TT> <BR><I><TT><FONT COLOR="#990000"> # an accurate description in the change text.</FONT></TT></I> <BR><TT><FONT COLOR="#990000"> p4 change -o > xx.tmp</FONT></TT> <BR><TT><FONT COLOR="#990000"> (</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo "/^Files:/+1,\$v/$filename/d"</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo "/<enter description here>/s/.*/ $description/"</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo w</FONT></TT> <BR><TT><FONT COLOR="#990000"> echo q</FONT></TT> <BR><TT><FONT COLOR="#990000"> ) | ed - xx.tmp</FONT></TT> <BR><TT><FONT COLOR="#990000"> p4 submit -i < xx.tmp</FONT></TT> <BR><TT><FONT COLOR="#990000"> rm xxx.tmp</FONT></TT> <P><FONT COLOR="#000000"><I>Perl</I> hackers won't see substantial gains by doing this in <I>perl</I>, and the 'ed' is actually easier from the shell script.</FONT> <BR> <LI> <FONT COLOR="#000000">The underlined part above is important, if the file you're working with is a giant log file, because giant text files should include this <U>hint</U> so that the files will be stored in more efficient ways.</FONT></LI> </DL> <HR> <LI> <A NAME="Should I check in my .log files?"></A><B>Should l check in my log files? What files might need to be modified and checked in as part of a build?</B></LI> <BR> <DL> <LI> There's no good answer for this - log files are usually huge, and can start eating space quickly. It's probably best to keep them in a disk area that's backed up regularly, but not to check them in if you can avoid it.</LI> <LI> If, however, you track overnight builds based on change numbers instead of labels, you might want to have a file somewhere in the <I>depot</I> that contains one-line entries of the form:</LI> <BR><TT><FONT COLOR="#990000"> change #12345 - overnight build for 07/22/98</FONT></TT> <BR><TT><FONT COLOR="#990000"> change #12366 - overnight build for 07/23/98</FONT></TT> <BR>and so on. That should be modified (appended to) by the build scripts using code similar to the example above. <U>It gets really hard to track which change corresponded to which nightly build, otherwise.</U> <BR> </DL> <HR> <LI> <A NAME="What should I back up when I put a release"></A><B>What should I back up when I put a release out?</B></LI> <BR> <BR>As a minimum, you'll need to back up the source files and build scripts in Perforce that created the product. This makes several assumptions: <DL> <LI> That the build environment (machine, OS, compilers, etc) are easily recreateable from media you have in-hand;</LI> <LI> That, if you only back up the top revision of those files, you won't need file revision history and the like.</LI> </DL> Usually, however, <U>you'll probably just want to back up the entire Perforce depot so that you have a complete snapshot of everything</U>, and then if you restore from this backup you can refer to current labels such as the <I>product release label</I> you made when you released the product. (You can also refer to intermediate build labels if you choose, since they'll be included in the backup.) <P>To make the most comprehensive backup, you'll need to do the following: <DL> <LI> Checkpoint the Perforce database and save the checkpoint and the subdirectory $P4ROOT/depot with all its contents:</LI> <P><TT><FONT COLOR="#990000"> cd $P4ROOT</FONT></TT> <BR><TT><FONT COLOR="#990000"> p4d -r . -jc</FONT></TT> <P><FONT COLOR="#000000">Note that this is documented in the "FAQ for System Administrators" and also on-line in the Perforce manuals.</FONT> <LI> <FONT COLOR="#000000">Save a copy of the Perforce executables you're using, server and client versions.</FONT></LI> <LI> <FONT COLOR="#000000">Do a full system backup of the build machine on which the product was created. This isn't so much to get the source onto media as to get compilers, OS, etc.</FONT></LI> <LI> <FONT COLOR="#000000">Put this all onto a medium you trust, that'll be around for a while.</FONT></LI> <LI> <FONT COLOR="#000000">Write the <I>product release label,</I> the one that you created when you built the product, on the media case. Include with the case a hard-copy printout of the exact commands you typed to create the media, e.g. <I>created using "tar rvf".</I></FONT></LI> </DL> <I><FONT COLOR="#000000">The second and third steps are slight overkill for many situations, but if you wanna be incredibly paranoid, that's the approach to take.</FONT></I> <BR> <HR> <LI> <A NAME="How do I make an escrow copy of the files (that"></A><B>How do I make an "escrow" copy of the files (that built the release) for the off-site vaults?</B></LI> <BR> <BR>You can use a label to create an initial fileset that you'll archive. It's pretty easy: <DL> <LI> Create a label using <TT><FONT COLOR="#990000">p4 label, <I>escrowlabelname1.0, </I></FONT></TT>that maps in only the directories you want. (You'll edit the label specification to either include as many directories as you choose, or perhaps the exact file list itself.)</LI> <LI> Populate the label, using:</LI> <P><TT><FONT COLOR="#990000"> p4 labelsync -l <I>escrowlabelname1.0</I> //buildclientname/...#have</FONT></TT> <P>or, <P><TT><FONT COLOR="#990000"> p4 labelsync -l <I>escrowlabelname1.0 </I> @release_source_label</FONT></TT> <P>The first example relies on the build client having the exact files, at this moment, that it did when it created the build; the second example relies on having a label of all the source files that were built to create the release. <LI> Create a new Perforce client that maps its client area to an empty part of the disk, and on that new client run the command:</LI> <BR><TT><FONT COLOR="#990000"> p4 sync @<I>escrowlabelname1.0</I></FONT></TT> <LI> Back up this new area.</LI> <LI> Run <TT><FONT COLOR="#990000">p4 label <I>escrowlabel1.0</I> </FONT></TT> and mark the label "locked". This will prevent accidently updates of that label.</LI> </DL> When release 2.0 goes out and you need to do this again, follow the same steps. When you create the label, however, use the original (release 1.0) escrowlabel as a template, using<TT><FONT COLOR="#990000"> p4 label -t <I>escrowlabel1</I>.0 </FONT></TT> escrowlabel2.0 .</OL> <I>Last modified on 21 September 1998.</I> </BODY> </HTML>
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#1 | 684 | jim_dowsett | Beginning point for UTA se1 CSE5324 summer 2001 version control/file sharing project. | ||
//public/perforce/faq/build.html | |||||
#4 | 112 | Laura Wingerd |
Publish "Branching FAQ" (pull in changes from //guest/laura_wingerd/perforce/faq/... ) |
||
#3 | 27 | Jeff Bowles | Adding important comment. | ||
#2 | 20 | Jeff Bowles | Cleaning up numbered lists. | ||
#1 | 1 | laura |
Add FAQs. (Still needs index page.) |