Copyright

Copyright (C) 1997 Capella Computers Ltd.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.

Version

This is version 0.1 of the wrappers, dated 7/97. It has just emerged from preliminary testing and is being used by the developers in Capella Computers Ltd. It is likely that numerous small problems will be detected and corrected during the next few months. When no change will be made to the system for a reasonable period of time (two months?), the first stable release (1.0) will be "formally" declared.

If you have any questions, discovered any bugs, would like the most up-to-date version, or have any other sort of feedback, contact the author, oren@capella.co.il.

Introduction

This document gives a brief overview of the perl p4 wrappers. The purpose of these wrappers is to implement a particular development process on top of the generic p4 services. This process is an adaptation of the process enforced by aegis. Aegis is a free software available on the net, and is much more robust and complete then this set of wrappers. If you develop on UNIX platforms and you can use NFS to link the developer's workstations, you should probably use aegis.

A deliberate design choice made when implementing these wrappers was not to attempt any form of co-existance with other practices. This allowed the wrappers to rely on a particular structuring of the depot, thereby allowing better protection schemes and higher-level baviour. As a result, the wrappers completely replace the built-in p4 interface, with a few notable exceptions which will be listed below.

The wrappers are implemented in perl and as such are rather portable. However, they invoke a few external programs (such as cp, rm and mkdir). While the access to these programs is isolated, it is not exactly trivial to port the system to DOS, for example. However, you can obtain all the necessary programs (including a shell) as free software from GNU project or as commercial software (from MKS, for example).

Basic Concepts

The overview for aegis contains a good tutorial of the basic concepts. It is also recommended that you become familiar with the basic p4 concepts before you read this overview.

The definitive sources of the system reside in //depot/baseline/....
The baseline always "works". In practical terms it means that there is a perl script, verify.pl, in the top level directory of the baseline, and that running it is guaranteed to succeed. Note that this file is part of the baseline and can be changed like any other source file.
Development is done in terms of jobs. The best way to view a job is as a transaction on the sources database. Like a transaction, it must ensure that it brings the database to a consistent state (as defined by verify.pl). It is atomic - either it is applied in full or it is rejected completely. And like in any database, the transactions are serialized.
The integration of a job into the baseline creates a new system version. The result of serializing the jobs is to create a system history which is a linear sequence of such versions, where it seems as if each job begun with the version created by the previous job, made some changes, and created the next version.

Job Life Cycle

The trick is, of course, how to enable a team of programmers to work on several jobs at once and still maintain the above benefits. The best way to understand this process is to follow the life cycle of a job.

new

This is a status of a newly created job. At this point, the job exists as a simple p4 job.

work

Only a single developer can work on a job (a developr may, of course, work on several jobs simultanously). For the developer to be able to work, a new branch is created and populated with the current baseline version. The new branch is named //depot/user/job-attempt/.

The attempt number is incremented whenever a developer undoes everything, returning the job to the new state. If the same developer starts development later on, p4 demands we use a different branch and directory name, hence the count.

The developer's work is done inside a p4 change. This change is submitted to p4 when the job is submitted to review.

review

When development is done, and the job is verified, it is moved to this state. At this stage, all files modified by the job are locked, so no two active jobs which are at this stage or onwards can touch the same file. This is necessary since such jobs will be sent to integration without further synchronization with the baseline.

Sadly, it is not possible to use the built-on p4 file locking mechanism for this, since it only allows associating a lock with an active change which actually tries to modify the locked files. Instead, there's a dummy p4 job which holds in its description the list of all locked files.

fail

The system supports the notion of a code review. At the moment, anyone (even the developer itself) can change the review status of a job. Aegis does a much better job here (as in some other places) by limiting this to developers which are specially denoted as reviewers, and also optionally blocks a developer from reviewing himself.

At any rate, a review may fail the job. In this case, the developer is expected to resume working on it. Note that this means that a new change will be opened, and this change does not necessarily touch all the files which were touched by the previous one, which means the built-in p4 reports for change file lists are not sufficient by themselves to describe what has been done in a particular job.

pass

At this stage, the job comes to the attention of the integrator. In aegis, this can be any developer who has been assigned the task. In p4, you need to specify a special user (called integ) to handle the task. This means an extra p4 license. However, Perforce were kind enough to give us (Capella Computers) an extra license so we could rotate the task between the developers.

At any rate, the integrator chooses between the jobs at this stage to decide which job to integrate next. This may have consequences on developers working on other changes, so the choice is not always easy.

integ

Only one job can be at this stage at any given time. This is the job that is being integrated into the baseline. The integration consists of opening a new change, in which the development branch is merged into the baseline. Note that this is a simple merge - there are never any conflicts (due to the file locks mentioned before). In theory, it should be possible to merge more then one branch in the same change, though this is not currently implemented. Again, there is no danger of conflicts.

The resulting baseline version is re-verified (using verify.pl). This is necessary since the verification done by the developer was on his machine, and may have passed due to his special development environment or due to other external factors. The integration verification is the last line of defense - any problem that gets by it will go straight into the baseline.

Once it passes verification, the change is submitted, a new baseline version is created, the job file locks are released, and there's no going back. Of course, you can easily recreate a previous version of the baseline. But development from this point on would build on the version just created.

The baseline version created is given the label Lversion which can be used to refer to this version in various commands (e.g., p4fp). Note that the version number is different from the job number, and is always increasing by one for each integration.

clear

Since aegis assumes file-system access to all development hosts, it cleans up all the development directories when a job is integrated. One of the nice points about p4 is that it works through any Tcp/Ip connection. This means that the developer needs to do something on his own host to clean things up - both the development branch and the development directory. Note that since there's no point in keeping history of the branch files, the branch files are truly obliterated.

done

Once a job is cleared, there's nothing left to do. It is still possible to edit the job description, but any changes to the sources require opening a new job.

Commands

Copyright

In order to follow the spirit of the GPL, the following command, useless as they may seem to developers, are provided:

p4notice: This displays a short copyright notice, and refers to the copy of the GPL which is supposed to be distributed with the software.

Environment

There are a several environment variables which are used by the wrappers. With the exception of P4JOB, they are all mandatory.

P4ROOT: The root directory of the client's view.
P4PATH: Where the wrappers are installed.
P4PORT: The Tcp/Ip address (including the port) of the p4 server.
P4CLIENT: The identity of the current client.
P4USER: The identity of the current user.
P4JOB: The number of the default job (optional). Serves as the default job number for any command which takes such an argument.

The environment can be manipulated by the following commands:


p4el: List all relevant environment variables.
p4ej [job]: Print (if given no argument) or set the default job; a value of '-' means unsetting the variable altogether.
p4eu [user]: Print (if given no argument) or set the current user id. This command should not be used by most users.

There are additional environment variables used, which are not p4 specific.

TEMP: Where to create temporary directories. Mandatory.
EDITOR: How to edit text files. Is only used by p4je, but you really want to set it anyway (is used by a lot of utilities).
DIFF: How to diff text files. Used by p4rdiff and by some p4 commands. The safe bet is to set it - after all, any system has a reasonable built-in diff, even if you need to get it of the net.
MERGE: How to merge text files. You can do without it if you don't ask for a merged version during conflict resolution, which is a good idea anyway. Using the edit option is sufficient, really.

Report

The following are read-only operations which should be available for all users on all clients, unless stated otherwise.

p4dh

List all the changes in the depot, from recent to ancient. The output is the raw output of p4. Todo: format output as a report.

p4dv [release-specifier]

Print depot baseline directory. This requires you to have a copy of the baseline on the current client. Such a copy can be obtained by using p4fg. It takes as argument any valid p4 release specifier, such as @Lversion-number.

p4fh [file-pattern] ...

Display file history (as a list of jobs). Todo: list the system version number as well as the job number. It takes as argument any file pattern recognized by p4. Typically you just give it the path name(s) of the files in the development or baseline directories, which p4 translates automatically to its internal file name specification.

p4jd [job]

Get job details. It could probably be in a better format.

p4je [job]

Edit job data. This one is very dangerous. As long as you only change the description, you are safe; otherwise you can really shoot yourself in the foot. Todo: provide a safer way to edit just the description.

p4jf [job]

Get list of job file operations - that is, the files changed in this job, and the operation performed on each (add/delete/edit). If the job is in work mode, this will only work on the development machine. Otherwise, it will work anywhere (very important for reviewers).

p4jh [job]

History of a job (a list of changes). Same problems as the depot history command.

p4jl [filter] ...

List jobs. By default, shows all jobs, but accepts a multitude of filtering commands. These commands are applied in the order they are given. A '+' turns on a set of jobs, a '-' turns it off. If the first command is a '+', then it is assumed you start with the empty set; if it is a '-', it is assumed you start with the complete set. The '+/-Status' command turns on/off the listing of jobs of any status. '+/-New' controls listing of new jobs, and likewise for any other job status.

It is also possible to filter by user: '+/-u user' will turn on/off the set of jobs for the specified user, where 'User' means all users, and 'Me' means the current user. You can mix filtering by status and by user; for example, p4jl -done -u me lists all the jobs which aren't done and are owned by the current user.

Any of the keyworks can be shortened to its first letter; the previous example can be written as p4jl -d -u m.

Just for fun, a set of shell functions for common cases is provided:

p4jln: Is equivalent to p4jl +n
p4jlw: Is equivalent to p4jl +w
p4jlr: Is equivalent to p4jl +r
p4jlf: Is equivalent to p4jl +f
p4jlp: Is equivalent to p4jl +p
p4jli: Is equivalent to p4jl +i
p4jlc: Is equivalent to p4jl +c
p4jld: Is equivalent to p4jl +d
p4jlu: Is equivalent to p4jl +u; requires an argument.
p4jlm: Is equivalent to p4jl +u me

p4uh [user]

History of a user (a list of changes). Same problems as the depot history command.

p4ul

all users. Again, the unformatted output of the p4 command.

p4vl

List all versions of the depot, which job created them, who was the developr and so on.

The following are file commands which are not commonly used. Unlike any other command, they take as first argument either baseline or a job number (which is P4JOB by default). They use this argument to construct the necessary prefix for the file pattern passed to p4. By default, this pattern is ....

When applied to files of a job, these commands work only for the developer on the development client if the job is in work state, and for anyone anywhere if the job is in review/pass/fail state. In the second case, the (default) pattern ... is taken to mean all the files opened in this job, and not all the files. In order to get all the files, specify the .... (add another '.').

p4fg [job|baseline] [file-pattern][release-specifier] ...

Get depot file(s). For example, p4fg baseline @L7 would bring version 7 of the whole depot to the current client.

If you got files you want to get rid of, the only way is to get them with a release specifier of '#none'. This removes the files from the client.

p4fp [job|baseline] [file-pattern][release-specifier] ...

Print depot file(s). For example, p4fp baseline verify.pl@L7 would print the contents of verify.pl as it was in version 7 of the system.

p4fr [job|baseline] [file-pattern][release-specifier] ...: Refresh unopened file(s). For example, p4fr verify.pl would refresh verify.pl from the depot, in case it was deleted by mistake using rm, for example.

New Jobs

p4jn title [issue] ...: Create a new job. The first argument is the job's title (if it contains multiple words, put quotes around it so it will be a single argument), and a list of issue id (for recording relevant entries in an external issue-tracking system).
p4jnu [job]: Undo new job creation. This only works on new jobs, and onliterates them from the system. This is the only way to delete a job.

Work Jobs

With the exception of p4wb, all these commands will only work for the developer on the development client, for obvious reasons.

Job Commands

p4wb [job]: Start working on a new job. Creates the branch, creates a development directory in the current client and populates it. Will work for any developer anywhere (except integ, which isn't really a developer). The current user and client become the development user and client for this job.
p4wbu [job]: Undo working on a job. Removes the branch and the development directory; all work is lost forever, unless previouly backed up. Increases the attempt counter by one.
p4we [job]: End working on a job. It first verifies the job (as per p4wv), and then locks all relevant files and submits the job for review.
p4weu [job]: Undo end of working on a job. Which means you've just remembered something which will cause the reviewer to fail the job anyway, so you quickly withraw the submission and fix it, hoping nobody noticed :-) Or, you haven't remembered, and the job has been failed; you need to restart work to fix whatever is wrong.
p4ws [job]: Synchronize a development directory with updated baseline. This expects the developer to interact with resolve to handle potential conflict. The best way is to edit each file - p4 is kind enough to insert very clear markers into the source files, which allow resolving the conflicts using any text editor.
p4wv [job]: Verify work on a job. Basically, a predictor whether p4we will work or not, but can also be used as an easy way to invoke make or whatever.

File Commands

p4fd [job] file-name...: Open a file for deletion. Informs p4 that the specified file will not be included in the version created by this job. Will remove the specified file.
p4fdiff [job] [diff-flags] file-name...: Diff edited file from baseline. This one also works for the integrator when the job is in integ mode, as it is useful for last-moment reviews. P4 allows certain flags to be specified for diff in this command; see the description of the diff command.
p4fe [job] file-name...: Open a file for editing. Informs p4 that the specified file will be changed in the version created by this job. It will make the file writable (files are read-only by default).
p4fn [job] file-name...: Open a new file for editing. Informs p4 that the specified file will be added in the version created by this job. Does not actually create or modify the file.
p4fu [job] file-name...: Undo operation on an opened file. Informs p4 that whatever has been said about the file is not true, whatever the operation was. p4 restores the file to its original state (which means refreshing it if it was deleted or edited). It will not delete files which were added.

Review Job

These commands can be run by any developer (except integ, which is not really a decveloper). It would have been nice to limit them just to developers designated as reviewers, and prevent a developer from reviewing himself, but as fooling the system is so easy (just set P4USER), what's the point?

p4rdiff [job] [diff-flags] file-name...: View difference for a file for a job. P4 is not kind enough to let us difference any two files in the depot (not that I know of, anyway), so you first have to get all the modified files from the development branch by using p4fg job. The bonus is that you can use any diff program, and give it any flags you like.
p4rfail [job]: Fail a reviewed job; additional work should be done by the developer. In theory, there should be no "bugs" where (assuming that the verify.pl script runs a good set of regression tests). Even so, there are issues of style, design, performance, and so on. While failing their jobs won't make you popular with other developers, passing bad code won't make you popular with the integrator.
p4rpass [job]: Pass a review job (clearing it for integration). By this the reviewer is taking as much responsibility for the submitted code as the developer. This keeps the reviewers honest :-)

Integration Job

p4ib [job]: Start integration of a job. This creates a local integration directory (ruining whatever copy of the baseline you had), and merges the changes from the development branch. Due to p4 bug #350, this causes getting all the added branch files as well, so you'll have to get rid of them manually later. This bug is really a pain in the rear echelon.
p4ibu [job]: Undo begin integration of a job. Restores the local baseline copy to the latest version, removes the integration change, and so on.
p4ie [job]: End integration of a job. This is the point of no return. It does a verify first, of course, as per p4iv, for what it is worth. If a bug made it this far, it can bask in the warm fuzzy certainty it will take a lot of time and effort to root it out...
p4iv [job]: Verify integration of a job. Basically, run verify.pl and hope it checks the right things.

Clear Job

p4ic [job]: Clear integrated job directories. As you've noticed, the above process leaves a lot of junk after the job is integrated. All the reviewers and the integrator need to do p4fg job #none, for starters. Then the developer can destroy the whole works, obliterating the branch.

Locks

In principle, none of these commands should ever be used. The world being an imperfect place, it is very probable that sometimes scripts will crash leaving hanging locks and other inconsistencies. The following commands allow access to the lock operations, so it would be easy to recover from broken/hanging locks. Other inconsistencies require manual care using the raw p4 interface.

p4ld: Lock the whole depot. Note that this only stops operations which try to lock the whole depot; operations that only lock a single job will succeed.
p4ldu: Unlock the whole depot.
p4lf: List all locked files by all jobs.
p4lj [job]: Lock a single job. Note that this will succeed regardless of whether the whole depot is locked.
p4lju [job]: Unlock a single job.
p4ll: List all (job, depot) locks.

Starting

Set Up New Depot

Start with a completely blank depot.
Initialize the relevant environment variables. The file lib/p4init contains a base you can start from. You should probably include this initialization in your shell's startup file (.profile or whatever).
Make sure you set the P4SHIFTER environment variable before invoking p4funcs. This will allow you to switch to being another user - in particular, the integrator (integ).
Use p4eu integ to become the integrator. You should probably move to the lib directory so invoking the p4 client directly would be easy. It is intentionally not placed in the path.
Run p4 depot depot and create the depot (optional).
Run p4 user and create the user integ.

Run p4 protect and set up the protection scheme. It should look like:

read integ ip-mask //depot/...
write integ ip-mask //depot/baseline/...
super integ ip-mask //depot/...
read user ip-mask //depot/...
write user ip-mask //depot/user/...
...

Run p4 client to create your client. The default view is fine. Make sure that the root agrees with the P4ROOT environment.
Create the first version of the verify.pl file:
- mkdir -p $P4ROOT/baseline
- $EDITOR $P4ROOT/baseline/verify.pl
  Anything will do in there, as long as it doesn't call die or exit.
- Add it using p4 add //depot/baseline/verify.pl.
- Submit the first change using p4 submit.
Run p4eu your-name to become yourself again.

Add New Developer

In order to add a new developer (and his client) to the system:

As integ, add the new user to the protection lists, using p4 protect.
As the new developer, copy the p4init file into the shell startup file, with the appropriate customizations. Apply it.
Call p4 user to create the user.
Call p4 client to create the client.

Caveats

This is the second try at getting the system right. The first was intended to be a much thinner set of wrappers on top of p4, with the goal to use as much of the built-in p4 commands as possible. This turned out to be a huge headache, due to problems with almost anything - change lists, file locks, job status...

The result is that you have to do things through the wrappers exclusively. It is true that a few of them are just calls to built-in p4 commands, but you've got to go through them in order to get the arguments right.

An exception to the above is setting things up. There's no automated way to set up the initial dept, create a new user, a new client, and so on. This is not too bad since such operations are rare.

A second problem is that these wrappers go only so far. There is some missing functionality, the output of commands is not consistent (some is the raw outpout from p4, some is formatted reports), and so on.

The implementation is not very fast, either. It is not that they are terribly slow, but compared to the lightning-quick response of the raw p4 interface, they seem a drag. Most of the delays are due to the horribly inefficent way locks are implemented. If p4 allows locking unopened files (or even better, nonexisting files), this could be speeded up by a huge factor.