Perforce Defect Tracking Integration Advanced Administrator's Guide

Contents

• 1. Introduction
• 2. Select the issues to replicate
  • 2.1. TeamTrack examples
  • 2.2. Bugzilla example
• 3. Allowing users to create issues in Perforce
  • 3.1. The replicate_job_p function
  • 3.2. The prepare_issue function
• 4. Migrating to the defect tracker from Perforce jobs
  • 4.1. Planning for migration
  • 4.2. How to migrate
  • 4.3. Provide names and e-mail addresses for Perforce users
  • 4.4. Create defect tracker user accounts
  • 4.5. Make Perforce jobs consistent with the jobspec
  • 4.6. Select the jobs to migrate
  • 4.7. Translate jobs to the new jobspec
  • 4.8. Transform jobs to issues
  • 4.9. Summary of translation, transformation, and preparation steps
  • 4.10. Migrate
  • 4.11. Handle migration errors
• 5. Integrating with multiple Perforce servers
• 6. Querying the TeamTrack database
• A. References
• B. Document History

1. Introduction

This manual is the Perforce Defect Tracking Integration Advanced Administrator's Guide. It describes methods of administering the P4DTI that are outside the scope of the Perforce Defect Tracking Integration Administrator's Guide because they require a higher level of expertise (see "Required experience" below).

This document is intended for P4DTI administrators. Ordinary users of the defect tracker or Perforce should read the Perforce Defect Tracking Integration User's Guide.

This guide does not describe how to install, maintain, or perform ordinary configuration and administration of the P4DTI. Read the Perforce Defect Tracking Integration Administrator's Guide.

Required experience

To do the things described in this guide, you must have the following experience:

• Perforce: You must be expert with jobspecs and jobs, confident about backing up and restoring from checkpoints, able to set up a duplicate or test Perforce server. You need good working knowledge of the relevant sections of the Perforce System Administrator's Guide.
• The Python programming language: You must be reasonably competent with Python; in particular, able to write functions that transform dictionaries.
• The defect tracker: You must know the defect tracker's database schema. You must have detailed knowledge of your workflow and how the workflow puts constraints on the issue data. You must able to make backups and set up a duplicate or test defect tracker.

2. Select the issues to replicate

The P4DTI selects the issues that get replicated from the defect tracker using the replicate_p function. To select the issues that get replicated, edit this function in the file config.py in the installation directory.

Normally, the P4DTI replicates all issues created or modified after the start_date, but you can modify this function to further restrict the issues. See section 5.1, "P4DTI configuration", of the Perforce Defect Tracking Integration Administrator's Guide for more details of the start_date parameter.

Note: after an issue starts being replicated, it continues to be replicated, even if it no longer matches the replicate_p criteria.

The function is passed one argument, the issue, in the form of a Python dictionary that maps field names (strings) to field values. If the function returns 1, the issue is replicated. If it returns 0, the issue is not.

The replicate_p function does not have to check the start_date configuration parameter. The start_date is always honored by the P4DTI, in addition to any test made by replicate_p.

3. Allowing users to create issues in Perforce

By default, the P4DTI requires users to create issues using the defect tracker. It ignores jobs created in Perforce. The P4DTI can be configured to transform new jobs created in Perforce into new issues in the defect tracker.

To cause the P4DTI to replicate jobs created in Perforce, add two functions to the end of config.py: replicate_job_p and prepare_issue.

Note: this feature is not available for TeamTrack 4.5. You must upgrade to TeamTrack 5.0 or later.

3.1. The replicate_job_p function

The replicate_job_p function chooses the jobs in Perforce that get added to the defect tracker. The function takes the newly-created job as its argument, in the form of a Python dictionary that maps field names (strings) to field values (strings). If it returns 1, the job is added to the defect tracker and replicated thereafter. If it retuns 0, the job is not replicated.

Note: after a job starts being replicated it, continues to be replicated, even if it no longer matches the replicate_job_p criteria.

If you want all newly-created jobs to be translated and added to the defect tracker, write a replicate_job_p function as follows:

def replicate_job_p(job):
    return 1

3.2. The prepare_issue function

The P4DTI does its best to build the new issue, but the defect tracker might require fields that can't be translated automatically from fields in the job. So the P4DTI calls the prepare_issue function to complete the transformation of a new job in Perforce to an issue in the defect tracker, by supplying values for fields that can't be translated automatically.

The function takes two arguments (issue, job):

issue

The partly-constructed issue, in the form of a Python dictionary that maps field names (strings) to field values. For example, in TeamTrack this might be:

{
    "STATE": 17,
    "OWNER": 9,
    "TITLE": "The widgets are specified in imperial units (ft/lb/s).\n",
    "DESCRIPTION": "But 80% of customers use ISO 31, so we should use m/kg/s.\n"
    "PRIORITY": 32,
    "PROJECTID": 17,
    "ISSUETYPE": 0,
    "SUBMITTER": 9,
}

job

The job, in the form of a Python dictionary that maps field names (strings) to a field values (strings). For example:

{
    "Job": "job000123",
    "State": "verified",
    "Owner": "gdr",
    "Date": "2001/10/11",
    "Title": "The widgets are specified in imperial units (ft/lb/s).\n",
    "Description": "But 80% of customers use ISO 31, so we should use m/kg/s.\n",
    "Priority": "high",
    "Project": "widget",
}

The function must modify issue so that it is valid for the defect tracker. The following sections explain what "valid" means for TeamTrack (section 3.2.1) and Bugzilla (section 3.2.2).

4. Migrating to the defect tracker from Perforce jobs

Migration is the process of loading your Perforce jobs into the defect tracker.

Note: this feature is not available for TeamTrack 4.5. You must upgrade to TeamTrack 5.0 or later.

4.1. Planning for migration

4.1.1. Limitations of this procedure

The migration procedure has an important limitation: you can't keep your old jobspec. You have to migrate to a system in which the P4DTI chooses your jobspec for you based on the fields you replicate from the defect tracker.

If you need to keep your old jobspec, then you must use the "advanced configuration" approach. This is outside the scope of the manual, but is documented in [GDR 2000-10-16, 8.6]. Contact Perforce technical support if you need to do this.

4.1.2. Configuring and testing the P4DTI and migration

1. Create a Perforce server for testing.

2. Create a defect tracker for testing. If you have existing defect tracking data, make a copy that includes your workflow and a representative sample of your existing issues.

3. Set up and test the P4DTI on the test Perforce server and test defect tracker, following the instructions in the Perforce Defect Tracking Integration Administrator's Guide. Test the integrated system and verify that the workflow is correct.

4. Record the jobspec created by the P4DTI (for example, by running a command like p4 jobspec -o > saved-jobspec, or by printing it out). This jobspec is used in configuring the translation step of the migrator. (See section 4.7, "Translating to the new jobspec".)

5. Replace the test Perforce server with a server that has a copy of all your jobs and fixes. (Create a checkpoint of your main Perforce server and restore it onto a new empty Perforce server. See chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.)

6. Set up and test migration from the test Perforce server to the test defect tracker, as described in section 4.2, "How to migrate".

7. Apply any workflow changes you made on your test defect tracker to your main defect tracker server.

When you are confident that you have the correct configuration, proceed to section 4.1.3, "Going live with the P4DTI with migration".

4.1.3. Going live with the migrated data

After you have configured and tested the P4DTI with migration, follow these steps to make the system available to your users:

1. Ask your users to stop using the defect tracker and Perforce server.

2. Make a backup up of your Perforce repository and your defect tracker database, in case there are any serious problems.

   • Perforce: make a checkpoint of your Perforce server; see chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.

   • TeamTrack: start the TeamTrack Administrator; select "Copy Database" from the "File" menu; see section 16, "Copying a Database", of the TeamTrack Administrator Manual 5.5.

   • Bugzilla: run mysqldump -u user database > backup-file. See section 4.4.1, "Database Backups", of the MySQL ManuaL.

3. Edit your config.py file to point to your main Perforce and defect tracker servers.

4. Run the migration and check the results (4.10).

5. Start the P4DTI.

4.2. How to migrate

Migration cannot be done automatically by the P4DTI, because the data in Perforce is in a very different format from the defect tracker, and the way your workflow is implemented in Perforce is likely to be different from the way it's implemented in the defect tracker.

Here is an overview of migration. The details of each step are covered in the following sections.

1. Prepare the Perforce data for migration:

   1. Ensure that users in Perforce match users in the defect tracker (4.3 and 4.4).

   2. Make Perforce jobs consistent with the current jobspec so the P4DTI can transform them correctly (4.5).

2. Write Python functions in your config.py to do the following:

   1. Select the jobs to migrate (4.6).

   2. Translate these jobs so they match the jobspec you plan to use (4.7).

   3. Satisfy any defect tracker constraints on submitted issues (4.8).

3. Migrate the selected jobs to the defect tracker (4.10).

4.3. Provide names and e-mail addresses for Perforce users

Give every Perforce user a name and e-mail address, to enable the P4DTI to correctly match users in Perforce to users in TeamTrack (see section 3.5, "User accounts" of the Perforce Defect Tracking Integration Administrator's Guide), or automatically make user accounts in Bugzilla for each Perforce user (see section 4.4, "Making defect tracker user accounts").

4.4. Create defect tracker user accounts

In your defect tracker, create a user account for every Perforce user who originates or works on issues.

4.5. Make Perforce jobs consistent with the jobspec

If you change the jobspec, Perforce doesn't update its existing jobs to match, so some jobs might be inconsistent with the revised jobspec. Examples: a "required" field that is missing from some jobs; a "select" field for which some jobs have an illegal value. These problems prevent the P4DTI from reading the jobs.

To find inconsistencies, go to the P4DTI installation directory and run the command python check_jobs.py (it takes about a minute for every thousand jobs on a typical system). For example:

$ python check_jobs.py
(P4DTI-10081)  Job 'job000024' doesn't match the jobspec:
(P4DTI-7087)  Error detected at line 11.
Value for field 'Status' must be one of open/suspended/closed.

Fix these jobs so that they match the jobspec. When this is done, you'll get the following output:

$ python check_jobs.py
(P4DTI-10092)  All jobs match the jobspec.

4.6. Select the jobs to migrate

To select the jobs to migrate, add the function migrate_p to the end of config.py.

The migrate_p(job) function determines whether to migrate a job to the defect tracker. The job argument is the job being considered for migration (in the old jobspec), in the form of a dictionary that maps field names (strings) to field values (strings). If it returns 1, the job is migrated. If it returns 0, the job is not migrated.

4.6.1. Examples of migrate_p

Example 1. To migrate all jobs:

def migrate_p(job):
    return 1

Example 2. To migrate only open jobs:

def migrate_p(job):
    return job["Status"] == "open"

Example 3. To migrate only jobs in the "horseshoe" and "nail" projects:

def migrate_p(job):
    return job["Project"] in ["horseshoe", "nail"]

4.7. Translate jobs to the new jobspec

Add the function translate_jobspec to the end of config.py.

The translate_jobspec function translates a job from the old jobspec (before migration) to the new jobspec (after migration). The old job is passed as the single argument, in the form of a dictionary that maps field names (strings) to field values (strings). The new job must be returned in the same form (it's OK to update and return the argument).

Job: job000123
Status: closed
User: gdr
Date: 2001/10/11
Description:
        The widgets are specified in imperial units (ft/lb/s).
        But 80% of customers use ISO 31, so we should use m/kg/s.
Priority: high
Project: widget

Job: job000123
State: verified
Owner: gdr
Date: 2001/10/11
Title:
        The widgets are specified in imperial units (ft/lb/s).
Description:
        But 80% of customers use ISO 31, so we should use m/kg/s.
Priority: high
Project: widget

Job: job000123
Status: closed
User: gdr
Date: 2001/10/11
Description:
        The widgets are specified in imperial units (ft/lb/s).
        But 80% of customers use ISO 31, so we should use m/kg/s.
Priority: high
Project: widget

Job: job000123
Status: closed
Assigned_To: gdr
Date: 2001/10/11
Summary:
        The widgets are specified in imperial units (ft/lb/s).
Description:
        But 80% of customers use ISO 31, so we should use m/kg/s.
Resolution: FIXED
Priority: P1
Product: widget

4.8. Transform jobs to issues

Jobs translated to the new jobspec might not be suitable for submission to the defect tracker. The defect tracker might have additional constraints, or require fields that can't be translated from fields in the job. You must write a prepare_issue function as described in section 3.2, "The prepare_issue function".

4.9. Summary of translation, transformation, and preparation steps

Figure 1 shows a summary of the translate, transform, and preparation steps of the migration process:

1. Translate the job from the old jobspec to the new jobspec using the translate_jobspec function.

2. The P4DTI transforms the job to an issue by converting each field in replicated_fields and any system fields.

3. Prepare for submission to the defect tracker using the prepare_issue function.

Figure 1. The translate, transform, and prepare steps. The example data in the figure is for TeamTrack.

4.10. Migrate

1. Make a checkpoint of the Perforce server as described in chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.

2. Go to the P4DTI installation directory and run the command python migrate.py. If it succeeds, it prints output like this. (If you see no output, then your migrate_p function isn't matching any jobs.)

   2001-11-14 12:34:13  (P4DTI-8920)  Migrated job 'job000001' to issue 'BUG00001'.
   2001-11-14 12:34:13  (P4DTI-8920)  Migrated job 'job000002' to issue 'BUG00002'.
   2001-11-14 12:34:13  (P4DTI-8206)  -- Added fix for change 3326 with status closed.
   ...
   2001-11-14 12:35:02  (P4DTI-8953)  Migration completed.

   Notes for Bugzilla users:

   • The P4DTI locks the Bugzilla database while migrating, so no-one can use Bugzilla until migration completes.

   • If you've specified a value for bugzilla_directory, you'll see the message "Running processmail for %d bugs..." after the last job is migrated and before the "Migration completed" message appears. This is because Bugzilla's processmail script is being run for each migrated issue. (If you don't want notifications to be sent when migrating, set bugzilla_directory=None while migrating and reset it afterwards.)

3. Handle errors (see section 4.11). When you've fixed the problem, run python migrate.py again. Provide the command-line option -s JOBNAME to start migrating at a particular job. For example, if migration failed because of a problem job000123, then continue migrating with python migrate.py -s job000123.

4. In the defect tracker, verify that all the Perforce jobs you want to replicate are present and that the values in all fields are correct. If not, fix the problems by hand, or edit the migrate_p, translate_jobspec and prepare_issue functions to correct the problem, and run migration again.

5. Replace the existing jobs in Perforce with jobs replicated from the defect tracker's database by refreshing them as described in section 9.2, "Refreshing jobs in Perforce", of the Perforce Defect Tracking Integration Administrator's Guide.

6. Check the consistency of the replicated data by running the consistency checker as described in section 7.3, "Checking data consistency", of the Perforce Defect Tracking Integration Administrator's Guide. If this consistency check fails, fix the problems by hand, or restore your Perforce server from the checkpoint you made earlier, edit the migrate_p, translate_jobspec and prepare_issue functions to correct the problem and run migration again.

7. Start the P4DTI as described in section 5.5, "Starting and stopping the replicator manually", of the Perforce Defect Tracking Integration Administrator's Guide.

4.11. Handle migration errors

These kinds of error are likely to be encountered during migration:

• The P4DTI can't translate a user. For example:

  (P4DTI-9231)  Translating job field 'User' (value 'spong') to issue field 'assigned_to'...
  (P4DTI-921X)  Migrating job 'job000174'...
  Bugzilla module error (P4DTI-514X)  There is no Bugzilla user corresponding to Perforce user 'spong'.

  This can happen even if you translated all Perforce users in section 4.4, because a deleted Perforce user might still appear in fields in jobs.

  Choose the most appropriate of these four solutions:

  • Add the user to Perforce and to the defect tracker. (You need a spare Perforce licence and a spare TeamTrack licence if migrating to TeamTrack.)

  • Bugzilla: Add the user to Bugzilla (only) and edit the job so that the user's e-mail address appears instead of the user name. (The P4DTI can match this address to the user in Bugzilla without the user needing to exist in Perforce.)

  • TeamTrack: Add the user to TeamTrack (only) with the same user name as in Perforce. (You need a spare TeamTrack licence.)

  • Edit the job and replace the non-existent user with a real user.

• The P4DTI can't transform a job to an issue; for example:

  (P4DTI-9231)  Translating job field 'State' (value 'suspended') to issue field 'STATE'...
  (P4DTI-921X)  Migrating job 'job000174'...
  (P4DTI-6426)  No TeamTrack state corresponding to Perforce state 'suspended'.

  The translate_jobspec function has failed to translate all values in Perforce. (In the example error, it must translate the Perforce status suspended to a TeamTrack state, as shown in the example in section 4.7.1.)

• The P4DTI can't create an issue in the defect tracker; for example:

  (P4DTI-921X)  Migrating job 'job000174'...
  (P4DTI-522X)  Can't create Bugzilla bug without component field.

  The prepare_issue function has failed to meet all the defect tracker constraints. (In the example error, it needs to provide a value for the component field, as shown in the example in section 3.2.4.)

5. Integrating with multiple Perforce servers

The P4DTI can be set up to replicate defects from a single defect tracker to multiple Perforce servers. For example, if you have several development groups, each with their own Perforce server, you can set up the P4DTI to replicate the issues for each group to the appropriate Perforce server.

Note 1: this feature is not available for TeamTrack 4.5 or early TeamTrack 5.0 releases. You must upgrade to TeamTrack build 50207 or later.

Note 2: the P4DTI doesn't support integration between one Perforce server and multiple defect tracking servers.

To replicate to multiple Perforce servers:

1. Install a copy of the P4DTI for each Perforce server, as described in readme.txt.

2. Configure each copy of the P4DTI. The following configuration parameters must be different for each copy:

   • rid, so the replicators can distinguish the issues they are responsible for.
   • sid and p4_server_description, so the defect tracker can report the Perforce server to which the issue is replicated.
   • p4_port, so each replicate can find its Perforce server. (If you have different users and passwords on each Perforce server, change p4_user and p4_password as well).
   • replicate_p, so that the replicators don't try to replicate the same issues.
   • changelist_url and job_url (if you use these features), so that the defect tracker can provide links to the correct server to provide more information changelists and jobs.

3. Start and manage each P4DTI as described in the Perforce Defect Tracking Integration Administrator's Guide.

Note that there's no way to move issues from one Perforce server to another. Once an issue is replicated to one Perforce server then continues to be replicated to that server, even if it later matches the replicate_p function for another replicator. If it's doubtful which development group is going to work on an issue, then you must design your workflow so that issues don't get replicated until it's clear who's going to work on them.

For example, you might have a replicate_p function like this:

def replicate_p(issue):
    return (issue["PROJECTID"] in [4,7,11]
            and issue["STATE"] == 19)

(where state 19 is "Assigned"), so that you get an opportunity to take a decision on the issue and possibly move it to a different project before it gets replicated.

Multiple Perforce server example

You have Perforce servers for your development groups in France and Germany and a single TeamTrack server. Projects 4, 7 and 11 are worked on in France and projects 5, 6, 9 and 10 are worked on in Germany. So set up one P4DTI copy with the following in config.py:

rid = "replicator_fr"
sid = "france"
p4_port = "perforce.company.fr:1666"
p4_server_description = "French development group"
changelist_url = "http://perforce.company.fr/cgi-bin/perfbrowse?@describe+%d"
job_url = "http://perforce-company.fr/cgi-bin/perfbrowse?@job+%s"
def replicate_p(issue):
    return issue["PROJECTID"] in [4,7,11]

and the other with the following:

rid = "replicator_de"
sid = "germany"
p4_port = "perforce.company.de:1666"
p4_server_description = "German development group"
changelist_url = "http://perforce.company.de:8080/%d?ac=10"
job_url = None
def replicate_p(issue):
    return issue["PROJECTID"] in [5,6,9,10]

6. Querying the TeamTrack database

To write replicate_p and prepare_issue functions, you need to know the key values for the tables in the TeamTrack database.

You can examine the TeamTrack database using Microsoft SQL Server, or you can use the teamtrack_query.py script that comes with the P4DTI.

Use the TeamTrack database schema [TeamShare 2002-01-30a] to determine which table you need to query and how to understand the output.

To display the contents of the TeamTrack database:

1. Edit config.py and specify teamtrack_server, teamtrack_user and teamtrack_password.

2. Run the command python teamtrack_query.py TABLE to query a table and show all fields for all records.

   Run the command python teamtrack_query.py TABLE FIELD1 FIELD2 ... to find the contents of a table and restrict the output to the named fields.

   Add the option -q QUERY to restrict the output to the rows matching the query.

Query example 1

You want to replicate issues from the projects "Horseshoe" and "Nail". The TeamTrack database schema [TeamShare 2002-01-30a] says that the value in an issue's PROJECTID field is the ID field of a record in the PROJECTS table. So you run this command:

> python teamtrack_query.py PROJECTS ID NAME
+------+----------------+
|  ID  |  NAME          |
+------+----------------+
|  1   |  Base project  |
|  2   |  Horseshoe     |
|  3   |  Cart          |
|  4   |  Nail          |
|  ... |  ...           |
+------+----------------+

So you can write:

def replicate_p(issue):
    return issue["PROJECTID"] in [2,4]

Query example 2

When a job is created in Perforce, you want to create an issue in TeamTrack with issue type "BUG". The TeamTrack database schema [TeamShare 2002-01-30a] says that an issue type is the ID field of a record in the SELECTIONS table where the PREFIX field gives the prefix for the issue id. So you run the command:

> python teamtrack_query.py SELECTIONS ID PREFIX NAME
+------+----------+------------------------+
|  ID  |  PREFIX  |  NAME                  |
+------+----------+------------------------+
|  0   |          |  (None)                |
|  1   |  BUG     |  Bug report            |
|  2   |  ENH     |  Enhancement proposal  |
|  ... |  ...     |  ...                   |
+------+----------+------------------------+

So you can write:

def prepare_issue(issue):
    issue["ISSUETYPE"] = 1

A. References

B. Document History

This document is copyright © 2001 Perforce Software, Inc. All rights reserved.

Redistribution and use of this document in any form, with or without modification, is permitted provided that redistributions of this document retain the above copyright notice, this condition and the following disclaimer.

This document is provided by the copyright holders and contributors "as is" and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the copyright holders and contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this document, even if advised of the possibility of such damage.