p4review.py #12

  • //
  • guest/
  • perforce_software/
  • sdp/
  • dev/
  • Server/
  • Unix/
  • p4/
  • common/
  • bin/
  • p4review.py
  • View
  • Commits
  • Open Download .zip Download (18 KB)
#!/usr/bin/python
#==============================================================================
# Copyright and license info is available in the LICENSE file included with
# the Server Deployment Package (SDP), and also available online:
# https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/LICENSE
#------------------------------------------------------------------------------

# Based initially on //public/perforce/utils/reviewd/p4review.py#4
#
# Perforce review daemon
#
# This script emails descriptions of new changelists and/or new or modified
# jobs to users who have expressed an interest in them.  Users express
# interest in reviewing changes and/or jobs by setting the "Reviews:" field
# on their user form (see "p4 help user").  Users are notified of changes
# if they review any file involved in that change. Users are notified of
# job updates if they review "//depot/jobs". (This value is configurable
# - see the <jobpath> configuration variable, below).
#
# If run directly with the <repeat> configuration variable = 1, the script
# will sleep for "sleeptime" seconds and then run again.  On UNIX you can
# run the script from cron by setting <repeat> = 0 and adding the following
# line to the cron table with "crontab -e:"
#
#        * * * * * /path/to/p4review.py
#
# This will run the script every minute.  Note that if you use cron you
# should be sure that the script will complete within the time allotted.
#
# The CONFIGURATION VARIABLES below should be examined and in some
# cases changed.
#
#
# Common errors and debugging tips:
#
# -> Error: "command not found" (Windows) or "name: not found" (UNIX) errors.
#
#     - On Windows, check that "p4" is on your PATH or set:
#       p4='"c:/program files/perforce/p4"' (or to the appropriate path).
#       (NOTE the use of " inside the string to prevent interpretation of
#       the command as "run c:/program with arguments files/perforce/p4...")
#
#     - On UNIX, set p4='/usr/local/bin/p4' (or to the appropriate path)
#
# -> Error: "You don't have permission for this operation"
#
#     - Check that the user you set os.environ['P4USER'] to (see below)
#       has "review" or "super" permission via "p4 protect".
#       This user should be able to run "p4 -u username counter test 42"
#       (this sets the value of a counter named "test" to 42)
#
# -> Error: "Unable to connect to SMTP host"
#
#     - check that the mailhost is set correctly - try "telnet mailhost 25"
#       and see if you connect to an SMTP server.  Type "quit" to exit
#
# -> Problem: The review daemon seems to run but you don't get email
#
#     - check the output of "p4 counters" - you should see a counter named
#       "review"
#     - check the output of "p4 reviews -c changenum" for a recent change;
#       if no one is reviewing the change then no email will be sent.
#       Check the setting of "Reviews:" on the user form with "p4 user"
#     - check that your email address is set correctly on the user form
#       (run "p4 reviews" to see email addresses for all reviewers,
#       run "p4 user" to set email address)
#
# -> Problem: Multiple job notifications are sent
#
#     - the script should be run on the same machine as the Perforce
#       server; otherwise time differences between the two machines can
#       cause problems. This is because the job review mechanism uses a
#       timestamp. The change review mechanism uses change numbers, so
#       it's not affected by this problem.


import sys, os, string, re, time, smtplib, traceback
import ConfigParser

config = ConfigParser.RawConfigParser()

#######################################################################
#####                                                             #####
#####  CONFIGURATION VARIABLES: Modify in p4review.cfg as needed. #####
#####                                                             #####

SDP_INSTANCE = sys.argv[1]
config.read('/p4/common/config/p4_%s.p4review.cfg' % (SDP_INSTANCE))

SECTION="%s" % SDP_INSTANCE
debug = int(config.get(SECTION, 'debug'))
administrator = config.get(SECTION, 'administrator')
mailhost = config.get(SECTION, 'mailhost')
repeat = int(config.get(SECTION, 'repeat'))
sleeptime = int(config.get(SECTION, 'sleeptime'))
limit_emails = int(config.get(SECTION, 'limit_emails'))
limit_description = int(config.get(SECTION, 'limit_description'))
notify_changes = int(config.get(SECTION, 'notify_changes'))
notify_jobs = int(config.get(SECTION, 'notify_jobs'))
bcc_admin = int(config.get(SECTION, 'bcc_admin'))
send_to_author = int(config.get(SECTION, 'send_to_author'))
reply_to_admin = int(config.get(SECTION, 'reply_to_admin'))
maildomain = config.get(SECTION, 'maildomain')
complain_from = config.get(SECTION, 'complain_from')
jobpath = config.get(SECTION, 'jobpath')
datefield = config.get(SECTION, 'datefield')
servername = config.get(SECTION, 'servername')

P4PORT = os.environ['P4PORT']

if maildomain == 'None':
    maildomain = None

os.system("/p4/%s/bin/p4_%s login -a < /p4/common/config/.p4passwd.${P4SERVER}.admin > /dev/null" % (SDP_INSTANCE, SDP_INSTANCE))
    # This user must have Perforce review privileges (via "p4 protect")

p4 = '/p4/%s/bin/p4_%s' % (SDP_INSTANCE, SDP_INSTANCE)
    # The path of your p4 executable. You can use
    # just 'p4' if the executable is in your path.
    # NOTE: Use forward slashes EVEN ON WINDOWS,
    # since backslashes have a special meaning in Python)

#############                                                    ##########
#############           END OF CONFIGURATION VARIABLES           ##########
#############                                                    ##########
###########################################################################

bcc_admin = bcc_admin and administrator # don't Bcc: None!
if administrator and reply_to_admin:
  replyto_line='Reply-To: '+administrator+'\n'
else:
  replyto_line=''


def complain(mailport,complaint):
  '''
  Send a plaintive message to the human looking after this script if we
  have any difficulties.  If no email address for such a human is given,
  send the complaint to stderr.
  '''
  complaint = complaint + '\n'
  if administrator:
    mailport.sendmail(complain_from,[administrator],\
      'Subject: Perforce Review Daemon Problem\n\n' + complaint)
  else:
    sys.stderr.write(complaint)

def check_length(body):
  '''
  Prevent changelist with descriptions of greater than limit_description
  from being sent.
  '''
  if len(body) > limit_description:
  	truncate_body = """\
  
  ...
  WARNING: Changelist/Job truncated! REASON: Too many characters.
  Raise limit_description value in Review Daemon. To see the full change
  or job description, run 'p4 describe -s <chg#>' or 'p4 job -o <job#>'
  directly against the Perforce server.
  
  """
  	body = body[:limit_description]
	body = body + truncate_body
  return body    

def mailit(mailport, sender, recipients, message):
  '''
  Try to mail message from sender to list of recipients using SMTP object
  mailport.  complain() if there are any problems.
  '''
  if debug:
    if not administrator:
      print 'Debug mode, no mail sent: would have sent mail ' \
            + 'from %s to %s' % (sender,recipients)
      return
    print 'Sending mail from %s to %s (normally would have sent to %s)' \
           % (sender,administrator,recipients)
    message = message + '\nIN DEBUG MODE: would normally have sent to %s' \
              % recipients
    recipients =  administrator      # for testing or initial setup
  try:
    failed = mailport.sendmail(sender, recipients, message)
  except:
    failed = string.join(apply(traceback.format_exception,sys.exc_info()),'')

  if failed:
    complain( mailport, 'The following errors occurred:\n\n' +\
               repr(failed) +\
              '\n\nwhile trying to email from\n' \
              + repr(sender) + '\nto ' \
              + repr(recipients) + '\nwith body\n\n' + message)


def set_counter(mailport,counter,value):
  if debug: print 'setting counter %s to %s' % (counter,repr(value))
  set_result = os.system('%s counter %s %s > /dev/null' % (p4,counter,value))
  if set_result !=0:
    complain(mailport,'Unable to set review counter - check user %s ' \
                       + 'has review privileges\n(use p4 protect)"' \
                       % os.environ['P4USER'])


def parse_p4_review(command,ignore_author=None):
  reviewers_email = []
  reviewers_email_and_fullname = []

  if debug>1: print 'parse_p4_review: %s' % command
  for line in os.popen(command,'r').readlines():
    if debug>1: print line
    # sample line: james <james@perforce.com> (James Strickland)
    #              user   email                fullname
    try:
      (user,email,fullname) = re.match( r'^(\S+) <(\S+)> \((.+)\)$', line).groups()

      if maildomain:      # for those who don't use "p4 user" email addresses
        email= '%s@%s' % (user, maildomain)

      if user != ignore_author:
        reviewers_email.append(email)
        reviewers_email_and_fullname.append('"%s" <%s>' % (fullname,email))
    except:
      print("Error:", sys.exc_info())
      continue

  if debug>1: print reviewers_email, reviewers_email_and_fullname
  return reviewers_email,reviewers_email_and_fullname


def change_reviewers(change,ignore_author=None):
  '''
  For a given change number (given as a string!), return list of
  reviewers' email addresses, plus a list of email addresses + full names.
  If ignore_author is given then the given user will not be included
  in the lists.
  '''
  return parse_p4_review(p4 + ' reviews -c ' + change,ignore_author)


def review_changes(mailport,limit_emails=100):
  '''
  For each change which hasn't been reviewed yet send email to users
  interested in reviewing the change.  Update the "review" counter to
  reflect the last change reviewed.  Note that the number of emails sent
  is limited by the variable "limit_emails"
  '''
  if debug:
    no_one_interested=1
    current_change=int(os.popen(p4 + ' counter change').read())
    current_review=int(os.popen(p4 + ' counter review').read())
    print 'Looking for changes to review after change %d and up to %d.' \
           % (current_review, current_change)

    if current_review==0:
      print 'The review counter is set to zero.  You may want to set\
it to the last change with\n\n  %s -p %s -u %s counter review %d\n\nor \
set it to a value close to this for initial testing. (The -p and -u may \
not be necessary, but they are printed here for accuracy.)'\
% (p4,os.environ['P4PORT'],os.environ['P4USER'],current_change)
  change = None

  for line in os.popen(p4 + ' review -t review','r').readlines():
    # sample line: Change 1194 jamesst <js@perforce.com> (James Strickland)
    #              change #    author   email             fullname
    if debug: print line[:-1]
    try:
      (change,author,email,fullname) = re.match( r'^Change (\d+) (\S+) <(\S+)> \(([^\)]+)\)', line).groups()

      if maildomain: # for those who don't use "p4 user" email addresses
        email= '%s@%s' % (author, maildomain)

      if send_to_author:
        (recipients,recipients_with_fullnames) = change_reviewers(change)
      else:
        (recipients,recipients_with_fullnames) = change_reviewers(change,author)

      if bcc_admin: recipients.append(administrator)

      if debug:
        if recipients:
          no_one_interested=0
          print ' users interested in this change: %s' % recipients
        else:
          print ' no users interested in this change'
      if not recipients: continue  # no one is interested

      message = 'From: ' + fullname + ' <' + email + '>\n' +\
                'To: ' + string.join(recipients_with_fullnames,', ') + '\n' +\
                'Subject: %s:%s change ' % (servername, P4PORT) + change + ' for review\n' +\
                replyto_line +\
                '\n' +\
                check_length(os.popen(p4 + ' describe -s ' + change,'r').read())

      mailit(mailport, email, recipients, message)
      limit_emails = limit_emails - 1
      if limit_emails <= 0:
        break
    except:
      print("Error:", sys.exc_info())
      continue


  if debug and change and no_one_interested:
    print 'No users were interested in any of the changes above - perhaps \
    no one has set the Reviews: field in their client spec?  (please see \
    p4 help user").'

  # if there were change(s) reviewed in the above loop, update the counter
  if change: set_counter(mailport,'review',change)


def job_reviewers(jobname,ignore_author=None):
  '''
  For a given job, return list of reviewers' email addresses,
  plus a list of email addresses + full names.
  If ignore_author is given then the given user will not be included
  in the lists.
  '''
  return parse_p4_review(p4 + ' reviews ' + jobpath,ignore_author)
           # not the most efficient solution...


def review_jobs(mailport,limit_emails=100):
  '''
  For each job which hasn't been reviewed yet send email to users
  interested in reviewing the job.  Update the "jobreview" counter to
  reflect the last time this function was evaluated.  Note that the number
  of emails sent is limited by the variable "limit_emails" - ***currently
  this causes extra job notifications to be dropped...not optimal...
  '''
  start_time = int(os.popen(p4 + ' counter jobreview').read())
  query_time = int(time.time())
  start_time_string = \
     time.strftime('%Y/%m/%d:%H:%M:%S',time.localtime(start_time))
  query_time_string = \
     time.strftime('%Y/%m/%d:%H:%M:%S',time.localtime(query_time))
  query = \
     '%s>%s&%s<=%s' % (datefield, start_time_string, datefield,\
                       query_time_string)

  if debug:
    no_one_interested=1
    print 'Looking for jobs to review after\n%s \
          (%d seconds since 1 Jan 1970 GMT) \
          and up to\n%s (%d seconds since 1 Jan 1970 GMT).' \
          % (start_time_string, start_time, query_time_string, query_time)

  jobname=None

  for line in os.popen(p4 + ' jobs -e "' + query + '"','r').readlines():
    # sample line: job000001 on 1998/08/10 by james *closed* 'comment'
    #              jobname      date          author
    if debug: print line[:-1]
    try:
      (jobname,author) = re.match( r'^(\S+) on \S+ by (\S+)', line).groups()
      match = re.match( r'^\S+\s+<(\S+)>\s+\(([^\)]+)\)', \
      os.popen(p4 + ' users ' + author,'r').read() )
      if match:
        (email,fullname) = match.groups()
        if maildomain:     # for those who don't use "p4 user" email addresses
          email= '%s@%s' % (author, maildomain)
      else:
        email = administrator
        fullname = "Unknown user: " + author
        complain(mailport,'Unkown user %s found in job %s' % (author,jobname))

      if send_to_author:
        (recipients,recipients_with_fullnames) = job_reviewers(jobname)
      else:
        (recipients,recipients_with_fullnames) = job_reviewers(jobname,author)

      if bcc_admin: recipients.append(administrator)

      if debug:
        if recipients:
          no_one_interested=0
          print ' users interested in this job: %s' % recipients
        else:
          print ' no users interested in this job'
      if not recipients: continue  # no one is interested

      message = 'From: ' + fullname + ' <' + email + '>\n' +\
                'To: ' + string.join(recipients_with_fullnames,', ') + '\n' +\
                'Subject: %s:%s job ' % (servername, P4PORT) + jobname + ' for review\n' +\
                replyto_line +\
                '\n'
      job_body = ''
      for line in os.popen(p4 + ' job -o ' + jobname,'r').readlines():
        if line[0] != '#': job_body = job_body + line
      message = message + check_length(job_body)

      mailit(mailport, email, recipients, message)
      limit_emails = limit_emails - 1
      if limit_emails <= 0:
        complain( mailport, 'email limit exceeded in job review \n- extra jobs dropped!')
        break
    except:
      print("Error:", sys.exc_info())
      continue

  if debug and jobname and no_one_interested:
      print 'No users were interested in any of the jobs above - \
             perhaps no one has set the Reviews: field in their client\
             spec to include the "jobpath", namely "%s".  Please see "p4 \
             help user").' % jobpath
  set_counter(mailport,'jobreview',query_time)

def loop_body(mailhost):
  # Note: there's a try: wrapped around everything so that the program won't
  # halt.  Unfortunately, as a result you don't get the full traceback.
  # If you're debugging this script, remove the special exception handlers
  # to get the real traceback, or figure out how to get a real traceback,
  # by importing the traceback module and defining a file object that
  # will take the output of traceback.print_exc(file=mailfileobject)
  # and mail it (see the example in cgi.py)
  if debug: print 'Trying to open connection to SMTP (mail) \
                   server at host %s' % mailhost
  try:
    mailport=smtplib.SMTP(mailhost)
  except:
    sys.stderr.write('Unable to connect to SMTP host "' + mailhost \
                      + '"!\nWill try again in ' + repr(sleeptime) \
                      + ' seconds.\n')
  else:
    if debug: print 'SMTP connection open.'
    try:
      if notify_changes: review_changes(mailport,limit_emails)
      if notify_jobs: review_jobs(mailport,limit_emails)
    except:
      complain(mailport,'Review daemon problem:\n\n%s' % \
                  string.join(apply(traceback.format_exception,\
                  sys.exc_info()),''))
    try:
      mailport.quit()
    except:
      sys.stderr.write('Error while doing SMTP quit command (ignore).\n')


if __name__ == '__main__':
  if debug: print 'Entering main loop.'
  while(repeat):
    loop_body(mailhost)
    if debug: print 'Sleeping for %d seconds.' % sleeptime
    time.sleep(sleeptime)
  else:
    loop_body(mailhost)
  if debug: print 'Done.'
# Change User Description Committed
#14 30937 Robert Cowham Update p4review2.py to work with Python3
Add basic test harness.
Delete p4review.py which is Python2 and update docs.
#13 27722 C. Thomas Tyler Refinements to @27712:
* Resolved one out-of-date file (verify_sdp.sh).
* Added missing adoc file for which HTML file had a change (WorkflowEnforcementTriggers.adoc).
* Updated revdate/revnumber in *.adoc files.
* Additional content updates in Server/Unix/p4/common/etc/cron.d/ReadMe.md.
* Bumped version numbers on scripts with Version= def'n.
* Generated HTML, PDF, and doc/gen files:
  - Most HTML and all PDF are generated using Makefiles that call an AsciiDoc utility.
  - HTML for Perl scripts is generated with pod2html.
  - doc/gen/*.man.txt files are generated with .../tools/gen_script_man_pages.sh.

#review-27712
#12 27325 C. Thomas Tyler Replaced hard-coding of SDP instance setting, so we don't assume
it is Instance 1.
#11 20170 Russell C. Jackson (Rusty) Moved password and users into the config directory to allow for instance specific
users and passwords. Ran into a case where two different teams were sharing the same
server hardware and needed this type of differentiation. Surprised that we haven't hit
this sooner.

Also defaulted mkdirs to use the numeric ports since this is the most common
installation.
#10 16029 C. Thomas Tyler Routine merge to dev from main using:
p4 merge -b perforce_software-sdp-dev
#9 13906 C. Thomas Tyler Normalized P4INSTANCE to SDP_INSTANCE to get Unix/Windows
implementations in sync.

Reasons:
1. Things that interact with SDP in both Unix and Windows
environments shoudn't have to account for this obscure
SDP difference between Unix and Windows.  (I came across
this doing CBD work).

2. The Windows and Unix scripts have different variable
names for defining the same concept, the SDP instance.
Unix uses P4INSTANCE, while Windows uses SDP_INSTANCE.

3. This instance tag, a data set identifier, is an SDP concept.
I prefer the SDP_INSTANCE name over P4INSTANCE, so I prpose
to normalize to SDP_INSTANCE.

4. The P4INSTANCE name makes it look like a setting that might be
recognized by the p4d itself, which it is not.  (There are other
such things such as P4SERVER that could perhaps be renamed as
a separate task; but I'm not sure we want to totally disallow
the P4 prefix for variable names. It looks too right to be wrong
in same cases, like P4BIN and P4DBIN.  That's a discussion for
another day, outside the scope of this task).

Meanwhile:
* Fixed a bug in the Windows 2013.3 upgrade script that
was referencing undefined P4INSTANCE, as the Windows
environment defined only SDP_INSTANCE.

* Had P4INSTANCE been removed completely, this change would
likely cause trouble for users doing updates for existing
SDP installations.  So, though it involves slight technical debt,
I opted to keep a redundant definition of P4INSTANCE
in p4_vars.template, with comments indicating SDP_INSTANCE should be
used in favor of P4INSTANCE, with a warning that P4INSTANCE
may go away in a future release.  This should avoid unnecessary
upgrade pain.

* In mkdirs.sh, the varialbe name was INSTANCE rather than
SDP_INSTANCE.  I changed that as well.  That required manual
change rather than sub/replace to avoid corrupting other similar
varialbe names (e.g.  MASTERINSTANCE).

This is a trivial change technically (a substitute/replace, plus
tweaks in p4_vars.template), but impacts many files.
#8 12923 C. Thomas Tyler Routine merge down from main to dev.
Resolved with 'p4 resolve -as', no merges or conflicts.
#7 12169 Russell C. Jackson (Rusty) Updated copyright date to 2015

 Updated shell scripts to require an instance parameter to eliminate the need
 for calling p4master_run.    Python and Perl still need it since you have to set the
environment for them to run in.

 Incorporated comments from reviewers. Left the . instead of source as that seems
more common in the field and has the same functionality.
#6 12028 C. Thomas Tyler Refreshed SDP dev branch, merging down from main.
#5 11503 Russell C. Jackson (Rusty) Changed SECTION to text instead of integer.
#4 11477 Russell C. Jackson (Rusty) Updated to use /usr/bin/env python

Added workshop header.

 Changed cfg to config.
#3 11474 Russell C. Jackson (Rusty) Had to move the cfg directory to the metadata volume and link it under the instance
directory to provide the proper separation in a shared volume environment. The
 instance specific vars cannot be in a shared directory since they need to be different
 on each node using the shared volume.

 Since the files moved back to the instance directory, I changed the names back to:
   instance_vars
   p4review.cfg
 to keep things simple.

 Also moved p4_vars.template to SDP/Server/Unix/p4/common/cfg so that it doesn't get
 copied to the /p4/common/bin folder. Plus, it makes more sense for it to be in that
 directory in the SDP structure.
#2 11466 Russell C. Jackson (Rusty) Initial work to simplify p4_vars and remove cluster stuff.

 Testing of named instances surfaced some bugs that are in prod sdp,
 now fixed in dev.

 Added three triggers from RCJ SDP

 Moved p4review.cfg into the new /p4/common/cfg to go along with the
 instance_vars files. mkdirs.sh now generates an instance_p4review.cfg
 as well.

 Removed incremental p4verify to clean up a bit. It didn't support replicas
 and was really never used.

 All port settings now live in <instance>_vars file. You set what you want
 the ports to be in mkdirs.sh. There is no more fancy logic to try to guess
 what the port should be. You set it, and that is what it is.

 Remaining to do is to updated scripts to not need p4master_run. Saved
 that work for later since this is tested and works.
#1 10638 C. Thomas Tyler Populate perforce_software-sdp-dev.
//guest/perforce_software/sdp/main/Server/Unix/p4/common/bin/p4review.py
#1 10148 C. Thomas Tyler Promoted the Perforce Server Deployment Package to The Workshop.