CheckCaseTrigger.py #1

  • //
  • guest/
  • perforce_software/
  • sdp/
  • dev/
  • Unsupported/
  • Samples/
  • triggers/
  • CheckCaseTrigger.py
  • View
  • Commits
  • Open Download .zip Download (11 KB)
#!/p4/common/python/bin/python3
# Version 2.0.5

"""
CaseCheckTrigger.py

Example Usage (if server is standard SDP and has appropriate environment defaults for P4PORT and P4USER):

    checkcase change-submit //... "python3 /p4/common/bin/triggers/CheckCaseTrigger.py %change% "

Option where a specific user is specified to avoid trigger (e.g. git-fusion-user)

    CheckCaseTrigger change-submit //... "/p4/common/bin/triggers/CheckCaseTrigger.py %changelist% myuser=%user%"

Old style specifying params as keywords:

    checkcase change-submit //... "python3 c:\perforce\scripts\CheckCaseTrigger.py %changelist% port=%serverport% user=perforce"

NOTE:
    'mysuser' is only used to exclude the 'git-fusion-user' from this check.
    Even if you are not using Git Fusion we recomend setting the trigger up this
    way just in case you install Git Fusion in future.

Sample output:
    Submit validation failed -- fix problems then use 'p4 submit -c 1234'.
    'CheckCaseTrigger' validation failed:

    Your submission has been rejected because the following files
    are inconsistent in their use of case with respect to existing
    directories

    Your file:         '//depot/dir/test'
    existing file/dir: '//depot/DIR'

    Note: This trigger requires P4Triggers.py and the P4Python API.
"""

from __future__ import print_function

import P4
import P4Triggers
import os
import re
import subprocess
import sys

class CheckCaseTrigger(P4Triggers.P4Trigger):
    """CaseCheckTrigger is a subclass of P4Trigger. Use this trigger to ensure
       that your depot does not contain two filenames or directories that only
       differ in case.
       Having files with different case spelling will cause problems in mixed
       environments where both case-insensitive clients like Windows and case-
       sensitive clients like UNIX access the same server.
    """

    def __init__(self, *args, maxErrors=10, **kwargs):
        kwargs['charset'] = 'none'
        kwargs['api_level'] = 71
        
        fileFilter = None
        if 'filefilter' in kwargs:
            fileFilter = kwargs['filefilter']
            del kwargs['filefilter']
        
        P4Triggers.P4Trigger.__init__(self, **kwargs)
        self.parse_args(__doc__, args)

        # need to reset the args in case a p4config file overwrote them
        for (k, v) in kwargs.items():
            if k != "log":
                try:
                    setattr(self.p4, k, v)
                except:
                    self.logger.error("error setting p4 property: '%s' to '%v'" % (k, v))

        self.map = None
        if fileFilter:
            try:
                with open(fileFilter) as f:
                    self.map = P4.Map()
                    for line in f:
                        self.map.insert(line.strip())
            except IOError:
                self.logger.error("Could not open filter file %s" % fileFilter)
        
        self.maxErrors = maxErrors
        self.depotCache = {}
        self.dirCache = {}
        self.fileCache = {}

    def add_parse_args(self, parser):
        """Specific args for this trigger - also calls super class to add common trigger args"""
        parser.add_argument('change', help="Change to validate - %%change%% argument from triggers entry.")
        super(CheckCaseTrigger, self).add_parse_args(parser)

    def setUp(self):
        info = self.p4.run_info()[0]
        if "unicode" in info and info["unicode"] == "enabled":
            self.p4.charset = "utf8"
        self.p4.exception_level = 1 # ignore WARNINGS like "no such file"
        self.p4.prog = "CheckCaseTrigger"

        self.USER_MESSAGE="""

    Your submission has been rejected because the following files
    are inconsistent in their use of case with respect to existing
    directories

    """

        self.BADFILE_FORMAT="""
      Your file:         '%s'
      existing file/dir: '%s'
      """

    def validate(self):
        """Here the fun begins. This method overrides P4Trigger.validate()"""
        badlist = {}
        files = self.change.files

        self.filterRenames(files)

        for file in files:
            action = file.revisions[0].action
            if self.map and self.map.includes(file.depotFile):
                continue
            if action not in ("add", "branch", "move/add"):
                continue

            mismatch = self.findMismatch(file.depotFile)
            if mismatch:
                badlist[file.depotFile] = mismatch

         # end for
      
        if len(badlist) > 0:
            self.report(badlist)
    
        return len(badlist) == 0

    # Method canonical 
    # IN:  string, utf8 compatible
    # OUT: unicode string, all lower case
    def canonical(self, aString):
        if self.p4.charset == "utf8":
            return unicode(aString, "utf8").lower()
        else:
            return aString.lower()

    # Method filterRenames:
    # Removes pairs of files that only differ in case
    # where one action is branch, and the other delete
    def filterRenames(self, files):
        branches = [x for x in files if x.revisions[0].action == 'branch']
        deletes = [x.depotFile.lower() for x in files if x.revisions[0].action == 'delete']
        
        for f in branches:
            if f.depotFile.lower() in deletes:
                files.remove(f)

    # This method returns either a depot or directory that differs from 
    # the supplied path only in case, or None if no mismatch is found
    def findMismatch(self, path):
        path = path[2:] # cut the // at the beginning of the depot off
        dirs = path.split('/')
        depot = dirs.pop(0)
        file = dirs.pop()

        depotMismatch = self.findDepotMismatch(depot)
        if (depotMismatch):
            return depotMismatch

        oldDirs = dirs[:]
        dirMismatch = self.findDirMismatch('//' + depot, dirs)
        if (dirMismatch):
            return dirMismatch

        fileMismatch = self.findFileMismatch(depot, oldDirs, file)
        if (fileMismatch):
            return fileMismatch

        return None

    # This method looks for a depot mismatch
    # It caches the depots as it found them
    # IN: depot name (as provided by the change list
    # OUT: None if (no mismatch found) else (stored depot name)

    def findDepotMismatch(self, depot):
        canonicalDepot = self.canonical(depot)

        # if the depot is in the cache, the cached version is authorative
    
        if canonicalDepot in self.depotCache:
            if self.depotCache[canonicalDepot] == depot:
                return None
            else:
                return '//' + self.depotCache[canonicalDepot]

        # depot not found in cache. Populate the cache
    
        mismatch = None
        for d in self.p4.run_depots():
            dname = d["name"]
            cd = self.canonical(dname)
            self.depotCache[cd] = dname
            if canonicalDepot == cd and depot != dname:
                mismatch = '//' + dname

        return mismatch
    # end findDepotMismatch

    # Look for matching or mismatching directories
    # Recursive algorithm, descending down the directory paths
    # Caching directories as we find them
    # 

    def findDirMismatch(self, top, dirs):
        if len(dirs) == 0:
            return None

        aDir = dirs.pop(0)
        path = top + '/' + aDir
        cpath = self.canonical(path)

        if cpath in self.dirCache :
            # negative lookups are cached with the value set to False
            if (self.dirCache[cpath] == path) or (self.dirCache[cpath] == False):
               return self.findDirMismatch(path, dirs)
            else:
                # it is a mismatch, either with a previously stored name, or, worse
                # with a previous file in the same change list
            
                return self.dirCache [cpath]
            # end if
        # end if

        # so, it is not in the cache. Let's populate the cache and check
        # while we are at it
        
        mismatch = None
        for d in self.p4.run_dirs(top + "/*"):
            d = d["dir"]  # result is in tagged mode, single entry "dir"=>directory name
            cd = self.canonical (d)
            self.dirCache[cd] = d
            
            if cd == cpath:
                # we found the dir entry. Is it the right case?
                
                if d == path:
                    # so far so good, lets step one directory level down
                    
                    mismatch = self.findDirMismatch(path, dirs)
                else:
                    # oh-ho, we found a mismatch
                    mismatch = d
                # end if d == path
            # end if cd == cpath
        # end for

        if not mismatch:
            # enter as a negative match
            self.dirCache [cpath] = False

        return mismatch

    def findFileMismatch(self, depot, dirs, file):
        base = '//' + depot + '/'
        if (len(dirs) > 0):
            base += '/'.join(dirs)
            base += '/'

        name = base + file
        cname = self.canonical(name)
        if cname in self.fileCache:
            if (self.fileCache[cname] == name):
                return None # all is well, but cannot happen, add would fail
            else:
                return self.fileCache[cname]

        # so the file is not in the cache. Let's load the cache
        for f in self.p4.run_files(base + "*"):
            if "delete" not in f["action"]:
                f = f["depotFile"]
                cf = self.canonical(f)
                self.fileCache[cf] = f
                if (cf == cname and f != name):
                    return f

        # so it is not in the file cache
        self.fileCache[cname] = name
        return None

    def report(self, badfiles):
        msg = self.USER_MESSAGE
        for (n, (file, mismatch)) in enumerate(badfiles.items()):
            if n >= self.maxErrors:
                break
            msg += self.BADFILE_FORMAT % (file, mismatch)

        self.message(msg)

    def run(self):
        """Runs trigger"""
        try:
            self.logger.debug("CheckCaseTrigger firing")
            self.setupP4()
            return self.parseChange(self.options.change)
        except Exception:
            return self.reportException()
        return 0

# If called from the command line, go in here

if __name__ == "__main__":
    # Generate new args - parsing out port=123 style way of specifying parameters intended for p4 properties
    kwargs = {}
    args = []
    for arg in sys.argv[1:]:
        p = arg.split("=", 1)
        if len(p) == 1:
            args.append(arg)
        else:
            kwargs[p[0]] = p[1]

    # Example of how to exclude the 'git-fusion-user'
    # Note: Need to remove 'mysuser' after test as it's not a valid P4 argument.
    if 'myuser' in kwargs: 
        if kwargs['myuser'] == 'git-fusion-user':
            sys.exit(0)
        else:
            del kwargs['myuser']

    # If the changelist description contains the text BYPASS_CASE_CHECK, bypass the
    # case check logic.
    # Grab the changelist description, and scan for the bypass token string.
    # If the token is detected, silently and immediately exit with a happy 0 exit code.
    changelist = sys.argv[1]
    cmd = "%s -ztag -F %%desc%% describe -f -s %s" % (os.getenv('P4BIN','p4'), changelist)
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
    (changeDesc, err) = p.communicate()
    p_status = p.wait()

    if (re.search (b'BYPASS_CASE_CHECK', changeDesc, re.MULTILINE)):
        sys.exit(0)

    ct = CheckCaseTrigger(*args, maxErrors=10, **kwargs)
    sys.exit(ct.run())
# Change User Description Committed
#10 29571 Andy Boutte Adding correct spacing after .
 and ,  to increase readability
#9 29320 kathy_rayburn Fix bug the allowed adds of directories that differed only by case.
Add test cases to verify that this bug is fixed.
Add test case to verify that "p4 files -i" works as expected.
Add ability to run test cases on a case-insensitive server.
#8 29174 kathy_rayburn Fix for directory case-only rename bug.
Extensive modification of CheckCaseTrigger.py
Addition of test cases to TestCheckCaseTrigger.py
#7 29066 kathy_rayburn #review-29062
CheckCaseTrigger.py performance changes and bug fix for file/directory case comparisons.
Also, always call p4 describe -s, not p4 describe.
#6 29043 Andy Boutte SDP-824 - Update to CheckCaseTrigger to support python3 and Helix Core in unicode mode
#5 27726 C. Thomas Tyler Generated HTML and PDF from adoc for Unsupported folder.

Corrected Makefile so 'make clean' also removes *.html files.

Added missing doc tags in CheckCaseTrigger.py.
#4 27722 C. Thomas Tyler Refinements to @27712:
* Resolved one out-of-date file (verify_sdp.sh).
* Added missing adoc file for which HTML file had a change (WorkflowEnforcementTriggers.adoc).
* Updated revdate/revnumber in *.adoc files.
* Additional content updates in Server/Unix/p4/common/etc/cron.d/ReadMe.md.
* Bumped version numbers on scripts with Version= def'n.
* Generated HTML, PDF, and doc/gen files:
  - Most HTML and all PDF are generated using Makefiles that call an AsciiDoc utility.
  - HTML for Perl scripts is generated with pod2html.
  - doc/gen/*.man.txt files are generated with .../tools/gen_script_man_pages.sh.

#review-27712
#3 27104 C. Thomas Tyler CheckCaseTrigger.py v2.1.3:
* Added an optional 'allowbypass=no' parameter to enforce the case
check policy strictly, disallowing the bypass.
* If bypass is allowed, the BYPASS_CASE_CHECK feature is now
documented in that it appears in the message users see when the
trigger fires, instructing them how to bypass the trigger, but
warning of the perils of doing so.
* Generally improved the user message when the trigger fires,
prescribing next steps for the user, with different options
output depending on whether the bypass is enabled.
* Changed sample trigger call to avoid calling a random interpreter
in the PATH on Linux systems, where the best practice is to use the
shebang line in the script. (The Windows sample still illustrates
calling an interpreter from the PATH, as that is the best practice
on Windows.)
* Made some internal coding style consistency tweak (no functional
impact).

#review-27105
#2 26681 Robert Cowham Removing Deprecated folder - if people want it they can look at past history!
All functions have been replaced with standard functionality such as built in LDAP,
or default change type.
Documentation added for the contents of Unsupported folder.
Changes to scripts/triggers are usually to insert tags for inclusion in ASCII Doctor docs.
#1 26652 Robert Cowham This is Tom's change:

Introduced new 'Unsupported' directory to clarify that some files
in the SDP are not officially supported. These files are samples for
illustration, to provide examples, or are deprecated but not yet
ready for removal from the package.

The Maintenance and many SDP triggers have been moved under here,
along with other SDP scripts and triggers.

Added comments to p4_vars indicating that it should not be edited
directly. Added reference to an optional site_global_vars file that,
if it exists, will be sourced to provide global user settings
without needing to edit p4_vars.

As an exception to the refactoring, the totalusers.py Maintenance
script will be moved to indicate that it is supported.

Removed settings to support long-sunset P4Web from supported structure.

Structure under new .../Unsupported folder is:
   Samples/bin             Sample scripts.
   Samples/triggers        Sample trigger scripts.
   Samples/triggers/tests  Sample trigger script tests.
   Samples/broker          Sample broker filter scripts.
   Deprecated/triggers     Deprecated triggers.

To Do in a subsequent change: Make corresponding doc changes.
//guest/perforce_software/sdp/dev/Server/Unix/p4/common/bin/triggers/CheckCaseTrigger.py
#6 25715 Robert Cowham Refactor CheckCaseTrigger to work in SDP trigger style - and fix SDP failures.
       Added modified version of Sven's test harness which works (for Mac at least where
some tests must be skipped due to filesystem being case insensitive).
#5 24510 C. Thomas Tyler Enhanced CheckCaseTrigger.py so BYPASS_CASE_CHECK override feature
works even if the defaultChangeType configurable is set to
restricted.
#4 24460 C. Thomas Tyler Tweak to case check trigger to allow user to bypass the safety
feature by including the token BYPASS_CASE_CHECK in the changelist
description.

This is likely useful just after initial rollout of the CaseCheckTrigger
on an existing server that already contains case inconsistencies.
Bypassing the trigger may be required to do some cleanup of existing
data.

If this trigger is deployed on a new server initially, the bypass may
not be needed.
#3 21120 C. Thomas Tyler Corrected shebang line in CheckCaseTrigger.
Added manual-update version id to replace keyword tag.
#2 21098 C. Thomas Tyler SDP-ified:
* Changed sample path to reference SDP /p4/common/bin/triggers
location.
* Changed shebang line to use SDP standard python, which includes
P4Python.
* Removed the '$Id:' RCS keywordt ag line, as RCS tags aren't allowed
in the SDP (since SDP scripts live in many Perforce servers).
* Changed file time from text+kx to text+x.
* Updated copyright up thru 2016.
* One minor cosmetic tweak to doc text.

#review-21099
#1 21097 C. Thomas Tyler Branched CheckCase trigger into the SDP.
//guest/robert_cowham/perforce/utils/triggers/CheckCaseTrigger.py
#7 19940 Robert Cowham Tabs->spaces, adjust some other whitespace
#6 19939 Robert Cowham Update with latest changes by Sven etc.
#5 8050 Robert Cowham P4Python 2009.1
#4 8049 Robert Cowham Whitespace only change and comments.
Made indents standard and removed tabs
#3 8048 Robert Cowham Add comment about installing
#2 8046 Robert Cowham Latest change from Sven
#1 7531 Robert Cowham Personal branch
//guest/sven_erik_knop/P4Pythonlib/triggers/CheckCaseTrigger.py
#4 7379 Sven Erik Knop Added output to a log file.
The default is the send output to p4triggers.log in the P4ROOT directory, this can be overridden with the parameter log=<path>
Also, errors now cause the trigger to fail with sensible output first.
#3 7372 Sven Erik Knop Rollback Rename/move file(s).
To folder "perforce" is needed.
#2 7370 Sven Erik Knop Rename/move file(s) again - this time to the right location inside a perforce directory.
#1 7367 Sven Erik Knop New locations for the Python triggers.
//guest/sven_erik_knop/perforce/P4Pythonlib/triggers/CheckCaseTrigger.py
#1 7370 Sven Erik Knop Rename/move file(s) again - this time to the right location inside a perforce directory.
//guest/sven_erik_knop/P4Pythonlib/triggers/CheckCaseTrigger.py
#1 7367 Sven Erik Knop New locations for the Python triggers.
//guest/sven_erik_knop/triggers/CheckCaseTrigger.py
#3 7219 Sven Erik Knop First attempt for renamer support, not finished yet, therefore disabled.
#2 7218 Sven Erik Knop Updated CheckCaseTrigger.py to fix problems with files within directories.

The trigger would not detect case problems for files that are located in
subdirectories. Unintentional side effect of modifying the dirs list recursively
when checking for mismatched directories.
The solution was simple: make a copy of the directory list for the file check.
#1 6413 Sven Erik Knop Added some P4Python-based Perforce triggers.

P4Triggers.py is the based class for change trigger in Python modelled on
Tony Smith's Ruby trigger with the same name.

CheckCaseTrigger.py is a trigger that ensures that no-one enters a file
or directory with a name only differing by case from an existing file. This
trigger is Unicode aware and uses Unicode-comparison of file names, so it
can be used on nocase-Unicode based Perforce servers, which cannot catch
the difference between, say, "�re" and "�re" at the moment.

clienttrigger.py is a simple trigger that modifies the option "normdir" to
"rmdir" for new client specs only. It is meant as a template to create more
complex default settings like standard views.