In This Package:

dbsvn.py

Go to the documentation of this file.

#!/usr/bin/env python
"""

Usage examples
~~~~~~~~~~~~~~

::

   ./dbsvn.py --help
       ## full list of options and this help text 

   ./dbsvn.py ~/catdir -M
       ## check catalog and skip commit message test    

   ./dbsvn.py ~/catdir -m "test commit message  dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000  "
       ## check catalog and commit message 

This script performs basic validations of SVN commits intended to lead to DB updates, 
it is used in two situations:

#. On the SVN server as part of the pre-commit hook that allows/denies the commit
#. On the client, to allow testing of an intended commit before actually attempting the commit as shown above

NB this script DOES NOT perform commits, it only verifies them


How this script fits into the workflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

   cd ; svn co http://dayabay.ihep.ac.cn/svn/dybaux/catalog/tmp_offline_db 
         ## TEMORARILY DYBAUX NEEDS SOME CLEAN UP ... SO USE THE BELOW URL

   cd ; svn co http://dayabay.phys.ntu.edu.tw/repos/newtest/catalog/tmp_offline_db/
         ## check out catalog as left by the last updater

   ./db.py offline_db rdumpcat ~/tmp_offline_db     
         ## rdumpcat current offline_db on top of the SVN checkout and look for diffs

   svn diff ~/tmp_offline_db
         ## COMPLAIN LOUDLY IF YOU SEE DIFFS HERE BEFORE YOU MAKE ANY UPDATES     

   ./db.py tmp_joe_offline_db rdumpcat ~/tmp_offline_db       ## NB name switch
         ## write DBI catalog on top of working copy ~/tmp_offline_db 
   
   svn diff ~/tmp_offline_db     
         ## see if changed files are as you expect

   ./dbsvn.py ~/tmp_offline_db 
         ## use this script to check the "svn diff" to see if looks like a valid DBI update

   ./dbsvn.py ~/tmp_offline_db -m "Updating dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000 " 
         ## fails as annotation link refers to dummy path, no such package and no change to that file at that revision 

   ./dbsvn.py ~/tmp_offline_db -m "Annotation link  dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README "
         ## check the "svn diff" and intended commit message, fails as no revision  

   ./dbsvn.py ~/tmp_offline_db -m "Annotation link  dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README@10000 "
         ## fails as no change to that file at that revision 

   ./dbsvn.py ~/tmp_offline_db -m "Annotation link  dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README@9716 " 
         ## succeeds 

   svn ci ~/tmp/offline_db -m  "Updating dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000 " 
         ## attempt the actual commit 
 

What is validated by :file:`dbsvn.py`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#. The commit message, eg "Updating dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000 "

   #. must provide valid dybsvn reference which includes dybgaudi/trunk package path and revision number  

#. Which files (which represent tables) are changed

   #.  author must have permission for these files/tables 
   #.  change must effect DBI file/tablepairs (payload, validity)

#. What changes are made:
 
   #. must be additions/subtractions only  (allowing subtractions is for revertions) 
   #. note that LOCALSEQNO (a DBI bookkeeping table) is a special case 

    
Rationale behind these validations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#. valid DBI updates
#. establish provenance and purpose
   
    #. what purpose for the update
    #. where it comes from (which revision of which code was used) 
    #. precise link to producing code and documentation 


Commit denial
~~~~~~~~~~~~~~

This script is invoked on the SVN server by the pre-commit hook (shown below)
if any directories changed by the commit start with "catalog/". 
If this script exits normally with zero return code, the commit is 
allowed to proceed.

On the other hand, if this script returns a non-zero exit code, 
for example if an assert is tickled, then the commit is denied and stderr 
is returned to the failed committer.


OVERRIDE commits
~~~~~~~~~~~~~~~~~

Administrators (configured using -X option on the server) can 
use the string "OVERRIDE" in commit messages to short circuit validation.
This is needed for non-standard operations, currently:

#. adding/removing tables 

A commit like the below from inside catalog will fail, assuming that the **dayabay** svn identity 
is not on the admin list::

   svn --username dayabay ci -m "can dayabay use newtest OVERRIDE "



Deployment of pre-commit hook on SVN server 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Only SVN repository administrators need to understand this section.

The below commands are an example of creating a bash pre-commit wrapper. 
After changing the TARGET and apache user identity, the commands can be
used to prepare the hook.
Note that the pre-commit script is invoked by the server in a bare environment, 
so any customizations must be propagated in, deploy the code::

    cd ; svn co http://dayabay.ihep.ac.cn/svn/dybsvn/dybgaudi/trunk/DybPython/python/DybPython
    export TARGET=/var/scm/repos/newtest/hooks/pre-commit
    sudo bash -c "cp $HOME/DybPython/{dbsvn,svndiff}.py $(dirname $TARGET)/  && chown nobody.nobody  $(dirname $TARGET)/{dbsvn,svndiff}.py "

    DBSVN_XREF=/var/scm/svn/dybsvn python dbsvn.py HOOK    ## check the hook is customized as desired
    DBSVN_XREF=/var/scm/svn/dybsvn python dbsvn.py HOOK  | sudo bash -c "cat - > $TARGET && chmod ugo+x $TARGET && chown nobody.nobody $TARGET " 


A pre-commit hook testing harness is available in bash functions :env:`trunk/svn/svnprecommit.bash`


Typical Problems with the Hook 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Mainly for admins**

If the precommit hook is mis-configured the likely result is that attempts to 
commit will hang. For example the :file:`dbsvn.py` invokation in the hook script needs to have:

#. a valid admin user (SVN identity) 
#. local filesystem repository path for the cross reference `-r` option

The default cross reference path is the dybsvn URL which might hang on the server 
as the user(root/nobody/...) that runs the SVN repository normally does not 
have user permissions to access sibling repository dybsvn. (have switched to non-interactive now) 



"""


import sys, os, re
from svndiff import Diff, SVN, SVNLook

log = lambda msg:sys.stderr.write(msg+"\n")
cmd = lambda _:os.popen(_).read().rstrip()

class Hook(dict):
    tmpl = """#!/bin/bash
export LD_LIBRARY_PATH=%(LD_LIBRARY_PATH)s
export SVNLOOK=%(SVNLOOK)s
    
dbi=0
dirs=$(%(SVNLOOK)s dirs-changed $1 --transaction $2 )
for dir in $dirs ; do
    case $dir in 
        catalog*) dbi=1 ;; 
    esac
    echo "dir $dir dbi $dbi " >&2    
done 

# skip DBI validation if none of the dirs-changed start with "catalog"
[ "$dbi" == "0" ] && exit 0 

%(PYTHON)s $(dirname $0)/dbsvn.py $* -X %(USER)s -r %(XREF)s
exit $?
"""
    def __init__(self, *args, **kwargs):
         dict.__init__(self, *args, **kwargs)
         self.update( LD_LIBRARY_PATH=os.environ['LD_LIBRARY_PATH'], SVNLOOK=cmd("which svnlook"),  PYTHON=cmd("which python"), USER=os.getlogin(), XREF=os.environ.get('DBSVN_XREF','http://dayabay.ihep.ac.cn/svn/dybsvn') ) 
    __str__ = lambda self:self.tmpl % self





class Msg(dict):
    """
    A valid DBI update commit message contains a link of the form::

        dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README@9716 

    The link provides crucial association between a DB update 
    and the package and revision of the code used in the preparation 
    and invokation of the update.

    Dummy links will be denied, a real path that was modified at the specified 
    revision is required.

    """
    baseurl="http://dayabay.ihep.ac.cn/svn/dybsvn"
    tracurl="http://dayabay.ihep.ac.cn/tracs/dybsvn"
    _ptn = re.compile("(?P<link>dybsvn:source:(?P<path>dybgaudi/trunk/(?P<relpath>\S*))\@(?P<rev>\d*))\s*")
    logurl = property(lambda self:os.path.join(self.tracurl,"log", self['path'] ))

    def validate_annotation_xref( self, refpath ):
        """ 
        Check that the annotation link points to a valid changed annotation file in xref repository  
        """
        revision = self.get('rev', None)
        path     = self.get('path', None)
        assert revision and path , "failed to find annotation link path and revision in commit message "

        if refpath.startswith("http:"):
            xmd = SVN( path=refpath , revision=revision )         
        elif os.path.exists(refpath):
            xmd = SVNLook( repo_path=refpath , revision=revision )
        else:
            assert 0, "refpath is invalid %s " % refpath
        changed = xmd.changed

        errmsg = lambda:"\n".join([
                    "INVALID COMMIT MESSAGE",
                    "referenced repository path \"%s\" was not touched by revision \"%s\"" % ( path, revision) , 
                    "the below paths were changed in this revision... ",] + changed + 
                    ["", "check trac url %s to find valid links/revisions " % self.logurl ,"" ]  )
        assert path in changed, errmsg()

    def __init__(self, msg ):
        dict.__init__(self)
        self.msg = msg
        m = self._ptn.search( msg )
        assert m, "INVALID COMMIT MESSAGE %s " % self.msg + self.__doc__
        self.update( m.groupdict() )

    def __repr__(self):
        return "Msg %s %r " % ( self.msg, dict.__repr__(self) ) 


class DBIValidate(list):
    """
    Basic validation of commit that represents an intended DB update
    """
    def __init__(self, diff, msg , author ):
         self.diff = diff
         self.msg = msg
         self.author = author
         self.tabledict = dict([(c.name,c.smry) for c in self.diff.children])

    tables = property(lambda self:[c.name for c in self.diff.children])
    exts   = property(lambda self:[c.ext  for c in self.diff.children])

    def validate_msg(self):
        pass

    def validate_update( self):
        """
        Current checks do not verify tail addition  
        """
        tdict = self.tabledict 
        tabs = tdict.keys()
        print tabs
        print tdict

        assert 'LOCALSEQNO' in tabs, "No LOCALSEQNO in %s " % tabs
        assert tdict['LOCALSEQNO'] == "-+", "Unexpected LOCALSEQNO change %r " % tdict 
        tabs = filter(lambda t:t != 'LOCALSEQNO', tabs )
        assert len(tabs) % 2 == 0 , "An even number of changed tables is required %r " % tabs
        vlds = filter( lambda t:t[-3:] == 'Vld' , tabs )
        assert len(vlds) == len(tabs)/2 , "Need equal number of payload and validity table changes "
        for vld in vlds:
            pay = vld[:-3]
            assert pay in tabs, "Vld table %s is not paired " % vld 

    def __call__(self):
        self.validate_msg()
        self.validate_update()


def main():
    from optparse import OptionParser
    op = OptionParser(usage=__doc__ )
    op.add_option("-v", "--verbose", action="store_true" )
    op.add_option("-m", "--message", help="Commit message to be validated, client side only. Default %default "  )
    op.add_option("-r", "--refpath", help="URL or filesystem path of cross reference checking repository. Default %default "  )
    op.add_option("-M", "--no-message-chk",  action="store_true",  help="Skip Commit message check, client side only. Default %default "  )
    op.add_option("-a", "--author",  help="Author identity, client side only. Default %default "  )
    op.add_option("-X", "--admins",  help="Comma separated list of SVN identity names of admins, server side. Default %default "  )
    op.set_defaults( verbose=False, message="no-message", author="unknown", admins="", no_message_chk=False , refpath="http://dayabay.ihep.ac.cn/svn/dybsvn" )
    (opts_ , args) = op.parse_args()
    opts = vars(opts_)
    if len(args)==0:args = [os.getcwd()]    ## default to current working directory 

    if len(args) == 1:
        if args[0] == "HOOK":
            print Hook()
            sys.exit(0)
        cmd = SVN(path=args[0], msg=opts['message'], author=opts['author'] ) 
    elif len(args) == 2:
        cmd = SVNLook(repo_path=args[0], txn_name=args[1], )
    else:
        print __doc__
        sys.exit(1) 
    
    admins = opts['admins'].split(",")


    author = cmd.author
    msg_ = cmd.msg 

    ## commit messages containing OVERRIDE by authors in the administrators list 
    ## short circuit the validations ... this is needed for non-standard operations such
    ## as adding or removing tables from the catalog 
    if msg_.find("OVERRIDE") > -1:
         if author in admins:
             sys.exit(0) 
         else:
             log("your identity %r is not in the admin users list  %r so you cannot use the OVERRIDE control  " % (author, admins) ) 
             sys.exit(1)


    lines = cmd.diff.split("\n")

    if not opts['no_message_chk']:
        msg = Msg( msg_ )
        log("commit msg: %r author: %r admins: %r  " % (msg,author,admins) )
        msg.validate_annotation_xref( opts['refpath'] )
    else:
        msg = None 



    maxl = 1000
    for i,line in enumerate(lines[0:maxl]):
        log("[%2d] %s " % (i+1, line))
    if len(lines) > maxl:
        log("TOTAL of %d lines truncated to max %d " %( len(lines), maxl) )  


    diff = Diff(lines)
    diff.dump()

    dbiv = DBIValidate( diff, msg, author )
    dbiv()

if __name__=='__main__':
    main()

---