Dariusz on Software

Perforce is a commercial, centralised VCS (Version Control System). I started using it to exchange data with customer's codebase. Internal development is using GIT, snapshots are stored inside Perforce.

Checkout of full tree requires to download many gigabytes of data thus was not an option. I was interested only in small part of whole repository. Subtree checkout syntax is not very obvious (had to google for it):

p4 sync //path/to/directory/...

I think, this hint may be useful to someone that must (want?)* use this VCS implementation.

(*) Who will pay 700 USD per user for Subversion replacement? Am I missing something valuable in P4 that is worth of this money? Commercial support :-)?

Status command is very important part of any VCS (Version Control System) local interface. It allows you to check your workspace state and ensure correct commit will be created.

Perforce is a commercial VCS that is similar to CVS (revisions per file) and SVN (global revisions of whole tree). It's support for status command is very clumsy. Let's check how we can emulate status with this tool using small script:

echo === extra files not tracked by Perforce ===
find . '!' -type d '!' -executable | p4 -x - have 2>&1 | grep 'not on client' | \
 sed '/\/moc_/d;/\.so/d;/\.o /d;/Makefile /d;/\.a /d'

echo === Current changelist ===
p4 changelist -o

As you can see we implemented "ignore" mechanism in above script (by sed filtering). "Extra files not tracked by Perforce" reflects "?"-status from CVS/SVN. "Current changelist" reflects "A,D,U"-statuses from CVS/SVN.

By using such script you can ensure your commit contains all files from your workspace, thus will be buildable on other developers' machines.

Perforce is a commercial version control system that seems to share some CVS and SVN features. Besides his shortcomings and weaknesses it has very useful feature: automatic merge tracking.

But sometimes you want to know what commits were already merged and what not. The simplest method to obtain such information is to try merge commits separately (so called cherry-picks) and then check merge results reported by "p4 resolve". It's very slow and boring method. Can it be automated?

Script below uses method described above but automatically. You can easily check merge status:

cd dest-branch
p4-merge-status //path/to/branch

Also you can narrow commits to be checked using weird P4 syntax (to save time):

p4-merge-status //path/to/branch @CL1,@CL2

Here's the script:

#!/bin/sh

BRANCH=$1
RANGE=$2

p4 changes $BRANCH/...$RANGE | sort -n | awk -v BRANCH=$BRANCH '
BEGIN {
        system("p4 revert ... >/dev/null 2>&1")
}
{
        cl=$2;
        pcl=cl-1;
        line=$0;

        cmd = "p4 integrate -Dt " BRANCH "/...@" pcl ",@" cl " ... 2>&1";
        while(cmd | getline > 0) {
                if(/already integrated/) {
                        print " + " line;
                        next;
                }
        }
        close(cmd)

        both = 0
        theirs = 0
        conflict = 0

        cmd = "p4 resolve -af 2>&1";
        while(cmd | getline > 0) {
                if (/^Diff/) {
                        both += ($9 + 0)
                        theirs += ($6 + 0)
                        conflict += ($12 + 0)
                }
        }
        close(cmd)

        if (both >= theirs) {
                print " + " line " both=" both ", theirs=" theirs ", conflict=" conflict;
                next;
        }
        # system("p4 revert ... >/dev/null 2>&1")
        print " - " line " both=" both ", theirs=" theirs ", conflict=" conflict;
}
END {
        system("p4 revert ... >/dev/null 2>&1")
}
'

And sample results:

$ p4-merge-status ../dev @165853,@166120
 + Change 165853 on 2011/04/19 by abc@def 'artf69528 fixed (assigned actio'
 - Change 165863 on 2011/04/20 by abc@def 'Artf69596: After this change Re' both=0, theirs=2, conflict=0
 - Change 165865 on 2011/04/20 by abc@def 'E2E>NMC>PVR> PVR node is displa' both=0, theirs=21, conflict=0
 + Change 165881 on 2011/04/20 by dariusz.cieslak@dc 'Merging changes' both=9, theirs=0, conflict=0
 - Change 165884 on 2011/04/20 by abc@def 'artf68460 : additional popup im' both=0, theirs=16, conflict=0

Legend:

• "+": change was already merged
• "-" change was not merged yet
• both: number of chunks that are the same on both source and destination branch
• theirs: number of chunks only on source branch
• conflict: number of chunks with conflicts

Above data allows you to estimate merge effort (and risk) before starting real merge.

The script is not fast because it's using P4 commands to merge changes. Perforce uses network heavily and that's why it's so slow for operations that other systems (SVN, GIT) do locally.

However: it's better than nothing, I hope you will find above script useful.

I've been assigned recently a task to prepare development process for two teams that are working on separate version control systems (GIT and Perforce in my case). One of important parts of this task is to create effective method of syncing codebases between both storages.

Of course we have git-p4 tool, but my requirements are a bit complicated for this tool:

• Only subset of whole GIT repository will be stored in P4
• GIT repository already exists with some history (the same for P4)

so I decided to write small script that will do at least P4 -> GIT sync.

My first attempt was:

• Sync GIT with main repository: git pull
• Sync dir to latest sync point: p4 sync -f subdir/...@$CL1
• Reload local changes from GIT: git reset --hard
• make files acceptable by P4: find subdir -type f -print0 | xargs --null chmod u-w
• Learn P4 about local GIT changes based on $CL1: p4 diff -se subdir/... | p4 -x - edit
• Inspect local changes from P4 point of view: p4 diff -du subdir/...
• Merge latest changes from P4 up to $CL2: p4 sync subdir/...@$CL2
• Resolve potential conflicts: p4 resolve -af
• make files acceptable by GIT: find subdir -type f -print0 | xargs --null chmod u+w
• Add missing files: git add .
• Build + tests
• Upgrade GIT repo: git commit -am "subdir merged up to CL $CL2"

But I noticed that merges performed by P4 aren't fast nor accurate. A developer from my team suggested that it may be useful to import P4 in smaller chunks in order to allow do bisection if there's a bug in imported codebase.

Then I created simple script:

#!/bin/sh

CL1=$1
CL2=$2

p4 changes ...@$CL1,@$CL2 | sort -n | awk -v CL1=$CL1 '
BEGIN {
    print "p4 sync mw/...@" CL1
    print "p4 sync ui/...@" CL1
}

{
    CL=$2

    print "p4 sync mw/...@" CL
    print "p4 sync ui/...@" CL
    print "git add -A ."
    print "p4 changes -l  ...@" CL ",@" CL " | git commit -a -F -"
}
'

It creates mirror of subset of P4 history and creates GIT commit per every CL. After that import merges can be done inside GIT (fast with good algorithms).