Releasing Software is not just packing latest version

The aims for releasing procedure designed:

• allow for testing window before release date
• have the possibility for examine released version to test for reported bugs
• possibility to manage existing releases (hot-fixing critical bugs)

Prepare Release Candidate Branch

We want to stabilize and test some snapshot of current development branch. That’s why I’m forking few days before release RC branch from “master” and switch (it will be used for hot-fixing):

git branch RELEASE_0.2.1_branch
git push origin RELEASE_0.2.1_branch
git co RELEASE_0.2.1_branch

I’m marking RC state (RC=Release Candidate) to be able to see changes done for released version:

git tag RELEASE_0.2.1_RC
git push --tags

Anyone familiar with advanced CVS usage will see similarities to tagging for CVS merge purposes. GIT tracks merges automatically, however marking branch starting point is a good idea.

As you can see I created simple naming convention schema to manage releases.

Prepare Release

Time after creating RC branch and before releasing from this branch is the time for testers to do their job and sweep out as many defects as possible. Fixes are added directly on release branch (we will port them to master later).

When tests are done we are preparing release and tag current version by RELEASE_0.2.1

git commit -a -m "version number changed (#XYZ)"
git tag RELEASE_0.2.1
git push --tags

By this tag we will be able to inspect exact version that was sent to our clients.

Prepare hotfix

Sometimes out safety net composed of automated test suite and testing team fails and we have to fix errors reported from production. That is the purpose for release candidate branch. First, switch to correct branch that supports hot-fixed release:

git branch --track RELEASE_0.2.1_branch origin/RELEASE_0.2.1_branch
==OR==: git co RELEASE_0.2.1_branch; git pull origin

Tag current version that goes in this hotfix by RELEASE_0.2.1_hotfix_YYYYMMDD:

git commit -a -m "version number changed (#XYZ)"
git tag RELEASE_0.2.1_hotfix_YYYYMMDD
git push --tags

As you might noticed naming convention is based on branch name. Thanks to this convention we can answer the following questions:

• what hot-fixes were prepared for release X?
• what is the latest hotfix for release X?
• what was delivered in latest hotifix of release X?
• etc.

Porting back changes from branch to master

Sometimes changes made on branch will be useful for next releases. You can easily merge them back to master:

$ git co master
$ git merge RELEASE_0.2.1
$ git push

GIT tracks what have been merged already so you can merge/cherry-pick in both directions.

Typical Usage

What changes were included in latest hotfix compared to previous one:

 $ git diff RELEASE_0.2.1_hotfix_date1 RELEASE_0.2.1_hotfix_date2

What changes were added in new release:

$ git log  RELEASE_0.2..RELEASE_0.2.1
$ git diff RELEASE_0.2..RELEASE_0.2.1

svn merge -c -19203 https://REPO_URL

In example above you do reverse merge of 19203 revision (note “-” sign before revision number). After that merge:

• Inspect if workspace compiles without errors
• If it’s OK: commit local changeset
• Notify 19203 committer about the change

In order to fix broken commit original author should do the opposite:

svn merge -c 19203 https://REPO_URL

(note there’s no “-” before revision number). Then:

• Correct changeset and test if workspace is not broken
• If it’s OK: commit local changeset (it will be nice to show original revision number in a comment)

The property I’m going to present today is “mime-type“. It describes file content in similar way to HTTP header “Content-type” telling svn client how to handle the file. Typical values are: “text/plain”, “application/octet-stream”. Especially first part of mime-type is important:

• text/*: line-by-line merges are used, diffs are generated
• any other prefix: no text merges prepared

If you do not set this properly right you may end up with messed binary file (end-of-line conversions) or non-mergable changes in text file (that is marked as binary by mistake).

Of course during adding a file to workspace one can forget to set those properties correctly. Here auto-props comes with help. Auto-props are applied when “svn add” command (from command line or from GUI) is issued. Configuration is placed inside “~/.subversion/config” file. Here’s my config fragment from one of projects.

[auto-props]
*.csv = svn:mime-type=text/plain
*.java = svn:mime-type=text/plain
*.sql = svn:mime-type=text/plain
*.sql = svn:keywords=Author Date Id Revision URL
*.jar = svn:mime-type=application/octet-stream

Besides mime-type svn:keywords is being set in the example. It controls which keywords are expanded in source files.

Questions

Important knowledge that may be required by developers doing updates may be summarized in few sentences:

• Who changed recently that line of code?
• When this method have been changed?
• Why algorithm works that way?

There’s simple method of automatically saving and retrieving this kind of information: Subversion (or any other version control system). How?

Answers

There’s nice feature of version control system that is not the most frequent used but is very useful: annotation/blame. This special view shows you for a file:

• Who changed this line?
• When this line was changed?
• Revision number of commit => Log entry => Bug tracker task ID => rationale (Why)

After locating such information you may have better understanding of source code.

How to check annotation using different tools:

• svn annotate filename
• git annotate filename
• bzr annotate filename
• Eclipse: Team / Show Annotation

The problem

Looks simple, but there’s a “quirk” here. If you are doing massive code changes (to enforce n+1th coding standard) you are overwriting original source code authors and information. Thus annotation (and log) becomes useless.

That’s why I’m asking you:

Do not reformat whole files on commit, PLEASE!

• any commit done to trunk is protected by set of automated tests that should run 100% clean (more critical project will use framework for a continuous build process like Cruise Control or PQM).
• “risky” tasks are done using peer code review process based on version control software

What to review?

We believe the most efficient method of reviewing code is to look at changesets (unified diffs). Then reviewer can see exact changes done by another developer to meet task criteria and codebase to review is minimized to really important parts of code (changes with some lines of context). That’s why we believe forming proper changesets is very important requirement.

Commit methods

There are mainly three methods of inserting new commits into trunk (specified in our implementation by special “target” field in bug tracker):

• Direct commit (target=master) without code review: used mainly for simplest functionality without much risk involved. Minimal administration burden, minimal merge conflict.
• Patch Based Review (target=patch) using PMS functionality for one-way code review: in this scenario only one developer is supposed to modify patch, another one is only reviewing. At least two developers are involved, merge conflict minimized by basing on trunk (patch is always re-based on trunk during development).
• Shared branch (target=branch) using VCS: Development is branched on separate branch and reviewved on this branch. Many developers can commit on this branch/review before merge. There is higher merge conflict risk for long development on branch.

A convention used by us: patch / branch name should contain noun from task summary and task id separated by a dash:

noun-task_id

for example: invoice-2351. This way we can easily find topic branches and precisely point to taks specification (by a task id).

Peer Code Review

Where’s peer code review here? Three modes of commit, three review configurations follows:

• Direct commit: changeset is reviewed after commit on trunk (as I said lower risk encourages us to be flexible)
• Patch-based (read-only) changeset view: patch is sent to another team member for review by email or shared patches repository
• Classical “topic branch”: read/write review: team member switches to topic branch and checks latest commits, he may also place comments in source code

Old, Good TODOs

What is a result of code review? The most efficient way to track problems found is to insert comments inside source code (in comments, near problematic code location). We prepared a convention of well-known TODO tags in source code: TODO:nick description. “nick” is abbreviation of comment recipient, description should be short and valuable. Here’s the example:

/*
TODO:DCI VAT report summary should show taxes grouped by rates,
not the case now
*/

Recipient then scans all source code for such TODOs and fix problems found (thus removes TODO in the same commit). This way:

• we can connect problem report with problem fix in one changeset
• we can track who entered comment and who fixed it (from SVN history)

Summary

Code inspections were found to be very effective method of raising quality of software (the IBM story). I believe informal implementation called peer review plays very well with other agile tools we are using for developing better software.

The possible reason for freezing commits:

• creating releases from trunk
• creating releases from branch
• commit freeze during merge from different branch
• move to different version control system
• massive changes expected that may cause many conflicts

Creating releases

If you want to make minor changes related to release and block any other (probably more risky) changes to be accidentially introduced you needn’t freeze commits. The more efficient solution here is to fork a branch. On separate branch you can do any justification you need to build the binaries for release.

Merging

For time-consuming merges (especially when many conflicts are present) it’s tempting to prevent commits on target branch to minimize problems related to local development during merged changes commit. I think merging person should perform frequent updates insted from current branch and match merged changes to current trunk state.

Switch version control software / repositories

Switching between version control system is a big change in development team. One has to learn new toolset to operate efficiently with new version control system. Postponing commits on old repository is not required. Those changes could be reapplied later by creating patch from missing changesets and applying them on new repository. Path format is a standard that allow to move changesets beteen different repositories.

Summary

In my opinion temporary blocking commits (so called “commit freeze”) is not a good idea. Agile methodology (the one we use at Aplikacja.info) requires frequent information sharing. There are alternatives that have lower impact on development and not get in the way for normal code flow.

svn propedit -r <revision-number-here> --revprop svn:log "<new log message>" <url>

That’s all. Pretty simple.

I’m describing here Subversion usage best practices that come from my experience with Subversion (or other version control systems) collected on many projects. Some suggestions are related strictly to Subversion, some are general.

Commit complete, logical changesets

Always commit complete patch. “Complete” means the commit:

• shouldn’t have compilation errors
• shouldn’t lower successful test cases counter
• should have precise purpose (one fixed bug, implemented feature, as so on…)

Why complete changesets are so important? You can easily then:

• reapply only selected bugfixes on another branch (sometimes it’s called cherry-picking)
• undo change that made system unstable by simple reverted merge

Sometimes you have many different changes (example: two bugfixes, one feature added) in your workspace and it seems the easiest way is to commit them in one big commit. DON’T. You can use “svn diff”, “patch” commands to split your local changeset into few parts and commit them separately. How? It’s simple (but not obvious at first glance):

1. svn diff > ../changeset.diff
2. $EDITOR ../changeset.diff
3. Leave only changes to be removed from working copy, delete remaining changes
4. patch -p0 -R <  ../changeset.diff
5. Now you have yor workspace without changes to be removed, retest, commit
6. patch -p0 <  ../changeset.diff
7. Now you have only changed that previously were skipped, retest, commit

Special tool have been implemented that can automate above flow: PMS (Patch Management System).

Do not commit workspace partially

If you commit only subset of files from your working copy you risk breaking the build. The proper way for splitting working copy into separate commits is to reverse patch your changeset (see previous practice examples).

Similar problem appears when you forget to add to versioned files new file (and it appears as “?” on “svn status” listing). Use “svn status” frequently and properly set svn:ignored where necessary.

Continuous integration simplified: commit often, update very often

If you postpone your changes more that two days it’s very likely someone will use obsolete API that you’re refactoring and someone will get compilation errors. Try to make atomic but small commit to minimize such risk.

When you develop a feature it’s very likely you will use obsolete API that some other developer is refactoring now. Update very often to see fresh system state.

Have you noticed above paragraphs are very similar? You have to make fast information flow. Making long-living branches will postpone new code from being integrated. Frequent merges and commits are implementation of continuous integration. Every integration problem (as API change mentioned in this section) will be visible as soon as possible (when it is stable enough to be committed).

Someone can point out that frequent commits can destroy trunk stability – see next section for answer.

Shared branches must be stable, Really

When you are working on isolated topic branch you don’t need to ensure it’s stable enough to be committed. It’s not needed to compiler, either. But if anyone will have access with you to this branch/trunk you must ensure it will remain stable. Stable branch encourage frequent updates and it’s base to Continuous integration.

How can we define “stable” criteria for local changeset to be committed:

• Code compiles with no errors
• No additional failing tests (no regression!)
• Ensure no accidental whole-file reformatting by viewing diffs (svn di)
• Ensure no forgotten unversioned files (svn status = ?) in your repository (it may break compilation for someone else)

Use meaningful commit comments

A typical error (yes, error) made by novice programmers is empty or “asdf”-style commit comments. The information what was committed is hidden in commit changeset only. It will make code inspection harder. I suggest to enter at least in commit comment:

• Issue tracker ID if preset (to link commit to issue fixed)
• short one-line description (can be copied from issue tracker)

Then anyone can look at fresh commits:

svn log -l 10

and see what’s going on now in the project.

Care about merge and annotate – DO NOT reformat whole file

Sometimes when I want to trace who are responsible for particular change in file I’m discovering by annotate that whole file have been changed recently by one person. Why? This programmer reformatted whole file and committed big changeset. Now file looks better but at what price? Original committer information is lost.

Similar problem occurs when you merge such changeset (from topic branch to trunk for instance). There are many conflicts because merge changeset overlap with changesets committed previously on trunk.

Subversion like many other popular version control systems loses commit information on merge (merged person is visible instead of original authors of merged changesets). If you expect to preserve such information use GIT. (Thanks Michal to pointing this out: Subversion >= 1.5 tracks merges).

Setup auto properties correctly

In mixed systems environment there are many formats of storing files. EOL (end of line) standards differs between operating systems. Subversion allows to convert-on-fly EOL style. In order to make use of such feature you have to set “svn:mime-type: text/plain“.

Setting such properties on every added files can be boring, so auto properties have been introduced in Subversion. You can map (locally) properties to extensions in your ~/.subversion/config:

[auto-props]
*.csv = svn:mime-type=text/plain
*.java = svn:mime-type=text/plain
*.sql = svn:mime-type=text/plain

Setup svn:ignore in repository

Some files are required to live in project directory but shouldn’t be committed into SVN (*.class, *.ear, …). General “ignore” mechanism marks such files (or file masks) to be invisible for SVN (not reported as unknown files “?”). Typically CVS used .cvsignore file, Bazaar reads .bzrignore, SVN reads svn:ignore property AND local user preferences stored in ~/.subversion/config.

Do not depend on local developer ignore settings in ~/.subversion/config. Attach svn:ignore tag to proper directories to be present on every working copy. It will make to add new filters easier (by commiting new properties once to Subversion).

svn merge -r17007:17249 http://hostname/svn/path

I got the error: svn: Malformed URL for repository. The fix is to add user name in SVN URL:

svn merge -r17007:17249 http://dcieslak@hostname/svn/path

Is this a Subversion bug? Probably the problem is that my username placed in SVN config file has form: domainname\username. Maybe SVN merge cannot handle that format?