Subversion migration notes

STATUS: Complete. As of the early hours of Jan 1st 2007, all the GNOME CVS modules were migrated to Subversion repositories.

Thanks to everyone for their support :)

My developer CVS checkout stopped working. How do I fix it?

Shortly after the migration cut-off time, CVS-via-SSH for normal developers will no longer be available and attempts to update or work with an existing checkout will likely result in a connection error. If you replace the contents of your CVS/Root files in all of your CVS working directories to point to the AnonymousCVS mirror, you will still be able to use 'cvs diff' etc to draw patches from your old working directories. Something like the following should do it (use with care): 

$ for i in `find . -name CVS -type d`; do echo ":pserver:anoncvs@anoncvs.gnome.org:/cvs/gnome" > $i/Root; done

The changecvsroot.py script (available from a jhbuild checkout) can be used for this as well.

'cvs2svn' doesn't migrate CVSROOT/modules to 'svn:external's, and it doesn't seem to be something that can be automated, so it'll be up to the maintainers to add these themselves after the migration, or to decide on alternative ways of doing things, such as packaging the dependencies (e.g. libbackground) separately or whatever. I expect most maintainers will just add 'svn:externals' attributes, at least to begin with.

For example, the nautilus maintainer (or one of its developers) will need to add the following: 

$ cd nautilus-trunk
$ svn propset svn:externals "libbackground http://svn-test.gnome.org/svn/gnome-control-center/trunk/libbackground" .
$ svn commit -m "Add externals"

This will need to be repeated for any branches still being maintained.

For reference, the CVSROOT/modules file can be seen here. A large number of the entries in the file refer to out-of-date modules and aliases that are no longer used. Additionally, I found that several of the modules that were including the old 'macros' directory no longer actually need it to build. As well as forcing maintainers/developers to understand and review their use of 'svn:externals', this will also be a good opportunity for some impromptu housekeeping.

Bonsai data

Due to a bug in svndbadmin, there will be a few small holes in the bonsai database. I have filed two bug reports about this here and here and created a workaround patchs which allows this tool to skip over revisions it can't handle and continue instead of balking, so we get a fuller history with less chunks missing. Once those bugs are properly resolves, the holes can be filled with a rebuild.

Misc suggestion

How far have we got?

We have a new, dedicated Subversion server, graciously provided and hosted by Canonical Ltd, in London, UK. Thanks again to the guys at Canonical for this.

The results of the live migration are here:

http://svn-test.gnome.org/migration/status.xml

Completed tasks:

In progress:

  • Confirming details of backup strategy offered by Canonical. (query #26293 logged with Canonical RT, awaiting their advice)
  • Make global and per-repository post-commit scripts web-readable.

How does the migration script work?

See for yourself here.

How is the migration order determined?

See for yourself here.

How do I checkout one of the test subversion repositories?

See the notes on the Subversion page.

Where do I report problems?

Either directly to me (ross@golder.org) or to the GNOME infrastructure list (gnome-infrastructure@gnome.org).

We would welcome feedback from people checking out and building copies of modules using subversion to ensure integrity. I've checked out and used a few modules and they seem to be fine. However, we are aware that some modules may be missing folders that were previously provided by the CVS 'modules' file feature. The same can be done with 'svn:externals', but it currently seems that this will need to be done manually by the module maintainers after the migration. For more info, see this post.

What's your plan for going live?

  • Switch CVS to read-only (at 23:59UTC) on the agreed/announced cut-off date. Set perms on container:/cvs/gnome to read-only (recursively) to prevent writes to CVS during migration. Start rsync over to new Subversion server.
  • Clear out the /svn hierarchy (except for README and bin!), empty /svn/tmp and the 'bonsai' tables. Clear out the 'logs', 'tests' and 'tmp' folders in /usr/local/svn-migration.
  • Wait for rsync to finish and remove the symlinks (gnomemeeting, xpdf etc) in /cvs/gnome.
  • Kick off the migration script.
  • Wait for the first module (test) to finish migrating. Check read/write access to ensure it all works normally.
  • Wait for the the web modules to finish migrating (they should be in the priority list). Activate and test new subversion-based website commit-to-update scripts.
  • Cut-off commit access to /svn/beast on container and rsync it into place alongside the other migrated modules. Run 'svndbadmin' on it.
  • Wait for all hell to break loose.

  • Somehow switch Bugzilla checkout to use SVN -- OlavVitters

  • Change sysadmin checkouts to use SVN

and whatever else depends on CVS.

I have set up socket.gnome.org as a slave LDAP server (self-issued a cert for 'progress.gnome.org' and configured LDAP to slave from button). Label, the master, replicates changes to it via the SSL-encrypted LDAP connection (port 636). This ensures that subversion doesn't become unavailable or laggy should there be connectivity problems between Canonical and RedHat, or downtime on the LDAP master.

The /cvs/gnome repository is manually rsynced from container. It is not (yet?) automatically kept in sync in any way.

  • Write some really cool post-commit hooks scripts (later). Should be a doddle with subversion, python and things like 'svnlook'.
    • A global post-commit that looks up the repository being commit to in a lookup table, and if there is an e-mail address for it (e.g. maintainer, or project commits list), sends a commit log. This should somehow contain the option to include/suppress the full patch (which could be sent as an attachment, for easier 'save to disk').
    • A modified post-commit that looks up the language code of a translation file being commit, and sends a summary or full patch to a nominated e-mail address (if found). This would be useful for language teams and their co-ordinators.

I do not know if anyone has seen this one before but here is a (post commit hook friendly) python script to generate an RSS feed of the subversion version history http://base-art.net/Articles/18/

Old status report (16th December 2006)

The last remaining issues have now been ironed out, so the subversion checkouts available to test will now be indicative of the results of the live migration.

Developers should now be able to check out subversion modules and commit modifications to them. Only test commits should be made to subversion. Anything you commit to subversion before the live migration *will* be lost at live migration time and any time before then that we deem necessary to do more tests of the migration scripts. Commits will trigger the pre-commit and post-commit scripts, but notices to the 'svn-commits' mailing list and CIA will be marked as 'testcommit'.

Please note that modules that previously relied on 'CVSROOT/modules' entries to include libraries into a checkout will have to add their own 'svn:external' properties, and/or update their Makefiles accordingly. I will try to write up a reference example of how you might do this.

We now have 606 modules to migrate (down from 800+ thanks to the sterling efforts of Vincent Untz). If you spot any other modules that you know are now redundant, please mention them. The full migration now runs in about 36 hours.

For those concerned by the modules marked in red on the test migration status pages, these are 'false negatives' caused by inconsistent linefeeds, certain RCS-expanded tags and CVSROOT/modules included directories. The migration script compares a clean CVS HEAD checkout to a subversion trunk checkout using a recursive 'diff'. The migration script makes some crude attempts to neutralise the causes of the these three problems before running the diff. Those that have failed have been manually checked several times to confirm that they do not show real differences.

The only module that showed a difference that caused me some concern was that the 'livecd-project' module contained a '.css' file which contained a load of binary data and couldn't be run with the same 'cvs2svn' command-line options as all the other modules. And relevant 'svn:eol-style' attributes and/or 'svn:mime-type' attributes will need to be applied manually after the migration for this module.

PLEASE help ensure this migration is bulletproof. Check out 'trunk' copies of your favourite modules and check that it builds cleanly. I have spent a lot of time and effort ensuring that this migration is successful. I do not want anyone to find a problem during or after the migration that we could have found and fixed before it. Please direct any questions or problem reports to 'gnome-infrastructure@gnome.org' or to me directly (RossGolder). If any serious issues arise, they will be posted to 'gnome-hackers@gnome.org' for further consideration and the migration will be postponed again indefinitely.

Also, if possible, please check a couple of recent/useful branches out. We know that it may be possible that there will be some broken historical branches/tags due to the system clock being reset at various points during the CVS server's lifetime, but a lot of effort has gone into minimising the problems associated with that and we expect *most* branches/tags to be sensible. For good measure, we will make sure the CVS archives remain available in a read-only state indefinitely.

-- RossGolder

Previous (failed) migration notes

MIGRATION CANCELLED :(

CVS cut off was: Friday 14th July 2006 at 23:59UTC

CVS re-enabled: Sunday 16th July 2006 at 03:50UTC

UPDATE: A second problem was found caused by the work-around to the first problem. Due to the absence of 'svn:eol-style' tags for text files, checkouts of text files on non-unix platforms were not platform-independent wrt EOL style, and were not the same as they were when people tested the repositories before the workaround. It was decided to cancel the migration, do some more thorough testing and re-schedule for another weekend. A few modules will be left in Subversion in the meantime.

OLD UPDATE: A problem was found during the initial run with certain SVN properties not being set by the migration script, resulting in corrupt binary files (e.g. PNGs, WAVs) etc on checkout. The script has been fixed and re-tested, and the initial migration stopped and restarted. It was restarted at around 09:00UTC.

You can follow the progress of the migration by watching the files here. The main script output is here. See below for links to the 3 priority lists.

Dealing with the fallout of the cancelled migration

Unfortunately, several modules took commits once converted, and now need to be maintained as subversion modules until the rest are converted (hopefully very soon!). Most were able to re-apply their changes to CVS, but for various reasons the following were not:

  • beast (timj)

The plan here is to write, test and run a 'svndump-postprocess-undump' script to add the necessary 'svn:eol-style' properties to the text files retrospectively.

Unfortunately, as normal GNOME devs only have SSH access to use *either* CVS or Subversion on the server, and it's currently set to CVS, special temporary access needs to be given to the module maintainers of the above modules so that they can continue to work on their module until a successful migration of the remaining CVS modules is complete. So, before anyone else asks, that means we will *not* be honouring requests to add new subversion modules for people who have been waiting for subversion to go live.

Yes, very confusing and not an ideal situation. We hope to rectify it a.s.a.p.

The following modules also suffered fallout:

  • glib - gnomebug:347908

Background/History

The board asked the GNOME sysadmin team to investigate setting up a Subversion repository of the GNOME source code.

http://mail.gnome.org/archives/gnome-infrastructure/2005-June/msg00009.html

Migration was first scheduled for 17th March 2006:

http://mail.gnome.org/archives/gnome-announce-list/2006-February/msg00027.html

However, it was post-poned due to problems found in ordering of the commit history in the resulting repositories. Bugs in clock-skew detection and correction in the cvs2svn script and and branch/tag naming conflicts in the repository were the problem.

http://mail.gnome.org/archives/gnome-announce-list/2006-March/msg00025.html

The problems were fixed and the migration was then re-scheduled for Friday 16th June 2006 at 23:59UTC.

http://mail.gnome.org/archives/devel-announce-list/2006-May/msg00004.html

However, this fell too soon before GUADEC, so we decided to delay it another month to allow GUADEC to clear. The GNOME CVS to Subversion migration is currently scheduled for Friday 14th July 2006 at 23:59UTC.

http://mail.gnome.org/archives/gnome-hackers/2006-June/msg00009.html

This migration turned out to be a miserable failure, but we learned of a few more problems (at the last minute) that didn't show up in testing, and thought of a few better ways to do things, so we can be more sure of success on the next attempt.

http://mail.gnome.org/archives/gnome-hackers/2006-July/msg00011.html

New 'go live' proposal date: 29th December 2006. Announced:

Attic/SubversionMigration (last edited 2013-11-23 00:33:13 by WilliamJonMcCann)