Contents
Whiteboard
Scripts and goodies for the conversion are at
git clone ssh://MYUSERNAME@git.gnome.org/git/gnome-git-migration.git
You can also see the cgit view of that repository.
Requirements
Data Integrity
One of the primary concerns that Olav (justifiably) has is making sure that the conversion proceeds accurately and we don't find problems after the fact when they are going to be hard to fix up.
Issues that he was worried about included:
Newline handling. With cvs => svn conversion we had some issues where images were corrupted because of different newline handling in cvs and svn. Do checkouts match byte-per-byte between the svn repository and the converted repository?
- History on branches. Apparently in some cases people have done things like 'svn rm trunk; svn mv branches/development trunk'. Do such branches migrate reasonably? Is history correct when migrated?
Blame accuracy. Does 'svn blame' output reasonably match the new '<foo> blame'? Of course, none of 'git blame' 'git blame -M' or 'git blame -C' exactly match 'svn blame', so there's some work to figure out the appropriate metric.
What procedures do we follow to verify the integrity of the converted repositories?
Strategy:
- After a repository is converted, compare the final heads.
TODO: do we really need to compare the "blame" output?
Commit Tracking
Olav considers it necessary to be able to track reliably and unforgeably who introduced code into the repository.
This would presumably be a look-aside log that recorded the ssh user ID for each push along with identifying information for the push. (The commit ID for each pushed head or whatever.)
Olav suggests 'hg pushlog' as a comparison.
Beyond that tools to retrieve the information and figure out who pushed (was the first to push) a particular commit would be nice, but as long as we *could* write them, probably not immediately necessary.
Strategy:
- Use a post-push hook to log the ssh account, the refs that got updated and to what.
Status:
- A post-receive script to write changed refs with pusher/time into $GIT_DIR/gnome_pushlog has been implemented. If we actually needed to reference the information, we'd need to write tools. But that can be done later.
Server Usage
Estimating the server resources would be needed to figure out where we would host a git.gnome.org.
Some factors to consider:
- IO utilization (seeks, bandwidth, extent to which things can be cached by giving the server a lot of memory, etc.)
- CPU/Memory utilization
- Disk space utilization ... how does it compare? How would it evolve over time? Do we need scripting of git-pack-refs or whatever?
Data can presumably be obtained from freedesktop.org about usage and server load and we can try to extrapolate to the GNOME load.
Pre-commit checks
How we would go about migrating the pre-commit hooks we have in svn currently. Are there any things that would be hard to do with git?
SVN dependencies
What depends upon svn.gnome.org, and what is needed to migrate them?
- buildbot
- automated gnome-web rebuilds
- commit mails
- damned-lies
- Anything else run out of commit hooks (review)x
Status:
Review done: /PerModuleHooks
- Basic commit mails in place - still needs improvement
- CIA (IRC bot) notification working
JHBuild
Is the current jhbuild git support good enough? How does the disk space usage compare for a complete checkout of the 2.26 moduleset between git and svn?
There are a few bug reports that would benefit the eye of a git expert:
Status:
- Current jhbuild support works OK, could certainly use ome more love
- Spot checks indicate that the modules clones with git are at most slightly bigger than svn checkouts, even with the full history in clone
Build bot
build.gnome.org integration with the VCS is done in two parts, and largely based on stuff that would be needed elsewhere anyway.
The first part is the notification of changes, this is currently done subscribing to svn-commits-list and parsing incoming mails (see code). Those mails should contain: revision hash, author, module, commit message; also the list of added/modified/removed files is nice as it allows to avoid rebuilds for translation changes. The email format itself doesn't need to be identical, a mail header X-Commit-Mail-Format: foobar would be ideal to identify them.
The second part of the integration is pointing back to the commit, currently it points to viewvc (example); so it just needs a template URL where $module and $hash could be replaced.
check if tests were made mixing buildbot and git for the gstreamer switch
How does it work for translators?
What are the special requirements for translators:
- Is git too hard for translators?
- The basic instructions are similar to SVN, and should be easy to follow.
- How does the translator workflow work with git?
- Is the checkout time and disk space usage reasonable?
- The clone time is longer with git compared to SVN checkouts. The disk space usage is not significantly bigger (in many cases it is smaller). Shallow clones do not save significant disk space advantages, though they save in clone time (about 30%). Example with gtk+ (git clone: 3.12min, git shallow clone: 2.00min, svn checkout: 40s, size around 170MB). It is good for translators to keep repositories of modules as they clone, then pull while updating translations.
Is transifex to a point where we could deploy it for GNOME?
You can find practical instructions for GNOME translators, a discussion and ideas at GitMigration/Translators.
Web infrastructure
Is cgit good enough to replace how people use viewvc? (What would deploying gitorious bring in the future in addition?)
Status:
- cgit is up and operational. No major deficiencies have been noted yet.
SVN hooks
hooks.tar.gz - The current set of SVN hooks for each module in svn.gnome.org, as well as a "bin" directory with scripts that get called from the hooks.
FedericoMenaQuintero put the hooks above in a Git repository here: http://gitorious.org/projects/gnome-svn-hooks/repos/mainline
Website update hooks: http://svn.gnome.org/viewvc/sysadmin-bin/trunk/gnomeweb/
Olav said about those hooks: These are running on various machines. If you sudo to gnomeweb on master, you can see the config for just that machine in ~/etc/website-triggers. Then also make sure the hooks for those in ~/hooks don't rely on SVN specific stuff (for all machines which use the website update scripts).
git clone git://gitorious.org/gnome-svn-hooks/mainline.git
Plan of action
There are some main areas that need work:
- Convert the repositories from Subversion to Git, or reuse the ones from git-mirror.gnome.org.
- Convert the commit hooks we have
- Ensure that jhbuild works with git.gnome.org
- Ensure that cgit.gnome.org works, or some other web-based front-end
- Provide an infrastructure for people who need git repositories to be created (for public or personal modules)
- Write documentation so people know how to transition to Git
- For the future, enable Gitorious or some other Git-based collaboration tool
Repository conversion
We can use git-svn (see John Carr's details on this).
Alternatively, we could reuse the repositories that are already converted in git-mirror.gnome.org. This would be convenient for people who already have work based on them.
TODO: Are the repos in git-mirror.gnome.org adequate?
TODO: Which modules use SVN externals? Where is this a good/bad idea, and what do we do in each case? For example, we could turn libbackground into a real library, or make it part of the gnome-desktop library.
Commit hooks
We have to port our current set of commit hooks to Git. Our hooks and a "bin" directory with scripts are here:
git clone git://gitorious.org/gnome-svn-hooks/mainline.git
TODO: Correlate the hooks with our list of requirements.
TODO: Create git-commits-list@gnome.org.
Done: Find which modules have hooks specific to them, instead of just generic hooks. See /PerModuleHooks
TODO: Port the svn commit hooks in the "bin" directory above to the Git machinery.
JHBuild
TODO: Figure out what needs to be done for jhbuild
Web front-end
FIXED: Set up cgit and point it to our repositories.
TODO: Is a setup like freedesktop.org's, which also groks ~username/git/reponame.git applicable to GNOME? Or should we wait for a real Gitorious installation to avoide giving people shell accounts?
cgit TODO
FIXED: Show diffs between revisions; right now all the repo history is there, but for some reasons diffs/patches aren't working.
FIXED: GNOME Theming/Branding
- Remove pagination in repo list (elsewhere?)
Fix the sort algorithm for branch and tag names to match GNOME conventions. For example, TOMBOY_0_12_0 > TOMBOY_0_8_0.
Maintenance infrastructure
TODO: We need a way for people to create public repositories ("new-gnome-module"), similar to the new-svn-repos script we have.
TODO: See below in the documentation section, we need a document to let people know how to push to new repositories or how to seed them from existing repositories.
Migration docs
TODO: Write document on suggested workflows for maintainers
We can use, base or link to An overview of recommended workflows with git and How to maintain git
Instructions about how to use Git can be found in Git.
TODO: Write document on suggested workflow for translators (do sparse checkouts make sense? do people really check out a single language.po file?) Draft document available at GitMigration/Translators.
TODO: Write document on how to request that a new repository be created, and how to populate it from an existing repository. "git push --all --tags" may be all people need, plus something to set up a remote.
TODO: Write general document on how the svn->git migration was done, for other projects that may find it useful.
suggestion for the main page of git.gnome.org: look at git website - git quick start, with two common operations directly on the front page: clone+patch+commit and import a new project
TODO: Update the Wiki pages. There are a lot of pages that refer to Subversion or svn in the wiki pages. Most conversions are trivial, in the sense that they only need simple susbtitutions:
SVN/svn/subversion -> git
trunk -> master
svn url -> git url
svn cmds -> git cmds (no change in workflow)
The idea is to convert them more or less at the same time as the GitMigration itself. At the very least the first six in the the following list of pages should be converted:
MaintainersCorner: trivial
MaintainersCorner/Releasing: +/- trivial but local commit + tag + push + failure?
ReleasePlanning/MakingARelease: trivial + proofread
AccountPolicy: trivial + proofread
NewSVNRepos: -> Git/NewRepository: extra work
Infrastructure: trivial
Sysadmin/Servers/Container: Update by the sysadmins?
GitForGnomeDevelopers: add a comment
TwoPoint*/*: worth? web redirect for svn url?
GnomeGames/Contributing: trivial, link to Git?
GnomeWeb/Library/ServerDetails: Not trivial, add a comment
GnomeDeveloperKit/BuildingPackages: themselves
GTK+/TaskAPIDocMigration: trivial
Sabayon: trivial, but should link to the Git instructions
ModuleMaintenanceWorkspaces/ProjectWebsiteStatus: Needs extra work
Straw/HowtoBuildFromSource: Trivial but leave the external Git repository
- ...
Future
TODO: Set up Gitorious.
TODO: Give UI love to Gitorious! Talk to the KDE people; they are interested in this as well.
Milestones
1. Working scratch-repositories and notification
- We have git.gnome.org with git-daemon for public cloning (git://git.gnome.org/modulename)
We have git.gnome.org with ssh for pushing (git push ssh://user@git.gnome.org/git/modulename blah:blah)
- We have one or two scratch repositories there (a few test commits are fine)
- One can push to the scratch repositories, and the following happens:
git-commits-list@gnome.org is set up and mailed to
- the repositories and their state appear in the web interface of cgit.gnome.org
2. Guinea pigs with converted repositories
- We convert a few existing repositories from maintainers who agree to testing the new git.gnome.org setup.
HubertFiguiere volunteer the "niepce" module for the tests.
- We let them report problems for some period of time, and fix them.
Who does what
Done: compare the existing module hooks to see which ones are generic and which ones are special: /PerModuleHooks
- Figuring out svn-all-fast-export and testing it - handles branches and tags better than git svn, doesn't create empty commits for the svn cp operation that was used to create the branch or tag
- Writing a small script to checkout and compare all tags and branch heads of a converted git repo to the corresponding svn checkout.
- Setting up cgit when we get the play-pen up and running
- Done: A playpen git.gnome.org (rsync'ed svn repos, created gitadmin group, created el5 git-1.6.0.6 rpm, cgit rpm)
- Done: CSS-themed cgit to look like a gnome website
- Convert commit hook scripts
- cgit theming
Volunteers available to help
ShaunMcCance - not an admin type, but happy to develop tools, conversion scripts, automated verifiers, etc.
LuisMedinas - I can help test/migrate some modules, already updated prepare-ChangeLog script to work on git >= 1.6.x on svn/releng trunk.
HubertFiguiere - I can help test. So above for the Guinea Pig repository.
RubenVermeersch We need gitorious, probably even more than cgit :-). Willing to help figure that out if needed.
MattEnright Willing to help test, in addition to cgit/gitorious-type works.
SantiBejar I would like to help in migration docs and testing.
SimosXenitellis I would like to help in the documentation, and assist translators in a smooth transition.
Schedule
Date for migration is April 16, 2009 (see http://mail.gnome.org/archives/gnome-infrastructure/2009-March/msg00064.html announcement).
See also Schedule.
Tools
svn-all-fast-export
By Thiago Macieira and Chris Lee:
http://repo.or.cz/w/svn-all-fast-export.git
Config file: see the end of http://mail.kde.org/pipermail/kde-scm-interest/2007-December/000150.html
Misc
Gitorious is like Github, but free. It hosts itself, so we could adapt it easily as a web frontend for git.gnome.org with collaboration tools.
References
John Carr's page about how git-mirror.gnome.org is generated
Git for Gnome developers - we'll need to update this once the migration is done