Database Migration and Backup

Launchpad blueprint: db-migration-and-backup

There are (at least) three cases where serializing the log database to some external format would be nice:

• Migration - if we change the DB schema or change storage solution entirely

• Backups - Backups? Who needs backups? And note that backups are only useful if they are stored in a format you can read in the future

• Debugging - Some times it is simply nice to get a human readable dump of the entire DB

• Testing - we could ship small log samples that we import prior to running some test cases, this way we don't have to populate the DB inside the unit testing code

Implementation

First and foremost we need to version the database in some way. There are two key factors:

• What DB-backend do we use; Storm, Querymancer, ? They are strictly not required to use the same DB layout
• What is the version number of the schema? Ie. not the version number of Zeitgeist itself, but that of the schema

Each DB backend should for each version implement some serializer class that outputs in some serialization format set in stone. I propose that we use whatever Tracker use for their backups, I believe it is Turtle or N3. Then we need a deserializer and an importer.

GUI

It would be nice with a super simple GUI tool. Really just a dialog with OK/Cancel buttons, a progress bar and some message stating that DB migration is about to be performed and that it might thrash the system while it runs.

Tasks

Here follows a work break down of the tasks needed to accomplish this:

1. Define a way to version the DB.
2. Define serialization format
3. Write framework code for choosing the right migration path given two DBs with given versions
4. Write serializer for querymancer_engine.py
5. Write deserializer for querymancer_engine.py
6. Write importer leveraging points 3-5
7. Write simple GUI