Camel.FolderSummaryDisk

This is the 'disk' part of the 'disksummary' branch. i.e. the main reason this branch exists, although it was used also as an excuse to try out other ideas and clean up other problems.

It uses libdb to store each record, with the key being the UID.

Base class

The base class implements all of the functions required to store the summary in database records in a persistent way. Including maintaining the view indices.

This is a concrete class that can be used as-is, or subclassed if additional data needs to be stored. In this way it is equivalent to the Evolution/Camel.FolderSummary class, for storing records on disk.

 struct _CamelFolderSummaryDiskClass {
        CamelFolderSummaryClass parent_class;
 
        void (*encode)(CamelFolderSummaryDisk *, struct _CamelMessageInfoDisk *mi, struct _CamelRecordEncoder *);
        int (*decode)(CamelFolderSummaryDisk *, struct _CamelMessageInfoDisk *mi, struct _CamelRecordDecoder *);
 
        void (*sync)(CamelFolderSummaryDisk *, GPtrArray *changes, CamelException *ex);
 };

It adds record encoding and decoding methods which translate a binary blob of data into a CamelMessageInfo structure.

It also adds a sync method, to ensure changes are flushed. Internally it catches changes to each messageinfo and keeps them in a list. After a timeout, it processes this list - updating secondary indices (i.e. views) for any changes, and then writing everything out. Backends can hook onto this virtual sync method to also store any changes to physical storage. i.e. the IMAPX backend uses this opportunity to write flag changes to the server.

Methods

Note that new and construct are not passed the Evolution/CamelDS.ViewSummary that Evolution/CamelDS.FolderSummary requires. They get it from the folder's store, which has it. Probably CamelDS.FolderSummary should do the same.

 CamelFolderSummaryDisk *camel_folder_summary_disk_construct(CamelFolderSummaryDisk *cds, struct _CamelFolder *folder);
 CamelFolderSummaryDisk *camel_folder_summary_disk_new(struct _CamelFolder *folder);

This will trigger a sync, it is an immediate sync; it is normally called internally, or from a folder sync call.

 void camel_folder_summary_disk_sync(CamelFolderSummaryDisk *cds, CamelException *ex);

Because the indices are B-Tree based, there are more features available for scanning than the basic iterator interface provides. This function lets you get iterators which jump directly to records or scan the records from the end. See the source for more discussion - only for implementor use.

 const CamelMessageInfo *camel_message_iterator_disk_get(void *mitin, guint32 flags0, guint32 flags1, CamelException *ex);

Caching

Individual decoded records are cached in memory, those currently in use are stored in a hash table until they are unreffed finally. Any outstanding changes delay the final unref, so at unref time the records are already synchronised to disk.

The tables themselves are also cached, or rather, they are limited to how many may be open at a given time. This is mostly to get around a bug in libdb that assigns (multiple) new file descriptors to databases open within the same file. An upgrade of libdb may fix this problem and remove the need for this cache.

Tables

There are multiple tables stored in each database file. There will be one for each folder (representing the 'root view'), and one for each view of each folder.

A per-store Evolution/CamelDS.ViewSummary#CamelDS.ViewSummaryDisk object manages the database environment for each store. It also has a database-global lock for access to all database functions - this must be used to enforce single-threaded access to each database (environment?), to avoid deadlocks. It also maintains its own table for all views.

Root Table

The root table will be given a name which matches the folder name.

 folder/name

The records in this table will consist of a key being the UID of the message, which will be a string representation of an integer value, and the data record which contains the CamelMessageInfo in an encoded form. The Evolution/CamelDS.FolderSummary.uid_cmp() function is used as the key comparator for the B-Tree index.

Records are stored as binary encodings of the various data parameters. To try to reduce some of the versioning problems present in the Evolution/Camel.FolderSummary records, the record is split into sections. Each section then has records only ever added to the end of it. This should allow transparent forward compatability of the record format, with only conditional code required for backward compatability. Note that each subclass will implement its own section if it wants to add data, not extend an existing section, so versioning is also detached from the class heirarchy.

For efficiency - since records will be read and written one at a time, potentially many times, and because size is less of an issue (since libdb adds significant overhead anyway), records are stored in a more native sized format, i.e. no compression takes place. Evolution/CamelDS.Misc#Camel.Record utilities are used to encode and decode each record.

Each record is split into 2 basic sections too, one for the static data, and one for the variable. It may make sense to split this into different tables, to reduce the amount of data which must be re-written when status changes.

View tables

Each view is given a name that begins with the folder name, followed by the view-id with a separating 0x01 octet. The 0x01 value was chosen because it cannot occur in normal UTF-8 names, and using a simple strcmp based sort will properly order the list of folders in the view table.

 folder/name\01view-id

The records in this table ONLY consist of key fields, which are simply the UIDs of the messages in this view. The data field is not used, and will be zero-length.

Note that this means the secondary index cannot be automatically created using libdb's functionality. This was tried initially but it leads to a lot of extra overhead - records need to be decoded multiple times.

Currently, the view table is updated whenever the summary is synchronised. The ChangeInfo from the root view is processed, one message at a time, and any messages matching (or newly unmatching) a given view are added (or removed) from that view. The ChangeInfo is then propagated to the per-view ChangeInfo data stored in the CamelFolderViewDisk.view structure. And finally to the folder, after processing is complete.

Table Notes

It probably makes sense to change the folder name part of the tables to be unique id's, which do not change when the folder is renamed. Otherwise the table and all sub-views need to be renamed - a process which isn't very simple with libdb, and just a hassle anyway. Instead perhaps the view table could have a mapping from the folder id to the folder name, and just change it there.

Camel.FolderViewDisk

The CamelFolderViewDisk object adds to the per-view data a database handle where the records for this view are stored.

 struct _CamelFolderViewDisk {
        CamelFolderView view;
        DB *db;
        int usecount:31;
        int unused:1;
        EDListNode ln;
 };
 
 #define CFSD_VIEW_FROM_UNUSED(x) ((CamelFolderViewDisk *)(((char *)x)-G_STRUCT_OFFSET(CamelFolderViewDisk, ln)))

The strange-looking macro lets the ln list-node structure be offset from the beginning of the data record, when stored in the unused list.

The rest of the fields are used to manage the database handle, to ensure too many handles are not open at one time, leading to resource starvation.

Performance

Because of the incremental nature of the algorithms employable because we are using sorted folder summaries, the efficiency is pretty good.

Even with minimal caching, the data-access-coherence leads to pretty decent performance. Startup time is negligable (sans any database recovery and checking archive logs), and random and linear access to the records is relatively fast. Loading all messages into memory is only about 1/2 the speed of having them all already in memory with the old interfaces.

Given that all interfaces are incremental, and views have persistently maintained indexes, almost all of the per-record overhead can be hidden from a user by updating the UI more incrementally.

Notes

I tried adding some transactions to the various libdb calls, but it seems you need to use them always or never. It also seems to greatly affect reliability if the code crashes. Seeminly causing the database to require manual intervention more often.

As it currently stands, the reliability seems 'ok', but sometimes database recovery can get tied in knots and can only be fixed by removing the database files.

The archive logs are not flushed automatically - a libdb upgrade would add this feature. There is no code to do this manually either, although one or the other must be done. Otherwise, they grow without bound, both taking disk space, and significantly impacting on the startup time of the database environment.

I've applied a couple of libdb patches from sleepycat to get it to this state, although i'd recommend a libdb update to a newer version anyway, it has some additional features and bugfixes which should be of benefit.