Camel.Local

CamelLocalProvider implements several related local-storage backends. Maildir, MH, and Berkeley Mailbox (mbox).

It provides a common base class of folder, foldersummary and store, with some virtual methods so that common code can be re-used between all of them.

It isn't decisive as to whether the local super-class really adds anything but overhead to the code ...

Camel.LocalStore

The local store class adds a couple of virtual methods, to get filenames used for folders and meta-data files.

This is used by the base class so that it can implement many of the basic store methods directly, renaming, deleting, and creating the base storage location for folders.

Other facilities that the local store class provides is the setup of the persistent properties files on trash and junk folders, and keeping track of the root directory of the store in a convenient manner.

Camel.LocalFolder

The local folder class adds virtual methods for locking folders, and a factory method for instantiating the folder summary object to be used for this folder. It initialises and maintains a set of filenames used to find various resources, indexes, folder files, etc.

It provides a persistent property, CAMEL_LOCAL_FOLDER_INDEX_BODY which indicates whether or not the folder is to be indexed or not.

It drives some of the methods, refresh_info and sync, implementing the lock, summary-sync-unlock calls, so implementing classes only need to implement the summary methods. Likewise for rename, and delete.

It also implements a default property handler for describing the folder.

Camel.LocalSummary

The local summary class adds virtual methods for checking the summary matches the folder, and synchronising any changes from the summary to the folder.

It will do some index management if one is set on the summary.

Camel.MBOX

The Mbox provider implements a Berkeley Mailbox backend. This backend supports multiple folders which are stored in a tree of directories. All folders can contain folders or messages.

This is the default backend used by Evolution mail.

There are many more details than can be covered here that should be checked against the source, for example, how the various 'summarising', 'indexing', 'checking', and appending processes work.

Mbox URI format

The mbox uri will be of the form:

 mbox:///path#folder
 mbox:/path#folder

; path : Is the path to a directory containing the folders. It is treated as an absolute path. ; folder : Is the full name of the folder.

All paths are in normal URI format, i.e. separated by /.

Mbox folder layout

Since Linux filesystem directories can generally only contain files, and files cannot generally contain directories, a special directory format is required to represent folders.

Each folder containing messages is represented by a file containing the messages in Berkeley Mailbox format. Then each folder which contains subfolders has a directory, with the same name as the folder, but with ".sbd" appended to it. There are also some meta-data files associated with each folder.

So for a folder imaginatively named "Folder".

 Folder
 Folder.ev-summary
 Folder.cmeta
 Folder.ibex.index
 Folder.ibex.index.data

Where; ; .ev-summary : Is the summary file representing an index of all messages in the folder. ; .cmeta : Stores any per-folder state or persistent properties. ; .ibex.index, .ibex.index.data : The Evolution/Camel.Index index and data files for content indexing.

If the folder has subfolders, a subfolder directory will also be created:

 Folder.sbd/
 Folder.sbd/Subfolder1

etc.

Mbox format

The format is 'standard' Berkeley Mailbox format. That is, RFC822 messages are concatenated together, separated by "From " lines; lines which begin with "From ", followed by some other information.

Although the mbox code tries to format the rest of this line in a way compatible with other mail clients, there is no standard, so this is not guaranteed to work.

Any message content which begins a line with "From " is munged to become ">From ". Note that this is a munging process, not an escaping process - it is never reversed.

A slight variation of the Berkeley Mailbox format (among the many slight variations), is for this process to be two-way, that is "From " -> ">From " -> "From" on reading. Infact it needs to take into account any number of ">"'s before the "From", so that it is properly reversible.

This wouldn't be terribly difficult to implement, but it hasn't been so far.

Mbox Locking

All operations are locked using all locking mechanisms, as described in Evolution/Camel.Misc#Camel.Lock.

This does reduce performance quite a bit when appending many messages, as each is appended using a separate lock - Evolution/Camel.Folder.copy_to could be implemented in a way which reduced this overhead however. It has little effect on most other normal operations.

There could be some argument to be made, that since this backend is designed for mail which is at the sole use of Evolution, it doesn't need locking. Many users would disregard this and lose mail though.

Mbox X-Evolution

The X-Evolution header is used to assign a persistent unique identifier to every message in the mailbox. If a mailbox is opened and messages do not have one assigned, it must be added.

It also has room for storing 16 bits of message flags.

It used to also store the tags and user-flags, but these were removed since it usually requires a full mailbox rewrite if these change. Normal flags can be written with a simple seek and write of that header value.

 X-Evolution: UUUUUUUU-FFFF

; UUUUUUUU : An 8 character hexadecimal value which represents the numerical UID of this particlar message ; FFFF : A 4 character hexdecimal value which represents the flags of the message.

A typical example might be:

 X-Evolution: 0000002-0002

This message is message number 2, and it is marked DELETED (CAMEL_MESSAGE_DELETED = 1<<1).

Mbox bugs

There is a rather nasty but that crops up from time to time whereby the "index and summary mismatch even after sync". That is because the sync process doesn't perform a full synchronisation, it only checks for the presence of messages in the summary. Ones that are missing are not properly updated. The Evolution/CamelDS.Local disksummary branch version fixes this with all new algorithms.

Camel.Spool

The spool provider is a subclass of the Mbox provider, overriding some of the classes and methods to provide a different directory layout and some different properties.

This has two modes of operation, and how it acts depends on what path it is given. If it points to a file, then the store creates a single 'Inbox' folder to represent the messages in this file.

If it points to a directory, then it creates a tree of folders which it finds at that path. Folders are identified as being valid when they begin with "From ". These folders, unlike the mbox folders, can only contain messages or folders, but not both. This is how pine/elm/mutt work with mbox backends.

Apart from these limitations, some differences in the way the Evolution/#Spool X-Evolution header is written means that you should avoid this backend unless you are just visiting old archived data from other programs, or really need to use the other program to access the mail still. It will be somewhat slower than using any other backend.

Spool URI format

 spool:///path#folder
 spool:/path#folder

This is basically the same as the Evolution/#Mbox URI format.

Spool folder layout

When in directory mode, spools directories and folders use native Linux folders and files. Hence the limitation that folders contain other folders or messages but never both.

 Folder
 FolderWithSub/Subfolder

The meta-data files are stored elsehwere, relative to the Evolution/Camel.Session.storage_path.

It is a bit convoluted, but I think will be something like:

 /home/user/.evolution/mail/spool/home/user/.evolution/mail/local/FolderWithSub_Subfolder.cmeta
 /home/user/.evolution/mail/spool/home/user/.evolution/mail/local/FolderWithSub_Subfolder.ibex.index
 ...

For a physical folder location of

 /home/user/.evolution/mail/local/FolderWithSub/Subfolder

This should really be fixed to be something flatter.

Spool locking

Spool mailboxes are locked only using "dot" locking. This seems to be the most widely used locking mechanism for local delivery agents, and since this is usually delivered via NFS, most other locking schemes will often fail.

Spool X-Evolution

This is the same as the Evolution/#Mbox X-Evolution header.

There are some slight differences as to when it is written however. Since spools are likely to be altered in between locks held by Evolution, the mailbox must be updated immediately with any missing X-Evolution headers. This process will be followed whenever the mailbox is altered outside of evolution.

Camel.Maildir

The maildir backend implements, surprise surprise, maildir format storage. This is a simple and safe storage mechanism that uses a special directory structure and filenames to implement a lock-less mail storage. This can be of great benefit for NFS mounted home directories where locking is often nonexistant or unreliable.

Maildir folder layout

Maildir folders are all Linux folders but with 3 special sub-directories where the messages are held. Because of this, subfolders can just be normal folders too.

The three folders are cur, new, and tmp.

 Folder/
 Folder/cur
 Folder/new
 Folder/tmp

Subfolders are easily created:

 Folder/Subfolder
 Folder/Subfolder/cur
 ...

Because this was really only ever designed to work with Evolution, the metadata is also stored here, using the same names as Mbox:

 Folder.cmeta
 Folder.ev-summary
 ...

Some other maildir backends, for some unknown reason, try to display these.

Maildir format

The folders themselves contain a single file per message. The filename itself consists of a base part, and a flags part. The base filename of the message is used as the UID of the message. The flags part is additional persistant storage of some of the Evolution/Camel.FolderSummary system flags.

The filenames follow an earlier definition of the 'standard' for Maildir.

 time.pid_inc.hostname

; time : Result of time(0) as an integer. ; pid : The pid of the current process, as an integer. ; inc : A number incremented by 1 every time a message is created. ; hostname : The hostname of the current host, if it can be determined.

The goal of this naming scheme is to create a unique name - no matter which process or machine the name is created on. This isn't a very strong scheme, and newer maildir code uses a different scheme, although we do some checks for uniqueness before using any name.

Maildir++

We support Maildir, not Maildir++. Maildir++ seems to be a way to structure subdirectores which differs significantly from the way we do. Each has their advantages, but we just do it differently. Maybe someone can add Maildir++ support if they really care.

Note that Maildir doesn't actually specify a multi-directory layout, that I can ascertain (or didn't when the code was written). The problem with incomplete 'standards'.

Camel.MH

For completness, a MH backend was written too. MH stores mail as numbered files in directories.

MH was NEVER designed for concurrent use by different processes, let alone different machines. Nothing is ever locked, it is even less standardised than MH, and there's no mechanism for storing message state.

Unless you have a specific need to talk to archived MH mail, never use this, use Maildir instead.

So, if you need to know more, read the code.

Notes

Just looking through any of these implementations is a great advertisement for removing the CamelFolderInfo interfaces from Evolution/Camel.Store.

They're a pretty good advertisement for rewriting the Evolution/Camel.FolderSummary interfaces too - which was done on the disksummary branch. For example, the way UID's are assigned to messages being added to the summary is quite convoluted, depending on whether various summary-global variables are set or not.

Apps/Evolution/Camel.Local (last edited 2013-08-08 22:50:00 by WilliamJonMcCann)