Camel.Provider

CamelProvider is a structure which describes a mail storage or transport backend, or both. It is basically a plugin mechanism.

For a client, the provider structure gives enough information that can be used to present the user with questions required to configure it. It includes a list of supported authentication mechanisms which can be used to connect to the services, and provider-supplied callbacks for comparing it's URIs.

This is the first information which is loaded by an external mail backend, so because of it's importance, I will cover it in minute detail.

struct CamelProvider

First, the structure itself:

 typedef enum {
        CAMEL_PROVIDER_STORE,
        CAMEL_PROVIDER_TRANSPORT,
        CAMEL_NUM_PROVIDER_TYPES
 } CamelProviderType;
 
 extern char *camel_provider_type_name[CAMEL_NUM_PROVIDER_TYPES];
 
 typedef struct {
        char *protocol;
        char *name;
        char *description;
        char *domain;
 
        int flags, url_flags;
 
        CamelProviderConfEntry *extra_conf;
        CamelProviderAutoDetectFunc auto_detect;
 
        CamelType object_types[CAMEL_NUM_PROVIDER_TYPES];
        
        GList *authtypes;
        
        CamelObjectBag *service_cache[CAMEL_NUM_PROVIDER_TYPES];
        GHashFunc url_hash;
        GCompareFunc url_equal;
 
        char *translation_domain;
 
        const char *license;
        const char *license_file;
 
        void *priv;
 } CamelProvider;

• protocol: The URI protocol that this provider implements. These must be unique, short, lower-case strings which describe the provider. e.g. imap for the IMAP backend. Note that Camel's URI's are not necessarily equivalent to any defined or standardised URL's (e.g. IMAP-URL).

• name: A short name of the provider. Could just be the same as the protocol with friendly case. e.g. IMAP

• description: A longer description describing what the provider is used for. e.g. "For reading and storing mail on IMAP servers."

• domain: The application domain of the provider. This will normally be mail, to indicate the backend supports mail operations. news is also available, although it is treated the same.

• flags: Flags describing the provider. See the flags description below.

• url_flags: Flags describing the URI format. See the url flags description below.

• extra_conf: This is an array of optional configuration information for the CamelStore part of the provider. This lists the options, their type, and descriptions that can be added to the URI as URI parameters. With the e-plugin part of evolution, this is no longer required.

• auto_detect: This is a callback that is called before any of the information in the extra_conf table is used. It will create a hash table of values to substitute for the defaults listed in extra_conf and can be used by a backend to pre-configure various options. With e-plugin this is also no longer required.

• object_types: This is an array of the CamelType of the CamelStore and/or CamelTransport for this implementation. These are passed to Evolution/Camel.Object.new() by Evolution/Camel.Session.get_service() to instantiate new services as they are requested by the application.

• authtypes: The full set of authentication types supported by this backend. Note that this is the list of authentication types that the backend knows how to implement, not the authentication types supported by a given server - Evolution/Camel.Service.get_auth_types() is used for that purpose.

• service_cache: Is a Evolution/Camel.Object#Camel.ObjectBag holding all active services. This will be setup by the camel_provider_register function using the supplied hash table functions.

• url_hash: A hashing function, that given a Evolution/Camel.URL will generate a hash of the significant parts of this URL required to identify unique service instances.

• url_equal: As above, but for checking if URL's are equal.

• translation_domain: This translation domain is used to translate any of the plain text strings in the CamelProvider structure. They must be registered appropriately in the camel_provider_module_init callback in the provider library.

• license, license_file: Old license gconf key and descriptive text file. This shouldn't be used anymore as all backends must be GPL.

• priv: Some private data for the provider. Can be used as a mechanism to pass provider-specific data from the provider initialisation function to plugins. Where there is some other out-of-band mechanism (i.e. common header files) to share the structure of the information.

flags

And now for the various flag bits. Some historic cruft here.

• CAMEL_PROVIDER_IS_REMOTE: The provider access remote data. This means it needs a connection and so forth. This isn't used so much anymore, it is only used to sort the providers in the folder tree and for the vFolder options to include local or remote folders. It should be set appropriately still.
• CAMEL_PROVIDER_IS_LOCAL: It isn't a remote provider? This is a redundant flag which is no longer used, do not use it in new code.
• CAMEL_PROVIDER_IS_EXTERNAL: This is an external provider. Originally this meant that the provider should not be listed in the folder tree. This is no longer implemented, do not use it in new code.
• CAMEL_PROVIDER_IS_SOURCE: This indicates that the provider is only a source for mail, and should not be accessed as a fully capable mail storage. This is used by the POP provider. IS_SOURCE providers will be listed in the account editor as a receiving mail type, but will not be shown in the folder tree.
• CAMEL_PROVIDER_IS_STORAGE: The provider supports storage of mail, i.e. read/write, multi-folder access. Like IMAP.

• CAMEL_PROVIDER_SUPPORTS_SSL: This is a connection based provider which also supports SSL. The URI parameter use_ssl will be set appropriately to one of three options. never means do not use ssl, whenever-possible is a stupidly named option that indicates TLS should be used, and always, another stupidly named option which means SSL must be used. The names and some of the behaviour is historic cruft that marketing didn't let us fix.

• CAMEL_PROVIDER_HAS_LICENSE: A rather redundant bit to say that the license member is set (a NULL check would suffice). Do not use this.

• CAMEL_PROVIDER_DISABLE_SENT_FOLDER: This means that the provider has its own implicit sent folder, and that client-side sent-folder processing should not be carried out. Any per-account sent folder setting UI should also be disabled.

Note that if adding new flags you should only add them to the end of the structure. Otherwise binary compatability with plugin modules could be affected - not that this is a very big problem with source-level compatability in Free Software.

url_flags

The url_flags setting tells the client code how to configure the basic URI components and how to display a UI suitable to configure them. This is a not-very-successful mechanism used to control some reasonably complex configurable UI components in a simple way.

The URI fields that are controlled in this way are as follows:

• USER: The username field of the URI.
• AUTH: The authmech field.
• PASSWORD: The passwd field.
• HOST: The host field.
• PORT: The port field.
• PATH: The path field.

Then for each field, there are 3 options. As well as the CAMEL_URL_* flag definitions that the CamelProvider implementers use, there are corresponding macros that client code can use to test these conditions. The macro's should be used to test the conditions since certain combinations imply others (e.g. NEED implies ALLOWS).

• CAMEL_URL_ALLOW_*, CAMEL_PROVIDER_ALLOWS(provider, flags): This means that the given field can be set on the URI. The UI should present input boxes for these fields as appropriate.
• CAMEL_URL_NEED_*, CAMEL_PROVIDER_NEEDS(provider, flags): The value must be supplied and must not be empty. For example it is hard to access an IMAP backend with no server. This can be used by the UI to implement basic input validation.
• CAMEL_URL_HIDDEN_*, CAMEL_PROVIDER_HIDDEN(provider, flags): The value should be set on the URI, and must be maintained when editing accounts, but no UI should be presented for it. This can be used by backends for example, that perform some sort of auto-discovery of various settings based on some other setting.

And finally there are two other bits that implementers can set and client code can test.

• CAMEL_URL_FRAGMENT_IS_PATH: When setting a folder-path on a Evolution/Camel.Store URI, the path should be set on the URI fragment, and not the URI path. This is used in most file-related backends, because they use the URI path for the path to the file or directory instead. This is a bit of a hack that really only exists because there are not the appropriate virtual methods on Evolution/Camel.Store to resolve folders by URI.

• CAMEL_URL_PATH_IS_ABSOLUTE: I think this means that paths must start with /. This isn't used anymore though. Do not use it in new code

extra_conf

The extra_conf field is used to specify an array of options for this provider. Infact it will only be options for the Evolution/Camel.Store part of the provider, as they are only ever displayed on the receiving options page of the account editor.

This stuff predates EPlugin, although due to it's simplicity there are probably reasons to keep it around. It still interacts with EPlugin, so the pressure to remove it is minimal.

Basically each item is presented in a UI one line at a time. Additionally, all items can be encased in a single level 'section', which is usually displayed in the UI using a frame. The sections must be defined; no items can be defined outside of sections.

 typedef enum {
        CAMEL_PROVIDER_CONF_END,
        CAMEL_PROVIDER_CONF_SECTION_START,
        CAMEL_PROVIDER_CONF_SECTION_END,
        CAMEL_PROVIDER_CONF_CHECKBOX,
        CAMEL_PROVIDER_CONF_CHECKSPIN,
        CAMEL_PROVIDER_CONF_ENTRY,
        CAMEL_PROVIDER_CONF_LABEL,
        CAMEL_PROVIDER_CONF_HIDDEN
 } CamelProviderConfType;
 
 typedef struct {
        CamelProviderConfType type;
        char *name, *depname;
        char *text, *value;
 } CamelProviderConfEntry;

The way this works is as follows:

• The type field specifies the type of the entry
• The name field specifies the name of the entry; all items except _END items must have a name.

• depname specifies which checkbox is used to enable or disable this item. If the checkbox matching depname is enabled, then this item must be sensitised.

• text is the label to display for this item; all items except _END items must have a label.
• value specifies a default. The format of this depends on the type.

type is the type of the config entry:

• CAMEL_PROVIDER_CONF_END: Use this to define the last configuration item in the array.

• CAMEL_PROVIDER_CONF_SECTION_START: Use this to define the start of a section. The name must be supplied, which should be common across all providers for common features - see camel/providers/imap/camel-imap-provider.c for typical examples. text is the human-readable label for this section (frame).

• CAMEL_PROVIDER_CONF_SECTION_END,: All sections must have an END marker once their items are defined.

• CAMEL_PROVIDER_CONF_CHECKBOX: This is a checkbox item holding an on/off true/false value. text is the label of the checkbox, and value will be "0" or "1" to indicate the default value of the checkbox when creating a new account.

• CAMEL_PROVIDER_CONF_CHECKSPIN: This input box consists of a checkbox plus a spinbox (a number entry box). value is a string describing the checkbox state and controls for the numerical input. The format is 'on,min,def,max'. On is 'y' or 'n', to indicate if the checkbox is enabled. min, def, and max are floating point values specifying the minimum, default and maximum input values allowed.

• CAMEL_PROVIDER_CONF_ENTRY: This is a normal text entry box.

• CAMEL_PROVIDER_CONF_LABEL: This is a hack provided for Exchange Connector. It allows the labels of the hostname or username entry boxes to be overriden. Set the label to override as the name, and set text to the new label.

• CAMEL_PROVIDER_CONF_HIDDEN: This is a hidden configuration option. This is no longer implemented, do not use it in new or any other code.

There are a couple of defines for standard defaults, although all of these fields are normally handled by the URL options above, so there is rarely any need to use them.

 #define CAMEL_PROVIDER_CONF_DEFAULT_USERNAME  { CAMEL_PROVIDER_CONF_LABEL, "username", NULL, N_("User_name:"), NULL }
 #define CAMEL_PROVIDER_CONF_DEFAULT_HOSTNAME  { CAMEL_PROVIDER_CONF_LABEL, "hostname", NULL, N_("_Host:"), NULL }
 #define CAMEL_PROVIDER_CONF_DEFAULT_PATH      { CAMEL_PROVIDER_CONF_ENTRY, "path", NULL, N_("_Path:"), "" }

Then we get to the auto-detect function.

 typedef int (*CamelProviderAutoDetectFunc) (CamelURL *url, GHashTable **auto_detected, CamelException *ex);

Again, because of EPlugin, the need for this function is much reduced. However, it can still be used to override the defaults on a per-service basis.

At the provider end, if defined, auto_detect will be invoked with the current URL describing the user's settings. It can then add name-value pairs to a GHashTable for any default values it wishes to override.

In the client code, this should be called before presenting the extended options. When generating the option UI, a value is first looked up in the hash table, if a value is present, then that should be used to initialise the default. If it is not, then just use the defaults provided in the extra_conf table.

Provider use

In addition to defining the CamelProvider type, there are some utilities for querying providers, initialising the provider system, and other uses.

Client functions

Before any functions in libcamel-provider are used, the library must be initialised. This should be called after libcamel is initialised as well.

 void camel_provider_init(void);

Another function which can be used to build user interfaces to choose between backend types is the listing function. If load is true, then the providers not already loaded are loaded. Normally this would be used unless you are loading specific providers manually.

 GList *camel_provider_list(gboolean load);

auto_detect is just a helper function to invoke the auto_detect callback.

 int camel_provider_auto_detect(CamelProvider *provider, CamelURL *url,
                                GHashTable **auto_detected, CamelException *ex);

Internal functions

These functions are normally used by Evolution/Camel.Session or internally to resolve services, but they may also be used by client code for lower-level access.

get will resolve a URI to a provider. In fact, it only takes the protocol part of the URI and uses that to look up the provider from the plugin list. The provider is loaded if it isn't already loaded.

 CamelProvider *camel_provider_get(const char *url_string, CamelException *ex);

And load will load a specific shared library directly, given the full path to it. It will call the camel_provider_module_init function which must be defined within the shared library; it is then up to the provider to register any new providers.

 void camel_provider_load(const char *path, CamelException *ex);

Plugin functions

camel_provider_module_init is not a function in libcamel-provider, it is the signature of the initialisation function that the provider module must export. See any of the existing providers or the examples.

 void camel_provider_module_init(void);

Note that the provider is not given any arguments, specifically, no CamelSession is supplied. This is because providers are global code-resources, not active, live objects. This should be taken into account when allocating any provider resources. No remote connections or other session related resources must be allocated from module initialisation functions; they should be kept as stateful information on the active Evolution/Camel.Service objects.

Inside the provider's module_init callback, it should call camel_provider_register for all of the providers the provider implements. The protocol field of the providers registered should be set to match the URI prefix in the Evolution/#.urls file.

 void camel_provider_register(CamelProvider *provider);

Provider installation and initialisation

Providers are essentially plugins, installed in a specific location with a specific filename. The mechanism is very simple but adequate enough, although they must be loaded before anything other than their URI prefix can be determined.

.so

Providers must be installed in a the specific provider installation directory. This can be queried using pkg-config for the camel_providerdir directory. It will normally be a version-specific directory, currently tied to the EDS API version (there are some problems with this though).

The providers must be shared libraries, installed with a name such as 'libnmap.so'. The '.so' part is mandatory.

.urls

Inside the camel provider installation directory, there must be a filename which matches the '.so' filename, but ending in '.urls' instead. Inside this file is listed each URI protocol that this provider implements, one per line.

At camel_provider_init time, these '.urls' files will be loaded and a table created so that it can later resolve URI's to provider modules.

Initialisation

When they are loaded (which may not be at application startup), the providers will be initialised by calling the camel_provider_module_init function, as described above.

By convention only, within Evolution, these will always be called from the 'main' thread, i.e. the Gtk main loop thread. This allows gtk dependent code (and others such as bonobo) to run safely. Note however that this is the ONLY function which is guaranteed to be called from this thread, any other functions on the live objects, including the auto_detect function may be invoked from any thread.

Examples

It all sounds a bit more complex than it really is, here are some examples (or example).

Example: Mail storage backend

We'll define a new mail backend called 'nmap'. I'll use the gcc extended syntac for structure initialisation here (no idea if this is C99 or not), as it makes it a little more readable.

We're just using the default camel url hash functions for simplicity.

Don't get too excited, this is just an example of what might be a real protocol, it's options probably aren't correct, and it wont be filled out.

 static CamelProvider imap_provider = {
        .protocol = "nmap",
        .name = N_("NMAP"),
        .description = N_("For reading and storing mail via the NMAP protocol."),
        .domain = "mail",
        .flags = CAMEL_PROVIDER_IS_REMOTE | CAMEL_PROVIDER_IS_SOURCE | CAMEL_PROVIDER_IS_STORAGE,
        .url_flags = CAMEL_URL_NEED_USER | CAMEL_URL_NEED_HOST | CAMEL_URL_NEED_PORT,
        .extra_conf = nmap_conf_entries,
        .translation_domain = GETTEXT_PACKAGE
        .url_hash = camel_url_hash;
        .url_equal = camel_url_equal;
 };

Now, we'll just create a simple option that lets the user specify a command to execute to connect to the service, rather than using TCP. We copy the same setting from the IMAP code intentionally - for consistency, and so that any plugins can automatically attach to the same section using the same path.

When configured, these options become URI parameters. Also note that the depname is used for the command field; this means that this field will be de-sensitised if the option is disabled, which is exactly what we want.

 CamelProviderConfEntry nmap_conf_entries[] = {
        { CAMEL_PROVIDER_CONF_SECTION_START, "cmdsection", NULL,
          N_("Connection to Server") },
        { CAMEL_PROVIDER_CONF_CHECKBOX, "use_command", NULL,
          N_("_Use custom command to connect to server"), "0" },
        { CAMEL_PROVIDER_CONF_ENTRY, "command", "use_command",
          N_("Command:"), "ssh -C -l %u %h exec /usr/sbin/nmapd" },
        { CAMEL_PROVIDER_CONF_SECTION_END },
        { CAMEL_PROVIDER_CONF_END }
 };

And now comes our provider init callback. We just setup any fields we cannot setup using static assignments, and register it. We also need to register the gettext package so that dgettext can find our translation domain. We support no authentication types.

 void
 camel_provider_module_init(void)
 {
        nmap_provider.object_types[CAMEL_PROVIDER_STORE] = camel_nmap_store_get_type();
        nmap_provider.object_types[CAMEL_PROVIDER_TRANSPORT] = camel_nmap_transport_get_type();
 
        bindtextdomain(GETTEXT_PACKAGE, NMAP_LOCALEDIR);
        bind_textdomain_codeset(GETTEXT_PACKAGE, "UTF-8");
 
        camel_provider_register(&nmap_provider);
 }

Note that the various gettext related defines depend on the installation settings. The various man pages and the gettext library info pages have details.

So that's about it. We then compile this file and all the implementation into a libcamel-nmap.so file, create a libcamel-namp.urls file containing a single line of nmap and install it into the camel_providerdir.

Example: Specifying a SASL authentication type

Many connection oriented protocols support SASL authentication modules. You just need to specify that auth is required, and then list those that the protocol supports.

Add CAMEL_PROVIDER_ALLOW_AUTH to the flags field, then add whatever SASL modules are required to the authtypes list in the module init function, they can be queried by name, see Evolution/Camel.SASL.

        nmap_provider.authtypes = g_list_append(nmap_provider.authtypes, camel_sasl_authtype("CRAM-MD5"));
        nmap_provider.authtypes = g_list_append(nmap_provider.authtypes, camel_sasl_authtype("PLAIN"));

Note that some authtypes may not be compiled in, you may need to verify the authtype exists (KERBEROS_V4 for example). If you just want to list all SASL authtypes implemented, just use:

        nmap_provider.authtypes = camel_sasl_authtype_list(FALSE);

Specifying if you want the PLAIN authtype or not.

Also see Evolution/Camel.Service for information about CamelServiceAuthType.

Example: Specifying an alternative authentication type

Often a given backend will have a non-SASL authentication default, or other possibilities. You can specify whatever authentication types are supported by adding custom entries to the authtype list. CAMEL_PROVIDER_ALLOW_AUTH also needs to be added to flags.

 CamelServiceAuthType nmap_password_authtype = {
        .name = N_("Password"),
        .description = N_("This option will connect to server using a plaintext password."),
        .authproto = "",
        .need_password = TRUE
 };

Note that this would normally be defined non-static; the CamelService implementation will also have to access it, although it could also just walk the CamelProvider list.

Also note the use of the empty authproto. This means this is the default connection method, and corresponds to an empty or missing Evolution/Camel.URL auth mechanism.

Then in the module init function, you just add it to the auth type list:

        nmap_provider.authtypes = g_list_append(nmap_provider.authtypes, &nmap_password_authtype);

Another example of a special authentication type might be an anonymous connection:

 CamelServiceAuthType nmap_anonymous_authtype = {
        .name = N_("Anonymous"),
        .description = N_("This option will connect to the server anonymously."),
        .authproto = "anon",
        .need_password = FALSE
 };

Is is up to the Evolution/Camel.Store or Evolution/Camel.Transport implementation to interpret these, so you can add whatever you like.

Notes

Although it has served quite well, CamelProvider could use a bit of work.

For starters, CamelProvider should really be a first-class Evolution/Camel.Object rather than a static structure. This could allow for more dynamic information queries based on object properties or other means.

If Camel had a plugin system, it would make sense for this to be just one type of plugin. In that event, the .urls file would be replaced by a more descriptive one; it should be possible to list information about a provider without loading it from disk for example.

Given the Evolution now has EPlugin, the requirements for providers to self-describe themselves is lessened. Infact much of the UI-related information including most of the url_flags bits could probably be dropped if this was enforced.

The way store + transport providers is implemented and used is a bit messy. Perhaps some other mechanism is required to define them.

Any unified account work may also impact on providers; the current URI-based configuration mechanism is problematic. It makes it difficult if not impossible to modify active objects. Again if CamelProvider was an active object, perhaps this configuration information would be specified differently, instead of using a URI. Perhaps a ServiceConfig object with it's own serialisation interface should be used, and also used inside any Unified Account system (remember, Camel code cannot use GObjects directly).

Given that much UI stuff is going into libedataserverui anyway, perhaps some complimentary factory methods need to be defined for creating UI's for editing settings in a re-usable way.