Shotwell Architecture Overview: Runtime Monitoring and Auto-import

Shotwell 0.8 introduced two related features, runtime monitoring of the library directory and auto-import from the library directory. The first is enabled automatically. The second is optional and must be enabled by the user.

The requirements sound pretty simple:

• For runtime monitoring, detect if a file has changed in the library directory. If so, update Shotwell’s library.
• Also for runtime monitoring, detect if a file has been deleted. If so, update Shotwell’s library.
• For auto-import, detect if a new file has been added to the directory. If so, automatically import it into Shotwell.

Additionally, there is a desire to have two aspects of runtime monitoring: a one-time startup scan that sweeps the directory tree and reports everything found, and monitoring itself, which reports changes from then on.

There’s a lot more to it, but that’s the crux.

Runtime Monitoring

Directory Monitoring

GLib’s directory monitoring API only allows for a directory’s immediate contents to be monitored. In order to monitor a directory heirarchy, a monitor must be enabled in each directory. This means if a directory is created, a monitor must be added to it.

For this level of functionality, the DirectoryMonitor class was created. DirectoryMonitor is given a directory to monitor and options to monitor all subdirectories (recurse) and to enable monitoring (with this turned off, only the startup scan is performed).

DirectoryMonitor is a fairly neutral class in Shotwell -- it knows nothing of photos or media or JPEG. It merely provides a service to scan and monitor files. To extend DirectoryMonitor into Shotwell, the LibraryMonitor class inherits from DirectoryMonitor and overrides key protected virtual methods. These virtual methods are called by DirectoryMonitor to notify subclasses of filesystem objects detected during the startup scan and of changes to those objects at runtime. Like the strategy for Shotwell’s data architecture, these virtual methods in turn fire complementary signals so external observers may be notified as well.

In addition, DirectoryMonitor stores a File object and its associated FileInfo for every file and directory it discovers while monitoring. (They’re removed if the file is deleted.) This can grow quite large, but allows for the filesystem to be traversed and examined without blocking I/O. Since many operations require knowing simple details about the files (such as modification time or file size) this keeps the main UI thread responsive. It also allows for loops to be detected in the filesystem due to symbolic links (which DirectoryMonitor supports and will traverse).

More information about DirectoryMonitor can be found by reading the comment at the start of the source file. The signals and virtual methods should be self-explanatory. They key to understand them is to understand that the "discovered" methods are only called during the startup scan.

Another important point is that DirectoryMonitor does everything in its power to report changes to the filesystem in the order they were reported via its installed monitors. However, DirectoryMonitor will also attempt to deduce aggregate operations and report them as one. For example, in earlier versions of GLib, a move-file operation would be reported as a delete followed by a create, even if the file was merely being renamed. DirectoryMonitor will put these two operations together (using its stored FileInfo) and report the move properly.

Library Monitoring

Using DirectoryMonitor as its base, LibraryMonitor converts the reported file operations into Shotwell objects and performs the proper operations on them to reflect the change. For example, if a file is deleted, the associated Photo object is marked offline.

Because different media have different concerns with how to deal with changes to their files, LibraryMonitor keeps a list of MediaMonitors, one for each type of media (today, Photos and Videos). MediaMonitor has an interface that’s similar to DirectoryMonitor’s upcalls. LibraryMonitor determines which MediaMonitor is appropriate for the file and invokes it appropriately. VideoMonitor has a more-or-less understandable implementation, reacting to each change in a file in a straightforward way.

PhotoMonitor is complicated because a Photo may have two backing files (the master and an editable, which is used when Photos -> Edit with External Editor is invoked). The PhotoMonitor must distinguish between the two and set Photo’s state accordingly.

Another aspect of MediaMonitor is that, for performance reasons, it wants to queue up changes and propagate them throughout the system all at once. For example, if a photo editor is saving changes to a file, several alteration notifications may be sent to Shotwell prior to the file being closed. Instead of updating the internal database with each alteration, MediaMonitor queues up the changes in an internal list, which is acted upon in cycles. Note that signal aggregation is used whenever possible. For example, if a directory is moved, all its files must be renamed internally. The queue issues all these changes at once, rather than one at a time. This is done by using a transaction controller.

Auto-Import

Auto-import is a special case of runtime monitoring. The notification being watched for is a file being created. If the file does not correspond to a media object (which would result in it being marked online), the file is examined. If it looks like the type of file Shotwell would be interested in, its queued for import.

Auto-import piggybacks on the BatchImport system. It merely creates BatchImportJobs for each file and submits it to a BatchImport object.