File & Data Storage

The question of where and how photos keeps files and data has stalked us for some time. This page is an effort to approach it in a systematic manner.

Potential Goals

These aren't decided on and may conflict with one another.

Primary goals:

• Allow storage on remote and removable storage devices
  • Could be all photos or subsets
  • Could involve caching/syncing part of a larger photo collection locally
• Allow backups and data migration
• Data recovery when Photos isn't present, including:
  • Original image files
  • Edits
  • Manually created albums?

Related questions:

• Should Photos manage how files are organized on the filesystem?
• Should the images that are presented by Photos be visible as files on the filesystem?
• How to handle the case where someone has invested time in organizing their photos on the filesystem?
• To what extent should Photos interoperate with other photo editors, managers and operating systems?

• Should it be possible to store files and data so that they are picked up by 3rd party cloud providers (NextCloud, Dropbox, etc)?

Relevant Art

GNOME Photos

• Picks up images from ~/Pictures.
• No way to specify another folder to watch.
• Edits are saved as sidecar files beside the original images.
• Photos can be organized as albums, but there's no way to save/migrate/export this data.

Mac Photos

• Photos are stored in library files - each one is a single file that contains a set of images and metadata. The default library is located in the Pictures folder in the home directory.

  • It is possible to disable this behavior, so that photos can be viewed and organized on the filesystem. This has some side-effects.

• A library is set as the system library. This is where photos from iCloud are synced.
• It's possible to switch to a different library by:
  • Opening a library file from Finder.
  • Changing the default library from the settings.
  • Holding the option key when launching Photos, which shows a choose library dialog.

Windows Photos

• Collection is a chronological view of the content from the OneDrive Pictures folder (there's a setting to show photos from all locations instead).

• It can optionally highlight the most recently imported batch of photos, and auto-generated tags.

• Folders allows browsing photos on the filesystem. It shows directories and OneDrive accounts that have been added as "sources", either through the settings or by importing them; and highlights the most recently imported batch of photos.

• You can import from a folder or a device:
  • Importing a local folder adds the directory to the list of sources.
  • Importing from a device - the dialog shows the destination directory; it also allows changing the destination location (selecting a destination which isn't a source adds it as one). Import also has an option to delete the originals.
• When editing there are options to save or save a copy - so it's quite transparent as far as the file system is concerned.

Lightroom

• Files are manually imported. Imported files remain in their original locations on the filesystem.
• Imported files can be organized using collections and exported.
• Metadata about files is stored in catalogs - these are database files. You can have multiple catalogs and have to specify a location for each one. It's possible to export a subset of the current catalog as a new one.
  • It sounds like catalogs can be a bit fragile, since they are largely composed of links to files across the filesystem.
• Edits can be stored as sidecar files, although this is off by default.

Darktable

• Files are manually imported. Imported files remain in their original locations on the filesystem.
• Imported files can be organized as collections.
• Edits are stored as sidecar files by default.

Shotwell

• Local content under the top-level source directory can be optionally auto-detected, or only manually added. The top-level source directory defaults to ~/Pictures. Opting for auto-detection with 56G worth of JPEG and RAW images on an Intel i7 with SSD froze Shotwell 0.26.4 for close to an hour.

• Folders that were added as sources can be browsed through the sidebar.
• You can import from a directory, a device or another application:
  • Importing a local directory adds the directory to the list of sources in the sidebar.
  • Importing from a device copies the content to a sub-directory under the top-level source directory. The format of the sub-directory defaults to hierarchical year/month/day structure.
  • Used to support importing the image library from F-Spot. F-Spot's demise has led to this code being retired recently.
• Uses timestamps to group photos into events. Events can be merged together.

• Doesn't support albums per se.

• Edits are stored inside a SQLite database in ~/.local/share/shotwell/data/photo.db. No support for sidecar files.
• Can detect renamed or moved files, as long as they stay within the top-level source and their content, hence the checksum, hasn't changed.

Comparison

¹ Although you can export an image and then edit it elsewhere

(Presence of a particular feature on this table isn't necessarily a positive result.)

Discussion

Tentative Design

Comments