Future Gnome VFS

(Very first attempt to structure the Gnome VFS plans ChristianKellner)

Document based API

There seem to be some consens that a future Gnome VFS should be document based and not try to copy the POSIX I/O functions. Let's try to outline the how such a API should look like to fit the needs of diffrent applications:

queryInterface()-style approach

  • "The other suggestion I think is important for working on the VFS is to have a queryInterface()-style approach, where a backend is only required to support a very limited set of operations (such as "get entire file contents"), but can be queried for optional features such as "supports write", "supports move", "supports symlinks", etc. and those features would then be used only if available." (HavocPennington)

  • Basic functions that every VFS module MUST implement:

    • get capabilities () / query interface ()
    • read entire document ()

    Functions that every VFS module SHOULD implement:

    • save entire document ()
    • move document ()
    • trash document ()
    • get file info () (not so sure about these two)
    • set file info ()

    Functions that every VFS module MAY implement: (do we need 3 categories?)

(This is just a very first and very poor attempt - Please HELP! ChristianKellner)

  • To me it doesn't sound like a good idea that "read entire document ()" must be supported. This would exclude any kind of "streaming" module. -- -- SebastianRittau 2006-01-17 17:57:51

Common VFS Layer

Discussion

ThomasFitzsimmons: Why don't we eliminate the need for the gnome-vfs APIs by finishing and adopting Gnome VFS Mount?

HavocPennington: An async/gnome-vfs/http type API can wrap POSIX file APIs, but the reverse isn't true without breaking POSIX semantics in major ways (and keeping you from implementing things you need, like progress bars). The desktop file API can really be extremely simple: get/put user-visible documents/data. For config files and caches and such, using the vfs isn't even right; you want a guaranteed local/fast filesystem or an API like gconf. The way to think about a proper gnome-vfs would be a document load/save API more than as a streams API. This is ignoring things like Storage.

PaoloMaggi: I agree with hp, we really need a document load/save API. We already have some useful code in libeel (in particular eel_read_entire_file_async) that IMO should be moved to gnome-vfs (eventually adding some way to hook data transformation, like encoding conversion, etc.). This function, with its sync pair, should solve the document load API problem. About the document save API problem, we (PaoloBorelli and I) are working on this problem for gedit and we hope to be able to propose a generic document save (with backup) API useful also for other GNOME apps. Though, ATM, it only supports sync saving (not async). Note that ATM saving remote documents in a sane way (I mean without losing permissions, ownership, etc) using gnome-vfs is not so easy due to the limitations of the various gnome-vfs modules.

PaoloBorelli: Substantially I agree with what Havoc and Paolo Maggi said above... not much to add about it. However I want to describe what would would fit our needs in gedit, so that needs of a potential consumer of the api are clearer. Ideally what we would like to see is two functions: vfs_load and vfs_save(_with_backup). To get a look at how the current gedit code to obtain something like this looks like, take a peek at gedit-document.c (loading) and at http://bugzilla.gnome.org/attachment.cgi?id=35830&action=view for a preliminary version of the saving code. Our needs are:

  • Dealing with files must be async and non-blocking, so that the GUI doesn't freeze. On the other hand it must be smart enough to get a good throughput: for instance at the moment we have to special case local files and load them synchronously because using the eel_read_entire_file_async is too slow. Obviously we would prefer if this kind of thing was transparent.
  • A file operation must provide info about its progress so that for instance a progress bar can be displayed. Ideally also this kind of progress dialogs should standardazed, but in a separate library so that non-GUI apps can make use of the vfs api.
  • A file operation must be cancellable
  • A file operation must be resumable (think of "A backup couldn't be created, continue anyway?")
  • Loading a file information should be a "one step" operation and also collect available informations about the file: for instance we want to know if a file is read-only and its mime type so that we can pick the proper syntax highlighting... at the moment we have to stat() the file before reading it etc which is a mess.
  • Saving a file should allow more complex operations than a plain write: for instance we need to create a backup. It must also ensure that the file is not corrupted, for instance if we truncate it, start to refresh the contents and the operation fails. Dealing with this in all the cases is not trivial (think of symlinks etc)
  • Saving should be a sort of negotiation: the app makes a list of requirements (be able of preserving permissions, make a backup, etc) and the vfs returns a list of the things it is able to fulfill, so that the user can be prompted to continue anyway or to abort the operation.

I guess that's enough babbling from me for now...

RaphaelBosshard: I'd like a simplified way to manage files. Default handlers for duplicate filenames, moving to trash and so on. Currently that stuff is implemented in nautilus-file-operations, but it may be interesting for other applications too, without re-implementing. Copying from nautilus is sometimes quite tricky; everything is tightly integrated there.

NicolasTrangez: Maybe this is some dumb idea, but what about a standard document format for applications using GnomeVFS? This way a document-centered approach could be easier to achieve I guess. The "document" the application receives can be compared to CDATA in XML, so GnomeVFS does not care about how the actual data is stored, but it does store this document data in some standard format. This way additional document properties can be easily added too in a standardized way. A sample of this (XML, although I'm not saying at all this should be done in XML, its just the easiest way to explain it I think): 

<GnomeVfsDocument>
    <Metadata>
        <AuthorInfo>
            <Name>NicolasTrangez</Name>
        </AuthorInfo>
        <Comment>Blah</Comment>
        <Todo>Enhance this a bit</Todo>
    </Metadata>
    <DocumentData>Application gets only this when "opening" the file, can be anything: binary data, strings,...
Even exactly the same thing the application stores now</DocumentData>
</GnomeVfsDocument>

SeanMiddleditch: Just a note that I've started discussion on a true cross-desktop VFS system. The wiki for it is at http://freedesktop.org/wiki/Software/dvfs.

OlivierSessink on the simplified API: it should be capable of doing some optional extra actions as well:

  • create a backup file as in filename~ (allow user interaction if that fails)
  • check if the document on disk already exist (if the user intended to create a new document allow interaction)
  • check if the document on disk is changed (allow user interaction if the file is indeed changed on disk)

Another useful addition would be to have a function to create a local copy of a VFS file such that mmap() and any other local function work, and a function that synchronizes the local copy back to VFS. I followed the discussion on the GIMP development list on the usage of gnomeVFS, and they certainly need such functionality.

OlivierSessink on the document based approach: that really fits well into the object approach that gnome is taking (every directory an object, every file an object). The document based / object approach works for any document that is a single independent unit. However, many programs work with multiple files as unity, or even directory trees as unity. For example a mathematical simulation program might have multiple files that define different parts of the simulation. Or a familiar example: a tree with .c and .h files, a Makefile, autoconf macro's etc. This discussion is beyond gnomeVFS (hmm is there a wiki on the fileselector or nautilus?), but it is important to remember that not all documents are single independent objects.

GrundigLinux on las commentary from HavocPennington: As you said, things like gnome-vfs and KIO IMO suffers has a huge design hole: it divides the filesystem namespace and doesn't lets other apps. DVFS "solves" it proposing to create "special cli commands vfs-cat, vfs-cp, vfs-move" and a dvfs -> FUSE backend. I don't think this is a good idea (requiring to modify every app on eart...) I think FUSE can solve part of the problems that gnome-vfs and KIO (and dvfs) try to solve, but it won't solve some of the problems proposed here (It's not what it has to do, either). I think we could divide the funcionality here in two big parts: One, letting people accesing SMB volumes, remote SSH filesystems (FUSE), and the other an (library) API on top of the existing standard VFS (a freedesktop standard, DVFS "redone"?) to do all the document-related work. FUSE right now is not very "user-oriented" but it is definitively cool and it already has a ssh-fs (which, I've heard, is a great nfs replacement for most of uses) and lots of filesystems and lots of users, and it's already mainstream in linux since 2.6.14 or so i think: Gnome could implement all the "user-side" and still it'd be easier than redoing gnome-vfs, and it'd work in other desktops. And there're BSD people who already has implemented FUSE-compatible functionality and windows seems to have ways of doing it too (userspace filesystems are a "must have" for modern desktop operative system these days, every os is implemeting it or it's in their TODO list). There's also a lot of work going on in the Linux VFS which will allow doing cool things (per-process filesystem namespaces) which maybe can be used to do interesting things in desktops.

JohnNilsson: Not knowing the exact problem that gnome-vfs is supposed solve, and thinking that if not following POSIX, why do files at all? So I did some resarch on interesting articles on allternatives approches, and general thinking on the subject. Posting them here in no particular order:

As mentiond above, Storage, why not start with that? How would gnome-vfs relate to Storage?

NorbertFrese: The Project Portland also has a VFS-Task. It aims to provide a generic interface to VFS-functions through scripts and a library interface.

JoachimNoreiko: This might not be the right place for this suggestion, but I think the following should be possible in GNOME: Suppose a user opens a document, say in gedit. Then,

  • if the user renames the file in Nautilus, the gedit window updates with the new filename seamlessly
  • if the user moves the file in Nautils, the gedit window updates with the new path seamlessly

  • if another process changes the file (my use case is ChangeLog CVS conflicts btw), gedit is notified and offers to reload the file if no user changes have been made. (If there are unsaved changes, I guess you're in trouble...)

jdv: It might be nice if the VFS did some magic to make copying, moving, transcoding (...) files appear instantaneous. Take the use case of copying a movie from disk disk 1 to disk 2: a copying dialog would still inform the user of how much data has really been transfered. To Nautilus and the thumbnailer, however, it would appear as if the movie is already fully present on disk 2. If totem was launched, it too would be able to open the file as if it was already copied in full. In the background the VFS would start by copying the file sequentially from start to finish, but allowing for rescheduling depending on what files and bytes applications like a movie thumbnailer or totem needed. Similarly, when ripping a CD it should be possible to start listening to the ripped songs right away, even if that means feeding raw (WAV) data to the music player (I believe Be used to sell a device that did this). Also, as a related idea, perhaps a VFS should allow for smart coordinating of copying and movie large files. Say if I have two disks, each with 1 GB unused; if on both disks there is a file 2 GB big. If I cut and pasted one of these files from the one disk to another, Nautilus shouldn't give me an error. Instead it should tell me it can't move the file until space is freed on the second disk. Now when you cut and paste the 2GB file from the first to the second disk, suddenly both move operations should start simultaneously, and the VFS should be smart about making these operations succeed.

<<DateTime: execution failed [Bad timestamp u'timestamp': could not convert string to float: timestamp] (see also the log)>>: Alexander Larsson posted plans for moving to a new VFS architecture here. It appears that Alexander would like to see a new VFS close to the glib level so that its API (but not necessarily its filesystem implementations) would be cross platform and its dependencies noncontroversial. In his original e-mail, Alexander also proposed to move away from URIs to a new path-based naming scheme (which perhaps would not be exposed to users). In later messages he also seemed to consider (at some level) using IRIs instead.

Attic/GnomeVfsPlans (last edited 2013-11-26 20:33:26 by WilliamJonMcCann)