Writing a Custom Data Provider

As mentioned, a dataprovider can be a DataSource, a DataSink, or TwoWay (which is both a source, and a sink of data). If you are developing on an installed copy of Conduit then put your DataProvider in ~/.config/conduit/modules/ otherwise put them in the appropriate dataprovider or directory. The filename must end with Module.py.

The latest example dataprovider code descrbed on this page is always located at :

• The current SVN Source Code

• The current API Documentation

The test dataproviders also contain examples on how to do most everything :

• Test dataproviders

This page will not describe a few of the more esoteric parts of the module.

Introduction

We will create an example dataprovider demonstrating how to implement support for two-way sync to/from a device/webpage/application. We will also utilize a custom datatype, and demonstrate how to perform conversions on that datatype. This should illustrate how Conduit uses conversions to allow different dataproviders to be synchronized.

This example module does not do anything particuarly interesting, however is does form a useful skeleton upon which to base your own dataproviders.

Example - TwoWay Data Provider

Declaration

   1 MODULES = {
   2     "ExampleDataProviderTwoWay" :   { "type": "dataprovider" },
   3     "ExampleConverter"          :   { "type": "converter" }
   4 }

The most important thing to define is the dictionary MODULES which describes the DataProvider and any converters which your dataprovider may need to ineract with conduit's built in DataTypes. Of course if you only use built in DataTypes then you do not need to provide any converters.

This example DataProvider uses a custom DataType, and subsequently defines a converter which converts the DataType to text (which is the lowest common denominator, and generally allows you to synchronize most things together.

   1 class ExampleDataProviderTwoWay(DataProvider.TwoWay):
   2     """
   3     An example dataprovider demonstrating how to partition
   4     funtionality in such a way
   5     """
   6 
   7     _name_ = "Example Dataprovider"
   8     _description_ = "Demonstrates a Twoway Dataprovider"
   9     _category_ = conduit.dataproviders.CATEGORY_MISC
  10     _module_type_ = "twoway"
  11     _in_type_ = "exampledata"
  12     _out_type_ = "exampledata"
  13     _icon_ = "applications-internet"

Conduit uses class properties to describe the capabilities of the dataprovider.

Configuration

You must override configure() , and add the _configurable_ = True class property if you wish to present a configuration dialog to the user.

   1     def configure(self, window):
   2         import gtk
   3         import conduit.gtkui.SimpleConfigurator as SimpleConfigurator
   4         
   5         def set_foo(param):
   6             self.foo = int(param)
   7         
   8         items = [
   9                     {
  10                     "Name" : "Value of Foo:",
  11                     "Widget" : gtk.Entry,
  12                     "Callback" : set_foo,
  13                     "InitialValue" : str(self.foo)
  14                     }                    
  15                 ]
  16         #We just use a simple configuration dialog
  17         dialog = SimpleConfigurator.SimpleConfigurator(window, self._name_, items)
  18         #This call blocks
  19         dialog.run()

If you only need to get a few simple values then you may use the DataProvider.DataProviderSimpleConfigurator( class. Give it a dicionary including the following values

• Name: The text to display in the label

• Widget: The widget to include in the dialog, Gtk.Entry, Gtk.CheckButton are supported

• Callback: A funtion which is called, in which the value in the widget is stored

This will create a dialog like

   1         #Define the items in the configure dialogue
   2         items = [
   3                     {
   4                     "Name" : "Foo",
   5                     "Widget" : gtk.Entry,
   6                     "Callback" : set_foo,
   7                     "InitialValue" : self.foo
   8                     },
   9                     {
  10                     "Name" : "Bar",
  11                     "Widget" : gtk.CheckButton,
  12                     "Callback" : set_bar,
  13                     "InitialValue" : self.bar
  14                     }                    
  15                 ]
  16         dialog = DataProvider.DataProviderSimpleConfigurator(window, self.name, items)
  17         dialog.run()

You may specify more than one dictionary in items if you want multiple fields.

Refresh and returning data

   1     def refresh(self):
   2         DataProvider.TwoWay.refresh(self)
   3         self.data = [str(random.randint(1,100)) for i in range(10)]

Refresh is guarenteed to be called before get_all(). It is called in a thread which is separate from the main thread so this function can block and take a long time. Due to pygtk not being entirely thread safe do not attempt to do anything to the gui. If Refresh is unsuccessful then raise a RefreshError exception otherwise just return. Call the base class refresh method to let the GUI know we are busy

   1     def get_all(self):
   2         DataProvider.TwoWay.get_all(self)
   3         return self.data

The get_all() method will be called after refresh(). It should return a list of the locally unique identifiers (LUID) of the items to synchronize. DataSources should always call the base classes get_all() method. LUIDs must be strings.

get()

   1     def get(self, LUID):
   2         DataProvider.TwoWay.get(self, LUID)
   3         data = self._get_data(LUID)
   4         data.set_UID(LUID)
   5         return data

get() will be called once for every LUID provided by get_all(). It is also executed in a separate thread to the main GUI so the same rules apply as refresh(). If there is a non fatal (i.e dont stop the whole process) error then raise a SyncronizeError. If there is a fatal (i.e. stop getting data immediately) then raise a SyncronizeFatalError.

Each piece of data returned from get() should have a LUID associated with it (call data.set_UID()), and the mtime and open_uri should have been set (if appropriate).

put()

• General Put Instructions

Put is called for each data to be stored in the dataprovider. It cant rely on the state of the dataprovider in which data is stored, nor can it assume that it has put the data before. For these reasons it must check for the following conditions, and react accordingly

• We have never seen this data before (LUID = None), was this because;
  • The data really is new
  • The user re-installed Conduit and deleted all records of previously put data, therefor make sure we dont overwrite newer data with old
• We have seen this data before, and remember where we put it last time (LUID != None), we must therefor check
  • Is the previous data still there?
    • If not, is this because the user deleted it, and do they want us to put it back?
    • If so, have they modified it from another location, and would overwriting it be a conflict?
  • If overwrite is true, then we are getting called because the user cleared the previous conflict, and we should trust their judgement

put() should return a Record identifier represeting the put data. If this is a two-way dataprovider, and it is not too costly operation, its easiest to return something like the following;

   1     return self.get(LUID).get_rid()

Otherwise you should manually construct a Rid with at least the uid corresponding to the data which was returned, e.g

   1     return conduit.datatypes.Rid(uid=LUID, mtime=None, hash=None)

configuration

In order to save the dataproviders settings between use of the application we need to override the get_configuration() function.

   1     def get_configuration(self):
   2         return {"pages" : self.pages}

The function should return a dict, the key is the name of the instance variable in which the setting to be saved is stored. For example self.pages stores a list of page names, so the key is pages. At present strings, bools, ints or lists of strings can be safely saved. Their values are restored automatically when they are loaded, overriding set_configuration() is not necessary (in almost all cases).

If more complicated settings needs to be saved, and providing you implement your own set_configuration() method, you may place whatever data you want in the dictionary providing it is still one of the basic types outlined above.

The final method which must be implemented is get_UID(). This should return a string which represents the dataprovider in such a way that the following scenario is possible

1. User adds dp to canvas, configures and syncs
2. User deletes dp from canvas
3. User re-adds dp as in (1)
4. No data is duplicated because the UID in (1) and 3() was the same

The UID should represent the dateprovider without being too tied to its instance configuration. Simple changes to the dp configuration should not change the UID and hence cause duplicate data to be synchronized

   1     def get_UID(self):
   2         return "Example"

Custom Datatype

We define a simple data-type. Note that the _name_ must match what is specified in the DataProviders in/out_type class property.field of the MODULES dictionary.

   1 class ExampleDataType(DataType.DataType):
   2     _name_ = "exampledata"
   3     def __init__(self, uri, **kwargs):
   4         DataType.DataType.__init__(self)
   5         self.data = kwargs.get("data",0)
   6 
   7     def __str__(self):
   8         return self.get_string()
   9 
  10     def get_hash(self):
  11         return hash(self.data)
  12 
  13     def get_string(self):
  14         return "string %d" % self.data

We also provide a converter which renders the ExampleDataType to a file so that it may be synchronize with any other DataProvider with "file" listed as its in_type

   1 class ExampleConverter:
   2     def __init__(self):
   3         self.conversions =  {    
   4                             "exampledata,file"   : self.exampledata_to_file
   5                             }
   6                             
   7 
   8     def exampledata_to_file(self, data, **kwargs):
   9         f = Utils.new_tempfile(
  10                         contents=data.get_string()
  11                         )
  12         return f

Notice how we preserve some information in the conversion, such as renaming the file to be something more useful. Dont worry about setting the mtime on the file, or transferring the LUID to the new instance, conduit does that for us.