JHBuild Module Refactor

The current JHBuild module type system has a number of limitations that it would be nice to eliminate.

Current Scheme

A jhbuild.modtypes.base.Package instance in JHBuild defines a state machine that describes how to build modules of that type. The state machine is operated like so:

the initial state is set to "start"
if the current state is "done", then we stop.
a method by the name "do_statename()" is called on the Package instance.
the method returns a tuple of the form (nextstate, errormessage, list-of-alternative-nextstates).
if errormessage is None, then the state is set to nextstate, and we loop to step 2
if errormessage is not None, then we display the error message, and ask the user the following:
- whether they want to ignore the error, in which case we set the state to nextstate and loop to step 2
- whether they want to retry, in which case we leave the state the same and loop to step 2
- whether they want to pick one of the alternative next states, and loop to step 2
- other options (e.g. start shell)

The current set of module types are arranged in the following heirarchy:

Package (abstract)
- AutogenModule (abstract)
  - CVSModule
    - MozillaModule
    - GCJModule
    - GDBModule
  - SVNModule
  - ArchModule
- MetaModule
- Tarball

The subclassing is done in order to share implementation of various build states.

Problems

Some of the problems with the current scheme include:

Luis wants support for modules using a Python distutils setup.py script instead of an autogen.sh and Makefiles (see bug #308328). His implementation adds support for distutils modules hosted via CVS by subclassing CVSModule. To support distutils modules hosted in Subversion, Arch or tarballs, we'd need three more module types duplicating much of the logic.
- indicates that the current heirarchy of module types is not sufficient -- we need an easy way to get every combination of download scheme (tarball, CVS, Subversion, Arch, etc) and build scheme (autogen.sh, setup.py, etc).
edges of the state graph are not available in a declarative form -- the information is only available by running the state machine.
handling of skipped build stages is handled in the previous state. For example, the AutogenModule.do_start() method reads as follows in order to support flags like --no-network:
- ```
   1 def do_start(self, buildscript):
   2     builddir = self.get_builddir(buildscript)
   3     if not buildscript.config.nonetwork: # normal start state
   4         return (self.STATE_CHECKOUT, None, None)
   5     elif buildscript.config.nobuild:
   6         return (self.STATE_DONE, None, None)
   7     elif buildscript.config.alwaysautogen or \
   8              not os.path.exists(os.path.join(builddir, 'Makefile')):
   9         return (self.STATE_CONFIGURE, None, None)
  10     elif buildscript.config.makeclean:
  11         return (self.STATE_CLEAN, None, None)
  12     else:
  13         return (self.STATE_BUILD, None, None)
```
The complexity goes up with the number of consecutive stages that can possibly be skipped.
- Ideally the build state would be able to test whether it should be skipped itself. The build loop could then easily move on to the next build stage (easy if (2) is addressed.
The states are implemented as Python methods. It would be nice if we could describe state methods in a less verbose manner.
- string.Template maybe?
The xorg guys want a way to enable/disable certain modules based on architecture or OS. Ideally it would be easy to easily add such a feature without modifying all module types, or adding functionality to the base class.

New Scheme

Declarative state transition information

A combination of method decorators and a metaclass (or class advisors) might provide a clean way to externalise the state machine information. Something like this:

   1 class CVSModule:
   2     @nextstate('configure')
   3     @skipif('nonetwork')
   4     def do_checkout(self):
   5         ...
   6 
   7     @nextstate('clean', onerror='force_checkout')
   8     @skipif('nobuild')
   9     def do_configure(self):
  10         ...
  11 
  12     @nextstate('build', onerror='force_checkout')
  13     @skipif('nobuild')
  14     @skipif('!makeclean')
  15     def do_clean(self):
  16         ...
  17 
  18     @nextstate('check', onerror='force_checkout')
  19     @skipif('nobuild')
  20     def do_build(self):
  21         ...
  22 
  23     @nextstate('install', onerror='force_checkout')
  24     @skipif('nobuild')
  25     @skipif('!makecheck')
  26     def do_check(self):
  27         ...
  28 
  29     @nextstate('done')
  30     @skipif('nobuild')
  31     def do_install(self):
  32         ...

So the state machine would be executed like so:

If any of the @skipif conditions hold, pretend the stage succeeded. Otherwise, run the method.
If the stage succeeded, move on to the state given in @nextstate decorator.
If an exception was raised (particular exceptions?), ask the user what to do, presenting the onerror state(s) as options.

Status

Done:

CVSModule, ArchModule and SVNModule have been removed.
New version control interface added as jhbuild.versioncontrol, with additional support for Darcs and GIT.
AutogenModule is now used for CVS, Subversion, Arch, Darcs and GIT modules directly.
Repositories can be registered in .modules files with a general purpose <repository> element.
If do_* methods have next_state/error_states attributes, they are used to perform the state transitions.
New .modules file syntax to allow deprecation of <cvsmodule>, <svnmodule> and <archmodule> elements (<autotools>).
New "tarball" version control type, allowing tarball builds to use the same build code as the other autotools based module types. Only bootstrap.modules has been converted so far.

Todo:

Support for @skipif or similar in module type definitions. Convert remaining state functions and remove old system.
Look at supporting Python distutils modules (e.g. LDTP) (bug 308328)
Support for Perl module build infrastructure (bug 342638)

CategoryJhbuild

Attic/JhbuildModuleRefactor (last edited 2013-11-25 16:28:34 by WilliamJonMcCann)