Build Brigade

The Build Brigade is an effort to automate discovery and reporting of Gnome build errors and
to make testing of the development version easy for everyone

The Gnome Buildbot

This is a continuous integration system based on Buildbot adapted to build all Gnome modules through jhbuild:

http://build.gnome.org

Our Goals

Roughly in order of priority:

  • Find Gnome build errors quickly: when a commit breaks the build, we should be able to find out that it happened and what caused it.

  • Find regressions quickly: use unit tests (e.g. check), functional tests (e.g. dogtail) and integration tests (e.g. dogtail) to find out when functionality breaks.

  • Make life easier for testers: build RPM and Debian packages from Git so that testers can install and use them. Virtual machine images are easy to use and growing popular.

  • Profile performance: discover when code runs slower than before

  • Generate coverage information: find out gaps in our unit tests.

  • Upload generated API documentation.

Detailed requirements obtained mainly from Luis Villa's experience with MicroTinder can be found here:

Existing efforts

Social aspects

  • Building trust with maintainers
    • We might get powers to revert patches to fix the build
    • Use HACKING file as policy for what to do in case of breakage
  • Educate maintainers in improving test coverage of their modules

Todo

This is a list of tasks that need to be done. If you think you can help with any of them, please, let us know, your help would be very welcomed ;-). After each task there is a set of attributes between parenthesis that state for (priority, difficulty, technology, asigneee). If you want to collaborate in a task which is already assigned, you can contact the people the task is assigned to. Otherwise, if you want to help in an unassigned task, you can talk to us using the mailing list and the IRC channel, or you can contact me (IagoToral) directly.

  • Build error notification using a mailing-list, similar to the gnome-commits one (high)
  • Improve Check based unit tests integration into buildbot view. (medium, easy, check-builbot-python-html, IagoToral)

    • Improve test results visualization.
    • Get counts of passed and failed tests.
  • dbus needs to be configured with --enable-tests --enable-asserts for the 'make check' results to be meaningful, but any production build should not have these flags
  • Improve current buildbot view (low)
    • Right now is synchronized with library.gnome.org view, but a review would be good to check if we missed something
    • It would be good to also expose additional information, ie:
      • Right now it shows the moduleset being used, it would be good to also write down the metasets (So: gnome-apps-3.0: "meta-gnome-core" "meta-gnome-apps-features")

Long term ideas

  • Add a way to test patches.
  • Integrate with Bugzilla for automated testing of patches.
  • Build Gnome with -Wall -Werror
  • Notify i18n teams of new strings. Maybe even having a string-freeze check that gets activated in freeze period.
    • ClaudeParoz: That's already implemented with Damned Lies. Each string-freeze break is reported to gnome-i18n ML.

  • Add a Makefile target 'make bugreport' to as many modules as possible, which knows where to send the bugreport (because of the variety of bug tracking systems in use). Making sure every HACKING file has the right information would be a good start.
  • Integration with SuSE Build Service

Done

Build Tools

Tinderbox 3

  • Pro: widely used (e.g. Mozilla)
  • Con: not actively maintained

JHBuild

JHbuild builds Gnome from git (before: svn/cvs) or tarballs

  • Pro: can already build Gnome
  • Pro: can work with multiple revision control systems
  • Pro: can be used in conjunction with other systems e.g. have JHBuild do the checking out and Tinderbox/BuildBot do the build

BuildBot

  • Pro: actively maintained
  • Pro: flexible
  • Pro: supports having dependency information

Garnome

Garnome builds Gnome from latest tarballs (both stable and unstable branch)

  • Pro: actively maintained
  • Pro: handles dependencies
  • Pro: reliably builds Gnome
  • Con: doesn't support building from SVN/Git

Requirements

  • Using existing JHBuild module set information -- duplicating module information is bad

Niceties

  • Don't rebuild modules that haven't changed
  • Use ccache
  • Support running "make check" or other test infrastructure

Questions

  • Should we build everything from scratch each time, or reuse existing builds?

ChristianKirbach: This cannot easily be answered. Updating the sources is faster than checking out everything each time. Keeping the old object modules, i.e. not running 'make clean', usually saves a lot of building time. However, cvs sometimes cannot update properly and gives up because of conflicts. I have sometimes experienced that after successfully updating the sources autogen must be run again for a successful build. I have also rarely seen that make clean actually was required. So, for fast building we probably want some fallback system like

  1. Update sources and build. (If updating fails jump to d?)
  2. If build fails run autogen and build.

  3. If build fails run make clean and build.

  4. If build fails purge the sources, check them out, run autogen and build.

  5. If build finally fails the module is definitely broken. Reasons can be a broken module or e.g. an API change in one of its dependencies.

I assume steps b and c recover the build in 90% of all failing cases.

JoseDapenaPaz: an easy solution is combining "compiling caches" (like ccache) and a "copy" update model (see my post about implementing checkout modes in jhbuild).

  • When a build fails, should we use the previous build and continue, or stop?

JoseDapenaPaz: I would definitely use the previous build. We cannot taint all the modules because of every single change. Anyway, it would be interesting that Continuous Integration could warn that an error compiling a dependency has happened.

Meetings

Background

These ideas have grown out of a presentation (CI BoF 2006) and a subsequent discussion at GUADEC 2006.

Contact us

You can contact us through:

Who are we?

BuildBrigade (last edited 2010-12-09 16:53:17 by AlejandroPiñeiro)