BuildStream quick start guide (DRAFT!!)

What is BuildStream?

BuildStream is a command-line tool that executes software build and integration pipelines.

It supports a wide range of use cases: currently you can produce Debian packages, Linux virtual machine images, and Flatpak application bundles, and it's possible to extend BuildStream through its plugin system to produce any type of output you like.

All builds are performed in a controlled sandbox with an emphasis on reproducibility. In particular, builds are not allowed to do network access within the sandbox. This allows build results to be shared between multiple developers.

BuildStream does not mandate that you use any particular automation tool, but builds can easily be automated using GitLab CI, Jenkins, BuildBot or with many other tools.

Concepts

The bst commandline tool operates within a project and a project contains one or more elements. The elements each describe how to build one component from its sources.

Projects can be small or large, and can depend on other BuildStream projects. We recommend keeping each BuildStream project in its own Git repository.

A project is marked by a project.conf file which sets up project-wide configuration options. The syntax used is YAML. The directory containing project.conf is considered to be the top-level project directory, and you must always run bst commands from this directory.

Elements usually live a subdirectory of the project named elements. Each element is represented on disk by a .bst file which again uses YAML syntax. An element has various attributes that authors can control, but BuildStream aims to support the "don't repeat yourself" principle and so provide sensible default values are provided that should fit many cases.

Here is an example element, which is taken from the Freedesktop SDK project and describes how to build the GNU Nano text editor:

kind: autotools
description: GNU nano

depends:
  - filename: bootstrap-import.bst
    type: build
  - filename: base/pkg-config.bst

sources:
  - kind: tar
    url: http://ftp.gnu.org/gnu/nano/nano-2.8.7.tar.xz
    ref: fbe31746958698d73c6726ee48ad8b0612697157961a2e9aaa83b4aa53d1165a

You can see at a glance that the upstream source tarball is referenced in the sources section. Note that the ref field here stores the SHA256 checksum of the contents of the file, which BuildStream uses to validate the tarball after fetching it. The source declares that is of kind: tar. BuildStream supports various other types of sources too. The list of built-in sources can be found here. Support for more types of source can be added by writing plugins.

There are no build instructions written here because GNU Nano uses a standard build system that BuildStream already supports. The first line (kind: autotools) instructs BuildStream that it should fill in the build instructions for this element using the autotools element plugin. There are various ways to override the defaults which are described in the "Composition" section of the reference manual.

The depends section is a little harder to read at a glance. The filenames listed here are other elements within the same project. One of these provides the pkg-config tool which is built in a similar way. The other provides a binary sysroot which is built as part of a separate project. BuildStream runs all builds inside a sandbox without any access to the host, and it doesn't provide any standard way of making standard programs like a UNIX Shell available in the sandbox. All build commands require various tools to function, at minimal a shell but usually also a C compiler and a whole array of other possible tools. It is up to the individual project to provide these, and so the first element of a project will normally be an import element that pulls prebuilt binaries from somewhere.

Building a Project

This section gives an example of using BuildStream to build an existing BuildStream project, using the Freedesktop SDK (1.8) project as our example. The BuildStream reference guide discusses this topic in more detail in the Using BuildStream section.

The Freedesktop SDK project exists primary to provide runtimes suitable for use with the Flatpak application distribution tool. It also allows building bootable VM images, which is what we will do here.

First you need to clone the project's Git repository, and enter the correct directory:

    git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk
    cd freedesktop-sdk/sdk

Type ls to list the files in this directory. You should see a file named project.conf, which defines the global project configuration. You will also see an elements/ subdirectory which contains many files each with the extension .bst. Each of these files defines one of the elements which will make up the build pipeline.

We are going to build a tiny Linux VM image using the element vm/minimal-vm-image-x86_64.bst.

TO WRITE:

• show a pipeline
• build it
• build VM image
• discuss caching
• run VM image

Authoring a Project

TO WRITE:

• Netsurf web browser as a Flatpak app, built with BuildStream.

• See https://mail.gnome.org/archives/buildstream-list/2018-February/msg00011.html

• See also: https://gitlab.com/BuildStream/buildstream-examples/merge_requests/1

Further reading

The BuildStream reference manual lives at https://buildstream.gitlab.io/buildstream/

There are more links available on the project wiki at https://wiki.gnome.org/Projects/BuildStream

Pointers for some specific use cases are documented below.

Producing virtual machine images

This can be done with the x86image plugin from the bst-external plugins package. This plugin should be considered a proof of concept rather than a production-ready tool in itself.

Producing Debian packages

This can be done with the dpkg_build and dpkg_deploy elements from the bst-external plugins package.

When doing this you will want to ensure that you import the correct version of Debian into the sandbox before building anything. You could use conditionals to create several variants of your pipeline that build packages against different versions of Debian, and other distributions such as Ubuntu which also use DPKG. Once more plugins are available which produce packages for other distributions you will be able to use the same mechanism to produce packages for these other distributions as well.