BuildStream performance

Introduction

This page should be used to track ongoing work to monitor and improve the performance of BuildStream.

Work should be categorised so that all bugs relating to one aspect of BuildStream's functionality are discussed together.

Generally useful links:

• Buildstream benchmarking framework, and its automated test pipelines.

• Pyflame produces pretty flamegraphs.

• MR!185 "Make plugins debuggable with pdb etc." could make profiling easier as it prevents BuildStream from forking off subprocesses to run jobs.

• All issues carrying label Optimization

1. Startup time

Time spent starting the Python interpreter and importing the 'buildstream' module. A good testcase is bst --help.

Fixed issues:

• #172 pkg_resources module causes a delay of between 0 and many seconds every time `bst` starts up

2. Pipeline initialization

Time to load the .bst files, calculate cache keys, contact remote caches and check availability in the local cache. A good testcase is `bst show`.

Fixed issues:

• #173 Element state/cache key handling

3. Staging area construction

The 'Staging dependencies' task that occurs as part of bst shell|checkout|build. The easiest way to test is bst shell on an element that has no integration commands.

Profiling with Pyflame shows Python path/string processing operations high on the profile.

Caching manifests (#82) could save time here.

Fixed issues:

• #174 Python os.realpath() function is too slow

There has been discussion of using sandboxfs to construct the staging area more quickly.

4. Integration and build commands

Mostly the speed of the FUSE sandbox that the integration commands run on top of.

5. Automated tests / CI

Speed of BuildStream's test suite and integration test suite, plus CI speed for projects using BuildStream.

Fixes:

• Using `bst checkout --hardlinks` in test suite

6. Other

Speed of `bst checkout`

OSTree optionally calls fsync() while doing a checkout, disabling this seems to speed things up in some cases.