Artifact Cache Sharing

BuildStream currently uses ostree for the artifact cache, which is responsible for archiving the result of each individual build.

Currently we do not have any settings or code for pushing build results to a remote artifact cache or pulling artifacts from there.

Implement artifact cache pushing

We expect to be able to accomplish pushing to a remotely hosted ostree repository using ostree-push. This should allow the remote to configure (in any way that it wants to) who has the right to push to the remote artifact cache by authenticating users by their public ssh keys.

  • Configuration needs to be added to BuildStream to configure the remote artifact server in a user config file

  • The artifact cache needs to gain some API to push a locally created artifact to a remote cache
  • The scheduler needs to gain a new Queue for the purpose of pushing artifacts to a remote, which can be parallelized with other build tasks (local builds need not wait on publishing artifacts which already exist locally).

Status: Done

Implement artifact cache fetching

Using the same configurations above, the artifact cache needs to gain a fetch semantic for attempting to download an existing artifact by it's cache key from the remote server.

Here, instead of implementing a new scheduler Queue, we need the Element's assemble method to first make an attempt at obtaining a remote artifact and download it, only building the artifact locally if a remote artifact could not be downloaded.

Bonus points are awarded for detecting network timeouts and tracking that in some local state, so that we do not waste time trying to download from an unreachable artifact server.

Note: It will be desirable to attempt fetching artifacts in a separate scheduler queue as this can potentially make the pipeline more performant, but this can be attempted as a separate enhancement activity.

Status: Done

Document artifact server

The artifact server should ideally not be an implementation requiring code, the only requirement really is that one is able to host an ostree repository, ideally with ostree-push capability.

An explanation, possibly including a small and simple HTTP server written in python as an example (instead of referring to big scary apache), should be added to the BuildStream documentation to ensure this process is clear.

Status: Done

Projects/BuildStream/Roadmaps/Initial2017/ArtifactSharing (last edited 2018-01-03 11:42:05 by SamThursfield)