Guide for New Release Team Members
Welcome to the release-team, and thanks for being willing to serve GNOME. This page will try to give you a quick overview of the tasks the release-team performs (or, preferably, delegates to the community and simply ensures the completion of), but note that tasks tend to evolve over time and having people spearhead new projects/tasks to improve the overall release process is always welcome.
Estimated Time Requirements
It's hard to give an exact feel for how much time it takes to be a release-team member, in particular because different members vary so much on how much time they put into it. Team members can always focus on certain tasks and sink a lot of time into them (revamping the release set scripts, pushing to build consensus around new policies, showstopper nagging, starting a new project like the patch squad, etc.).
However, as a rough estimate, the three most time-taking tasks ignoring special projects are (a) new module decisions, (b) approving freeze breaks, and (c) making releases. The last two are split up among all team members, but if you can pull your share in those three areas and keep up with the mailing lists, that should be about 95% of the time that you need. Read the specific tasks for general guidelines on how long each takes.
(See also https://mail.gnome.org/archives/release-team/2008-January/msg00035.html for a more detailed explanation of time requirements.)
Getting Started
To get started, simply subscribe to a few mailing lists and start reading a lot of background information.
Subscribe:
Discussions in Discourse: Go to Notifications Tags; under "Watched" add "release-team". (Discourse superseded our release-team mailing list in August 2020.)
Freeze Break requests in GitLab: Go to freeze-breaks; click "Star"; enable "Watch".
Security issues in GitLab: Go to security; click "Star"; enable "Watch".
(Optional) build issues in Git:ab: Go to gnome-build-meta; click "Star"; enable "Watch".
Subscribe to the following mailing lists if you aren't already:
devel-announce-list (d-a-l); shame on you if you aren't already subscribed to this.
desktop-devel-list (d-d-l); some feel d-d-l is somewhat time-taking to try to keep up with, but it is pretty important to try to do as a release-team member.
Join:
- #release-team on irc.gnome.org
Read:
- This page
General information for Maintainers (even if you are not a module maintainer yourself)
Requirements for Modules in the GNOME release sets.
It's also a good idea to familiarize yourself with the release-team pages.
Ask for:
ftpadmin permissions (accounts@gnome.org)
- set as developer for the release engineering group (any existing r-t member can do this)
You can also join the #release-team channel, where many things are happening -- including many non-release-team discussion!
Replacements
Membership is normally by invite and recommendation when one person leaves. Thus, you are probably on the team at the recommendation of the most recent member to leave, though team size is not fixed so this need not be the case. It is important to remember that new members are only added upon approval of the team, so when you leave and suggest a replacement (by email to r-t), please don't tell the replacement they have the job until the rest of the team has approved; also, be prepared to answer any questions about your nomination when you suggest a replacement. Please remember to update the membership page when you resign and when your replacement is approved.
Note that the GNOME Foundation board has the power to select release-team members and influence its decisions, but they usually don't fix stuff if it isn't broken. See also the membership page, which may have more info.
General Guidelines
- We tend to have a formal meeting once a cycle on IRC to discuss new module decisions mainly, but also to divide up tasks and discuss policy/procedure changes. We plan the meeting by email on r-t. We will likely move to twice per cycle (also meeting just after the big .0 release to discuss the next cycle). We try to keep meetings under an hour, though we haven't been particularly successful.
- The release-team subscribes to the "just do it" philosophy for tasks that should be underway or finished but for whatever reason aren't. Thus, if you see one of the standard tasks hasn't been done and no one is doing it (e.g. sending out a reminder, approving or rejecting a freeze break, ensuring release notes are written, making a release, etc.), feel free to just do it.
- We try to work by community consensus. Thus if you have a new proposal or want to modify existing procedure (for example, to change the requirements for allowed external dependencies), you must first try to build consensus in the community (on d-d-l) and verify with the release-team that the community consensus is behind the change before you finally start enforcing the change. Note that anyone from the community can propose changes to the process, the release-team's job is really just to discover community consensus about the release cycle and enforce it.
- Important release-team is up to date with the whole of GNOME. No matter if something is communicated via blogs/planet/some bug/desktop-devel-list/random mailing list/some IRC channel. Not everyone has to follow everything. Still, as a whole, we should be aware what is going on so we can act on it. By the time someone tries to raise something to release-team, ideally we should've already thought of a solution. As such, important to be able to deal with a lot of information.
Tasks we are responsible to ensure someone does
The release-team has responsibility for a broad range of tasks. These include:
- Creating a schedule
- Sending out periodic reminders
- Making Releases
- Freeze Break Approvals/Rejections
- Ensuring release notes are written
- Defining the module lists for release sets
- Ensure Roadmapping is done
- Monitoring Quality
Creating a schedule
This task is only needed once every 6 months, and is typically done by one person. It might take a few hours to review old schedules (e.g. 2.17), old proposals (e.g. 2.9 cycle), and upcoming events to get a good feel for what is needed. But then it's pretty easy to create a proposal. Using the schedule creation releng script is recommended. Once created, it can be proposed to the community by emailing d-d-l. Once agreed upon, it should be put on the wiki and announced at d-a-l.
It'd be really nice to get some historical information on timings, such as Mark researched in his 2.10 proposal (e.g. number of weeks between freezes for each cycle, total number of weeks in each cycle, and maybe the reasons we changed over time) and put it up online to make future schedules easier to create.
Sending out periodic reminders
Someone should send a reminder to d-a-l about upcoming tarball due dates and freezes. Other occasional reminders (e.g. module proposal process) are helpful too. It is also important to keep the summarized Schedule up-to-date. Only takes a few minutes, typically old reminders can be used nearly verbatim (be careful to update the relevant information if you copy-and-paste from an old announcement, though!)
Making Releases
I think we average a point release every two-three weeks, and each can be a bit of a bear to make. Details on this process are on our Making A Release guide. About 98% of the time required for making a release is used in smoketesting the release (i.e. building and doing basic testing of the release). It can take an hour or so to create a basic moduleset to build with all the appropriate tarballs, and then it typically takes several hours to build all the modules in a release (though most of this is wait time for the computer to download/compile/link so you can do other work during this time as long as you remember to check on it occasionally). There's unfortunately a big unknown factor to it: any build problems mean you need to hunt down the relevant maintainers and ask for fixes. The GARNOME people tend to be really helpful in terms of finding issues and tracking down maintainers, though, and will often beat you to the task, so that tends to lighten the burden.
We try to split up this task, so each team member only needs to do one or two releases per cycle.
You can try to learn this process without fear of messing anything up, since 98% of the time is smoketesting and only the remaining 2% (running scripts on the server) make anything official. To do so, become familiar with using jhbuild, look at one of the recent past generated modulesets, and try following the "on testing machine" directions (other than the 'svn commit' step!) from the Making a Release guide.
Note that you need FTP access for uploading the modulesets. You can apply for that by sending an email to the accounts team.
Freeze Break Approvals/Rejections
Read over the explanations of what each freeze means and the process on requesting freeze breaks. Approving/rejecting freeze breaks might sound scary at first when you realize you are not an expert in the module requesting a freeze break, but remember that it is impossible for the release-team to have expertise in everything (we have over 100 modules). Thus, our job is not to ensure technical correctness but more to (a) make sure that maintainers have done due diligence weighing the pros and cons of breaking the freeze, (b) to ensure some level of uniformity among maintainers as to what is considered freeze break worthy, and (c) help slow down the rate of major change (and eventually even minor changes) in the lead up to a release.
Approving and Rejecting freeze break requests is an inexact science, and is really a gut feeling call by release-team members. Some general guidelines and reminders (much of this taken is from Murray and Mark's advice to me in Feb' 2005 when I asked them how this worked):
- Approvals from two different team members means the request is approved. Less than two approvals mean it is not approved. Rejections are basically counted the same as a non-response from a release-team member (and vice-versa), except that the reasoning for the rejection might influence other release-team member's decisions.
- We're trying to slow down the rate of major change in the lead up to a release. We start by slowing down the rate of large scale changes and changes that affect others and coming closer to the release we lock down all changes.
- We frown upon changes that might cause regressions, changes that have impact on other parts of the release, people rushing to get things in at the last minute without giving themselves a chance to get it right etc. etc.
- We try to avoid making technical decisions; those really are for the maintainers and the community to decide.
- We try to weigh the impact of the bug as well as the likelihood for regressions. If the patch is small and fixes something important, it's less likely to cause problems elsewhere (and if it does, it will be more readily fixed). If it's a trivial bug affecting almost no one that is being fixed, it should likely be rejected even if the change is Obviously Correct. If it's a highly repeatable crasher affecting everyone, then even a large patch isn't going to make things much worse (though such a case worries us a lot during e.g. Hard Code Freeze).
- We also look over the patch itself to make sure we fully understand what's being proposed and its impact and we try and catch anything obvious in the patch the maintainer has missed. E.g. if you spotted a leak, it would be "tut, tut, you're trying to rush this in"
- We're trying to teach maintainers to make this decision on their own. Ideally, the maintainer only asks after weighing up the pros and cons and we're just a rubber stamp. So, we look for sane, considered requests from sane, considered maintainers.
- It is not uncommon for release-team members to wait a day hoping that someone else on the release-team feels more qualified to respond to a specific request. If no one else responds, however, you should definitely feel that your opinion is better than nothing.
- Special cases may need special consideration (e.g. language bindings main job is to try to keep up with new APIs, though it can be difficult, so it may make sense to cut them a bit of slack in that area...of course, not blanket "you can add APIs whenever you want" slack)
Ensuring release notes are written
We try to delegate this to the Engagement team. This typically means contacting them and following up to ensure someone is working on it.
Defining the module lists for release sets
This basically means following the long emails on d-d-l about proposed modules and trying to get community consensus on the issues. Make sure to read up on the module proposal rules and judgement criteria. This task also involves a big debate among release-team members in our semi-annual meeting. Don't forget to try out the modules as well, and try to check up on whether they are following the release rules and schedule.
Ensure Roadmapping is done
This is a relatively new area, but we basically want to ensure that the RoadMap Gang collects a set of plans (near, intermediate, and far future) and creates a RoadMap out of it. At a basic level, this is just bottom-up planning, but we hope that it will help us with release notes, increase the awareness about future steps of the project inside and outside the community, and get us to discuss more often where we want to go.
Monitoring Quality
This is a somewhat fuzzy requirement of the team, which often means we don't do it as well as we'd like. We need to generally have a feel of the quality of the modules in the release set, and take measures to try to improve any problematic areas. This can mean virtually anything, but concrete steps often involve talking with the bugsquad and downstream distributors, basic build nagging (trying to delegate to the new BuildBrigade), and showstopper bugs nagging (trying to delegate to the Bugsquad), and even starting IRC hackfests to try to fix specific issues.
Tasks that it would be nice to tackle
Find ways to make the relase-team feel more inclusive and increase community participation (see Mark's old suggestion for some basics; we've moved in that direction but need to move further)
- Find ways to improve communication, particularly with distro packagers. (Our communication is such that some may feel it necessary to remain an inactive member of the team just to be able to follow the mailing list; one past member explicitly stated this concern. We should fix that.)
- More thorough monitoring for freeze breaks (rather than the haphazard hope-someone-notices-and-tells-us procedure we use now)
- Monitoring developer modules for whether they document new API
- Automated API/ABI checks (feature, ui, and string would be great too, though they might be harder?)