Bugzilla 4.4 upgrade

This page lists all the information we gathered during the upgrade from version 3.4 to 4.4.6 in early 2015.

Server setup

1. The production machine will be bugzilla-new.gnome.org

2. The directory where Bugzilla is hosted is /srv/http/bugzilla.gnome.org following Apache's standardization SOP

3. Members of the bugzilla group will be able to access the machine, run checksetup.pl and restart httpd

4. Upstream code is maintained at https://git.gnome.org/browse/bugzilla-gnome-org-upstream (hooks are in place for automatically updating the live instance)

5. Extensions and other GNOME customizations are available at https://git.gnome.org/browse/bugzilla-gnome-org-customizations (hooks are in place for automatically updating the live instance). Additionally symlinks are in place for Bugzilla to pick up changes to skins/contrib or template/en/custom accordingly.

6. For debugging purposes, hopefully output in data/errorlog is useful

7. Interaction between ftpadmin and Bugzilla will use XML-RPC!

bugzilla-gnome-org-{upstream,customizations} Git repositories

Note: the bugzilla-test VM became the production instance in February 2018, the only active repository branch is production which updates the production bugzilla.gnome.org instance. The information below is kept for historical reasons.

Two repositories have been set up in order to properly manage and keep track of changes deployed on both the upstream and customizations trees. The repositories have the following set of branches you should use when working with GNOME's Bugzilla instance, specifically:

1. master (bound to the test instance that will be deployed in the future for testing new features, GNOME themes or any other customization before it reaches production)
2. production (currently updating bugzilla-new)

The Git workflow as of today (in a pre-production-upgrade-to-4.4 environment) consists in:

1. Pushing to the production branch for directly testing the proposed changes and fixes

The Git workflow will change when the bugzilla-web (the current machine hosting Bugzilla 3.4) machine will be shutdown and bugzilla-new will take its place. In that case:

1. Pushing to the production branch should first require testing the proposed changes and fixes on the testing instance

2. The general rule will be pushing to the master branch, testing your changes on the testing machine and then merging / cherry-picking to production

3. Once the testing environment will be set up the production and master branches of both repositories should be in sync so that the same code is deployed on both the production and testing environments

Additional Git related details

Mozilla Bugzilla's upstream repository tracks both the extensions directory and the .htaccess file we do manage with Puppet. It's very important the production branch remains cleansed from both the resources listed above when a merge from that remote takes place. (usually when a new release is out and we want to upgrade our local code to it)

When merging a new release Git will complain with something like:

CONFLICT (modify/delete): extensions/Example/lib/WebService.pm deleted in HEAD and modified in release-4.4.7. Version release-4.4.7 of extensions/Example/lib/WebService.pm left in tree.

Don't worry as you will just have to launch your merge tool of choice and resolve the conflicts making sure Git is aware the files were deleted on purpose. That action usually corresponds to the Deleted (D) flag of the git mergetool of your choice.

Processing a new Bugzilla release upgrade through the bugzilla-gnome-org-upstream repository can be achieved this way:

git clone https://git.gnome.org/browse/bugzilla-gnome-org-upstream && cd bugzilla-gnome-org-upstream
git checkout 4.4
git remote add upstream https://github.com/bugzilla/bugzilla
git pull upstream release-4.4.9
git push

Once done, login to bugzilla.gnome.org, disable httpd and puppet and run checksetup.pl for the upgrade to happen.

Note: Before any upgrade, check whether the upstream diff includes changes to files (especially templates) that have been copied and customized in bugzilla-gnome-org-upstream. These upstream changes need to be incorporated. For templates this will also require updating TemplateOverride's sha256 sums (see below) to not make checksetup.pl fail.

Upgrade step by step

1. Upgrade machine to RHEL 6 (making sure the whole setup is reproduced through Puppet)

2. Add a custom page that reports Bugzilla is under maintenance plus a HTTP header (using mod_header) (CustomHeader: Bugzilla is being upgraded, ETA is ...)

3. Lock tables on the bugs database and dump it

4. Move data/ from the old instance to bugzilla-new

5. Update bugs ACL on the MySQL master server, update database password and host access

6. The bugs database is currently not being replicated on our MySQL slave thus import it over to the slave so that they are in sync again and the replica will magically be back when the upgrade will start

7. Adjust localconfig database details to match the ones previously updated
8. Update IP on gnome.org DNS zone and point it to bugzilla-new's IP
9. Update the bugzilla-test.gnome.org vhost, rename it bugzilla.gnome.org and make sure production certificates and variables are used
10. Run checksetup.pl and monitor the upgrade process (ETA is around 24 to 36 hours). If this step fails then see the Customizations section - there may be a clue about what is wrong
11. Change ftpadmin script in sysadmin-bin module, change BUGZILARPC to TRUE and remove the comment preceding the line. This code has NOT been tested with 4.4, only with 4.2. Commit the change!

Bugzilla administration tasks:

1. Disable NOTXIMIAN and GNOME1.x for new bugs in https://bugzilla.gnome.org/editvalues.cgi?field=resolution

After the upgrade:

1. Verify the functionality of the list-abusers script
2. Set up a test instance called bugzilla-test.gnome.org
3. Make sure the SSL certificate for bugzilla-test and bugzilla-attachments-test is used

4. Add Git hooks in place from the master branch of both -customizations and -upstream repositories

5. Add a Nagios check for the bugzilla-test SSL certificate
6. Decommission bugzilla-web (add unused IPs on the "Free IPs" area on the gnome.org DNS zone file, remove the VM from autostart, shut it down, remove it from the daily backups rotation)

Customizations

TemplateOverrides

This extension was added to aid us in tracking changes in templates we have overridden. When a change happens in original template then it might be necessary to backport the changes to our override. To detect the changes, this extension keeps sha256 sums of original templates in Digests.pm and compares the saved sum with the actual sum computed during checksetup.

During checksetup, this extension might bail out the process with one of following errors:

1. "Original ... (at ...) has changed since last checksetup. Please check if the changes should be backported to overridden templates and update the digest in %digests variable with ..."
   • Primary checksetup failure reason during upgrade and quite self-explanatory.
2. "Following overrides are missing their digests: ..."
   • Programming mistake.
   • Either add the sha256sums for listed files to Digests.pm or somebody already removed them from Digests.pm, but forgot the remove the overrides.
3. "Following files have digests, but no overrides for them exist: ..."
   • Programming mistake.
   • Either remove the sums from Digests.pm or somebody forgot to actually add the overrides to git.
4. "Following templates are not overriding anything: ..."
   • This should not happen when upgrading to new patch release. It might happen when doing some major update. It rather needs a case by case approach.
5. "Original template for ... not found - should not happen"
   • Sounds like a bug in extension.

ExtensionDependencies

This extension has two purposes:

1. Documentation
   • Each extension says which extensions it needs at runtime to work.
   • Dependencies can be circular, it doesn't matter at runtime.
2. Kind of sanity check
   • Just checking if needed extensions are loaded and enabled.

It is implemented by adding gnome_deps function to each extension, which return a list of extension names, example in DescribeUser extension

Possible errors:

1. "<EXT> has unsatisfied dependency on <DEP> extension - <DEP> is not available"

   • Either somebody forgot to remove the dependency on <DEP> from <EXT> or forgot to add <DEP> to git.

2. "<EXT> has unsatisfied dependency on <DEP> extension - <DEP> is disabled"

   • Either somebody forgot to remove the dependency on <DEP> from <EXT> or forgot to enable the extension - its enabled function returns false.

   • I'm not really sure if this one can happen actually.