Migrating from Intltool to Gettext

It used to be that Gettext could only translate strings in source code, and some limited other formats. Since about version 0.18, Gettext has been adding support for a lot more file formats that are used in modern GNOME desktop applications. This has slowly been making Intltool obsolete, which was invented to bridge that gap between what Gettext supported and what applications needed.

This is a migration guide for moving your Intltool-using project to pure Gettext.

Prerequisites

First make sure that you are prepared to support a recent enough version of Gettext. Here are the versions of Gettext in which support was added for files that Intltool was previously used for:

  • Glade 3.x / GtkBuilder XML - 0.18.3

  • .desktop file - 0.19
  • GSettings schema - 0.19

  • AppData - 0.19.6

  • Custom XML formats with ITS rules - 0.19.7

(This information taken from Gettext's release announcements, http://savannah.gnu.org/news/?group_id=425)

Conversely, these are the Gettext versions packaged by major Linux distributions at the time of writing:

  • Debian Jessie (Stable) - 0.19.3
  • Fedora 25 - 0.19.8.1
  • Ubuntu 16.04 LTS - 0.19.7
  • Ubuntu 14.04 LTS - 0.18.3
  • openSUSE 13.2 - 0.19.2
  • Linux Mint - 0.18.3

If you're working on a core GNOME package, you don't have to be concerned, but third-party application developers may want to check that they are not making their package impossible to build on a platform that they still want to support.

Remove Intltool

Start by removing mentions of Intltool from anywhere throughout your whole build chain:

  • In autogen.sh, make sure the "intltoolize" program is not invoked.

  • In configure.ac, remove any use of the IT_PROG_INTLTOOL macro.

  • In Makefile.am, remove any mention of intltool-extract.in, intltool-merge.in, intltool-update.in, and m4/intltool.m4; these might be listed in EXTRA_DIST or DISTCLEANFILES or MAINTAINERCLEANFILES.

  • Also in Makefile.am, remove @INTLTOOL_DESKTOP_RULE@ and @INTLTOOL_XML_RULE@.

  • If you have any distribution-specific packaging, such as RPM spec files, remove any build dependency on Intltool from there as well.

Upgrade to a modern Gettext

Gettext has the peculiarity that if you ask for a particular version of it in your build system, then you get exactly that version, and not any later version. So depending on what platforms you want to support (see above), you may want to pick the highest version that's common between all your supported platforms, even if you don't need the latest functionality from that version. Otherwise, you wouldn't be able to take advantage of as many bug fixes as possible. For the rest of this document, we'll assume that 0.19.7 is the version you're upgrading to.

  • In configure.ac, edit any use of AM_GNU_GETTEXT_VERSION to refer to your chosen version: AM_GNU_GETTEXT_VERSION([0.19.7]).

  • If you have anything referring to AM_GNU_GLIB_GETTEXT, that's very old; replace it with AM_GNU_GETTEXT.

  • In po/POTFILES.in, remove any annotations in square brackets like [type: gettext/glade].

    • This notation is Intltool-specific.

You will also need a po/Makevars file, if you don't have one yet. Many Intltool-using programs didn't. A template for this file called po/Makevars.template is installed by modern versions of Gettext's setup program "autopoint".

  • Run autopoint to create po/Makevars.template.

  • Optional: consider adding po/Makevars.template to your .gitignore file.
  • Edit po/Makevars.template to your liking; the comments should be self-explanatory.

  • On the XGETTEXT_OPTIONS line in po/Makevars, the full GLib-approved list of options would be something like --from-code=UTF-8 --keyword=_ --keyword=N_ --keyword=C_:1c,2 --keyword=NC_:1c,2 --keyword=g_dngettext:2,3 --add-comments and customize to reflect whatever you use in your sources.

    • Make sure to include --from-code=UTF-8 if your source strings are encoded in UTF-8, which they probably are.

  • Rename po/Makevars.template to po/Makevars.

Add rules for merging translations back into non-source files

Some files have translatable strings extracted at build time, but the translations themselves are fetched at runtime; these include source code, GtkBuilder XML files, and GSettings schema files. Other files need to have translations merged back into them at build time, before they are installed: these include .desktop files and AppData files. (If you don't have any files that need merging, then you can skip this section.) As an example, we'll use a .desktop file called com.example.Foo.desktop.

The way it works is that Gettext extract translatable strings from a .desktop.in file into your .pot file. The translators translate those strings and put them in the .po files, and so your build process needs to merge the translated strings back into the .desktop file. You commit the .desktop.in file to version control, and not the .desktop file. This is how Intltool worked as well internally, but with Gettext it's a transparent make rule.

  • In Makefile.am, make sure com.example.Foo.desktop.in is listed in EXTRA_DIST or an Automake variable with a dist_ prefix, and com.example.Foo.desktop is not.

  • Also in Makefile.am, make sure com.example.Foo.desktop is listed in CLEANFILES.

  • In po/POTFILES.in, make sure com.example.Foo.desktop is listed.
  • Add the following make rule: 

com.example.Foo.desktop: com.example.Foo.desktop.in
    $(AM_V_GEN)$(MSGFMT) --desktop --template $< -d $(top_srcdir)/po -o $@
  • (with the usual caveat that a tab character is required in make rules, so beware when copy-pasting)
  • Make sure the .desktop file is built before the po/ directory.

    • In your toplevel Makefile.am, ensure that your SUBDIRS variable lists po after the directory that the .desktop file is in. (If you're using non-recursive Make, then that would look like this: SUBDIRS = . po).

  • Make sure to remove any Intltool-specific notation from your .desktop or XML files, such as keys or XML attributes that start with underscores.

For an XML format such as AppData, you would replace --desktop by --xml in the make rule above.

Check your work

The upshot of all this should be that you get exactly the same results with one less dependency. So, check your .pot file by first making a backup copy and then updating it: run make -C po mypotfile.pot-update. (That is, if your .pot file is named po.mypotfile.pot, append -update to the end of it.) Diff the updated .pot file against the backup copy; you may see formatting differences, but you should not see any strings being removed.

Addendum: Translate other XML formats

These instructions are outdated and need to be rewritten to use https://www.gnu.org/software/gettext/manual/html_node/Preparing-ITS-Rules.html instead of ITS Tool.

There are still other, more obscure XML formats that Gettext does not handle. For these cases, we have http://itstool.org/.

ITS Tool extracts translatable strings from XML files using ITS rules, which are a W3C standard; Intltool did the same thing, but using an ad-hoc notation which made some XML files invalid.

(It's planned for Gettext to support ITS rules natively in the future, in which case this section will become obsolete. This section does contain a few annoying hoops to jump through, and it would be great if those went away with Gettext ITS rule support.)

For this example, we'll use a MIME type description file called foo-mime.xml. Following this example should make it possible for people to still build your package even if they don't have ITS Tool installed; it would only be a hard requirement for building from a fresh checkout from version control.

  • In configure.ac, check for ITS Tool: add AC_PATH_PROG([ITSTOOL], [itstool], [notfound]).

    • The last argument ensures that the build doesn't fail if it isn't found.

  • In Makefile.am, make sure that foo-mime.xml, foo-mime.xml.in, and foo-mime.pot are all listed in EXTRA_DIST or in Automake variables with dist_ prefixes.

    • Particularly the latter two, foo-mime.xml.in and foo-mime.pot, should not be installed by "make install", so dist_noinst_DATA would be a good place to list them.

  • In po/POTFILES.in, add foo-mime.pot.
    • (This imports all strings in the "sub" .pot file into the main one.)
  • Add the following rules to Makefile.am: 

foo-mime.pot: foo-mime.xml.in
    $(AM_V_GEN)$(ITSTOOL) -o $@ $<

foo-mime.xml: foo-mime.xml.in
    $(AM_V_GEN)$(ITSTOOL) -j $< -o $@ $(top_srcdir)/po/*.gmo
  • (with the usual caveat that a tab character is required in make rules, so beware when copy-pasting)

  • In Makefile.am, add foo-mime.xml and foo-mime.pot to MAINTAINERCLEANFILES.

  • Edit foo-mime.xml.in and remove the underscore notation that Intltool required.

    • In the case of MIME type descriptions, that means changing all <_comment> elements to <comment>. Now the .xml.in file should be a valid MIME type description by itself.

  • Commit foo-mime.pot to your version control system.
    • (It's unfortunate that you have to check in a generated file, and this will hopefully be unnecessary after Gettext grows native support for ITS rules.)
  • In your toplevel Makefile.am, make sure the po/ directory is processed before the directory that contains the MIME type file.

Now, build and check that:

  • All the translatable strings from your XML file made it into foo-mime.pot;

  • No strings that should not be translated were put into foo-mime.pot;

  • The output .xml file is correctly generated;
  • All the strings from foo-mime.pot made it into the main .pot file.

In case the first two conditions are not satisfied, then ITS Tool's default ITS rules were not enough, and you need to write a custom ITS rule. The default rule is to translate text inside of elements, and skip text inside attribute values. For MIME types, this works fine; only the <comment> elements need to be translated, and the translations become <comment xml:lang="en"> elements as they are merged back in.

A custom ITS rule set will look basically like this: 

<its:rules xmlns:its="http://www.w3.org/2005/11/its" version="1.0">
  <its:translateRule translate="no" selector="//some-element"/>
  <its:translateRule translate="yes" selector="//@some-attribute"/>
  <!-- Add more as needed -->
</its:rules>

That is, you can tell it to ignore text inside of some elements, and you can tell it not to ignore text inside of some attributes. You can also use any other XPath expressions inside the selector attributes, so it can be as complicated as you need it to be.

Tell ITS Tool to use your custom rules by adding an -i myrules.its flag to the $(ITSTOOL) invocation in the first make rule above (the rule that builds the .pot file from the .xml.in file.) You can see an example of complicated ITS rules in action, in the data/ directory of the gtksourceview project.

MigratingFromIntltoolToGettext (last edited 2017-09-11 07:23:06 by PiotrDrag)