Translating user interface of SumatraPDF
SumatraPDF is the best PDF/eBook/Comic Book viewer for Windows. It’s small, fast, full of features, free and open-source.
It became popular enough that it made sense to translate the UI for non-English users. Currently we support 72 languages.
This article describes how I designed and implemented a translation system in SumatraPDF, a native win32 C++ Windows application.
Hard things about translating the UI
There are 2 hard things about translating an application
- code for translation system (extracting strings to translate, translate strings from English to user’s language)
- translating them into many languages
Currently there are 381 strings in SumatraPDF subject to translation. It’s important that the system requires the least amount of effort when adding new strings to translate.
Every string that needs to be translated is marked in .cpp or .h file with one of two macros:
_TRA("Rename")_TRN("Open")
I have a script that extracts those strings from source files. Mine is written in Go but it could just as well be Python or JavaScript. It’s a simple regex job.
_TR stands for “translation”.
_TRA(s) expands into const char* trans::GetTranslation(const char* str) function which returns str translated to current UI language.
We auto-detect language at startup based on Windows settings and allow the user to explicitly set UI language.
For English we just return the original string.
If a string to be translated is e.g. a part of const char* array[], we can’t use trans::GetTranslation().
For cases like that we have _TRN() which expands to English string. We have to write code to translate it at some point.
Adding new strings is therefore as simple as wrapping them in _TRA() or _TRN() macros.
Translating strings into many languages
Now that we’ve extracted strings to be translated, we need to translate them into 72 languages.
SumatraPDF is a free, open-source program. I don’t have a budget to hire translators. I don’t have a budget, period.
The only option was to get help from SumatraPDF users.
It was vital to make it very easy for users to send me translations. I didn’t want to ask them, for example, to download some translation software.
Design and implementation of AppTranslator web app
I designed it to be generic but I don’t think anyone else is using it.
- 4k lines of Go server code
- 451 lines of html code
- a single dependency: bootstrap CSS framework (the project is old)
It’s simple because I don’t want to spend a lot of time writing translation software. It’s just a side project in service of the goal of translating SumatraPDF.
Login is exclusively via GitHub.
It doesn’t even use a database. Like in Redis, changes are stored as a series of operations in an append-only log. We keep the whole state in memory and re-create it from the log at startup.
Main operation is translate a string from English to language X represented as [kOpTranslation, english string, language, translation, user who provided translation].
When user provides a translation in the web UI, we send an API call to the server which appends the translation operation to the log.
Simple and reliable.
Because the code is written in Go, it’s very fast and memory efficient. When running it uses mere megabytes of RAM. It can comfortably run on the smallest 256 MB VPS server.
I backup the log to S3 so if the server ever fails, I can re-install the program on a new server and re-download the translations from S3.
I provide RSS feed for each language so that people who provide translations can monitor for new strings to be translated.
Sending strings for translation and receiving translations
So I have a web app for collecting translations and a script that extracts strings to be translated from source code.
How do they connect?
AppTranslator has an API for submitting the current set of strings to be translated in the simplest possible format: a line for each string (I ensure there are no newlines in the string itself by escaping them with \n)
API is password protected because only I can submit the strings.
The server compares the strings sent with the current set and records a difference in the log.
It also sends a response with translations. Again the simplest possible format:
AppTranslator: SumatraPDF
651b739d7fa110911f25563c933f42b1d37590f8
:%s annotation. Ctrl+click to edit.
am:%s մեկնաբանություն: Ctrl+քլիք՝ խմբագրելու համար:
ar:ملاحظة %s. اضغط Ctrl للتحرير.
az:Qeyd %s. Düzəliş etmək üçün Ctrl+düyməyə basın.
As you can see:
- a string to translate is on a line starting with :
- is followed by translations of that strings in the format:
${lang}: ${translation}
An optimization: 651b739d7fa110911f25563c933f42b1d37590f8 is a hash of this response. If I submit this hash with my request and translations didn’t change on the server, the response is empty.
Implementing C++ part of translation system
So now I have a text file with translation downloaded from the server. How do I get a translation in my C++ code?
As with everything in SumatraPDF, I try to do things in a simple and efficient way.
The whole Translation.cpp is only 239 lines of code.
The core of translation system is const char* trans::GetTranslation(const char* s); function.
I embed the translations in exact the same format as received from AppTranslator in the executable as data file in resources.
If the UI language is English, we do nothing. trans::GetTranslation() returns its argument.
When we switch the language, we load the translations from resources and build an index:
- an array of English strings
- an array of corresponding translations
Both arrays use my own
StrVec class optimized for storing an array of strings.
To find a translation we scan the first array to find an index of the string and return translation from the second array, at the same index.
Linear scan seems like it would be slow but it isn’t.
Resizing dialogs
I have a few dialogs defined in SumatraPDF.rc file.
The problem with dialogs is that position of UI elements is fixed.
A translated string will almost certainly have a different size than the English string which will mess up fixed layout.
Thankfully someone wrote DialogSizer that smartly resizes dialogs and solves this problem.
The evolution of a solution
No AppTranslator
My initial implementation was simpler. I didn’t yet have AppTranslator so I stored the strings in a text file in repository in the same format as what I described above.
People would download it, make changes using a text editor and send me the file via email which I would then checkin.
It worked for a while but it became worse over time. More strings, more languages created more work for me to manually manage e-mail submissions.
I decided to automate the process.
Code generation
My first implementation of C++ side used code generation instead of embedding the text file in resources.
My Go script would generate C++ source code files with static const char* [] arrays.
This worked well but I decided to improve it further by making the code use the text file with translations embedded in the app.
The main motivation for the change was to open a possibility of downloading latest translations from the server to fix the problem of translations not being all ready when I build the release executable.
I haven’t done that yet but it’s now easier to implement given that the format of strings embedded in the exe is the same as the one I can download from AppTranslator.
Only utf-8
SumatraPDF started by using both WCHAR* Unicode strings and char* utf8 strings.
For that reason the translation system had to support returning translation in both WCHAR* and char* version.
Over time I refactored the code to use mostly utf8 and at some point I no longer needed to support WCHAR* version.
That made the code even smaller and reduced memory usage.
The experience
I’m happy how things turned out.
AppTranslator proved to be reliable and hassle free. It runs for many years now and collected 35440 string translations from users.
I automated everything so that all I need to do is to periodically re-run the script that extracts strings from source code, uploads them to AppTranslator and downloads latest translations.
One problem is that translations are not always ready in time for release so I make a release and then people start translating strings added since last release.
I’ve considered downloading the latest translations from the server, in addition to embedding them in an executable at the time of building the app.
Would I do the same today?
While AppTranslator is reliable and doesn’t require on-going work, it would be better to not have to run a server at all.
The world has changed since I started SumatraPDF.
Namely: people are comfortable using GitHub and you can edit files directly in GitHub UI. It’s not a great experience but it works.
One option would be to generate a translation text file for each language, in this format:
:first untranslated string
:second untranslated string
:first translated string
translation of first string
:second translated string
translation of second string
Untranslated strings are listed at the top, to make it easier to find.
A link would send a translator directly to edit this file in GitHub UI.
When translator saves translations, it creates a PR for me to review and merge.
The roads not taken
But why did you re-invent everything? You should do X instead.
All other X that I know about suck.
Using per-language .rc resource files
Traditional way of localizing / translating Window GUI apps is to store all strings and dialog definitions in an .rc file. Each language gets its own .rc file (or files) and the program picks the right resource based on a language.
This doesn’t solve the 2 hard problems:
- having an easy way to add strings for translations
- having an easy way for users to provide translations
XML horror show
There was a dark time when the world was under the iron grip of XML fanaticism.
Everything had to be an XML file even when it was the worst possible solution for the problem.
XML doesn’t solve the 2 hard problems and a string storage format is an absolute nightmare for human editing.
GNU gettext
There’s a C library gettext that uses .po files.
This is much saner solution than XML horror show. .po files are relatively simple text format. The code is already written.
Warning: tooting my own horn.
My format is better. It’s easier for people to edit, it’s easier to write code to parse it.
This looks like many times more than 239 lines of code.
Ok, gettext probably does a bit more than my code, but clearly nothing than I need.
It also doesn’t solve the 2 hard problems. I would still have to write code to extract strings from source code and build a way to allow users to translate them easily.
