Overhaul translation deployment

README.md

@@ -45,6 +45,94 @@ To use linuxdeploy-plugin-standalone, download the official AppImage, make it ex
 linuxdeploy-plugin-qt will look for Qt libraries in the library directory `usr/lib/` and deploy the Qt plugins and other resources for these. This means that if linuxdeploy or another tool haven't been run on the AppDir yet, i.e., no Qt libraries have been deployed yet, linuxdeploy-plugin-qt won't be able to recognize which plugins and resources have to be deployed, and will return an error.
 
 
+### Translations
+#### Qt Translations
+Translation of Qt libraries (usually accessible at `/usr/share/qt{5,6}/translations/`) is split into the following categories:
+
+| Category                        | CLI enable                  | CLI disable                    | Env variable                     |
+| ------------------------------- | --------------------------- | ------------------------------ | -------------------------------- |
+| Individual library translations | `--individual-translations` | `--no-individual-translations` | `TRANSLATIONS_INDIVIDUAL=YES/NO` |
+| Merged library translations     | `--merged-translations`     | `--no-merged-translations`     | `TRANSLATIONS_MERGED=YES/NO`     |
+
+Individual library translations copy over individual `.qm` files into standard translation directory (`<AppDir>/share/translations`, retrievable by calling `QLibraryInfo::path(QLibraryInfo::TranslationsPath)` from within program). For example, if the program is using Core and Multimedia modules, `qtbase_cs.qm`, `qtmultimedia_cs.qm`, `qtbase_de.qm`, `qtmultimedia_de.qm`... will get copied over.
+
+Merged library translations will produce a `qt_<lang>.qm` file into standard translation directory. This is consistent with for example how `windeployqt.exe` Qt official deployer deploys translations.
+
+By default, all available translations matching the Qt libraries used are deployed. The list of deployed languages can be restricted by supplying a comma separated list of language codes (codes matching filenames in `/usr/share/qt{5,6}/translations/`) with `--qt-languages` or `$TRANSLATION_LANGUAGES`.
+
+Note that `--qt-languages` and `$TRANSLATION_LANGUAGES` only affect Qt's own translations. Program provided translations are not affected.
+
+#### App Translations
+App translation handling is program specific. For best results, make sure to configure the build system of the program to be deployed [as described in AppImage documentation](https://docs.appimage.org/packaging-guide/from-source/native-binaries.html#using-the-build-system-to-build-the-basic-appdir) when deploying from source.
+
+It is best to test translations before distributing the AppImage. This can be done by
+
+1. Making sure the locale to be tested is loaded on glibc Linux
+
+   Here are some resources on the topic: [Arch Linux (Arch Wiki)](https://wiki.archlinux.org/title/Locale), [Debian](https://wiki.debian.org/Locale), [Alpine](https://wiki.alpinelinux.org/wiki/Locale), [Void Linux](https://docs.voidlinux.org/config/locales.html), [Gentoo](https://wiki.gentoo.org/wiki/Localization/Guide).
+2. Override the `LC_MESSAGES` or `LANG` variable while executing the appimage from a terminal by either prepending `<VAR>=<LANG> ./myappimage.AppImage`:
+
+   ```
+   LC_MESSAGES=cs_CZ.UTF-8 ./myappimage-x86_64.AppImage
+   ```
+
+   or by issuing `export` before running the AppImage:
+
+   ```
+   export LC_MESSAGES=cs_CZ.UTF-8
+   ./myappimage-x86_64.AppImage
+   ```
+
+Make sure that the tested program does indeed provide translations for the overridden locale.
+
+linuxdeploy-plugin-qt provides a flag to add a symlink to program translations to `TranslationsPath` (to `<AppDir>/usr/translations`):
+
+| Category                        | CLI enable                   | CLI disable                     | Env variable                      |
+| ------------------------------- | ---------------------------- | ------------------------------- | --------------------------------- |
+| Symlink app translations        | `--app-symlink-translations` | `--no-app-symlink-translations` | `TRANSLATIONS_SYMLINK_APP=YES/NO` |
+
+#### Recommendations
+linuxdeploy-plugin-qt enables individual library translations and symlink app translations and disabled merged library translations by default for backwards compatibility.
+
+If the program was written with for example with `windeployqt.exe` in mind, merged library translations and symlink app translations should do the job.
+
+You can try enabling and disabling these flags to see which are required for the program being packaged to load translations.
+
+#### Recommendations to application developers
+Load Qt translations with
+
+```cpp
+translator.load("qt_" + language, QLibraryInfo::path(QLibraryInfo::TranslationsPath));
+```
+
+or
+
+```cpp
+translator.load(QLocale::system(), "qt", "_", QLibraryInfo::path(QLibraryInfo::TranslationsPath));
+```
+
+This should work with linux distro packages, `windeployqt` deployed `.exe` files and with linuxdeploy-plugin-qt.
+
+For program translations, the easiest way of distributing translations in regard to deploying it (with linuxdeploy-plugin-qt or other tools) is to bundle them into the executable as a [Qt resource](https://doc.qt.io/qt-6/resources.html). The rest of this section concerns the more complicated solution, which is installing compiled translations alongside the executable.
+
+For program translations, you have the freedom of choosing translation directory, but be aware that the [recommended building process](https://docs.appimage.org/packaging-guide/from-source/native-binaries.html#using-the-build-system-to-build-the-basic-appdir) uses prefix of `/usr` and `DESTDIR` to install program files into AppDir.
+
+If you try to load translations from the directory your build system thinks it installs them into at configure time, it will try to load translations from host, not from the appimage.
+
+One solution is to load directories relative to `QLibraryInfo::path(QLibraryInfo::PrefixPath)` instead of `/usr` or build system prefix. See [standard paths](#standard-paths) for a list of standard paths recognized by Qt.
+
+Another solution is to load from path relative to `QCoreApplication::applicationDirPath()`.
+
+It is wise to try several directories for loading app translations. Some reasonable picks include:
+
+```cpp
+// Good for windeployqt and macdeployqt
+QLibraryInfo::path(QLibraryInfo::TranslationsPath)
+// windeployqt-esque
+QCoreApplication::applicationDirPath() + "/translations"
+// Not Windows friendly, good in combination with some other dirs
+QLibraryInfo::path(QLibraryInfo::PrefixPath) + "/share/" + QCoreApplication::applicationName() + "/translations"
+```
 
 ### Environment variables
 
@@ -62,6 +150,34 @@ Just like all linuxdeploy plugins, the Qt plugin's behavior can be configured so
 - `$EXTRA_PLATFORM_PLUGINS=platformA;platformB`: Platforms to deploy in addition to `libqxcb.so`. Platform must be available from `QT_INSTALL_PLUGINS/platforms`.
   - To support Wayland, add `libqwayland-egl.so;libqwayland-generic.so`
 
+**Translations:**
+- `$TRANSLATIONS_INDIVIDUAL=YES/NO`
+- `$TRANSLATIONS_MERGED=YES/NO`
+- `$TRANSLATIONS_SYMLINK_APP=YES/NO`
+- `$TRANSLATION_LANGUAGES=comma separated language list`
+
+See [translations](#translations) for an explanation of the env variables.
+
 QML related:
 - `$QML_SOURCES_PATHS`: directory containing the application's QML files — useful/needed if QML files are "baked" into the binaries. linuxdeploy-plugin-qt will look for all imported QML modules and include them. `$QT_INSTALL_QML` is prepended to this list internally.
 - `$QML_MODULES_PATHS`: extra directories containing imported QML files (normally doesn't need to be specified).
+
+## Developer details
+### Standard paths
+Here are standard Qt lookup paths of appimage contents (same in Qt5 and Qt6):
+
+| Path type                | Path                                       |
+| -----------------------: | ------------------------------------------ |
+| `PrefixPath`             | `/tmp/.mount_<random_id>/usr`              |
+| `DocumentationPath`      | `/tmp/.mount_<random_id>/usr/doc`          |
+| `HeadersPath`            | `/tmp/.mount_<random_id>/usr/include`      |
+| `LibraryExecutablesPath` | `/tmp/.mount_<random_id>/usr/libexec`      |
+| `BinariesPath`           | `/tmp/.mount_<random_id>/usr/bin`          |
+| `PluginsPath`            | `/tmp/.mount_<random_id>/usr/plugins`      |
+| `QmlImportsPath`         | `/tmp/.mount_<random_id>/usr/qml`          |
+| `ArchDataPath`           | `/tmp/.mount_<random_id>/usr`              |
+| `DataPath`               | `/tmp/.mount_<random_id>/usr`              |
+| `TranslationsPath`       | `/tmp/.mount_<random_id>/usr/translations` |
+| `ExamplesPath`           | `/tmp/.mount_<random_id>/usr/examples`     |
+| `TestsPath`              | `/tmp/.mount_<random_id>/usr/tests`        |
+| `SettingsPath`           | `/tmp/.mount_<random_id>/usr`              |

src/CMakeLists.txt

@@ -31,7 +31,7 @@ add_library(linuxdeploy-plugin-qt_util STATIC util.cpp util.h)
 target_include_directories(linuxdeploy-plugin-qt_util PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
 target_link_libraries(linuxdeploy-plugin-qt_util linuxdeploy_core args Threads::Threads)
 
-add_executable(linuxdeploy-plugin-qt main.cpp qt-modules.h qml.cpp qml.h deployment.h)
+add_executable(linuxdeploy-plugin-qt main.cpp qt-modules.h qml.cpp qml.h deployment.h translation-deploymant.cpp translation-deploymant.h)
 target_link_libraries(linuxdeploy-plugin-qt linuxdeploy_core args nlohmann_json::nlohmann_json linuxdeploy-plugin-qt_util Threads::Threads)
 set_target_properties(linuxdeploy-plugin-qt PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/bin")
 target_compile_definitions(linuxdeploy-plugin-qt

src/deployment.h

@@ -114,53 +114,3 @@ inline bool createAppRunHook(appdir::AppDir &appDir) {
 
     return true;
 }
-
-inline bool
-deployTranslations(appdir::AppDir &appDir, const fs::path &qtTranslationsPath, const std::vector<QtModule> &modules) {
-    if (qtTranslationsPath.empty() || !fs::is_directory(qtTranslationsPath)) {
-        ldLog() << LD_WARNING << "Translation directory does not exist, skipping deployment";
-        return true;
-    }
-
-    ldLog() << "Qt translations directory:" << qtTranslationsPath << std::endl;
-
-    auto checkName = [&appDir, &modules](const fs::path &fileName) {
-        if (!strEndsWith(fileName.string(), ".qm"))
-            return false;
-
-        // always deploy basic Qt translations
-        if (strStartsWith(fileName.string(), "qt_") && fileName.filename().string().size() >= 5 &&
-            fileName.filename().string().size() <= 6)
-            return true;
-
-        for (const auto &module : modules) {
-            if (!module.translationFilePrefix.empty() && strStartsWith(fileName.string(), module.translationFilePrefix))
-                return true;
-        }
-
-        return false;
-    };
-
-    for (fs::directory_iterator i(qtTranslationsPath); i != fs::directory_iterator(); ++i) {
-        if (!fs::is_regular_file(*i))
-            continue;
-
-        const auto fileName = (*i).path().filename();
-
-        if (checkName(fileName))
-            appDir.deployFile(*i, appDir.path() / "usr/translations/");
-    }
-
-    const auto& appDirTranslationsPath = appDir.path() / "usr/translations";
-    for (auto& i: fs::recursive_directory_iterator(appDir.path())) {
-        if (!fs::is_regular_file(i) || pathContainsFile(appDirTranslationsPath, i))
-            continue;
-
-        const auto fileName = i.path().filename();
-
-        if (strEndsWith(fileName.string(), ".qm"))
-            appDir.createRelativeSymlink(i, appDir.path() / "usr/translations" / fileName);
-    }
-
-    return true;
-}

src/main.cpp

@@ -16,6 +16,7 @@
 #include "qt-modules.h"
 #include "util.h"
 #include "deployment.h"
+#include "translation-deploymant.h"
 #include "deployers/PluginsDeployerFactory.h"
 
 namespace fs = std::filesystem;
@@ -26,6 +27,67 @@ using namespace linuxdeploy::log;
 using namespace linuxdeploy::plugin::qt;
 
 
+// These classes are a hack to be able to get --feature/--no-feature flags
+// where latter flags override former ones.
+class TrueToggleFlag : public args::Flag {
+    private:
+        bool &value;
+
+    public:
+        TrueToggleFlag(args::Group &group, const std::string &name,
+          const std::string &help, args::Matcher &&matcher,
+          bool &value)
+            : args::Flag(group, name, help, std::move(matcher)),
+              value(value) {}
+
+        virtual void ParseValue(const std::vector<std::string> &v) override {
+            args::Flag::ParseValue(v); // keeps Matched()/Get() bookkeeping intact
+            value = true;
+        }
+};
+
+class FalseToggleFlag : public args::Flag {
+    private:
+        bool &value;
+
+    public:
+        FalseToggleFlag(args::Group &group, const std::string &name,
+          const std::string &help, args::Matcher &&matcher,
+          bool &value)
+            : args::Flag(group, name, help, std::move(matcher)),
+              value(value) {}
+
+        virtual void ParseValue(const std::vector<std::string> &v) override {
+            args::Flag::ParseValue(v); // keeps Matched()/Get() bookkeeping intact
+            value = false;
+        }
+};
+
+
+class CustomArgumentParseError : public std::runtime_error {
+    using std::runtime_error::runtime_error;
+};
+
+
+static bool yesNoArg(const char *envVar, std::string_view value) {
+    std::string lowercase;
+    lowercase.reserve(value.size());
+
+    std::transform(value.begin(), value.end(), std::back_inserter(lowercase),
+      [](unsigned char c){ return std::tolower(c); }
+    );
+
+    if (lowercase == "yes" || lowercase == "y" || lowercase == "on" ||
+        lowercase == "1" || lowercase == "true")
+        return true;
+    if (lowercase == "no" || lowercase == "n" || lowercase == "off" ||
+        lowercase == "0" || lowercase == "false")
+        return false;
+
+    throw CustomArgumentParseError("Unknown value for env variable \"" +
+                                   std::string(value) + "!");
+}
+
 int main(const int argc, const char *const *const argv) {
     // set up verbose logging if $DEBUG is set
     if (getenv("DEBUG"))
@@ -44,6 +106,26 @@ int main(const int argc, const char *const *const argv) {
                                                   "Extra Qt module to deploy (specified by name, filename or path)",
                                                   {'m', "extra-module"});
 
+    args::ValueFlag<std::string> qtLanguages(parser, "language list",
+                                             "Comma separated list of Qt languages to install (does not apply to "
+                                             ".qm files provided by program)", {"qt-languages"});
+    bool individualTranslations = true;
+    bool appTranslations = true;
+    bool mergedTranslations = false;
+
+    TrueToggleFlag yesIndividualTranslations(parser, "", "Enable individual translations",
+                                             {"individual-translations"}, individualTranslations);
+    FalseToggleFlag noIndividualTranslations(parser, "", "Disable individual translations",
+                                             {"no-individual-translations"}, individualTranslations);
+    TrueToggleFlag yesAppTranslations(parser, "", "Enable symlinking app translations to standard directory",
+                                      {"app-symlink-translations"}, appTranslations);
+    FalseToggleFlag noAppTranslations(parser, "", "Disable symlinking app translations to standard directory",
+                                      {"no-app-symlink-translations"}, appTranslations);
+    TrueToggleFlag yesMergedTranslations(parser, "", "Enable producing of merged qt_*.qm translation files",
+                                         {"merged-translations"}, mergedTranslations);
+    FalseToggleFlag noMergedTranslations(parser, "", "Disable producing of merged qt_*.qm translation files",
+                                         {"no-merged-translations"}, mergedTranslations);
+
     args::Flag pluginType(parser, "", "Print plugin type and exit", {"plugin-type"});
     args::Flag pluginApiVersion(parser, "", "Print plugin API version and exit", {"plugin-api-version"});
 
@@ -290,12 +372,67 @@ int main(const int argc, const char *const *const argv) {
                 return 1;
     }
 
-    ldLog() << std::endl << "-- Deploying translations --" << std::endl;
-    if (!deployTranslations(appDir, qtTranslationsPath, qtModulesToDeploy)) {
-        ldLog() << LD_ERROR << "Failed to deploy translations" << std::endl;
+    // deployTranslations() might need a temporary directory. It is placed here
+    // to make sure it lives long enough, because files from it will be deployed.
+    TempDir lconvertTemporaryDirectory;
+
+    try {
+        if (!yesIndividualTranslations && !noIndividualTranslations) {
+            const char *individualTranslationsEnv = getenv("TRANSLATIONS_INDIVIDUAL");
+            if (individualTranslationsEnv != nullptr) {
+                individualTranslations = yesNoArg("TRANSLATIONS_INDIVIDUAL",
+                                                  individualTranslationsEnv);
+            }
+        }
+        if (!yesMergedTranslations && !noMergedTranslations) {
+            const char *mergedTranslationsEnv = getenv("TRANSLATIONS_MERGED");
+            if (mergedTranslationsEnv != nullptr) {
+                mergedTranslations = yesNoArg("TRANSLATIONS_MERGED", mergedTranslationsEnv);
+            }
+        }
+        if (!yesAppTranslations && !noAppTranslations) {
+            const char *appTranslationsEnv = getenv("TRANSLATIONS_SYMLINK_APP");
+            if (appTranslationsEnv != nullptr) {
+                mergedTranslations = yesNoArg("TRANSLATIONS_SYMLINK_APP", appTranslationsEnv);
+            }
+        }
+    } catch (const CustomArgumentParseError & exc) {
+        std::cerr << exc.what() << std::endl;
         return 1;
     }
 
+    TranslationDeploymentType translationDeploymentType = 0;
+    if (individualTranslations)
+        translationDeploymentType |= TranslationDeployment::individual;
+    if (appTranslations)
+        translationDeploymentType |= TranslationDeployment::user_symlink;
+    if (mergedTranslations)
+        translationDeploymentType |= TranslationDeployment::merged;
+
+    std::vector<std::string> languages = split(qtLanguages.Get(), ',');
+
+    if (qtLanguages) {
+        languages = split(qtLanguages.Get(), ',');
+    } else {
+        const char *languagesEnv = getenv("TRANSLATION_LANGUAGES");
+        if (languagesEnv != nullptr) {
+            languages = split(languagesEnv, ',');
+        }
+    }
+
+    if (translationDeploymentType == 0) {
+        ldLog() << std::endl << "-- Skipping translation deployment on user request --" << std::endl;
+    } else {
+        ldLog() << std::endl << "-- Deploying translations --" << std::endl;
+        if (!deployTranslations(appDir, qtTranslationsPath, qtModulesToDeploy,
+                                translationDeploymentType, languages,
+                                lconvertTemporaryDirectory))
+        {
+            ldLog() << LD_ERROR << "Failed to deploy translations" << std::endl;
+            return 1;
+        }
+    }
+
     ldLog() << std::endl << "-- Executing deferred operations --" << std::endl;
     if (!appDir.executeDeferredOperations()) {
         ldLog() << LD_ERROR << "Failed to execute deferred operations" << std::endl;