refactor(web): unbundle lexical model worker from compiled artifact

[mcdurdin]

This change unbundles the lexical model worker from keymanweb.js, and provides it alongside as a separate .js file.

This is groundwork to address issue #13862 with Keyman for Android, where importScripts appears to be unreliable in a Worker context; this unreliability has increased dramatically as of Chrome 152. Note: file:// is unreliable, but http[s]:// has proven reliable (see #16628).

Unbundling of the worker was on the roadmap for Keyman but this change has been brought forward, originally due to the need to mitigate #13862.

• Rename several classes and files to clarify difference between worker and factory that constructs worker
• Rework worker factories to actually implement WorkerFactory interface
• Fixup pathing issues where http:/host was returned instead of http://host
• Rename worker-thread output from worker-main.js to worker-thread.js, resolving confusion with separate worker-main module
• Add path resolution for worker-thread.js to Keyman Engine
• Tweak build scripts and unit tests to work with the new unbundled paths

Fixes: #13862
Build-bot: release

User Testing

• TEST_IOS: Verify that Keyman for iOS predictive text continues to work
• TEST_ANDROID: Verify that Keyman for Android predictive text continues to work
• TEST_WEB: Run a basic smoke test on KeymanWeb, verifying that basic behaviors in the test pages continue to work correctly, particularly the Predictive Text manual test page.

[keymanapp-test-bot]

User Test Results

Test specification and instructions

• ⬜ TEST_IOS (OPEN)
• ⬜ TEST_ANDROID (OPEN)
• ⬜ TEST_WEB (OPEN)

Results Template

# Test Results

* **TEST_IOS (OPEN):** notes
* **TEST_ANDROID (OPEN):** notes
* **TEST_WEB (OPEN):** notes

[mcdurdin]

@jahorton FYI this is what I was playing with in terms of unbundling the worker; I have not yet tackled the path questions (need some special casing for unit tests, but for normal use it should be easier I hope, relative to keymanweb.js init path). And of course the Android bypass of importScript is really a separate question altogether.

Are you keen to take this on? Or would you prefer that I keep working on this?

[mcdurdin]

Per @ermshiperete, who kindly compared the two implementations of this:

#16574 is based on an earlier commit of #16565. Since both are in draft they may be incomplete, and I didn't check what from #16565 would also be in #16574, so some of my comments below might be valid for the other PR as well.

In #16565

• I like the renaming of the classes and files which I think makes it clearer what they do.
• Also getting rid of the distinction between NodeWorker and SourcemappedWorker, although I'm not entirely sure what we loose with the change.
• Renaming to *Factory makes things much clearer.
• Good cleanup of comments, making them more consistent with reality.
• Gets rid of build-wrapper.js and build-polyfiller.js, thus simplifying build process.

#16574

• looks like it might better deal with the location of the workerthread file.
• It directly uses new NodeWorkerFactory() instead of a global variable which IMO is clearer and sufficient in this case.
• It uses constructInstance in tests, passing the path to the .js file instead of adding a constructTestInstance function.
• Small changes/fixes/adjustments in tests, e.g. https://github.com/keymanapp/keyman/pull/16574/changes#diff-50ab7c3b74bb0c0236fecb9adc0e21d668cd0340c070ba46b5bc2ddaa9d31106R38
• Adds a separate init() function to LanguageProcessor so that we don't have to pass the sourcepath in the constructor - I'm not sure what I like better.

---

My plan is to use #16565, apply the review comments on this PR, and port the following improvements over from #16574:

1. Eliminate the factory variables and use new ...Factory() pattern throughout
2. Refactor constructInstance and path management for worker-thread.js -- this will simplify fix(android): concatenate worker and model file to load together #16586 also
3. Review test patches in refactor(web): fully unbundles worker with engine-init configurable source path #16574 and apply if necessary (not clear that they are necessary?)

[jahorton]

• Adds a separate init() function to LanguageProcessor so that we don't have to pass the sourcepath in the constructor - I'm not sure what I like better.

Note that architecturally, the engine's path configuration is not ready - or even near ready - when LanguageProcessor is first instantiated within the engine's current bootstrapping process.

[jahorton]

• Also getting rid of the distinction between NodeWorker and SourcemappedWorker, although I'm not entirely sure what we loose with the change.

Since we were bundling the worker previously, when interactively debugging, there was no way to provide sourcemaps without also bundling them. This led to a massive filesize spike, so we only bundled sourcemaps for 'debug' builds.

Unbundling the worker allows us to draw in the worker sourcemaps from a separate file, so we don't really lose anything here - as long as we remember to copy the sourcemap file into the Android and iOS bundles in addition to the newly spun-off worker script file.

[jahorton]

Past the other comment I made, I'd like to verify that the worker's sourcemaps show up properly when debugging the Android and iOS apps. Will get to that later today.

[jahorton]

This solution appears to be bypassing the standard KeymanEngine path-configuration process, instead determining the .sourcePath it would have constructed in advance?

[mcdurdin]

Yes. This is making the (hopefully fairly safe) assumption that worker-thread.js can be found at the same URL path as keymanweb.js; there's a bit of a chicken-and-egg with the pattern of constructing the predictive text worker factory before KeymanEngine is constructed. Not seeing a cleaner way of implementing this at present (although perhaps 'worker-thread.js' belongs in a constant somewhere)

[jahorton]

My version of the PR just initialized the actual worker later, using the configured path as a parameter for the constructInstance method.

[jahorton]

The Android build checks out, but I'm not getting sourcemaps on iOS yet. Looking into that now.

[jahorton]

OK, fixed the iOS engine sourcemapping.