Use production editor regions for Java EOF completion in existing LSP

.github/workflows/embedded-languages.yml

@@ -0,0 +1,41 @@
+name: Embedded language consumer
+on:
+  pull_request:
+  push:
+    branches: [master]
+jobs:
+  diagnostics:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-java@v5
+        with:
+          distribution: temurin
+          java-version: '21.0.9'
+      - uses: actions/checkout@v6
+        with:
+          repository: opaopa6969/unlaxer-parser
+          ref: a3d1b131f1509361e94924e64c0f2cf65da9fb5a
+          path: .integration/unlaxer-parser
+      - name: Install source-map and diagnostic APIs
+        working-directory: .integration/unlaxer-parser
+        run: mvn -B -pl unlaxer-common,unlaxer-dsl -am install -DskipTests -Dgpg.skip=true
+      - name: Install production Tiny with current default dependencies
+        run: mvn -B install -DskipTests -Dtinyexpression.skipRailroad=true -Dgpg.skip=true
+      - name: Default LSP and DAP compatibility
+        working-directory: tools/tinyexpression-p4-lsp-vscode
+        run: mvn -B clean test -Dtest=TinyExpressionP4LanguageServerExtTest,TinyExpressionP4DebugAdapterExtTest
+      - name: Real javac through existing opt-in LSP consumer
+        working-directory: tools/tinyexpression-p4-lsp-vscode
+        run: |
+          mvn -B clean package -Pembedded-languages -Dgpg.skip=true -Dtest=ProductionEmbeddedDiagnosticsTest,TinyExpressionP4LanguageServerExtTest,TinyExpressionP4DebugAdapterExtTest
+          python3 scripts/check-embedded-lsp.py
+      - uses: actions/setup-node@v6
+        with:
+          node-version: '24'
+      - name: Extension setting and TypeScript checks
+        working-directory: tools/tinyexpression-p4-lsp-vscode
+        run: |
+          npm ci --ignore-scripts
+          npm run compile
+          npm test

README.en.md

@@ -299,6 +299,8 @@ Full specification: [docs/language-guide.md](docs/language-guide.md)
 
 ## LSP / DAP
 
+The existing **Java** LSP consumer offers [opt-in embedded Java diagnostics and EOF completion](tools/tinyexpression-p4-lsp-vscode/EMBEDDED-LANGUAGES.md). Its current Tiny grammar adaptation and the fixed-revision Java/Rust bridge fixtures have separate validation scopes; they do not establish a Rust Tiny LSP consumer.
+
 The [tinyexpression-p4-lsp-vscode](tools/tinyexpression-p4-lsp-vscode/README.md) VS Code extension provides:
 
 - Syntax highlighting and semantic tokens

README.md

@@ -329,6 +329,8 @@ VS Code 拡張 [tinyexpression-p4-lsp-vscode](tools/tinyexpression-p4-lsp-vscode
 - 宣言・import・method・`.tecatalog` を使う補完とホバー
 - DAP デバッグ（6 バックエンドのパリティ比較）
 
+既存の **Java** LSP consumer へ埋め込み Java 診断・EOF 補完を接続する [opt-in 設定と検証範囲](tools/tinyexpression-p4-lsp-vscode/EMBEDDED-LANGUAGES.md) も提供します。現行 Tiny grammar への接続と、固定版の Java / Rust 共通 bridge fixture は別の検証対象です。Rust Tiny LSP consumer の対応を示すものではありません。
+
 DAP 0.2.33 は生成AST上の停止・ブレークポイント・実ランタイム評価に加え、
 人向けの期待値付き構文診断と、LLM/editor向けの構造化診断dataに対応します。
 `launch.json` の `variables` は `CalculationContext` に型付きで注入され、選択バックエンド、

tools/tinyexpression-p4-lsp-vscode/EMBEDDED-LANGUAGES.md

@@ -0,0 +1,31 @@
+# Embedded Java diagnostics in the existing P4 server
+
+This is an explicit migration profile. The default dependencies follow the current root POM (unlaxer 3.3.0-SNAPSHOT and Tiny 2.1.0); the diagnostics feature remains disabled by default, and P4/ubnfc and DAP behavior are unchanged. The profile uses the new source-map/provider API from unlaxer-parser 3.3.0-SNAPSHOT (validated at a3d1b131) and JDK 21.0.9. It never evaluates a Tiny formula or Java body; the external javac adapter disables annotation processing.
+
+Build unlaxer-parser's `unlaxer-common` and `unlaxer-dsl` modules with `mvn -pl unlaxer-common,unlaxer-dsl -am install -DskipTests -Dgpg.skip=true`. Install this repository's root artifact using the current root dependencies with `mvn install -DskipTests -Dtinyexpression.skipRailroad=true -Dgpg.skip=true`. Then, from this directory:
+
+```sh
+mvn clean package -Pembedded-languages -Dgpg.skip=true
+```
+
+Point `tinyExpressionP4Lsp.server.jarPath` at the resulting server jar, use JDK 21.0.9, and enable `tinyExpressionP4Lsp.embeddedLanguageDiagnostics`. Other LSP clients can set `initializationOptions.embeddedLanguageDiagnostics: true`. Enabling it with a server that lacks the profile fails initialization explicitly. Merely including the profile does not enable analysis.
+
+The existing `TinyExpressionP4LanguageServerExt` now sends untouched, versioned full documents to an optional diagnostics provider. The production FormulaInfo and P4 parsers construct all section regions; javac diagnostics from each Java body are mapped to original UTF-16 positions. Updating later sections does not reuse first-section cache results. Stale updates are rejected, save revalidates, and close clears diagnostics. Invalid or incomplete region extraction and unavailable providers emit `EMBEDDED_UNAVAILABLE`, not empty success. Inside an owned Java region, completion now uses the same production editor binding and returns source-mapped javac edits, including at unfinished host EOF. Outside Java, existing first-section Tiny completion, formatting and other methods retain their prior behavior.
+
+The bridge is a namespaced port of unlaxer-parser production editor bridge commit f2fc6faa (the strict bridge originated in PR #470), adapted to the current Tiny grammar revision 0d84f0dc. The shared Java/Rust bridge contract is independently tested in that repository at the fixed Tiny revision `f86ce8a5ab0ab7d23fdba2cd477187df8aa7607c`; this current-consumer adaptation has a separate validation scope. This repository's existing LSP consumer is Java; no Rust LSP consumer is claimed here. Foreign dependency diagnostics need a later consumer extension. Workspace ownership is rejected explicitly by this single-document adapter.
+
+The validation targets are distinct:
+
+| Tiny source revision | Hosts / consumer | Independent evidence |
+| --- | --- | --- |
+| `f86ce8a5ab0ab7d23fdba2cd477187df8aa7607c` | Strict Java / Rust production bridge in unlaxer-parser [#470](https://github.com/opaopa6969/unlaxer-parser/pull/470) | 16 shared inputs / 177 independent observations per host, including original code-point / UTF-16 positions and query edits. |
+| `f86ce8a5ab0ab7d23fdba2cd477187df8aa7607c` | Editor Java / Rust production bridge in unlaxer-parser [#479](https://github.com/opaopa6969/unlaxer-parser/pull/479) | Nine shared inputs / 29 independent observations per host; edited unfinished inputs still fail strict parsing. These fixed-grammar fixtures do not certify current long-fence consumer parity. |
+| `0d84f0dc5c0331c09f46350cceb21cb57ee25d79` | Existing **Java** Tiny P4 LSP consumer, [#252](https://github.com/opaopa6969/tinyexpression/pull/252) / [#254](https://github.com/opaopa6969/tinyexpression/pull/254) | The paired PRs add opt-in javac diagnostics and EOF completion with current retained-source long fences and original UTF-16 positions. No current Rust Tiny LSP consumer or current-long-fence Java/Rust protocol parity is implemented or verified by this profile. |
+
+Run `mvn test -Pembedded-languages -Dtest=ProductionEmbeddedDiagnosticsTest,TinyExpressionP4LanguageServerExtTest,TinyExpressionP4DebugAdapterExtTest` here. The integration test runs real javac on production fixture 69 (ten formulas, two Java bodies), checks both error positions, a second-section-only update, Unicode/CRLF, stale updates, save, close and unavailable extraction. Run the 40 existing LSP/DAP tests without the profile as a separate clean build to verify the default dependency path. Use an isolated Maven repository if other checkouts install Tiny artifacts with different generated AST versions.
+
+The explicit editor entry certifies leading CodeBlock prefixes using the real production parser. Recognition-only synthetic closing text never enters the snapshot or edits. Incomplete Java diagnostics and EOF completion stay inside the original body; valid siblings remain usable when another formula fails. The strict parser is unchanged and still reports unfinished formulas as FAILED. The packaged stdio smoke additionally verifies the Unicode/CRLF unfinished-body completion edit at line 3, characters 41–44.
+The new completion method is optional on the dependency-neutral SPI. The default build does not load a provider or alter its Tiny completion behavior. Both diagnostics and completion use the same immutable full snapshot under the existing server's lifecycle lock, so an intervening update cannot relabel an old result with a new version.
+
+
+Current Tiny grammar support includes the retained-source long CodeBlock form: exact-width standalone fences keep shorter triple backticks opaque inside Java text. The adapter uses the production LongCodeFence layout after the production parser certifies the block. FormulaInfo hash-comment masking never rewrites an opaque Java body. The current Tiny grammar revision is carried in FormulaInfo/Tiny region identities; the Java profile identity remains `lang/java@0.1.0`.

tools/tinyexpression-p4-lsp-vscode/README.md

@@ -19,6 +19,8 @@ for TinyExpression formulas using the P4 grammar (UBNF-generated, type-safe).
   - `_tinyP4AstNodePath` — breadth-first path through the AST
   - `parity.*` — 6-backend evaluation comparison (JAVA_CODE / AST_EVALUATOR / DSL_JAVA_CODE / P4_AST / P4_DSL)
 
+Opt-in embedded Java diagnostics and EOF completion in the existing **Java** LSP consumer are described in [EMBEDDED-LANGUAGES.md](EMBEDDED-LANGUAGES.md). That document distinguishes the current Tiny grammar adaptation from the fixed-revision Java/Rust bridge fixtures; it does not claim a Rust Tiny LSP consumer.
+
 ## Requirements
 
 - Java 21+

tools/tinyexpression-p4-lsp-vscode/package.json

@@ -296,6 +296,11 @@
       "type": "object",
       "title": "TinyExpression P4 LSP",
       "properties": {
+        "tinyExpressionP4Lsp.embeddedLanguageDiagnostics": {
+          "type": "boolean",
+          "default": false,
+          "description": "Diagnose and complete embedded Java in every FormulaInfo section without executing user code. Requires the explicitly built embedded-languages server profile and JDK 21.0.9. Changing this setting restarts the server."
+        },
         "tinyExpressionP4Lsp.server.javaPath": {
           "type": "string",
           "default": "java",

tools/tinyexpression-p4-lsp-vscode/pom.xml

@@ -217,6 +217,23 @@
     </build>
 
     <profiles>
+        <!-- Explicit migration profile; default P4/ubnfc and DAP dependencies follow the root POM. -->
+        <profile>
+            <id>embedded-languages</id>
+            <dependencies><dependency><groupId>org.unlaxer</groupId><artifactId>unlaxer-common</artifactId><version>${unlaxer.version}</version></dependency></dependencies>
+            <build>
+                <resources><resource><directory>src/embedded/resources</directory></resource></resources>
+                <plugins><plugin>
+                    <groupId>org.codehaus.mojo</groupId><artifactId>build-helper-maven-plugin</artifactId>
+                    <executions>
+                        <execution><id>embedded-sources</id><phase>generate-sources</phase><goals><goal>add-source</goal></goals>
+                            <configuration><sources><source>src/embedded/java</source></sources></configuration></execution>
+                        <execution><id>embedded-tests</id><phase>generate-test-sources</phase><goals><goal>add-test-source</goal></goals>
+                            <configuration><sources><source>src/embedded-test/java</source></sources></configuration></execution>
+                    </executions>
+                </plugin></plugins>
+            </build>
+        </profile>
         <profile>
             <id>p4-smoke</id>
             <build>

tools/tinyexpression-p4-lsp-vscode/scripts/check-embedded-lsp.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""Verify the packaged existing launcher, opt-in provider service, and actual javac diagnostics."""
+import json
+import pathlib
+import subprocess
+import threading
+import queue
+
+root = pathlib.Path(__file__).resolve().parents[1]
+messages = queue.Queue()
+with (root / 'target/embedded-stdio.log').open('wb') as errors:
+    process = subprocess.Popen(['java', '--enable-preview', '-jar', str(root / 'target/tinyexpression-p4-lsp-server.jar')], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=errors)
+    def read():
+        try:
+            while True:
+                headers = {}
+                while True:
+                    line = process.stdout.readline()
+                    if not line:
+                        raise EOFError('server exited')
+                    if line == b'\r\n':
+                        break
+                    key, value = line.decode('ascii').split(':', 1)
+                    headers[key.lower()] = value.strip()
+                messages.put(json.loads(process.stdout.read(int(headers['content-length']))))
+        except BaseException as error:
+            messages.put(error)
+    threading.Thread(target=read, daemon=True).start()
+    def send(method, params, request_id=None):
+        message = {'jsonrpc': '2.0', 'method': method, 'params': params}
+        if request_id is not None:
+            message['id'] = request_id
+        body = json.dumps(message, ensure_ascii=False).encode('utf-8')
+        process.stdin.write(f'Content-Length: {len(body)}\r\n\r\n'.encode('ascii') + body)
+        process.stdin.flush()
+    def receive(predicate):
+        while True:
+            message = messages.get(timeout=40)
+            if isinstance(message, BaseException):
+                raise message
+            if predicate(message):
+                return message
+    try:
+        send('initialize', {'capabilities': {}, 'initializationOptions': {'embeddedLanguageDiagnostics': True}}, 1)
+        initialize = receive(lambda value: value.get('id') == 1)
+        assert initialize['result']['capabilities']['experimental']['embeddedLanguageDiagnostics']['enabled']
+        send('initialized', {})
+        uri = 'file:///production.formulainfo'
+        send('textDocument/didOpen', {'textDocument': {'uri': uri, 'languageId': 'tinyexpressionP4', 'version': 1, 'text': (root / 'src/embedded-test/resources/type-error.txt').read_text()}})
+        published = receive(lambda value: value.get('method') == 'textDocument/publishDiagnostics')['params']
+        assert published['version'] == 1, published
+        diagnostics = [item for item in published['diagnostics'] if item.get('source') == 'tinyexpression-java']
+        assert [(item['range']['start']['line'], item['range']['start']['character']) for item in diagnostics] == [(41, 9), (69, 9)], published
+        assert all(item['code'] == 'compiler.err.prob.found.req' and item['severity'] == 1 for item in diagnostics)
+        partial = (root / 'src/embedded-test/resources/partial-eof.txt').read_bytes().decode('utf-8')
+        send('textDocument/didChange', {'textDocument': {'uri': uri, 'version': 2}, 'contentChanges': [{'text': partial}]})
+        assert receive(lambda value: value.get('method') == 'textDocument/publishDiagnostics')['params']['version'] == 2
+        send('textDocument/completion', {'textDocument': {'uri': uri}, 'position': {'line': 3, 'character': 44}}, 3)
+        completions = receive(lambda value: value.get('id') == 3)['result']
+        target = next(item for item in completions if item['label'] == 'target')
+        assert target['textEdit'] == {'range': {'start': {'line': 3, 'character': 41}, 'end': {'line': 3, 'character': 44}}, 'newText': 'target'}, target
+        send('textDocument/didClose', {'textDocument': {'uri': uri}})
+        assert receive(lambda value: value.get('method') == 'textDocument/publishDiagnostics')['params']['diagnostics'] == []
+        send('shutdown', {}, 2)
+        receive(lambda value: value.get('id') == 2)
+        send('exit', {})
+        print('PASS: packaged Tiny P4 launcher -> real javac -> two original FormulaInfo diagnostic positions -> partial EOF completion -> close clear')
+    finally:
+        process.kill()
+        process.wait(timeout=5)