LightOnOCR-2-1B fails to load in Docker deployment: missing LightOnOcrForConditionalGeneration in bundled transformers

[tatankam]

Description

I'm running SIE using the official Docker deployment with the image image: ghcr.io/superlinked/sie-server:latest-cuda12-default

When trying to use the model lightonai/LightOnOCR-2-1B, model loading fails with the following error:

ModelLoadFailedError: Model 'lightonai/LightOnOCR-2-1B' failed to load (DEPENDENCY, attempts=1):
ImportError: cannot import name 'LightOnOcrForConditionalGeneration' from 'transformers'

Investigation

Inside the running container:

import transformers
print(transformers.__version__)

returns:

4.57.6

and:

from transformers import LightOnOcrForConditionalGeneration

fails with:

ImportError: cannot import name 'LightOnOcrForConditionalGeneration'

The bundled Transformers package used by SIE is:

/app/bundle-libs/transformers

I also verified that there is no lighton_ocr model implementation present under:

/app/bundle-libs/transformers/models

Expected behavior

Since lightonai/LightOnOCR-2-1B appears in the available model catalog, I would expect it to either:

1. Load successfully, or
2. Be excluded/marked as unsupported when the required backend is not available.

Question

Is lightonai/LightOnOCR-2-1B currently supported by the Docker release of SIE?

If not, is support planned in a future image/version, or is there a recommended workaround?

Thanks a lot
Mario

[jturner421]

I am seeing the same thing. I attempted to use this server tag: ghcr.io/superlinked/sie-server:v0.6.19-cuda12-transformers5. The Lighton model loads and works, but it very slow.

[svonava]

Interesting, checking!

[svonava]

@jturner421 what hardware are you running on and what perf are you seeing? we will do some more optimization here

[svonava]

Update: we have merged a broader set of generative OCR performance improvements, and they will be included in the next release.

On our canonical NVIDIA L4 benchmark (500 requests at concurrency 4), LightOnOCR-2-1B improved from 125.7s to 20.8s p50 latency and from 450.7s to 44.4s p99 latency, while page throughput increased from 0.024 to 0.164 pages/s. That is approximately 6.0× lower median latency, 10.1× lower p99 latency, and 6.8× higher page throughput.

The same optimized serving path now also covers PaddleOCR-VL-1.5 and GLM-OCR. Their new canonical L4 results are:

• PaddleOCR-VL-1.5: 12.4s p50, 26.0s p99, 0.275 pages/s
• GLM-OCR: 16.8s p50, 67.2s p99, 0.199 pages/s

We also reran the full 1,403-page quality benchmark for all three models; the quality gates passed with results comparable to the previous serving path.

[jturner421]

@svonava Sorry it took so long to get back to you. I am running this on hardware that you have not benchmarked. I am using an Ubuntu 24.04 LTS server with an Nvidia RTX 5090. Although this is technically a consumer level card, it does have 32GB of VRAM. Using this setup, I am able to serve inference, OCR, and embeddings. I've attached a screenshot of Nvidia-smi for reference.

I don't expect you to be able to check against this. But I do think that the updates that you've made that will be in the next release will be helpful. Have those already been checked into source? I ask because in order to run the project locally, I had to make some small changes to run the gateway locally to serve inference since I don't have Kubernetes set up. The project as it currently stands does not allow inference to be served unless it's on a local Mac or using Kubernetes.

[svonava]

Very cool - cc @huronat any idea when the OCR speedup gets released?

@jturner421 I'm jealous of the 5090, sitting here with 4090 locally still :-) Very curious what numbers you'll be getting once the next release hits.

[huronat]

@jturner421 hey, thanks for flagging, the OCR speedup optimized image has been rolled out with the v0.6.20 release and is already available as ghcr.io/superlinked/sie-server:v0.6.20-cuda12-sglang.
By the way, we're also carving the latest public release for v0.6.21, as well, soon. We'd love to hear your numbers on an RTX 5090/SM120 path.

[jturner421]

@huronat @svonava Are your OCR tests, metrics, and test documents publically available? I did not see them in the repo. I'd be happy to run a set of benchmarks against my config for you.

[huronat]

@jturner421 hey, thanks for running benchmarks. Currently we don't publish it onto the public OSS repo, but we do have measurement benchmarks on the SIE webpage, please find e.g.: https://superlinked.com/models/lightonai-lightonocr-2-1b

[svonava]

@tatankam is this now resolved? close if yes pls

[tatankam]

I updated ghcr.io/superlinked/sie-server:latest-cuda12-default so, is it the model still available on that image?
RequestError: Model 'lightonai/LightOnOCR-2-1B' not found

Instead if I use: "PaddlePaddle/PaddleOCR-VL-1.5:transformers" it works

[tatankam]

Bug: OCR model lightonai/LightOnOCR-2-1B not found after updating sie-server

Description

After updating the SIE server Docker image to the latest version:

docker pull ghcr.io/superlinked/sie-server:latest-cuda12-default

OCR extraction requests that previously worked now fail with:

RequestError: Model 'lightonai/LightOnOCR-2-1B' not found
Did you mean 'lightonai/mLateOn'?

The suggested model name (lightonai/mLateOn) does not appear to be related to the requested OCR model.

Environment

Server

docker image: ghcr.io/superlinked/sie-server:latest-cuda12-default

Updated to the latest available image on GHCR.

Client

sie-sdk==0.7.2

Python

Python 3.12

Reproduction

from pathlib import Path

from sie_sdk import SIEClient
from sie_sdk.model import Item

image_bytes = Path("../data/samples/Esempio.png").read_bytes()

client = SIEClient("http://myserver.my.net:8080")

result = client.extract(
    "lightonai/LightOnOCR-2-1B",
    Item(images=[image_bytes]),
)

Actual Behavior

The request fails with:

RequestError: Model 'lightonai/LightOnOCR-2-1B' not found
Did you mean 'lightonai/mLateOn'?

Full traceback:

---------------------------------------------------------------------------
RequestError Traceback (most recent call last)
Cell In[2], line 10
      6 image_bytes = Path("../data/samples/Esempio.png").read_bytes()
      7
      8 client = SIEClient("http://myserver.my.net:8080")
      9
---> 10 result = client.extract(
     11     "lightonai/LightOnOCR-2-1B",
     12     Item(images=[image_bytes]),
     13 )

File .../site-packages/sie_sdk/client/sync.py, line 3517, in SIEClient.extract
    handle_error(response)

File .../site-packages/sie_sdk/client/_shared.py, line 1247, in handle_error
    raise RequestError(...)

RequestError: Model 'lightonai/LightOnOCR-2-1B' not found
Did you mean 'lightonai/mLateOn'?

Expected Behavior

The model lightonai/LightOnOCR-2-1B should either:

1. Be available as before, or
2. Return a clear migration/deprecation message indicating the replacement model.

Additional Information

The issue started immediately after updating the server Docker image.

Could you confirm:

• Whether lightonai/LightOnOCR-2-1B is still supported?
• Whether model names have changed in the latest release?
• Whether an additional model synchronization/download step is now required after upgrading the server?

[tatankam]

So, the problem is that on the image ghcr.io/superlinked/sie-server:latest-cuda12-default
there isn't lightonai/LightOnOCR-2-1B

I pulled ghcr.io/superlinked/sie-server:latest-cuda12-transformers5 and there is lightonai/LightOnOCR-2-1B:transformers
so it works with the last one image ghcr.io/superlinked/sie-server:latest-cuda12-transformers5