[rikvanriel]

Fix +highlight worker bloat on large items

Viewing a large item with +highlight can use hundreds of MB of worker
memory per request. Measured locally on 450 KB of code: 8.2 s wall
time, 123 MB tracemalloc peak and +211 MB worker RSS. Under concurrent
crawler hits this grows workers to GBs until the OOM-killer finds them
(see issue #2307).

The view sends the whole item through pygments and builds one DOM node
per token, so cost grows with page size.

Highlight at most MAX_HIGHLIGHT_SIZE (102400) characters, cutting at
the last newline, and note on the page that the output is truncated.
Small items render exactly as before.

Feeding the input through pygments in chunks does not work, because
lexers lose their state at each chunk boundary, so multi-line strings
and comments split across chunks highlight wrongly; and the pipeline
still builds the whole tree before serializing, so memory would not
drop either. CPU stays the same because the whole text is still lexed.

After the cap, the same page renders in 0.95 s with 32 MB peak and
+2.7 MB RSS; small-page output is byte-identical.

Addresses the +highlight half of upstream issue #2307.

[roland-ruedenauer]

Assuming from your description, that the main problem is related to use of pygments, what content type caused pygments to use such excessive amounts of memory?

[rikvanriel]

It was some wiki text document that was well over 100kB in size.

[roland-ruedenauer]

It would be interesting to see how much memory would be used if pygments wasn't installed and the plaint text alternative converter would be used instead when rendering very large wiki content.

I'd suggest turning MAX_HIGHLIGHT_SIZE into a (defaulted) configuration parameter. This might also be done in a separate pull request.