Implement async load for route objects - #1417
Conversation
|
wait... this messing with the $include route command... i got plenty of error rail structure index out of range in well... let me set is as draft... |
|
@ginga81 do you want to try this? in route viewer try use minimal testing settings as i written above. |
|
alright, i think that one will fix it. although on my test its around ~4s slower... now first load is ~15s well... at least better than wrong looking route. |
|
Ready for review. |
|
All CPU cores are active, and I feel like loading times have become incredibly fast! I'll open a separate issue for this later, but even when shadows are turned off, the brightness doesn't behave as it used to—it stays bright even when entering a tunnel. |
|
ok i have added similar async load for object viewer. currently only for b3d and csv mesh. for .animated mesh need more test although it works pretty well... Note: i dont know if that route viewer like progress bar is needed here... maybe i will delete it and leave the window name progress only. |
|
X files are broken somehow, 
I'm not entirely certain why, not got time to debug this tonight, but at a probable guess, your material cache is broken- Hitting the last loaded object's materials or something? Definitely no sign of schwellen4.bmp in there. 
Randomly missing textures, very probably related to the above. These are stable throughout a reload, but if I close Route Viewer and re-open, different missing textures- I'd presume this is because the reload simply hits the cache. |
|
is there a link for that route? |
|
That commits may work... |
|
another fix and ram usage improvement. i hope its working on linux and macos... |
|
That looks better now. I'll try and test on Mac, but might be a few days. |
|
It seemed to work fine on my Ubuntu 24.04. |
However, the scene redraws when I change direction, and the frame rate subsequently drops to 10 fps.
It redraws the entire scene into the distance every time I change direction, and the frame rate does not recover afterwards.
|
The opaque faces seems too much, did you didn't set the object optimization mode to high? |
|
I wondered what "object optimization mode" was, so I checked the options and saw it was indeed set to "Low"... |
Add a private cache lock with Store/Report/Prune helpers, clone cache hits outside the lock, and record per-phase plugin load timings.
Route all cache inserts and failure reports through the locked helpers, and read the log via snapshots.
Decode outside the lookup lock with double-check registration, lock all allocator paths and readers, and prune object caches through locked helpers.
Mirror the main program hardening in ObjectViewer and RouteViewer, including log snapshots for all message readers.
Collect Structure declarations during parsing, decode them concurrently with auto-sized parallelism, then commit sequentially so loading screen progress stays monotonic and 100 percent still means playable.
Cycle.Ground and Cycle.Rail validate indices against committed Structure dictionaries. Replaying them after the object barrier restores baseline behavior.
…el decode, freeze-frame and train support
… and loading screen
Use per-file parser instances instead of static mutable state, so parallel Structure decoding cannot mix state across threads.
Unload textures and clear host object caches in DeInitialize before disposing the window; RouteViewer only deinitializes on the final close.
Load each unique object file once and serialize same-path texture registration, so parallel workers no longer parse and decode the same files redundantly.
Replace static host and hack state with per-load parser instances.
Replace static host with an instance field and drop the unused sound-folder hack.
Full ordered teardown (object caches, texture bytes, sound buffers) before loading a different route file.
Force LOH compaction during route-switch teardown and log managed memory plus cache counts before and after.
Null the static RouteData once parsing and applying finish.
…h check Commit 392ad68 removed the width*height*4 truncation copy in CreateTexture that had been masking over-allocation in all 12 decompressors (depth*sizeOfPlane + height*bps + width*bpp). Every DDS then threw ArgumentException (data bytes are not of the expected length) and rendered white. Allocate the exact plane size and defensively truncate in CreateTexture.
…ureInternal) textureCache could hold a null value when GetTexture failed (e.g. missing texture on BVE5/BVE6 routes), crashing on cachedTexture.MultipleFrames. Never cache failed decodes, purge poisoned null entries, guard null Origin in all TryGetValue paths, and route ObjectLibrary through the TryGetCachedTexture/StoreCachedTexture helpers with StoreCachedTexture hardened against null.
biSizeImage may legally be zero for uncompressed (BI_RGB) bitmaps, and pixels are read up to EOF, so the zero-size warning is a false positive there. Restrict it to compressed formats.
The Adam7 path copied 8 raw bytes per pixel into a Width*Height*4 output buffer, overrunning it and silently failing the whole texture. Keep each channel's high byte instead, mirroring the non-interlaced path.
Custom templates are legal in X files (3ds Max exporters emit metadata blocks such as KeyValuePair), but any data instance of an unknown template threw Unrecognised token and failed the whole object. Consume and skip such blocks instead; the nFaces==0 recovery path stays strict.
LoadStructureList decoded every structure file sequentially. Collect entries first, decode deduplicated files with Parallel.For (same DOP formula as the CSV parser), then commit per key in file order with identical semantics, including null entries for failed loads.
2ddf61a to
ac3965b
Compare
|
alright, the commits from the DDS parts is mostly for BVE5/6 routes. now it should as fast as openbve routes. There is still odd fps drop bug when turning on shadow feature when loading bve5/6 routes... idk what it cause.... |
|
Apple appears to be OK, but I haven't done much testing on this at the minute. Viewers seem to be broken on Apple though: That's somewhat odd, as it's just the default shader which is used on the main game which works OK :/ |
|
That fix might work? |
|
Merged, but I've just found this breaks the Object Viewer auto-reload. Might have to disable async for that? |
|
oh yeah its broken... but only when loading extensions.cfg i will trace back, seems easy enough to fix |
|
Definitely broken with just a plain animated file. This is what I noticed it with: (Added a couple of minor functions this morning, was just using a random couple of files to test them with) Timing related I'd assume. |
|
another minor bug, auto load not triggered when changing something inside .animated file that referenced in current loaded extensions.cfg, need manual reload. but changing someting in extensions.cfg works as expected. i think that bug not a big deal? |
|
can you test with #1421 ? |
3 participants
Route loading should covers both .csv and .rw . it was spent most of its time decoding object files one by one on the loader thread: every
Structure.*declaration calledHost.LoadObject()inline before parsing continued.This PR collects all
Structure.*declarations (Rail, Walls, Dikes, Forms, Roofs, Cracks, FreeObj, Beacon, Poles, Ground etc..) during parsing, decodes them concurrently withParallel.For, then commits the results sequentially in file order.Loading behaviour is unchanged in game, so there should be no object popping in game because it loaded late.
Worker count is automatic:
CPU <= 4 ? CPU - 1 : min(8, CPU), reserving one thread for the loading screen on small machines.Parsing the CSV/RW text is cheap, file I/O + object parsing + texture decode is the actual bottleneck. On high multi-core machines the object phase speeds up roughly with core count (currently capped at 8 logical core not physical core count, although disk/RAM speed may become the limit).
Thread-safety hardening
Shared state that was previously only touched single-threaded is now guarded:
HostInterface: add private cache lock withStore/Report/Prune/Clearhelpers, the cache hits are cloned outside the lock; cache inserts upsert instead of throwing on duplicates.Host(main program, ObjectViewer, RouteViewer): failure/missing-file sets updated atomically, messages emitted outside locks.TextureManager: texture decode happens outside the lookup lock with double-checked registration, all allocator paths and readers (load/unload/stats) are also locked.Testing
1fine-Hayabusa1v1.00.csv. The load is around ~11s with reload is ~4s