perf(db): resolve the observed-objects cache in one query - #623
Open
mrosseel wants to merge 1 commit into
Open
Conversation
Since brickbots#528 the observed-objects cache resolves every logged listing to its sky object id, one lookup per listing, and UIObjectDetails builds a new ObservationsDatabase -- so a new cache -- on every entry into the screen. brickbots#564 indexed catalog_objects, which took the per-lookup cost from a ~151k-row scan down to an index probe, so the stall is gone; what is left is a per-entry cost that still grows with the size of the log. - ObjectsDatabase.get_object_ids_by_listings() maps many listings in one chunked query - load_observed_objects_cache() calls it once instead of looping - UIObjectDetails shares one ObservationsDatabase per process, like the existing _catalog_db() handle, so opening the screen no longer rebuilds the cache at all Measured on a Pi 4 with the indexed catalog DB and 138 logged listings: 138 single lookups 7.2 ms, one bulk query 1.6 ms. On the same data with an unindexed DB (a catalog DB built before brickbots#564, before the on-open backfill runs) the loop costs 5.5 s and the single query 0.075 s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #564, from the other side of the same stall.
What is left after #564
#564 indexed
catalog_objects, so each listing lookup is now an index probe instead of a ~151k-row scan, and the ~1.4 s stall is gone. Two things from #528 remain:load_observed_objects_cache()still resolves one lookup per logged listing, so the cache build grows with the size of the log.UIObjectDetails.__init__still builds a newObservationsDatabase— and therefore a new cache — on every entry into the object details screen.Measured on a Pi 4 with the indexed catalog DB and 138 logged listings:
Small in absolute terms today, but it is paid on every screen entry and scales with the log. On the same 138 listings against a catalog DB that has no indexes yet — one built before #564, before
_ensure_catalog_object_indexes()backfills — the loop costs 5.5 s and the single query 0.075 s. That is how I found this: a device in the field took 6.5–8.5 s to open object details.Change
ObjectsDatabase.get_object_ids_by_listings()— maps many listings to object ids in one query, chunked at 400 pairs to stay inside SQLite's variable limit, written as row values so the(catalog_code, sequence)index resolves the whole chunk.load_observed_objects_cache()calls it once instead of looping.UIObjectDetailsshares oneObservationsDatabaseper process, the same pattern as the existing_catalog_db()handle, so opening the screen stops rebuilding the cache.No schema change, no catalog rebuild, no behaviour change: the resulting
observed_object_idsset is identical.Tests
tests/test_objects_db_listings.py(new): bulk lookup matches single lookups, unresolved listings are omitted, chunking works past one chunk.tests/test_observed_identity.py: the test double stubs the bulk seam too, and a new test asserts the cache resolves in exactly one call.ruff checkandruff format --checkpass. Full unit run on macOS: 1232 passed; the only failures are 5 intests/test_comets.py, from skyfield version skew in my local environment, untouched by this change.🤖 Generated with Claude Code