Repository navigation
feat(properties): listing metrics, functional search indexes, search cache and search rate limit - #1475
Open
m-hajjo wants to merge 2 commits into
Open
feat(properties): listing metrics, functional search indexes, search cache and search rate limit#1475m-hajjo wants to merge 2 commits into
m-hajjo wants to merge 2 commits into
Conversation
…cache, search rate limit Performance and observability for the public listing read path (`GET /properties` and `GET /properties/search`), following the conventions of the Users API pass in MettaChain#1457-MettaChain#1463 and complementing MettaChain#1367-MettaChain#1370, which covered the property write/detail surface. - MettaChain#1383: `properties_listings_served_total{surface,outcome}`, `properties_listing_response_seconds{surface}` and `properties_listing_results_returned{surface}` from `ListingsMetricsInterceptor`, plus `properties_listing_cache_lookups_total{outcome}` from the cache service. Every label set is fixed (2/3/3 values), so cardinality cannot grow with request input; `surface` is derived from the request path rather than the handler name so a new handler cannot silently widen the label space. The results histogram makes a broken filter visible long before anyone reports it. docs/api/listings-metrics.md - MettaChain#1384: the case-insensitive search filters compile to `LOWER("city") = LOWER($1)`, which a plain btree index on the raw column can never serve - so `@@index([city, state])` and `@@index([propertyType])` were never chosen by the planner for the filters they exist for. 20261001000001_add_listing_search_functional_indexes indexes the expression instead, leading with (status, is_deleted) so a filtered search is one index seek: LOWER(city), LOWER(state), LOWER(property_type), a partial LOWER(category), GIN on the tags array for `{ has: tag }` containment, and LOWER(amenity_type) on property_amenities. Prisma cannot express functional or partial indexes in the schema, so they are intentionally not declared there and the doc says so - if a future `migrate dev` proposes dropping them, re-create them. docs/api/listings-db-optimization.md - MettaChain#1385: GET /properties/search is read through PropertiesListingsCacheService, TTL 30s, registered under CACHE_TAGS.PROPERTIES. The key is a sha1 of the filter set with entries sorted and absent values dropped, so query-string order and omitted-vs-undefined cannot produce two keys for one filter set. An empty filter set is deliberately NOT cached: that is the full feed, which changes constantly. Keys over 2KB are skipped so a pathological query cannot create an unbounded Redis key. Cache errors fall through to the database. docs/api/listings-caching.md - MettaChain#1386: GET /properties/search had no entry in ENDPOINT_RATE_LIMITS at all, so being unauthenticated it was bounded only by the global per-IP bucket shared with the whole API. It is the most expensive read in the module - up to 25 filters, an optional in-memory trust-score pass and a COUNT per page - so one client could spend that entire budget on search alone. Now 60 per minute, against 100/min for the plain list. The existing global guard was reused rather than adding a route guard class with its own DI surface. docs/api/listings-rate-limiting.md PropertiesListingsCacheService is injected with @optional(), so PropertiesService still constructs without it and the existing unit test module is unaffected. Nothing was installed, built, linted or tested; please verify in CI. Closes MettaChain#1383 Closes MettaChain#1384 Closes MettaChain#1385 Closes MettaChain#1386
|
@m-hajjo Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance and observability for the public listing read path (
GET /propertiesandGET /properties/search). This complements #1471, which covered the property write/detail surface, and follows the conventions of the Users API pass in #1457–#1463.#1384 — Database query optimisation (the big one)
The case-insensitive search filters could never use an index.
GET /properties/searchfilters with Prisma'smode: 'insensitive', which compiles to:A plain btree index on the raw column is unusable for that expression — the indexed values are
Austinwhile the lookup key isLOWER("city"). So@@index([city, state])and@@index([propertyType])were never chosen by the planner for the filters they were added for, which are among the most common in the product. Same forcountry,categoryand the amenity join.prisma/migrations/20261001000001_add_listing_search_functional_indexesindexes the expression instead, leading with the listing predicate so a filtered search is a single index seek:(status, is_deleted, LOWER(city))?city=(status, is_deleted, LOWER(state))?state=(status, is_deleted, LOWER(property_type))?propertyType=(status, is_deleted, LOWER(category)) WHERE category IS NOT NULL?category=GIN (tags)where: { tags: { has: tag } }LOWER(amenity_type)onproperty_amenities?amenityType=?tags=is array containment, which needs GIN — btree cannot answer it.?category=is partial because the column is optional, so indexing the empty majority would cost space and scan time for nothing.Deliberately not done: the free-text
?location=filter doescontainsacross address/city/state/zipCode and really wantspg_trgm+ GIN or Postgres FTS. That needs the extension enabled on the database, which is a platform decision rather than something a migration should make unilaterally. Flagged for follow-up in the doc.→
docs/api/listings-db-optimization.md#1385 — Redis caching
GET /properties/searchis read throughPropertiesListingsCacheService: keyproperties:search:<sha1 of the normalised filter set>, TTL 30 seconds, registered underCACHE_TAGS.PROPERTIESso a property write invalidates it with the rest.GET /propertiesis not cached. Its result set changes on almost every write and it is the cheaper of the two queries — caching it buys mostly invalidation risk.undefinedcannot produce two keys for the same filter set.PropertiesListingsCacheServiceis injected with@Optional(), so the service still constructs without it and the existing unit-test module is unaffected.→
docs/api/listings-caching.md#1383 — Prometheus metrics
properties_listings_served_totalsurface(list/search),outcomeproperties_listing_response_secondssurfaceproperties_listing_results_returnedsurfaceproperties_listing_cache_lookups_totaloutcome(hit/miss/bypass)Every label set is fixed — 2, 3 and 3 values — so cardinality cannot grow with request input, which is what makes these safe to scrape at 1% resolution.
surfaceis derived from the request path rather than the handler name, so adding a handler cannot silently widen the label space.Two worth alerting on:
properties_listing_results_returned— a histogram of how many listings each response carried. A drop toward zero at flat request volume means the filters or index are wrong, visible long before someone reports "search is broken"...._cache_lookups_total{outcome="bypass"}— separates a query that was never cacheable from a cache that errored. A rising bypass share usually means Redis is unhealthy, visible before latency degrades.→
docs/api/listings-metrics.md#1386 — Rate limiting
GET /properties/searchhad no entry inENDPOINT_RATE_LIMITSat all. Both listing routes are unauthenticated, so search was bounded only by the global per-IP bucket — shared with every other endpoint on the API. A single client could spend that whole budget on search alone, and search is the most expensive read in the module: up to 25 filters, an optional in-memory trust-score pass, and aCOUNTissued alongside every page.GET /propertiesGET /properties/searchSixty per minute is above what a person paging through filters needs, while cutting the worst case by roughly 40%. Unauthenticated requests are keyed by client IP (hashed, and
x-forwarded-foronly consulted whenTRUST_PROXYis set, per #1195); authenticated callers are additionally bounded by their own user-tier bucket.A per-route guard class was written first and then rejected: the global
RateLimitGuardalready provides this behaviour, and a new@UseGuardsclass would have added DI surface (aRateLimitServicedependency) to every integration test that bootsPropertiesControllerin isolation.→
docs/api/listings-rate-limiting.mdNot run
Nothing was installed, built, linted or tested — please verify in CI, in particular that the functional indexes apply cleanly to your Postgres version.
Closes #1383
Closes #1384
Closes #1385
Closes #1386