Repository navigation
Remediation: ESPI REST controller modernization (post-#116) #119
Description
Issue #98 Remediation Plan (originally tracked as PR #115)
Re-aimed 2026-05-20: This plan was originally written against PR #115's branch. PR #115 was closed as superseded by PR #116 (merged 2026-05-20), which landed a renamed and slimmer subset of #115's scope. The dangerous bits that were the original blockers (
ManageControllershell exec,web/api/RetailCustomerControllerwrite methods,web/api/ServiceStatusController) did not merge and remain as.disabledfiles onmain.The plan below has been re-scoped to target what actually landed in
mainvia #116, plus pre-conditions for safely re-enabling the.disabledfiles later. File path references generally use the*RESTController.javanaming that exists on main; per-phase paths should still be verified at execution time.Source review: see
/review PR #115findings.
Organized into 8 phases (and one deferred bucket) — each phase is self-contained, can be a separate commit, and has its own acceptance criteria. Ordered by risk-first, then dependency (CI unblock → data correctness → structural cleanup → polish → tests).
Phase 1 — De-stream REST controllers (3–4 hr) ⚡ NEXT
Surfaced by PR #118 CI failure: MeterReadingControllerTest.shouldReturn200WhenExists throws java.util.ConcurrentModificationException because MeterReadingController returns a StreamingResponseBody, and the streaming worker thread races with the test thread on MockHttpServletResponse's LinkedCaseInsensitiveMap while Spring Security's OnCommittedResponseWrapper writes security headers.
StreamingResponseBody is used by every GET endpoint PR #116 added (9 controllers, ~30+ endpoints): ApplicationInformationRESTController, CustomerAccountRESTController, CustomerRESTController, IntervalBlockRESTController, ElectricPowerQualitySummaryRESTController, ReadingTypeRESTController, UsagePointController, UsageSummaryRESTController, MeterReadingController.
Why drop it (not just patch the test): What's on main today isn't real streaming — it's StreamingResponseBody wrapping JAXB-marshalled bytes. The service builds the full DTO graph, JAXB marshals it to a complete byte[], and only then does the streaming worker dribble that already-buffered payload out. The whole response is in heap before the first byte hits the wire, so removing StreamingResponseBody loses zero real streaming benefit — including on /Subscription/* endpoints. In exchange, the "fake streaming" costs: race conditions with Spring Security headers (the CME), per-request WebAsyncManager worker thread overhead, permanent asyncDispatch() tax on every MockMvc test, harder operational debugging, forced chunked transfer encoding. It's premature optimization that introduced concrete bugs. Prerequisite for Phases 2 and 3 — subscription scoping is much simpler against sync controllers, and Atom link/UUID work is much cleaner without async thread-context concerns. Also unblocks CI — until this lands, the CME re-surfaces randomly and fails arbitrary PRs.
- 1.1
ApplicationInformationRESTController— replaceResponseEntity<StreamingResponseBody>returns withResponseEntity<String>(already-marshalled XML) orResponseEntity<EntryDto>/ResponseEntity<FeedDto>+ JAXB message converter - 1.2
CustomerAccountRESTController - 1.3
CustomerRESTController - 1.4
IntervalBlockRESTController - 1.5
ElectricPowerQualitySummaryRESTController - 1.6
ReadingTypeRESTController - 1.7
UsagePointController - 1.8
UsageSummaryRESTController - 1.9
MeterReadingController - 1.10 Remove all
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;lines that are no longer referenced - 1.11 Update corresponding MockMvc tests: drop any
request().asyncStarted()/MvcResult/asyncDispatch()wrappers (added or about to be added to work around the CME) - 1.12 Run
mvn verify -pl openespi-datacustodian -am3× to confirm no remaining flakes - 1.13 Confirm
MeterReadingControllerTest.shouldReturn200WhenExistspasses deterministically
Acceptance: No StreamingResponseBody references remain in openespi-datacustodian controllers; full datacustodian test suite passes 3 consecutive clean runs; no MockMvc test uses asyncDispatch (unless a deliberately-async endpoint exists, which currently it does not).
Production utility implementations of /Subscription/* and /Batch/Bulk/* likely DO need real streaming — a year of 15-min interval data across many usage points for one customer can exceed 100 MB, and a bulk export across thousands of customers is unbounded. But "real streaming" is not StreamingResponseBody + JAXB. Real streaming means:
- Open a streaming JPA cursor (
@QueryHints(@QueryHint(name=HINT_FETCH_SIZE, value="100"))returningStream<Entity>inside a@Transactional(readOnly=true)) orJdbcTemplate.queryForStream(...) - Use a StAX
XMLStreamWriterpointed atresponse.getOutputStream() - Write the Atom feed envelope, then per-entry: marshal one DTO →
flush()→ advance the cursor - Write the feed footer and close the stream
That's the architecture that delivers actual memory bounding and progressive transmission. It belongs in production utility forks of this codebase; this Sandbox / reference implementation doesn't need it (response sizes are small by definition of "sandbox", and the test/CI cost of streaming far outweighs the benefit). When a fork does need it, implement it endpoint-by-endpoint with explicit StAX + cursor — do not reach for StreamingResponseBody as a shortcut, since it delivers the test-flake costs without the memory benefits.
Phase 2 — Subscription scoping (data-leakage fix) (3–5 hr)
Currently a third-party token gets unfiltered global lists. Model on UsagePointController which does this correctly.
For each controller below, the pattern is:
- Inject
SubscriptionService(or whateverUsagePointControlleruses) - Resolve subscription from access token via
ApiAccessValidator.requireSubscriptionId(authHeader) - For non-admin scopes, filter the result set to entities belonging to that subscription's authorized usage points
- Admin scope retains unfiltered access
- 2.1
IntervalBlockRESTControllerlist endpoints - 2.2
MeterReadingControllerlist endpoints - 2.3
ElectricPowerQualitySummaryRESTControllerlist endpoints - 2.4
ReadingTypeRESTController— note ReadingType is shared metadata; verify scoping semantics with the spec before filtering - 2.5
UsageSummaryRESTControllerlist endpoints - 2.6
CustomerRESTControllerlist endpoints - 2.7
CustomerAccountRESTControllerlist endpoints - 2.8 Add MockMvc test per controller: third-party token sees only their subscription's data; admin token sees all
Acceptance: Each list endpoint has an integration test proving that two different subscription tokens get disjoint result sets, and admin sees the union.
Phase 3 — ESPI 4.0 Atom + JSON compliance (4–6 hr)
The most important compliance fix. Touches both code and conventions.
- 3.1
BaseExportService.createServiceStatusEntry(if present on main — verify) — replaceselfLink.setHref("ServiceStatus")/upLink.setHref("")with the canonical patternhttps://{host}/DataCustodian/espi/1_1/resource/{resource}/{id}. Inject host from config (espi.base-urlor equivalent). - 3.2 Audit every
createAtomEntry(title, dto)callsite inDtoExportServiceImpland ensure the entry has:rel="self"href with full resource URIrel="up"href with full parent collection URIrel="related"hrefs where applicable
- 3.3 Replace all
UUID.randomUUID()in Atom-id generation with UUID Version 5 (deterministic from resource URI). Use the project's existingespiIdGeneratorServiceif available; otherwise add a helper. - 3.4 Verify entry IDs serialize as
urn:uuid:xxxxxxxx-xxxx-5xxx-xxxx-xxxxxxxxxxxx - 3.5 Remove
produces = APPLICATION_JSON_VALUEfrom every ESPI resource endpoint on main:ApplicationInformationRESTController,AuthorizationController,CustomerAccountRESTController,CustomerRESTController,ElectricPowerQualitySummaryRESTController,IntervalBlockRESTController,MeterReadingController,ReadingTypeRESTController,UsagePointController,UsageSummaryRESTController. Keep XML only —APPLICATION_XML_VALUEorAPPLICATION_ATOM_XML_VALUEas appropriate. - 3.6 Add an integration test that marshals one feed + one entry per resource and validates against
openespi-common/src/main/resources/schema/ESPI_4.0/espi.xsdusing aValidatorfromSchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI).newSchema(espiXsd)
Acceptance: All ESPI resource feeds/entries pass XSD validation; Atom IDs are UUID v5; no JSON content-type is offered for ESPI resources.
Phase 4 — ID type cleanup (1–2 hr)
Wider than originally scoped — Long retailCustomerId is spread across MVC controllers on main, not just one method.
- 4.1
DtoExportService.createRetailCustomerEntry(Long retailCustomerId)→UUID retailCustomerId(verify signature still exists on main) - 4.2
openespi-datacustodian/.../web/custodian/AssociateUsagePointControllerlines 65 / 74 —Long retailCustomerId→UUID - 4.3
openespi-datacustodian/.../web/custodian/RetailCustomerController:85—Long retailCustomerId→UUID - 4.4
openespi-datacustodian/.../web/customer/CustomerDownloadMyDataControllerlines 46 / 64 —Long retailCustomerId→UUID - 4.5
openespi-datacustodian/.../web/customer/MeterReadingController:45—Long retailCustomerId→UUID - 4.6
openespi-datacustodian/.../web/customer/UsagePointController:88—Long retailCustomerId→UUID - 4.7
openespi-thirdparty/.../web/MeterReadingController:45—Long retailCustomerId→UUID - 4.8
openespi-thirdparty/.../web/UsagePointController:80—Long retailCustomerId→UUID - 4.9
openespi-thirdparty/.../web/custodian/RetailCustomerController:90—Long retailCustomerId→UUID - 4.10 Final sweep:
git grep -nE "Long\\s+\\w*[Cc]ustomerId" openespi-datacustodian openespi-common openespi-thirdpartyreturns empty - 4.11 Verify
RetailCustomerEntityPK is UUID per CLAUDE.md — if it's stillLong, that's a separate (larger) issue: file it as a new issue
Acceptance: No Long-keyed RetailCustomer references in datacustodian, common, or thirdparty modules. All controller path variables and service signatures use UUID.
Phase 5 — Constructor + facade structural cleanup (2–3 hr)
Verify scope on main first — the renamed/slimmed #116 may have already partially addressed this. If DtoExportServiceImpl constructor on main is no longer 11-arg, scale this phase down.
- 5.1 Verify on main: count
DtoExportServiceImplconstructor parameters. If ≤4, mark this phase N/A. If still large:- 5.1a (preferred) Switch DTO tests that enumerate dependencies to Mockito
@InjectMocksso they no longer passnull, null, null, … - 5.1b Extract a small
MarshallerOnlyExporterfromDtoExportServiceImplthat the DTO tests can construct cheaply
- 5.1a (preferred) Switch DTO tests that enumerate dependencies to Mockito
- 5.2 Verify on main: is
DtoExportServiceFacadepresent? If yes, either delegate all feed/entry methods toDtoExportServiceImpl, or removeDtoExportServiceImplfrom the facade's deps and centralize logic in the facade. Do not keep both code paths. - 5.3 Verify: are there still two
createServiceStatusEntryimplementations? If yes, collapse into one. - 5.4 Confirm
@Qualifier("dtoExportServiceImpl")resolves correctly (default Spring bean name should match)
Acceptance: Adding a new dependency to DtoExportServiceImpl does not require touching DTO test files. At most one createServiceStatusEntry exists.
Phase 6 — Pagination & behavior polish (2 hr)
Verify each item against main first. Several originally-listed controllers (TimeConfigurationController, RetailCustomerController.index(), BatchController) are still .disabled on main — those sub-items should move into the Deferred bucket below.
Applicable on main:
- 6.1
ReadingTypeRESTControllerlist endpoint — implementlimit/offset(pass to a paginated repository call) or drop the parameters - 6.2
OffsetPageable.getPageNumber()(if present on main) — replaceoffset / limitwithMath.floorDiv(offset, limit); add a unit test for non-multiple offsets - 6.3
ApiAccessValidator.requireSubscriptionId(if present on main) — return401 UNAUTHORIZEDfor missing/empty token,403 FORBIDDENonly for valid-but-insufficient scope
Acceptance: Pagination parameters either work or are removed; HTTP status codes follow spec.
Phase 7 — CustomerMapper correctness (1 hr)
Verify scope on main first — confirm CustomerMapper.toDto(RetailCustomerEntity) exists on main with the issue. If the method wasn't merged, defer to when it gets re-introduced.
- 7.1
CustomerMapper.toDto(RetailCustomerEntity)— null-safe name concatenation (skip nulls, trim, returnnullrather than"null null") - 7.2 Stop populating
OrganisationDto.organisationNamewith a person name. Either leave it null when the source is aRetailCustomer, or split the mapping into a person-customer branch and an organisation-customer branch - 7.3 Add a unit test covering: all-null name, first-only, last-only, both, organisation case
- 7.4 Design question to raise: should
RetailCustomerEntityeven marshal as aCustomer(different schema)? If not, deprecate this mapper method and route consumers to the proper resource
Acceptance: Tests cover null permutations; no "null null" artifact possible.
Phase 8 — Test fortification (3 hr)
- 8.1 XSD-validation integration test (Phase 3.6) confirmed running in
mvn verify - 8.2 Subscription-scoping tests from Phase 2.8 confirmed running
- 8.3 Run full
mvn verify -pl openespi-common,openespi-datacustodian -amclean
Acceptance: All new tests green; no regressions in existing suite.
Deferred — Pre-conditions for re-enabling .disabled controllers
These items were originally Phase 2 blockers for PR #115 but did NOT land in main via #116. They live in .disabled files. Apply these fixes as part of the same PR that re-enables each file — never re-enable as-is.
- D.1
web/api/ManageRESTController.java.disabled— removeresetDatabase/initializeDatabaseREST endpoints before re-enabling. Two options:- Option A (preferred): delete them entirely; move admin operations to a non-HTTP CLI
- Option B: gate behind
SCOPE_DataCustodian_Admin_Accessand@Profile("!prod"), and never echo stderr into HTTP response
- D.2
web/api/RetailCustomerRESTController.java.disabled— add@PreAuthorize("hasAuthority('SCOPE_DataCustodian_Admin_Access')")tocreate(),update(),delete()before re-enabling (or delete those methods if they're still 501 stubs) - D.3
web/api/ServiceStatusRESTController.java.disabled— replaceauthHeader.substring(7)with a guarded extraction (startsWith("Bearer ")ANDlength() > 7AND non-empty token, else 401) before re-enabling - D.4
web/api/ApplicationInformationController.createApplicationInformation(if/when added) — add@Validand bean-validation constraints onclientSecret,redirectUri,scope,tosUri,clientName - D.5
BatchRESTController.java.disabled— when re-enabling:subscriptionUsagePoint(...)must filter by the subscription's specific usage points, not delegate tosubscription(...)upload()/.bulk()must log when called (log.warn("Unimplemented endpoint called: {}", …)) before returning 501downloadCollection(@PathVariable Long retailCustomerId, …)→ useUUID(also covered by Phase 4)- Streaming approach for
/Batch/Bulk/*and/Batch/Subscription/*— these endpoints can legitimately return 100+ MB in production utility deployments. If/when this Sandbox repo needs real streaming for them, implement it as StAXXMLStreamWriter+ a streaming JPA cursor (orJdbcTemplate.queryForStream) marshalling one entry at a time. Do not wrap JAXB-marshalled bytes inStreamingResponseBody— that's the fake streaming pattern Phase 1 just removed; it has no memory benefit and brings back the CME footgun. For Sandbox/CI purposes, a synchronous response of bounded test-data size is fine; the streaming path is only worth the engineering investment if a production fork pulls this code as-is.
- D.6
TimeConfigurationRESTController.java.disabled— when re-enabling, honorlimit/offsetor drop them - D.7
ServiceStatusController— replace hardcodedapplicationStatus = "0"with aDataCustodianApplicationStatus.OKconstant/enum - D.8 Audit token logging in
AuthorizationService.findByAccessTokenand any new auth-handling code — ensure raw tokens are never logged
Acceptance: Every .disabled file ships with the corresponding security/correctness fix in the same PR.
Cross-cutting reviewer checklist (apply per commit)
- Edit-tool diffs only (no
sed/awkper CLAUDE.md) - No new JAXB on entities, no new JPA on DTOs
- No new
Long-keyed ESPI entities - AssertJ assertions chained per CLAUDE.md
-
mvn test -pl <touched-module> -amgreen before each commit - PR title uses conventional-commit format (
feat:,fix:,chore:, etc. with colon) to clear PR Validation
Suggested phase grouping into PRs (if breaking up)
- PR-A: Phase 1 (de-stream controllers) — mechanical refactor, unblocks CI, prerequisite for PR-B and PR-C
- PR-B: Phase 2 (data-leakage fix) — high-impact, focused review
- PR-C: Phase 3 (ESPI compliance) — touches many files, dedicated review
- PR-D: Phases 4 + 5 + 6 + 7 (cleanup & polish)
- PR-E: Phase 8 (test fortification) — could fold into A–D, or stand alone
Deferred items (D.1–D.8) ship one-by-one with whichever PR re-enables the corresponding .disabled file.
Total estimated effort: ~18–23 hours focused work on the 8 active phases, 5 PRs if split. Deferred items add ~3–5 hours when their respective .disabled files are re-enabled.
Change log
- 2026-05-20 Initial plan written against PR feat - Issue 98, implementing remaining controller functionality #115 review findings.
- 2026-05-20 Phase 2.5 added (de-stream controllers) after PR chore: ignore npm tooling artifacts in .gitignore #118 CI failure exposed the
StreamingResponseBody× Spring Security CME pattern. - 2026-05-20 Plan re-aimed at
mainafter PR feat - Issue 98, implementing remaining controller functionality #115 closed (superseded by feat: Modernize ESPI REST controllers (Issue #98, no auth-server migration) #116). Dropped original Phases 0 and 1 (no longer applicable). Promoted Phase 2.5 → Phase 1. Renumbered subsequent phases. Moved original Phase 2 security items to Deferred bucket (the dangerous code lives in.disabledfiles, not on main yet). Updated file path references to*RESTController.javawhere applicable. Widened Phase 4 ID cleanup scope to include MVC controllers across all three modules.
Activity
refactor: de-stream REST controllers to remove MockMvc CME flake (#120)
dfcoffin commentedon May 21, 2026
Phase 1 complete in #120 (merged 2026-05-21 as 797b4dd). CME flake resolved across all 9 controllers / 26 endpoints; CI now deterministic. Phase 2 (subscription scoping) is next.
dfcoffin commentedon May 22, 2026
Phase 2 — Data Leakage Fix
Companion to Issue #119, Phase 2. Audits the current active REST API surface
in
openespi-datacustodianafter PR #120 and PR #121, identifies the IDOR /cross-subscription data-leakage exposure, and proposes the fix design.
This is a design draft for issue-comment posting, not a committed artifact.
Several precursors (running auth-server, scope-grammar alignment, JSON output)
must land or be acknowledged before any code in this document is written.
Generated: 2026-05-22 (revised to opaque-token introspection model).
1. Active REST API surface
Base path:
/espi/1_1/resource(all controllers).Response bodies are
byte[]containing JAXB-marshalled XML. Controllers declareboth
application/xmlandapplication/jsonas producible, but only XML isactually emitted today. Real JSON support is a separate Phase 3 work item; the
JSON content-type advertisement is currently cosmetic and misleading.
1.1 GET endpoints
@PreAuthorize(as written)GET /UsagePointlimit=50,offset=0GET /UsagePoint/{usagePointId}GET /Subscription/{subscriptionId}/UsagePointlimit=50,offset=0GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}GET /MeterReadinglimit=50,offset=0GET /MeterReading/{meterReadingId}GET /ReadingTypelimit=50,offset=0GET /ReadingType/{readingTypeId}GET /IntervalBlocklimit=50,offset=0GET /IntervalBlock/{intervalBlockId}GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}/MeterReading/{meterReadingId}/IntervalBlockGET .../IntervalBlock/{intervalBlockId}(same prefix)GET /UsageSummarylimit=50,offset=0GET /UsageSummary/{usageSummaryId}GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}/UsageSummarylimit=50,offset=0GET .../UsageSummary/{usageSummaryId}GET /ElectricPowerQualitySummarylimit=50,offset=0GET /ElectricPowerQualitySummary/{electricPowerQualitySummaryId}GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}/ElectricPowerQualitySummarylimit=50,offset=0GET .../ElectricPowerQualitySummary/{electricPowerQualitySummaryId}GET /Customerlimit=50,offset=0GET /Customer/{customerId}GET /CustomerAccountlimit=50,offset=0GET /CustomerAccount/{customerAccountId}GET /ApplicationInformationGET /ApplicationInformation/{applicationInformationId}1.2 Mutating endpoints (CRUD)
@PreAuthorizePOST /Customer(XML/JSON in & out)PUT /Customer/{customerId}DELETE /Customer/{customerId}POST /CustomerAccount(JSON only)PUT /CustomerAccount/{customerAccountId}(JSON only)DELETE /CustomerAccount/{customerAccountId}POST /ApplicationInformation(JSON only)PUT /ApplicationInformation/{id}(JSON only)DELETE /ApplicationInformation/{id}1.3 Observations
Authentication authenticationis injected on most handlers but neveractually read — vestigial wiring from the original modernization.
produces = { APPLICATION_XML, APPLICATION_JSON }is declared everywhere, butthe body is JAXB-serialized XML bytes regardless. JSON content-type
negotiation is cosmetic; real JSON output requires Phase 3.
/Subscription/...handler carries a// TODO: implement subscription-based filteringcomment in the body — thesecurity check was deferred during modernization and never landed.
2. Threat model
A third-party application obtains an OAuth2 access token bound to subscription
S_a(granting access to retail customer A's data). With that token they cantoday exercise three distinct IDOR (CWE-639) paths:
GET /UsagePointreturns everyUsagePoint across all subscriptions.
GET /UsagePoint/{anyId}performs no ownershipcheck. UUIDv5 reduces guessability but does not eliminate it (especially
when IDs leak through Atom
selfLink/relatedURIs in other responses).GET /Subscription/{S_b}/UsagePointreturns all data; the server doesn't compare
{S_b}to the token's binding,doesn't filter by it, and the path variable is effectively decorative.
All three apply across UsagePoint, MeterReading, IntervalBlock, UsageSummary,
ElectricPowerQualitySummary, Customer, and CustomerAccount. ReadingType is
metadata (intentionally cross-subscription) and ApplicationInformation is
admin-only — both excluded.
3. Token model — opaque, NOT JWT
ESPI 4.0 requires opaque (reference) access tokens. JWT-style access tokens
are not permitted by the spec. This dictates the entire enforcement design:
It has bytes, not a structured payload.
RFC 7662 token introspection against the authorization server.
OpaqueTokenIntrospectoras the abstraction. Thedefault implementation (
SpringOpaqueTokenIntrospector) calls/oauth2/introspectand maps the response into anOAuth2AuthenticatedPrincipal.subscription_id,retail_customer_id, scope list, and admin status.These must be added to the auth-server's introspection response shape if
they aren't already.
4. Fix design
Two-layer defense, built around
OpaqueTokenIntrospector.4.1 Layer 1 — introspection-driven principal
A custom
OpaqueTokenIntrospectorbean wraps the default. It:by the token hash, TTL ≤ token remaining lifetime, ≤ 60s upper bound).
subscription_idand admin status from the response.EspiPrincipalcarrying(tokenId, subscriptionId, retailCustomerId, scopes, isAdmin).OAuth2AuthenticatedPrincipal, so it lands in theSecurityContext.Admin tokens (carrying the configured admin scope — exact string deferred to
the Phase 7 scope-grammar work) yield
isAdmin = trueand bypass binding.Non-admin tokens must carry a subscription binding in the introspection
response or the request is rejected
401 Unauthorized. Silent fall-through isexplicitly forbidden — that's the bug we're fixing.
4.2 Layer 2 —
ApiAccessValidatorhelperLocation:
openespi-datacustodian/src/main/java/.../security/ApiAccessValidator.java.The helper consumes the principal that the introspector populated — it does
not know or care that the underlying token is opaque. That's the layering
benefit.
4.3 Per-handler enforcement patterns
Three patterns, applied by endpoint shape:
/Subscription/{subscriptionId}/...enforcePathMatch(pathSubId, tokenSubId). Then filter the query bytokenSubId.GET /MeterReading)tokenSubIdviafindAllBySubscriptionId(...). Admin: unfiltered.GET /MeterReading/{id})enforceOwnership(entity.getSubscriptionId(), tokenSubId). Admin: unrestricted.4.4 Repository additions
Each affected repository gains
findAllBySubscriptionId(UUID, Pageable)(orequivalent named query). For nested resources (IntervalBlock → MeterReading →
UsagePoint → Subscription) the query joins the chain rather than denormalizing.
The data model audit (precursor — see §6) confirms whether
UsagePointEntitycarries a directsubscription_idFK or only transitivelyvia
RetailCustomerEntity. That determines whether repo methods are derivedqueries or explicit JPQL with joins.
4.5 Tests (per endpoint, three minimum)
401403200with only matching-subscription data (assert pagination andfiltering both apply; pagination without filtering is the silent-bypass mode)
A
MockOpaqueTokenIntrospectortest fixture returns a configurable principalwithout requiring a live auth-server. That fixture is the most important new
test infrastructure in Phase 2.
5. Sub-phase sequencing
openespi-authserverin dev; confirm it issues opaque tokens and that its introspection response carriessubscription_id(add it if not).OpaqueTokenIntrospectorandApiAccessValidator. Fix the 8/Subscription/...endpoints (rows 3, 4, 11, 12, 15, 16, 19, 20).ReadingType (rows 7, 8) and ApplicationInformation (rows 25, 26, M7–M9) are
intentionally exempt — metadata and admin-only respectively.
6. Precursors to resolve before coding 2a
openespi-authserverto issue opaquetokens (
OAuth2TokenFormat.REFERENCE), expose/oauth2/introspect, andinclude
subscription_id(and admin flag) in the introspection response.Register the data-custodian as a Resource Server pointing at it.
UsagePointEntity(and the otherleaf entities) carry a
subscription_idFK directly or only transitively.Dictates repository query shape.
SCOPE_FB_15_READ_3rd_partyshorthand and parses it identically on bothsides, or waits for Phase 7's spec-compliant FB-string grammar. The
introspector design is neutral; the
@PreAuthorizelines are not.produces = JSONdeclarations stay as-is during Phase 2 (real JSON outputis Phase 3). Otherwise the scope of this work expands considerably.
Resolving 1 and 2 is the first concrete work of 2.0.
dfcoffin commentedon May 22, 2026
Phase 2 is blocked by #122 (auth-server bring-up + opaque-token introspection). No Phase 2a code will be merged until #122's acceptance criteria (one end-to-end integration test minting a real token and reaching a data-custodian endpoint with a populated subscription binding) are met.
dfcoffin commentedon May 26, 2026
Follow-up checklist item from PR #116 notes
Surfaced in
PR_116_FOLLOWUP_NOTES.md(item 5) and folded into this remediation issue as it's part of ESPI REST controller correctness:MapStruct mappers don't populate
IdentifiedObjectbase fields. The entity→DTO mappers omit the inheritedIdentifiedObjectfields —id,selfLink,upLink,relatedLinks,published,updated— which are required for ESPI Atom feed compliance (self/up/related links and resource identity in every feed entry). Without them, marshalled Atom output is missing the link structure the ESPI 4.0 objective requires.Suggested acceptance:
id,selfLink,upLink,relatedLinks,published,updatedon the DTO from the entity (or via the Atom-feed assembly layer)espi.xsdwith self/up/related links presentdfcoffin commentedon Jun 5, 2026
Master remediation roadmap (2026-06-05) — sequenced to close all validly-open issues
Backlog triaged after the Phase-2.0 cross-service-contract chain landed (#122/#148/#149/#150/#157/#158/#160). This issue (#119) is the canonical roadmap home — kept current here rather than in a repo doc.
Closed during triage: #112 (obsolete — SAS stays its own module), #132 (resolved —
StreamingResponseBodyremoved in #119 Phase 1, CME gone).Already done on
main: #119 Phase 1 (de-streaming) ✅.Waves (DC-first; TP last per project lead)
controll-update-status.md;@accountSecurityServiceref already gone), fix: Cross-platform timestamp test failures due to OS timestamp resolution differences #78 (isAfter→isAfterOrEqualTo, 5 tests — fixes Windows-local failures), Configure Spotless Maven plugin or remove CI formatting check #117 (Spotless: configure or remove the CI check).UsagePointController, built on the mergedEspiScopeOpaqueTokenIntrospector/DC scope enforcement vocabulary doesn't match AS-issued ESPI FB scopes — customer tokens 403 on all resources #157. Two-token disjoint-result tests.Long retailCustomerId→UUID). → closes Verify Entity Field Types and Lengths Match ESPI 4.0 XSD Schemas #101, ApplicationInformation schema drift from ESPI 4.0 XSD across MySQL/PostgreSQL/H2 V1 migrations #123.chk_risk_score_range; re-enablePostgreSqlTestcontainersIntegrationTest) → Complete #123 schema-repair sync: align AS V1 schema with application code column names #145 (V8 rename, re-enable V3–V6, drop Flyway cap) → Rebuild AS OAuth2 flow integration suite on AbstractAuthserverIntegrationTest #151 (rebuild OAuth2-flow IT on the Repair AS integration test infrastructure (pre-existing pre-#122) #149 harness) → Auth-server tests are broken AND excluded from CI — repair test infrastructure, then re-enable in CI #129 (repair broad AS unit suite + full suite into CI). → closes AS PostgreSQL migration V2 attaches chk_risk_score_range to wrong table #152, Complete #123 schema-repair sync: align AS V1 schema with application code column names #145, Rebuild AS OAuth2 flow integration suite on AbstractAuthserverIntegrationTest #151, Auth-server tests are broken AND excluded from CI — repair test infrastructure, then re-enable in CI #129..disabledre-enablement: AuthorizationController stub bodies need implementation + per-TP filtering #141 (Authorization GET bodies + per-TP filtering) + re-enable the 7 DC.disabledcontrollers each with its Remediation: ESPI REST controller modernization (post-#116) #119-Deferred security fix + Remediation: ESPI REST controller modernization (post-#116) #119 Phases 5–8. → closes AuthorizationController stub bodies need implementation + per-TP filtering #141, Complete Get Controller Functionality #98, Remediation: ESPI REST controller modernization (post-#116) #119.* ReadingType is shared metadata — confirm scoping semantics against the spec before filtering.
dfcoffin commentedon Jun 5, 2026
⭐ CANONICAL BUILD PLAN — DC ESPI Resource Surface (GET-first) — 2026-06-05
(Supersedes the high-level wave note above. Source: Don's authoritative API-surface xlsx — DC Usage Root/XPath/Collection + Customer XPath sheets.)
Direction
Build the canonical ESPI resource API surface first (ROOT + XPath, collection + individual), via a generic
EspiResourceControllerbase +ScopeResolver. Subscription scoping becomes intrinsic (theScopeStrategy), ESPI/Atom/XSD compliance is the base's emit contract, andBatch/Subscription|Bulk|RetailCustomercompose over the surface. This single workstream delivers #98 and subsumes #119 Ph2–Ph8 + #101 (validated continuously).Confirmed decisions
RetailCustomerEntitymigratesLong→UUIDin Foundation (no special-casing; UUID everywhere).Sequence + estimates (engineering-days; one PR per numbered item)
F — Foundation (~4 d): F1
EspiResourceControllerbase (2.0) · F2ScopeResolver/ScopeStrategy(1.0) · F3ParentChain+ResourceDescriptor+RetailCustomer UUID migration (0.5) · F4 XSD+disjoint-token test harness (0.5)E — Energy (~5.5 d): E1 UsagePoint+MeterReading+ReadingType triad (2.0) · E2 IntervalBlock (0.5) · E3 EPQS (0.5) · E4 UsageSummary (0.5) · E5 LocalTimeParameter (0.5) · E6 RetailCustomer (0.5) · E7 Subscription (0.5) · E8 ProgramIdMappings (0.5)
B — Batch composition (~2 d): B1 Batch/Subscription (1.0) · B2 Batch/Bulk (0.5) · B3 Batch/RetailCustomer(+/UsagePoint) (0.5)
P — PII (~5.5 d): P1 Customer (1.5) · P2–P9 CustomerAccount, CustomerAgreement, ServiceSupplier, ServiceLocation, EndDevice, Meter, Statement, ProgramDateIdMappings (0.5 ea = 4.0)
A — Status/admin GET (~1.75 d): A1 ReadServiceStatus (0.25) · A2 ApplicationInformation GET (0.5) · A3 Authorization GET + per-TP filter [#141] (1.0)
X — Cross-cutting (~4.5 d): X1 Security matchers (1.0) · X2 end-to-end XPath-tree IT (1.0) · X3 entity↔XSD drift [#101] (1.5) · X4 JSON-removal/Atom sweep [#119 Ph3] (1.0)
Total ≈ 23–25 eng-days for the full GET surface (delivers #98; subsumes #119 Ph2–8 + #101). Deferred admin CRUD writes track: +5–8 d, scheduled later.
Estimates are effort-days (your issues' unit); wall-clock with implement-and-review cadence is materially shorter.
chore(#119): defer ESPI resource CRUD writes — delete write/admin con…
dfcoffin commentedon Jun 5, 2026
🔧 Correction to the canonical build plan — RetailCustomer is NOT an ESPI resource
Per ESPI authority:
RetailCustomeris not defined inespi.xsdand has no Atom/XML stream — it is only the local correlation between ESPI data and a user account. Consequences for the plan above:RetailCustomer→UUID migration. It staysLong. The previously-proposed PR-0 (and Remediation: ESPI REST controller modernization (post-#116) #119 Phase 4's "Long retailCustomerId→UUID") are dropped. UUID v5 applies to real ESPI resources only.RetailCustomeris aLongpath-container / scope key, not a Phase-E resource slice. There is nothing to marshal, so drop the E6 "RetailCustomer" resource. The energy resource slices are: E1 UsagePoint+MeterReading+ReadingType triad, IntervalBlock, EPQS, UsageSummary, LocalTimeParameter, Subscription, ProgramIdMappings.ParentChainhandles mixed key types:RetailCustomer=Long, every real ESPI resource =UUID.customerResourceURI=Batch/RetailCustomer/{Long}(already non-UUID viaEspiBatchUri.retailCustomerId).GET /RetailCustomer(+/{id}) endpoints are admin/local-account correlation → deferred admin track, not the ESPI resource surface.Net effect: Foundation loses its riskiest item (no PK migration). Revised next step = go straight to F1 (base
EspiResourceController). Estimate drops ~0.5–1 d.dfcoffin commentedon Jun 5, 2026
🔧 Correction — ESPI has no pagination (CMD/DMD are machine-to-machine)
Per ESPI authority: the standard does not support pagination. Connect-My-Data and Download-My-Data are machine-to-machine bulk exchanges, so there is no
limit/offset(nor OpenSearchmax-results/start-index).EspiResourceControllerhas NO pagination — a collection endpoint returns the full scoped Atom feed.@RequestParam limit/offsetcurrently on the live GET controllers (UsagePoint, Customer, CustomerAccount, MeterReading, …) are a non-ESPI deviation; they are removed as each resource migrates onto the base (Phase E) — no separate cleanup PR.OffsetPageable, limit/offset) is dropped.published-min/max,updated-min/max) remain, on the feeds where the spec defines them (the DC→TP notification contract test + TP-receiver stub #158 notification URIs already usepublished-min/max). These are content filters, not pagination.dfcoffin commentedon Jun 5, 2026
🔧 Refinement — ESPI query parameters (REQ.21.6.2.8, p.176–177)
The standard defines feed query parameters and states: "Only published-max/min and updated-max/min are required to be supported." Full list:
published-min/max,updated-min/max,max-results,start-index,start-after,depth.Design for the F1 base
EspiResourceController:published-min,published-max,updated-min,updated-max(applied as repository range queries onIdentifiedObject.published/updated).max-results,start-index,start-after(pagination) anddepth. ESPI does not require pagination — CMD/DMD are machine-to-machine. (Supersedes the earlier "no query params / full feed only" note: the timestamp filters ARE required; only pagination/depth are skipped.) The pre-existing non-ESPIlimit/offsetparams are removed as resources migrate onto the base.Net F1 contract: full scoped Atom feed, optionally narrowed by the four required timestamp filters; no pagination.
dfcoffin commentedon Jun 5, 2026
🔧 Refinement — optional query params codified as optional DB-query knobs
The optional REQ.21.6.2.8 params (
max-results,start-index,start-after,depth) are DB-query parameters that certification checks only for data-format conformance / acceptance, not filtering behavior. Decision: accept all four as optional, null-safe query knobs bound into the DB query — so the endpoint conforms to the full ESPI parameter set without pagination being required behavior.F1 base controller binds an
EspiQueryOptionsvalue object via@RequestParam(required=false)under the ESPI names:published-min/max,updated-min/maxonIdentifiedObject.published/updated.max-results→ query limit,start-index→ offset (both trivially honored);start-after→ id cursor anddepth→ feed-expansion are accepted best-effort/no-op (documented as format-only, not behavior-tested).Net: endpoint accepts the complete REQ.21.6.2.8 set (no 400s), required filters filter, optional params are present-but-optional DB knobs. Supersedes the earlier "do not implement the optional params" note — they are accepted/codified-optional rather than rejected.
dfcoffin commentedon Jun 5, 2026
🔧 Clarification — ALL query parameters are optional (incl. the date filters)
No required-vs-optional split in request binding. Every REQ.21.6.2.8 parameter is optional on the request; the spec's "required to be supported" means the DC implements the date-filter behavior, not that a caller must supply it.
EspiQueryOptionsis all-nullable; every param binds@RequestParam(required = false).400for omitting (or supplying) any param.published/updated min/maxare honored as WHERE predicates (DC supports them);max-results→limit,start-index→offset honored;start-after/depthaccepted best-effort/no-op.Supersedes the "required four / optional four" framing — the only distinction is behavioral depth when a param happens to be supplied, not request-time requiredness.
dfcoffin commentedon Jun 6, 2026
✅ F1 scope clarification — reuse the existing (tested) Atom/DTO/IdentifiedObject layer
The Atom-envelope ↔
IdentifiedObjectmapping and the DTO marshalling are already implemented and tested inopenespi-common(this is what aligned the DB DDL withespi.xsd/customer.xsd). Per REQ.21.6.2, identity/naming lives in the Atom<entry>envelope (mRID→atom:idurn:uuid:,description→atom:title,published/updated,self/up/relatedlinks) — consistent with the near-emptyIdentifiedObjectcomplexType (extendsObject+extension, adds onlybatchItemInfo).F1 therefore re-implements NONE of that — it is purely an orchestration layer over the proven
UsageExportService/CustomerExportService+ mappers + Atom DTOs. F1's only new code:EspiResourceControllerbase (ROOT + XPath, collection + individual; delegates marshalling to the existing export services).ScopeResolver/ScopeStrategy(energy usage-point set, PII retail-customer set, ReadingType reachability).ParentChainvalidation (mixed keys: RetailCustomer=Long, resources=UUIDv5).EspiQueryOptions(all-optional REQ.21.6.2.8 params → repo query).Net: F1 is lighter/lower-risk than the original 2.0 d estimate — composition, not creation.
dfcoffin commentedon Jun 6, 2026
🔧 E1 entry-assembly contract — atom:link rules (Don, from a GBA certification feed)
The existing marshalling renders links present in the DTO but does not populate them — link/URI assembly is E1's job, layered on the tested marshalling.
atom:linkappears ONLY on<entry>, never on<feed>(Sandbox). The CMD-certification feed-levelrel="related" type="text/xml"certificate link is NOT required for the Sandbox → omit. So a feed =id+title+updated+ entries.<entry>carries:rel="self",rel="up", and onerel="related"per reachable related resource whose FB is in the token scope (gated viaScopeResolver/EspiScope). Related set per REQ.21.6.2: UsagePoint→{MeterReading, UsageSummary, ElectricPowerQualitySummary, LocalTimeParameters, ProgramIdMappings}; MeterReading→{ReadingType, IntervalBlock}; etc.typeattribute rule:espi-feed/{Resource}when the href has no index (a collection URL ending in the resource name);espi-entry/{Resource}when the href ends in an id. ⇒self→espi-entry/…,up→espi-feed/…,related→by its href shape./Subscription/{subscriptionId}/…;atom:id=urn:uuid:{mRID}.Open for E1: the GBA cert feed uses the default (unprefixed) Atom namespace; the current tested
UsageExportServiceemitsatom:-prefixed Atom. Confirm whether to switch the marshaller to default-ns or keep the prefix before E1.F1 (the scope core) is unaffected by all of the above and remains the input the related-link FB-gating consumes.
feat(#119 F1): resource-server scope foundation + dev XML capture/sam…
dfcoffin commentedon Jun 6, 2026
🔍 Audit: atom:link
<<link>>graph vs entity relationships + DDLCross-referenced the NAESB 4.0 UML
<<link>>associations against the JPA entities and the Flyway DDL, to verify the entry-assembler can actually resolve each related resource id.Energy (E2–E4): ready ✅ — entity navigation present for UsagePoint↔{MeterReading, ElectricPowerQualitySummary, UsageSummary, LocalTimeParameters} and MeterReading↔{IntervalBlock, ReadingType}. One gap: no
ProgramIdMappingsentity → #168 (E2 emits its other 4 related links and omits ProgramIdMappings until then).PII (P-phase): blocked on entity mappings ❌ — the DDL has the FKs, but
CustomerAgreement/CustomerAccount/ServiceLocation/EndDevice/Meterentities don't map them: CustomerAccount↔CustomerAgreement, CustomerAgreement↔{ServiceLocation, ServiceSupplier, ProgramDateIdMappings}, ServiceLocation↔{EndDevice, Meter, LocalTimeParameters}. So the PII XPath chain isn't navigable and those related links can't be emitted → #169 (pure JPA mapping of existing FKs; DDL unchanged; prerequisite for the P-phase). Already-mapped/OK: Customer↔CustomerAccount/Statement/TimeConfiguration, *↔Statement, UsageSummary↔Statement (cross-domain).Net: E1/E2 (energy) proceed now; #168 + #169 land before the resources that need them. Both are #101/#123-adjacent.
fix: local sandbox bring-up — DC↔AS introspection alignment + TP atom…
dfcoffin commentedon Jun 11, 2026
E1 of the DC ESPI Resource Surface (GET-first) plan is now tracked in #187 — resource engine + Atom entry-assembly (GET-first), first slice = UsagePoint→MeterReading→ReadingType triad, fully scope-enforced (consumes the F1/#167
ScopeResolver). Namespace gate cleared (keep explicitatom:/espi:prefixes).