Skip to content

Remediation: ESPI REST controller modernization (post-#116) #119

Open

Description

@dfcoffin

Issue #98 Remediation Plan (originally tracked as PR #115)

Re-aimed 2026-05-20: This plan was originally written against PR #115's branch. PR #115 was closed as superseded by PR #116 (merged 2026-05-20), which landed a renamed and slimmer subset of #115's scope. The dangerous bits that were the original blockers (ManageController shell exec, web/api/RetailCustomerController write methods, web/api/ServiceStatusController) did not merge and remain as .disabled files on main.

The plan below has been re-scoped to target what actually landed in main via #116, plus pre-conditions for safely re-enabling the .disabled files later. File path references generally use the *RESTController.java naming that exists on main; per-phase paths should still be verified at execution time.

Source review: see /review PR #115 findings.

Organized into 8 phases (and one deferred bucket) — each phase is self-contained, can be a separate commit, and has its own acceptance criteria. Ordered by risk-first, then dependency (CI unblock → data correctness → structural cleanup → polish → tests).


Phase 1 — De-stream REST controllers (3–4 hr) ⚡ NEXT

Surfaced by PR #118 CI failure: MeterReadingControllerTest.shouldReturn200WhenExists throws java.util.ConcurrentModificationException because MeterReadingController returns a StreamingResponseBody, and the streaming worker thread races with the test thread on MockHttpServletResponse's LinkedCaseInsensitiveMap while Spring Security's OnCommittedResponseWrapper writes security headers.

StreamingResponseBody is used by every GET endpoint PR #116 added (9 controllers, ~30+ endpoints): ApplicationInformationRESTController, CustomerAccountRESTController, CustomerRESTController, IntervalBlockRESTController, ElectricPowerQualitySummaryRESTController, ReadingTypeRESTController, UsagePointController, UsageSummaryRESTController, MeterReadingController.

Why drop it (not just patch the test): What's on main today isn't real streaming — it's StreamingResponseBody wrapping JAXB-marshalled bytes. The service builds the full DTO graph, JAXB marshals it to a complete byte[], and only then does the streaming worker dribble that already-buffered payload out. The whole response is in heap before the first byte hits the wire, so removing StreamingResponseBody loses zero real streaming benefit — including on /Subscription/* endpoints. In exchange, the "fake streaming" costs: race conditions with Spring Security headers (the CME), per-request WebAsyncManager worker thread overhead, permanent asyncDispatch() tax on every MockMvc test, harder operational debugging, forced chunked transfer encoding. It's premature optimization that introduced concrete bugs. Prerequisite for Phases 2 and 3 — subscription scoping is much simpler against sync controllers, and Atom link/UUID work is much cleaner without async thread-context concerns. Also unblocks CI — until this lands, the CME re-surfaces randomly and fails arbitrary PRs.

  • 1.1 ApplicationInformationRESTController — replace ResponseEntity<StreamingResponseBody> returns with ResponseEntity<String> (already-marshalled XML) or ResponseEntity<EntryDto> / ResponseEntity<FeedDto> + JAXB message converter
  • 1.2 CustomerAccountRESTController
  • 1.3 CustomerRESTController
  • 1.4 IntervalBlockRESTController
  • 1.5 ElectricPowerQualitySummaryRESTController
  • 1.6 ReadingTypeRESTController
  • 1.7 UsagePointController
  • 1.8 UsageSummaryRESTController
  • 1.9 MeterReadingController
  • 1.10 Remove all import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody; lines that are no longer referenced
  • 1.11 Update corresponding MockMvc tests: drop any request().asyncStarted() / MvcResult / asyncDispatch() wrappers (added or about to be added to work around the CME)
  • 1.12 Run mvn verify -pl openespi-datacustodian -am 3× to confirm no remaining flakes
  • 1.13 Confirm MeterReadingControllerTest.shouldReturn200WhenExists passes deterministically

Acceptance: No StreamingResponseBody references remain in openespi-datacustodian controllers; full datacustodian test suite passes 3 consecutive clean runs; no MockMvc test uses asyncDispatch (unless a deliberately-async endpoint exists, which currently it does not).

Production utility implementations of /Subscription/* and /Batch/Bulk/* likely DO need real streaming — a year of 15-min interval data across many usage points for one customer can exceed 100 MB, and a bulk export across thousands of customers is unbounded. But "real streaming" is not StreamingResponseBody + JAXB. Real streaming means:

  1. Open a streaming JPA cursor (@QueryHints(@QueryHint(name=HINT_FETCH_SIZE, value="100")) returning Stream<Entity> inside a @Transactional(readOnly=true)) or JdbcTemplate.queryForStream(...)
  2. Use a StAX XMLStreamWriter pointed at response.getOutputStream()
  3. Write the Atom feed envelope, then per-entry: marshal one DTO → flush() → advance the cursor
  4. Write the feed footer and close the stream

That's the architecture that delivers actual memory bounding and progressive transmission. It belongs in production utility forks of this codebase; this Sandbox / reference implementation doesn't need it (response sizes are small by definition of "sandbox", and the test/CI cost of streaming far outweighs the benefit). When a fork does need it, implement it endpoint-by-endpoint with explicit StAX + cursor — do not reach for StreamingResponseBody as a shortcut, since it delivers the test-flake costs without the memory benefits.


Phase 2 — Subscription scoping (data-leakage fix) (3–5 hr)

Currently a third-party token gets unfiltered global lists. Model on UsagePointController which does this correctly.

For each controller below, the pattern is:

  1. Inject SubscriptionService (or whatever UsagePointController uses)
  2. Resolve subscription from access token via ApiAccessValidator.requireSubscriptionId(authHeader)
  3. For non-admin scopes, filter the result set to entities belonging to that subscription's authorized usage points
  4. Admin scope retains unfiltered access
  • 2.1 IntervalBlockRESTController list endpoints
  • 2.2 MeterReadingController list endpoints
  • 2.3 ElectricPowerQualitySummaryRESTController list endpoints
  • 2.4 ReadingTypeRESTController — note ReadingType is shared metadata; verify scoping semantics with the spec before filtering
  • 2.5 UsageSummaryRESTController list endpoints
  • 2.6 CustomerRESTController list endpoints
  • 2.7 CustomerAccountRESTController list endpoints
  • 2.8 Add MockMvc test per controller: third-party token sees only their subscription's data; admin token sees all

Acceptance: Each list endpoint has an integration test proving that two different subscription tokens get disjoint result sets, and admin sees the union.


Phase 3 — ESPI 4.0 Atom + JSON compliance (4–6 hr)

The most important compliance fix. Touches both code and conventions.

  • 3.1 BaseExportService.createServiceStatusEntry (if present on main — verify) — replace selfLink.setHref("ServiceStatus") / upLink.setHref("") with the canonical pattern https://{host}/DataCustodian/espi/1_1/resource/{resource}/{id}. Inject host from config (espi.base-url or equivalent).
  • 3.2 Audit every createAtomEntry(title, dto) callsite in DtoExportServiceImpl and ensure the entry has:
    • rel="self" href with full resource URI
    • rel="up" href with full parent collection URI
    • rel="related" hrefs where applicable
  • 3.3 Replace all UUID.randomUUID() in Atom-id generation with UUID Version 5 (deterministic from resource URI). Use the project's existing espiIdGeneratorService if available; otherwise add a helper.
  • 3.4 Verify entry IDs serialize as urn:uuid:xxxxxxxx-xxxx-5xxx-xxxx-xxxxxxxxxxxx
  • 3.5 Remove produces = APPLICATION_JSON_VALUE from every ESPI resource endpoint on main: ApplicationInformationRESTController, AuthorizationController, CustomerAccountRESTController, CustomerRESTController, ElectricPowerQualitySummaryRESTController, IntervalBlockRESTController, MeterReadingController, ReadingTypeRESTController, UsagePointController, UsageSummaryRESTController. Keep XML only — APPLICATION_XML_VALUE or APPLICATION_ATOM_XML_VALUE as appropriate.
  • 3.6 Add an integration test that marshals one feed + one entry per resource and validates against openespi-common/src/main/resources/schema/ESPI_4.0/espi.xsd using a Validator from SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI).newSchema(espiXsd)

Acceptance: All ESPI resource feeds/entries pass XSD validation; Atom IDs are UUID v5; no JSON content-type is offered for ESPI resources.


Phase 4 — ID type cleanup (1–2 hr)

Wider than originally scoped — Long retailCustomerId is spread across MVC controllers on main, not just one method.

  • 4.1 DtoExportService.createRetailCustomerEntry(Long retailCustomerId) → UUID retailCustomerId (verify signature still exists on main)
  • 4.2 openespi-datacustodian/.../web/custodian/AssociateUsagePointController lines 65 / 74 — Long retailCustomerId → UUID
  • 4.3 openespi-datacustodian/.../web/custodian/RetailCustomerController:85 — Long retailCustomerId → UUID
  • 4.4 openespi-datacustodian/.../web/customer/CustomerDownloadMyDataController lines 46 / 64 — Long retailCustomerId → UUID
  • 4.5 openespi-datacustodian/.../web/customer/MeterReadingController:45 — Long retailCustomerId → UUID
  • 4.6 openespi-datacustodian/.../web/customer/UsagePointController:88 — Long retailCustomerId → UUID
  • 4.7 openespi-thirdparty/.../web/MeterReadingController:45 — Long retailCustomerId → UUID
  • 4.8 openespi-thirdparty/.../web/UsagePointController:80 — Long retailCustomerId → UUID
  • 4.9 openespi-thirdparty/.../web/custodian/RetailCustomerController:90 — Long retailCustomerId → UUID
  • 4.10 Final sweep: git grep -nE "Long\\s+\\w*[Cc]ustomerId" openespi-datacustodian openespi-common openespi-thirdparty returns empty
  • 4.11 Verify RetailCustomerEntity PK is UUID per CLAUDE.md — if it's still Long, that's a separate (larger) issue: file it as a new issue

Acceptance: No Long-keyed RetailCustomer references in datacustodian, common, or thirdparty modules. All controller path variables and service signatures use UUID.


Phase 5 — Constructor + facade structural cleanup (2–3 hr)

Verify scope on main first — the renamed/slimmed #116 may have already partially addressed this. If DtoExportServiceImpl constructor on main is no longer 11-arg, scale this phase down.

  • 5.1 Verify on main: count DtoExportServiceImpl constructor parameters. If ≤4, mark this phase N/A. If still large:
    • 5.1a (preferred) Switch DTO tests that enumerate dependencies to Mockito @InjectMocks so they no longer pass null, null, null, …
    • 5.1b Extract a small MarshallerOnlyExporter from DtoExportServiceImpl that the DTO tests can construct cheaply
  • 5.2 Verify on main: is DtoExportServiceFacade present? If yes, either delegate all feed/entry methods to DtoExportServiceImpl, or remove DtoExportServiceImpl from the facade's deps and centralize logic in the facade. Do not keep both code paths.
  • 5.3 Verify: are there still two createServiceStatusEntry implementations? If yes, collapse into one.
  • 5.4 Confirm @Qualifier("dtoExportServiceImpl") resolves correctly (default Spring bean name should match)

Acceptance: Adding a new dependency to DtoExportServiceImpl does not require touching DTO test files. At most one createServiceStatusEntry exists.


Phase 6 — Pagination & behavior polish (2 hr)

Verify each item against main first. Several originally-listed controllers (TimeConfigurationController, RetailCustomerController.index(), BatchController) are still .disabled on main — those sub-items should move into the Deferred bucket below.

Applicable on main:

  • 6.1 ReadingTypeRESTController list endpoint — implement limit/offset (pass to a paginated repository call) or drop the parameters
  • 6.2 OffsetPageable.getPageNumber() (if present on main) — replace offset / limit with Math.floorDiv(offset, limit); add a unit test for non-multiple offsets
  • 6.3 ApiAccessValidator.requireSubscriptionId (if present on main) — return 401 UNAUTHORIZED for missing/empty token, 403 FORBIDDEN only for valid-but-insufficient scope

Acceptance: Pagination parameters either work or are removed; HTTP status codes follow spec.


Phase 7 — CustomerMapper correctness (1 hr)

Verify scope on main first — confirm CustomerMapper.toDto(RetailCustomerEntity) exists on main with the issue. If the method wasn't merged, defer to when it gets re-introduced.

  • 7.1 CustomerMapper.toDto(RetailCustomerEntity) — null-safe name concatenation (skip nulls, trim, return null rather than "null null")
  • 7.2 Stop populating OrganisationDto.organisationName with a person name. Either leave it null when the source is a RetailCustomer, or split the mapping into a person-customer branch and an organisation-customer branch
  • 7.3 Add a unit test covering: all-null name, first-only, last-only, both, organisation case
  • 7.4 Design question to raise: should RetailCustomerEntity even marshal as a Customer (different schema)? If not, deprecate this mapper method and route consumers to the proper resource

Acceptance: Tests cover null permutations; no "null null" artifact possible.


Phase 8 — Test fortification (3 hr)

  • 8.1 XSD-validation integration test (Phase 3.6) confirmed running in mvn verify
  • 8.2 Subscription-scoping tests from Phase 2.8 confirmed running
  • 8.3 Run full mvn verify -pl openespi-common,openespi-datacustodian -am clean

Acceptance: All new tests green; no regressions in existing suite.


Deferred — Pre-conditions for re-enabling .disabled controllers

These items were originally Phase 2 blockers for PR #115 but did NOT land in main via #116. They live in .disabled files. Apply these fixes as part of the same PR that re-enables each file — never re-enable as-is.

  • D.1 web/api/ManageRESTController.java.disabled — remove resetDatabase / initializeDatabase REST endpoints before re-enabling. Two options:
    • Option A (preferred): delete them entirely; move admin operations to a non-HTTP CLI
    • Option B: gate behind SCOPE_DataCustodian_Admin_Access and @Profile("!prod"), and never echo stderr into HTTP response
  • D.2 web/api/RetailCustomerRESTController.java.disabled — add @PreAuthorize("hasAuthority('SCOPE_DataCustodian_Admin_Access')") to create(), update(), delete() before re-enabling (or delete those methods if they're still 501 stubs)
  • D.3 web/api/ServiceStatusRESTController.java.disabled — replace authHeader.substring(7) with a guarded extraction (startsWith("Bearer ") AND length() > 7 AND non-empty token, else 401) before re-enabling
  • D.4 web/api/ApplicationInformationController.createApplicationInformation (if/when added) — add @Valid and bean-validation constraints on clientSecret, redirectUri, scope, tosUri, clientName
  • D.5 BatchRESTController.java.disabled — when re-enabling:
    • subscriptionUsagePoint(...) must filter by the subscription's specific usage points, not delegate to subscription(...)
    • upload() / .bulk() must log when called (log.warn("Unimplemented endpoint called: {}", …)) before returning 501
    • downloadCollection(@PathVariable Long retailCustomerId, …) → use UUID (also covered by Phase 4)
    • Streaming approach for /Batch/Bulk/* and /Batch/Subscription/* — these endpoints can legitimately return 100+ MB in production utility deployments. If/when this Sandbox repo needs real streaming for them, implement it as StAX XMLStreamWriter + a streaming JPA cursor (or JdbcTemplate.queryForStream) marshalling one entry at a time. Do not wrap JAXB-marshalled bytes in StreamingResponseBody — that's the fake streaming pattern Phase 1 just removed; it has no memory benefit and brings back the CME footgun. For Sandbox/CI purposes, a synchronous response of bounded test-data size is fine; the streaming path is only worth the engineering investment if a production fork pulls this code as-is.
  • D.6 TimeConfigurationRESTController.java.disabled — when re-enabling, honor limit/offset or drop them
  • D.7 ServiceStatusController — replace hardcoded applicationStatus = "0" with a DataCustodianApplicationStatus.OK constant/enum
  • D.8 Audit token logging in AuthorizationService.findByAccessToken and any new auth-handling code — ensure raw tokens are never logged

Acceptance: Every .disabled file ships with the corresponding security/correctness fix in the same PR.


Cross-cutting reviewer checklist (apply per commit)

  • Edit-tool diffs only (no sed/awk per CLAUDE.md)
  • No new JAXB on entities, no new JPA on DTOs
  • No new Long-keyed ESPI entities
  • AssertJ assertions chained per CLAUDE.md
  • mvn test -pl <touched-module> -am green before each commit
  • PR title uses conventional-commit format (feat:, fix:, chore:, etc. with colon) to clear PR Validation

Suggested phase grouping into PRs (if breaking up)

  • PR-A: Phase 1 (de-stream controllers) — mechanical refactor, unblocks CI, prerequisite for PR-B and PR-C
  • PR-B: Phase 2 (data-leakage fix) — high-impact, focused review
  • PR-C: Phase 3 (ESPI compliance) — touches many files, dedicated review
  • PR-D: Phases 4 + 5 + 6 + 7 (cleanup & polish)
  • PR-E: Phase 8 (test fortification) — could fold into A–D, or stand alone

Deferred items (D.1–D.8) ship one-by-one with whichever PR re-enables the corresponding .disabled file.


Total estimated effort: ~18–23 hours focused work on the 8 active phases, 5 PRs if split. Deferred items add ~3–5 hours when their respective .disabled files are re-enabled.


Change log

Activity

  1. self-assigned this
    on May 20, 2026
  2. added
    taskA general task
    PolishClean up unnecessary logic or code smells
    NAESB ESPI StandardRequired to comply with the NAESB ESPI standard
    schema-complianceData elements comply with their appropriate ESPI schema definitions
    blockingBlocks other work or CI
    ESPI 4.0Touches the NAESB ESPI 4.0 implementation
    on May 20, 2026
  3. dfcoffin commented on May 21, 2026

    @dfcoffin
    ContributorAuthor

    Phase 1 complete in #120 (merged 2026-05-21 as 797b4dd). CME flake resolved across all 9 controllers / 26 endpoints; CI now deterministic. Phase 2 (subscription scoping) is next.

  4. dfcoffin commented on May 22, 2026

    @dfcoffin
    ContributorAuthor

    Phase 2 — Data Leakage Fix

    Companion to Issue #119, Phase 2. Audits the current active REST API surface
    in openespi-datacustodian after PR #120 and PR #121, identifies the IDOR /
    cross-subscription data-leakage exposure, and proposes the fix design.

    This is a design draft for issue-comment posting, not a committed artifact.
    Several precursors (running auth-server, scope-grammar alignment, JSON output)
    must land or be acknowledged before any code in this document is written.

    Generated: 2026-05-22 (revised to opaque-token introspection model).


    1. Active REST API surface

    Base path: /espi/1_1/resource (all controllers).
    Response bodies are byte[] containing JAXB-marshalled XML. Controllers declare
    both application/xml and application/json as producible, but only XML is
    actually emitted today
    . Real JSON support is a separate Phase 3 work item; the
    JSON content-type advertisement is currently cosmetic and misleading.

    1.1 GET endpoints

    # Controller HTTP method + path Path params Query params (optional) @PreAuthorize (as written) Leakage risk
    1 UsagePoint GET /UsagePoint – limit=50, offset=0 Admin · FB_15 · FB_16 · FB_36 HIGH — root collection, no subscription scope
    2 UsagePoint GET /UsagePoint/{usagePointId} usagePointId – Admin · FB_15 · FB_16 · FB_36 HIGH — any token with scope can read any UP
    3 UsagePoint GET /Subscription/{subscriptionId}/UsagePoint subscriptionId limit=50, offset=0 FB_15 · FB_16 · FB_36 MED — TODO comment; subscriptionId ignored
    4 UsagePoint GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId} subscriptionId, usagePointId – FB_15 · FB_16 · FB_36 MED — same, subscriptionId ignored
    5 MeterReading GET /MeterReading – limit=50, offset=0 Admin · FB_15 · FB_16 · FB_36 HIGH
    6 MeterReading GET /MeterReading/{meterReadingId} meterReadingId – Admin · FB_15 · FB_16 · FB_36 HIGH
    7 ReadingType GET /ReadingType – limit=50, offset=0 Admin · FB_15 · FB_16 · FB_36 LOW — metadata, not customer data
    8 ReadingType GET /ReadingType/{readingTypeId} readingTypeId – Admin · FB_15 · FB_16 · FB_36 LOW
    9 IntervalBlock GET /IntervalBlock – limit=50, offset=0 Admin · FB_15 · FB_16 · FB_36 HIGH
    10 IntervalBlock GET /IntervalBlock/{intervalBlockId} intervalBlockId – Admin · FB_15 · FB_16 · FB_36 HIGH
    11 IntervalBlock GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}/MeterReading/{meterReadingId}/IntervalBlock subscriptionId, usagePointId, meterReadingId – FB_15 · FB_16 · FB_36 MED — filters by meterReadingId only
    12 IntervalBlock GET .../IntervalBlock/{intervalBlockId} (same prefix) + intervalBlockId – FB_15 · FB_16 · FB_36 MED — TODO; no relationship validation
    13 UsageSummary GET /UsageSummary – limit=50, offset=0 Admin · FB_15 · FB_16 · FB_36 HIGH
    14 UsageSummary GET /UsageSummary/{usageSummaryId} usageSummaryId – Admin · FB_15 · FB_16 · FB_36 HIGH
    15 UsageSummary GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}/UsageSummary subscriptionId, usagePointId limit=50, offset=0 FB_15 · FB_16 · FB_36 MED — TODO; both path IDs ignored
    16 UsageSummary GET .../UsageSummary/{usageSummaryId} + usageSummaryId – FB_15 · FB_16 · FB_36 MED
    17 ElectricPowerQualitySummary GET /ElectricPowerQualitySummary – limit=50, offset=0 Admin · FB_15 · FB_16 · FB_36 HIGH
    18 ElectricPowerQualitySummary GET /ElectricPowerQualitySummary/{electricPowerQualitySummaryId} electricPowerQualitySummaryId – Admin · FB_15 · FB_16 · FB_36 HIGH
    19 ElectricPowerQualitySummary GET /Subscription/{subscriptionId}/UsagePoint/{usagePointId}/ElectricPowerQualitySummary subscriptionId, usagePointId limit=50, offset=0 FB_15 · FB_16 · FB_36 MED
    20 ElectricPowerQualitySummary GET .../ElectricPowerQualitySummary/{electricPowerQualitySummaryId} + electricPowerQualitySummaryId – FB_15 · FB_16 · FB_36 MED
    21 Customer GET /Customer – limit=50, offset=0 Admin · FB_15 HIGH
    22 Customer GET /Customer/{customerId} customerId – Admin · FB_15 HIGH
    23 CustomerAccount GET /CustomerAccount – limit=50, offset=0 Admin · FB_15 HIGH
    24 CustomerAccount GET /CustomerAccount/{customerAccountId} customerAccountId – Admin · FB_15 HIGH
    25 ApplicationInformation GET /ApplicationInformation – – Admin · ThirdParty_Admin LOW — admin-only by design
    26 ApplicationInformation GET /ApplicationInformation/{applicationInformationId} applicationInformationId – Admin · ThirdParty_Admin LOW

    Important caveat on the @PreAuthorize column. The scope-string shorthand
    in the source (e.g. SCOPE_FB_15_READ_3rd_party) is not the NAESB ESPI 4.0
    Function Block grammar
    (FB=4_5_15;IntervalDuration=3600;...). The
    shorthand was introduced during the original Spring Boot modernization without
    a spec mapping. Aligning the scope grammar — both auth-server issuance and
    resource-server matching — is a precursor / parallel task (tracked separately
    as Phase 7 in #119) and is out of scope for Phase 2 itself, but Phase 2's
    introspection layer has to be neutral to whichever form survives that work.

    1.2 Mutating endpoints (CRUD)

    # Controller HTTP method + path @PreAuthorize Leakage risk
    M1 Customer POST /Customer (XML/JSON in & out) Admin · FB_15_WRITE Write-side IDOR
    M2 Customer PUT /Customer/{customerId} Admin · FB_15_WRITE Cross-customer overwrite possible
    M3 Customer DELETE /Customer/{customerId} Admin · FB_15_WRITE Same
    M4 CustomerAccount POST /CustomerAccount (JSON only) Admin · FB_16_WRITE Same
    M5 CustomerAccount PUT /CustomerAccount/{customerAccountId} (JSON only) Admin · FB_16_WRITE Same
    M6 CustomerAccount DELETE /CustomerAccount/{customerAccountId} Admin · FB_16_WRITE Same
    M7 ApplicationInformation POST /ApplicationInformation (JSON only) Admin only OK
    M8 ApplicationInformation PUT /ApplicationInformation/{id} (JSON only) Admin only OK
    M9 ApplicationInformation DELETE /ApplicationInformation/{id} Admin only OK

    1.3 Observations

    • Authentication authentication is injected on most handlers but never
      actually read
      — vestigial wiring from the original modernization.
    • produces = { APPLICATION_XML, APPLICATION_JSON } is declared everywhere, but
      the body is JAXB-serialized XML bytes regardless. JSON content-type
      negotiation is cosmetic; real JSON output requires Phase 3.
    • Every /Subscription/... handler carries a
      // TODO: implement subscription-based filtering comment in the body — the
      security check was deferred during modernization and never landed.

    2. Threat model

    A third-party application obtains an OAuth2 access token bound to subscription
    S_a (granting access to retail customer A's data). With that token they can
    today exercise three distinct IDOR (CWE-639) paths:

    1. Root collection enumeration — GET /UsagePoint returns every
      UsagePoint across all subscriptions.
    2. Root by-id guessing — GET /UsagePoint/{anyId} performs no ownership
      check. UUIDv5 reduces guessability but does not eliminate it (especially
      when IDs leak through Atom selfLink / related URIs in other responses).
    3. Path-bound endpoints ignore path — GET /Subscription/{S_b}/UsagePoint
      returns all data; the server doesn't compare {S_b} to the token's binding,
      doesn't filter by it, and the path variable is effectively decorative.

    All three apply across UsagePoint, MeterReading, IntervalBlock, UsageSummary,
    ElectricPowerQualitySummary, Customer, and CustomerAccount. ReadingType is
    metadata (intentionally cross-subscription) and ApplicationInformation is
    admin-only — both excluded.


    3. Token model — opaque, NOT JWT

    ESPI 4.0 requires opaque (reference) access tokens. JWT-style access tokens
    are not permitted by the spec. This dictates the entire enforcement design:

    • The resource server (data-custodian) cannot read claims from the token.
      It has bytes, not a structured payload.
    • To learn what the token grants, the resource server must call
      RFC 7662 token introspection against the authorization server.
    • Spring Security 7 provides OpaqueTokenIntrospector as the abstraction. The
      default implementation (SpringOpaqueTokenIntrospector) calls
      /oauth2/introspect and maps the response into an
      OAuth2AuthenticatedPrincipal.
    • The introspection response is the only authoritative source for
      subscription_id, retail_customer_id, scope list, and admin status.
      These must be added to the auth-server's introspection response shape if
      they aren't already.

    4. Fix design

    Two-layer defense, built around OpaqueTokenIntrospector.

    4.1 Layer 1 — introspection-driven principal

    A custom OpaqueTokenIntrospector bean wraps the default. It:

    1. Calls the auth-server's introspection endpoint (with response caching keyed
      by the token hash, TTL ≤ token remaining lifetime, ≤ 60s upper bound).
    2. Extracts subscription_id and admin status from the response.
    3. Constructs an EspiPrincipal carrying (tokenId, subscriptionId, retailCustomerId, scopes, isAdmin).
    4. Returns it as the OAuth2AuthenticatedPrincipal, so it lands in the
      SecurityContext.

    Admin tokens (carrying the configured admin scope — exact string deferred to
    the Phase 7 scope-grammar work) yield isAdmin = true and bypass binding.
    Non-admin tokens must carry a subscription binding in the introspection
    response or the request is rejected 401 Unauthorized. Silent fall-through is
    explicitly forbidden — that's the bug we're fixing.

    4.2 Layer 2 — ApiAccessValidator helper

    Location: openespi-datacustodian/src/main/java/.../security/ApiAccessValidator.java.

    @Component
    public class ApiAccessValidator {
    
        public UUID requireSubscriptionId(Authentication auth) {
            EspiPrincipal principal = (EspiPrincipal) auth.getPrincipal();
            if (principal.isAdmin()) return null; // null = unrestricted
            UUID sub = principal.subscriptionId();
            if (sub == null) {
                throw new ResponseStatusException(UNAUTHORIZED,
                    "token lacks subscription binding");
            }
            return sub;
        }
    
        public void enforcePathMatch(UUID pathSubId, UUID tokenSubId) {
            if (tokenSubId == null) return; // admin
            if (!tokenSubId.equals(pathSubId)) {
                throw new ResponseStatusException(FORBIDDEN, "subscription mismatch");
            }
        }
    
        public void enforceOwnership(UUID entitySubId, UUID tokenSubId) {
            if (tokenSubId == null) return;
            if (!tokenSubId.equals(entitySubId)) {
                throw new ResponseStatusException(FORBIDDEN,
                    "resource not in caller's subscription");
            }
        }
    }

    The helper consumes the principal that the introspector populated — it does
    not know or care that the underlying token is opaque. That's the layering
    benefit.

    4.3 Per-handler enforcement patterns

    Three patterns, applied by endpoint shape:

    Endpoint shape Enforcement
    /Subscription/{subscriptionId}/... enforcePathMatch(pathSubId, tokenSubId). Then filter the query by tokenSubId.
    Root collection (e.g. GET /MeterReading) Non-admin: scope to tokenSubId via findAllBySubscriptionId(...). Admin: unfiltered.
    Root by-id (e.g. GET /MeterReading/{id}) Non-admin: load entity, then enforceOwnership(entity.getSubscriptionId(), tokenSubId). Admin: unrestricted.

    4.4 Repository additions

    Each affected repository gains findAllBySubscriptionId(UUID, Pageable) (or
    equivalent named query). For nested resources (IntervalBlock → MeterReading →
    UsagePoint → Subscription) the query joins the chain rather than denormalizing.

    The data model audit (precursor — see §6) confirms whether
    UsagePointEntity carries a direct subscription_id FK or only transitively
    via RetailCustomerEntity. That determines whether repo methods are derived
    queries or explicit JPQL with joins.

    4.5 Tests (per endpoint, three minimum)

    • Missing subscription binding on non-admin token → 401
    • Path or entity subscription ≠ token subscription → 403
    • Match → 200 with only matching-subscription data (assert pagination and
      filtering both apply; pagination without filtering is the silent-bypass mode)

    A MockOpaqueTokenIntrospector test fixture returns a configurable principal
    without requiring a live auth-server. That fixture is the most important new
    test infrastructure in Phase 2.


    5. Sub-phase sequencing

    Sub-phase Scope Risk addressed
    2.0 (precursor) Stand up openespi-authserver in dev; confirm it issues opaque tokens and that its introspection response carries subscription_id (add it if not). Without this, all of 2a–2c is design-only and untestable end-to-end.
    2a Wire the custom OpaqueTokenIntrospector and ApiAccessValidator. Fix the 8 /Subscription/... endpoints (rows 3, 4, 11, 12, 15, 16, 19, 20). Closes "subscriptionId ignored" hole.
    2b Add subscription-scoped filtering to the 12 root GET endpoints (rows 1, 2, 5, 6, 9, 10, 13, 14, 17, 18, 21, 22, 23, 24). Closes "any FB scope reads everything" hole.
    2c Ownership checks on Customer / CustomerAccount mutations (M1–M6). Closes write-side IDOR.

    ReadingType (rows 7, 8) and ApplicationInformation (rows 25, 26, M7–M9) are
    intentionally exempt — metadata and admin-only respectively.


    6. Precursors to resolve before coding 2a

    1. Auth-server up in dev. Configure openespi-authserver to issue opaque
      tokens (OAuth2TokenFormat.REFERENCE), expose /oauth2/introspect, and
      include subscription_id (and admin flag) in the introspection response.
      Register the data-custodian as a Resource Server pointing at it.
    2. Data model audit. Confirm whether UsagePointEntity (and the other
      leaf entities) carry a subscription_id FK directly or only transitively.
      Dictates repository query shape.
    3. Scope-grammar holding pattern. Decide whether Phase 2 keeps the existing
      SCOPE_FB_15_READ_3rd_party shorthand and parses it identically on both
      sides, or waits for Phase 7's spec-compliant FB-string grammar. The
      introspector design is neutral; the @PreAuthorize lines are not.
    4. JSON-output expectations. Confirm with stakeholders that the cosmetic
      produces = JSON declarations stay as-is during Phase 2 (real JSON output
      is Phase 3). Otherwise the scope of this work expands considerably.

    Resolving 1 and 2 is the first concrete work of 2.0.

  5. dfcoffin commented on May 22, 2026

    @dfcoffin
    ContributorAuthor

    Phase 2 is blocked by #122 (auth-server bring-up + opaque-token introspection). No Phase 2a code will be merged until #122's acceptance criteria (one end-to-end integration test minting a real token and reaching a data-custodian endpoint with a populated subscription binding) are met.

  6. dfcoffin commented on May 26, 2026

    @dfcoffin
    ContributorAuthor

    Follow-up checklist item from PR #116 notes

    Surfaced in PR_116_FOLLOWUP_NOTES.md (item 5) and folded into this remediation issue as it's part of ESPI REST controller correctness:

    MapStruct mappers don't populate IdentifiedObject base fields. The entity→DTO mappers omit the inherited IdentifiedObject fields — id, selfLink, upLink, relatedLinks, published, updated — which are required for ESPI Atom feed compliance (self/up/related links and resource identity in every feed entry). Without them, marshalled Atom output is missing the link structure the ESPI 4.0 objective requires.

    Suggested acceptance:

    • MapStruct mappers populate id, selfLink, upLink, relatedLinks, published, updated on the DTO from the entity (or via the Atom-feed assembly layer)
    • A marshalled feed entry validates against espi.xsd with self/up/related links present
  7. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    Master remediation roadmap (2026-06-05) — sequenced to close all validly-open issues

    Backlog triaged after the Phase-2.0 cross-service-contract chain landed (#122/#148/#149/#150/#157/#158/#160). This issue (#119) is the canonical roadmap home — kept current here rather than in a repo doc.

    Closed during triage: #112 (obsolete — SAS stays its own module), #132 (resolved — StreamingResponseBody removed in #119 Phase 1, CME gone).

    Already done on main: #119 Phase 1 (de-streaming) ✅.

    Waves (DC-first; TP last per project lead)

    * ReadingType is shared metadata — confirm scoping semantics against the spec before filtering.

  8. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    ⭐ CANONICAL BUILD PLAN — DC ESPI Resource Surface (GET-first) — 2026-06-05

    (Supersedes the high-level wave note above. Source: Don's authoritative API-surface xlsx — DC Usage Root/XPath/Collection + Customer XPath sheets.)

    Direction

    Build the canonical ESPI resource API surface first (ROOT + XPath, collection + individual), via a generic EspiResourceController base + ScopeResolver. Subscription scoping becomes intrinsic (the ScopeStrategy), ESPI/Atom/XSD compliance is the base's emit contract, and Batch/Subscription|Bulk|RetailCustomer compose over the surface. This single workstream delivers #98 and subsumes #119 Ph2–Ph8 + #101 (validated continuously).

    Confirmed decisions

    1. All IDs are UUID v5 — RetailCustomerEntity migrates Long→UUID in Foundation (no special-casing; UUID everywhere).
    2. ReadingType = shared measurement metadata, scoped by reachability (visible iff referenced by an in-scope MeterReading), kept with the UsagePoint/MeterReading triad for flow/cost/commodity evaluation + FB-based UsagePoint selection.
    3. UsageSummary = the ESPI 3.2 rename of ElectricPowerUsageSummary (covers electricity, natural gas, water).
    4. CRUD POST/PUT/DELETE deferred — these are admin/sandbox-DB-management APIs (secondary). Existing ESPI resource write controllers are being deleted now; the surface is GET-only until the admin-CRUD track is scheduled.

    Sequence + estimates (engineering-days; one PR per numbered item)

    F — Foundation (~4 d): F1 EspiResourceController base (2.0) · F2 ScopeResolver/ScopeStrategy (1.0) · F3 ParentChain+ResourceDescriptor+RetailCustomer UUID migration (0.5) · F4 XSD+disjoint-token test harness (0.5)
    E — Energy (~5.5 d): E1 UsagePoint+MeterReading+ReadingType triad (2.0) · E2 IntervalBlock (0.5) · E3 EPQS (0.5) · E4 UsageSummary (0.5) · E5 LocalTimeParameter (0.5) · E6 RetailCustomer (0.5) · E7 Subscription (0.5) · E8 ProgramIdMappings (0.5)
    B — Batch composition (~2 d): B1 Batch/Subscription (1.0) · B2 Batch/Bulk (0.5) · B3 Batch/RetailCustomer(+/UsagePoint) (0.5)
    P — PII (~5.5 d): P1 Customer (1.5) · P2–P9 CustomerAccount, CustomerAgreement, ServiceSupplier, ServiceLocation, EndDevice, Meter, Statement, ProgramDateIdMappings (0.5 ea = 4.0)
    A — Status/admin GET (~1.75 d): A1 ReadServiceStatus (0.25) · A2 ApplicationInformation GET (0.5) · A3 Authorization GET + per-TP filter [#141] (1.0)
    X — Cross-cutting (~4.5 d): X1 Security matchers (1.0) · X2 end-to-end XPath-tree IT (1.0) · X3 entity↔XSD drift [#101] (1.5) · X4 JSON-removal/Atom sweep [#119 Ph3] (1.0)

    Total ≈ 23–25 eng-days for the full GET surface (delivers #98; subsumes #119 Ph2–8 + #101). Deferred admin CRUD writes track: +5–8 d, scheduled later.

    Estimates are effort-days (your issues' unit); wall-clock with implement-and-review cadence is materially shorter.

  9. added a commit that references this issue on Jun 5, 2026
  10. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    🔧 Correction to the canonical build plan — RetailCustomer is NOT an ESPI resource

    Per ESPI authority: RetailCustomer is not defined in espi.xsd and has no Atom/XML stream — it is only the local correlation between ESPI data and a user account. Consequences for the plan above:

    1. No RetailCustomer→UUID migration. It stays Long. The previously-proposed PR-0 (and Remediation: ESPI REST controller modernization (post-#116) #119 Phase 4's "Long retailCustomerId→UUID") are dropped. UUID v5 applies to real ESPI resources only.
    2. RetailCustomer is a Long path-container / scope key, not a Phase-E resource slice. There is nothing to marshal, so drop the E6 "RetailCustomer" resource. The energy resource slices are: E1 UsagePoint+MeterReading+ReadingType triad, IntervalBlock, EPQS, UsageSummary, LocalTimeParameter, Subscription, ProgramIdMappings.
    3. The base controller's ParentChain handles mixed key types: RetailCustomer = Long, every real ESPI resource = UUID. customerResourceURI = Batch/RetailCustomer/{Long} (already non-UUID via EspiBatchUri.retailCustomerId).
    4. The standalone GET /RetailCustomer(+/{id}) endpoints are admin/local-account correlation → deferred admin track, not the ESPI resource surface.

    Net effect: Foundation loses its riskiest item (no PK migration). Revised next step = go straight to F1 (base EspiResourceController). Estimate drops ~0.5–1 d.

  11. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    🔧 Correction — ESPI has no pagination (CMD/DMD are machine-to-machine)

    Per ESPI authority: the standard does not support pagination. Connect-My-Data and Download-My-Data are machine-to-machine bulk exchanges, so there is no limit/offset (nor OpenSearch max-results/start-index).

    • F1 base EspiResourceController has NO pagination — a collection endpoint returns the full scoped Atom feed.
    • The @RequestParam limit/offset currently on the live GET controllers (UsagePoint, Customer, CustomerAccount, MeterReading, …) are a non-ESPI deviation; they are removed as each resource migrates onto the base (Phase E) — no separate cleanup PR.
    • Remediation: ESPI REST controller modernization (post-#116) #119 Phase 6 pagination item (OffsetPageable, limit/offset) is dropped.
    • Distinct from pagination: ESPI date-range content filters (published-min/max, updated-min/max) remain, on the feeds where the spec defines them (the DC→TP notification contract test + TP-receiver stub #158 notification URIs already use published-min/max). These are content filters, not pagination.
  12. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    🔧 Refinement — ESPI query parameters (REQ.21.6.2.8, p.176–177)

    The standard defines feed query parameters and states: "Only published-max/min and updated-max/min are required to be supported." Full list:
    published-min/max, updated-min/max, max-results, start-index, start-after, depth.

    Design for the F1 base EspiResourceController:

    • Support (REQUIRED): timestamp content filters — published-min, published-max, updated-min, updated-max (applied as repository range queries on IdentifiedObject.published/updated).
    • Do NOT implement (OPTIONAL per spec): max-results, start-index, start-after (pagination) and depth. ESPI does not require pagination — CMD/DMD are machine-to-machine. (Supersedes the earlier "no query params / full feed only" note: the timestamp filters ARE required; only pagination/depth are skipped.) The pre-existing non-ESPI limit/offset params are removed as resources migrate onto the base.

    Net F1 contract: full scoped Atom feed, optionally narrowed by the four required timestamp filters; no pagination.

  13. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    🔧 Refinement — optional query params codified as optional DB-query knobs

    The optional REQ.21.6.2.8 params (max-results, start-index, start-after, depth) are DB-query parameters that certification checks only for data-format conformance / acceptance, not filtering behavior. Decision: accept all four as optional, null-safe query knobs bound into the DB query — so the endpoint conforms to the full ESPI parameter set without pagination being required behavior.

    F1 base controller binds an EspiQueryOptions value object via @RequestParam(required=false) under the ESPI names:

    • REQUIRED (functional WHERE predicates): published-min/max, updated-min/max on IdentifiedObject.published/updated.
    • OPTIONAL (codified, null-safe; applied only if present): max-results → query limit, start-index → offset (both trivially honored); start-after → id cursor and depth → feed-expansion are accepted best-effort/no-op (documented as format-only, not behavior-tested).

    Net: endpoint accepts the complete REQ.21.6.2.8 set (no 400s), required filters filter, optional params are present-but-optional DB knobs. Supersedes the earlier "do not implement the optional params" note — they are accepted/codified-optional rather than rejected.

  14. dfcoffin commented on Jun 5, 2026

    @dfcoffin
    ContributorAuthor

    🔧 Clarification — ALL query parameters are optional (incl. the date filters)

    No required-vs-optional split in request binding. Every REQ.21.6.2.8 parameter is optional on the request; the spec's "required to be supported" means the DC implements the date-filter behavior, not that a caller must supply it.

    • EspiQueryOptions is all-nullable; every param binds @RequestParam(required = false).
    • Zero params → full scoped Atom feed. Any subset is valid — never a 400 for omitting (or supplying) any param.
    • When present: published/updated min/max are honored as WHERE predicates (DC supports them); max-results→limit, start-index→offset honored; start-after/depth accepted best-effort/no-op.

    Supersedes the "required four / optional four" framing — the only distinction is behavioral depth when a param happens to be supplied, not request-time requiredness.

  15. dfcoffin commented on Jun 6, 2026

    @dfcoffin
    ContributorAuthor

    ✅ F1 scope clarification — reuse the existing (tested) Atom/DTO/IdentifiedObject layer

    The Atom-envelope ↔ IdentifiedObject mapping and the DTO marshalling are already implemented and tested in openespi-common (this is what aligned the DB DDL with espi.xsd/customer.xsd). Per REQ.21.6.2, identity/naming lives in the Atom <entry> envelope (mRID→atom:id urn:uuid:, description→atom:title, published/updated, self/up/related links) — consistent with the near-empty IdentifiedObject complexType (extends Object+extension, adds only batchItemInfo).

    F1 therefore re-implements NONE of that — it is purely an orchestration layer over the proven UsageExportService/CustomerExportService + mappers + Atom DTOs. F1's only new code:

    1. EspiResourceController base (ROOT + XPath, collection + individual; delegates marshalling to the existing export services).
    2. ScopeResolver/ScopeStrategy (energy usage-point set, PII retail-customer set, ReadingType reachability).
    3. ParentChain validation (mixed keys: RetailCustomer=Long, resources=UUIDv5).
    4. EspiQueryOptions (all-optional REQ.21.6.2.8 params → repo query).
    5. 404/403 semantics.

    Net: F1 is lighter/lower-risk than the original 2.0 d estimate — composition, not creation.

  16. dfcoffin commented on Jun 6, 2026

    @dfcoffin
    ContributorAuthor

    🔧 E1 entry-assembly contract — atom:link rules (Don, from a GBA certification feed)

    The existing marshalling renders links present in the DTO but does not populate them — link/URI assembly is E1's job, layered on the tested marshalling.

    • atom:link appears ONLY on <entry>, never on <feed> (Sandbox). The CMD-certification feed-level rel="related" type="text/xml" certificate link is NOT required for the Sandbox → omit. So a feed = id + title + updated + entries.
    • Each <entry> carries: rel="self", rel="up", and one rel="related" per reachable related resource whose FB is in the token scope (gated via ScopeResolver/EspiScope). Related set per REQ.21.6.2: UsagePoint→{MeterReading, UsageSummary, ElectricPowerQualitySummary, LocalTimeParameters, ProgramIdMappings}; MeterReading→{ReadingType, IntervalBlock}; etc.
    • type attribute rule: espi-feed/{Resource} when the href has no index (a collection URL ending in the resource name); espi-entry/{Resource} when the href ends in an id. ⇒ self→espi-entry/…, up→espi-feed/…, related→by its href shape.
    • hrefs are the XPath form under /Subscription/{subscriptionId}/…; atom:id = urn:uuid:{mRID}.

    Open for E1: the GBA cert feed uses the default (unprefixed) Atom namespace; the current tested UsageExportService emits atom:-prefixed Atom. Confirm whether to switch the marshaller to default-ns or keep the prefix before E1.

    F1 (the scope core) is unaffected by all of the above and remains the input the related-link FB-gating consumes.

  17. added a commit that references this issue on Jun 6, 2026
  18. dfcoffin commented on Jun 6, 2026

    @dfcoffin
    ContributorAuthor

    🔍 Audit: atom:link <<link>> graph vs entity relationships + DDL

    Cross-referenced the NAESB 4.0 UML <<link>> associations against the JPA entities and the Flyway DDL, to verify the entry-assembler can actually resolve each related resource id.

    Energy (E2–E4): ready ✅ — entity navigation present for UsagePoint↔{MeterReading, ElectricPowerQualitySummary, UsageSummary, LocalTimeParameters} and MeterReading↔{IntervalBlock, ReadingType}. One gap: no ProgramIdMappings entity → #168 (E2 emits its other 4 related links and omits ProgramIdMappings until then).

    PII (P-phase): blocked on entity mappings ❌ — the DDL has the FKs, but CustomerAgreement/CustomerAccount/ServiceLocation/EndDevice/Meter entities don't map them: CustomerAccount↔CustomerAgreement, CustomerAgreement↔{ServiceLocation, ServiceSupplier, ProgramDateIdMappings}, ServiceLocation↔{EndDevice, Meter, LocalTimeParameters}. So the PII XPath chain isn't navigable and those related links can't be emitted → #169 (pure JPA mapping of existing FKs; DDL unchanged; prerequisite for the P-phase). Already-mapped/OK: Customer↔CustomerAccount/Statement/TimeConfiguration, *↔Statement, UsageSummary↔Statement (cross-domain).

    Net: E1/E2 (energy) proceed now; #168 + #169 land before the resources that need them. Both are #101/#123-adjacent.

  19. dfcoffin commented on Jun 11, 2026

    @dfcoffin
    ContributorAuthor

    E1 of the DC ESPI Resource Surface (GET-first) plan is now tracked in #187 — resource engine + Atom entry-assembly (GET-first), first slice = UsagePoint→MeterReading→ReadingType triad, fully scope-enforced (consumes the F1/#167 ScopeResolver). Namespace gate cleared (keep explicit atom:/espi: prefixes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ESPI 4.0Touches the NAESB ESPI 4.0 implementationNAESB ESPI StandardRequired to comply with the NAESB ESPI standardPolishClean up unnecessary logic or code smellsblockingBlocks other work or CIschema-complianceData elements comply with their appropriate ESPI schema definitionstaskA general task

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions