feat(datasource-pylon): Pylon datasource for the Ruby agent (EXT-4) - #369
Conversation
…341) * feat(pylon): scaffold datasource gem with config and resilient client Story 1 of the Pylon datasource (EXT-5). Adds the forest_admin_datasource_pylon gem skeleton: Zeitwerk autoloading, typed error hierarchy with an APIError carrying HTTP status and parsed body, configurable logger, Configuration with api_key validation, and a Faraday client authenticating with a Bearer token plus a GET /me health check. The Faraday middleware order is deliberate and differs from the Mambu Payments gem: raise_error sits outside the JSON parser so errors carry an already-parsed body, and retry sits innermost so it can observe raw statuses. Behind raise_error the retry middleware never sees a 429 and retry_statuses silently does nothing. Non-idempotent verbs are only retried on 429, where Pylon rejected the request before processing it. Wires the package into the CI lint, test and coverage jobs. The semantic-release publish pipeline is intentionally left untouched until Story 9, so an incomplete gem is never pushed to RubyGems. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…agination (#347) * feat(pylon): add PylonIssue read-only collection with cursor pagination Registers the first Pylon collection: issues in list + record-detail mode, backed by POST /issues/search and GET /issues/{id}. Forest asks for an offset/limit window while Pylon only hands out the next page of a cursor, so CursorWalker walks pages until the window is covered then slices. The walk is capped (20 pages / 5000 records, with a truncation warning) because /issues/search allows 20 requests per minute and an uncapped deep-offset walk would spend an agent's whole budget on one list view. It also stops defensively on an empty page or a cursor that does not advance. The schema follows the live API rather than the ticket: Pylon has no priority field, first_response_time/resolution_time are RFC3339 timestamps and not durations, and /issues/search exposes no sort parameter, so no column is sortable and translate_sort is not ported from Zendesk. Nested account/requester/assignee/team objects are flattened into id columns until the related collections exist. Search and Count default to disabled in BaseCollection, the inverse of the Zendesk template, since both land with the condition-tree translator. Until then a condition the collection cannot honour is dropped with a warning naming what was discarded, so an unfiltered result set does not read as a filtered one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#351) * feat(datasource): translate Forest filters for Pylon search PylonIssue honoured no filter but the primary-key short-circuit, so a segment or a UI filter returned unfiltered rows that looked filtered. - Query::ConditionTreeTranslator maps a condition tree onto the structured JSON filter of POST /issues/search: native OR through subfilters, a depth-3 guard, and timezone-aware date bounds. Anything the API cannot express raises instead of being dropped. - Issue::ApiFilters transcribes Pylon's per-field allow-list and is the single source of truth: define_schema derives every filter_operators from it, so the schema can no longer advertise a filter the API refuses. Declaring the bare comparisons on date columns lets the toolkit rewrite Today / Previous* into a pair of bounds, so time_range is never emitted. - filter.search is forwarded as search_text (enable_search). - extract_id_lookup also pulls the id leaf out of a top-level AND and returns the leftover conditions, applied in memory. Forest sends AND(id equal X, scope) on a record detail as soon as a scope is set, and id is not a Pylon filter field. - translate_sort / timezone_for ported from the Zendesk base collection. /issues/search has no sort parameter, so PylonIssue's allow-list is empty and a requested order is reported rather than silently swallowed. Count stays disabled: Pylon exposes neither a count endpoint nor a total, so counting means walking pages against a 20 req/min budget on every list view. Moved to the hardening story that owns throttling. 176 examples, 0 failures, 100% line coverage, rubocop clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gem was registered in build.yml and .rubocop.yml but not in .releaserc.js, so semantic-release never bumped its version, never built it and never pushed it, while CI stayed green: the omission has no failing check to surface it. Added to the three places a package has to appear -- the prepareCmd sed, the successCmd build and push, and the git assets, so the bumped version.rb travels with the release commit. query/filter_value.rb requires active_support/core_ext/time/zones and calls Time.use_zone, which only the monorepo Gemfile was providing. The zendesk and mambu datasources both declare activesupport >= 6.1; without it, installing the gem on its own gives a datasource that raises on the first date filter it translates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(pylon): client endpoints for accounts, contacts, users and teams Adds search/list/fetch methods for the four new resources, routed through shared private helpers; search_issues and fetch_issue now delegate to the same helpers with unchanged behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(pylon): embed the issue conversation thread
Ports the Zendesk comments embedder to Pylon: the messages of an issue are
read through GET /issues/{id}/messages and embedded as a structured array
column on PylonIssue.
The thread is asked for without a limit, so Pylon answers with the whole
conversation in one request; a page would have handed back the oldest
messages and cut the recent ones off. Authors are flattened from the payload
Pylon already nests in each message, so no author lookup is spent at all.
The fan-out is bounded like the primary-key lookups of this collection: the
endpoint allows 20 requests per minute and a thread costs one request per
row, so rows past MAX_MESSAGE_EMBEDS are left at nil -- unknown, never the
empty list, which would read as "this issue has no message". A thread that
cannot be read degrades the same way instead of failing the page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: bound every job and harden the unixODBC install The workflow declared no timeout anywhere, so a job that hangs pins the run for GitHub's 6 hour default. A hung `apt-get` on the two Snowflake lint legs did exactly that: `test` needs `lint`, and `coverage` needs `test`, so one stuck leg of the matrix froze the whole pipeline and no test ever ran. Lint, test and coverage now carry a timeout, and the apt step that hangs carries its own, tighter one, plus retries on the mirror and a noninteractive frontend. A failing leg is now visible in minutes and can be re-run. Deploy is left unbounded on purpose: cutting semantic-release off in the middle of pushing thirteen gems is worse than waiting for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit c5df079)
* feat(pylon): client endpoint for custom fields GET /custom-fields takes a mandatory object_type, so the definitions of each collection are read by their own call. The walk carries the parameter on every page: dropped on the second request, it would answer for another object type or with a 400. Degrades to an empty list, like the message thread: this is read while the agent boots, and a token missing the permission has to cost the operator the custom columns rather than the whole datasource. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(pylon): write operations (CRUD)
Open the datasource to writes: a `writes` mixin on the client, one method
per Pylon endpoint, and create/update/delete on every collection, through
a mechanism shared by the three collection bases.
What may be written is `is_read_only` on the column, the single source of
truth the payload builder reads, the way `api_filters` already is for
filtering. Pylon's own `is_read_only` is now honoured on a custom field,
and a value is written back through the list of `{slug, value}` entries
the API takes, `values` for a multiselect and the option slug for a
select.
The verbs Pylon exposes no endpoint for -- no POST or DELETE on a user,
no DELETE on a team -- refuse with a message rather than the contract's
NotImplementedError, which the agent answers as an unexpected 500. So do
the fields it only accepts in one direction: `body_html` on a create,
`state` on an update, and the like, dropped when they ask for nothing and
refused when the operator really changed them.
A filter-driven update or delete resolves its ids exactly or refuses:
an id filter is answered without a request, anything else goes through
the collection's own list so the scope applies, and a selection wider
than one pass of writes is refused rather than written halfway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gins (EXT-12) (#366) * feat(pylon): close and create-with-notification action plugins Two plugins at parity with the Ruby Zendesk ones, built on the write primitives of the previous commit. CloseIssue moves the selected issues to a state, `closed` unless told otherwise -- a custom status slug is taken as readily, Pylon accepting one wherever it accepts a standard state. One variant per scope rather than the four Zendesk builds, there being a single terminal state here. Each id is written under its own rescue and named in the message, so a batch that only half applied never reads as a plain success. CreateIssueWithNotification opens an issue and delivers its first message. Pylon says the delivery outright where Zendesk infers it from a public comment: `destination_metadata.destination` names the channel, and no metadata at all is what leaves the issue internal, which is what the "send as internal note" checkbox writes. The form carries no Type field, `POST /issues` taking none, and the priority it does carry is never read back -- no Pylon read returns one. Both find the issues to act on through one option: `issue_id_field` names a column of the host collection, and its absence falls back to the primary keys, which is what an action registered on PylonIssue acts on. Snooze is left out until its endpoint is confirmed against a live organization; IssueTargets and the messages module are already shared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…comments Review follow-up on the previous commit, no behaviour change. Building the fields hash with to_h removes both the reassignment of a key while the hash is being iterated and the now useless dup of it, and the comments are cut back to the two non-obvious points: the key? read and the copy before the write. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings the 29 commits main gained since 1.36.2, the merge base. Three files were touched on both sides and conflicted, all three because main registered forest_admin_datasource_graphql_hasura where this branch registered forest_admin_datasource_pylon. Every resolution is additive, both packages kept: - .releaserc.js: prepareCmd, successCmd and the git assets list - .github/workflows/build.yml: lint matrix, test matrix, coverage files - .rubocop.yml: the Gemspec/RequireMFA exclusion main also put forest_admin_rails back in the test matrix and moved the coverage job to Ruby 4.0; both come over untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(datasource): proactive throttling for the pylon datasource (EXT-13) Space requests out inside each endpoint's documented budget instead of discovering the quota as a 429. RateLimits holds the per-endpoint figures read off the Pylon API reference, one window per endpoint since Pylon meters per endpoint rather than per token. RateLimiter keeps a sliding window per bucket, reserves the slot a request will take under a mutex and waits outside it, so concurrent callers spread over distinct slots rather than waking onto the same one. Throttle is the Faraday middleware, placed inside `retry` so a replay waits for a slot like a first attempt. The wait is bounded (5s): past that the window is saturated by more than this agent's own traffic, so the request goes out and the 429 retry stays the backstop. Queueing behind a full window would trade a retry the client already handles for a request the operator watches spin. Also corrects the rate-limit figures the comments and one operator-facing message asserted. They were 4x to 15x below what Pylon documents: the search endpoints grant 120 requests a minute rather than 20, the record reads 300 rather than 60, and no endpoint this gem touches sits at the 10 a minute the story assumed. The caps built on those figures are unchanged and re-justified on what actually bounds them - sequential round-trips and unbounded payloads, not the quota. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…on (EXT-17) (#375) * refactor(pylon): extract the faraday connection builder Everything but the timeouts is the same on every connection the client needs, so the builder takes those and holds the rest -- middleware order and the shared limiter included -- in one place. No behaviour change: the single connection is now its memoized call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Number filter the agent cast to Infinity or NaN reached `to_i`, which raises, and then the JSON encoder, which raises too: a 500 either way, on a value the operator typed. It is refused with a 400 naming the field. An action reading its issue ids off a column deduplicates them: that column is not a key, so two selected records could name the same issue, which was then written twice and counted twice in what the action reported back. `collect_pages` stops on a cursor it has already followed, not only on one that did not move: a cursor cycling over three pages walked them again until the page cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MAX_ID_LOOKUPS bounded a selection rather than a page: the ids were truncated to the first 20 before the window was applied, so an offset at or past 20 answered an empty page from records that exist, and a wider selection lost its tail with nothing but a server-side log to say so. The window is now taken off the ids before any of them is read, which also spends one request per record the caller asked to see rather than one per record in the selection. A residual condition takes that away, which records the window holds being known only once they are all read, so past the cap that one selection is refused rather than answered with a fraction of itself. Reading by id moves to its own mixin, the collection having reached the class-length cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A scope that is neither a symbol nor a string raised NoMethodError from `to_sym`, hiding the unknown-scope error that names what was passed. An email template title is both the enum value and the key its content is looked up by, so it has to name one template: a duplicate made the first unreachable and sent the other one's content under its name, and a template titled like the sentinel of the "pick none" option could never be picked at all. Both are configuration, so both are refused at registration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`project` served the whole record when the projection named no column of the collection, so a row asked for as `account:name` alone came back carrying every native column too. The relation is embedded onto the row afterwards; what `project` owns is the columns, and a projection naming none asks for none. Only a nil projection still means the record as it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pylon exposes no count endpoint and no total, so no collection ever passed `countable: true`: the cursor-backed ones refuse `aggregate` outright rather than count the pages they walked, and the two holding their whole dataset are not advertised as countable either. The flag was carried by the constructor and exercised by one spec testing the flag itself. Recorded on EXT-7, whose scope still lists Count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EXT-17 bounded each boot-time introspection per request, not the three of them together: a Pylon that hangs cost the full boot bound once per object type, so a Rails boot could wait around a minute where the ticket promised twenty seconds. The first failure now stands for the rest, which it does in practice: a Pylon that is down, or a token missing the permission, fails the two that follow the same way. `fetch_custom_fields` degrades to nil rather than to an empty list so the introspector can tell that apart from an organization that simply defined no custom field, which says nothing about the next object type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Last open item of EXT-13. Covers configuration, the five collections and what each one can be written through, the action plugins, and the two things an operator needs to know before deploying: what Pylon cannot do and how this refuses it rather than answering something that looks right, and how the per-endpoint throttling behaves under saturation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The caveat said a scope on PylonIssue does not bound what the action closes, on the grounds that the state is written through the client. The ids are resolved first, through the collection the action sits on, and the agent intersects the operator's scope into the filter that reads them: so mounted on PylonIssue, the scope bounds exactly what it closes. What the caveat describes is the host-collection form, where the ids come off a column and that column is the authority. Also names the discrepancy the uncapped batch has with the collections themselves, which refuse a filter-driven write past MAX_WRITE_REQUESTS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pylon has no join, so a related record is read through its own endpoint, which takes no field list: the foreign collection hands back everything it holds and the embedder nested all of it. A row asked for as `account:name` came back carrying every column of the account. The agent redacts a projection per collection since #365, dropping the paths through relations the caller may not read, so the columns served that way were also the ones their permissions had taken out. Nothing downstream cuts them back: RelationCollectionDecorator returns early when the projection is unchanged, which it is here, no relation being emulated. The nested record is now cut to the fields the projection named, the way `project` cuts the row it sits on. This is the relation half of cc01551, which made the same argument for the native columns. The primary key is unioned back in rather than assumed: the route's projection carries it, but the nested record is what the serializer reads an included resource's id off, and a caller inside the agent may project without it. By hand rather than through `Projection#with_pks`, which also walks the relations of the projection it is given and dereferences their schema with no nil guard -- a path reaching through a relation the collection does not declare would raise there, where it used to be ignored. The sub-projections are rebuilt rather than read off the argument: `list` is also called with a plain array of field names, and a relation read from one of those would go unprojected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`api_key` is a bearer token behind an attr_reader, and nothing prints a Configuration on purpose: what reaches an `inspect` is a Rails error page, or a `logger.debug` of something holding one. Masking the Configuration is one path of three. The token also rides in the headers of the client's two memoized Faraday connections, and `Faraday::Connection#inspect` prints those in clear, so a Client leaks it whatever the Configuration does about its own. Every collection reaches the same connections through the `@datasource` the toolkit's Collection keeps. Configuration, Client and Datasource each mask their own; together they cut every path from an object this package hands out to the credential. Each keeps what made it worth printing -- the base url, the collection names -- and the Datasource one also spares the recursive dump the default walks into, a datasource and its collections pointing at each other. The spec asserts the whole graph rather than the Configuration alone, so a new holder cannot reopen a path quietly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`extract_data` hands an envelope carrying no `data` straight back, which is what a read of an unwrapped body wants. `Array()` then split that envelope into [key, value] pairs, and the collection serialized them into rows holding nothing: a page that looks answered and is empty, on every endpoint read as a list. The write path already refuses exactly this shape, and for the reason that applies here too -- `extract_written` says the collection would serialize the envelope into a record with no id. The read half now says the same, one extractor per shape, naming the endpoint that broke the contract. An absent or null `data` stays the empty answer it is: Pylon spells "no record" that way, and a search matching nothing is not a broken contract. The two best-effort callers keep degrading rather than raising: a malformed thread costs the conversation column, and a malformed custom-fields response boots the datasource on its native schema. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The walk compared the next cursor to the previous one alone, so it caught a cursor that did not move and nothing wider. A cycle spanning two pages ran until MAX_PAGES or MAX_RECORDS cut it short, and handed the same records back several times over as if they were distinct rows. `Client#collect_pages` already keeps the whole set, that being the hardening b9cb7d1 landed on it; the walk now stops on the same terms. The loop moved into a private `collect` to keep `walk` at what it does -- bound the window and slice it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PylonUser and PylonTeam read their endpoint in one response, so the records in hand are every record Pylon holds and a count over them is the figure a server-side count would have given. `FetchAllCollection#aggregate` already answered exactly for that reason, but the collections never called `enable_count`, so the route short-circuited and the panel showed no total it could have served on a request the list makes anyway. They are the exception the refusal was written around: every cursor-backed collection stays uncountable, counting the pages a walk collected being a fraction of a collection presented as the whole of it. Nothing else reads the flag -- Count and CountRelated are its only two readers in the agent, and no OneToMany points at either collection, so no related count is opened by this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ConditionTreeBranch#match` recursed through `every_leaf` / `some_leaf`, which walk all the way down to the leaves. The `all?` of an `And` was therefore applied to the leaves of an `Or` nested inside it, and the aggregator of that inner branch was simply lost: `And(a, Or(b, c))` was evaluated as `And(a, b, c)`. Two ways that surfaced. A scope combined with an "any of" filter -- the shape every route builds, `ConditionTreeFactory.intersect` flattening same-aggregator children only -- silently dropped the rows matching one side of the union. And a nil-unsafe comparison paired with a presence check, the guard a datasource wraps `less_than` / `greater_than` in, stopped guarding: read as `any?`, the presence check answered false on the null and the comparison was evaluated anyway, raising NoMethodError. Recursing through `match` fixes both. For a branch holding leaves only the two forms are equivalent -- a leaf yields itself to either walker -- so nothing changes there, and an empty branch keeps answering what it did: `Or []` false, `And []` true. Only three call sites read this outside the toolkit: the validation decorator, and the two in-memory passes of the Pylon datasource. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plugin writes through the client rather than through the collection, being registered on the host collection rather than on PylonIssue, so none of the caps `Writes` defends applied to it: `apply_state` looped over every id it was handed, one PATCH apiece, sequentially. On a bulk action over a wide selection that is a run the request times out on, leaving part of the selection moved and reporting which part to nobody, `apply_state` only naming what landed once the loop returns. Selected on PylonIssue itself the ids are bounded by the cursor walk, at 5000; on a business collection carrying issue ids they are bounded by nothing at all. The batch is now capped at `MAX_TARGETS`, read off the very constant a filter-driven `update` or `delete` already spends, so the two move together. Past it the run is refused before its first write, with a message naming the count and the cap. The cap counts the issues named rather than the records selected: a column of issue ids is not a key, so a hundred host records naming ten issues stays a batch of ten. The README documented the fan-out as a known limitation; it now documents the bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Review of the assembled branch at
Also: the latest Macroscope finding is not addressed — |
Four findings from the review of the assembled branch, each with the spec that pins it. - update() dropped every read-only and unknown key of a patch and then returned, so a patch reduced to nothing was reported as a successful edit -- the route re-reads the record and hands back the values it already had, showing the operator their change reverting with no reason given. Refused now, naming the keys. A dropped key alongside a real edit is still dropped, and a patch naming no field at all still asks for nothing. - RetryPolicy honoured a Retry-After of up to a full minute on each of three retries, parking the calling thread for three minutes on a request the Forest server had long since timed out, and multiplied by the 20 sequential calls one list view may spend. DEFAULT_MAX_INTERVAL is 12 rather than 65, the figure that keeps the sleeps of a whole request under a minute; past it a 429 surfaces as an error instead of being waited out. - Client#fetch_all ignored the pagination block, so the day Pylon paginates GET /users or GET /teams its first page would silently become "everything" and FetchAllCollection would keep calling its filter, sort and count exact over a fraction of the collection. The cursor is followed now, and refused rather than truncated past the page cap. - A residual combined with a page window was pinned by nothing: every residual spec used a filter without a page, so a regression slicing the ids before applying the residual would have answered pages with records silently missing and passed the suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in .github/workflows/build.yml, resolved in favour of main. #380 replaced the lint job's per-package matrix with a single job running one root `bundle exec rubocop`, so the third place this branch registered the package no longer exists. Its two other entries auto-merged and are kept: the test matrix and the "Send coverage" files list, 13 packages each. Nothing is lost by dropping the lint entry -- the root .rubocop.yml has no Include, so one rubocop run already walks packages/**, which is what main's CLAUDE.md now says to rely on. That same CLAUDE.md edit fixes the step-5 instruction this branch had followed, so the guidance and the workflow agree again. Verified on the merged tree: toolkit 485, customizer 703, pylon 815, agent 1179, active_record 203 examples, 0 failures; rubocop 911 files, 0 offense. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`base_url` is operator-supplied, and an egress proxy fronting Pylon is spelled `https://user:pass@proxy.internal`: that user-info is a credential, and both places that print the url printed it in clear -- a Rails error page or a `logger.debug` of a Client or a Configuration. Configuration#redacted_url is now what the two inspects print. The finding named Client#inspect alone; Configuration#inspect printed `@base_url` the same way, so masking only the one it named would have left the other path open. Cut with a regexp rather than through URI: nothing validates that the base url parses, and an inspect raising on the way to a Rails error page would replace the page with its own failure. A spec pins that, and one pins a url carrying no user-info being printed as it is -- placing the deployment is what makes an inspect worth printing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The eighth review pass, verified one by one. Six of its twelve findings led
to a change; the six others are answered in thread.
- `collect_pages` treated two of its four stops as the end of the
collection. An empty page advertising a cursor, and a cursor answered
twice, both left records behind and `fetch_all` still called the result
exact -- the same hole the previous commit had closed for the page cap
alone. Only `next_cursor` being nil is Pylon saying there is nothing
more; every other stop is now a truncation, which the caller reads as a
warning or as a refusal.
- `coerce_number` reached an integer through a Float, so a custom field
holding more than 2**53 was displayed and compared with its last digits
rewritten. Parsed as an integer first, in base 10 explicitly: Ruby reads
`Integer("012")` as octal, and the suggested patch would have renumbered
every zero-padded value Pylon holds.
- `FilterValue` checked the timezone stripped and then stored it unstripped,
so a padded identifier passed the blank guard, failed the zone lookup and
fell back to UTC -- a day boundary off by the offset.
- `CloseIssue` registered one action per scope through `add_action`, which
keys them by name: two variants configured with the same name left the
second overwriting the first, so the panel offered one action scoped for
the wrong selection and nothing said the other had gone. Refused at
registration, like the plugin's other silent misconfigurations.
Two outside the package, both found by verifying a finding rather than in
it:
- `Charts#compute_value` indexed `result[0]['value']` while
`Aggregation#apply` answers an empty selection with no row at all, so a
filtered value chart matching nothing was a 500 where the figure is zero.
Reachable from every collection that aggregates in memory, `PylonUser`
and `PylonTeam` being the first to do so exactly.
- `Aggregation` compared `Max` with `<`, so it kept the smallest value it
saw: every Max chart over an in-memory aggregation answered with the
minimum. No spec covered Max at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… read `:action_name` taken as a Symbol was registered as one, and a Symbol name breaks the agent rather than the action: `GeneratorAction.get_action_slug` calls `strip` on it, and `GeneratorCollection` sorts the names of a collection's actions, which raises `comparison of Symbol with String failed` as soon as one sits beside a String. Schema generation, so the whole agent, for a plugin option typed the way the `:scopes` option already accepts. Coerced through `to_s`, like `normalize_scopes` does -- which also makes `refuse_colliding_names` read `:Resolve` and `'Resolve'` as the one name they are, where it had just let them register as two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🎉 This PR is included in version 1.41.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |

Integration branch for EXT-4 —
forest_admin_datasource_pylon, feature parity with the Zendesk datasource.The ten stories of the epic are Done and merged,
mainis merged in, and the branch touches exactly three files outside the new package.What is in the branch
A new
forest_admin_datasource_pylonpackage: 44 files underlib, 24 spec files, 786 examples, 100.0% line coverage (1686/1686) and 94.5% branch coverage. Ten story PRs, all reviewed and merged into this branch:PylonIssueread-only, cursor paginationThat is every sub-issue of EXT-4 and every PR ever opened against this branch. Re-checked against
Linear on 2026-08-27: EXT-4 has exactly these ten children, all
Done, and no eleventh — so thetable above is the whole epic. EXT-4 itself is the only thing left
In Progress, this PR being whatcloses it.
Three follow-ups were filed out of the review rather than fixed here. They are not sub-issues of
EXT-4 — each is repo-wide, not Pylon's — and they are listed so the table is the full Linear picture:
.releaserc.jschains all fourteensuccessCmdpushes with;Shape of the package:
client.rb+client/writes.rbover the Pylon API with aretry_policy, arate_limiter/rate_limits/throttletrio metering each endpoint in its own sliding window, abase_collectionspecialised intocursor_collection/fetch_all_collection, one directory per collection (account,contact,issue,team,user) each with its ownapi_filters/schema_definition/serializer, aquery/layer translating condition trees into Pylon filters,pagination/cursor_walker,schema/custom_fields_introspector, and two action plugins.Review pass on the whole branch
65190b9fmergesmain; the two conflicts were both the#368sort-decorator cherry-pick, resolved in favour ofmain, so those files have left this diff. Fifteen commits then close out the review of the assembled branch — these are direct commits, so they appear in no story PR above:b9cb7d15Numberfilter cast toInfinity/NaN; deduplicate action issue ids; stopcollect_pageson a cursor already followeda93474b1MAX_ID_LOOKUPSbounds a page rather than a selection24951097cc015514373f94decountable:flag nothing opts into7033ba1dbd55e8ba2a1b4d6d67cd786a1fd03cbd96a0575aCloseIssueand what does not0a2d4e4dversion.rbwith the other packagesd75bab2cidconditions of anand, on both the read and the write path524cc95eb6bd085bCloseIssueDefects fixed
b9cb7d15— aNumberfilter the agent cast toInfinityorNaNreachedto_i, which raises, and then the JSON encoder, which raises too: a 500 either way, on a value the operator typed. Now refused with a 400 naming the field. Same commit: an action reading issue ids off a column deduplicates them, andcollect_pagesstops on a cursor it has already followed rather than only on one that did not move.a93474b1—MAX_ID_LOOKUPSbounded a selection where it should bound a page: the ids were truncated to the first 20 before the window was applied, so an offset at or past 20 answered an empty page from records that exist. The window is now taken off the ids before any of them is read, which also spends one request per record the caller asked to see. A lookup carrying a residual condition cannot be paged that way and is refused past the cap rather than answered with a fraction of itself.24951097— a plugin scope that is neither a symbol nor a string raisedNoMethodErrorinstead of the error naming it; duplicate email-template titles, and a template titled like the "pick none" sentinel, are now refused at registration instead of silently sending the wrong message.cc015514—projectserved the whole record when the projection named no column of the collection, so a row asked for asaccount:namecame back carrying every native column too. Relevant now thatmainhas landed serve only the columns of collections the caller may read. It also changed what an empty projection returns, which is not that case at all;2a1b4d6dseparates the two.7033ba1d— EXT-17 bounded each boot-time introspection per request, not the three of them together, so a Pylon that hangs could add about a minute to a Rails boot rather than the twenty seconds intended. The first failure now stands for the object types after it.Also
373f94de— drops thecountable:flag: no collection ever passed it, Pylon having no count endpoint and no total.bd55e8ba— the package README, last open item of EXT-13.Macroscope: 21 review threads across seven passes, every one answered in thread. The seventh pass landed one finding, on
base_collection.rb:107, and it was real: fixed ind75bab2c, described below. Two of the last four findings were real and are fixed (67cd786a,1fd03cbd); one is a false positive (nil.to_iis0, so the guard already re-raises) and one is not reachable through the agent stack, which rewrites the date operators upstream — both answered with the reasoning. Thenormalize_templatesthread was Macroscope reading a stale index; the method has been there since24951097.Recorded on Linear rather than fixed here:
SnoozeIssue(EXT-12) andaggregate/ Count (EXT-7) are in the epic scope but were never delivered; both are commented on their story with what exists instead. EXT-4's rate-limit figures were low by a factor of 4 to 15 and have been corrected againstrate_limits.rb. The three findings declined as repo-wide now have tickets of their own, so they outlive these threads: EXT-19 (.releaserc.jschaining with;), EXT-20 (no datasource gemspec declares the Forest gems) and EXT-21 (a non-positive page limit).Behaviour outside the new package
Three files, each additively, to register the gem:
.github/workflows/build.yml(test matrix, coveragefiles:),.releaserc.js(prepareCmd,successCmd, git assets) and.rubocop.yml(MFA opt-out, theversion.rbstring cops, and the per-file metric excludes the package actually trips).Two entries in
build.yml, not three: #380 replaced the lint job's per-package matrix with a single rootbundle exec rubocop, which is where the merge conflict was and what219d8bccresolves. That PR also rewrote step 5 ofCLAUDE.mdto say the lint job needs no entry, so the guidance and the workflow agree again -- the four points ofCLAUDE.mdare covered against its current text.Verification
Re-run locally on
cbaa5619, Ruby 4.0.2:forest_admin_datasource_pylon: 827 examples, 0 failures, line coverage 100.0% (1755/1755).forest_admin_agent1180,forest_admin_datasource_toolkit486,forest_admin_datasource_customizer703,forest_admin_datasource_active_record203 — all 0 failures. RuboCop over the whole repo: 911 files, 0 offense.forest_admin_datasource_mongoidcould not be run locally (its gems are not installed on this machine); CI covers it..releaserc.jswas loaded through node to confirm bothprepareCmdandsuccessCmdare valid concatenations, and both YAML files were parsed.forest_admin_datasource_pylonis unclaimed on RubyGems, so the first publish creates it.Three fixes from the seventh pass
Local re-run on
b6bd085b, Ruby 4.0.2: 794 examples, 0 failures, line coverage 100.0% (1695/1695), branch 94.3%, RuboCop 71 files / 0 offense.d75bab2c— anandnames the records all of its conditions name, so twoidleaves name their intersection. Both the lookup short-circuit and the write-resolution bound kept only the first and left the second to be applied later, which put the cap around the wider set: a selection of 25 ids narrowed to 3 was refused over the 25 it named, after reading all 25 one request each. Worth being exact about the failure — the refusal is an explicitUnsupportedOperatorError, never a short page, so this over-refused rather than under-delivered.extract_id_lookupnow partitions theidleaves out of the branch and intersects them, leaving the residual,ensure_residual_appliable!andguard_nil_comparisonsuntouched; disjoint sets reduce to the empty lookup with no request at all.writes.rb#filtered_idsmoves with it, that count being whatrefuse_unresolvable_selectionis worded around — its own comment had been excusing the same shape. Narrow on the read side: onlyPylonIssuereaches theandbranch, cursor collections short-circuiting on a bare leaf alone.524cc95e—IssueTargets.resolve_issue_idsrescuedStandardErrorand answered[], which the action reports as "No Pylon issue selected." That swallowed the datasource's own refusals: a bulkCloseIssueonPylonIssue, with a scope or segment set and more thanMAX_ID_LOOKUPSrecords selected, raises insidelistand the operator was told nothing was selected about a selection they can see they made. A Pylon outage read the same way. AValidationErrornow travels, the agent surfacing its message; everything else still degrades, so a renamed column is still "no issue selected" rather than a stack trace in the panel.b6bd085b— the README'sCloseIssuecaveat had the scope question backwards. It said a scope onPylonIssuedoes not bound what the action closes, on the grounds that the state is written through the client. The ids are resolved first, through the collection the action sits on, andActions#get_record_selectionintersects the operator's scope into the filter that reads them — so mounted onPylonIssue, the scope bounds exactly what it closes. What the caveat describes is the host-collection form, where the ids come off a column and that column is the authority. As written it told an operator a scope bought them nothing when it bought them the whole selection.Human review, and
mainmerged inThree commits since the seventh pass.
a08ab420answers the four findings @matthv raised on the assembled branch,219d8bccmergesmain,e1865da7closes the last open Macroscope thread of the previous pass.a08ab420219d8bccmain, resolving thebuild.ymlconflicte1865da7inspectThe four findings
updatereported a success on a patch it dropped entirely.writable_attributesdrops read-only and unknown keys, and a patch reduced to nothing hitreturn if attributes.empty?: the route then re-reads the record and hands back the values it already had, so the operator watched their change revert with no reason given -- the exact answer the module's preamble says it refuses. Refused now, naming the keys. Deliberately narrow: a dropped key alongside a real edit is still dropped and the edit performed, and a patch naming no field at all (a form whose every field is a relation) still asks for nothing. The refusal also stays settled before the ids are, so a wide selection with an unwritable patch is refused for its patch and spends no request.Retry-Afterwhenever it fits undermax_interval, somax_retries: 3against a per-minute quota meant up to 180s of blocking sleep, plus theThrottlewaits inside the retry -- on a request the Forest server had long since timed out, and multiplied by the 20 sequential calls one list view may spend.DEFAULT_MAX_INTERVALis 12 rather than 65, which is the figure that keeps the sleeps of a whole request under a minute (3 x 12plus4 x RateLimiter::DEFAULT_MAX_WAIT= 56s); past it a 429 surfaces as an error instead of being waited out. A spec computes that bound rather than asserting the constant, so it cannot drift silently. The trade is stated in the code and in the README, with the snippet to choose the other way.Client#fetch_allignored the response'spaginationblock.FetchAllCollectionbuilds its exactness guarantee -- including the package's only exactaggregate-- on "the records in hand are all of them", so the day Pylon paginatesGET /usersorGET /teamsits first page would silently become everything. The cursor is followed now, the way/messagesalready was, and refused rather than truncated pastMAX_COLLECTED_PAGES: a count answered short and presented as exact is the one answer this datasource refuses everywhere else. Zero cost today -- nopaginationblock is advertised, so the walk leaves after the first page and spends the single request it always did.records_by_id-- residual first,page_windowsecond -- was pinned by nothing. The added example was checked against the regression rather than only against green: flipping the implementation to slice the ids first makes it fail withexpected [i3], got [i2].The
inspecthardening, which the previous description had left as a follow-up and which the eighth Macroscope pass then raised as its own finding:base_urlis operator-supplied, and an egress proxy fronting Pylon is spelledhttps://user:pass@proxy.internal. Both places that printed the url printed that user-info in clear.Configuration#redacted_urlis now what the twoinspects print -- the finding namedClient#inspectalone, butConfiguration#inspectprinted@base_urlthe same way. Cut with a regexp rather than parsed: nothing validates that the base url parses, and aninspectraising on the way to a Rails error page would replace the page with its own failure.Open items before merge
e17c4328, and 33498265681 on the merge commit219d8bcc-- the latter being what validates the resolved workflow, the lint job now being the single root RuboCop run of ci(build): halve CI wall clock by parallelizing jobs and caching gems #380 while pylon keeps its two test jobs and its coverage entry. The run forcbaa5619is in flight at the time of writing. Earlier: 33158331718 on3e38d094, 33084670532 onb6bd085b, both successful on Ruby 3.4 and 4.0.e17c4328,cbaa5619), six are answered in thread. What it caught, and what it got wrong:collect_pagestreated an empty page advertising a cursor, and a cursor answered twice, as the end of the collection -- sorefuse_past_caphad closed the exactness hole for the page cap alone. Onlynext_cursorbeing nil is Pylon saying there is nothing more; every other stop is now a truncation the caller reads as a warning or as a refusal.2**53read through a Float (parsed as an integer now, in base 10 explicitly --Integer("012")is 10, so the suggested patch would have renumbered every zero-padded value); a timezone checked stripped and stored unstripped, falling back to UTC on a padded identifier;CloseIssueletting two variants configured with the same name overwrite each other in the collection's action hash; and an:action_namegiven as a Symbol reaching the schema, whereget_action_slugcallsstripon it andGeneratorCollectionsorts it against a String -- schema generation, so the whole agent, not the action.Charts#compute_valueindexedresult[0]['value']whileAggregation#applyanswers an empty selection with no row, so a filtered value chart matching nothing was a 500 where the figure is zero -- reachable from every collection that aggregates in memory. AndAggregationcomparedMaxwith<, so every Max chart over an in-memory aggregation answered with the minimum, on every datasource; no spec coveredMaxat all. Both fixed, with specs.messagesis not a relation: it is a single column whosecolumn_typeis[MESSAGE_THREAD_SCHEMA], a nested object shape the agent projects whole, and every field the finding names (is_private,author_email, both author ids) is a declared member of that column's own type -- covered by the operator's permission on the column. No agent path produces themessages:sub-projection it assumes. The comparison withRelationEmbedderis what misleads: that one re-projects because it embeds records of another collection, carrying their own permissions.update_summary_in_placeguards withif value, sofalseis skipped and noTrueClass/FalseClasscomparison ever happens. It answers wrongly rather than raising -- a BooleanMincan never reportfalse-- which is filed with theSum/Avg-over-a-String finding as EXT-23, both being the same method applying an operation the column type cannot take.andcarrying anidis refused instead of resolved, on the read and the write path (EXT-22). It fails closed, andConditionTreeFactory.groupflattens one level, so every shape the agent builds keeps itsidat the top level.Date/Dateonlyoperators the validator rejects are described in the comment aboveTIME_OPS, with the toolkit contradiction tracked as PRD-989. And one left as it is:Time/DateTimefilter bounds losing their fractional seconds is unreachable through the agent,Transforms::Timesalready handing over second-truncated strings from the sameiso8601call one layer up.function-parameters(count = 4), 16function-complexity, onefile-complexityonbase_collection.rband oneidentical-codeon the scaffoldGemfiles -- which are identical becauseCLAUDE.mdasks for them to be mirrored. The repo carries no.qltyconfig,qlty checkreports No blocking issues, and the three raised on this pass's own code (collect_pages,refuse_truncated_walk,log_truncated_walk) are all the 4-parameter threshold, where each parameter is a value the message or the walk names. Resolved so that an open thread on this PR means feedback still to answer, rather than advisory noise.CloseIssue's batch is uncapped: one request per selected record, so a wide bulk selection is a long sequential run the request may time out on, leaving what it already closed closed. Stated in the README (96a0575a, reworded inb6bd085b); putting a cap on a bulk action is a product call, not a defect fix. Worth knowing thatWrites::MAX_WRITE_REQUESTSrefuses exactly this shape at 20 requests on the collection itself, so the datasource currently answers the same question two ways.NIL_UNSAFE_OPERATORSandresidual_leaf_appliable?are derived from two different sets, so the guard is narrower than what the residual admits and only the upstreamOperatorsEquivalencerewrite closes the gap. Not reachable today; deriving the guard from the equivalence touches the filtering path of every collection.Configuration#validate!checks neither thatbase_urlparses nor that it is https, so a typo surfaces as a rawURIerror at the first request and anhttp://base url would carry the bearer token in clear on the wire. Refusinghttpwould break local mock servers, so this wants a warning rather than a refusal, in its own change. (What aninspectprints is no longer part of this item:e1865da7closes that half.)🤖 Generated with Claude Code
Note
Add
ForestAdminDatasourcePylondatasource with API client, collections, and pluginsForestAdminDatasourcePylongem, exposingIssue,Account,Contact,User, andTeamcollections backed by the Pylon API.Pagination::CursorWalkerto handle cursor-based pagination.CloseIssueandCreateIssueWithNotificationplugins for issue management actions.ConditionTreeBranch#matchin condition_tree_branch.rb to preserve nested branch structure during matching.ConditionTreeBranch#matchfix changes how nestedAnd/Orbranches are evaluated; verify existing toolkit consumers depending on flattened leaf evaluation are unaffected.Changes since #369 opened
ForestAdminDatasourcePylonpackage [e1865da]ForestAdminDatasourcePylon::Plugins::CloseIssue.name_forto coerce action names to strings and added documentation comments [cbaa561]ForestAdminDatasourcePylon::Plugins::CloseIssue[cbaa561]Macroscope summarized 219d8bc.