Conversation
Coverage Report for CI Build 30911640308Coverage decreased (-0.06%) to 98.265%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
❌ 26 blocking issues (33 total)
|
![qltysh[bot]](https://avatars.githubusercontent.com/in/890766?s=60&v=4){kind=small}
| format.json { render json: { results: @results, pagination: @pagination, errors: @errors } } | ||
| format.turbo_stream { render :results } | ||
| format.html { render :results } | ||
| end |
There was a problem hiding this comment.
| @results = NormalizeTimdexResults.new(raw_results, @enhanced_query[:q]).normalize | ||
| @filters = extract_filters(response) | ||
| @append_results = @results | ||
| @load_more = load_more_from_pagination(@pagination) |
| @errors = data[:errors] | ||
| @pagination = data[:pagination] | ||
| @show_primo_continuation = data[:show_primo_continuation] | ||
| @load_more = data[:load_more] |
| def combine_errors(*error_arrays) | ||
| all_errors = error_arrays.compact.flatten | ||
| all_errors.any? ? all_errors : nil | ||
| end |
| has_more: has_more?(state, display_count), | ||
| total_hits: total_hits | ||
| } | ||
| } |
| ].slice(offset, per_page) || [] | ||
| { results: results, hits: 4, errors: nil, show_continuation: false } | ||
| end | ||
| timdex_fetcher = lambda do |offset:, per_page:, query: nil| |
Member
Author
There was a problem hiding this comment.
Making this change fails the tests. It is incorrect.
| assert_equal 2, timdex_calls.length | ||
| test 'fetch stops when duplicate source chunks do not grow the ordered result set' do | ||
| calls = [] | ||
| fetcher = lambda do |offset:, per_page:, query: nil| |
There was a problem hiding this comment.
Found 2 issues:
1. Unused block argument - per_page. If it's necessary, use _ or _per_page as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]
2. Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]
Member
Author
There was a problem hiding this comment.
Making this change fails the tests. It is incorrect.
| called << [:timdex, offset, per_page] | ||
| test 'fetch defaults to 50 results per source when env var is not set' do | ||
| per_page_seen = [] | ||
| fetcher = lambda do |offset:, per_page:, query: nil| |
There was a problem hiding this comment.
Found 2 issues:
1. Unused block argument - offset. If it's necessary, use _ or _offset as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]
2. Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]
Member
Author
There was a problem hiding this comment.
Making this change fails the tests. It is incorrect.
| primo_all = (1..3).map { |i| { title: "P#{i}", score: 1.0 - (i * 0.01), api: 'primo', identifier: "p#{i}" } } | ||
| timdex_all = (1..5).map { |i| { title: "T#{i}", score: 1.0 - (i * 0.01), api: 'timdex', identifier: "t#{i}" } } | ||
|
|
||
| primo_fetcher = lambda do |offset:, per_page:, query: nil| |
Member
Author
There was a problem hiding this comment.
Making this change fails the tests. It is incorrect.
| { results: primo_all.slice(offset, per_page) || [], hits: primo_all.length, errors: nil, | ||
| show_continuation: false } | ||
| end | ||
| timdex_fetcher = lambda do |offset:, per_page:, query: nil| |
Member
Author
There was a problem hiding this comment.
Making this change fails the tests. It is incorrect.
There was a problem hiding this comment.
Pull request overview
This PR replaces merged “all tab” pagination with a reranked, cached candidate pool (via the reranker gem) and updates the UI to use a Turbo Stream “Load more results” interaction across results pages.
Changes:
- Introduces
MergedSearchServicestate caching + reranking flow for the “all” tab, with new ENV controls (ALL_TAB_RESULTS_PER_SOURCE,ALL_TAB_SCORER,LOAD_MORE_MAX_RESULTS). - Switches results rendering from pagination to a shared “load more” control and Turbo Stream append/replace responses.
- Removes
MergedSearchPaginatorand updates/expands tests to reflect the new behavior.
Reviewed changes
Copilot reviewed 19 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
app/models/merged_search_service.rb |
Reworks all-tab orchestration to cache candidates, rerank with reranker, and serve stable “load more” slices. |
app/controllers/search_controller.rb |
Adds Turbo Stream responses and load-more parameter handling; adapts all-tab to display-count based fetching. |
app/views/search/results.html.erb |
Renders results via shared item partial + load-more control instead of pagination. |
app/views/search/results.turbo_stream.erb |
Turbo Stream append/replace template for load-more updates. |
app/views/search/results_geo.html.erb |
Updates geodata results to use load-more UI and results-list target. |
app/views/search/results_geo.turbo_stream.erb |
Turbo Stream append/replace template for geodata load-more updates. |
app/views/search/_load_more.html.erb |
New shared “Load more results” control for all tabs. |
app/views/search/_result_item.html.erb |
New shared wrapper partial to render either Primo or TIMDEX result markup. |
app/javascript/loading_spinner.js |
Extends spinner behavior to cover Turbo Stream load-more interactions. |
app/assets/stylesheets/partials/_results.scss |
Adjusts list styling to support numbered result lists. |
app/assets/stylesheets/partials/_pagination.scss |
Adds styling for the new .load-more control. |
Gemfile |
Adds reranker gem from GitHub tag. |
Gemfile.lock |
Locks reranker and enumerable-stats. |
README.md |
Documents new ENV variables for all-tab reranking and load-more behavior. |
AGENTS.md |
Updates internal agent guidance to reflect reranking + load-more approach. |
.env.test |
Adds test ENV defaults for reranking and load-more. |
test/models/merged_search_service_test.rb |
Replaces paginator-oriented tests with reranking/state-cache/load-more tests. |
test/models/merged_search_paginator_test.rb |
Removes tests for deleted paginator. |
test/controllers/search_controller_test.rb |
Updates assertions for load-more UI and controller delegation changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.
Suppressed comments (7)
app/models/merged_search_service.rb:93
ensure_ordered_resultscan get stuck retrying the same offsets when a source returns duplicate records: offsets are derived fromstate[:*_results].length, butdedupe_recordscan keep that length from increasing, so the next loop iteration (and later load-more requests) will re-request the same offset again. This also makeshas_more?stay true even when no new ordered keys can be produced.
def ensure_ordered_results(state, display_count:, stable_count:, per_source:)
state = rerank_state(state, stable_count: stable_count) if state[:ordered_keys].empty? && any_results?(state)
while state[:ordered_keys].length < display_count && sources_available?(state)
previous_count = state[:ordered_keys].length
state = fetch_next_source_chunks(state, per_source: per_source)
state = rerank_state(state, stable_count: stable_count)
break if state[:ordered_keys].length == previous_count
end
app/models/merged_search_service.rb:182
parallel_fetchjoins threads but does not surface exceptions raised inside fetcher threads. If a fetcher raises, the exception will be swallowed and the service will treat the source as skipped (nil), masking real failures and potentially caching incomplete state.
# Fetch from Primo and TIMDEX concurrently.
#
# WARNING: exceptions raised inside threads do not automatically propagate;
# callers should account for this.
#
# @param primo_offset [Integer, nil] next Primo offset, or nil to skip Primo
# @param timdex_offset [Integer, nil] next TIMDEX offset, or nil to skip TIMDEX
# @param per_page [Integer] number of results to request per source
# @return [Array<Hash, Hash>] [primo_response, timdex_response]
def parallel_fetch(primo_offset:, timdex_offset:, per_page:)
primo = nil
timdex = nil
threads = []
if primo_offset && primo_offset < Analyzer::PRIMO_MAX_OFFSET
threads << Thread.new do
primo = @primo_fetcher.call(offset: primo_offset, per_page: per_page, query: @enhanced_query)
end
end
if timdex_offset
threads << Thread.new do
timdex = @timdex_fetcher.call(offset: timdex_offset, per_page: per_page, query: @enhanced_query)
end
end
threads.each(&:join)
[primo, timdex]
app/models/merged_search_service.rb:211
configured_scorerdefaults tozipper, but the fallback for an unrecognizedALL_TAB_SCORERvalue currently returnsZscoreScorer. That makes misconfiguration silently change behavior instead of falling back to the documented default.
case ENV.fetch('ALL_TAB_SCORER', 'zipper').downcase
when 'zscore' then Reranker::ZscoreScorer.new
when 'zipper' then Reranker::ZipperMergeScorer.new
when 'simple' then Reranker::SimpleScorer.new
when 'random' then Reranker::RandomScorer.new
else Reranker::ZscoreScorer.new
end
app/models/merged_search_service.rb:221
state_cache_keyalways usesENV['ALL_TAB_RESULTS_PER_SOURCE']forper_source, butfetchallows overridingper_source:via an argument. If callers use that argument (tests already do), different per-source fetch sizes will share the same cache key and can reuse/overwrite incompatible cached state.
def state_cache_key
query = @enhanced_query.except(:page).merge(
tab: @active_tab,
scorer: ENV.fetch('ALL_TAB_SCORER', 'zipper'),
boost_sources: ENV.fetch('ALL_TAB_BOOST_SOURCES', ''),
per_source: ENV.fetch('ALL_TAB_RESULTS_PER_SOURCE', '50')
)
"#{CacheKeyGenerator.call(query)}/all-tab-load-more"
app/views/search/_load_more.html.erb:12
- For non-
alltabs, the load-more link advancespagein the URL and relies on Turbo Stream appends to build a growing list. A refresh or shared URL like...?tab=primo&page=2will render only that page’s results (not pages 1+2), but the UI will still show an ordered list starting at 1 and a "Load more" link to page 3—effectively skipping the first page.
<% if @active_tab == 'all' %>
<% params_copy.delete(:page) %>
<% params_copy[:load_count] = @load_more[:next_count] %>
<% else %>
<% params_copy[:page] = @load_more[:next_page] %>
<% end %>
app/controllers/search_controller.rb:50
- Turbo Stream responses were added for both standard and Geo results, but controller tests only assert the presence of the load-more container in HTML. Consider adding at least one test that requests
format: :turbo_streamand asserts the response includes the expectedturbo-streamactions (append toresults-listand replaceload-more).
# Render the response in HTML or JSON format
respond_to do |format|
format.json { render json: { results: @results, pagination: @pagination, errors: @errors } }
format.turbo_stream { render :results }
format.html { render :results }
end
app/javascript/loading_spinner.js:59
- The new load-more handlers use optional chaining on
#search-results, but the same file still has unguardeddocument.getElementById('search-results').classList...calls in theturbo:frame-renderhandler and the tab-click handler. If#search-resultsis not present (e.g., Geo results view), those paths can still throw at runtime; consider making all spinner toggles consistently null-safe.
document.addEventListener('turbo:before-stream-render', function(event) {
if (window.pendingFocusAction === 'load-more') {
document.getElementById('search-results')?.classList.remove('spinner');
window.pendingFocusAction = null;
}
});
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces our Reranker gem, but keeps the Zipper merge default. This sets us up for future algorithms, but allows us to separate this structural change from algorithm selection.
It also reworks pagination to be load more that relies heavily on a new redis cache approach. This is quite a bit simpler than our initial approach because we are dropping support for jumping to arbitrary pages.
It also introduces a first pass at reintroducing result numbers. A separate ticket has been created and assigned to Dave to adjust as needed and ensure it works well in mobile sized viewports.
It tried to break this into separate PRs, but failed.
Developer
Accessibility
New ENV
Approval beyond code review
Additional context needed to review
E.g., if the PR includes updated dependencies and/or data
migration, or how to confirm the feature is working.
Code Reviewer
Code
added technical debt.
Documentation
(not just this pull request message).
Testing