Skip to content

git: fail --git indexing when a batch is not stored - #120

Open
vladwing wants to merge 1 commit into
facebookexperimental:mainfrom
vladwing:git-range-failed-batches
Open

vladwing wants to merge 1 commit into
facebookexperimental:mainfrom
vladwing:git-range-failed-batches

Conversation

@vladwing

Copy link
Copy Markdown
Contributor

A --git or tree run that fails to store a batch of files logs the failure, prints "Indexing completed successfully!" and exits 0. In two of three runs of the v7.3..v7.4-rc1 merge window, an inserter hit lance's "Too many concurrent writers" retry limit on the types and functions tables, and the database came out 512 processed_files rows short with exit status 0. insert_batch_and_mark() already leaves a failed batch's files unmarked, but only a later run over a range containing those files reads them again, and nobody reruns a range that reported success.

Such a run now exits non-zero and says how many files it left unindexed:

  • insert_batch_and_mark() returns the run's stored-batch count when every fact table and the mark were stored, and None otherwise.
  • The range and tree inserters count failed batches and their files, and GitTupleStats carries both counts. An inserter task that panics counts as one failed batch, with the files of the batch it was storing. The periodic optimization check moves from insert_batch_and_mark() into the inserters, after the batch is marked or failed, so a panic there does not count marked files.
  • process_git_range() and process_git_tree() finish as before (summary, optimization, call_edges index), then return an error such as "2 batches (512 files) were not stored; indexing again reads them". Their summaries say "Incomplete" and give the same count. process_git_tree() returns the error before forget_older_processed_files(), record_index_build() and record_tree_sha(), because those record that the whole tree was read.

The change covers the file batches only. A failed git_commits insert in process_commits_pipeline() is still only logged.

semcode-index calls both functions with ?, so a failed run no longer prints the success line. Branch mode stops without recording the branch as indexed, and the branches after it wait for the next run. Running the same command again reads only the unmarked files. The other callers already handle an error:

  • semcode --reindex-if-stale now exits with the error and leaves the index marked stale. Before, record_index_build() ran anyway, the stale flag cleared, and semcode answered from an index with holes.
  • semcode's auto-indexing of the current commit no longer prints "Indexing complete"; its warning carrying the error shows with SEMCODE_DEBUG=warn.
  • semcode-mcp reports a failed index instead of a completed one.

insert_batch_and_mark_reports_a_failed_mark checks the return value with processed_files dropped, and the existing per-table test now checks it with each fact table dropped. a_run_with_an_unstored_batch_fails checks that tree and range runs over a one-file repository fail, record no tree SHA while the types table is missing, and succeed once it exists. unstored_batches_error_counts_batches_and_files checks the message.

A `--git` or tree run that fails to store a batch of files logs the
failure, prints "Indexing completed successfully!" and exits 0. In two
of three runs of the v7.3..v7.4-rc1 merge window, an inserter hit
lance's "Too many concurrent writers" retry limit on the `types` and
`functions` tables, and the database came out 512 `processed_files` rows
short with exit status 0. `insert_batch_and_mark()` already leaves a
failed batch's files unmarked, but only a later run over a range
containing those files reads them again, and nobody reruns a range that
reported success.

Such a run now exits non-zero and says how many files it left unindexed:

- `insert_batch_and_mark()` returns the run's stored-batch count when
  every fact table and the mark were stored, and `None` otherwise.
- The range and tree inserters count failed batches and their files, and
  `GitTupleStats` carries both counts. An inserter task that panics
  counts as one failed batch, with the files of the batch it was
  storing. The periodic optimization check moves from
  `insert_batch_and_mark()` into the inserters, after the batch is
  marked or failed, so a panic there does not count marked files.
- `process_git_range()` and `process_git_tree()` finish as before
  (summary, optimization, `call_edges` index), then return an error such
  as "2 batches (512 files) were not stored; indexing again reads them".
  Their summaries say "Incomplete" and give the same count.
  `process_git_tree()` returns the error before
  `forget_older_processed_files()`, `record_index_build()` and
  `record_tree_sha()`, because those record that the whole tree was
  read.

The change covers the file batches only. A failed `git_commits` insert
in `process_commits_pipeline()` is still only logged.

`semcode-index` calls both functions with `?`, so a failed run no
longer prints the success line. Branch mode stops without recording the
branch as indexed, and the branches after it wait for the next run.
Running the same command again reads only the unmarked files. The other
callers already handle an error:

- `semcode --reindex-if-stale` now exits with the error and leaves the
  index marked stale. Before, `record_index_build()` ran anyway, the
  stale flag cleared, and semcode answered from an index with holes.
- semcode's auto-indexing of the current commit no longer prints
  "Indexing complete"; its warning carrying the error shows with
  `SEMCODE_DEBUG=warn`.
- semcode-mcp reports a failed index instead of a completed one.

`insert_batch_and_mark_reports_a_failed_mark` checks the return value
with `processed_files` dropped, and the existing per-table test now
checks it with each fact table dropped.
`a_run_with_an_unstored_batch_fails` checks that tree and range runs
over a one-file repository fail, record no tree SHA while the `types`
table is missing, and succeed once it exists.
`unstored_batches_error_counts_batches_and_files` checks the message.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant