Repository navigation
Conversation
A `--git` or tree run that fails to store a batch of files logs the failure, prints "Indexing completed successfully!" and exits 0. In two of three runs of the v7.3..v7.4-rc1 merge window, an inserter hit lance's "Too many concurrent writers" retry limit on the `types` and `functions` tables, and the database came out 512 `processed_files` rows short with exit status 0. `insert_batch_and_mark()` already leaves a failed batch's files unmarked, but only a later run over a range containing those files reads them again, and nobody reruns a range that reported success. Such a run now exits non-zero and says how many files it left unindexed: - `insert_batch_and_mark()` returns the run's stored-batch count when every fact table and the mark were stored, and `None` otherwise. - The range and tree inserters count failed batches and their files, and `GitTupleStats` carries both counts. An inserter task that panics counts as one failed batch, with the files of the batch it was storing. The periodic optimization check moves from `insert_batch_and_mark()` into the inserters, after the batch is marked or failed, so a panic there does not count marked files. - `process_git_range()` and `process_git_tree()` finish as before (summary, optimization, `call_edges` index), then return an error such as "2 batches (512 files) were not stored; indexing again reads them". Their summaries say "Incomplete" and give the same count. `process_git_tree()` returns the error before `forget_older_processed_files()`, `record_index_build()` and `record_tree_sha()`, because those record that the whole tree was read. The change covers the file batches only. A failed `git_commits` insert in `process_commits_pipeline()` is still only logged. `semcode-index` calls both functions with `?`, so a failed run no longer prints the success line. Branch mode stops without recording the branch as indexed, and the branches after it wait for the next run. Running the same command again reads only the unmarked files. The other callers already handle an error: - `semcode --reindex-if-stale` now exits with the error and leaves the index marked stale. Before, `record_index_build()` ran anyway, the stale flag cleared, and semcode answered from an index with holes. - semcode's auto-indexing of the current commit no longer prints "Indexing complete"; its warning carrying the error shows with `SEMCODE_DEBUG=warn`. - semcode-mcp reports a failed index instead of a completed one. `insert_batch_and_mark_reports_a_failed_mark` checks the return value with `processed_files` dropped, and the existing per-table test now checks it with each fact table dropped. `a_run_with_an_unstored_batch_fails` checks that tree and range runs over a one-file repository fail, record no tree SHA while the `types` table is missing, and succeed once it exists. `unstored_batches_error_counts_batches_and_files` checks the message.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
--gitor tree run that fails to store a batch of files logs the failure, prints "Indexing completed successfully!" and exits 0. In two of three runs of the v7.3..v7.4-rc1 merge window, an inserter hit lance's "Too many concurrent writers" retry limit on thetypesandfunctionstables, and the database came out 512processed_filesrows short with exit status 0.insert_batch_and_mark()already leaves a failed batch's files unmarked, but only a later run over a range containing those files reads them again, and nobody reruns a range that reported success.Such a run now exits non-zero and says how many files it left unindexed:
insert_batch_and_mark()returns the run's stored-batch count when every fact table and the mark were stored, andNoneotherwise.GitTupleStatscarries both counts. An inserter task that panics counts as one failed batch, with the files of the batch it was storing. The periodic optimization check moves frominsert_batch_and_mark()into the inserters, after the batch is marked or failed, so a panic there does not count marked files.process_git_range()andprocess_git_tree()finish as before (summary, optimization,call_edgesindex), then return an error such as "2 batches (512 files) were not stored; indexing again reads them". Their summaries say "Incomplete" and give the same count.process_git_tree()returns the error beforeforget_older_processed_files(),record_index_build()andrecord_tree_sha(), because those record that the whole tree was read.The change covers the file batches only. A failed
git_commitsinsert inprocess_commits_pipeline()is still only logged.semcode-indexcalls both functions with?, so a failed run no longer prints the success line. Branch mode stops without recording the branch as indexed, and the branches after it wait for the next run. Running the same command again reads only the unmarked files. The other callers already handle an error:semcode --reindex-if-stalenow exits with the error and leaves the index marked stale. Before,record_index_build()ran anyway, the stale flag cleared, and semcode answered from an index with holes.SEMCODE_DEBUG=warn.insert_batch_and_mark_reports_a_failed_markchecks the return value withprocessed_filesdropped, and the existing per-table test now checks it with each fact table dropped.a_run_with_an_unstored_batch_failschecks that tree and range runs over a one-file repository fail, record no tree SHA while thetypestable is missing, and succeed once it exists.unstored_batches_error_counts_batches_and_fileschecks the message.