Repository navigation
fix: progress bar, console output, and interactive mode improvements - #2
Merged
Merged
Conversation
- transferCollection now fetches documents page-by-page (page size = batchSize) using startAfter pagination instead of loading the entire collection at once with query.get(). This distributes network I/O evenly across the transfer, making the progress bar advance linearly instead of stalling during a long initial fetch then jumping. - Fix counting bug: countSubcollectionsForDoc now respects maxDepth to match transfer behavior. Previously, the count included docs beyond maxDepth that would never be transferred, causing the progress bar total to be too high.
- Separate each top-level collection with clear formatting - Show unique subcollection names found per collection (e.g. orders, reviews, settings) so user can verify what's being scanned - Show excluded subcollection names so user can verify --exclude works - Show subcollection document counts per collection - Show live "Scanning <collection>..." indicator during counting - Guard all process.stdout.write calls against JSON mode to prevent garbled output when using --json - Add onSubcollectionExcluded callback to CountProgress interface
Interactive mode now includes all options available in CLI/config modes: - After basic prompts (source, dest, collections, subcollections, dry run, merge), offers "Configure additional options?" with a checkbox of all available options grouped logically - Each selected option gets its appropriate prompt type (input, number, confirm, etc.) - Available options: exclude patterns, where filters, parallel, batch size, document limit, max depth, rate limit, clear destination, delete missing, transform file, rename collections, ID prefix/suffix, webhook, skip oversized, detect conflicts, verify counts, verify integrity, retries At the end of the interactive session, user can choose to: - Execute transfer (existing behavior) - Save as INI config file - Save as JSON config file Config serialization generates proper INI (with sections and commented defaults for unset options) or JSON format. The saved file can then be used with `fscopy -f <file>`. Also updates cli.ts to handle the new InteractiveResult return type and exit gracefully when saving instead of executing.
There was a problem hiding this comment.
Pull request overview
This PR improves the Firestore transfer CLI’s user experience and correctness by making transfer progress more linear (via paginated reads), aligning document counting with transfer behavior (notably maxDepth), and expanding interactive mode to configure and persist more options.
Changes:
- Implement paginated document fetching in
transferCollectionto avoid loading entire collections at once and to make progress advance steadily. - Fix counting logic to respect
maxDepth, and enhance counting console output (including excluded subcollections) while guarding live output for non-interactive modes. - Expand interactive mode to cover advanced options and allow “execute vs save config” at the end, with CLI support for the new return type.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/transfer/transfer.ts | Switches to page-by-page query pagination for transfer reads and moves limit handling into pagination. |
| src/transfer/count.ts | Makes subcollection counting respect maxDepth and adds a callback for excluded subcollections. |
| src/orchestrator.ts | Improves counting/deletion progress output and adds helpers for live progress rendering. |
| src/interactive.ts | Implements full interactive flow including advanced options and config save (INI/JSON) paths. |
| src/cli.ts | Adapts CLI to the new interactive return type and exits early on “save”. |
| src/tests/interactive.test.ts | Updates interactive tests for new prompts and return shape. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Gate progressBar.start() with canWriteProgress check - Use process.stdout.columns + isTTY in clearLine - Extend INI/JSON parsers and mergeConfig for advanced fields - Include advanced fields in INI serialization - Use explicit confirm queue in interactive tests - Add pagination boundary tests for transfer - Add maxDepth + onSubcollectionExcluded tests for count
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…rops 0 values - Remove yargs defaults for retries, verify, rateLimit, skipOversized, detectConflicts, maxDepth, verifyIntegrity so file config values are actually used when CLI args are not explicitly passed - Replace Number.parseInt(...) || undefined with parseIntOption() helper to preserve valid 0 values in INI config parsing
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
transferCollectionnow fetches documents page-by-page (batchSizeper page) usingstartAfterpagination instead of loading the entire collection withquery.get(). This distributes network I/O evenly, making the progress bar advance linearly instead of stalling during a long initial fetch then jumping.maxDepth—countSubcollectionsForDocnow respectsmaxDepthto match transfer behavior. Previously the total was too high when using--max-depth, causing the progress bar to never reach 100% naturally.process.stdout.writecalls are guarded against JSON mode.fscopy -f.??fallback works correctly. Fixed INI parsing of0values being dropped byparseInt(...) || undefined.