Repository navigation
Conversation
AmirHossein-84
added a commit
to AmirHossein-84/DigUp
that referenced
this pull request
Oct 18, 2026
… localization - SQLite & Search Layer (PRs ARahim3#16, ARahim3#38, ARahim3#82, ARahim3#85, ARahim3#94, ARahim3#111): * StatementCache with sqlite3_reset statement recycling and reentrant transactions * PRAGMA mmap_size = 1GB, synchronous = NORMAL, cache_size = -64000 * Zero-copy row.copyBlob direct vector loading into searcher matrix halves * Unicode NFC normalization for cross-platform and CJK/accented file paths * Chunked IN queries with UNION ALL to eliminate N+1 roundtrips * CJK Han character detection, Han text budgeting, and Han-aware snippet generation * SIMD8<Float> acceleration in Math+Portable.swift without heap allocations * Float16Shim for Intel Mac x86_64 IEEE 754 binary16 compatibility - Crawler, Rules, Path Normalization & Extractors (PRs ARahim3#33, ARahim3#42, ARahim3#56, ARahim3#60, ARahim3#79, ARahim3#80, ARahim3#86, ARahim3#88, ARahim3#91, ARahim3#101, ARahim3#107, ARahim3#109, Issues ARahim3#67, ARahim3#97, ARahim3#105, ARahim3#127): * Fix Windows path slash/drive normalization in DirectoryWalk preventing 0-file crawl * Fix case-insensitive folder matching on Windows in FolderSelection * Fix Crawler to never treat user-selected search roots (e.g. ~) as project roots * Exclude secrets/keys (.env*, .pem, .kdbx, SSH keys, passwd) in Rules * Honor standard CACHEDIR.TAG 43-byte signature when crawling * Skip build/intermediate directories (.venv, target, .build, .next, dist, bin, obj) * Drop FileProvider download status lookups for 10-50x faster cloud folder crawls * OfficeText extractor for .xlsx, .pptx, .ods, .odp XML documents * Headless LibreOffice .doc extractor on Linux and local Notion SQLite workspace extractor * Safe typed CGPDFDictionaryGetString for PDF metadata preventing crashes * Preserve OCR keywords on image-only PDFs * Outlook PST reader utility crate in pst/ - Indexing Pipeline & Runtime (PRs ARahim3#49, ARahim3#53, ARahim3#62, ARahim3#64, ARahim3#71, ARahim3#72, ARahim3#114, Issue ARahim3#3): * Batched document text embeddings (up to 16 docs per pass) * Cumulative IndexProgress tracking across multi-pass indexing * DocumentCache for extracted chunks to prevent redundant file disk reads * Linux URLSession SIGABRT fix in FileFetcher * ImageLoader parallel prefetching and batched image embeddings * Apple Neural Engine CoreML video frame encoding support * Scoped incremental subtree sync when changed paths are known * Autoreleasepool wrapping in background worker extraction to eliminate memory leaks - Daemon, Lifecycle & Policies (PRs ARahim3#24, ARahim3#27, ARahim3#31, ARahim3#36, ARahim3#51, ARahim3#82, Issues ARahim3#22, ARahim3#68): * HelperRestartPolicy for clean stop/resume transitions * Backlog starvation recovery and --resume-pending in IndexService * Configurable indexing hours window policy * Opt-in Low Power Mode indexing to protect laptop thermals and battery * Win32 OpenProcess(SYNCHRONIZE) parent monitor to prevent zombie helpers on Windows * Correct 0-hit search behavior when all search roots are removed - UI, Cross-Platform Shell, Localization & Build (PRs ARahim3#15, ARahim3#20, ARahim3#37, ARahim3#39, ARahim3#58, ARahim3#65, ARahim3#74, ARahim3#81, ARahim3#84, ARahim3#112): * DigUpLocalization framework with en, zh-Hans, zh-Hant string catalogs * AppSearch engine and launcher for macOS App and Tauri desktop shells * KeyCommands and QuickLookController shared handling * Preserved search field text during IME composition * Search panel fade transitions and @observable code editor selection * Hardened build scripts with pinned SHA-256 vendor checksum verification
Owner
|
Thanks for the work. As on #59, the beta's packages replace the tarball and I'm not adding a LibreOffice dependency for .doc, so I'm closing the PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Current
mainhas portable engine and CLI source that builds on Linux, but no checked Linux CLI archive or Linux CI path. Legacy.docfiles are classified for indexing and then fail because the portable reader does not handle them. The README still describes Linux as forthcoming.What changed
.doc, using an isolated temporary profile and stripping the UTF-8 BOM before chunking. The built-in.docx,.odt, and.rtfreaders remain in place.The macOS menu bar app and its readers are unchanged. Linux uses the current local EmbeddingGemma 2, SQLite, and hybrid search engine.
Relationship to earlier Linux work
Related to follow-up issue #59, original proposal #5, and the first Linux PR #6. This branch starts directly from current
ARahim3/DigUp:main; it does not depend on #6 merging. PR #6 targets an older architecture and currently conflicts withmain. It includes presentation-deck and package-format work that this PR does not claim to replace.Verification
7df5685: release build, 88 tests in 11 suites, checksum, CLI smoke check, and artifact upload..doctest passed. A 41 MB archive passed an isolated containerdigup --helpcheck with PDFium and llama.cpp loading from beside the binary.bash -nandgit diff --checkpassed.The unfiltered
swift testran 90 tests and reported five failures in existingImageTests: HEIC decoding and TIFF/WebP orientation with Ubuntu 24.04's FFmpeg 6.1. Those fixtures need FFmpeg 8.1. CI explicitly excludes that suite for now. Model-backed reference vectors and macOS tests were not run for this change.Remaining limits
This is a CLI archive, not a Linux desktop app or published release. It targets the Ubuntu 24.04 ABI and needs system glibc, C++/OpenMP libraries and FFmpeg for media. There is no
.deb,.rpm, aarch64 archive, presentation-deck or RAW reader, or OCR for exact words in scanned pages/screenshots. LibreOffice Writer is optional and needed only for legacy.docfiles. Issue #59 tracks the remaining Linux work.