From 864968c9b0e5c4c20202e112bf6dce8bc6c55bdf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Orhan=20AYDO=C4=9EDU?= Date: Tue, 30 Dec 2025 05:45:31 +0300 Subject: [PATCH 1/3] Update .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index f041e78..1c00535 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,4 @@ CLAUDE.md # Config files (contain secrets) .nonedb !.nonedb.example +/db From 4b9b952318ae250cf7b54d8555c9940fd378027d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Orhan=20AYDO=C4=9EDU?= Date: Sat, 24 Oct 2026 01:37:23 +0300 Subject: [PATCH 2/3] v3.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Concurrency and durability Database-level locking. Every operation holds a lock on its database for its whole duration (shared for reads, exclusive for writes), across processes and instances. Concurrent inserts, updates and deletes no longer lose records or reuse keys (K-01). Caches can't go stale. A generation counter in the lock file's state record invalidates every cached index, meta and field/spatial index of a database as soon as another process writes it (K-07, Y-11, Y-13). Atomic file writes. Index, meta and other JSON files are written to a temp file and renamed; a crash can no longer leave a truncated index (Y-14). Unencodable data (invalid UTF-8, NAN, INF) is rejected before anything is written (K-03). Self-healing indexes. A missing, corrupt or out-of-date key index is rebuilt from the data file; deletions append a tombstone line so deleted records stay deleted after a rebuild. A torn last line (crash during an append) is cut off. Records are verified against their key when read. Lock file handles are opened close-on-exec: programs started by the process don't inherit them (an inherited copy would keep a lock held after the process died in the middle of a write). Write journal. Field indexes, spatial indexes and shard counters are rebuilt automatically if a writer dies half-way through an operation (the journal entry lives in the lock file's state record). Idempotent buffer flush. A crash during a buffer flush can no longer lose or duplicate records (K-08). Tested with real SIGKILL crashes and multi-process stress tests (tests/Regression). Performance No per-operation checks for old versions. The lock file is read on every lock acquisition anyway; its state record says whether the database was already checked for pre-3.2 leftovers, whether the write buffer holds records and whether the index journal has entries. Normal reads and writes therefore don't stat buffer files, glob for old files or re-derive the index list. Pre-3.0 data files are converted when their (missing) index is first needed. Lock file handles stay open between operations (one per database, at most 64; never shared with a forked child, never trusted after the file was replaced), and the change counter is written once per write session with a single fixed-size write. Loaded key indexes are trusted until another process writes the database (no stat per read). Query builder: where()/orWhere() filters are compiled once per query (operators validated, $like/$regex compiled), sort keys are computed once per record, and reading records in file order no longer sorts the offset map when it is already in order. new noneDB() no longer resolves the script directory unless a relative dbDir needs it, and a dbDir given in the config array skips resolving the file's one. Query sorting uses array_multisort on precomputed sort keys (same order as before, stable on every PHP version). query()->nearest(k) combined with other filters reads the nearest candidates in growing batches instead of putting every record in distance order (same results; it used to sort the whole database since k is applied after the other filters). Security secretKey is always honoured: the persistent hash cache is per key (.nonedb_hash_cache_, mode 0600) and can no longer let a wrong key read data or "poison" the right key (S-01, S-02, S-03, S-14). The old .nonedb_hash_cache file is ignored (migrateKey() removes it). The key is never shown by print_r/var_dump/var_export, and instances can't be serialized (S-09). New database directories get an .htaccess (Require all denied) and an empty index.html (S-06). noneDB::migrateKey($dir, $old, $new) re-keys existing databases (also those from before 3.2, without leaving anything under the old key). A secretKey given in the config array is always validated as given: null is rejected also when there is no .nonedb file (it was reported as a missing config file) and in dev mode (it was silently replaced by the predictable dev mode key). Config values are validated (shardSize, lockTimeout, lockRetryDelay, autoCompactThreshold). Storage engine Pre-3.0 files are detected safely: a record whose first field is data is no longer mistaken for the old format and destroyed (K-02). Sharding: keys come from meta.nextKey and the shard size stored in each database's meta, so reopening with a different shardSize is safe (Y-10); migration never reuses keys (K-06); key lookups on sharded databases see new records (Y-09); empty shards are removed by compact(). Auto-compaction triggers when deleted/superseded records exceed the threshold relative to the file (not to the key counter) (O-22). Key filters accept integers and digit strings only ('abc', null, true, 1.9 match nothing) and are combined (AND) with the other filters (Y-15, Y-16). null filters behave the same in find, update and delete (Y-17). Malformed update() input is rejected (Y-18). New dropDB($name). Field and spatial indexes Indexed fields are recorded in {db}.nonedb.indexes; field names may contain any characters (e-mail, şehir, first.name) (Y-08). New shards, null values and type changes are indexed correctly (Y-12). Index files written by older versions are rebuilt on first use. Batch operations write each index file once (P-02). Spatial: indexes work on sharded databases (Y-02), cover new shards (Y-05) and survive migration (Y-04); buffered inserts are indexed (Y-01); the R-tree can no longer loop forever (Y-03). nearest() no longer cuts through a rounding tie at its search radius: a record just outside the radius whose distance rounds to the same value (and has the smaller key) was skipped although it sorts first. Spatial queries: correct results near the antimeridian and the poles (Y-06); nearest() with maxDistance returns the matches it finds (Y-07); withinDistance() measures to the nearest point of a geometry (a point inside a polygon is at distance 0); withinPolygon() accepts MultiPolygon and checks edges, not only vertices; queries without hits return immediately (O-24, O-34). GeoJSON validation rejects NaN/INF, strings as coordinates, self-intersecting rings and holes outside their shell; ring closure is compared numerically. Query builder where() calls are combined with AND (they used to overwrite the same field) (Y-21). orWhere() without where() returns only the records matching an OR group (it returned everything) (Y-20). Operators compare by type: numbers (and numeric strings) numerically, strings lexically; null, booleans and arrays never satisfy $gt/$gte/$lt/$lte/between (O-10). Unknown operators and invalid operands ($in without an array, a broken $regex) throw, whether or not there are records (O-11). like(), $like and $regex are safe: like patterns are literal except a leading ^ / trailing $; invalid regular expressions throw InvalidArgumentException (Y-19). sort() is a stable total order (missing/null first, then booleans, numbers, strings, arrays); several sort() calls sort by several fields (Y-22, O-12). count(), withDistance(), nearest()+where(), groupBy() key types, search() (Unicode, ignores key) fixed (O-01, O-04..O-08). withDistance() measures like withinDistance()/nearest() (0 inside a polygon, else to the nearest point of the geometry); it used the centroid, so a polygon could pass a 1 km filter and show a larger _distance. The direct $db->between() compares like query()->between() (it used PHP's loose comparison, so e.g. true was "between" 100 and 500). removeFields() removes fields from the stored records only; select/except/join/withDistance output never reaches storage (K-04, Y-26). update(), delete() and removeFields() run selection and write under one lock; the reads of one query (get(), count(), aggregates) run under one shared lock, so they see one state of the database. Other getDBs() lists each database once, includes sharded databases with their total size (O-03). limit([]) and sort([]) return [] (D-05). Typo "paramters" fixed (D-06). Removed dead code (legacy .idx key index, unused helpers); the query builder no longer uses Reflection. Examples use examples/_bootstrap.php, run from any directory and can be run repeatedly. Distance and centroid memo caches keep working when full (the older half is dropped) (D-08). Docs: join() example corrected (the record is attached under the foreign database's name), and the docs were checked against the code (return shapes, error messages, whereNot()/notBetween() semantics, spatial distance options). Benchmark scripts (tests/*_benchmark.php, tests/sleekdb_comparison.php) use their own temporary directory and key; performance_benchmark.php and sleekdb_comparison.php used to delete every file in the project's db/ directory. Sizes can be passed on the command line (php tests/performance_benchmark.php 100,1000). New tests/operation_benchmark.php measures single-operation latency and can compare two versions; docs/BENCHMARKS.md was re-measured (3.2 vs 3.1 on the same machine). Test Results PHP 8.5 compatible. Tested on PHP 7.4, 8.2, 8.4 and 8.5 (Linux and macOS); CI runs 7.4 to 8.5. PHPUnit now fails on deprecations, so a new PHP version's deprecations show up in CI. The tests no longer call Reflection*::setAccessible() on PHP 8.1+ (deprecated in 8.5), tests that kill child processes don't go through a shell (with dash as /bin/sh the shell was killed and the child kept running), and the dev mode test runs in its own process (its constant leaked into later tests). 1084 tests (about 19,000 assertions), new suite tests/Regression (Regression): one or more tests per review finding, multi-process tests, real SIGKILL crash tests, differential fuzzing against an independent model, lock-state tests. Expectations of existing tests changed only where a documented behaviour change requires it (database name validation, secretKey validation, limit([])/sort([]), insert([]), orWhere() and repeated where() semantics). --- .github/workflows/tests.yml | 22 +- .gitignore | 3 + CHANGES.md | 194 + README.md | 65 +- docs/API.md | 170 +- docs/BENCHMARKS.md | 459 +- docs/CONFIGURATION.md | 130 +- docs/QUERY.md | 147 +- docs/SPATIAL.md | 59 +- docs/index.html | 4 +- examples/_bootstrap.php | 28 + examples/aggregation.php | 5 +- examples/chaining.php | 5 +- examples/database-management.php | 40 +- examples/delete.php | 12 +- examples/find.php | 5 +- examples/getdb-info.php | 5 +- examples/insert.php | 5 +- examples/multiple-insert.php | 5 +- examples/performance.php | 31 +- examples/query-methods.php | 5 +- examples/sharding.php | 41 +- examples/update.php | 7 +- examples/utility-methods.php | 5 +- noneDB.php | 10933 +++++++--------- phpunit.xml | 6 +- tests/Feature/CheckDBTest.php | 39 +- tests/Feature/ConfigurationTest.php | 92 +- tests/Feature/CreateDBTest.php | 14 +- tests/Feature/FindTest.php | 10 +- tests/Feature/InsertTest.php | 18 +- tests/Feature/NewChainingMethodsTest.php | 17 +- tests/Feature/NewMethodsEdgeCasesTest.php | 8 +- tests/Feature/NewMethodsShardedTest.php | 12 +- tests/Feature/QueryDocumentationTest.php | 6 +- tests/Feature/ShardedFieldIndexTest.php | 4 +- tests/Feature/ShardingTest.php | 4 +- tests/Feature/UtilityMethodsTest.php | 6 +- tests/Integration/CRUDWorkflowTest.php | 14 +- tests/Integration/ConcurrencyTest.php | 28 +- tests/Integration/EdgeCasesTest.php | 106 +- .../Regression/ConcurrencyRegressionTest.php | 155 + tests/Regression/CrashRecoveryTest.php | 183 + tests/Regression/DifferentialFuzzTest.php | 425 + tests/Regression/FieldIndexRegressionTest.php | 188 + tests/Regression/LockStateRegressionTest.php | 193 + tests/Regression/PublicApiCoverageTest.php | 150 + tests/Regression/QueryRegressionTest.php | 445 + tests/Regression/RegressionTestCase.php | 141 + .../SecurityConfigRegressionTest.php | 371 + tests/Regression/SpatialRegressionTest.php | 345 + tests/Regression/StorageRegressionTest.php | 307 + tests/Unit/LimitTest.php | 5 +- tests/noneDBTestCase.php | 14 +- tests/operation_benchmark.php | 80 + tests/operator_benchmark.php | 6 +- tests/performance_benchmark.php | 71 +- tests/sleekdb_comparison.php | 28 +- tests/sleekdb_vs_nonedb_benchmark.php | 23 +- tests/spatial_benchmark.php | 14 +- 60 files changed, 9282 insertions(+), 6631 deletions(-) create mode 100644 examples/_bootstrap.php create mode 100644 tests/Regression/ConcurrencyRegressionTest.php create mode 100644 tests/Regression/CrashRecoveryTest.php create mode 100644 tests/Regression/DifferentialFuzzTest.php create mode 100644 tests/Regression/FieldIndexRegressionTest.php create mode 100644 tests/Regression/LockStateRegressionTest.php create mode 100644 tests/Regression/PublicApiCoverageTest.php create mode 100644 tests/Regression/QueryRegressionTest.php create mode 100644 tests/Regression/RegressionTestCase.php create mode 100644 tests/Regression/SecurityConfigRegressionTest.php create mode 100644 tests/Regression/SpatialRegressionTest.php create mode 100644 tests/Regression/StorageRegressionTest.php create mode 100644 tests/operation_benchmark.php diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 5b9edf9..61e8acd 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -14,7 +14,7 @@ jobs: strategy: fail-fast: false matrix: - php-version: ['7.4', '8.0', '8.1', '8.2', '8.3'] + php-version: ['7.4', '8.0', '8.1', '8.2', '8.3', '8.4', '8.5'] steps: - name: Checkout code @@ -39,18 +39,24 @@ jobs: uses: actions/cache@v4 with: path: ${{ steps.composer-cache.outputs.dir }} - key: ${{ runner.os }}-composer-${{ hashFiles('**/composer.lock') }} - restore-keys: ${{ runner.os }}-composer- + # composer.lock is not committed, so the cache follows composer.json + key: ${{ runner.os }}-php${{ matrix.php-version }}-composer-${{ hashFiles('composer.json') }} + restore-keys: ${{ runner.os }}-php${{ matrix.php-version }}-composer- - name: Install dependencies run: composer install --prefer-dist --no-progress --no-interaction - - name: Create test database directory - run: mkdir -p db - - - name: Run PHPUnit tests + - name: Run PHPUnit tests (Unit, Feature, Integration, Regression) run: vendor/bin/phpunit --testdox --colors=always + - name: Run the examples (each must finish without warnings) + run: | + for f in examples/*.php; do + case "$f" in examples/_bootstrap.php|examples/performance.php) continue;; esac + out=$(php -d error_reporting=E_ALL -d display_errors=1 "$f" 2>&1) || { echo "$out"; exit 1; } + if echo "$out" | grep -qiE "warning|notice|deprecated|fatal|uncaught"; then echo "$f:"; echo "$out"; exit 1; fi + done + - name: Cleanup test databases if: always() - run: rm -rf db/*.json + run: rm -rf tests/test_db examples/db diff --git a/.gitignore b/.gitignore index 1c00535..a3c39e3 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,6 @@ CLAUDE.md .nonedb !.nonedb.example /db + +# Example databases +/examples/db/ diff --git a/CHANGES.md b/CHANGES.md index 85c85bb..ed1a443 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,199 @@ # noneDB Changelog +## v3.2.0 (2026-10-23) + +### Reliability, security and correctness release + +This release fixes every finding of the full code review (see `CODE_REVIEW.md`). Most changes are +internal; the few behaviour changes are listed under **Upgrade notes** below. + +--- + +### Upgrade notes (read before upgrading) + +1. **secretKey is validated.** An empty, missing, non-string or placeholder key + (`CHANGE_THIS_TO_A_SECURE_RANDOM_STRING`, ...) now throws `InvalidArgumentException`. + If your databases were created with the placeholder key, re-key them once: + ```php + noneDB::migrateKey('/path/to/db', 'CHANGE_THIS_TO_A_SECURE_RANDOM_STRING', bin2hex(random_bytes(32))); + // then put the new key in .nonedb + ``` +2. **Database names are validated, not sanitized.** Only `A-Z a-z 0-9`, space, `'` and `-` are allowed. + Other characters (`_`, `.`, `!`, non-ASCII, ...) throw `InvalidArgumentException` instead of being + silently removed (`"my_db"` used to become `"mydb"`, and `"çalışan"` became `"alan"`, colliding with + another database). Data written under the old, sanitized name is still there: use that name + (e.g. `"mydb"`) or rename your code's names to valid ones. +3. **Relative `dbDir` in `.nonedb` is resolved against the `.nonedb` file** (`./db/` used to follow the + running script, `db/` the working directory). Web pages and CLI/cron scripts therefore share one + database directory. In a config array a relative `dbDir` follows the entry script. If an old + directory exists and the new one doesn't, it keeps being used and an `E_USER_DEPRECATED` notice + tells you to move it. +4. **A config array is merged over the `.nonedb` file** (as the documentation always said). Before, an + array without `secretKey` silently ran with an empty key. +5. **The write buffer is off by default** (`bufferEnabled: false`). Direct writes are now faster and + always visible to other processes. Buffers left by older versions are flushed automatically. +6. **Keys never change.** `compact()` keeps keys on sharded databases too (it used to renumber them), + and drops shards that became empty. +7. **Reads never create anything.** `find()`, `count()`, `getDBs('name')`, `join()`, spatial queries... + on a database that doesn't exist return an empty result (or `false` when `autoCreateDB` is off) and + create no files. Only writes (`insert`, `createDB`, ...) create databases. +8. **Floats keep their type.** `2.0` is stored as `2.0` and read back as a float (it used to come back + as the integer `2`). Already stored values are unchanged. +9. **`lockTimeout` is honoured.** Waiting longer than `lockTimeout` seconds for a database lock throws + `RuntimeException` (it used to fall back to waiting forever). `0` waits forever. +10. **`noneDB.php` no longer sets `memory_limit` to `-1`** when it is included. +11. **Lock files carry state.** Each database has a `{db}.nonedb.lock` file holding a small state record + (change counter, pending buffer, index journal). Don't delete lock files while processes use the + database. Old-version leftovers (pre-3.2 buffer files, missing index list) are looked for once per + database, not on every operation. Don't run 3.1 and 3.2 on the same database directory; if you go + back to 3.1 and later return to 3.2, delete the `*.nonedb.lock` files first (with no process + running) so that check runs again. + +### Concurrency and durability + +- **Database-level locking.** Every operation holds a lock on its database for its whole duration + (shared for reads, exclusive for writes), across processes and instances. Concurrent inserts, + updates and deletes no longer lose records or reuse keys (K-01). +- **Caches can't go stale.** A generation counter in the lock file's state record invalidates every + cached index, meta and field/spatial index of a database as soon as another process writes it + (K-07, Y-11, Y-13). +- **Atomic file writes.** Index, meta and other JSON files are written to a temp file and renamed; + a crash can no longer leave a truncated index (Y-14). Unencodable data (invalid UTF-8, `NAN`, `INF`) + is rejected before anything is written (K-03). +- **Self-healing indexes.** A missing, corrupt or out-of-date key index is rebuilt from the data file; + deletions append a tombstone line so deleted records stay deleted after a rebuild. A torn last line + (crash during an append) is cut off. Records are verified against their key when read. +- Lock file handles are opened close-on-exec: programs started by the process don't inherit them (an + inherited copy would keep a lock held after the process died in the middle of a write). +- **Write journal.** Field indexes, spatial indexes and shard counters are rebuilt automatically if a + writer dies half-way through an operation (the journal entry lives in the lock file's state record). +- **Idempotent buffer flush.** A crash during a buffer flush can no longer lose or duplicate records (K-08). +- Tested with real `SIGKILL` crashes and multi-process stress tests (`tests/Regression`). + +### Performance + +- **No per-operation checks for old versions.** The lock file is read on every lock acquisition anyway; + its state record says whether the database was already checked for pre-3.2 leftovers, whether the + write buffer holds records and whether the index journal has entries. Normal reads and writes + therefore don't stat buffer files, glob for old files or re-derive the index list. Pre-3.0 data files + are converted when their (missing) index is first needed. +- Lock file handles stay open between operations (one per database, at most 64; never shared with a + forked child, never trusted after the file was replaced), and the change counter is written once per + write session with a single fixed-size write. +- Loaded key indexes are trusted until another process writes the database (no `stat` per read). +- Query builder: `where()`/`orWhere()` filters are compiled once per query (operators validated, + `$like`/`$regex` compiled), sort keys are computed once per record, and reading records in file order + no longer sorts the offset map when it is already in order. +- `new noneDB()` no longer resolves the script directory unless a relative `dbDir` needs it, and a + `dbDir` given in the config array skips resolving the file's one. +- Query sorting uses `array_multisort` on precomputed sort keys (same order as before, stable on every + PHP version). +- `query()->nearest(k)` combined with other filters reads the nearest candidates in growing batches + instead of putting every record in distance order (same results; it used to sort the whole database + since k is applied after the other filters). + +### Security + +- `secretKey` is always honoured: the persistent hash cache is per key + (`.nonedb_hash_cache_`, mode 0600) and can no longer let a wrong key read data or + "poison" the right key (S-01, S-02, S-03, S-14). The old `.nonedb_hash_cache` file is ignored + (`migrateKey()` removes it). +- The key is never shown by `print_r`/`var_dump`/`var_export`, and instances can't be serialized (S-09). +- New database directories get an `.htaccess` (`Require all denied`) and an empty `index.html` (S-06). +- `noneDB::migrateKey($dir, $old, $new)` re-keys existing databases (also those from before 3.2, without + leaving anything under the old key). +- A `secretKey` given in the config array is always validated as given: `null` is rejected also when + there is no `.nonedb` file (it was reported as a missing config file) and in dev mode (it was silently + replaced by the predictable dev mode key). +- Config values are validated (`shardSize`, `lockTimeout`, `lockRetryDelay`, `autoCompactThreshold`). + +### Storage engine + +- Pre-3.0 files are detected safely: a record whose first field is `data` is no longer mistaken for the + old format and destroyed (K-02). +- Sharding: keys come from `meta.nextKey` and the shard size stored in each database's meta, so + reopening with a different `shardSize` is safe (Y-10); migration never reuses keys (K-06); key lookups + on sharded databases see new records (Y-09); empty shards are removed by `compact()`. +- Auto-compaction triggers when deleted/superseded records exceed the threshold relative to the file + (not to the key counter) (O-22). +- Key filters accept integers and digit strings only (`'abc'`, `null`, `true`, `1.9` match nothing) and + are combined (AND) with the other filters (Y-15, Y-16). `null` filters behave the same in `find`, + `update` and `delete` (Y-17). Malformed `update()` input is rejected (Y-18). +- New `dropDB($name)`. + +### Field and spatial indexes + +- Indexed fields are recorded in `{db}.nonedb.indexes`; field names may contain any characters + (`e-mail`, `şehir`, `first.name`) (Y-08). New shards, `null` values and type changes are indexed + correctly (Y-12). Index files written by older versions are rebuilt on first use. +- Batch operations write each index file once (P-02). +- Spatial: indexes work on sharded databases (Y-02), cover new shards (Y-05) and survive migration + (Y-04); buffered inserts are indexed (Y-01); the R-tree can no longer loop forever (Y-03). +- `nearest()` no longer cuts through a rounding tie at its search radius: a record just outside the + radius whose distance rounds to the same value (and has the smaller key) was skipped although it sorts + first. +- Spatial queries: correct results near the antimeridian and the poles (Y-06); `nearest()` with + `maxDistance` returns the matches it finds (Y-07); `withinDistance()` measures to the nearest point of + a geometry (a point inside a polygon is at distance 0); `withinPolygon()` accepts MultiPolygon and + checks edges, not only vertices; queries without hits return immediately (O-24, O-34). +- GeoJSON validation rejects NaN/INF, strings as coordinates, self-intersecting rings and holes outside + their shell; ring closure is compared numerically. + +### Query builder + +- `where()` calls are combined with AND (they used to overwrite the same field) (Y-21). +- `orWhere()` without `where()` returns only the records matching an OR group (it returned everything) (Y-20). +- Operators compare by type: numbers (and numeric strings) numerically, strings lexically; `null`, + booleans and arrays never satisfy `$gt/$gte/$lt/$lte/between` (O-10). Unknown operators and invalid + operands (`$in` without an array, a broken `$regex`) throw, whether or not there are records (O-11). +- `like()`, `$like` and `$regex` are safe: like patterns are literal except a leading `^` / trailing `$`; + invalid regular expressions throw `InvalidArgumentException` (Y-19). +- `sort()` is a stable total order (missing/null first, then booleans, numbers, strings, arrays); several + `sort()` calls sort by several fields (Y-22, O-12). +- `count()`, `withDistance()`, `nearest()`+`where()`, `groupBy()` key types, `search()` (Unicode, + ignores `key`) fixed (O-01, O-04..O-08). `withDistance()` measures like `withinDistance()`/`nearest()` + (0 inside a polygon, else to the nearest point of the geometry); it used the centroid, so a polygon + could pass a 1 km filter and show a larger `_distance`. +- The direct `$db->between()` compares like `query()->between()` (it used PHP's loose comparison, so + e.g. `true` was "between" 100 and 500). +- `removeFields()` removes fields from the stored records only; `select/except/join/withDistance` + output never reaches storage (K-04, Y-26). +- `update()`, `delete()` and `removeFields()` run selection and write under one lock; the reads of one + query (`get()`, `count()`, aggregates) run under one shared lock, so they see one state of the database. + +### Other + +- `getDBs()` lists each database once, includes sharded databases with their total size (O-03). +- `limit([])` and `sort([])` return `[]` (D-05). Typo "paramters" fixed (D-06). +- Removed dead code (legacy `.idx` key index, unused helpers); the query builder no longer uses Reflection. +- Examples use `examples/_bootstrap.php`, run from any directory and can be run repeatedly. +- Distance and centroid memo caches keep working when full (the older half is dropped) (D-08). +- Docs: `join()` example corrected (the record is attached under the foreign database's name), and the + docs were checked against the code (return shapes, error messages, `whereNot()`/`notBetween()` + semantics, spatial distance options). +- Benchmark scripts (`tests/*_benchmark.php`, `tests/sleekdb_comparison.php`) use their own temporary + directory and key; `performance_benchmark.php` and `sleekdb_comparison.php` used to delete every file + in the project's `db/` directory. Sizes can be passed on the command line (`php tests/performance_benchmark.php 100,1000`). + New `tests/operation_benchmark.php` measures single-operation latency and can compare two versions; + `docs/BENCHMARKS.md` was re-measured (3.2 vs 3.1 on the same machine). + +### Test Results + +- **PHP 8.5 compatible.** Tested on PHP 7.4, 8.2, 8.4 and 8.5 (Linux and macOS); CI runs 7.4 to 8.5. + PHPUnit now fails on deprecations, so a new PHP version's deprecations show up in CI. The tests no + longer call `Reflection*::setAccessible()` on PHP 8.1+ (deprecated in 8.5), tests that kill child + processes don't go through a shell (with dash as `/bin/sh` the shell was killed and the child kept + running), and the dev mode test runs in its own process (its constant leaked into later tests). +- **1084 tests** (about 19,000 assertions), new suite `tests/Regression` (Regression): + one or more tests per review finding, multi-process tests, real `SIGKILL` crash tests, differential + fuzzing against an independent model, lock-state tests. +- Expectations of existing tests changed only where a documented behaviour change requires it + (database name validation, secretKey validation, `limit([])`/`sort([])`, `insert([])`, `orWhere()` and + repeated `where()` semantics). + +--- + + ## v3.1.0 (2025-12-29) ### Major: Spatial Indexing + MongoDB-Style Comparison Operators diff --git a/README.md b/README.md index ba82563..6f270d7 100755 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # noneDB -[![Version](https://img.shields.io/badge/version-3.1.0-orange.svg)](CHANGES.md) +[![Version](https://img.shields.io/badge/version-3.2.0-orange.svg)](CHANGES.md) [![PHP Version](https://img.shields.io/badge/PHP-7.4%2B-blue.svg)](https://php.net) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) -[![Tests](https://img.shields.io/badge/tests-970%20passed-brightgreen.svg)](tests/) -[![Thread Safe](https://img.shields.io/badge/thread--safe-atomic%20locking-success.svg)](#concurrent-access) +[![Tests](https://img.shields.io/badge/tests-1084%20passed-brightgreen.svg)](tests/) +[![Concurrency](https://img.shields.io/badge/multi--process-safe-success.svg)](#concurrent-access) **noneDB** is a lightweight, file-based NoSQL database for PHP. No installation required - just include and go! @@ -15,13 +15,13 @@ - **O(1) key lookups** - JSONL storage with byte-offset indexing - **Spatial indexing** - R-tree for geospatial queries (v3.1) - **MongoDB-style operators** - `$gt`, `$gte`, `$lt`, `$lte`, `$ne`, `$in`, `$nin`, `$exists`, `$like`, `$regex`, `$contains` (v3.1) -- **Auto-sharding** for large datasets (500K+ tested) -- **Thread-safe** - atomic file locking for concurrent access +- **Auto-sharding** for large datasets +- **Multi-process safe** - database-level locking, crash-safe writes, self-healing indexes - **Method chaining** - fluent query builder interface ## Requirements -- PHP 7.4 or higher +- PHP 7.4 or higher (tested on 7.4 up to 8.5) - Write permission on database directory ## Installation @@ -76,7 +76,7 @@ $db->delete("users", ["name" => "John"]); ## Configuration -Create a `.nonedb` file in your project root: +Create a `.nonedb` file in your project root (outside the web root): ```json { @@ -86,11 +86,18 @@ Create a `.nonedb` file in your project root: } ``` +- `secretKey` must be a non-empty string and not one of the example placeholders (32+ random characters + recommended: `php -r "echo bin2hex(random_bytes(32));"`). It makes database **file names** + unguessable; it does **not** encrypt the data — keep `dbDir` out of the web root. +- A relative `dbDir` (`./db/` or `db/`) is resolved against the `.nonedb` file's directory, so web pages + and CLI/cron scripts use the same databases. +- To change the key of existing databases use `noneDB::migrateKey($dbDir, $oldKey, $newKey)`. + **Or programmatically:** ```php $db = new noneDB([ - 'secretKey' => 'your_secure_key', + 'secretKey' => getenv('NONEDB_SECRET'), // your own random key (example placeholders are rejected) 'dbDir' => '/path/to/db/' ]); ``` @@ -104,6 +111,11 @@ $db = new noneDB(); > See [docs/CONFIGURATION.md](docs/CONFIGURATION.md) for all options. +### Database names + +Names may contain `A-Z a-z 0-9`, space, apostrophe (`'`) and hyphen (`-`). Anything else throws +`InvalidArgumentException` (names are never silently changed). `"0"` is a valid name. + --- ## Query Builder @@ -195,13 +207,15 @@ $inArea = $db->withinBBox("restaurants", "location", 28.97, 41.00, 29.00, 41.03) | Operation | 10K Records | 100K Records | |-----------|-------------|--------------| -| find(key) | **0.03 ms** | **0.05 ms** | -| find(filter) | 50 ms | 520 ms | -| insert (batch) | 290 ms | 3.1 s | +| find(key) | **0.05 ms** | **0.06 ms** | +| find(key), first lookup of a process | 3 ms | 4 ms | +| find(filter) | 32 ms | 336 ms | +| insert (batch) | 121 ms | 1.3 s | +| update (filter, 20% of records) | 40 ms | 455 ms | | count() | **< 1 ms** | **< 1 ms** | -| withinDistance | 10-20 ms | 50-100 ms | -> Benchmarks on Apple Silicon. Key lookups are O(1) with byte-offset indexing. +> PHP 8.2 on an Apple M2, including the per-operation database lock. Key lookups are O(1) with +> byte-offset indexing. Details, 3.1 comparison and SleekDB comparison: [docs/BENCHMARKS.md](docs/BENCHMARKS.md). --- @@ -222,11 +236,28 @@ vendor/bin/phpunit --testsuite Feature ## Concurrent Access -noneDB uses atomic file locking (`flock()`) for thread-safe operations: +Each operation locks its database (`flock()` on `{db}.nonedb.lock`): shared for reads, exclusive for +writes, across processes and instances. + +- **No lost updates** - concurrent inserts/updates/deletes are serialized; keys are never reused +- **Fresh caches** - a generation counter invalidates cached indexes when another process writes +- **Crash safety** - files are replaced atomically; a writer killed mid-operation leaves nothing + half-written: key, field and spatial indexes are repaired from the data on the next access +- **Lock timeout** - waiting longer than `lockTimeout` seconds throws `RuntimeException` + +The lock file also holds a small state record (change counter, pending buffer, index journal), so these +checks cost no extra file access. Don't delete `*.nonedb.lock` files while processes use the databases. + +This is verified by multi-process stress tests and real `SIGKILL` crash tests in `tests/Regression`. + +--- + +## Upgrading from 3.1 -- **No lost updates** - concurrent writes are serialized -- **Read consistency** - reads wait for ongoing writes -- **Crash safety** - locks auto-release on process termination +Read the [v3.2.0 upgrade notes](CHANGES.md#upgrade-notes-read-before-upgrading): secretKey and +database names are validated, a relative `dbDir` follows the `.nonedb` file, buffering is off by default +and `compact()` no longer renumbers keys. Don't run 3.1 and 3.2 on the same database directory at the +same time. --- diff --git a/docs/API.md b/docs/API.md index f9eada6..a551d03 100644 --- a/docs/API.md +++ b/docs/API.md @@ -2,7 +2,13 @@ Complete reference for all noneDB methods and operations. -**Version:** 3.1.0 +**Version:** 3.2.0 + +**Database names** may contain `A-Z a-z 0-9`, space, `'` and `-`; every method throws +`InvalidArgumentException` for anything else (names are never changed silently). `"0"` is valid. + +**Locking:** every method locks its database for its whole duration (shared for reads, exclusive for +writes). Waiting longer than `lockTimeout` seconds throws `RuntimeException`. --- @@ -36,7 +42,26 @@ Check if database exists. Creates it if `autoCreateDB` is true. ```php $exists = $db->checkDB("users"); -// Returns: true or false +// Returns: true or false (false for null / "") +``` + +### dropDB() + +Delete a database and all of its files (data, shards, indexes, buffers, metadata). + +```php +$result = $db->dropDB("users"); +// ["success" => true, "files" => 7] or ["success" => false, "files" => 0, "error" => "Database does not exist"] +``` + +### migrateKey() (static) + +Re-key existing databases (all files of every database created with `$oldKey` are renamed). Run it +while nothing else uses the databases, then put the new key in your config. + +```php +$result = noneDB::migrateKey('/var/data/db', $oldKey, $newKey); +// ["success" => true, "databases" => ["users", "orders"], "files" => 14, "error" => null] ``` ### getDBs() @@ -55,11 +80,14 @@ $dbs = $db->getDBs(true); // ... // ] -// Single database info +// Single database info (false if it doesn't exist; never creates it) $info = $db->getDBs("users"); // ["name" => "users", "createdTime" => 1703123456, "size" => "2,5 KB"] ``` +Only databases of the current `secretKey` are listed; sharded databases appear once, with the total +size of their shards. Sizes use a comma as decimal separator (`"2,5 KB"`). + --- ## CRUD Operations @@ -97,8 +125,18 @@ $db->insert("users", ["key" => "value"]); // Invalid data $db->insert("users", "not an array"); // ["n" => 0, "error" => "insert data must be array"] + +// Empty data, or a mix of a list and named fields +$db->insert("users", []); // ["n" => 0, "error" => "insert data must not be empty"] +$db->insert("users", [0 => ["a" => 1], "name" => "x"]); // ["n" => 0, "error" => "insert data must be either one record or a list of records"] + +// Values JSON can't represent (invalid UTF-8, NAN, INF): nothing is written +$db->insert("users", ["v" => NAN]); // ["n" => 0, "error" => "insert data cannot be encoded as JSON ..."] ``` +Keys are assigned in order and are **never reused or changed** (not by deletes, not by `compact()`). +Floats keep their type (`2.0` is read back as `2.0`). + ### find() Find records matching filter. @@ -119,7 +157,17 @@ $results = $db->find("users", ["key" => 0]); $results = $db->find("users", ["key" => [0, 1, 2]]); ``` -**Returns:** Array of records with `key` field added. +**Matching rules** (shared by `find()`, `update()` and `delete()`): + +- Every filter field must exist in the record and be **identical** (`===`): `"25"` does not match `25`, + `null` matches only a field that exists with the value `null`, arrays must be equal as a whole. +- `key` accepts an integer, a digit string (`"5"`) or a list of them; anything else matches nothing. + It is combined with the other filters (AND). +- Any integer (normally `0`), `null` and `[]` select every record. Other non-array filters return `false`. + +**Returns:** Array of records with `key` field added (in key order, or in the order of a `key` list). +Returns `[]` for a database that doesn't exist (`false` if `autoCreateDB` is off); reading never +creates a database. ### update() @@ -146,6 +194,10 @@ $result = $db->update("users", [ ]); ``` +The first element must be a filter array and `set` a non-`key` array of fields; otherwise +`["n" => 0, "error" => "Please check your update parameters"]`. An empty `set` changes nothing and +returns the number of matching records. + ### delete() Delete matching records. @@ -158,7 +210,7 @@ $result = $db->delete("users", ["name" => "John"]); // By key $result = $db->delete("users", ["key" => [0, 2]]); -// All records +// All records (an empty filter matches everything) $result = $db->delete("users", []); ``` @@ -176,35 +228,44 @@ $query = $db->query("users"); | Method | Description | |--------|-------------| -| `where($filters)` | AND filter with operator support | -| `orWhere($filters)` | OR filter | +| `where($filters)` | AND filter with operator support; several calls are all applied (AND) | +| `orWhere($filters)` | OR group: matches if all `where()` groups match **or** any `orWhere()` group does | | `whereIn($field, $values)` | Value in array | | `whereNotIn($field, $values)` | Value not in array | -| `whereNot($filters)` | Not equal | +| `whereNot($filters)` | Excludes records where **any** of the given fields equals its value (`===`); several calls all apply | | `like($field, $pattern)` | Pattern match | | `notLike($field, $pattern)` | Pattern not match | | `between($field, $min, $max)` | Range (inclusive) | -| `notBetween($field, $min, $max)` | Outside range | +| `notBetween($field, $min, $max)` | Not inside the range: also returns records without the field or with a value that can't be compared (e.g. `null`, `true`, text vs numbers) | | `search($term, $fields)` | Full-text search | ### Comparison Operators ```php +// Several operators on one field go into one array (a PHP array can't repeat a key) $db->query("users")->where([ - 'age' => ['$gt' => 18], // Greater than - 'age' => ['$gte' => 18], // Greater than or equal - 'age' => ['$lt' => 65], // Less than - 'age' => ['$lte' => 65], // Less than or equal - 'role' => ['$ne' => 'guest'],// Not equal - 'role' => ['$eq' => 'admin'],// Equal (explicit) - 'status' => ['$in' => ['active', 'pending']], // In array - 'status' => ['$nin' => ['banned', 'deleted']], // Not in array - 'email' => ['$exists' => true], // Field exists - 'name' => ['$like' => 'John'], // Pattern match - 'email' => ['$regex' => '@gmail.com$'], // Regex - 'tags' => ['$contains' => 'featured'] // Array/string contains -]); -``` + 'age' => ['$gte' => 18, '$lt' => 65], // Range + 'role' => ['$ne' => 'guest'], // Not equal + 'status' => ['$in' => ['active', 'pending']], // In array (strict) + 'email' => ['$exists' => true], // Field exists + 'name' => ['$like' => '^John'], // Pattern (only ^ and $ are special) + 'tags' => ['$contains' => 'featured'] // Array element / substring +])->get(); + +// ...or one where() per condition: all of them apply +$db->query("users") + ->where(['age' => ['$gt' => 18]]) + ->where(['age' => ['$lte' => 65]]) + ->where(['email' => ['$regex' => '@gmail\\.com$']]) // regular expression, case-insensitive + ->get(); +``` + +- `$gt`, `$gte`, `$lt`, `$lte` (and `between()`) compare numbers and numeric strings numerically and + other strings lexically. `null`, booleans, arrays, and a number against a non-numeric string never + match. +- `$eq`, `$ne`, `$in`, `$nin` use strict comparison (`===`). +- An unknown operator (e.g. a typo like `$gtee`), `$in`/`$nin` without an array or an invalid `$regex` + throws `InvalidArgumentException`, also when there are no records to check. ### Modifiers @@ -212,12 +273,12 @@ $db->query("users")->where([ |--------|-------------| | `select($fields)` | Include only specified fields | | `except($fields)` | Exclude specified fields | -| `sort($field, $order)` | Sort results | +| `sort($field, $order)` | Sort results; further calls add tie-breaking keys | | `limit($count)` | Limit results | | `offset($count)` | Skip records | -| `join($db, $localKey, $foreignKey)` | Join databases | -| `groupBy($field)` | Group results | -| `having($aggregate, $op, $value)` | Filter groups | +| `join($db, $localKey, $foreignKey, $alias = null)` | Attach the matching foreign record under `$alias` (default: the foreign database name; it replaces a field of that name in the result only). With duplicate foreign keys the last record wins | +| `groupBy($field)` | Group results (values keep their type: `1`, `"1"` and `1.5` are different groups) | +| `having($aggregate, $op, $value)` | Filter groups: `count`, `sum:f`, `avg:f`, `min:f`, `max:f`. Ignored without `groupBy()` or for unknown aggregates/operators | ### Terminal Methods @@ -233,8 +294,9 @@ $db->query("users")->where([ | `min($field)` | Minimum value | | `max($field)` | Maximum value | | `distinct($field)` | Unique values | -| `update($set)` | Update matches | -| `delete()` | Delete matches | +| `update($set)` | Update matches (sort/offset/limit choose which ones) | +| `delete()` | Delete matches (sort/offset/limit choose which ones) | +| `removeFields($fields)` | Remove fields from the **stored** matching records | --- @@ -289,9 +351,12 @@ $info = $db->getShardInfo("users"); ### compact() +Rewrites data files without deleted and superseded records. Keys never change; shards that became +empty are removed. Usually not needed: compaction also runs automatically. + ```php $result = $db->compact("users"); -// ["success" => true, "freedSlots" => 150, ...] +// ["success" => true, "freedSlots" => 150, "totalRecords" => 44850, "sharded" => true, "newShardCount" => 5] ``` ### migrate() @@ -316,7 +381,8 @@ $db->getShardSize(); // 10000 ```php $result = $db->createFieldIndex("users", "email"); -// ["success" => true] +// ["success" => true, "indexed" => 150, "values" => 120, "error" => null] (+ "shards" => N when sharded) +// Calling it again for an indexed field rebuilds the index ``` ### hasFieldIndex() @@ -326,10 +392,22 @@ $exists = $db->hasFieldIndex("users", "email"); // true/false ``` +### getFieldIndexes() / rebuildFieldIndex() / getKeysFromFieldIndex() + +```php +$db->getFieldIndexes("users"); // ["fields" => ["email", "e-mail"], "sharded" => false] +$db->rebuildFieldIndex("users", "email"); // same result as createFieldIndex() +$db->getKeysFromFieldIndex("users", "status", "active"); // [0, 4, 7] or null without an index +``` + +Field names may contain any characters. Indexes are maintained by every write and give exactly the +results of a full scan. + ### dropFieldIndex() ```php $result = $db->dropFieldIndex("users", "email"); +// ["success" => true, ...] or ["success" => false, "error" => "Index does not exist"] ``` --- @@ -340,7 +418,8 @@ $result = $db->dropFieldIndex("users", "email"); ```php $result = $db->createSpatialIndex("locations", "coords"); -// ["success" => true, "indexed" => 150] +// ["success" => true, "indexed" => 150, "error" => null] +// An existing index: ["success" => false, "indexed" => 0, "error" => "Spatial index already exists for field 'coords'"] // Records with invalid/missing GeoJSON are skipped (not indexed) // Always check 'indexed' count to verify @@ -364,6 +443,7 @@ $indexes = $db->getSpatialIndexes("locations"); ```php $result = $db->dropSpatialIndex("locations", "coords"); +// ["success" => true, "error" => null], also when there was no such index (unlike dropFieldIndex()) ``` ### rebuildSpatialIndex() @@ -404,7 +484,8 @@ $inside = $db->withinPolygon("locations", "coords", $polygon); ```php $result = $db->validateGeoJSON($geometry); -// ["valid" => true] or ["valid" => false, "error" => "..."] +// ["valid" => true, "error" => null, "type" => "Point"] +// ["valid" => false, "error" => "Longitude must be between -180 and 180", "type" => null] ``` --- @@ -418,10 +499,13 @@ $sorted = $db->sort($results, "name", "asc"); $sorted = $db->sort($results, "created_at", "desc"); ``` +Stable, total order: missing/`null` < booleans < numbers < strings < arrays (reversed for `desc`). +Returns `[]` for an empty array and `false` for a non-array. + ### limit() ```php -$limited = $db->limit($results, 10); +$limited = $db->limit($results, 10); // [] for an empty array, false for invalid arguments ``` ### first() / last() @@ -446,6 +530,9 @@ $results = $db->like("users", "name", "^John"); // Starts with $results = $db->like("users", "name", "son$"); // Ends with ``` +Case-insensitive. Only a leading `^` and a trailing `$` are special; every other character (`.`, `/`, +`*`, ...) matches itself. + ### between() ```php @@ -461,6 +548,21 @@ noneDB::disableStaticCache(); noneDB::enableStaticCache(); ``` +### Buffering and indexes + +```php +$db->enableBuffering(true); // off by default +$db->setBufferSizeLimit(1048576); // bytes (min 1KB) +$db->setBufferCountLimit(10000); // records (min 10) +$db->setBufferFlushInterval(30); // seconds, 0 = no time-based flush +$db->flush("users"); // ["success" => true, "flushed" => 12, "error" => null] +$db->flushAllBuffers(); // ["databases" => 2, "flushed" => 40] +$db->getBufferInfo("users"); // ["enabled" => true, ..., "buffers" => ["main" => ["exists" => true, "size" => 812, "records" => 12]]] + +$db->rebuildIndex("users"); // rebuild the key indexes from the data files (recovery tool) +$db->getIndexInfo("users"); // ["exists" => true, "totalRecords" => 120, "sharded" => false, ...] +``` + ### Configuration ```php diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 3de3158..3e0c73d 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,30 +1,38 @@ # noneDB Performance Benchmarks -Comprehensive performance benchmarks and database comparisons. +Performance measurements of noneDB 3.2.0, compared with 3.1.0 and SleekDB. -**Version:** 3.1.0 -**Test Environment:** PHP 8.2, macOS (Apple Silicon M-series) +**Version:** 3.2.0 +**Test environment:** PHP 8.2.0 (CLI), macOS 15.8, Apple M2, 16 GB RAM +**Measured:** 2026-10-24. The 3.1.0 numbers come from the same scripts, run on the same machine in the same session. + +All 3.2.0 numbers include the database lock every operation takes (shared for reads, exclusive for +writes) and crash-safe writes. 3.1.0 took no lock for reads and overwrote index files in place. --- ## Table of Contents 1. [Test Data Structure](#test-data-structure) -2. [v3.0+ Optimizations](#v30-optimizations) -3. [O(1) Key Lookup Performance](#o1-key-lookup-performance) -4. [Write Operations](#write-operations) -5. [Read Operations](#read-operations) -6. [Query & Aggregation](#query--aggregation) -7. [Method Chaining](#method-chaining) -8. [Spatial Query Performance](#spatial-query-performance) -9. [Storage & Memory](#storage--memory) -10. [SleekDB Comparison](#sleekdb-comparison) +2. [What Changed in 3.2](#what-changed-in-32) +3. [Single Operations (5,000 records)](#single-operations-5000-records) +4. [Key Lookup](#key-lookup) +5. [3.2 vs 3.1 at Scale](#32-vs-31-at-scale) +6. [Write Operations](#write-operations) +7. [Read Operations](#read-operations) +8. [Query & Aggregation](#query--aggregation) +9. [Method Chaining](#method-chaining) +10. [Comparison Operators](#comparison-operators) +11. [Spatial Query Performance](#spatial-query-performance) +12. [Storage & Memory](#storage--memory) +13. [SleekDB Comparison](#sleekdb-comparison) +14. [Running Benchmarks](#running-benchmarks) --- ## Test Data Structure -All benchmarks use 7 fields per record: +The performance and SleekDB benchmarks use 7 fields per record: ```php [ @@ -40,47 +48,95 @@ All benchmarks use 7 fields per record: --- -## v3.0+ Optimizations - -| Optimization | Improvement | -|--------------|-------------| -| **Static Cache Sharing** | 80%+ for multi-instance | -| **Batch File Read** | 40-50% for bulk reads | -| **Batch Update/Delete** | **25-30x faster** for bulk operations | -| **Single-Pass Filtering** | 30% for complex queries | -| **O(1) Sharded Key Lookup** | True O(1) for all database sizes | -| **O(1) Count** | **100-330x faster** (index metadata lookup) | -| **Hash Cache Persistence** | Faster cold startup | -| **atomicReadFast()** | Optimized index reads | - -### v3.1.0 Spatial Optimizations - -| Optimization | Improvement | -|--------------|-------------| -| **Parent Pointer Map** | O(1) parent lookup (was O(n)) | -| **Linear Split Algorithm** | O(n) seed selection (was O(n²)) | -| **Dirty Flag Pattern** | Single disk write per batch (was n writes) | -| **Distance Memoization** | Cached Haversine calculations | -| **Centroid Caching** | Cached geometry centroids | -| **Node Size 32** | Fewer tree levels and splits | -| **Adaptive nearest()** | Exponential radius expansion | +## What Changed in 3.2 + +| Change | Effect | +|--------|--------| +| **Database lock per operation** | Correct results with several processes; costs ~0.01-0.02 ms per call | +| **Lock file state record** | Generation counter, buffer flag, index journal and the one-time old-version check are read with the lock: no per-operation `stat`/`glob` | +| **Kept lock handles** | No `fopen`/`fclose` per operation (pid- and inode-checked) | +| **Atomic index writes** (temp file + rename) | A crash never leaves a half-written index; a single-record write costs ~0.15 ms more | +| **Batch inserts/updates** | Records encoded and appended in one write, each index written once: insert 10x, filtered update 4x faster | +| **Per-shard key lookup** | A cold lookup loads only the target shard's index: 5 ms instead of 156 ms at 500K | +| **Compiled query filters** | `where()`/`orWhere()` compiled once per query; `array_multisort` on precomputed sort keys | +| **Adaptive `query()->nearest()`** | With other filters, searches a growing radius instead of sorting the whole database | + +### Earlier optimizations + +| Optimization | Since | +|--------------|-------| +| JSONL storage with byte-offset index (O(1) key lookup) | 3.0 | +| Static cache shared across instances | 3.0 | +| Batch file read (64 KB buffered) | 3.0 | +| O(1) count from index metadata | 3.0 | +| R-tree spatial index: parent pointer map, linear split, dirty-flag batch writes, node size 32 | 3.1 | +| Distance and centroid memoization | 3.1 | --- -## O(1) Key Lookup Performance +## Single Operations (5,000 records) + +Milliseconds per call, best of 5 runs (`tests/operation_benchmark.php`). + +| Operation | 3.1.0 | 3.2.0 | +|-----------|-------|-------| +| insert (5,000 records, one call) | 175.4 | **17.2** | +| insert (one record) | **0.169** | 0.321 | +| find (all records) | 16.2 | **10.4** | +| find (by key) | **0.026** | 0.047 | +| find (filter) | 17.4 | **12.0** | +| count | **0.002** | 0.018 | +| query (where range + sort + limit) | 25.5 | **19.9** | +| update (by key) | **0.830** | 0.977 | +| update (filter, 1,250 records) | 102.6 | **25.0** | +| delete (by key) | **0.933** | 1.1 | +| find (filter on indexed field) | 4.0 | **3.2** | +| insert (one record, indexed field) | **1.2** | 1.5 | +| new instance + find (by key) | **0.029** | 0.056 | + +Bulk work is much faster. The smallest operations pay a fixed cost: the lock (all operations) and the +atomic index replacement (single writes). Those are what make 3.2 safe with concurrent processes and +crashes. + +--- -Key lookups are **constant time** regardless of database size after cache warm-up. +## Key Lookup -| Records | Cold (first access) | Warm (cached) | Notes | -|---------|---------------------|---------------|-------| -| 100 | 3 ms | **0.03 ms** | Non-sharded | -| 1K | 3 ms | **0.03 ms** | Non-sharded | -| 10K | 49 ms | **0.03 ms** | Sharded (1 shard) | -| 50K | 243 ms | **0.05 ms** | Sharded (5 shards) | -| 100K | 497 ms | **0.05 ms** | Sharded (10 shards) | -| 500K | 2.5 s | **0.16 ms** | Sharded (50 shards) | +`find($db, ['key' => $k])`. Cold = nothing cached (first lookup of a new process); warm = average of +200 lookups spread over all shards, after each shard's index was loaded once. -> **Key insight:** Cold time includes loading shard index. After cache warm-up, all lookups are near-instant. +| Records | Cold 3.1.0 | Cold 3.2.0 | Warm 3.1.0 | Warm 3.2.0 | +|---------|------------|------------|------------|------------| +| 100 | 3 ms | **0.21 ms** | **0.030 ms** | 0.045 ms | +| 1,000 | 3 ms | **0.45 ms** | **0.030 ms** | 0.046 ms | +| 10,000 | 9 ms | **3 ms** | **0.030 ms** | 0.054 ms | +| 50,000 | 22 ms | **5 ms** | **0.042 ms** | 0.057 ms | +| 100,000 | 37 ms | **4 ms** | 0.060 ms | **0.057 ms** | +| 500,000 | 156 ms | **5 ms** | 0.206 ms | **0.066 ms** | + +A cold lookup in 3.2 reads one shard's index, so it stays ~5 ms at any size. + +--- + +## 3.2 vs 3.1 at Scale + +Same script (`tests/performance_benchmark.php`), same machine. + +| Operation | 100K 3.1.0 | 100K 3.2.0 | 500K 3.1.0 | 500K 3.2.0 | +|-----------|------------|------------|------------|------------| +| insert() | 3.6 s | **1.3 s** | 18.8 s | **7.1 s** | +| update() (city filter) | 2.1 s | **455 ms** | 19 s | **2.4 s** | +| delete() (department filter) | 2.4 s | **831 ms** | 20.8 s | **4.4 s** | +| find(all) | 598 ms | **376 ms** | 2.7 s | **1.8 s** | +| find(key), first after insert | 490 ms | **<1 ms** | 2.5 s | **<1 ms** | +| find(filter) | 514 ms | **336 ms** | 2.7 s | **1.8 s** | +| distinct() | 586 ms | **361 ms** | 3 s | **2.1 s** | +| sort() + limit | 1.3 s | **452 ms** | 7.1 s | **2.7 s** | +| search() | **736 ms** | 849 ms | **4.2 s** | 4.6 s | +| select() | 864 ms | **596 ms** | 4.6 s | **3.6 s** | +| complex chain | 676 ms | **484 ms** | 3.7 s | **2.7 s** | + +`search()` is slightly slower because it now matches Unicode text case-insensitively (`İzmir` = `izmir`). --- @@ -88,11 +144,11 @@ Key lookups are **constant time** regardless of database size after cache warm-u | Operation | 100 | 1K | 10K | 50K | 100K | 500K | |-----------|-----|-----|------|------|-------|-------| -| insert() | 7 ms | 25 ms | 289 ms | 1.5 s | 3.1 s | 16.5 s | -| update() | 1 ms | 11 ms | 120 ms | 660 ms | 1.5 s | 11.3 s | -| delete() | 2 ms | 13 ms | 144 ms | 773 ms | 1.7 s | 12.5 s | +| insert() | 6 ms | 7 ms | 121 ms | 596 ms | 1.3 s | 7.1 s | +| update() | <1 ms | 4 ms | 40 ms | 216 ms | 455 ms | 2.4 s | +| delete() | 2 ms | 8 ms | 78 ms | 426 ms | 831 ms | 4.4 s | -> Update/delete use batch operations for efficient bulk modifications (single index write per shard) +> `update()` changes the ~20% of records in one city, `delete()` the ~20% in one department. --- @@ -100,11 +156,9 @@ Key lookups are **constant time** regardless of database size after cache warm-u | Operation | 100 | 1K | 10K | 50K | 100K | 500K | |-----------|-----|-----|------|------|-------|-------| -| find(all) | 3 ms | 23 ms | 48 ms | 268 ms | 602 ms | 2.7 s | -| find(key) | <1 ms | <1 ms | 49 ms | 243 ms | 497 ms | 2.5 s | -| find(filter) | <1 ms | 4 ms | 50 ms | 252 ms | 515 ms | 2.6 s | - -> **find(key)** first call includes index loading. Subsequent calls: ~0.05ms (see O(1) table above) +| find(all) | <1 ms | 3 ms | 28 ms | 169 ms | 376 ms | 1.8 s | +| find(key) | <1 ms | <1 ms | <1 ms | <1 ms | <1 ms | <1 ms | +| find(filter) | <1 ms | 3 ms | 32 ms | 163 ms | 336 ms | 1.8 s | --- @@ -113,15 +167,13 @@ Key lookups are **constant time** regardless of database size after cache warm-u | Operation | 100 | 1K | 10K | 50K | 100K | 500K | |-----------|-----|-----|------|------|-------|-------| | count() | **<1 ms** | **<1 ms** | **<1 ms** | **<1 ms** | **<1 ms** | **<1 ms** | -| distinct() | <1 ms | 4 ms | 49 ms | 270 ms | 590 ms | 2.9 s | -| sum() | <1 ms | 4 ms | 49 ms | 261 ms | 588 ms | 3 s | -| like() | <1 ms | 5 ms | 57 ms | 311 ms | 670 ms | 3.4 s | -| between() | <1 ms | 4 ms | 53 ms | 288 ms | 628 ms | 3.2 s | -| sort() | <1 ms | 8 ms | 105 ms | 565 ms | 1.3 s | 7.1 s | -| first() | <1 ms | 4 ms | 50 ms | 285 ms | 589 ms | 2.9 s | -| exists() | <1 ms | 4 ms | 49 ms | 272 ms | 588 ms | 3 s | - -> **count()** uses O(1) index metadata lookup - no record scanning required! +| distinct() | <1 ms | 3 ms | 34 ms | 170 ms | 361 ms | 2.1 s | +| sum() | <1 ms | 3 ms | 31 ms | 167 ms | 354 ms | 2.1 s | +| like() | <1 ms | 4 ms | 38 ms | 209 ms | 432 ms | 2.5 s | +| between() | <1 ms | 4 ms | 43 ms | 229 ms | 502 ms | 2.7 s | +| sort() | <1 ms | 4 ms | 38 ms | 211 ms | 452 ms | 2.7 s | +| first() | <1 ms | 3 ms | 31 ms | 168 ms | 364 ms | 1.8 s | +| exists() | <1 ms | 3 ms | 32 ms | 173 ms | 370 ms | 1.9 s | --- @@ -129,111 +181,106 @@ Key lookups are **constant time** regardless of database size after cache warm-u | Operation | 100 | 1K | 10K | 50K | 100K | 500K | |-----------|-----|-----|------|------|-------|-------| -| whereIn() | <1 ms | 4 ms | 53 ms | 302 ms | 657 ms | 3.6 s | -| orWhere() | <1 ms | 4 ms | 55 ms | 316 ms | 673 ms | 3.5 s | -| search() | <1 ms | 5 ms | 61 ms | 350 ms | 762 ms | 4.2 s | -| groupBy() | <1 ms | 4 ms | 52 ms | 307 ms | 657 ms | 3.5 s | -| select() | <1 ms | 5 ms | 57 ms | 400 ms | 854 ms | 4.5 s | -| complex chain | <1 ms | 5 ms | 60 ms | 322 ms | 684 ms | 3.6 s | - -> **Complex chain:** `where() + whereIn() + between() + select() + sort() + limit()` +| whereIn() | <1 ms | 3 ms | 35 ms | 227 ms | 419 ms | 2.5 s | +| orWhere() | <1 ms | 4 ms | 39 ms | 220 ms | 460 ms | 2.5 s | +| search() | <1 ms | 7 ms | 76 ms | 398 ms | 849 ms | 4.6 s | +| groupBy() | <1 ms | 3 ms | 35 ms | 209 ms | 434 ms | 2.4 s | +| select() | <1 ms | 4 ms | 38 ms | 277 ms | 596 ms | 3.6 s | +| complex chain | <1 ms | 4 ms | 39 ms | 235 ms | 484 ms | 2.7 s | --- -## Comparison Operators (v3.1.0) +## Comparison Operators -MongoDB-style operators add minimal overhead: +`tests/operator_benchmark.php`, best of 3 runs (records with 10 fields, including a GeoJSON point). | Operator | 100 | 500 | 1K | 5K | |----------|-----|-----|-----|-----| -| `$gt` | 0.7 ms | 3 ms | 6 ms | 33 ms | -| `$gte + $lte` (range) | 0.7 ms | 3.4 ms | 7 ms | 38 ms | -| `$in` (2 values) | 0.7 ms | 3.3 ms | 6.5 ms | 36 ms | -| `$nin` (2 values) | 0.7 ms | 3.5 ms | 7 ms | 37 ms | -| `$ne` | 0.7 ms | 3 ms | 6 ms | 36 ms | -| `$like` | 0.7 ms | 3.4 ms | 7 ms | 39 ms | -| `$exists` | 0.6 ms | 3.2 ms | 6 ms | 37 ms | -| Complex (4 operators) | 0.7 ms | 3.5 ms | 7.5 ms | 43 ms | - -> Operators add <1ms overhead per operation. Linear scaling with record count. +| `$gt` | 0.68 ms | 2.9 ms | 5.8 ms | 29 ms | +| `$gte + $lte` (range) | 0.72 ms | 3.5 ms | 7.1 ms | 36 ms | +| `$in` (2 values) | 0.6 ms | 2.8 ms | 5.6 ms | 30 ms | +| `$nin` (2 values) | 0.6 ms | 2.8 ms | 5.7 ms | 29 ms | +| `$ne` | 0.59 ms | 2.8 ms | 5.5 ms | 30 ms | +| `$like` | 0.64 ms | 3 ms | 6 ms | 32 ms | +| `$exists` | 0.61 ms | 2.8 ms | 5.7 ms | 30 ms | +| Complex (4 operators) | 0.8 ms | 3.7 ms | 7.5 ms | 40 ms | --- ## Spatial Query Performance -Tested with R-tree spatial index (v3.1.0): - | Operation | 100 | 500 | 1K | 5K | |-----------|-----|-----|-----|-----| -| createSpatialIndex | 2.4 ms | 34 ms | 81 ms | 423 ms | -| withinDistance (10km) | 3.1 ms | 16 ms | 32 ms | 166 ms | -| withinBBox | 0.7 ms | 4.5 ms | 7 ms | 38 ms | -| nearest(10) | 1.9 ms | 1.3 ms | 2 ms | 2.4 ms | +| createSpatialIndex | 2.6 ms | 23 ms | 61 ms | 434 ms | +| withinDistance (10 km) | 1.1 ms | 5.2 ms | 10.5 ms | 64 ms | +| withinBBox | 0.31 ms | 1 ms | 2.1 ms | 10 ms | +| nearest(10) | 0.33 ms | 1.4 ms | 2.6 ms | 3.1 ms | -> Spatial queries use R-tree indexing for O(log n + k) performance where k = matching records. +> The 10 km circle covers most of the test area, so `withinDistance` returns most records; its cost is +> dominated by the matches, not by the R-tree. ### Spatial + Operator Combination | Query Type | 100 | 500 | 1K | 5K | |------------|-----|-----|-----|-----| -| withinDistance only | 3 ms | 16 ms | 32 ms | 166 ms | -| + where (simple) | 3 ms | 17 ms | 34 ms | 174 ms | -| + where (`$gte`) | 3 ms | 17 ms | 33 ms | 175 ms | -| + where (`$in`) | 3 ms | 16.5 ms | 33 ms | 174 ms | -| + range (`$gte` + `$lte`) | 3 ms | 17 ms | 36 ms | 212 ms | -| + complex (4 operators) | 3 ms | 16.5 ms | 34.5 ms | 179 ms | -| + sort + limit | 3 ms | 17 ms | 35 ms | 183 ms | -| nearest + operators + limit | 8 ms | 4.5 ms | 7 ms | 12 ms | - -> Spatial + operator combinations add minimal overhead. R-tree filtering happens first, then operators applied to candidates. +| withinDistance + where (simple) | 1.1 ms | 5.5 ms | 11 ms | 64 ms | +| + where (`$gte`) | 1.1 ms | 5.7 ms | 11 ms | 66 ms | +| + where (`$in`) | 1.1 ms | 5.6 ms | 12.5 ms | 74 ms | +| + range (`$gte` + `$lte`) | 1.2 ms | 6 ms | 12.4 ms | 70 ms | +| + complex (4 operators) | 1.2 ms | 5.8 ms | 11.5 ms | 66 ms | +| + sort + limit | 1.6 ms | 8 ms | 15 ms | 88 ms | +| nearest(50) + operators + limit | 1.6 ms | 5.5 ms | 3.9 ms | 7.8 ms | + +> Since 3.2 `nearest(k)` returns the k nearest records **that match the other filters** (3.1 cut to k +> first and then filtered, which could return fewer or wrong records). With a spatial index this uses a +> growing search radius, so it stays fast as the database grows. --- ## Storage & Memory -| Records | File Size | Peak Memory | +| Records | Data Size | Peak Memory | |---------|-----------|-------------| -| 100 | 10 KB | 2 MB | -| 1,000 | 98 KB | 4 MB | -| 10,000 | 1 MB | 8 MB | -| 50,000 | 5 MB | 34 MB | -| 100,000 | 10 MB | 134 MB | -| 500,000 | 50 MB | ~600 MB | +| 100 | 12 KB | 4 MB | +| 1,000 | 127 KB | 6 MB | +| 10,000 | 1.3 MB | 34 MB | +| 50,000 | 6.5 MB | 124 MB | +| 100,000 | 13 MB | 238 MB | +| 500,000 | 66 MB | ~1.2 GB | + +> Peak memory of the benchmark process, including the generated test data and full result sets +> (`find(all)` of 500K records); a single key lookup needs only one shard's index. --- ## SleekDB Comparison -### Why Choose noneDB? +`tests/sleekdb_comparison.php` with SleekDB 2.15. -noneDB v3.0+ excels in **bulk operations** and **large datasets**: +### Why Choose noneDB? | Strength | Performance | |----------|-------------| -| **Bulk Insert** | **8-10x faster** than SleekDB | -| **Find All** | **8-66x faster** at scale | -| **Filter Queries** | **20-80x faster** at scale | -| **Update Operations** | **15-40x faster** on large datasets | -| **Delete Operations** | **5-23x faster** on large datasets | -| **Count Operations** | **90-330x faster** (O(1) index lookup) | -| **Complex Queries** | **22-70x faster** at scale | -| **Large Datasets** | Handles 500K+ records with auto-sharding | -| **Thread Safety** | Atomic file locking for concurrent access | -| **Static Cache** | Cross-instance cache sharing | +| **Bulk Insert** | **12-23x faster** than SleekDB | +| **Find All** | **10-75x faster** | +| **Filter Queries** | **11-53x faster** | +| **Update Operations** | **9-48x faster** | +| **Delete Operations** | **3.5-21x faster** | +| **Count Operations** | **7-450x faster** (O(1) index lookup) | +| **Complex Queries** | **7-18x faster** | +| **Large Datasets** | Auto-sharding, 500K+ records | +| **Concurrency** | Database-level locks across processes, crash recovery | | **Spatial Queries** | R-tree indexing (SleekDB: none) | -**Best for:** Bulk operations, analytics, batch processing, filter-heavy workloads, count operations, geospatial queries - ### When to Consider SleekDB? | Scenario | SleekDB Advantage | |----------|-------------------| -| **High-frequency key lookups** | <1ms vs ~500ms cold (file-per-record architecture) | +| **First key lookup of a process** | ≤2 ms vs up to 15 ms at 100K (noneDB loads one shard index first; later lookups take ~0.05 ms) | | **Very low memory** | Lower RAM usage | -> **Note:** SleekDB stores each record as a separate file, making single-record lookups instant but bulk operations slow. - ---- +> SleekDB stores each record as a separate file, so a single-record lookup is one file read while bulk +> operations have to open every file. ### Architectural Differences @@ -241,131 +288,63 @@ noneDB v3.0+ excels in **bulk operations** and **large datasets**: |---------|---------|--------| | **Storage** | One JSON file per record | JSONL + byte-offset index | | **ID Access** | Direct file read (O(1)) | Index lookup + seek | -| **Bulk Read** | Traverse all files | Single file read | +| **Bulk Read** | Traverse all files | Single file read per shard | | **Sharding** | None | Automatic (10K+) | -| **Cache** | Per-query | Static cross-instance | -| **Indexing** | None | Byte-offset (.jidx) | -| **Spatial Index** | None | R-tree (v3.1) | - ---- +| **Cache** | Per-query | Static cross-instance, validated per write | +| **Indexing** | None | Byte-offset (.jidx), field indexes | +| **Spatial Index** | None | R-tree | ### Detailed Benchmark Results -#### Bulk Insert - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | 7ms | 24ms | **noneDB 3x** | -| 1K | 26ms | 250ms | **noneDB 10x** | -| 10K | 306ms | 2.89s | **noneDB 9x** | -| 50K | 1.59s | 12.4s | **noneDB 8x** | -| 100K | 3.34s | 30.76s | **noneDB 9x** | - -#### Find All Records - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | 3ms | 28ms | **noneDB 8x** | -| 1K | 7ms | 286ms | **noneDB 42x** | -| 10K | 65ms | 2.71s | **noneDB 42x** | -| 50K | 300ms | 16.83s | **noneDB 56x** | -| 100K | 595ms | 39.03s | **noneDB 66x** | - -#### Find by Key (Single Record - Cold) - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | 3ms | <1ms | SleekDB | -| 1K | 3ms | <1ms | SleekDB | -| 10K | 55ms | <1ms | **SleekDB** | -| 50K | 287ms | <1ms | **SleekDB** | -| 100K | 561ms | <1ms | **SleekDB** | - -> **Note:** SleekDB's file-per-record design gives O(1) key lookup. noneDB must load shard index first (but subsequent lookups are O(1) with cache). - -#### Find with Filter - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | <1ms | 10ms | **noneDB 24x** | -| 1K | 4ms | 94ms | **noneDB 25x** | -| 10K | 49ms | 998ms | **noneDB 20x** | -| 50K | 254ms | 13.18s | **noneDB 52x** | -| 100K | 524ms | 41.64s | **noneDB 79x** | - -#### Count Operations - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | <1ms | <1ms | **noneDB 4x** | -| 1K | <1ms | 1ms | **noneDB 11x** | -| 10K | <1ms | 9ms | **noneDB 90x** | -| 50K | <1ms | 51ms | **noneDB 330x** | -| 100K | <1ms | 96ms | **noneDB 258x** | - -> **v3.0 Optimization:** noneDB uses O(1) index metadata lookup for count() - no record scanning! - -#### Update Operations - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | 1ms | 20ms | **noneDB 15x** | -| 1K | 11ms | 188ms | **noneDB 17x** | -| 10K | 118ms | 2.14s | **noneDB 18x** | -| 50K | 669ms | 20.91s | **noneDB 31x** | -| 100K | 1.53s | 61.27s | **noneDB 40x** | - -#### Delete Operations - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | 2ms | 10ms | **noneDB 5x** | -| 1K | 15ms | 105ms | **noneDB 7x** | -| 10K | 150ms | 1.27s | **noneDB 8x** | -| 50K | 839ms | 14.61s | **noneDB 17x** | -| 100K | 1.75s | 40.01s | **noneDB 23x** | - -#### Complex Query (where + sort + limit) - -| Records | noneDB | SleekDB | Winner | -|---------|--------|---------|--------| -| 100 | <1ms | 12ms | **noneDB 27x** | -| 1K | 4ms | 114ms | **noneDB 30x** | -| 10K | 55ms | 1.2s | **noneDB 22x** | -| 50K | 295ms | 15.33s | **noneDB 52x** | -| 100K | 591ms | 41.3s | **noneDB 70x** | - ---- - -### Summary - -| Use Case | Winner | Advantage | -|----------|--------|-----------| -| **Bulk Insert** | **noneDB** | 3-10x faster | -| **Find All** | **noneDB** | 8-66x faster | -| **Find with Filter** | **noneDB** | 20-79x faster | -| **Update** | **noneDB** | 15-40x faster | -| **Delete** | **noneDB** | 5-23x faster | -| **Complex Query** | **noneDB** | 22-70x faster | -| **Count** | **noneDB** | 4-330x faster (O(1) index lookup) | -| **Spatial Queries** | **noneDB** | R-tree indexing (SleekDB: none) | -| **Find by Key (cold)** | **SleekDB** | O(1) file access | - -> **Choose noneDB** for: Bulk operations, large datasets, filter queries, update/delete workloads, complex queries, count operations, geospatial queries -> -> **Choose SleekDB** for: High-frequency single-record lookups by ID (cold cache scenarios) +| Operation | 100 noneDB | 100 SleekDB | 1K noneDB | 1K SleekDB | 10K noneDB | 10K SleekDB | 50K noneDB | 50K SleekDB | 100K noneDB | 100K SleekDB | +|-----------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----| +| Bulk Insert | 2 ms | 21 ms | 7 ms | 175 ms | 120 ms | 1.97 s | 714 ms | 11.94 s | 1.25 s | 24.35 s | +| Find All | <1 ms | 5 ms | 3 ms | 37 ms | 32 ms | 419 ms | 181 ms | 2.28 s | 372 ms | 27.98 s | +| Find by Key (cold) | <1 ms | <1 ms | <1 ms | <1 ms | 3 ms | <1 ms | 4 ms | <1 ms | 15 ms | 2 ms | +| Find with Filter | <1 ms | 4 ms | 3 ms | 47 ms | 32 ms | 411 ms | 174 ms | 2.41 s | 697 ms | 36.73 s | +| Count | <1 ms | <1 ms | <1 ms | <1 ms | <1 ms | 4 ms | <1 ms | 17 ms | <1 ms | 34 ms | +| Update | <1 ms | 7 ms | 5 ms | 73 ms | 41 ms | 974 ms | 229 ms | 4.89 s | 444 ms | 21.47 s | +| Delete | 2 ms | 6 ms | 9 ms | 49 ms | 81 ms | 745 ms | 434 ms | 3.65 s | 937 ms | 19.72 s | +| Complex Query | <1 ms | 5 ms | 4 ms | 39 ms | 47 ms | 464 ms | 173 ms | 2.33 s | 372 ms | 6.52 s | + +### Performance Ratio (noneDB vs SleekDB) + +| Operation | 100 | 1K | 10K | 50K | 100K | +|-----------|------|------|------|------|------| +| Bulk Insert | **11.6x** | **23.4x** | **16.4x** | **16.7x** | **19.5x** | +| Find All | **10x** | **10.6x** | **13.2x** | **12.6x** | **75.2x** | +| Find by Key (cold) | 0.27x | 0.09x | 0.01x | 0.01x | 0.12x | +| Find with Filter | **11.1x** | **13.4x** | **12.9x** | **13.9x** | **52.7x** | +| Count | **7.2x** | **10.7x** | **64.5x** | **283.4x** | **447.9x** | +| Update | **9.3x** | **14.9x** | **23.6x** | **21.3x** | **48.3x** | +| Delete | **3.5x** | **5.4x** | **9.2x** | **8.4x** | **21x** | +| Complex Query | **6.8x** | **11.1x** | **9.9x** | **13.5x** | **17.5x** | + +> Find by Key (cold) is the first lookup after the data was written and the caches were cleared. Warm +> lookups take ~0.05 ms (see [Key Lookup](#key-lookup)). --- ## Running Benchmarks +Every script uses its own temporary directory and key; none of them touches your databases. + ```bash -# Performance benchmark +# Performance benchmark (default sizes 100 ... 500K; pass your own as a comma-separated list) php tests/performance_benchmark.php +php tests/performance_benchmark.php 100,1000,10000 -# SleekDB comparison (requires SleekDB installed) -php tests/sleekdb_comparison.php +# Single-operation latency; optionally compare another version of noneDB.php +php tests/operation_benchmark.php +php tests/operation_benchmark.php /path/to/other/noneDB.php 5 # best of 5 + +# Comparison operators + spatial combinations +php tests/operator_benchmark.php # Spatial benchmark php tests/spatial_benchmark.php + +# SleekDB comparison (requires the dev dependency rakibtg/sleekdb) +php tests/sleekdb_comparison.php +php tests/sleekdb_comparison.php 100,1000 ``` diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 54c5f40..7900842 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -2,7 +2,7 @@ Complete configuration options and security best practices. -**Version:** 3.1.0 +**Version:** 3.2.0 --- @@ -55,7 +55,7 @@ $db = new noneDB(); ```php $db = new noneDB([ - 'secretKey' => 'your_secure_key', + 'secretKey' => getenv('NONEDB_SECRET'), // your own random key (example placeholders are rejected) 'dbDir' => '/path/to/db/', 'autoCreateDB' => true, 'shardingEnabled' => true, @@ -65,15 +65,37 @@ $db = new noneDB([ ### 3. Mixed Configuration -Config file values can be overridden: +A config array is merged over the `.nonedb` file: values in the array win, everything else comes from the +file. When the array contains both `secretKey` and `dbDir`, the file is optional. ```php -// Base config from .nonedb, override dbDir +// secretKey (and the other settings) from .nonedb, dbDir overridden $db = new noneDB([ 'dbDir' => '/custom/path/' ]); ``` +### Where `.nonedb` is looked for + +1. The directory of the running script (`$_SERVER['SCRIPT_FILENAME']`) +2. The directory of `noneDB.php` +3. The Composer project root (when noneDB is installed under `vendor/`) +4. The current working directory + +The first file found is used for the whole process. Keep it **outside the web root**: it contains the +secret key. + +### Relative `dbDir` + +- In `.nonedb`, a relative `dbDir` (`./db/`, `../data/` or `db/`) is relative to **the `.nonedb` file's + directory**. Web pages (`public/index.php`) and CLI or cron scripts (`bin/job.php`) therefore use the + same databases. +- In a config array, a relative `dbDir` is relative to the entry script's directory. Absolute paths are + recommended. +- Before 3.2, `./db/` in the file was relative to the running script and `db/` to the working directory, + which split data between directories. If such an old directory exists and the new one doesn't, it is + still used and an `E_USER_DEPRECATED` notice asks you to move it. + --- ## Configuration Options @@ -82,9 +104,9 @@ $db = new noneDB([ | Option | Type | Default | Description | |--------|------|---------|-------------| -| `secretKey` | string | *required* | Secret key for hashing database names. **Must be unique and secure.** | -| `dbDir` | string | `./db/` | Directory for database files. Created automatically if doesn't exist. | -| `autoCreateDB` | bool | `true` | Auto-create databases on first use. Set `false` in production. | +| `secretKey` | string | *required* | Secret used to derive the database file names. Must be a non-empty string and not an example placeholder, otherwise the constructor throws `InvalidArgumentException`. Use 32+ random characters. | +| `dbDir` | string | *required* (dev mode: `db/` in the project root) | Directory for database files. Created automatically (with an `.htaccess` that denies web access and an empty `index.html`). See *Relative `dbDir`* above. | +| `autoCreateDB` | bool | `true` | Let writes (`insert`, `createFieldIndex`, ...) create missing databases. Reads never create anything. Set `false` in production. | ### Sharding Settings @@ -98,32 +120,54 @@ $db = new noneDB([ | Option | Type | Default | Description | |--------|------|---------|-------------| -| `autoCompactThreshold` | float | `0.3` | Trigger compaction when deleted > 30% of records. Range: 0.1-0.9. | +| `autoCompactThreshold` | float | `0.3` | Compact a data file when deleted/superseded records exceed this share of its records. Must be > 0. | ### Lock Settings | Option | Type | Default | Description | |--------|------|---------|-------------| -| `lockTimeout` | int | `5` | File lock timeout in seconds. | -| `lockRetryDelay` | int | `10000` | Lock retry delay in microseconds (10ms default). | +| `lockTimeout` | int | `5` | Seconds to wait for a database lock before throwing `RuntimeException`. `0` waits forever. | +| `lockRetryDelay` | int | `10000` | Microseconds between lock attempts (10ms default). Must be > 0. | ### Field Indexing (v3.0+) | Option | Type | Default | Description | |--------|------|---------|-------------| -| `fieldIndexEnabled` | bool | `true` | Enable field indexing for faster filtered queries. | +| `fieldIndexEnabled` | bool | `true` | Use field indexes in queries. Indexes are always kept up to date, so turning this back on is safe. | +| `spatialIndexEnabled` | bool | `true` | Allow creating and using spatial indexes. Existing ones are always kept up to date. | + +### Write Buffer + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `bufferEnabled` | bool | `false` | Collect inserts in a buffer file and add them in batches. Off by default since 3.2: direct inserts are fast and immediately visible. Buffered records are flushed before every read and write. | --- ## Security Best Practices -### 1. Secret Key Security +### 1. What the Secret Key Does (and doesn't) + +- Database files are named `{hash}-{name}.nonedb`, where `hash` = PBKDF2(name, secretKey). Without the + key the file names can't be guessed, e.g. by someone probing a web-accessible directory. +- The key does **not** encrypt anything: records are stored as plain JSON lines. Protect the directory + (outside the web root, file permissions) and the `.nonedb` file. +- A wrong key simply doesn't see the databases of another key. +- Anyone who can list the directory sees `{hash}-{name}` and can test guesses for the key offline + (PBKDF2-SHA256, 1000 rounds). A short or guessable key can be found this way: use 32+ random characters. +- To change the key of existing databases (for example away from the old example placeholder): + +```php +noneDB::migrateKey('/var/data/nonedb', $oldKey, $newKey); // renames every file of every database +``` + +### 2. Secret Key Handling ```php // BAD: Hardcoded secret $db = new noneDB(['secretKey' => 'mysecret']); -// GOOD: Environment variable +// GOOD: Environment variable (if it isn't set, getenv() returns false and the constructor throws) $db = new noneDB(['secretKey' => getenv('NONEDB_SECRET')]); // GOOD: Config file (not in git) @@ -140,7 +184,7 @@ $secretKey = bin2hex(random_bytes(32)); // openssl rand -hex 32 ``` -### 2. Protect Config File +### 3. Protect Config File Add to `.gitignore`: @@ -151,9 +195,10 @@ db/ *.nonedb.jidx ``` -### 3. Protect Database Directory +### 4. Protect Database Directory -Create `db/.htaccess`: +noneDB writes this `.htaccess` (and an empty `index.html`) when it creates `dbDir`. For an existing +directory, create `db/.htaccess`: ```apache # Deny all access @@ -174,7 +219,7 @@ location ~ \.nonedb { } ``` -### 4. Disable Auto-Create in Production +### 5. Disable Auto-Create in Production ```json { @@ -182,9 +227,10 @@ location ~ \.nonedb { } ``` -This prevents accidental database creation from typos or injection. +This prevents accidental database creation from typos or injection. Database names are validated +(`A-Z a-z 0-9`, space, `'`, `-`); anything else throws `InvalidArgumentException`. -### 5. Database Directory Location +### 6. Database Directory Location ```php // BAD: Inside web root @@ -225,8 +271,8 @@ When dev mode is enabled without config: | Option | Dev Mode Default | |--------|------------------| -| secretKey | `"development_secret_key_change_in_production"` | -| dbDir | `./db/` | +| secretKey | `"nonedb_dev_mode_key_" . md5()` — predictable by design | +| dbDir | `db/` in the project root (next to `vendor/` when installed with Composer, otherwise next to `noneDB.php`) | | autoCreateDB | `true` | | shardingEnabled | `true` | @@ -240,7 +286,8 @@ if (noneDB::isDevMode()) { ### Warning -Dev mode should **never** be used in production. It uses a known default secret key. +Dev mode should **never** be used in production: its key is predictable. Outside the CLI, noneDB raises +an `E_USER_NOTICE` once per process while dev mode is active. --- @@ -332,6 +379,13 @@ $db = new noneDB([ ]); ``` +Each database has a lock file `{hash}-{name}.nonedb.lock`. Besides the lock it holds a small state +record (change counter, pending write buffer, index journal) that every operation reads with the lock, +so cache validation and crash recovery need no extra file access. Lock files stay when a database is +dropped and their handles stay open between operations (one per database, at most 64 per process). +Don't delete them while processes use the databases; deleting them while nothing runs is harmless (the +one-time check for pre-3.2 leftovers then runs again). + ### Static Cache ```php @@ -431,6 +485,8 @@ $db->getShardSize(); // 10000 } ``` +Replace the placeholder key: noneDB refuses to start with it. + --- ## Troubleshooting @@ -453,19 +509,29 @@ chmod 755 db/ chmod 644 .nonedb ``` -### "Lock timeout" +### "timed out after Ns waiting for the lock" + +Another process held the database lock longer than `lockTimeout` (a long compaction or migration). ```php -// Increase timeout -$db = new noneDB(['lockTimeout' => 30]); +$db = new noneDB(['lockTimeout' => 30]); // or 0 to wait forever ``` -### "Database not accessible after upgrade" +### "'secretKey' is still the example placeholder" / "must be a non-empty string" -The most common cause is changed secretKey. Ensure you're using the same secretKey as before. +Generate a key (`php -r "echo bin2hex(random_bytes(32));"`). If databases already exist under the +placeholder key, move them first with `noneDB::migrateKey($dbDir, $oldKey, $newKey)`. -```php -// Database files are named: {hash}-{dbname}.nonedb -// The hash is generated from secretKey + dbname using PBKDF2 -// If you changed your secretKey, old database files won't be found -``` +### "invalid database name" + +Only `A-Z a-z 0-9`, space, `'` and `-` are allowed. Before 3.2 other characters were removed silently, so +data written as `"my_db"` is stored under the name `"mydb"`. + +### "Database not accessible after upgrade" + +- Changed secretKey: database files are named `{hash}-{name}.nonedb` with a hash derived from the key. + Use `noneDB::migrateKey()` to change keys instead of editing the config only. +- Relative `dbDir`: it now follows the `.nonedb` file. Look for an `E_USER_DEPRECATED` notice naming the + old directory, then move the files or use an absolute path. +- Went back to 3.1 and then returned to 3.2: 3.1 doesn't maintain the lock files' state. Stop all + processes, delete the `*.nonedb.lock` files and start again. diff --git a/docs/QUERY.md b/docs/QUERY.md index 187a1e5..0040049 100644 --- a/docs/QUERY.md +++ b/docs/QUERY.md @@ -2,7 +2,7 @@ Comprehensive documentation for noneDB's Query Builder API. -**Version:** 3.1.0 +**Version:** 3.2.0 --- @@ -95,9 +95,23 @@ noneDB supports MongoDB-style comparison operators in `where()` clauses. | `$in` | Value in array | `['category' => ['$in' => ['a', 'b']]]` | | `$nin` | Value not in array | `['tag' => ['$nin' => ['spam']]]` | | `$exists` | Field exists/not exists | `['email' => ['$exists' => true]]` | -| `$like` | Pattern matching | `['name' => ['$like' => 'John']]` | -| `$regex` | Regular expression | `['email' => ['$regex' => '@gmail.com$']]` | -| `$contains` | Array/string contains | `['tags' => ['$contains' => 'featured']]` | +| `$like` | Pattern matching (literal; only a leading `^` / trailing `$` are special) | `['name' => ['$like' => '^John']]` | +| `$regex` | Regular expression (case-insensitive, no delimiters) | `['email' => ['$regex' => '@gmail\\.com$']]` | +| `$contains` | Array element (strict) / substring | `['tags' => ['$contains' => 'featured']]` | + +**How values compare** + +- `$gt`, `$gte`, `$lt`, `$lte` (and `between()` / `notBetween()`): numbers and numeric strings compare + numerically (`"30" > 18`), other strings lexically (`'2024-02-01' > '2024-01-15'`). `null`, booleans, + arrays and a number against a non-numeric string never match. A missing field never matches. +- `$eq`, `$ne`, `$in`, `$nin`, `whereIn()`, `whereNot()` and plain `['field' => value]` filters use strict + comparison (`===`): `'25'` is not `25`. +- A filter value that is an array of operators applies all of them; any other array is compared as a + whole (`['addr' => ['city' => 'Ist']]` matches only that exact array). +- A mix of operators and plain keys in one value throws `InvalidArgumentException` as soon as `where()` + / `orWhere()` is called. An unknown operator (`$gtee`), `$in`/`$nin` without an array or an invalid + `$regex` throws it when the query runs, also on an empty database (filters are compiled once per + query, before records are read). ### Comparison Examples @@ -194,9 +208,9 @@ $featuredProducts = $db->query("products") ->where(['tags' => ['$contains' => 'featured']]) ->get(); -// Check if string field contains substring +// Check if string field contains substring (case-sensitive; use $like for case-insensitive matching) $techNews = $db->query("articles") - ->where(['title' => ['$contains' => 'technology']]) + ->where(['title' => ['$contains' => 'Technology']]) ->get(); ``` @@ -228,7 +242,12 @@ $activeAdmins = $db->query("users") ### where() -Primary filter method. Applies AND logic across all conditions. +Primary filter method. Applies AND logic across all conditions. Each `where()` call adds another group +that must also match, so conditions on the same field can be split over several calls: + +```php +$db->query("users")->where(['age' => ['$gte' => 18]])->where(['age' => ['$lt' => 65]]); +``` ```php // Simple equality @@ -243,7 +262,8 @@ $db->query("users")->where([ ### orWhere() -Adds OR conditions to the query. +Adds OR groups: a record matches if **all** `where()` groups match **or any** `orWhere()` group matches. +Without `where()`, only the `orWhere()` groups decide. ```php // Users who are admins OR have high reputation @@ -278,7 +298,8 @@ $db->query("products") ### whereNot() -Exclude records matching conditions. +Exclude records where **any** of the given fields equals its value (`===`): +`whereNot(['role' => 'guest', 'dept' => 'hr'])` drops guests and everyone in HR. Several calls all apply. ```php // All users except guests @@ -298,9 +319,12 @@ $db->query("users")->like('name', '^A')->get(); // Names ending with 'son' $db->query("users")->like('name', 'son$')->get(); -// Names containing 'john' (case-insensitive) +// Names containing 'john' (case-insensitive, UTF-8 aware) $db->query("users")->like('name', 'john')->get(); +// Everything except a leading ^ and a trailing $ is literal: this matches 'http://...' only +$db->query("links")->like('url', '^http://')->get(); + // Exclude spam titles $db->query("posts")->notLike('title', 'SPAM')->get(); ``` @@ -321,6 +345,10 @@ $db->query("users") ->get(); ``` +`between()` matches only comparable values inside the range (see the comparison rules above). +`notBetween()` returns every other record, including records without the field or with a value that +can't be compared (`null`, booleans, text against numbers). + ### search() Full-text search across fields. @@ -337,6 +365,9 @@ $db->query("articles") ->get(); ``` +Case-insensitive substring match on scalar values (`İzmir`, `IZMIR` and `izmir` match each other). +Without a field list, every field except `key` is searched. + --- ## Sorting & Pagination @@ -353,14 +384,20 @@ $db->query("users")->sort('name', 'asc')->get(); // Descending $db->query("products")->sort('price', 'desc')->get(); -// Sort by distance (spatial queries) +// Several keys: category first, then price +$db->query("products")->sort('category')->sort('price', 'desc')->get(); + +// Sort by distance (spatial queries; distances are in meters) $db->query("locations") - ->withinDistance('coords', $lon, $lat, 10) + ->withinDistance('coords', $lon, $lat, 10000) ->withDistance('coords', $lon, $lat) ->sort('_distance', 'asc') ->get(); ``` +Sorting is stable and uses a total order: missing/`null` first, then booleans, numbers, strings and +arrays (reversed for `desc`). + ### limit() / offset() Paginate results. @@ -431,18 +468,25 @@ $ordersWithUsers = $db->query("orders") ->join("users", "user_id", "key") // orders.user_id = users.key ->get(); -// Result includes user data in _joined field +// The matching user record is attached under the foreign database's name (or null if none matches) foreach ($ordersWithUsers as $order) { - echo $order['_joined']['name']; // User's name + echo $order['users']['name'] ?? '-'; // User's name } + +// Custom field name (alias) +$db->query("orders")->join("users", "user_id", "key", "buyer")->get(); // $order['buyer'] ``` +- The attached record replaces a field of the same name in the result only; stored records are never changed. +- If several foreign records have the same `$foreignKey` value, the last one wins. +- Joining a database that doesn't exist attaches `null` and creates nothing. + ### Multiple Joins ```php $data = $db->query("order_items") - ->join("orders", "order_id", "key") - ->join("products", "product_id", "key") + ->join("orders", "order_id", "key") // $item['orders'] + ->join("products", "product_id", "key") // $item['products'] ->get(); ``` @@ -460,13 +504,17 @@ $ordersByStatus = $db->query("orders") ->groupBy('status') ->get(); -// Returns: [ -// 'pending' => [...orders], -// 'completed' => [...orders], -// 'cancelled' => [...orders] +// Returns a list of groups: +// [ +// ['_group' => 'pending', '_items' => [...orders], '_count' => 12], +// ['_group' => 'completed', '_items' => [...orders], '_count' => 40], +// ... // ] ``` +Group values keep their type (`1`, `"1"` and `1.5` are separate groups); records without the field +are grouped under `null`. + ### having() Filter groups by aggregate conditions. @@ -481,10 +529,14 @@ $frequentCustomers = $db->query("orders") // Categories with average rating >= 4 $goodCategories = $db->query("products") ->groupBy('category') - ->having('avg', '>=', 4, 'rating') + ->having('avg:rating', '>=', 4) ->get(); ``` +Aggregates: `count`, `sum:field`, `avg:field`, `min:field`, `max:field`; operators: `>`, `<`, `>=`, `<=`, +`=`, `!=`. `having()` only applies to groups: without `groupBy()`, or with an unknown aggregate or +operator, the condition is ignored. + --- ## Spatial Queries @@ -494,7 +546,7 @@ noneDB supports geospatial queries with R-tree indexing. ### Creating Spatial Index ```php -// Create index on location field (required before spatial queries) +// Create index on location field (optional: spatial queries work without it, the index makes them fast) $db->createSpatialIndex("restaurants", "location"); // Check if index exists @@ -548,19 +600,20 @@ Find records within radius of a point. $nearby = $db->withinDistance("restaurants", "location", 28.9784, // longitude 41.0082, // latitude - 5 // radius in km + 5000 // radius in meters ); // Query builder $nearby = $db->query("restaurants") - ->withinDistance('location', 28.9784, 41.0082, 5) + ->withinDistance('location', 28.9784, 41.0082, 5000) // 5 km ->where(['open_now' => true]) ->get(); ``` ### withinBBox() -Find records within bounding box. +Find records whose geometry's bounding box overlaps the box (a point inside it, or a line/polygon that +reaches into or touches it). ```php // Direct method @@ -587,7 +640,7 @@ $closest = $db->nearest("restaurants", "location", 5 // number of results ); -// Query builder with distance +// Query builder: the 10 nearest records among those rated 4+ (k is applied after the other filters) $closest = $db->query("restaurants") ->nearest('location', 28.9784, 41.0082, 10) ->where(['rating' => ['$gte' => 4.0]]) @@ -625,17 +678,22 @@ $inPolygon = $db->query("restaurants") ### withDistance() -Add calculated distance to results. +Add `_distance` to every result: meters from the point to the geometry, measured like +`withinDistance()` / `nearest()` (0 for a point inside a polygon, otherwise to the geometry's nearest +point), rounded to 4 decimals. It is computed before sorting and kept by `select()`. + +In the query builder, `withinDistance()` and `nearest()` already add `_distance` (to their own center, +rounded to 2 decimals); `withDistance()` replaces it with the distance to its point. ```php $results = $db->query("restaurants") - ->withinDistance('location', $userLon, $userLat, 10) + ->withinDistance('location', $userLon, $userLat, 10000) // 10 km ->withDistance('location', $userLon, $userLat) ->sort('_distance', 'asc') ->get(); foreach ($results as $r) { - echo "{$r['name']}: {$r['_distance']} km away\n"; + echo "{$r['name']}: {$r['_distance']} m away\n"; } ``` @@ -644,7 +702,7 @@ foreach ($results as $r) { ```php // Food delivery app: find nearby open restaurants with delivery $restaurants = $db->query("restaurants") - ->withinDistance('location', $userLon, $userLat, 5) + ->withinDistance('location', $userLon, $userLat, 5000) ->where([ 'open_now' => true, 'delivery' => true, @@ -704,9 +762,13 @@ Updates matching records and returns result. $result = $db->query("users") ->where(['status' => 'inactive']) ->update(['status' => 'archived']); -// Returns: ['n' => 5, 'keys' => [1, 3, 7, 12, 15]] +// Returns: ['n' => 5] ``` +`sort()`, `offset()` and `limit()` choose which records are written (e.g. "the 10 oldest"); `select()`, +`except()` and `join()` don't matter. Selecting and writing happen under one database lock. +`update()`/`delete()` can't be combined with `groupBy()` (`LogicException`). + ### delete() Deletes matching records and returns result. @@ -715,7 +777,17 @@ Deletes matching records and returns result. $result = $db->query("logs") ->where(['created_at' => ['$lt' => strtotime('-30 days')]]) ->delete(); -// Returns: ['n' => 100, 'keys' => [...]] +// Returns: ['n' => 100] +``` + +### removeFields() + +Removes fields from the **stored** matching records (never affected by `select()`, `except()`, +`join()` or `withDistance()`). + +```php +$db->query("users")->where(['legacy' => true])->removeFields(['old_password', 'tmp']); +// Returns: ['n' => 12, 'fields_removed' => ['old_password']] ``` --- @@ -775,7 +847,7 @@ foreach ($db->query("orders")->where(['status' => 'completed'])->get() as $order ```php // Find nearby services with filters $services = $db->query("services") - ->withinDistance('location', $userLon, $userLat, 10) + ->withinDistance('location', $userLon, $userLat, 10000) ->where([ 'available' => true, 'rating' => ['$gte' => 4.0], @@ -859,11 +931,11 @@ $db->query("orders") ### 5. Use Spatial Indexes for Location Queries ```php -// Always create spatial index before queries +// Create the spatial index once $db->createSpatialIndex("restaurants", "location"); -// Then queries are O(log n) instead of O(n) -$db->withinDistance("restaurants", "location", $lon, $lat, 5); +// Then queries use the R-tree instead of scanning every record +$db->withinDistance("restaurants", "location", $lon, $lat, 5000); // 5 km (meters) ``` ### 6. Batch Operations @@ -902,7 +974,7 @@ $db->query("orders") ### Spatial Methods ```php -->withinDistance($field, $lon, $lat, $km) +->withinDistance($field, $lon, $lat, $meters) ->withinBBox($field, $minLon, $minLat, $maxLon, $maxLat) ->nearest($field, $lon, $lat, $k) ->withinPolygon($field, $polygon) @@ -958,6 +1030,7 @@ $db->query("orders") | Version | Changes | |---------|---------| +| 3.2.0 | `where()` calls combine with AND; `orWhere()` without `where()` fixed; type-aware comparisons; unknown operators throw; safe `like`/`$regex`; multi-key stable `sort()`; `nearest()` applies k after filters; `withDistance()` before sort and measured like `withinDistance()`; typed `groupBy()`; `removeFields()` on stored records only | | 3.1.0 | Added comparison operators ($gt, $gte, $lt, $lte, $ne, $eq, $in, $nin, $exists, $like, $regex, $contains) | | 3.1.0 | Added spatial indexing with R-tree (withinDistance, withinBBox, nearest, withinPolygon) | | 3.1.0 | Added withDistance() for distance calculations | diff --git a/docs/SPATIAL.md b/docs/SPATIAL.md index d877d98..f23f695 100644 --- a/docs/SPATIAL.md +++ b/docs/SPATIAL.md @@ -2,7 +2,7 @@ Comprehensive documentation for noneDB's geospatial capabilities using R-tree indexing. -**Version:** 3.1.0 +**Version:** 3.2.0 --- @@ -258,6 +258,10 @@ foreach ($nearby as $r) { } ``` +Distance is measured to the **nearest point of the geometry**: 0 for a point inside a polygon, the +closest vertex or edge otherwise (a Point is just that point). Results are sorted by distance. +Circles that cross the antimeridian (±180°) or reach a pole are searched correctly. + ### withinBBox() Find records within a bounding box. @@ -269,8 +273,13 @@ $inArea = $db->withinBBox( 28.97, 41.00, // minLon, minLat (SW corner) 29.00, 41.03 // maxLon, maxLat (NE corner) ); + +// A box crossing the antimeridian: minLon > maxLon +$pacific = $db->withinBBox("ships", "location", 170, -10, -170, 10); ``` +Returns geometries whose bounding box overlaps the box. + ### nearest() Find K nearest records to a point. @@ -292,11 +301,17 @@ echo "Nearest: " . $closest[0]['name']; ```php $closest = $db->nearest("restaurants", "location", 28.9784, 41.0082, 5, [ - 'includeDistance' => true, - 'maxDistance' => 10000 // Maximum distance in meters (10km) + 'includeDistance' => false, // default true for nearest(): each result has _distance in meters + 'maxDistance' => 10000 // Maximum distance in meters (10km) ]); ``` +`nearest()` adds `_distance` unless `'includeDistance' => false`; `withinDistance()` adds it only with +`'includeDistance' => true`. + +With `maxDistance`, the records within that distance are returned even when there are fewer than k. +`k = 0` returns `[]`; with fewer than k records in total, all of them are returned. + ### withinPolygon() Find records within a polygon boundary. @@ -316,6 +331,10 @@ $polygon = [ $inPolygon = $db->withinPolygon("restaurants", "location", $polygon); ``` +`$polygon` may be a Polygon (holes are excluded) or a MultiPolygon. A geometry matches when it lies +entirely inside: every vertex inside and no edge crossing the boundary (a line that leaves and re-enters +a concave polygon does not match). + --- ## Query Builder Integration @@ -341,8 +360,10 @@ $results = $db->query("restaurants") ### nearest() ```php +// The 10 nearest open restaurants: k is applied after the other filters $results = $db->query("restaurants") ->nearest('location', 28.9784, 41.0082, 10) + ->where(['open_now' => true]) ->get(); ``` @@ -365,9 +386,14 @@ $results = $db->query("restaurants") ->sort('_distance', 'asc') ->get(); -// Each result has _distance field in meters +// Each result has _distance in meters, computed before sort() ``` +`_distance` is measured like the spatial filters: 0 for a point inside a polygon, otherwise to the +geometry's nearest point (4 decimals). In the query builder `withinDistance()` and `nearest()` already +add `_distance` to their own center (2 decimals); `withDistance()` replaces it with the distance to its +point. + --- ## Combining Spatial with Filters @@ -704,12 +730,24 @@ if (!$validation['valid']) { | Error | Cause | Solution | |-------|-------|----------| -| "Spatial index already exists" | Duplicate createSpatialIndex | Check with hasSpatialIndex first | -| "Spatial index not found" | Query without index | Create index before querying | -| "Invalid GeoJSON" | Malformed geometry | Validate with validateGeoJSON() | -| "Ring must be closed" | Polygon not closed | Ensure first == last point | -| "Invalid longitude" | lon > 180 or < -180 | Use valid coordinates | -| "Invalid latitude" | lat > 90 or < -90 | Use valid coordinates | +| "Spatial index already exists for field 'x'" | Duplicate createSpatialIndex | Check with hasSpatialIndex first | +| "Missing type property", "Unknown geometry type: X", "Coordinates must be an array", ... | Malformed geometry | Validate with validateGeoJSON() | +| "Ring 0 must be closed (first and last positions must be identical)" | Polygon not closed | Ensure first == last point (compared numerically: `29` equals `29.0`) | +| "Ring 0 must have at least 4 positions" | Too few positions | A closed ring needs 4+ positions | +| "Ring 0 must not intersect itself" | Bow-tie / self-crossing ring | Fix the ring order | +| "Hole 1 must lie inside the outer ring" | Hole outside its shell | Fix the polygon | +| "Point coordinates must be finite numbers" | NaN, INF or strings | Use numbers | +| "Longitude must be between -180 and 180" | lon > 180 or < -180 | Use valid coordinates | +| "Latitude must be between -90 and 90" | lat > 90 or < -90 | Use valid coordinates | + +Queries work without a spatial index too (they scan the records); an index only makes them faster. +Records whose geometry is invalid are not indexed and never match spatial queries. + +### Sharded databases and field names + +Spatial indexes work the same on sharded databases (one R-tree per shard plus each shard's bounding +box for shard skipping), follow automatic migration to shards, and cover shards created later. +Field names may contain any characters (`home-loc`, `konum.ev`). --- @@ -717,6 +755,7 @@ if (!$validation['valid']) { | Version | Changes | |---------|---------| +| 3.2.0 | Sharded databases supported; buffered inserts indexed; antimeridian/pole searches; nearest() maxDistance returns partial results; distance to the nearest point of a geometry; MultiPolygon queries; edge-aware withinPolygon; stricter validation; crash-safe index maintenance | | 3.1.0 | Initial spatial indexing with R-tree | | 3.1.0 | GeoJSON validation | | 3.1.0 | withinDistance, withinBBox, nearest, withinPolygon | diff --git a/docs/index.html b/docs/index.html index ccf1832..adb6cf9 100644 --- a/docs/index.html +++ b/docs/index.html @@ -243,7 +243,7 @@
-
v3.1 - Now with Spatial Queries
+
v3.2 - Multi-process safe & crash-safe

The Zero-Dependency
Database for PHP

Not every project needs a database server.
For small to mid-scale applications, noneDB delivers the performance you need
without the infrastructure overhead.

@@ -280,7 +280,7 @@

Auto-Sharding

Automatically splits large datasets into shards for opt

-

Thread-Safe

Atomic file locking ensures safe concurrent access from multiple processes.

+

Multi-Process Safe

Database-level locking, atomic file replacement and self-healing indexes: no lost updates, even when a process is killed mid-write.

diff --git a/examples/_bootstrap.php b/examples/_bootstrap.php new file mode 100644 index 0000000..a318601 --- /dev/null +++ b/examples/_bootstrap.php @@ -0,0 +1,28 @@ + 'nonedb-examples-demo-key-not-for-production', + 'dbDir' => __DIR__ . '/db/', + ], $config)); +} + +/** + * Drop the given databases so every run starts from the same state + */ +function freshDatabases(noneDB $db, string ...$names): void +{ + foreach ($names as $name) { + $db->dropDB($name); + } +} diff --git a/examples/aggregation.php b/examples/aggregation.php index 85d9fa0..a8c9516 100644 --- a/examples/aggregation.php +++ b/examples/aggregation.php @@ -5,8 +5,9 @@ * Demonstrates count(), sum(), avg(), min(), and max() methods. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'employees', 'orders'); echo "=== noneDB Aggregation Methods Example ===\n\n"; diff --git a/examples/chaining.php b/examples/chaining.php index 511e126..c885b09 100644 --- a/examples/chaining.php +++ b/examples/chaining.php @@ -5,8 +5,9 @@ * Demonstrates the query builder pattern for fluent method chaining. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users'); echo "=== noneDB Method Chaining Example ===\n\n"; diff --git a/examples/database-management.php b/examples/database-management.php index 69a4b37..e69a7d2 100644 --- a/examples/database-management.php +++ b/examples/database-management.php @@ -5,8 +5,9 @@ * Demonstrates createDB(), checkDB(), and getDBs() methods. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users', 'orders', 'products', 'my-new-database', 'my-database', 'user\'s data', 'auto-created-db'); echo "=== noneDB Database Management Example ===\n\n"; @@ -20,12 +21,12 @@ // ----------------------------------------- echo "1. Create a New Database:\n"; -$result = $db->createDB("my_new_database"); -echo " createDB('my_new_database'): " . ($result ? "true (created)" : "false (exists)") . "\n"; +$result = $db->createDB("my-new-database"); +echo " createDB('my-new-database'): " . ($result ? "true (created)" : "false (exists)") . "\n"; // Try to create again -$result = $db->createDB("my_new_database"); -echo " createDB('my_new_database') again: " . ($result ? "true (created)" : "false (already exists)") . "\n"; +$result = $db->createDB("my-new-database"); +echo " createDB('my-new-database') again: " . ($result ? "true (created)" : "false (already exists)") . "\n"; // ----------------------------------------- // 2. Create Multiple Databases @@ -57,22 +58,25 @@ echo "\n4. Check Non-existing Database (Auto-create enabled):\n"; echo " Note: autoCreateDB is enabled by default.\n"; -$exists = $db->checkDB("auto_created_db"); -echo " checkDB('auto_created_db'): " . ($exists ? "true (auto-created)" : "false") . "\n"; +$exists = $db->checkDB("auto-created-db"); +echo " checkDB('auto-created-db'): " . ($exists ? "true (auto-created)" : "false") . "\n"; // ----------------------------------------- -// 5. Database Name Sanitization +// 5. Database Name Rules // ----------------------------------------- -echo "\n5. Database Name Sanitization:\n"; -echo " Only [A-Za-z0-9' -] allowed, others removed.\n\n"; +echo "\n5. Database Name Rules:\n"; +echo " Only A-Z a-z 0-9, space, apostrophe (') and hyphen (-) are allowed.\n"; +echo " Other characters are rejected with an InvalidArgumentException (since v3.2.0).\n\n"; -$db->createDB("test_db!"); // becomes "testdb" $db->createDB("my-database"); // hyphen is OK -$db->createDB("user's data"); // apostrophe is OK +$db->createDB("user's data"); // apostrophe and space are OK +echo " 'my-database' and \"user's data\" created\n"; -echo " 'test_db!' becomes 'testdb'\n"; -echo " 'my-database' stays 'my-database'\n"; -echo " \"user's data\" stays \"user's data\"\n"; +try { + $db->createDB("test_db!"); +} catch (InvalidArgumentException $e) { + echo " 'test_db!' rejected: " . $e->getMessage() . "\n"; +} // ========================================== // GET DATABASES @@ -132,7 +136,7 @@ // ----------------------------------------- echo "\n9. Check Non-existent Database Info:\n"; -$info = $db->getDBs("this_does_not_exist_xyz"); +$info = $db->getDBs("this-does-not-exist-xyz"); echo " Result: " . ($info === false ? "false (not found)" : json_encode($info)) . "\n"; // ========================================== @@ -177,6 +181,6 @@ echo "\n3. Security:\n"; echo " - Change \$secretKey before production!\n"; echo " - Protect /db directory with .htaccess\n"; -echo " - Validate/sanitize database names from user input\n"; +echo " - Validate database names from user input (invalid names throw InvalidArgumentException)\n"; echo "\n=== Database Management Examples Complete ===\n"; diff --git a/examples/delete.php b/examples/delete.php index eec10a1..08b4da9 100644 --- a/examples/delete.php +++ b/examples/delete.php @@ -5,8 +5,9 @@ * Demonstrates various ways to delete records. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users', 'logs', 'session', 'temp'); echo "=== noneDB Delete Example ===\n\n"; @@ -120,10 +121,9 @@ // 7. Deleted Records Become NULL // ----------------------------------------- echo "\n7. Technical Note - Deleted Records:\n"; -echo " - Deleted records become NULL internally\n"; -echo " - They are filtered from find() results\n"; -echo " - Use compact() to reclaim space\n"; -echo " - Keys may be reassigned after compact()\n"; +echo " - Deleted records disappear from find() immediately\n"; +echo " - Their bytes stay in the file until compaction (automatic, or compact())\n"; +echo " - Keys never change, also not after compact(); deleted keys are never reused\n"; // ----------------------------------------- // 8. Using compact() After Deletes diff --git a/examples/find.php b/examples/find.php index 6d095fe..c95adb1 100644 --- a/examples/find.php +++ b/examples/find.php @@ -5,8 +5,9 @@ * Demonstrates various ways to query and retrieve data. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users'); echo "=== noneDB Find Example ===\n\n"; diff --git a/examples/getdb-info.php b/examples/getdb-info.php index 0b866ee..578bd61 100644 --- a/examples/getdb-info.php +++ b/examples/getdb-info.php @@ -5,8 +5,9 @@ * Demonstrates getDBs() method for listing databases and getting info. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users', 'logs', 'products'); echo "=== noneDB Database Information Example ===\n\n"; diff --git a/examples/insert.php b/examples/insert.php index 62b2b0f..8c19d48 100644 --- a/examples/insert.php +++ b/examples/insert.php @@ -5,8 +5,9 @@ * Demonstrates single record insertion with various data types. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users', 'configs', 'products'); echo "=== noneDB Insert Example ===\n\n"; diff --git a/examples/multiple-insert.php b/examples/multiple-insert.php index 180a442..67c4213 100644 --- a/examples/multiple-insert.php +++ b/examples/multiple-insert.php @@ -5,8 +5,9 @@ * Demonstrates inserting multiple records at once. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'employees', 'orders', 'products'); echo "=== noneDB Multiple Insert Example ===\n\n"; diff --git a/examples/performance.php b/examples/performance.php index 488ef0c..651fe36 100644 --- a/examples/performance.php +++ b/examples/performance.php @@ -4,10 +4,10 @@ * Tests various operations with different dataset sizes */ -require_once __DIR__ . '/../noneDB.php'; +require __DIR__ . '/_bootstrap.php'; -$db = new noneDB(); -$testDb = 'perf_test_' . time(); +$db = exampleDb(); +$testDb = 'perf-test-' . time(); // database names: letters, digits, space, ' and - only function formatTime($ms) { if ($ms < 1) return '<1 ms'; @@ -29,21 +29,14 @@ function benchmark($callback) { return ['time' => $time, 'result' => $result]; } +// Data size of a database, all shards included (as reported by getDBs()) function getFileSize($db, $dbName) { - $reflection = new ReflectionClass($db); - $prop = $reflection->getProperty('dbDir'); - $prop->setAccessible(true); - $dbDir = $prop->getValue($db); - - $method = $reflection->getMethod('hashDBName'); - $method->setAccessible(true); - $hash = $method->invoke($db, $dbName); - - $file = $dbDir . $hash . '-' . $dbName . '.nonedb'; - return file_exists($file) ? filesize($file) : 0; + $info = $db->getDBs($dbName); + return is_array($info) ? $info['size'] : '-'; } -$sizes = [100, 1000, 10000, 50000, 100000]; +// Sizes to test: php performance.php 100,1000 (default: up to 100K records) +$sizes = isset($argv[1]) ? array_map('intval', explode(',', $argv[1])) : [100, 1000, 10000, 50000, 100000]; $results = []; echo "=== noneDB Performance Benchmark ===\n\n"; @@ -55,7 +48,7 @@ function getFileSize($db, $dbName) { echo "--- Testing with " . number_format($size) . " records ---\n"; $results[$size] = []; - $dbName = $testDb . '_' . $size; + $dbName = $testDb . '-' . $size; // Generate test data $data = []; @@ -252,12 +245,12 @@ function getFileSize($db, $dbName) { // Memory $results[$size]['memory'] = memory_get_peak_usage(true); echo "Peak Memory: " . formatMemory($results[$size]['memory']) . "\n"; - echo "File Size: " . formatMemory($results[$size]['filesize']) . "\n"; + echo "File Size: " . $results[$size]['filesize'] . "\n"; echo "\n"; // Cleanup this db - $db->delete($dbName, []); + $db->dropDB($dbName); gc_collect_cycles(); } @@ -315,7 +308,7 @@ function getFileSize($db, $dbName) { echo "|---------|-----------|-------------|\n"; foreach ($sizes as $size) { $sizeLabel = $size >= 1000 ? number_format($size) : $size; - echo "| {$sizeLabel} | " . formatMemory($results[$size]['filesize']) . " | " . formatMemory($results[$size]['memory']) . " |\n"; + echo "| {$sizeLabel} | " . $results[$size]['filesize'] . " | " . formatMemory($results[$size]['memory']) . " |\n"; } echo "\nBenchmark complete!\n"; diff --git a/examples/query-methods.php b/examples/query-methods.php index c474d04..c6f0674 100644 --- a/examples/query-methods.php +++ b/examples/query-methods.php @@ -5,8 +5,9 @@ * Demonstrates distinct(), like(), and between() methods. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users'); echo "=== noneDB Query Methods Example ===\n\n"; diff --git a/examples/sharding.php b/examples/sharding.php index 8ff6d4a..90c9af2 100644 --- a/examples/sharding.php +++ b/examples/sharding.php @@ -6,8 +6,9 @@ * and sharding behavior with large datasets. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'small-db', 'medium-db', 'with-deletions'); echo "=== noneDB Sharding Example ===\n\n"; @@ -27,14 +28,14 @@ echo "\n=== SMALL DATABASE (NOT SHARDED) ===\n\n"; // Create a small database -$db->insert("small_db", [ +$db->insert("small-db", [ ["id" => 1, "name" => "Item 1"], ["id" => 2, "name" => "Item 2"], ["id" => 3, "name" => "Item 3"], ]); -$info = $db->getShardInfo("small_db"); -echo "small_db info:\n"; +$info = $db->getShardInfo("small-db"); +echo "small-db info:\n"; echo " Sharded: " . ($info['sharded'] ? "true" : "false") . "\n"; echo " Total records: " . $info['totalRecords'] . "\n"; echo " Shards: " . $info['shards'] . "\n"; @@ -46,7 +47,7 @@ // For non-sharded database echo "1. Non-sharded database info:\n"; -$info = $db->getShardInfo("small_db"); +$info = $db->getShardInfo("small-db"); if ($info) { echo " " . json_encode($info, JSON_PRETTY_PRINT) . "\n"; } @@ -66,15 +67,15 @@ echo " " . json_encode($result) . "\n"; echo "\n2. Migrate small database (will create shards):\n"; -$result = $db->migrate("small_db"); +$result = $db->migrate("small-db"); echo " " . json_encode($result) . "\n"; // Check after migration -$info = $db->getShardInfo("small_db"); +$info = $db->getShardInfo("small-db"); echo " After migration - Sharded: " . ($info['sharded'] ? "true" : "false") . "\n"; echo "\n3. Try to migrate again (already sharded):\n"; -$result = $db->migrate("small_db"); +$result = $db->migrate("small-db"); echo " " . json_encode($result) . "\n"; // ========================================== @@ -83,7 +84,7 @@ echo "\n=== COMPACT ===\n\n"; // Create database with deletions -$db->insert("with_deletions", [ +$db->insert("with-deletions", [ ["id" => 1, "name" => "Keep 1"], ["id" => 2, "name" => "Delete me"], ["id" => 3, "name" => "Keep 2"], @@ -92,18 +93,18 @@ ]); echo "1. Before deletion:\n"; -echo " Records: " . $db->count("with_deletions") . "\n"; +echo " Records: " . $db->count("with-deletions") . "\n"; // Delete some records -$db->delete("with_deletions", ["name" => "Delete me"]); -$db->delete("with_deletions", ["name" => "Delete me too"]); +$db->delete("with-deletions", ["name" => "Delete me"]); +$db->delete("with-deletions", ["name" => "Delete me too"]); -echo "\n2. After deletion (records still show null internally):\n"; -echo " Visible records: " . $db->count("with_deletions") . "\n"; +echo "\n2. After deletion (deleted records are hidden immediately; their bytes stay in the file until compaction):\n"; +echo " Visible records: " . $db->count("with-deletions") . "\n"; // Compact to remove null entries echo "\n3. After compact:\n"; -$result = $db->compact("with_deletions"); +$result = $db->compact("with-deletions"); echo " Result: " . json_encode($result) . "\n"; echo " Freed slots: " . $result['freedSlots'] . "\n"; echo " Total records: " . $result['totalRecords'] . "\n"; @@ -128,12 +129,12 @@ } $start = microtime(true); -$db->insert("medium_db", $data); +$db->insert("medium-db", $data); $insertTime = round((microtime(true) - $start) * 1000, 2); echo " Insert time: " . $insertTime . " ms\n"; -$info = $db->getShardInfo("medium_db"); +$info = $db->getShardInfo("medium-db"); echo " Sharded: " . ($info['sharded'] ? "true" : "false") . "\n"; echo " Total records: " . $info['totalRecords'] . "\n"; @@ -146,7 +147,7 @@ // Find by key $start = microtime(true); -$result = $db->find("medium_db", ["key" => 250]); +$result = $db->find("medium-db", ["key" => 250]); $findTime = round((microtime(true) - $start) * 1000, 4); echo "Find by key 250:\n"; @@ -155,7 +156,7 @@ // Find by multiple keys $start = microtime(true); -$result = $db->find("medium_db", ["key" => [100, 200, 300, 400]]); +$result = $db->find("medium-db", ["key" => [100, 200, 300, 400]]); $findTime = round((microtime(true) - $start) * 1000, 4); echo "\nFind by keys [100, 200, 300, 400]:\n"; diff --git a/examples/update.php b/examples/update.php index a65e67f..675385b 100644 --- a/examples/update.php +++ b/examples/update.php @@ -5,8 +5,9 @@ * Demonstrates various ways to update records. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users'); echo "=== noneDB Update Example ===\n\n"; @@ -125,7 +126,7 @@ ["set" => ["key" => 999]] // ERROR: 'key' is reserved ]); echo " Result: " . json_encode($result) . "\n"; -// Output: {"n":0,"error":"Please check your update paramters"} +// Output: {"n":0,"error":"Please check your update parameters"} // ----------------------------------------- // 9. Show Final Data diff --git a/examples/utility-methods.php b/examples/utility-methods.php index 6bcfb5a..382b276 100644 --- a/examples/utility-methods.php +++ b/examples/utility-methods.php @@ -5,8 +5,9 @@ * Demonstrates first(), last(), exists(), sort(), and limit() methods. */ -include("../noneDB.php"); -$db = new noneDB(); +require __DIR__ . '/_bootstrap.php'; +$db = exampleDb(); +freshDatabases($db, 'users'); echo "=== noneDB Utility Methods Example ===\n\n"; diff --git a/noneDB.php b/noneDB.php index 129a6f3..d763ce9 100644 --- a/noneDB.php +++ b/noneDB.php @@ -11,30 +11,44 @@ * Respect for Labour - Emeğe Saygı */ -ini_set('memory_limit', '-1'); - class noneDB { - // Configuration file path (relative to dbDir or absolute) + // Configuration file name private static $configFile = '.nonedb'; private static $configLoaded = false; private static $configData = null; + private static $configFilePath = null; // Path of the loaded config file (null = none / dev mode) + private static $configFileDir = null; // Directory relative dbDir paths in the file resolve against + private static $scriptDirs = []; // SCRIPT_FILENAME + cwd -> resolved entry script directory + + // Placeholder keys shipped in the docs/templates; never accepted as a real secret + private static $placeholderKeys = [ + 'CHANGE_THIS_TO_A_SECURE_RANDOM_STRING', + 'YOUR_SECURE_RANDOM_STRING', + 'YOUR_SECURE_RANDOM_STRING_HERE', + 'GENERATE_UNIQUE_64_CHAR_STRING', + 'your_secure_key', + 'your_key', + ]; + + // secretKey values live outside the object so print_r/var_export/serialize can't leak them + private static $secretKeys = []; private $dbDir = null; // Set via .nonedb config file or constructor - private $secretKey = null; // Set via .nonedb config file or constructor private $autoCreateDB = true; + private $devMode = false; // Sharding configuration private $shardingEnabled=true; // Enable/disable auto-sharding - private $shardSize=10000; // Max records per shard (10K) - optimal for filter operations + private $shardSize=10000; // Records per shard for NEW sharded databases (existing ones keep the size stored in their meta) private $autoMigrate=true; // Auto-migrate legacy DBs to sharded format // File locking configuration - private $lockTimeout=5; // Max seconds to wait for lock + private $lockTimeout=5; // Max seconds to wait for a database lock (0 = wait forever) private $lockRetryDelay=10000; // Microseconds between lock attempts (10ms) - // Write buffer configuration - private $bufferEnabled=true; // Enable/disable write buffering + // Write buffer configuration (off by default since v3.2.0: direct writes are faster and always visible) + private $bufferEnabled=false; // Enable/disable write buffering private $bufferSizeLimit=1048576; // 1MB buffer size limit per buffer private $bufferCountLimit=10000; // Max records per buffer (safety limit) private $bufferFlushOnRead=true; // Flush buffer before read operations @@ -44,62 +58,78 @@ class noneDB { // Buffer state tracking (runtime) private $bufferLastFlush=[]; // Track last flush time per DB/shard private $shutdownHandlerRegistered=false; // Track if shutdown handler is registered - - // Performance cache (runtime) - v2.3.0 - private $hashCache=[]; // Cache dbname -> hash (PBKDF2 is expensive) - private $metaCache=[]; // Cache dbname -> meta data - private $metaCacheTime=[]; // Cache timestamps for TTL - private $metaCacheTTL=1; // Meta cache TTL in seconds (short for consistency) - - // Index configuration - v2.3.0 - private $indexEnabled=true; // Enable/disable primary key indexing - private $indexCache=[]; // Runtime cache for index data - private $shardedCache=[]; // Cache isSharded results - - // JSONL Storage Engine - v3.0.0 (JSONL-only, v2 format removed) - private $jsonlFormatCache=[]; // Cache format detection per DB - private $jsonlGarbageThreshold=0.3; // Trigger compaction when garbage > 30% - - // Static caches for cross-instance sharing - v3.0.0 - private static $staticIndexCache=[]; // Shared index cache across instances - private static $staticShardedCache=[]; // Shared isSharded results - private static $staticMetaCache=[]; // Shared meta data cache - private static $staticMetaCacheTime=[]; // Shared meta cache timestamps - private static $staticHashCache=[]; // Shared hash cache (PBKDF2 is expensive) - private static $staticFormatCache=[]; // Shared format detection cache - private static $staticFileExistsCache=[]; // Shared file_exists cache - v3.0.0 - private static $staticSanitizeCache=[]; // Shared dbname sanitization cache - v3.0.0 - private static $staticFieldIndexCache=[]; // Shared field index cache - v3.0.0 - private static $staticSpatialIndexCache=[]; // Shared spatial R-tree index cache - v3.1.0 - private static $staticGlobalSpatialIndexCache=[]; // Shared global spatial index cache - v3.1.0 + private static $bufferCounts=[]; // bufferPath => [size, count] so limits don't need a re-read per insert + private static $flushingNow=[]; // database base path => true while its buffer is being flushed + + // Performance caches (runtime). All keys are full file paths (or key-fingerprint|dbname for hashes), + // so instances with different dbDir/secretKey never see each other's entries. + private $hashCache=[]; // keyFingerprint|dbname -> hash (PBKDF2 is expensive) + private $metaCache=[]; // meta path -> meta data + private $metaCacheTime=[]; // kept for API compatibility (unused since v3.2.0) + + // Index configuration + private $indexEnabled=true; // Kept for API compatibility (enableIndexing / isIndexingEnabled) + private $indexCache=[]; // jidx path -> JSONL index + private $shardedCache=[]; // meta path -> bool + + // JSONL Storage Engine + private $jsonlFormatCache=[]; // data path -> bool (JSONL vs legacy v2) + private $jsonlGarbageThreshold=0.3; // Trigger compaction when garbage > 30% of live records + + // Static caches for cross-instance sharing + private static $staticIndexCache=[]; + private static $staticShardedCache=[]; + private static $staticMetaCache=[]; + private static $staticMetaCacheTime=[]; + private static $staticHashCache=[]; + private static $staticFormatCache=[]; + private static $staticFileExistsCache=[]; + private static $staticSanitizeCache=[]; + private static $staticFieldIndexCache=[]; + private static $staticSpatialIndexCache=[]; + private static $staticGlobalSpatialIndexCache=[]; + private static $staticGenerations=[]; // lock path -> generation the caches were validated against + private static $staticHashFilesLoaded=[]; // hash cache file -> true private static $staticCacheEnabled=true; // Enable/disable static caching - // Field indexing configuration - v3.0.0 + // Process-wide lock registry (shared by every instance so two instances never self-deadlock) + private static $lockRegistry=[]; // lock path -> ['fp', 'mode' => LOCK_SH|LOCK_EX, 'depth', 'dbname', 'gen', 'flags', 'bumped', 'pending'] while locked + private static $lockHandles=[]; // lock path -> ['fp' => resource, 'pid' => int]: unlocked handles kept open for the next operation + private $generations=[]; // per-instance view of $staticGenerations + private $hashFilesLoaded=[]; + + // Field indexing configuration private $fieldIndexEnabled = true; // Enable field-based indexing - private $fieldIndexCache = []; // Instance-level field index cache + private $fieldIndexCache = []; // field index path -> index - // Spatial indexing configuration - v3.1.0 + // Spatial indexing configuration private $spatialIndexEnabled = true; // Enable spatial R-tree indexing - private $spatialIndexCache = []; // Instance-level spatial index cache - private $globalSpatialIndexCache = []; // Instance-level global spatial index cache - private $rtreeNodeSize = 32; // Max entries per R-tree node (increased from 16 for fewer splits) - private $dirtySpatialIndexes = []; // Dirty flags for batch spatial writes + private $spatialIndexCache = []; // spatial index path -> index + private $globalSpatialIndexCache = []; // global spatial index path -> index + private $rtreeNodeSize = 32; // Max entries per R-tree node + private $dirtySpatialIndexes = []; // Dirty flags for batch spatial writes (flushed before a write lock is released) private $distanceCache = []; // Haversine distance memoization cache private $centroidCache = []; // Geometry centroid cache - // Persistent hash cache - v3.0.0 performance optimization - private $hashCacheFile = null; // Path to persistent hash cache file - private $hashCacheDirty = false; // Track if hash cache needs saving - private $hashCacheLoaded = false; // Track if persistent cache was loaded + private $keyFingerprint = null; // HMAC of the secret, used to bind caches to the key + private $dbBases = []; // dbname -> file name prefix (depends on dbDir and key) + private $dbBasesDir = null; // dbDir the prefixes were built for + + private static $devModeNoticeShown = false; /** * Constructor - load config and initialize static caches - * @param array|null $config Optional config array to override file config - * @throws \RuntimeException If config file is missing in production mode + * @param array|null $config Optional config array; merged over the .nonedb file when one exists + * @throws \RuntimeException If no configuration can be found outside dev mode + * @throws \InvalidArgumentException If secretKey is missing, empty, not a string or a placeholder */ public function __construct($config = null){ - // Load configuration $this->loadConfig($config); + $this->validateSecretKey(); + if(((int)ini_get('mbstring.func_overload')) & 2){ + // Byte offsets are computed with strlen(); an overloaded strlen() would corrupt them + throw new \RuntimeException('noneDB: mbstring.func_overload must be disabled'); + } // Link instance caches to static caches for cross-instance sharing if(self::$staticCacheEnabled){ @@ -112,85 +142,143 @@ public function __construct($config = null){ $this->fieldIndexCache = &self::$staticFieldIndexCache; $this->spatialIndexCache = &self::$staticSpatialIndexCache; $this->globalSpatialIndexCache = &self::$staticGlobalSpatialIndexCache; + $this->generations = &self::$staticGenerations; + $this->hashFilesLoaded = &self::$staticHashFilesLoaded; } + } - // Register shutdown function to flush dirty spatial indexes - register_shutdown_function([$this, 'flushSpatialIndexes']); + /** + * Never expose the secret in debug output + * @return array + */ + public function __debugInfo(){ + return [ + 'dbDir' => $this->dbDir, + 'secretKey' => '***', + 'autoCreateDB' => $this->autoCreateDB, + 'shardingEnabled' => $this->shardingEnabled, + 'shardSize' => $this->shardSize, + 'bufferEnabled' => $this->bufferEnabled, + 'devMode' => $this->devMode, + ]; } /** - * Load configuration from file or array - * Config file locations checked (in order): - * 1. .nonedb in project root (dirname of including script) - * 2. .nonedb in noneDB.php directory - * 3. .nonedb in dbDir - * - * @param array|null $config Optional config array - * @throws \RuntimeException If config file missing and not in dev mode + * Instances hold a secret and open resources; they can't be serialized + * @throws \LogicException */ - private function loadConfig($config = null){ - // If config array provided, use it directly - if(is_array($config)){ - $this->applyConfig($config); - return; - } + public function __serialize(): array { + throw new \LogicException('noneDB instances cannot be serialized'); + } - // If already loaded from file, just apply - if(self::$configLoaded && self::$configData !== null){ - $this->applyConfig(self::$configData); - return; + public function __unserialize(array $data): void { + throw new \LogicException('noneDB instances cannot be unserialized'); + } + + public function __clone(){ + $source = self::$secretKeys[$this->cloneSourceId] ?? null; + $this->cloneSourceId = spl_object_id($this); + if($source !== null){ + self::$secretKeys[$this->cloneSourceId] = $source; } + } - // Try to find config file - $configPaths = [ - dirname($_SERVER['SCRIPT_FILENAME'] ?? __DIR__) . '/' . self::$configFile, - __DIR__ . '/' . self::$configFile, - $this->dbDir . self::$configFile - ]; + private $cloneSourceId = null; - $configPath = null; - foreach($configPaths as $path){ - if(file_exists($path)){ - $configPath = $path; - break; - } - } + public function __destruct(){ + // Anything still dirty belongs to a lock we no longer hold; it was flushed on release. + unset(self::$secretKeys[spl_object_id($this)]); + } - if($configPath !== null){ - // Config file found - load it - $content = @file_get_contents($configPath); - if($content === false){ - throw new \RuntimeException("noneDB: Cannot read config file: {$configPath}"); - } + private function getSecretKey(){ + return self::$secretKeys[spl_object_id($this)] ?? null; + } - $data = @json_decode($content, true); - if($data === null && json_last_error() !== JSON_ERROR_NONE){ - throw new \RuntimeException("noneDB: Invalid JSON in config file: {$configPath}"); - } + private function setSecretKey($key){ + self::$secretKeys[spl_object_id($this)] = $key; + $this->cloneSourceId = spl_object_id($this); + $this->keyFingerprint = null; + $this->dbBases = []; + } - self::$configData = $data; + /** + * Load configuration. + * + * The .nonedb file is searched in: the entry script's directory, noneDB.php's directory, + * the Composer project root (when installed under vendor/) and the current working directory. + * A config array passed to the constructor is merged over the file (file is optional when the + * array already contains secretKey and dbDir). + * + * @param array|null $config Optional config array + * @throws \RuntimeException If config is missing and not in dev mode + */ + private function loadConfig($config = null){ + $arrayConfig = is_array($config) ? $config : []; + // A secretKey given in the array is used (and validated) as given, also null/false/'': + // never replaced by the file's key or the dev mode key + $keyGiven = array_key_exists('secretKey', $arrayConfig); + $arrayIsComplete = $keyGiven && isset($arrayConfig['dbDir']); + + // Load (and cache) the file config once per process + if(!self::$configLoaded){ + self::$configData = null; + self::$configFilePath = null; + self::$configFileDir = null; + + $configPath = self::findConfigFile(); + if($configPath !== null){ + $content = @file_get_contents($configPath); + if($content === false){ + if(!$arrayIsComplete){ + throw new \RuntimeException("noneDB: Cannot read config file: {$configPath}"); + } + } else { + $data = json_decode($content, true); + if(!is_array($data)){ + if(!$arrayIsComplete){ + throw new \RuntimeException("noneDB: Invalid JSON in config file: {$configPath}"); + } + } else { + self::$configData = $data; + self::$configFilePath = $configPath; + self::$configFileDir = dirname($configPath); + } + } + } self::$configLoaded = true; - $this->applyConfig($data); - return; } - // No config file found - // Check for development mode - $devMode = getenv('NONEDB_DEV_MODE') === 'true' - || getenv('NONEDB_DEV_MODE') === '1' - || (defined('NONEDB_DEV_MODE') && NONEDB_DEV_MODE === true); + if(self::$configData !== null){ + $fileConfig = self::$configData; + if(isset($arrayConfig['dbDir'])){ + unset($fileConfig['dbDir']); // overridden below; no need to resolve the file's one + } + $this->applyConfig($fileConfig, self::$configFileDir, true); + } + if(!empty($arrayConfig)){ + $this->applyConfig($arrayConfig, null, false); + } + if(self::$configData !== null || $arrayIsComplete || ($keyGiven && $this->dbDir !== null)){ + return; + } - if($devMode){ - // Dev mode - use defaults - self::$configLoaded = true; - self::$configData = []; - // Set default values for dev mode - $this->dbDir = __DIR__ . '/db/'; - $this->secretKey = 'nonedb_dev_mode_key_' . md5(__DIR__); + if(self::isDevModeEnabled()){ + $this->devMode = true; + $projectRoot = self::projectRoot(); + if($this->dbDir === null){ + $this->dbDir = $projectRoot . '/db/'; + } + if(!$keyGiven){ + // Deterministic for backward compatibility with databases created in dev mode before v3.2.0. + $this->setSecretKey('nonedb_dev_mode_key_' . md5(__DIR__)); + } + if(PHP_SAPI !== 'cli' && !self::$devModeNoticeShown){ + self::$devModeNoticeShown = true; + trigger_error('noneDB: development mode is active (predictable secretKey). Never use it in production.', E_USER_NOTICE); + } return; } - // Production mode without config file - throw error throw new \RuntimeException( "noneDB: Configuration file not found!\n" . "Create a '.nonedb' config file in your project root.\n" . @@ -199,25 +287,102 @@ private function loadConfig($config = null){ ); } + /** + * Candidate config file locations, in priority order + * @return array + */ + private static function configSearchPaths(): array { + $dirs = [self::scriptDir(), __DIR__]; + $root = self::projectRoot(); + if($root !== __DIR__){ + $dirs[] = $root; + } + $cwd = getcwd(); + if($cwd !== false){ + $dirs[] = $cwd; + } + $paths = []; + foreach(array_unique($dirs) as $dir){ + $paths[] = rtrim($dir, '/\\') . '/' . self::$configFile; + } + return $paths; + } + + private static function findConfigFile(): ?string { + foreach(self::configSearchPaths() as $path){ + if(is_file($path)){ + return $path; + } + } + return null; + } + + /** + * Absolute directory of the entry script (falls back to the working directory) + */ + private static function scriptDir(): string { + $script = $_SERVER['SCRIPT_FILENAME'] ?? ''; + $cwd = getcwd(); + // A relative SCRIPT_FILENAME depends on the working directory: both are part of the memo key + $memoKey = $script . "\0" . $cwd; + if(isset(self::$scriptDirs[$memoKey])){ + return self::$scriptDirs[$memoKey]; + } + $dir = null; + if($script !== ''){ + $real = realpath($script); + if($real !== false){ + $dir = dirname($real); + } else { + $realDir = realpath(dirname($script)); + if($realDir !== false){ + $dir = $realDir; + } + } + } + if($dir === null){ + $dir = $cwd !== false ? $cwd : __DIR__; + } + if(count(self::$scriptDirs) > 16){ + self::$scriptDirs = []; + } + return self::$scriptDirs[$memoKey] = $dir; + } + + /** + * Project root: the directory that contains vendor/ when installed with Composer, else noneDB.php's directory + */ + private static function projectRoot(): string { + $dir = str_replace('\\', '/', __DIR__); + $pos = strpos($dir, '/vendor/'); + if($pos !== false){ + return substr(__DIR__, 0, $pos); + } + return __DIR__; + } + + private static function isDevModeEnabled(): bool { + return getenv('NONEDB_DEV_MODE') === 'true' + || getenv('NONEDB_DEV_MODE') === '1' + || (defined('NONEDB_DEV_MODE') && NONEDB_DEV_MODE === true); + } + + private static function isAbsolutePath(string $path): bool { + return $path !== '' && ($path[0] === '/' || $path[0] === '\\' || preg_match('/^[A-Za-z]:[\\\\\/]/', $path) === 1); + } + /** * Apply configuration values to instance properties * @param array $config Configuration array + * @param string|null $baseDir Directory that relative dbDir values resolve against (null: the entry script's) + * @param bool $fromFile True when the values come from the .nonedb file */ - private function applyConfig(array $config){ - // Core settings - if(isset($config['secretKey'])){ - $this->secretKey = $config['secretKey']; + private function applyConfig(array $config, ?string $baseDir, bool $fromFile){ + if(array_key_exists('secretKey', $config)){ + $this->setSecretKey($config['secretKey']); } if(isset($config['dbDir'])){ - // Handle relative paths - $dbDir = $config['dbDir']; - if(substr($dbDir, 0, 2) === './'){ - $dbDir = dirname($_SERVER['SCRIPT_FILENAME'] ?? __DIR__) . '/' . substr($dbDir, 2); - } - if(substr($dbDir, -1) !== '/'){ - $dbDir .= '/'; - } - $this->dbDir = $dbDir; + $this->dbDir = $this->resolveDbDir((string)$config['dbDir'], $baseDir, $fromFile); } if(isset($config['autoCreateDB'])){ $this->autoCreateDB = (bool)$config['autoCreateDB']; @@ -228,7 +393,11 @@ private function applyConfig(array $config){ $this->shardingEnabled = (bool)$config['shardingEnabled']; } if(isset($config['shardSize'])){ - $this->shardSize = (int)$config['shardSize']; + $shardSize = (int)$config['shardSize']; + if($shardSize < 1){ + throw new \InvalidArgumentException("noneDB: 'shardSize' must be a positive integer"); + } + $this->shardSize = $shardSize; } if(isset($config['autoMigrate'])){ $this->autoMigrate = (bool)$config['autoMigrate']; @@ -236,34 +405,109 @@ private function applyConfig(array $config){ // Compaction settings if(isset($config['autoCompactThreshold'])){ - $this->jsonlGarbageThreshold = (float)$config['autoCompactThreshold']; + $threshold = (float)$config['autoCompactThreshold']; + if($threshold <= 0){ + throw new \InvalidArgumentException("noneDB: 'autoCompactThreshold' must be greater than 0"); + } + $this->jsonlGarbageThreshold = $threshold; } // Lock settings if(isset($config['lockTimeout'])){ - $this->lockTimeout = (int)$config['lockTimeout']; + $timeout = (int)$config['lockTimeout']; + if($timeout < 0){ + throw new \InvalidArgumentException("noneDB: 'lockTimeout' must be 0 (wait forever) or a positive number of seconds"); + } + $this->lockTimeout = $timeout; } if(isset($config['lockRetryDelay'])){ - $this->lockRetryDelay = (int)$config['lockRetryDelay']; + $delay = (int)$config['lockRetryDelay']; + if($delay < 1){ + throw new \InvalidArgumentException("noneDB: 'lockRetryDelay' must be a positive number of microseconds"); + } + $this->lockRetryDelay = $delay; + } + + // Write buffer + if(isset($config['bufferEnabled'])){ + $this->bufferEnabled = (bool)$config['bufferEnabled']; + } + + // Use indexes for queries (indexes are always kept up to date) + if(isset($config['fieldIndexEnabled'])){ + $this->fieldIndexEnabled = (bool)$config['fieldIndexEnabled']; + } + if(isset($config['spatialIndexEnabled'])){ + $this->spatialIndexEnabled = (bool)$config['spatialIndexEnabled']; } } /** - * Check if config file exists - * @return bool + * Resolve a configured dbDir to an absolute path ending with a slash. + * Relative paths (with or without a leading ./) are relative to the config file's directory + * (file config) or to the entry script's directory (array config). */ - public static function configExists(): bool { - $configPaths = [ - dirname($_SERVER['SCRIPT_FILENAME'] ?? __DIR__) . '/' . self::$configFile, - __DIR__ . '/' . self::$configFile - ]; + private function resolveDbDir(string $dbDir, ?string $baseDir, bool $fromFile): string { + if(!self::isAbsolutePath($dbDir)){ + $dotted = substr($dbDir, 0, 2) === './' || substr($dbDir, 0, 3) === '../' + || $dbDir === '.' || $dbDir === '..'; + $relative = preg_replace('#^(\./)+#', '', $dbDir); + $baseDir = $baseDir ?? self::scriptDir(); + $resolved = rtrim($baseDir, '/\\') . '/' . $relative; - foreach($configPaths as $path){ - if(file_exists($path)){ - return true; + // Before v3.2.0 "./x" in the file was resolved against the entry script and a plain "x" + // against the working directory. Keep using such a directory (with a deprecation notice) + // until the data is moved. + $legacyBase = is_dir($resolved) ? null : ($dotted ? ($fromFile ? self::scriptDir() : null) : getcwd()); + if(is_string($legacyBase)){ + $legacy = rtrim($legacyBase, '/\\') . '/' . $relative; + if(is_dir($legacy) && realpath($legacy) !== realpath($resolved)){ + trigger_error( + "noneDB: dbDir '{$dbDir}' is now resolved relative to " . ($fromFile ? 'the .nonedb file' : 'the entry script') + . " ({$baseDir}); using the old location {$legacy} for now. Move the directory or use an absolute path.", + E_USER_DEPRECATED + ); + $resolved = $legacy; + } } + $dbDir = $resolved; } - return false; + + if(substr($dbDir, -1) !== '/' && substr($dbDir, -1) !== '\\'){ + $dbDir .= '/'; + } + return $dbDir; + } + + /** + * Reject missing, empty, non-string and placeholder secret keys + * @throws \InvalidArgumentException + */ + private function validateSecretKey(){ + $key = $this->getSecretKey(); + if(!is_string($key) || trim($key) === ''){ + throw new \InvalidArgumentException( + "noneDB: 'secretKey' must be a non-empty string. Set it in your .nonedb file or constructor config " + . "(generate one with: php -r \"echo bin2hex(random_bytes(32));\")." + ); + } + if(in_array($key, self::$placeholderKeys, true)){ + throw new \InvalidArgumentException( + "noneDB: 'secretKey' is still the example placeholder. Replace it with a random secret " + . "(php -r \"echo bin2hex(random_bytes(32));\"). If you already have data, see noneDB::migrateKey()." + ); + } + if($this->dbDir === null){ + throw new \RuntimeException("noneDB: 'dbDir' is not configured"); + } + } + + /** + * Check if config file exists + * @return bool + */ + public static function configExists(): bool { + return self::findConfigFile() !== null; } /** @@ -282,6 +526,16 @@ public static function getConfigTemplate(): ?string { public static function clearConfigCache(): void { self::$configLoaded = false; self::$configData = null; + self::$configFilePath = null; + self::$configFileDir = null; + } + + /** + * True when development mode is enabled (NONEDB_DEV_MODE constant or environment variable) + * @return bool + */ + public static function isDevMode(): bool { + return self::isDevModeEnabled(); } /** @@ -297,64 +551,221 @@ public static function setDevMode(bool $enabled): void { } /** - * Destructor - save persistent hash cache - * v3.0.0 performance optimization + * Re-key existing databases: rename every file created with $oldKey to the name $newKey produces. + * + * File names are {hash}-{dbname}... where hash depends on the secretKey, so changing the key + * without this makes existing data invisible. Run it once, with no other process using the + * databases, then update secretKey in your config. Each database is locked while it is renamed. + * The old per-key hash cache and the pre-3.2 `.nonedb_hash_cache` file are removed. + * + * @param string $dbDir Database directory + * @param string $oldKey Key the databases were created with (the old placeholder is accepted here) + * @param string $newKey New secret key + * @return array ['success' => bool, 'databases' => string[], 'files' => int, 'error' => ?string] + */ + public static function migrateKey(string $dbDir, string $oldKey, string $newKey): array { + $result = ['success' => false, 'databases' => [], 'files' => 0, 'error' => null]; + if(trim($oldKey) === '' || trim($newKey) === ''){ + $result['error'] = 'Both keys must be non-empty strings'; + return $result; + } + if(in_array($newKey, self::$placeholderKeys, true)){ + $result['error'] = 'The new key is a placeholder; generate a random one'; + return $result; + } + if($oldKey === $newKey){ + $result['error'] = 'The new key is the same as the old key'; + return $result; + } + $dbDir = rtrim($dbDir, '/\\') . '/'; + if(!is_dir($dbDir)){ + $result['error'] = "Directory not found: {$dbDir}"; + return $result; + } + + // Databases created with the old key + $databases = []; + foreach(scandir($dbDir) ?: [] as $file){ + if(preg_match("/^([0-9a-f]{20})-([A-Za-z0-9' -]+)\\.nonedb(info|\\.meta)?$/", $file, $m) + && hash_pbkdf2('sha256', $m[2], $oldKey, 1000, 20) === $m[1]){ + $databases[$m[2]] = $m[1]; + } + } + + // Plan every rename first, refuse on any conflict + $plan = []; + $files = scandir($dbDir) ?: []; + foreach($databases as $dbname => $oldHash){ + $newHash = hash_pbkdf2('sha256', $dbname, $newKey, 1000, 20); + foreach($files as $file){ + // {hash}-{name}.*, {hash}-{name}_sN.*, and spatial files {hash}.nonedb.* / {hash}_sN.nonedb.* + $named = strpos($file, $oldHash . '-' . $dbname . '.') === 0 || strpos($file, $oldHash . '-' . $dbname . '_s') === 0; + $spatial = strpos($file, $oldHash . '.nonedb.') === 0 || preg_match('/^' . $oldHash . '_s\d+\.nonedb\./', $file) === 1; + if(!$named && !$spatial){ + continue; + } + $target = $newHash . substr($file, strlen($oldHash)); + if(file_exists($dbDir . $target)){ + $result['error'] = "Target file already exists: {$target}"; + return $result; + } + $plan[$dbname][] = [$file, $target]; + } + } + + foreach($plan as $dbname => $renames){ + $oldHash = $databases[$dbname]; + $lockPath = $dbDir . $oldHash . '-' . $dbname . '.nonedb.lock'; + // Databases from before v3.2 have no lock file: the one opened here is removed afterwards + clearstatcache(true, $lockPath); + $hadLock = file_exists($lockPath); + $fp = @fopen($lockPath, 'c+b'); + if($fp === false || !flock($fp, LOCK_EX)){ + if($fp !== false){ + fclose($fp); + } + if(!$hadLock){ + @unlink($lockPath); + } + $result['error'] = "Could not lock database '{$dbname}'"; + return $result; + } + try { + // Rename the lock file last so writers keep blocking on it until we are done + usort($renames, function($a, $b){ + return (substr($a[0], -12) === '.nonedb.lock') <=> (substr($b[0], -12) === '.nonedb.lock'); + }); + foreach($renames as $pair){ + if(!@rename($dbDir . $pair[0], $dbDir . $pair[1])){ + $result['error'] = "Failed to rename {$pair[0]}"; + return $result; + } + $result['files']++; + } + } finally { + flock($fp, LOCK_UN); + fclose($fp); + if(!$hadLock){ + @unlink($lockPath); + } + } + $result['databases'][] = $dbname; + } + + // Old caches can't be used with the new key (and the legacy one leaks names) + $oldFingerprint = substr(hash_hmac('sha256', 'nonedb-key-fingerprint-v1', $oldKey), 0, 16); + @unlink($dbDir . '.nonedb_hash_cache_' . $oldFingerprint); + @unlink($dbDir . '.nonedb_hash_cache'); + self::clearStaticCache(); + + $result['success'] = true; + return $result; + } + + /** + * HMAC fingerprint of the secret; binds caches to the key without revealing it */ - public function __destruct(){ - $this->savePersistentHashCache(); + private function getKeyFingerprint(){ + if($this->keyFingerprint === null){ + $this->keyFingerprint = substr(hash_hmac('sha256', 'nonedb-key-fingerprint-v1', (string)$this->getSecretKey()), 0, 16); + } + return $this->keyFingerprint; } /** - * Load hash cache from persistent storage - * v3.0.0 performance optimization: Eliminates PBKDF2 computation on subsequent requests + * Path of the persistent hash cache for this dbDir + secretKey. + * The file name is derived from the key, so different keys never share (or poison) a cache, + * and the file can't be guessed without the key. + */ + private function getHashCacheFile(){ + return $this->dbDir . '.nonedb_hash_cache_' . $this->getKeyFingerprint(); + } + + /** + * Load the persistent hash cache for this dbDir + key (once per process) * @return void */ private function loadPersistentHashCache(){ - if($this->hashCacheLoaded){ + $file = $this->getHashCacheFile(); + if(isset($this->hashFilesLoaded[$file])){ return; } - $this->hashCacheLoaded = true; + $this->hashFilesLoaded[$file] = true; - if($this->hashCacheFile === null){ - $this->hashCacheFile = $this->dbDir . '.nonedb_hash_cache'; + $fp = @fopen($file, 'rb'); + if($fp === false){ + return; + } + $content = ''; + if(flock($fp, LOCK_SH)){ + $content = stream_get_contents($fp); + flock($fp, LOCK_UN); } + fclose($fp); - if(file_exists($this->hashCacheFile)){ - $data = @file_get_contents($this->hashCacheFile); - if($data !== false && $data !== ''){ - $loaded = @json_decode($data, true); - if(is_array($loaded) && !empty($loaded)){ - // Merge into hash cache - foreach($loaded as $dbname => $hash){ - if(!isset($this->hashCache[$dbname])){ - $this->hashCache[$dbname] = $hash; - } - } - // Also update static cache if enabled - if(self::$staticCacheEnabled){ - self::$staticHashCache = $this->hashCache; - } - } + $loaded = json_decode((string)$content, true); + if(!is_array($loaded)){ + return; + } + $prefix = $this->getKeyFingerprint() . '|'; + foreach($loaded as $dbname => $hash){ + if(is_string($dbname) && is_string($hash) && preg_match('/^[0-9a-f]{20}$/', $hash)){ + $this->hashCache[$prefix . $dbname] = $hash; } } } /** - * Save hash cache to persistent storage - * v3.0.0 performance optimization: Persists PBKDF2 results across PHP requests + * Add one name -> hash entry to the persistent cache (locked read-merge-write). + * Entries whose database no longer exists are dropped. * @return void */ - private function savePersistentHashCache(){ - if(!$this->hashCacheDirty || empty($this->hashCache)){ + private function savePersistentHashCacheEntry($dbname, $hash){ + try { + $this->ensureDbDir(); + } catch(\RuntimeException $e){ + return; // the cache is optional + } + $file = $this->getHashCacheFile(); + $isNew = !file_exists($file); + $fp = @fopen($file, 'c+b'); + if($fp === false){ return; } - - if($this->hashCacheFile === null){ - $this->hashCacheFile = $this->dbDir . '.nonedb_hash_cache'; + if($isNew){ + @chmod($file, 0600); + } + if(!flock($fp, LOCK_EX)){ + fclose($fp); + return; + } + try { + $entries = json_decode((string)stream_get_contents($fp), true); + if(!is_array($entries)){ + $entries = []; + } + $entries[$dbname] = $hash; + foreach($entries as $name => $h){ + if(!is_string($h)){ + unset($entries[$name]); + continue; + } + $base = $this->dbDir . $h . '-' . $name; + if($name !== $dbname && !file_exists($base . '.nonedb') && !file_exists($base . '.nonedb.meta') && !file_exists($base . '.nonedbinfo')){ + unset($entries[$name]); + } + } + $json = json_encode($entries, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES); + if($json !== false){ + ftruncate($fp, 0); + rewind($fp); + fwrite($fp, $json); + fflush($fp); + } + } finally { + flock($fp, LOCK_UN); + fclose($fp); } - - @file_put_contents($this->hashCacheFile, json_encode($this->hashCache)); - $this->hashCacheDirty = false; } /** @@ -373,6 +784,13 @@ public static function clearStaticCache(){ self::$staticFieldIndexCache = []; self::$staticSpatialIndexCache = []; self::$staticGlobalSpatialIndexCache = []; + self::$staticGenerations = []; + self::$staticHashFilesLoaded = []; + self::$bufferCounts = []; + foreach(self::$lockHandles as $handle){ + @fclose($handle['fp']); + } + self::$lockHandles = []; } /** @@ -392,19 +810,21 @@ public static function enableStaticCache(){ } /** - * Cached file_exists check - v3.0.0 - * Reduces disk I/O by caching file existence checks + * Cached file_exists check. Entries are dropped whenever another process writes the database + * (see withLock), so the cache can't go stale across processes. * * @param string $path File path to check * @return bool True if file exists */ private function cachedFileExists($path){ if(!self::$staticCacheEnabled){ + clearstatcache(true, $path); return file_exists($path); } if(isset(self::$staticFileExistsCache[$path])){ return self::$staticFileExistsCache[$path]; } + clearstatcache(true, $path); $exists = file_exists($path); self::$staticFileExistsCache[$path] = $exists; return $exists; @@ -431,303 +851,667 @@ private function markFileNotExists($path){ } /** - * Invalidate file exists cache for a specific path - * @param string $path File path - */ - private function invalidateFileExistsCache($path){ - unset(self::$staticFileExistsCache[$path]); - } - - /** - * Sanitize database name - removes invalid characters - * Uses static cache to avoid redundant regex operations - v3.0.0 + * Validate a database name. + * Allowed characters: A-Z a-z 0-9, space, apostrophe and hyphen. Anything else is rejected + * (before v3.2.0 such characters were silently stripped, which let different names collide). * - * @param string $dbname Database name to sanitize - * @return string Sanitized database name + * @param mixed $dbname Database name + * @return string The (unchanged) database name + * @throws \InvalidArgumentException */ - private function sanitizeDbName($dbname){ - if(!self::$staticCacheEnabled){ - return preg_replace("/[^A-Za-z0-9' -]/", '', $dbname); + private function normalizeDbName($dbname){ + if(is_int($dbname)){ + $dbname = (string)$dbname; + } + if(!is_string($dbname)){ + throw new \InvalidArgumentException('noneDB: database name must be a string'); } if(isset(self::$staticSanitizeCache[$dbname])){ - return self::$staticSanitizeCache[$dbname]; + return $dbname; + } + if($dbname === '' || preg_match("/^[A-Za-z0-9' -]+$/", $dbname) !== 1){ + throw new \InvalidArgumentException( + "noneDB: invalid database name '{$dbname}'. Only letters A-Z/a-z, digits, space, apostrophe (') and hyphen (-) are allowed." + ); } - $sanitized = preg_replace("/[^A-Za-z0-9' -]/", '', $dbname); - self::$staticSanitizeCache[$dbname] = $sanitized; - return $sanitized; + if(self::$staticCacheEnabled){ + self::$staticSanitizeCache[$dbname] = true; + } + return $dbname; } /** - * hash to db name for security - * Uses instance-level caching + persistent cache to avoid expensive PBKDF2 recomputation - * v3.0.0 optimization: Loads from persistent cache on first access + * Hash a database name for its file names (PBKDF2-SHA256 keyed with secretKey). + * Cached per key fingerprint in memory and in a per-key file, so the secret is always honoured. */ private function hashDBName($dbname){ - // Check memory cache first (fastest) - if(isset($this->hashCache[$dbname])){ - return $this->hashCache[$dbname]; + $cacheKey = $this->getKeyFingerprint() . '|' . $dbname; + if(isset($this->hashCache[$cacheKey])){ + return $this->hashCache[$cacheKey]; } - // Load from persistent cache if not loaded yet $this->loadPersistentHashCache(); - if(isset($this->hashCache[$dbname])){ - return $this->hashCache[$dbname]; + if(isset($this->hashCache[$cacheKey])){ + return $this->hashCache[$cacheKey]; } - // Compute PBKDF2 hash (expensive: 1000 iterations) - $hash = hash_pbkdf2("sha256", $dbname, $this->secretKey, 1000, 20); - $this->hashCache[$dbname] = $hash; - $this->hashCacheDirty = true; + $hash = hash_pbkdf2("sha256", (string)$dbname, (string)$this->getSecretKey(), 1000, 20); + $this->hashCache[$cacheKey] = $hash; + $this->savePersistentHashCacheEntry((string)$dbname, $hash); return $hash; } - // ========================================== - // ATOMIC FILE OPERATIONS - // ========================================== + /** + * Common file-name prefix of every file belonging to a database: {dbDir}{hash}-{dbname} + */ + private function dbBase($dbname){ + if($this->dbBasesDir !== $this->dbDir){ + $this->dbBases = []; + $this->dbBasesDir = $this->dbDir; + } + return $this->dbBases[$dbname] ?? ($this->dbBases[$dbname] = $this->dbDir . $this->hashDBName($dbname) . '-' . $dbname); + } /** - * Atomically read a file with shared lock - * Prevents reading while another process is writing - * - * @param string $path File path - * @param mixed $default Default value if file doesn't exist - * @return mixed Decoded JSON data or default value + * Path of the main (non-sharded) data file */ - private function atomicRead($path, $default = null){ - clearstatcache(true, $path); + private function getDataPath($dbname){ + return $this->dbBase($dbname) . '.nonedb'; + } - if(!file_exists($path)){ - return $default; + /** + * Create the database directory (and protect it from direct web access) if needed + */ + private function ensureDbDir(){ + if(is_dir($this->dbDir)){ + return; } - - $fp = fopen($path, 'rb'); - if($fp === false){ - return $default; + if(!@mkdir($this->dbDir, 0755, true) && !is_dir($this->dbDir)){ + throw new \RuntimeException("noneDB: cannot create database directory {$this->dbDir}"); } + // Deny direct HTTP access on Apache; harmless elsewhere + @file_put_contents($this->dbDir . '.htaccess', "Require all denied\nDeny from all\n"); + @file_put_contents($this->dbDir . 'index.html', ''); + } - $startTime = microtime(true); - $locked = false; - - // Try to acquire shared lock with timeout - while(!$locked && (microtime(true) - $startTime) < $this->lockTimeout){ - $locked = flock($fp, LOCK_SH | LOCK_NB); - if(!$locked){ - usleep($this->lockRetryDelay); + // ========================================== + // DATABASE LOCKING AND STATE (v3.2.0) + // ========================================== + // + // Every database has a lock file {base}.nonedb.lock. Besides being the flock target it holds a small + // state record. It is read on every lock acquisition anyway, so the checks below cost no extra I/O: + // + // "{generation}[ v][ b][ j=main,0,3 | J]" space padded to LOCK_STATE_SIZE bytes + // + // generation bumped by the first write of every write session. A value other than the one this + // process last saw means another process changed the database: its cached state is dropped. + // v verified: the one-time checks for pre-3.2 leftovers (old buffer files, no index list) + // are done, so the flags below are authoritative and nothing is re-checked per operation. + // b the write buffer may hold records; they are flushed before the next read or write. + // j=... / J index journal: field/spatial indexes (and shard meta counters) of these segments (J: all) + // may lag behind the data because a writer died; the next lock holder rebuilds them. + // + // The record is overwritten in place with a fixed size (one small write: no truncate, no rename). + // Lock files are never deleted (dropDB keeps them), and their handles stay open between operations. + + private const LOCK_STATE_SIZE = 128; + private const LOCK_HANDLE_LIMIT = 64; + + private function getLockPath($dbname){ + return $this->dbBase($dbname) . '.nonedb.lock'; + } + + /** + * Run $fn while holding the database lock: shared for reads, exclusive for writes. + * + * The lock is process-wide and re-entrant. A nested exclusive request upgrades a shared lock. + * + * @param string $dbname Database name (already normalized) + * @param bool $exclusive + * @param callable $fn + * @return mixed Whatever $fn returns + */ + private function withLock($dbname, $exclusive, callable $fn){ + $path = $this->getLockPath($dbname); + + if(isset(self::$lockRegistry[$path])){ + if($exclusive && self::$lockRegistry[$path]['mode'] !== LOCK_EX){ + // flock upgrades are not atomic: another writer may run in between, so re-read the state + $this->acquireFlock(self::$lockRegistry[$path]['fp'], LOCK_EX, $dbname); + self::$lockRegistry[$path]['mode'] = LOCK_EX; + $this->loadLockState($dbname, $path); + } + self::$lockRegistry[$path]['depth']++; + try { + return $fn(); + } finally { + self::$lockRegistry[$path]['depth']--; } } - if(!$locked){ - // Fallback: blocking lock as last resort - $locked = flock($fp, LOCK_SH); - } - - if(!$locked){ - fclose($fp); - return $default; - } + $mode = $exclusive ? LOCK_EX : LOCK_SH; + $fp = $this->lockFile($path, $mode, $dbname); + self::$lockRegistry[$path] = [ + 'fp' => $fp, 'mode' => $mode, 'depth' => 1, 'dbname' => $dbname, + 'gen' => 0, 'flags' => [], 'bumped' => false, 'pending' => [], + ]; + $ok = false; try { - $content = stream_get_contents($fp); - if($content === false || $content === ''){ - return $default; - } - $data = json_decode($content, true); - // Check for JSON decode errors (corrupted data) - if($data === null && json_last_error() !== JSON_ERROR_NONE){ - return null; // Return null for corrupted JSON, not default - } - return $data !== null ? $data : $default; + $this->loadLockState($dbname, $path); + $this->prepareDb($dbname, $path); + $result = $fn(); + $ok = true; + return $result; } finally { - flock($fp, LOCK_UN); - fclose($fp); + try { + if(self::$lockRegistry[$path]['mode'] === LOCK_EX){ + $this->finishWriteSession($dbname, $path, $ok); + } + } finally { + $fp = self::$lockRegistry[$path]['fp']; + unset(self::$lockRegistry[$path]); + flock($fp, LOCK_UN); + $this->keepLockHandle($path, $fp); + } } } /** - * Fast atomic read optimized for index files - * v3.0.0 optimization: Skips clearstatcache and retry loop - * - Safe for index files that are read more often than written - * - Uses direct blocking lock instead of retry loop - * - * @param string $path File path - * @param mixed $default Default value if file doesn't exist - * @return mixed Decoded JSON data or default value + * End of an exclusive session: write pending spatial indexes, close the journal entries of this + * session, and make sure the generation moved so other processes drop their caches */ - private function atomicReadFast($path, $default = null){ - // Skip clearstatcache - safe for cached index paths - if(!file_exists($path)){ - return $default; + private function finishWriteSession($dbname, $path, $ok){ + if($ok){ + $this->flushSpatialIndexes(); + if(!empty(self::$lockRegistry[$path]['flags']['j'])){ + // Data, indexes and meta of every journaled segment are written + unset(self::$lockRegistry[$path]['flags']['j']); + $this->saveLockState($path); + } + } else { + // Partial failure: forget in-memory state, the files are the source of truth. + // A journal entry stays, so the next lock holder rebuilds what may be half-written. + $this->dirtySpatialIndexes = []; + $this->invalidateDbCaches($dbname); } - - $fp = @fopen($path, 'rb'); - if($fp === false){ - return $default; + if(!self::$lockRegistry[$path]['bumped']){ + $this->saveLockState($path); } + } - // Direct blocking LOCK_SH - faster than retry loop for read-heavy workloads - if(!flock($fp, LOCK_SH)){ + /** + * Open (or reuse) a lock file and lock it. + * A kept handle is only used in the process that opened it (a forked child would share the parent's + * locks), and any handle is only trusted while the path still names the same file: if the file was + * deleted or replaced meanwhile, the file the path names now is locked instead. + * @return resource + */ + private function lockFile($path, $mode, $dbname){ + $pid = getmypid(); + for($attempt = 0; ; $attempt++){ + $kept = self::$lockHandles[$path] ?? null; + unset(self::$lockHandles[$path]); + if($kept !== null && $kept['pid'] === $pid){ + $fp = $kept['fp']; + } else { + if($kept !== null){ + @fclose($kept['fp']); + } + $this->ensureDbDir(); + // "e": close-on-exec. A kept handle must not leak into programs this process starts: + // an inherited copy would keep a lock alive after this process died holding it. + $fp = @fopen($path, 'c+be'); + if($fp === false){ + throw new \RuntimeException("noneDB: cannot open lock file for database '{$dbname}'"); + } + } + try { + $this->acquireFlock($fp, $mode, $dbname); + } catch(\Throwable $e){ + fclose($fp); + throw $e; + } + $own = fstat($fp); + clearstatcache(true, $path); + $named = @stat($path); + if($named !== false && ($own === false || !$own['ino'] || !$named['ino'] + || ($own['ino'] === $named['ino'] && $own['dev'] === $named['dev']))){ + return $fp; + } + flock($fp, LOCK_UN); fclose($fp); - return $default; + if($attempt >= 10){ + throw new \RuntimeException("noneDB: the lock file of database '{$dbname}' keeps being replaced"); + } } + } - $content = stream_get_contents($fp); - flock($fp, LOCK_UN); - fclose($fp); - - if($content === false || $content === ''){ - return $default; + /** + * Keep an unlocked lock file handle open for the next operation (the oldest is closed past the limit) + */ + private function keepLockHandle($path, $fp){ + self::$lockHandles[$path] = ['fp' => $fp, 'pid' => getmypid()]; + if(count(self::$lockHandles) > self::LOCK_HANDLE_LIMIT){ + reset(self::$lockHandles); + $oldest = key(self::$lockHandles); + @fclose(self::$lockHandles[$oldest]['fp']); + unset(self::$lockHandles[$oldest]); } + } - $data = json_decode($content, true); - return $data !== null ? $data : $default; + /** + * Read the state record (caller holds the lock). If another process wrote the database since this + * process last looked, all cached state of the database is dropped. + */ + private function loadLockState($dbname, $path){ + $fp = self::$lockRegistry[$path]['fp']; + fseek($fp, 0); + $tokens = explode(' ', trim((string)fread($fp, self::LOCK_STATE_SIZE))); + $gen = (int)array_shift($tokens); + $flags = []; + foreach($tokens as $token){ + if($token === 'v' || $token === 'b'){ + $flags[$token] = true; + } elseif($token === 'J'){ + $flags['j']['*'] = true; + } elseif(strncmp($token, 'j=', 2) === 0){ + foreach(explode(',', substr($token, 2)) as $segment){ + if($segment !== ''){ + $flags['j'][$segment] = true; + } + } + } + } + if(!array_key_exists($path, $this->generations) || $this->generations[$path] !== $gen){ + $this->invalidateDbCaches($dbname); + $this->generations[$path] = $gen; + } + self::$lockRegistry[$path]['gen'] = $gen; + self::$lockRegistry[$path]['flags'] = $flags; } /** - * Atomically write a file with exclusive lock - * - * @param string $path File path - * @param mixed $data Data to write (will be JSON encoded) - * @param bool $prettyPrint Use JSON_PRETTY_PRINT - * @return bool Success + * Write the state record of a database held exclusively. The first write of a session also bumps + * the generation, so even a writer that dies half-way makes other processes drop their caches. */ - private function atomicWrite($path, $data, $prettyPrint = false){ - // Ensure directory exists - $dir = dirname($path); - if(!is_dir($dir)){ - mkdir($dir, 0755, true); + private function saveLockState($path){ + if(!self::$lockRegistry[$path]['bumped']){ + self::$lockRegistry[$path]['gen']++; + self::$lockRegistry[$path]['bumped'] = true; + $this->generations[$path] = self::$lockRegistry[$path]['gen']; + } + $flags = self::$lockRegistry[$path]['flags']; + $state = (string)self::$lockRegistry[$path]['gen']; + if(!empty($flags['v'])){ + $state .= ' v'; + } + if(!empty($flags['b'])){ + $state .= ' b'; } + if(!empty($flags['j'])){ + $segments = ' j=' . implode(',', array_keys($flags['j'])); + $state .= (isset($flags['j']['*']) || strlen($state . $segments) > self::LOCK_STATE_SIZE) ? ' J' : $segments; + } + $fp = self::$lockRegistry[$path]['fp']; + fseek($fp, 0); + if(fwrite($fp, str_pad($state, self::LOCK_STATE_SIZE)) !== self::LOCK_STATE_SIZE || !fflush($fp)){ + throw new \RuntimeException("noneDB: cannot write the lock file of database '" . self::$lockRegistry[$path]['dbname'] . "'"); + } + } - $fp = fopen($path, 'cb'); // Create if not exists, open for writing - if($fp === false){ - return false; + /** + * Change the state flag 'b' (write buffer may hold records); caller holds the exclusive lock + */ + private function setBufferFlag($dbname, $on){ + $path = $this->getLockPath($dbname); + if(!isset(self::$lockRegistry[$path]) || !empty(self::$lockRegistry[$path]['flags']['b']) === $on){ + return; } + if($on){ + self::$lockRegistry[$path]['flags']['b'] = true; + } else { + unset(self::$lockRegistry[$path]['flags']['b']); + } + $this->saveLockState($path); + } + + /** + * True when the state record of a locked database is authoritative (see 'v' above) + */ + private function isVerified($dbname){ + $path = $this->getLockPath($dbname); + return !empty(self::$lockRegistry[$path]['flags']['v']); + } - $startTime = microtime(true); - $locked = false; + /** + * After taking the lock: rebuild what a crashed writer left half-updated, and once per database + * look for pre-3.2 leftovers. Both need the exclusive lock and are rare. + */ + private function prepareDb($dbname, $path){ + $flags = self::$lockRegistry[$path]['flags']; + if(!empty($flags['v']) && empty($flags['j'])){ + return; + } + $this->withLock($dbname, true, function() use ($dbname, $path){ + // Re-read under the exclusive lock: another process may have done it meanwhile + if(empty(self::$lockRegistry[$path]['flags']['v'])){ + $this->verifyDb($dbname, $path); + } + if(!empty(self::$lockRegistry[$path]['flags']['j'])){ + $this->recoverPendingIndexes($dbname, $path); + } + }); + } - // Try to acquire exclusive lock with timeout - while(!$locked && (microtime(true) - $startTime) < $this->lockTimeout){ - $locked = flock($fp, LOCK_EX | LOCK_NB); - if(!$locked){ - usleep($this->lockRetryDelay); + /** + * One-time check of a database whose state record isn't verified yet (created before v3.2, or its + * lock file was deleted): pick up pre-3.2 write buffers and store the index list, so neither has to + * be looked for again + */ + private function verifyDb($dbname, $path){ + if($this->hasBuffer($this->getBufferPath($dbname)) || !empty($this->flushingFiles($dbname)) || !empty($this->legacyShardBufferFiles($dbname))){ + self::$lockRegistry[$path]['flags']['b'] = true; + } + $manifestPath = $this->getIndexManifestPath($dbname); + clearstatcache(true, $manifestPath); + if(!is_file($manifestPath)){ + unset($this->fieldIndexCache[$manifestPath]); + $manifest = $this->deriveIndexManifest($dbname); + if(!empty($manifest['fields']) || !empty($manifest['spatial'])){ + $this->saveIndexManifest($dbname, $manifest); } } + self::$lockRegistry[$path]['flags']['v'] = true; + $this->saveLockState($path); + } - if(!$locked){ - // Fallback: blocking lock as last resort - $locked = flock($fp, LOCK_EX); + /** + * Index journal: called before a segment's data changes. Needed when the database has secondary + * indexes, or meta counters (sharded) that follow the data. + */ + private function markIndexesPending($dbname, $shardId){ + $path = $this->getLockPath($dbname); + if(!isset(self::$lockRegistry[$path])){ + return; + } + $manifest = $this->getIndexManifest($dbname); + if($shardId === null && empty($manifest['fields']) && empty($manifest['spatial'])){ + return; } + $segment = $shardId === null ? 'main' : (string)$shardId; + if(isset(self::$lockRegistry[$path]['pending'][$segment])){ + return; + } + self::$lockRegistry[$path]['pending'][$segment] = true; + self::$lockRegistry[$path]['flags']['j'][$segment] = true; + $this->saveLockState($path); + } - if(!$locked){ - fclose($fp); - return false; + /** + * Rebuild field and spatial indexes of the segments a crashed writer left half-updated + * (caller holds the exclusive lock) + */ + private function recoverPendingIndexes($dbname, $path){ + $pending = self::$lockRegistry[$path]['flags']['j'] ?? []; + $known = $this->listSegments($dbname); + $segments = isset($pending['*']) ? $known : array_map(function($segment){ + return $segment === 'main' ? null : (int)$segment; + }, array_keys($pending)); + $manifest = $this->getIndexManifest($dbname); + foreach($segments as $shardId){ + if(!in_array($shardId, $known, true)){ + continue; + } + $this->repairSegmentIndex($dbname, $shardId); + foreach($manifest['fields'] as $field){ + $this->writeFieldIndex($dbname, $field, $this->buildSegmentFieldIndex($dbname, $shardId, $field), $shardId); + } + foreach($manifest['spatial'] as $field){ + $index = $this->buildSegmentSpatialIndex($dbname, $field, $shardId, + isset($manifest['spatialOptions'][$field]['nodeSize']) ? ['nodeSize' => $manifest['spatialOptions'][$field]['nodeSize']] : []); + $this->writeSpatialIndex($dbname, $field, $index, $shardId); + if($shardId !== null){ + $this->updateGlobalSpatialShard($dbname, $field, $shardId, $index); + } + } } + if($this->isSharded($dbname)){ + foreach($manifest['fields'] as $field){ + $this->rebuildGlobalFieldIndex($dbname, $field); + } + // A write may have died between its data and its meta update + $this->recountMeta($dbname); + // A migration may have died after switching to shards but before removing the old file + $legacy = $this->getDataPath($dbname); + clearstatcache(true, $legacy); + if(file_exists($legacy)){ + @unlink($legacy . '.jidx'); + @unlink($legacy); + $this->markFileNotExists($legacy); + unset($this->indexCache[$legacy . '.jidx'], $this->jsonlFormatCache[$legacy]); + foreach($manifest['fields'] as $field){ + $this->deleteFieldIndexFile($dbname, $field, null); + } + foreach($manifest['spatial'] as $field){ + @unlink($this->getSpatialIndexPath($dbname, $field, null)); + } + } + } + unset(self::$lockRegistry[$path]['flags']['j']); + $this->saveLockState($path); + } - try { - ftruncate($fp, 0); - rewind($fp); - $json = $prettyPrint - ? json_encode($data, JSON_PRETTY_PRINT) - : json_encode($data); - $written = fwrite($fp, $json); - fflush($fp); - return $written !== false; - } finally { - flock($fp, LOCK_UN); - fclose($fp); + /** + * flock with timeout (lockTimeout seconds; 0 waits forever) + * @throws \RuntimeException on timeout + */ + private function acquireFlock($fp, $mode, $dbname){ + if($this->lockTimeout <= 0){ + if(!flock($fp, $mode)){ + throw new \RuntimeException("noneDB: could not lock database '{$dbname}'"); + } + return; + } + $start = microtime(true); + while(!flock($fp, $mode | LOCK_NB)){ + if((microtime(true) - $start) >= $this->lockTimeout){ + throw new \RuntimeException("noneDB: timed out after {$this->lockTimeout}s waiting for the lock on database '{$dbname}'"); + } + usleep($this->lockRetryDelay); } } /** - * Atomically modify a file: read, apply callback, write - * This is the key method that prevents race conditions - * - * @param string $path File path - * @param callable $modifier Function that receives current data and returns modified data - * @param mixed $default Default value if file doesn't exist - * @param bool $prettyPrint Use JSON_PRETTY_PRINT for output - * @return array ['success' => bool, 'data' => modified data, 'error' => string|null] + * First write of an exclusive session: bump the generation right away (see saveLockState) */ - private function atomicModify($path, callable $modifier, $default = null, $prettyPrint = false){ - // Ensure directory exists - $dir = dirname($path); - if(!is_dir($dir)){ - mkdir($dir, 0755, true); + private function noteWrite(){ + foreach(self::$lockRegistry as $path => $entry){ + if($entry['mode'] === LOCK_EX && !$entry['bumped']){ + $this->saveLockState($path); + } } + } - // Open with c+ mode: read/write, create if not exists - $fp = fopen($path, 'c+b'); - if($fp === false){ - return ['success' => false, 'data' => null, 'error' => 'Failed to open file']; + /** + * Drop every cached entry that belongs to a database (index, meta, field/spatial indexes, file-exists) + */ + private function invalidateDbCaches($dbname){ + $prefix = $this->dbDir . $this->hashDBName($dbname); + $len = strlen($prefix); + $caches = [ + &$this->indexCache, &$this->shardedCache, &$this->metaCache, &$this->metaCacheTime, + &$this->jsonlFormatCache, &$this->fieldIndexCache, &$this->spatialIndexCache, + &$this->globalSpatialIndexCache, + ]; + foreach($caches as &$cache){ + foreach(array_keys($cache) as $key){ + if(strncmp((string)$key, $prefix, $len) === 0){ + unset($cache[$key]); + } + } + } + unset($cache); + foreach(array_keys(self::$staticFileExistsCache) as $key){ + if(strncmp((string)$key, $prefix, $len) === 0){ + unset(self::$staticFileExistsCache[$key]); + } } + foreach(array_keys(self::$bufferCounts) as $key){ + if(strncmp((string)$key, $prefix, $len) === 0){ + unset(self::$bufferCounts[$key]); + } + } + } - $startTime = microtime(true); - $locked = false; + // ========================================== + // ATOMIC FILE OPERATIONS + // ========================================== - // Try to acquire exclusive lock with timeout - while(!$locked && (microtime(true) - $startTime) < $this->lockTimeout){ - $locked = flock($fp, LOCK_EX | LOCK_NB); - if(!$locked){ - usleep($this->lockRetryDelay); - } + /** + * Encode JSON, returning false instead of producing invalid output + * @return string|false + */ + private function encodeJson($data, $prettyPrint = false, $extraFlags = 0){ + $flags = JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | $extraFlags; + if($prettyPrint){ + $flags |= JSON_PRETTY_PRINT; + } + try { + return json_encode($data, $flags | JSON_THROW_ON_ERROR); + } catch(\JsonException $e){ + return false; } + } - if(!$locked){ - // Fallback: blocking lock as last resort - $locked = flock($fp, LOCK_EX); + /** + * Encode one JSONL record line (without the trailing newline) + * @throws \InvalidArgumentException when the record can't be represented as JSON (invalid UTF-8, NAN, INF, ...) + */ + private function encodeRecord(array $record){ + // Keep floats floats: 2.0 / -0.0 are stored as 2.0 / -0.0, not as the ints 2 / 0 + $json = $this->encodeJson($record, false, JSON_PRESERVE_ZERO_FRACTION); + if($json === false){ + throw new \InvalidArgumentException('noneDB: record cannot be encoded as JSON (invalid UTF-8, NAN/INF or unsupported value): ' . json_last_error_msg()); } + return $json; + } - if(!$locked){ - fclose($fp); - return ['success' => false, 'data' => null, 'error' => 'Failed to acquire lock']; + /** + * Write a whole file atomically: temp file in the same directory + rename. + * Readers see either the old or the new content, never a partial file. + * @return bool + */ + private function writeFileAtomic($path, $content){ + $this->noteWrite(); + $dir = dirname($path); + if(!is_dir($dir) && !@mkdir($dir, 0755, true) && !is_dir($dir)){ + return false; + } + $tmp = $path . '.tmp' . bin2hex(random_bytes(4)); + $written = @file_put_contents($tmp, $content); + if($written === false || $written !== strlen($content)){ + @unlink($tmp); + return false; + } + if(!@rename($tmp, $path)){ + @unlink($tmp); + return false; } + clearstatcache(true, $path); + return true; + } + /** + * Append bytes to a file. Returns the offset the bytes start at, or false. + * A partial write is rolled back so the file never ends in a torn line. + * @return int|false + */ + private function appendToFile($path, $bytes){ + $this->noteWrite(); + $fp = @fopen($path, 'c+b'); + if($fp === false){ + return false; + } try { - // Read current content while holding lock - clearstatcache(true, $path); - $size = filesize($path); - - if($size === false || $size === 0){ - $currentData = $default; - } else { - rewind($fp); - $content = fread($fp, $size); - $currentData = ($content !== false && $content !== '') - ? json_decode($content, true) - : $default; - - if($currentData === null && $content !== 'null'){ - $currentData = $default; + if(fseek($fp, 0, SEEK_END) !== 0){ + return false; + } + $offset = ftell($fp); + if($offset === false){ + return false; + } + $length = strlen($bytes); + $done = 0; + while($done < $length){ + $n = fwrite($fp, $done === 0 ? $bytes : substr($bytes, $done)); + if($n === false || $n === 0){ + ftruncate($fp, $offset); + return false; } + $done += $n; } - - // Apply modification - $newData = $modifier($currentData); - - // Write modified data - ftruncate($fp, 0); - rewind($fp); - $json = $prettyPrint - ? json_encode($newData, JSON_PRETTY_PRINT) - : json_encode($newData); - $written = fwrite($fp, $json); fflush($fp); - - if($written === false){ - return ['success' => false, 'data' => null, 'error' => 'Failed to write data']; - } - - return ['success' => true, 'data' => $newData, 'error' => null]; + return $offset; } finally { - flock($fp, LOCK_UN); fclose($fp); + clearstatcache(true, $path); + } + } + + /** + * Read and decode a JSON file + * + * @param string $path File path + * @param mixed $default Value returned when the file doesn't exist or is empty + * @return mixed Decoded data, $default, or null when the file is corrupt + */ + private function atomicRead($path, $default = null){ + clearstatcache(true, $path); + if(!is_file($path)){ + return $default; + } + $content = @file_get_contents($path); + if($content === false || $content === ''){ + return $default; + } + $data = json_decode($content, true); + if($data === null && json_last_error() !== JSON_ERROR_NONE){ + return null; // corrupted + } + return $data !== null ? $data : $default; + } + + /** + * Encode and write a JSON file atomically. Nothing is touched when encoding fails. + * + * @param string $path File path + * @param mixed $data Data to write (will be JSON encoded) + * @param bool $prettyPrint Use JSON_PRETTY_PRINT + * @return bool Success + */ + private function atomicWrite($path, $data, $prettyPrint = false){ + $json = $this->encodeJson($data, $prettyPrint); + if($json === false){ + return false; } + return $this->writeFileAtomic($path, $json); } // ========================================== - // SHARDING HELPER METHODS + // STORAGE CORE (v3.2.0) // ========================================== + // + // A database is one "segment" when it is not sharded ({base}.nonedb + {base}.nonedb.jidx) or one + // segment per shard ({base}_sN.nonedb + .jidx) listed in {base}.nonedb.meta. Segment ids are null + // (non-sharded) or the shard id. Every write runs under the database's exclusive lock (withLock), + // reads under a shared lock, so index read-modify-write cycles are never interleaved. + // + // Data files are append-only JSONL. Every record line carries its "key"; deletions append a + // tombstone line {"__nonedb_deleted":key}. The .jidx index maps key => [offset, length] and stores + // "s", the data size it covers. If the data file is shorter than "s" (e.g. a crash between the two + // renames of a compaction) or the index is missing/corrupt, the index is rebuilt from the data. /** * Get the file path for a specific shard @@ -736,9 +1520,7 @@ private function atomicModify($path, callable $modifier, $default = null, $prett * @return string */ private function getShardPath($dbname, $shardId){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - return $this->dbDir . $hash . "-" . $dbname . "_s" . $shardId . ".nonedb"; + return $this->dbBase($dbname) . "_s" . $shardId . ".nonedb"; } /** @@ -747,9 +1529,14 @@ private function getShardPath($dbname, $shardId){ * @return string */ private function getMetaPath($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - return $this->dbDir . $hash . "-" . $dbname . ".nonedb.meta"; + return $this->dbBase($dbname) . ".nonedb.meta"; + } + + /** + * Data file of a segment (null = non-sharded database) + */ + private function segDataPath($dbname, $shardId){ + return $shardId === null ? $this->getDataPath($dbname) : $this->getShardPath($dbname, $shardId); } /** @@ -758,2973 +1545,1993 @@ private function getMetaPath($dbname){ * @return bool */ private function isSharded($dbname){ - $dbname = $this->sanitizeDbName($dbname); - - // Check cache first - if(isset($this->shardedCache[$dbname])){ - return $this->shardedCache[$dbname]; + $metaPath = $this->getMetaPath($dbname); + if(isset($this->shardedCache[$metaPath])){ + return $this->shardedCache[$metaPath]; } - // Use file_exists directly - shardedCache handles caching - $result = file_exists($this->getMetaPath($dbname)); - $this->shardedCache[$dbname] = $result; + clearstatcache(true, $metaPath); + $result = file_exists($metaPath); + $this->shardedCache[$metaPath] = $result; return $result; } - private function invalidateShardedCache($dbname){ - $dbname = $this->sanitizeDbName($dbname); - unset($this->shardedCache[$dbname]); + /** + * True when the database has data (non-sharded file or sharded meta) + */ + private function dbExists($dbname){ + return $this->cachedFileExists($this->getDataPath($dbname)) || $this->isSharded($dbname); } /** - * Read shard metadata with atomic locking + * Read shard metadata * @param string $dbname * @return array|null */ private function readMeta($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $path = $this->getMetaPath($dbname); - return $this->atomicRead($path, null); + $meta = $this->atomicRead($this->getMetaPath($dbname), null); + return is_array($meta) && isset($meta['shards']) && is_array($meta['shards']) ? $meta : null; } /** - * Get cached meta data with TTL support - * Avoids repeated file reads for frequently accessed meta + * Get cached meta data. The cache is dropped whenever another process writes the + * database (generation check in withLock), so no TTL is needed. * @param string $dbname * @param bool $forceRefresh Force refresh from disk * @return array|null */ private function getCachedMeta($dbname, $forceRefresh = false){ - $dbname = $this->sanitizeDbName($dbname); - $now = time(); + $metaPath = $this->getMetaPath($dbname); - if(!$forceRefresh && isset($this->metaCache[$dbname])){ - $cacheAge = $now - ($this->metaCacheTime[$dbname] ?? 0); - if($cacheAge < $this->metaCacheTTL){ - return $this->metaCache[$dbname]; - } + if(!$forceRefresh && isset($this->metaCache[$metaPath])){ + return $this->metaCache[$metaPath]; } $meta = $this->readMeta($dbname); if($meta !== null){ - $this->metaCache[$dbname] = $meta; - $this->metaCacheTime[$dbname] = $now; + $this->metaCache[$metaPath] = $meta; + return $meta; } - return $meta; - } - - /** - * Invalidate meta cache for a database - * @param string $dbname - */ - private function invalidateMetaCache($dbname){ - $dbname = $this->sanitizeDbName($dbname); - unset($this->metaCache[$dbname]); - unset($this->metaCacheTime[$dbname]); + unset($this->metaCache[$metaPath]); + return null; } /** - * Write shard metadata with atomic locking + * Write shard metadata (caller holds the exclusive lock) * @param string $dbname * @param array $meta * @return bool + * @throws \RuntimeException when the meta file can't be written */ private function writeMeta($dbname, $meta){ - $dbname = $this->sanitizeDbName($dbname); $path = $this->getMetaPath($dbname); - $result = $this->atomicWrite($path, $meta, true); - if($result){ - $this->invalidateMetaCache($dbname); + if(!$this->atomicWrite($path, $meta, true)){ + throw new \RuntimeException("noneDB: failed to write metadata of database '{$dbname}'"); } - return $result; + $this->metaCache[$path] = $meta; + $this->shardedCache[$path] = true; + $this->markFileExists($path); + return true; } /** - * Atomically modify shard metadata - * @param string $dbname - * @param callable $modifier - * @return array ['success' => bool, 'data' => modified meta, 'error' => string|null] + * Records per shard of a database (existing sharded databases keep the size stored in their meta) */ - private function modifyMeta($dbname, callable $modifier){ - $dbname = $this->sanitizeDbName($dbname); - $path = $this->getMetaPath($dbname); - $result = $this->atomicModify($path, $modifier, null, true); - if($result['success']){ - $this->invalidateMetaCache($dbname); + private function shardSizeOf($dbname){ + $meta = $this->isSharded($dbname) ? $this->getCachedMeta($dbname) : null; + if($meta !== null && isset($meta['shardSize']) && (int)$meta['shardSize'] > 0){ + return (int)$meta['shardSize']; } - return $result; + return $this->shardSize; } /** - * Get data from a specific shard with atomic locking - * Auto-migrates to JSONL format if needed (v3.0.0) - * @param string $dbname - * @param int $shardId - * @return array Returns {"data": [...]} format for backward compatibility + * Segment ids of a database: [null] when not sharded, the shard ids otherwise, [] if it doesn't exist */ - private function getShardData($dbname, $shardId){ - $path = $this->getShardPath($dbname, $shardId); - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname, $shardId); - - $jsonlIndex = $this->readJsonlIndex($dbname, $shardId); - if($jsonlIndex === null){ - return array("data" => []); - } - - // Read all records from JSONL - $allRecords = $this->readAllJsonl($path, $jsonlIndex); - - // Convert to {"data": [...]} format where array index is local key - $data = []; - foreach($allRecords as $record){ - if($record !== null && isset($record['key'])){ - $localKey = $record['key'] % $this->shardSize; - $data[$localKey] = $record; + private function listSegments($dbname){ + if($this->isSharded($dbname)){ + $meta = $this->getCachedMeta($dbname); + if($meta === null){ + throw new \RuntimeException("noneDB: metadata of sharded database '{$dbname}' is unreadable"); } + return array_map('intval', array_column($meta['shards'], 'id')); } - return array("data" => $data); - } - - /** - * Write data to a specific shard with atomic locking - * @param string $dbname - * @param int $shardId - * @param array $data - * @return bool - */ - private function writeShardData($dbname, $shardId, $data){ - $path = $this->getShardPath($dbname, $shardId); - return $this->atomicWrite($path, $data); + return $this->cachedFileExists($this->getDataPath($dbname)) ? [null] : []; } /** - * Atomically modify shard data - * @param string $dbname - * @param int $shardId - * @param callable $modifier - * @return array ['success' => bool, 'data' => modified data, 'error' => string|null] + * Segment that holds a key, or false if that shard doesn't exist + * @return int|null|false */ - private function modifyShardData($dbname, $shardId, callable $modifier){ - $path = $this->getShardPath($dbname, $shardId); - return $this->atomicModify($path, $modifier, array("data" => [])); + private function segmentForKey($dbname, $key){ + if(!$this->isSharded($dbname)){ + return null; + } + $shardId = intdiv($key, $this->shardSizeOf($dbname)); + return in_array($shardId, $this->listSegments($dbname), true) ? $shardId : false; } // ========================================== - // PRIMARY KEY INDEX SYSTEM (v2.3.0) + // JSONL SEGMENT INDEX // ========================================== /** - * Get path to index file for a database - * Index provides O(1) key lookups instead of O(n) shard scans + * Get JSONL index path * @param string $dbname + * @param int|null $shardId Null for non-sharded * @return string */ - private function getIndexPath($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - return $this->dbDir . $hash . "-" . $dbname . ".nonedb.idx"; + private function getJsonlIndexPath($dbname, $shardId = null){ + return $this->segDataPath($dbname, $shardId) . ".jidx"; + } + + private function newJsonlIndex(){ + return [ + 'v' => 3, + 'format' => 'jsonl', + 'created' => time(), + 'n' => 0, + 'd' => 0, + 't' => 0, + 's' => 0, + 'o' => [] + ]; } /** - * Read index file with caching + * Read a segment's JSONL index (byte offset map), repairing it when it is missing, corrupt or + * doesn't match the data file. * @param string $dbname - * @return array|null + * @param int|null $shardId + * @return array|null Null when the segment doesn't exist */ - private function readIndex($dbname){ - $dbname = $this->sanitizeDbName($dbname); - - // Check runtime cache first - if(isset($this->indexCache[$dbname])){ - return $this->indexCache[$dbname]; + private function readJsonlIndex($dbname, $shardId = null){ + $dataPath = $this->segDataPath($dbname, $shardId); + if(!$this->cachedFileExists($dataPath)){ + return null; + } + $path = $dataPath . '.jidx'; + // A cached index was checked when it was loaded, and is dropped as soon as another process writes + if(isset($this->indexCache[$path])){ + return $this->indexCache[$path]; } - $path = $this->getIndexPath($dbname); + // Missing or unreadable (pre-3.0 files have no index either): repaired or converted $index = $this->atomicRead($path, null); + if(!is_array($index) || !isset($index['o']) || !is_array($index['o'])){ + return $this->repairSegmentIndex($dbname, $shardId); + } - if($index !== null){ - $this->indexCache[$dbname] = $index; + // The data file can only grow between index writes; shorter means the index belongs to other data + if(isset($index['s'])){ + clearstatcache(true, $dataPath); + $size = @filesize($dataPath); + if($size !== false && $size < $index['s']){ + return $this->repairSegmentIndex($dbname, $shardId); + } } + $this->indexCache[$path] = $index; return $index; } /** - * Write index file and update cache + * Write a segment's JSONL index (caller holds the exclusive lock) * @param string $dbname * @param array $index + * @param int|null $shardId * @return bool + * @throws \RuntimeException when the index can't be written */ - private function writeIndex($dbname, $index){ - $dbname = $this->sanitizeDbName($dbname); + private function writeJsonlIndex($dbname, $index, $shardId = null){ + $dataPath = $this->segDataPath($dbname, $shardId); + $path = $dataPath . '.jidx'; + clearstatcache(true, $dataPath); + $size = @filesize($dataPath); + $index['s'] = $size === false ? 0 : $size; $index['updated'] = time(); - $path = $this->getIndexPath($dbname); - $result = $this->atomicWrite($path, $index, false); - - if($result){ - $this->indexCache[$dbname] = $index; + if(!$this->atomicWrite($path, $index)){ + unset($this->indexCache[$path]); + throw new \RuntimeException("noneDB: failed to write index of database '{$dbname}'"); } - - return $result; - } - - /** - * Invalidate index cache - * @param string $dbname - */ - private function invalidateIndexCache($dbname){ - $dbname = $this->sanitizeDbName($dbname); - unset($this->indexCache[$dbname]); + $this->indexCache[$path] = $index; + return true; } /** - * Build index from existing database data - * Called automatically on first key-based lookup if index doesn't exist - * @param string $dbname + * Rebuild a segment index from its data file (exclusive lock, upgrading a shared one). + * Re-checks first: another process may have repaired it while we waited for the lock. + * @param bool $force Rebuild even if the index on disk looks consistent * @return array|null */ - private function buildIndex($dbname){ - $dbname = $this->sanitizeDbName($dbname); - - $index = [ - 'version' => 1, - 'created' => time(), - 'updated' => time(), - 'totalRecords' => 0, - 'entries' => [] - ]; - - if($this->isSharded($dbname)){ - $meta = $this->getCachedMeta($dbname); - if($meta === null){ + private function repairSegmentIndex($dbname, $shardId, $force = false){ + return $this->withLock($dbname, true, function() use ($dbname, $shardId, $force){ + $dataPath = $this->segDataPath($dbname, $shardId); + $path = $dataPath . '.jidx'; + unset($this->indexCache[$path]); + clearstatcache(true, $dataPath); + if(!file_exists($dataPath)){ return null; } - - $index['sharded'] = true; - - foreach($meta['shards'] as $shard){ - $shardData = $this->getShardData($dbname, $shard['id']); - $baseKey = $shard['id'] * $this->shardSize; - - foreach($shardData['data'] as $localKey => $record){ - if($record !== null){ - $globalKey = $baseKey + $localKey; - // Store as [shardId, localKey] for sharded DBs - $index['entries'][(string)$globalKey] = [$shard['id'], $localKey]; - $index['totalRecords']++; + // A pre-3.0 {"data": [...]} file (it has no index) is converted, never scanned as JSONL + $this->ensureJsonlFormat($dbname, $shardId); + if(!$force){ + $index = $this->atomicRead($path, null); + if(is_array($index) && isset($index['o']) && is_array($index['o'])){ + $size = @filesize($dataPath); + if(!isset($index['s']) || ($size !== false && $size >= $index['s'])){ + $this->indexCache[$path] = $index; + return $index; } } } - } else { - $hash = $this->hashDBName($dbname); - $fullDBPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $jsonlIndex = $this->readJsonlIndex($dbname); - if($jsonlIndex === null){ - return null; + $index = $this->rebuildSegmentIndexFromData($dbname, $shardId); + if($shardId !== null){ + $this->recountMeta($dbname); } + return $index; + }); + } - $index['sharded'] = false; + /** + * Scan a data file and rebuild its index: the last line of a key wins, tombstones delete, + * a torn last line (crash during append) is cut off. Caller holds the exclusive lock. + * @return array + */ + private function rebuildSegmentIndexFromData($dbname, $shardId){ + $dataPath = $this->segDataPath($dbname, $shardId); + $old = $this->atomicRead($dataPath . '.jidx', null); - foreach($jsonlIndex['o'] as $key => $location){ - // Store just the key for non-sharded DBs - $index['entries'][(string)$key] = $key; - $index['totalRecords']++; + $offsets = []; + $lines = 0; + $tombstones = 0; + $maxKey = -1; + $offset = 0; + $fp = @fopen($dataPath, 'rb'); + if($fp !== false){ + while(($line = fgets($fp)) !== false){ + if(substr($line, -1) !== "\n"){ + break; // torn tail + } + $json = rtrim($line, "\r\n"); + $record = json_decode($json, true); + if(is_array($record)){ + if(array_key_exists('__nonedb_deleted', $record) && count($record) === 1){ + $k = (int)$record['__nonedb_deleted']; + unset($offsets[$k]); + $maxKey = max($maxKey, $k); + $tombstones++; + } elseif(isset($record['key']) && is_int($record['key'])){ + $offsets[$record['key']] = [$offset, strlen($json)]; + $maxKey = max($maxKey, $record['key']); + } + } + $offset += strlen($line); + $lines++; + } + fclose($fp); + } + clearstatcache(true, $dataPath); + if(file_exists($dataPath) && filesize($dataPath) > $offset){ + $fp = fopen($dataPath, 'c+b'); + if($fp !== false){ + ftruncate($fp, $offset); + fclose($fp); } } - $this->writeIndex($dbname, $index); - return $index; + ksort($offsets, SORT_NUMERIC); + $index = $this->newJsonlIndex(); + if(is_array($old) && isset($old['created'])){ + $index['created'] = $old['created']; + } + $index['o'] = $offsets; + $index['t'] = $tombstones; + $index['d'] = max(0, $lines - count($offsets) - $tombstones); + if($shardId === null){ + $index['n'] = max($maxKey + 1, is_array($old) && isset($old['n']) ? (int)$old['n'] : 0); + } else { + $base = $shardId * $this->shardSizeOf($dbname); + $index['n'] = $maxKey >= $base ? $maxKey - $base + 1 : 0; + } + $this->writeJsonlIndex($dbname, $index, $shardId); + return $this->indexCache[$this->getJsonlIndexPath($dbname, $shardId)]; } /** - * Get existing index or build it if missing - * @param string $dbname - * @return array|null + * Recompute per-shard and total counters in the meta file from the shard indexes */ - private function getOrBuildIndex($dbname){ - if(!$this->indexEnabled){ - return null; + private function recountMeta($dbname){ + $meta = $this->getCachedMeta($dbname, true); + if($meta === null){ + return; } - - $index = $this->readIndex($dbname); - if($index === null){ - $index = $this->buildIndex($dbname); + $size = (int)$meta['shardSize']; + $total = 0; + $maxKey = -1; + foreach($meta['shards'] as $i => $shard){ + $index = $this->readJsonlIndex($dbname, (int)$shard['id']); + $count = $index === null ? 0 : count($index['o']); + $meta['shards'][$i]['count'] = $count; + $total += $count; + if($index !== null && !empty($index['o'])){ + $maxKey = max($maxKey, max(array_keys($index['o']))); + } } - return $index; + $meta['totalRecords'] = $total; + $meta['nextKey'] = max((int)($meta['nextKey'] ?? 0), $maxKey + 1); + $this->writeMeta($dbname, $meta); } /** - * Update index after insert operation - * @param string $dbname - * @param array $keys Array of globalKey => localKey (or [shardId, localKey] for sharded) - * @param int|null $shardId Shard ID for sharded databases + * Detect if a data file is in JSONL format (vs the pre-3.0 single JSON document {"data": [...]}) + * @param string $path + * @return bool True if JSONL format */ - private function updateIndexOnInsert($dbname, array $keys, $shardId = null){ - if(!$this->indexEnabled){ - return; + private function isJsonlFormat($path){ + if(isset($this->jsonlFormatCache[$path])){ + return $this->jsonlFormatCache[$path]; } - - $index = $this->readIndex($dbname); - if($index === null){ - return; // No index yet, will be built on first read + if(!$this->cachedFileExists($path)){ + return true; } - $isSharded = $index['sharded'] ?? false; - - foreach($keys as $globalKey => $localKey){ - if($isSharded && $shardId !== null){ - $index['entries'][(string)$globalKey] = [$shardId, $localKey]; - } else { - $index['entries'][(string)$globalKey] = $localKey; + // A JSONL database always has its .jidx (or the index can be rebuilt from the lines). + // Only a file without index that decodes as exactly {"data": [...]} is the old v2 format; + // a JSONL record always has a "key" field, so a record whose first field is "data" is never mistaken. + $isJsonl = true; + if(!file_exists($path . '.jidx')){ + $content = @file_get_contents($path); + if($content !== false && $content !== ''){ + $decoded = json_decode($content, true); + if(is_array($decoded) && array_keys($decoded) === ['data'] && is_array($decoded['data'])){ + $isJsonl = false; + } } } - $index['totalRecords'] = count($index['entries']); - $this->writeIndex($dbname, $index); + $this->jsonlFormatCache[$path] = $isJsonl; + return $isJsonl; } /** - * Update index after delete operation + * Ensure a segment is in JSONL format (auto-migrates the pre-3.0 format) * @param string $dbname - * @param array $deletedKeys Array of deleted global keys + * @param int|null $shardId + * @return bool */ - private function updateIndexOnDelete($dbname, array $deletedKeys){ - if(!$this->indexEnabled){ - return; - } - - $index = $this->readIndex($dbname); - if($index === null){ - return; - } - - foreach($deletedKeys as $key){ - unset($index['entries'][(string)$key]); + private function ensureJsonlFormat($dbname, $shardId = null){ + $path = $this->segDataPath($dbname, $shardId); + if($this->isJsonlFormat($path)){ + return true; } - - $index['totalRecords'] = count($index['entries']); - $this->writeIndex($dbname, $index); + return $this->migrateToJsonl($path, $dbname, $shardId); } /** - * Find record by key using index (O(1) lookup) - * This is the core optimization - avoids loading entire shard - * @param string $dbname - * @param mixed $keyFilter Single key or array of keys - * @param array $index The index data - * @return array Found records with 'key' field added + * Migrate a pre-3.0 {"data": [...]} file to JSONL (exclusive lock) + * @param string $path Source file path + * @param string $dbname Database name + * @param int|null $shardId Shard ID or null for non-sharded + * @return bool Success */ - private function findByKeyWithIndex($dbname, $keyFilter, $index){ - $result = []; - $keys = is_array($keyFilter) ? $keyFilter : [$keyFilter]; - $isSharded = $index['sharded'] ?? false; - - foreach($keys as $globalKey){ - $globalKey = (int)$globalKey; - $keyStr = (string)$globalKey; - - if(!isset($index['entries'][$keyStr])){ - continue; // Key doesn't exist + private function migrateToJsonl($path, $dbname, $shardId = null){ + return $this->withLock($dbname, true, function() use ($path, $dbname, $shardId){ + unset($this->jsonlFormatCache[$path]); + if($this->isJsonlFormat($path)){ + return true; // migrated by someone else meanwhile } - $entry = $index['entries'][$keyStr]; + $data = json_decode((string)@file_get_contents($path), true); + if(!is_array($data) || !isset($data['data']) || !is_array($data['data'])){ + return false; + } - try { - if($isSharded){ - // Entry is [shardId, localKey] - use JSONL direct lookup for O(1) - // Note: JSONL shards store global keys, so use globalKey not localKey - $shardId = $entry[0]; - - $records = $this->findByKeyJsonl($dbname, $globalKey, $shardId); - if($records !== null && !empty($records)){ - $result = array_merge($result, $records); - } - } else { - // Entry is the key - use JSONL direct lookup - $records = $this->findByKeyJsonl($dbname, $globalKey); - if($records !== null && !empty($records)){ - $result = array_merge($result, $records); - } + $index = $this->newJsonlIndex(); + $base = ($shardId !== null) ? ($shardId * $this->shardSizeOf($dbname)) : 0; + $buffer = ''; + $offset = 0; + $slots = 0; + foreach($data['data'] as $localKey => $record){ + $slots = max($slots, (int)$localKey + 1); + if(!is_array($record)){ + continue; // deleted (null) or invalid slot: not stored, not garbage } - } catch(Exception $e){ - // Index might be corrupted, invalidate it - $this->invalidateIndexCache($dbname); - @unlink($this->getIndexPath($dbname)); - return null; // Signal to fall back to full scan + $key = $base + (int)$localKey; + unset($record['key']); + $record['key'] = $key; + $json = $this->encodeJson($record); + if($json === false){ + continue; + } + $index['o'][$key] = [$offset, strlen($json)]; + $buffer .= $json . "\n"; + $offset += strlen($json) + 1; } - } + $index['n'] = $slots; - return $result; + if(!$this->writeFileAtomic($path, $buffer)){ + return false; + } + $this->jsonlFormatCache[$path] = true; + $this->writeJsonlIndex($dbname, $index, $shardId); + return true; + }); } // ========================================== - // PUBLIC INDEX API (v2.3.0) + // RECORD I/O // ========================================== /** - * Enable or disable indexing - * @param bool $enable + * Batch read multiple JSONL records (one open, 64KB buffered sequential reads) + * @param string $path File path + * @param array $offsets [key => [offset, length], ...] + * @param bool $mismatch Set to true when a line doesn't hold the expected record + * @return array [key => record, ...] */ - public function enableIndexing($enable = true){ - $this->indexEnabled = (bool)$enable; - } + private function readJsonlRecordsBatch($path, $offsets, &$mismatch = false){ + $mismatch = false; + if(empty($offsets)){ + return []; + } - /** - * Check if indexing is enabled - * @return bool - */ - public function isIndexingEnabled(){ - return $this->indexEnabled; - } + $handle = @fopen($path, 'rb'); + if($handle === false){ + $mismatch = true; + return []; + } - /** - * Manually rebuild index for a database - * @param string $dbname - * @return array ['success' => bool, 'totalRecords' => int, 'time' => float] - */ - public function rebuildIndex($dbname){ - $start = microtime(true); - $this->invalidateIndexCache($dbname); - @unlink($this->getIndexPath($dbname)); + $records = []; + $bufferSize = 65536; + $buffer = ''; + $bufferStart = -1; + $bufferEnd = -1; + + // Read in file order. Appends keep the index in that order; updated records moved to the end. + $positions = []; + $inOrder = true; + $previous = -1; + foreach($offsets as $key => $location){ + $positions[$key] = $location[0]; + if($location[0] < $previous){ + $inOrder = false; + } + $previous = $location[0]; + } + if(!$inOrder){ + asort($positions, SORT_NUMERIC); + } - $index = $this->buildIndex($dbname); - $elapsed = (microtime(true) - $start) * 1000; + foreach($positions as $key => $position){ + $offset = (int)$position; + $length = (int)$offsets[$key][1]; + if($length <= 0){ + $mismatch = true; + continue; + } - if($index === null){ - return ['success' => false, 'error' => 'Failed to build index']; + if($offset >= $bufferStart && ($offset + $length) <= $bufferEnd){ + $json = substr($buffer, $offset - $bufferStart, $length); + } else { + fseek($handle, $offset, SEEK_SET); + $buffer = (string)fread($handle, max($length, $bufferSize)); + $bufferStart = $offset; + $bufferEnd = $offset + strlen($buffer); + $json = substr($buffer, 0, $length); + } + + $record = json_decode((string)$json, true); + if(!is_array($record) || !isset($record['key']) || $record['key'] !== (int)$key){ + $mismatch = true; + continue; + } + $records[$key] = $record; } - return [ - 'success' => true, - 'totalRecords' => $index['totalRecords'], - 'time' => round($elapsed, 2) . 'ms' - ]; + fclose($handle); + return $records; } /** - * Get index information for a database - * @param string $dbname - * @return array|null + * Read records of one segment, verified against their keys. A stale or corrupt index is + * rebuilt from the data file once. + * @param array|null $keys Keys to read (null = all) + * @return array [key => record] sorted by key */ - public function getIndexInfo($dbname){ - $index = $this->readIndex($dbname); - if($index === null){ - return null; + private function readSegmentRecords($dbname, $shardId, $keys = null){ + for($attempt = 0; $attempt < 2; $attempt++){ + $index = $attempt === 0 + ? $this->readJsonlIndex($dbname, $shardId) + : $this->repairSegmentIndex($dbname, $shardId, true); + if($index === null){ + return []; + } + if($keys === null){ + $offsets = $index['o']; + } else { + $offsets = []; + foreach($keys as $key){ + if(isset($index['o'][$key])){ + $offsets[$key] = $index['o'][$key]; + } + } + } + $mismatch = false; + $records = $this->readJsonlRecordsBatch($this->segDataPath($dbname, $shardId), $offsets, $mismatch); + if(!$mismatch){ + ksort($records, SORT_NUMERIC); + return $records; + } } + throw new \RuntimeException("noneDB: data file of database '{$dbname}' is corrupt and could not be repaired"); + } - return [ - 'exists' => true, - 'version' => $index['version'] ?? 1, - 'created' => $index['created'] ?? null, - 'updated' => $index['updated'] ?? null, - 'totalRecords' => $index['totalRecords'] ?? 0, - 'sharded' => $index['sharded'] ?? false, - 'path' => $this->getIndexPath($dbname) - ]; + /** + * Append records ([key => record]) to a segment and point the index at them. + * Everything is encoded before anything is written, so an unencodable record writes nothing. + * @throws \InvalidArgumentException|\RuntimeException + */ + private function appendSegmentRecords($dbname, $shardId, array &$index, array $records){ + if(empty($records)){ + return; + } + $this->markIndexesPending($dbname, $shardId); + $lines = []; + foreach($records as $key => $record){ + unset($record['key']); + $record['key'] = (int)$key; + $lines[$key] = $this->encodeRecord($record); + } + $buffer = implode("\n", $lines) . "\n"; + $dataPath = $this->segDataPath($dbname, $shardId); + $offset = $this->appendToFile($dataPath, $buffer); + if($offset === false){ + throw new \RuntimeException("noneDB: failed to write data of database '{$dbname}'"); + } + $this->markFileExists($dataPath); + foreach($lines as $key => $json){ + $index['o'][$key] = [$offset, strlen($json)]; + $offset += strlen($json) + 1; + } } /** - * Calculate shard ID from a global key - * @param int $key - * @return int + * Append tombstones for deleted keys, so an index rebuilt from the data won't resurrect them */ - private function getShardIdForKey($key){ - return (int) floor($key / $this->shardSize); + private function appendTombstones($dbname, $shardId, array $keys){ + if(empty($keys)){ + return; + } + $this->markIndexesPending($dbname, $shardId); + $buffer = ''; + foreach($keys as $key){ + $buffer .= '{"__nonedb_deleted":' . (int)$key . "}\n"; + } + if($this->appendToFile($this->segDataPath($dbname, $shardId), $buffer) === false){ + throw new \RuntimeException("noneDB: failed to write data of database '{$dbname}'"); + } } /** - * Calculate local key within a shard - * @param int $globalKey - * @return int + * Create an empty segment (data file + index) + * @param string $dbname + * @param int|null $shardId + * @return bool */ - private function getLocalKey($globalKey){ - return $globalKey % $this->shardSize; + private function createJsonlDatabase($dbname, $shardId = null){ + $path = $this->segDataPath($dbname, $shardId); + clearstatcache(true, $path); + if(!file_exists($path) && @touch($path) === false){ + throw new \RuntimeException("noneDB: cannot create data file of database '{$dbname}'"); + } + $this->markFileExists($path); + $this->jsonlFormatCache[$path] = true; + $index = $this->newJsonlIndex(); + if($shardId === null){ + $existing = $this->atomicRead($path . '.jidx', null); + if(is_array($existing) && isset($existing['o'])){ + return true; + } + } + $this->writeJsonlIndex($dbname, $index, $shardId); + return true; } // ========================================== - // JSONL STORAGE ENGINE (v2.4.0) - // O(1) key lookups with byte offset indexing + // WRITE PATHS (callers hold the exclusive lock) // ========================================== /** - * Detect if a database file is in JSONL format - * JSONL: Each line is a JSON object - * v2: {"data": [...]} - * @param string $path - * @return bool True if JSONL format + * Insert validated records. Assigns keys, writes data + indexes, updates field/spatial indexes + * and migrates to sharded format when the size threshold is reached. + * @param array $items Records without "key" + * @return array ['n' => int] or ['n' => 0, 'error' => string] */ - private function isJsonlFormat($path){ - // Check cache first (includes file existence) - if(isset($this->jsonlFormatCache[$path])){ - return $this->jsonlFormatCache[$path]; + private function insertRecords($dbname, array $items, $startKey = null){ + if(empty($items)){ + return ["n" => 0]; + } + if(!$this->dbExists($dbname)){ + if(!$this->autoCreateDB){ + return ["n" => 0, "error" => "Database does not exist"]; + } + $this->createDBUnlocked($dbname); } - if(!$this->cachedFileExists($path)){ - return false; + // With a start key the records get keys startKey, startKey+1, ... and keys that are already + // stored are skipped, so replaying the same insert after a crash never duplicates anything. + $keyed = null; + $endKey = null; + if($startKey !== null){ + $keyed = []; + foreach(array_values($items) as $i => $item){ + $keyed[(int)$startKey + $i] = $item; + } + $endKey = (int)$startKey + count($items); + foreach(array_keys($this->selectRecords($dbname, ['key' => array_keys($keyed)])) as $existing){ + unset($keyed[$existing]); + } } - $handle = fopen($path, 'rb'); - if($handle === false){ - return false; + if($this->isSharded($dbname)){ + $this->insertShardedRecords($dbname, $items, $keyed, $endKey); + return ["n" => count($items)]; } - // Read first 20 bytes to detect format - $header = fread($handle, 20); - fclose($handle); + $index = $this->readJsonlIndex($dbname, null); + if($index === null){ + $this->createJsonlDatabase($dbname, null); + $index = $this->readJsonlIndex($dbname, null); + } - // v2 format starts with {"data": - // JSONL starts with {"key": or just {" for record - $isJsonl = (strpos($header, '{"data":') === false && strpos($header, '{"data" :') === false); + if($keyed !== null){ + $records = $keyed; + $index['n'] = max((int)$index['n'], $endKey); + } else { + $records = []; + foreach($items as $item){ + $records[(int)$index['n']] = $item; + $index['n']++; + } + } + $this->appendSegmentRecords($dbname, null, $index, $records); + $this->writeJsonlIndex($dbname, $index, null); + $this->afterRecordsChanged($dbname, null, [], $this->withKeys($records)); - $this->jsonlFormatCache[$path] = $isJsonl; - return $isJsonl; + if($this->shardingEnabled && $this->autoMigrate && $index['n'] >= $this->shardSize){ + $this->migrateToSharded($dbname); + } + return ["n" => count($items)]; } /** - * Get JSONL index path - * @param string $dbname - * @param int|null $shardId Null for non-sharded - * @return string + * Insert into a sharded database. Keys come from meta.nextKey; key K lives in shard intdiv(K, shardSize). */ - private function getJsonlIndexPath($dbname, $shardId = null){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - if($shardId !== null){ - return $this->dbDir . $hash . "-" . $dbname . "_s" . $shardId . ".nonedb.jidx"; + private function insertShardedRecords($dbname, array $items, $keyed = null, $endKey = null){ + $meta = $this->getCachedMeta($dbname, true); + if($meta === null){ + throw new \RuntimeException("noneDB: metadata of sharded database '{$dbname}' is unreadable"); } - return $this->dbDir . $hash . "-" . $dbname . ".nonedb.jidx"; - } + $size = (int)$meta['shardSize']; + $nextKey = (int)$meta['nextKey']; - /** - * Read JSONL index (byte offset map) - * v3.0.0 optimization: Uses atomicReadFast for better performance - * @param string $dbname - * @param int|null $shardId - * @return array|null - */ - private function readJsonlIndex($dbname, $shardId = null){ - $path = $this->getJsonlIndexPath($dbname, $shardId); - $cacheKey = $path; + $byShard = []; + if($keyed !== null){ + foreach($keyed as $key => $item){ + $byShard[intdiv($key, $size)][$key] = $item; + } + $nextKey = max($nextKey, (int)$endKey); + $items = $keyed; + } else { + foreach($items as $item){ + $byShard[intdiv($nextKey, $size)][$nextKey] = $item; + $nextKey++; + } + } - // Check cache first - if(isset($this->indexCache[$cacheKey])){ - return $this->indexCache[$cacheKey]; + // Reserve the keys first: a crash later leaves a gap, never a reused key + $shardPos = []; + foreach($meta['shards'] as $i => $shard){ + $shardPos[(int)$shard['id']] = $i; + } + foreach($byShard as $shardId => $records){ + if(!isset($shardPos[$shardId])){ + $meta['shards'][] = ["id" => $shardId, "file" => "_s" . $shardId, "count" => 0, "deleted" => 0]; + $shardPos[$shardId] = count($meta['shards']) - 1; + } + $meta['shards'][$shardPos[$shardId]]['count'] += count($records); } + // Journal first: if we die after reserving, recovery recounts the meta from the shards + foreach(array_keys($byShard) as $shardId){ + $this->markIndexesPending($dbname, $shardId); + } + $previousMeta = $this->getCachedMeta($dbname); + $meta['nextKey'] = $nextKey; + $meta['totalRecords'] = (int)$meta['totalRecords'] + count($items); + $this->writeMeta($dbname, $meta); - // Use fast read for index files (skip clearstatcache + retry loop) - $index = $this->atomicReadFast($path, null); - if($index !== null){ - $this->indexCache[$cacheKey] = $index; + try { + foreach($byShard as $shardId => $records){ + $index = $this->readJsonlIndex($dbname, $shardId); + if($index === null){ + $this->createJsonlDatabase($dbname, $shardId); + $index = $this->readJsonlIndex($dbname, $shardId); + } + $this->appendSegmentRecords($dbname, $shardId, $index, $records); + $index['n'] = max((int)$index['n'], max(array_keys($records)) - $shardId * $size + 1); + $this->writeJsonlIndex($dbname, $index, $shardId); + $this->afterRecordsChanged($dbname, $shardId, [], $this->withKeys($records)); + } + } catch(\Throwable $e){ + // Nothing was appended for the failing shard; restore counters (keys stay reserved) + if($previousMeta !== null){ + $previousMeta['nextKey'] = $nextKey; + $this->writeMeta($dbname, $previousMeta); + $this->recountMeta($dbname); + } + throw $e; } - return $index; } /** - * Write JSONL index - * @param string $dbname - * @param array $index - * @param int|null $shardId - * @return bool + * Records with their "key" field set */ - private function writeJsonlIndex($dbname, $index, $shardId = null){ - $path = $this->getJsonlIndexPath($dbname, $shardId); - $index['updated'] = time(); - $this->indexCache[$path] = $index; - return $this->atomicWrite($path, $index); + private function withKeys(array $records){ + foreach($records as $key => $record){ + unset($records[$key]['key']); + $records[$key]['key'] = (int)$key; + } + return $records; } - // ==================== FIELD INDEX METHODS (v3.0.0) ==================== - /** - * Get field index file path - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID or null for non-sharded - * @return string Path to field index file + * Replace records ([key => full new record]) in one segment + * @return int Number of records written */ - private function getFieldIndexPath($dbname, $field, $shardId = null){ - $hash = $this->hashDBName($dbname); - $safeField = preg_replace('/[^a-zA-Z0-9_]/', '_', $field); - if($shardId !== null){ - return $this->dbDir . $hash . "-" . $dbname . "_s" . $shardId . ".nonedb.fidx." . $safeField; + private function updateSegmentRecords($dbname, $shardId, array $oldRecords, array $newRecords){ + if(empty($newRecords)){ + return 0; + } + $index = $this->readJsonlIndex($dbname, $shardId); + foreach(array_keys($newRecords) as $key){ + if(!isset($index['o'][$key])){ + unset($newRecords[$key], $oldRecords[$key]); + } } - return $this->dbDir . $hash . "-" . $dbname . ".nonedb.fidx." . $safeField; + if(empty($newRecords)){ + return 0; + } + $this->appendSegmentRecords($dbname, $shardId, $index, $newRecords); + $index['d'] += count($newRecords); + $this->writeJsonlIndex($dbname, $index, $shardId); + $this->afterRecordsChanged($dbname, $shardId, $oldRecords, $this->withKeys($newRecords)); + $this->maybeCompact($dbname, $shardId); + return count($newRecords); } /** - * Get cache key for field index - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID - * @return string Cache key + * Delete records ([key => record]) from one segment + * @return int Number of records deleted */ - private function getFieldIndexCacheKey($dbname, $field, $shardId = null){ - $key = $dbname . ':' . $field; + private function deleteSegmentRecords($dbname, $shardId, array $oldRecords){ + if(empty($oldRecords)){ + return 0; + } + $index = $this->readJsonlIndex($dbname, $shardId); + foreach(array_keys($oldRecords) as $key){ + if(!isset($index['o'][$key])){ + unset($oldRecords[$key]); + } + } + if(empty($oldRecords)){ + return 0; + } + $keys = array_keys($oldRecords); + $this->appendTombstones($dbname, $shardId, $keys); + foreach($keys as $key){ + unset($index['o'][$key]); + } + $index['d'] += count($keys); // the old record lines + $index['t'] = (int)($index['t'] ?? 0) + count($keys); // tombstone lines + $this->writeJsonlIndex($dbname, $index, $shardId); + $this->afterRecordsChanged($dbname, $shardId, $oldRecords, []); + if($shardId !== null){ - $key .= ':s' . $shardId; + $meta = $this->getCachedMeta($dbname, true); + foreach($meta['shards'] as $i => $shard){ + if((int)$shard['id'] === $shardId){ + $meta['shards'][$i]['count'] = max(0, $meta['shards'][$i]['count'] - count($keys)); + $meta['shards'][$i]['deleted'] += count($keys); + } + } + $meta['totalRecords'] = max(0, (int)$meta['totalRecords'] - count($keys)); + $meta['deletedCount'] = (int)($meta['deletedCount'] ?? 0) + count($keys); + $this->writeMeta($dbname, $meta); } - return $key; + + $this->maybeCompact($dbname, $shardId); + return count($keys); } /** - * Read field index from file - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID - * @return array|null Field index or null if not exists + * Keep field and spatial indexes in sync after records of a segment changed + * @param array $removed [key => old record] + * @param array $added [key => new record] */ - private function readFieldIndex($dbname, $field, $shardId = null){ - $cacheKey = $this->getFieldIndexCacheKey($dbname, $field, $shardId); - - if(isset($this->fieldIndexCache[$cacheKey])){ - return $this->fieldIndexCache[$cacheKey]; - } + private function afterRecordsChanged($dbname, $shardId, array $removed, array $added){ + $this->applyFieldIndexChanges($dbname, $shardId, $removed, $added); + $this->applySpatialIndexChanges($dbname, $shardId, $removed, $added); + } - $path = $this->getFieldIndexPath($dbname, $field, $shardId); - if(!file_exists($path)){ - return null; + /** + * Compact a segment once garbage exceeds the threshold (relative to live records) + */ + private function maybeCompact($dbname, $shardId){ + $index = $this->readJsonlIndex($dbname, $shardId); + if($index === null){ + return; } - - $index = $this->atomicRead($path, null); - if($index !== null){ - $this->fieldIndexCache[$cacheKey] = $index; + $garbage = (int)$index['d']; + if($garbage > 0 && $garbage > (count($index['o']) + $garbage) * $this->jsonlGarbageThreshold){ + $this->compactJsonl($dbname, $shardId); } - return $index; } /** - * Write field index to file - * @param string $dbname Database name - * @param string $field Field name - * @param array $index Field index data - * @param int|null $shardId Shard ID - * @return bool Success + * Compact a segment: rewrite live records (keys and order preserved), drop old versions and + * tombstones. Crash-safe: the data file is replaced first; if the index rename doesn't happen, + * the shorter data file is detected and the index rebuilt from it. + * @param string $dbname + * @param int|null $shardId + * @return array ['compacted' => int, 'freed' => int] */ - private function writeFieldIndex($dbname, $field, $index, $shardId = null){ - $path = $this->getFieldIndexPath($dbname, $field, $shardId); - $cacheKey = $this->getFieldIndexCacheKey($dbname, $field, $shardId); + private function compactJsonl($dbname, $shardId = null){ + return $this->withLock($dbname, true, function() use ($dbname, $shardId){ + $index = $this->readJsonlIndex($dbname, $shardId); + if($index === null){ + return ['compacted' => 0, 'freed' => 0]; + } + $path = $this->segDataPath($dbname, $shardId); + $records = $this->readSegmentRecords($dbname, $shardId); + $index = $this->readJsonlIndex($dbname, $shardId); - $index['updated'] = time(); - $this->fieldIndexCache[$cacheKey] = $index; - $this->markFileExists($path); + $newIndex = $this->newJsonlIndex(); + $newIndex['created'] = $index['created'] ?? time(); + $newIndex['n'] = $index['n']; + $buffer = ''; + $offset = 0; + foreach($records as $key => $record){ + $json = $this->encodeRecord($record); + $newIndex['o'][$key] = [$offset, strlen($json)]; + $buffer .= $json . "\n"; + $offset += strlen($json) + 1; + } - return $this->atomicWrite($path, $index); + if(!$this->writeFileAtomic($path, $buffer)){ + throw new \RuntimeException("noneDB: compaction of database '{$dbname}' failed while writing data"); + } + $this->writeJsonlIndex($dbname, $newIndex, $shardId); + + return ['compacted' => count($records), 'freed' => (int)$index['d']]; + }); } /** - * Delete field index file - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID - * @return bool Success + * Convert a non-sharded database to sharded format (keys preserved, indexes rebuilt per shard) + * @param string $dbname + * @return bool */ - private function deleteFieldIndexFile($dbname, $field, $shardId = null){ - $path = $this->getFieldIndexPath($dbname, $field, $shardId); - $cacheKey = $this->getFieldIndexCacheKey($dbname, $field, $shardId); + private function migrateToSharded($dbname){ + return $this->withLock($dbname, true, function() use ($dbname){ + if($this->isSharded($dbname)){ + return true; + } + $legacyPath = $this->getDataPath($dbname); + if(!$this->cachedFileExists($legacyPath)){ + return false; + } + $this->flushPendingBuffers($dbname); - unset($this->fieldIndexCache[$cacheKey]); - $this->markFileNotExists($path); + $index = $this->readJsonlIndex($dbname, null); + if($index === null){ + return false; + } + $records = $this->readSegmentRecords($dbname, null); + $size = $this->shardSize; + $maxKey = empty($records) ? -1 : max(array_keys($records)); + $nextKey = max((int)$index['n'], $maxKey + 1); + $numShards = max(1, intdiv(max($nextKey, 1) - 1, $size) + 1); - if(file_exists($path)){ - return @unlink($path); - } - return true; - } + $byShard = array_fill(0, $numShards, []); + foreach($records as $key => $record){ + $byShard[intdiv($key, $size)][$key] = $record; + } - /** - * Get list of indexed fields for a database - * @param string $dbname Database name - * @param int|null $shardId Shard ID - * @return array List of field names that have indexes - */ - private function getIndexedFields($dbname, $shardId = null){ - $hash = $this->hashDBName($dbname); - $pattern = $this->dbDir . $hash . "-" . $dbname; - if($shardId !== null){ - $pattern .= "_s" . $shardId; - } - $pattern .= ".nonedb.fidx.*"; + $meta = [ + "version" => 1, + "shardSize" => $size, + "totalRecords" => count($records), + "deletedCount" => 0, + "nextKey" => $nextKey, + "shards" => [] + ]; - $files = glob($pattern); - $fields = []; - foreach($files as $file){ - // Extract field name from path - if(preg_match('/\.fidx\.([^\/]+)$/', $file, $matches)){ - $fields[] = $matches[1]; + foreach($byShard as $shardId => $shardRecords){ + $slots = min($nextKey, ($shardId + 1) * $size) - $shardId * $size; + $shardIndex = $this->newJsonlIndex(); + $shardIndex['n'] = max(0, $slots); + $buffer = ''; + $offset = 0; + foreach($shardRecords as $key => $record){ + $json = $this->encodeRecord($record); + $shardIndex['o'][$key] = [$offset, strlen($json)]; + $buffer .= $json . "\n"; + $offset += strlen($json) + 1; + } + $shardPath = $this->getShardPath($dbname, $shardId); + if(!$this->writeFileAtomic($shardPath, $buffer)){ + throw new \RuntimeException("noneDB: migration of database '{$dbname}' failed while writing shard {$shardId}"); + } + $this->markFileExists($shardPath); + $this->jsonlFormatCache[$shardPath] = true; + $this->writeJsonlIndex($dbname, $shardIndex, $shardId); + + $meta['shards'][] = [ + "id" => $shardId, + "file" => "_s" . $shardId, + "count" => count($shardRecords), + "deleted" => 0 + ]; } - } - return $fields; - } - /** - * Check if a field has an index - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID - * @return bool True if index exists - */ - private function hasFieldIndex($dbname, $field, $shardId = null){ - $path = $this->getFieldIndexPath($dbname, $field, $shardId); - return file_exists($path); - } + // Flip to sharded only after every shard is safely on disk + $this->writeMeta($dbname, $meta); - /** - * Invalidate field index cache for a database - * @param string $dbname Database name - * @param string|null $field Specific field or null for all fields - * @param int|null $shardId Shard ID - */ - private function invalidateFieldIndexCache($dbname, $field = null, $shardId = null){ - if($field !== null){ - $cacheKey = $this->getFieldIndexCacheKey($dbname, $field, $shardId); - unset($this->fieldIndexCache[$cacheKey]); - } else { - // Invalidate all field indexes for this database - $prefix = $dbname . ':'; - foreach(array_keys($this->fieldIndexCache) as $key){ - if(strpos($key, $prefix) === 0){ - unset($this->fieldIndexCache[$key]); - } + // Verify, then remove the non-sharded files + $verified = 0; + foreach(array_keys($byShard) as $shardId){ + $verified += count($this->readSegmentRecords($dbname, $shardId)); } - } + if($verified !== count($records)){ + throw new \RuntimeException("noneDB: migration of database '{$dbname}' could not be verified"); + } + @unlink($legacyPath . '.jidx'); + @unlink($legacyPath); + @unlink($legacyPath . '.backup'); + $this->markFileNotExists($legacyPath); + unset($this->indexCache[$legacyPath . '.jidx'], $this->jsonlFormatCache[$legacyPath]); + + // Indexes: rebuilt per shard from the manifest + $this->rebuildIndexesAfterMigration($dbname); + return true; + }); } - // ==================== GLOBAL FIELD INDEX METHODS (Shard Skip) ==================== + // ========================================== + // WRITE BUFFER + // ========================================== + // Optional (off by default). Buffered records have no key yet; flushing renames the buffer + // first and then inserts its records through insertRecords(), so the normal indexes and + // sharding logic apply. Pending buffers (also those left by pre-3.2 versions) are flushed + // before every read and write, whether or not buffering is currently enabled. /** - * Get path for global field index file - * @param string $dbname Database name - * @param string $field Field name - * @return string Path to global field index file + * Get buffer file path of a database + * @param string $dbname + * @return string */ - private function getGlobalFieldIndexPath($dbname, $field){ - $hash = $this->hashDBName($dbname); - $safeField = preg_replace('/[^a-zA-Z0-9_]/', '_', $field); - return $this->dbDir . $hash . "-" . $dbname . ".nonedb.gfidx." . $safeField; + private function getBufferPath($dbname){ + return $this->dbBase($dbname) . ".nonedb.buffer"; } /** - * Get cache key for global field index - * @param string $dbname Database name - * @param string $field Field name - * @return string Cache key + * Check if buffer exists and has content + * @param string $bufferPath + * @return bool */ - private function getGlobalFieldIndexCacheKey($dbname, $field){ - return 'gfidx:' . $dbname . ':' . $field; + private function hasBuffer($bufferPath){ + clearstatcache(true, $bufferPath); + return file_exists($bufferPath) && filesize($bufferPath) > 0; } /** - * Read global field index (with static cache) - * @param string $dbname Database name - * @param string $field Field name - * @return array|null Index data or null if not exists + * Number of records in a buffer file; counted once, then tracked by size */ - private function readGlobalFieldIndex($dbname, $field){ - $cacheKey = $this->getGlobalFieldIndexCacheKey($dbname, $field); - - // Check static cache - if(isset($this->fieldIndexCache[$cacheKey])){ - return $this->fieldIndexCache[$cacheKey]; + private function getBufferRecordCount($bufferPath){ + clearstatcache(true, $bufferPath); + if(!file_exists($bufferPath)){ + return 0; } - - $path = $this->getGlobalFieldIndexPath($dbname, $field); - if(!$this->cachedFileExists($path)){ - return null; + $size = filesize($bufferPath); + if(isset(self::$bufferCounts[$bufferPath]) && self::$bufferCounts[$bufferPath][0] === $size){ + return self::$bufferCounts[$bufferPath][1]; } + $count = count($this->readBufferRecords($bufferPath)); + self::$bufferCounts[$bufferPath] = [$size, $count]; + return $count; + } - $content = file_get_contents($path); - if($content === false){ - return null; + /** + * Read all records from a buffer file (JSONL) + * @param string $bufferPath + * @return array + */ + private function readBufferRecords($bufferPath){ + $records = []; + $fp = @fopen($bufferPath, 'rb'); + if($fp === false){ + return $records; } - - $index = json_decode($content, true); - if($index === null){ - return null; + while(($line = fgets($fp)) !== false){ + $line = trim($line); + if($line === ''){ + continue; + } + $record = json_decode($line, true); + if(is_array($record)){ + $records[] = $record; + } } - - // Cache it - $this->fieldIndexCache[$cacheKey] = $index; - return $index; + fclose($fp); + return $records; } /** - * Write global field index - * @param string $dbname Database name - * @param string $field Field name - * @param array $metadata Index metadata - * @return bool Success + * True if the database has buffered records waiting (current or pre-3.2 layout) */ - private function writeGlobalFieldIndex($dbname, $field, $metadata){ - $path = $this->getGlobalFieldIndexPath($dbname, $field); - $metadata['updated'] = time(); - - $json = json_encode($metadata, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES); - $result = file_put_contents($path, $json, LOCK_EX); - - if($result !== false){ - // Update cache - $cacheKey = $this->getGlobalFieldIndexCacheKey($dbname, $field); - $this->fieldIndexCache[$cacheKey] = $metadata; - return true; + private function hasPendingBuffers($dbname){ + // Under the lock the state record says it (flag 'b', set before a buffer gets records and + // cleared after they are flushed), so normal operations don't touch the file system for this + if($this->isVerified($dbname)){ + return !empty(self::$lockRegistry[$this->getLockPath($dbname)]['flags']['b']); } - return false; + return $this->hasBuffer($this->getBufferPath($dbname)) || !empty($this->flushingFiles($dbname)) || !empty($this->legacyShardBufferFiles($dbname)); } /** - * Check if global field index exists - * @param string $dbname Database name - * @param string $field Field name - * @return bool True if exists + * Buffer files whose flush was interrupted: {buffer}.flushing.{firstKey} (and the pre-3.2 {buffer}.flushing) */ - private function hasGlobalFieldIndex($dbname, $field){ - $path = $this->getGlobalFieldIndexPath($dbname, $field); - return $this->cachedFileExists($path); + private function flushingFiles($dbname){ + $bufferPath = $this->getBufferPath($dbname); + $files = glob($bufferPath . '.flushing*') ?: []; + return array_values(array_filter($files, function($f){ + return preg_match('/\.nonedb\.buffer\.flushing(\.\d+)?$/', $f) === 1; + })); } /** - * Get target shards from global field index - * @param string $dbname Database name - * @param string $field Field name - * @param mixed $value Field value to search - * @return array|null Array of shard IDs or null if no global index + * Pre-3.2 per-shard buffer files (and interrupted flushes) of a database */ - private function getTargetShardsFromGlobalIndex($dbname, $field, $value){ - $globalMeta = $this->readGlobalFieldIndex($dbname, $field); - if($globalMeta === null || !isset($globalMeta['shardMap'])){ - return null; + private function legacyShardBufferFiles($dbname){ + if(!$this->isSharded($dbname)){ + return []; } - - $valueKey = $this->fieldIndexValueKey($value); - return $globalMeta['shardMap'][$valueKey] ?? []; + $files = glob($this->dbBase($dbname) . '_s*.nonedb.buffer*') ?: []; + return array_values(array_filter($files, function($f){ + return preg_match('/_s\d+\.nonedb\.buffer(\.flushing(\.\d+)?)?$/', $f) === 1; + })); } /** - * Add shard to global field index for a value - * @param string $dbname Database name - * @param string $field Field name - * @param mixed $value Field value - * @param int $shardId Shard ID to add + * Flush buffered records into the database (exclusive lock) + * @return int Number of records flushed */ - private function addShardToGlobalIndex($dbname, $field, $value, $shardId){ - $globalMeta = $this->readGlobalFieldIndex($dbname, $field); - if($globalMeta === null){ - return; // No global index exists + private function flushPendingBuffers($dbname){ + $base = $this->dbBase($dbname); + // Flushing inserts records, which can migrate to shards, which flushes again: don't re-enter + if(isset(self::$flushingNow[$base]) || !$this->hasPendingBuffers($dbname)){ + return 0; } + return $this->withLock($dbname, true, function() use ($dbname, $base){ + self::$flushingNow[$base] = true; + try { + return $this->flushPendingBuffersLocked($dbname); + } finally { + unset(self::$flushingNow[$base]); + } + }); + } - $valueKey = $this->fieldIndexValueKey($value); - - if(!isset($globalMeta['shardMap'][$valueKey])){ - $globalMeta['shardMap'][$valueKey] = []; - } + private function flushPendingBuffersLocked($dbname){ + { + $flushed = 0; + foreach($this->legacyShardBufferFiles($dbname) as $file){ + $flushed += $this->flushLegacyShardBuffer($dbname, $file); + } - if(!in_array($shardId, $globalMeta['shardMap'][$valueKey])){ - $globalMeta['shardMap'][$valueKey][] = $shardId; - $this->writeGlobalFieldIndex($dbname, $field, $globalMeta); + $bufferPath = $this->getBufferPath($dbname); + // Interrupted flushes first. A flush renames the buffer to {buffer}.flushing.{firstKey}: the + // rename records atomically which keys its records get, so replaying it is idempotent. + foreach($this->flushingFiles($dbname) as $file){ + $flushed += $this->flushBufferFile($dbname, $file, preg_match('/\.flushing\.(\d+)$/', $file, $m) ? (int)$m[1] : null); + } + if($this->hasBuffer($bufferPath)){ + if(!$this->dbExists($dbname)){ + $this->createDBUnlocked($dbname); + } + $startKey = $this->nextKeyOf($dbname); + $flushingPath = $bufferPath . '.flushing.' . $startKey; + if(!@rename($bufferPath, $flushingPath)){ + throw new \RuntimeException("noneDB: could not flush the write buffer of database '{$dbname}'"); + } + $flushed += $this->flushBufferFile($dbname, $flushingPath, $startKey); + } + unset(self::$bufferCounts[$bufferPath]); + $this->setBufferFlag($dbname, false); + $this->bufferLastFlush[$dbname] = time(); + return $flushed; } } /** - * Remove shard from global field index for a value (if no more records) - * @param string $dbname Database name - * @param string $field Field name - * @param mixed $value Field value - * @param int $shardId Shard ID to potentially remove + * Insert the records of one buffer file, starting at $startKey (null: pre-3.2 file, next free keys) + * @return int */ - private function removeShardFromGlobalIndex($dbname, $field, $value, $shardId){ - $globalMeta = $this->readGlobalFieldIndex($dbname, $field); - if($globalMeta === null){ - return; - } - - $valueKey = $this->fieldIndexValueKey($value); - - if(!isset($globalMeta['shardMap'][$valueKey])){ - return; - } - - // Check if this shard still has records with this value - $fieldIndex = $this->readFieldIndex($dbname, $field, $shardId); - if($fieldIndex !== null && isset($fieldIndex['values'][$valueKey]) && !empty($fieldIndex['values'][$valueKey])){ - return; // Still has records, don't remove + private function flushBufferFile($dbname, $file, $startKey){ + $records = $this->readBufferRecords($file); + if(!empty($records)){ + $result = $this->insertRecords($dbname, $records, $startKey); + if(isset($result['error'])){ + throw new \RuntimeException("noneDB: flushing the write buffer of '{$dbname}' failed: " . $result['error']); + } } + @unlink($file); + return count($records); + } - // Remove shard from this value's shard list - $globalMeta['shardMap'][$valueKey] = array_values( - array_filter($globalMeta['shardMap'][$valueKey], function($id) use ($shardId){ - return $id !== $shardId; - }) - ); - - // Remove empty value entries - if(empty($globalMeta['shardMap'][$valueKey])){ - unset($globalMeta['shardMap'][$valueKey]); + /** + * The key the next inserted record gets + */ + private function nextKeyOf($dbname){ + if($this->isSharded($dbname)){ + $meta = $this->getCachedMeta($dbname, true); + return $meta === null ? 0 : (int)$meta['nextKey']; } - - $this->writeGlobalFieldIndex($dbname, $field, $globalMeta); + $index = $this->readJsonlIndex($dbname, null); + return $index === null ? 0 : (int)$index['n']; } - /** - * Delete global field index file - * @param string $dbname Database name - * @param string $field Field name + * Flush a pre-3.2 shard buffer. Its records were already counted in the meta (count, totalRecords + * and nextKey) when they were buffered; they take the next free slots of their shard. + * @return int */ - private function deleteGlobalFieldIndex($dbname, $field){ - $path = $this->getGlobalFieldIndexPath($dbname, $field); - if(file_exists($path)){ - @unlink($path); + private function flushLegacyShardBuffer($dbname, $file){ + if(!preg_match('/_s(\d+)\.nonedb\.buffer(\.flushing(\.\d+)?)?$/', $file, $m)){ + return 0; + } + $shardId = (int)$m[1]; + $size = $this->shardSizeOf($dbname); + $index = $this->readJsonlIndex($dbname, $shardId); + if($index === null){ + $this->createJsonlDatabase($dbname, $shardId); + $index = $this->readJsonlIndex($dbname, $shardId); + } + // Same idempotent scheme as the current buffer: the first key goes into the file name + if(preg_match('/\.flushing\.(\d+)$/', $file, $km)){ + $startKey = (int)$km[1]; + } else { + $startKey = $shardId * $size + (int)$index['n']; + $renamed = preg_replace('/(\.flushing)?$/', '', $file, 1) . '.flushing.' . $startKey; + if(!@rename($file, $renamed)){ + throw new \RuntimeException("noneDB: could not flush a write buffer of database '{$dbname}'"); + } + $file = $renamed; + } + $records = array_values(array_filter($this->readBufferRecords($file), function($r){ + return is_array($r) && !array_key_exists('key', $r); + })); + if(!empty($records)){ + $keyed = []; + foreach($records as $i => $record){ + $keyed[$startKey + $i] = $record; + } + foreach(array_keys($keyed) as $key){ + if(isset($index['o'][$key])){ + unset($keyed[$key]); // stored before an interruption + } + } + $this->appendSegmentRecords($dbname, $shardId, $index, $keyed); + $index['n'] = max((int)$index['n'], $startKey + count($records) - $shardId * $size); + $this->writeJsonlIndex($dbname, $index, $shardId); + $this->afterRecordsChanged($dbname, $shardId, [], $this->withKeys($keyed)); + $this->recountMeta($dbname); } - // Clear cache - $cacheKey = $this->getGlobalFieldIndexCacheKey($dbname, $field); - unset($this->fieldIndexCache[$cacheKey]); + @unlink($file); + return count($records); } - // ==================== END FIELD INDEX METHODS ==================== - /** - * Migrate v2 format to JSONL format - * @param string $path Source file path - * @param string $dbname Database name - * @param int|null $shardId Shard ID or null for non-sharded - * @return bool Success + * Append validated records to the database's write buffer (exclusive lock) + * @return array */ - private function migrateToJsonl($path, $dbname, $shardId = null){ - if(!$this->cachedFileExists($path)){ - return false; + private function insertBuffered($dbname, array $validItems){ + if(!$this->dbExists($dbname)){ + if(!$this->autoCreateDB){ + return ["n" => 0, "error" => "Database does not exist"]; + } + $this->createDBUnlocked($dbname); } + $this->registerShutdownHandler(); - // Read v2 format - $content = file_get_contents($path); - if($content === false){ - return false; + $lines = ''; + foreach($validItems as $item){ + $lines .= $this->encodeRecord($item) . "\n"; } - $data = json_decode($content, true); - if(!isset($data['data']) || !is_array($data['data'])){ - return false; + $bufferPath = $this->getBufferPath($dbname); + if($this->shouldFlushBuffer($bufferPath, $dbname)){ + $this->flushPendingBuffers($dbname); } - - // Create JSONL format with byte offset index - $tempPath = $path . '.jsonl.tmp'; - $handle = fopen($tempPath, 'wb'); - if($handle === false){ - return false; + $count = $this->getBufferRecordCount($bufferPath); + $this->setBufferFlag($dbname, true); + if($this->appendToFile($bufferPath, $lines) === false){ + return ["n" => 0, "error" => "Failed to write buffer"]; } - - // Acquire exclusive lock - if(!flock($handle, LOCK_EX)){ - fclose($handle); - @unlink($tempPath); - return false; + clearstatcache(true, $bufferPath); + self::$bufferCounts[$bufferPath] = [filesize($bufferPath), $count + count($validItems)]; + if(!isset($this->bufferLastFlush[$dbname])){ + $this->bufferLastFlush[$dbname] = time(); } - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - - $offset = 0; - $baseKey = ($shardId !== null) ? ($shardId * $this->shardSize) : 0; - - foreach($data['data'] as $localKey => $record){ - if($record === null){ - $index['d']++; - continue; - } - - $globalKey = $baseKey + $localKey; - $record['key'] = $globalKey; - $json = json_encode($record, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; // Exclude newline - - fwrite($handle, $json); - - $index['o'][$globalKey] = [$offset, $length]; - $offset += strlen($json); - $index['n']++; + if($this->shouldFlushBuffer($bufferPath, $dbname)){ + $this->flushPendingBuffers($dbname); } + return ["n" => count($validItems)]; + } - flock($handle, LOCK_UN); - fclose($handle); - - // Atomic swap - if(!rename($tempPath, $path)){ - @unlink($tempPath); + /** + * Check if buffer needs flushing (size, count, or time based) + * @param string $bufferPath + * @param string $flushKey Key for tracking last flush time + * @return bool + */ + private function shouldFlushBuffer($bufferPath, $flushKey){ + if(!$this->hasBuffer($bufferPath)){ return false; } - - // Clear format cache - unset($this->jsonlFormatCache[$path]); - $this->jsonlFormatCache[$path] = true; - - // Write index - $this->writeJsonlIndex($dbname, $index, $shardId); - - return true; + if(filesize($bufferPath) >= $this->bufferSizeLimit){ + return true; + } + if($this->getBufferRecordCount($bufferPath) >= $this->bufferCountLimit){ + return true; + } + if($this->bufferFlushInterval > 0){ + $lastFlush = $this->bufferLastFlush[$flushKey] ?? time(); + if((time() - $lastFlush) >= $this->bufferFlushInterval){ + return true; + } + } + return false; } /** - * Read single record from JSONL file using byte offset - * O(1) complexity - * @param string $path File path - * @param int $offset Byte offset - * @param int $length Byte length - * @return array|null + * Register shutdown handler for flushing all buffers */ - private function readJsonlRecord($path, $offset, $length){ - $handle = fopen($path, 'rb'); - if($handle === false){ - return null; + private function registerShutdownHandler(){ + if($this->shutdownHandlerRegistered){ + return; } - - // Acquire shared lock - if(!flock($handle, LOCK_SH)){ - fclose($handle); - return null; + if($this->bufferAutoFlushOnShutdown){ + // Weak reference: the handler must not keep the instance alive until shutdown + $ref = \WeakReference::create($this); + register_shutdown_function(static function() use ($ref){ + $db = $ref->get(); + if($db !== null){ + $db->flushAllBuffers(); + } + }); + $this->shutdownHandlerRegistered = true; } + } - fseek($handle, $offset, SEEK_SET); - $json = fread($handle, $length); - - flock($handle, LOCK_UN); - fclose($handle); + // ========================================== + // FILTER MATCHING (find / update / delete) + // ========================================== - if($json === false){ - return null; + /** + * Normalize a "key" filter value to a list of int keys (invalid values match nothing) + * @return array + */ + private function normalizeKeyFilter($value){ + $values = is_array($value) ? $value : [$value]; + $keys = []; + foreach($values as $v){ + if(is_int($v)){ + $k = $v; + } elseif(is_string($v) && preg_match('/^(0|[1-9][0-9]*)$/', $v)){ + $k = (int)$v; + } else { + continue; + } + if($k >= 0 && !isset($keys[$k])){ + $keys[$k] = $k; + } } - - return json_decode($json, true); + return array_values($keys); } /** - * Batch read multiple JSONL records efficiently - v3.0.0 - * Opens file once and uses buffered reading for better performance - * @param string $path File path - * @param array $offsets Array of [key => [offset, length], ...] - * @return array Array of [key => record, ...] + * Strict equality filter used by find(), update() and delete(): + * every field must exist in the record and be identical (===) to the filter value. + * "key" filters compare against the record key. */ - private function readJsonlRecordsBatch($path, $offsets){ - if(empty($offsets)){ - return []; + private function matchesFilters(array $record, array $filters){ + foreach($filters as $field => $value){ + if($field === 'key'){ + if(!isset($record['key']) || !in_array($record['key'], $this->normalizeKeyFilter($value), true)){ + return false; + } + continue; + } + if(!array_key_exists($field, $record) || $record[$field] !== $value){ + return false; + } } + return true; + } - $handle = fopen($path, 'rb'); - if($handle === false){ + /** + * Records matching $filters, in key order (or in the order of a "key" filter) + * @return array [key => record] + */ + private function selectRecords($dbname, array $filters){ + $segments = $this->listSegments($dbname); + if(empty($segments)){ return []; } - // Acquire shared lock - if(!flock($handle, LOCK_SH)){ - fclose($handle); - return []; + $keyFilter = null; + if(array_key_exists('key', $filters)){ + $keyFilter = $this->normalizeKeyFilter($filters['key']); + unset($filters['key']); + if(empty($keyFilter)){ + return []; + } } - $records = []; - $bufferSize = 65536; // 64KB buffer - $buffer = ''; - $bufferStart = -1; - $bufferEnd = -1; - - // Sort offsets by position to minimize disk seeks - $sortedOffsets = $offsets; - uasort($sortedOffsets, function($a, $b){ - return $a[0] - $b[0]; - }); - - foreach($sortedOffsets as $key => $location){ - $offset = $location[0]; - $length = $location[1]; - - // Check if data is in current buffer - if($offset >= $bufferStart && ($offset + $length) <= $bufferEnd){ - // Read from buffer - $localOffset = $offset - $bufferStart; - $json = substr($buffer, $localOffset, $length); - } else { - // Need to read from file - fseek($handle, $offset, SEEK_SET); - - // Read enough data (at least the record, preferably more for next records) - $readSize = max($length, $bufferSize); - $buffer = fread($handle, $readSize); - $bufferStart = $offset; - $bufferEnd = $offset + strlen($buffer); - - // Extract the record - $json = substr($buffer, 0, $length); + $result = []; + if($keyFilter !== null){ + $bySegment = []; + foreach($keyFilter as $key){ + $segment = $this->segmentForKey($dbname, $key); + if($segment !== false){ + $bySegment[$segment === null ? '' : $segment][] = $key; + } } - - if($json !== false){ - $record = json_decode($json, true); - if($record !== null){ - $records[$key] = $record; + $found = []; + foreach($bySegment as $segment => $keys){ + $found += $this->readSegmentRecords($dbname, $segment === '' ? null : (int)$segment, $keys); + } + foreach($keyFilter as $key){ + if(isset($found[$key]) && $this->matchesFilters($found[$key], $filters)){ + $result[$key] = $found[$key]; } } + return $result; } - flock($handle, LOCK_UN); - fclose($handle); - - return $records; + $targets = $this->targetShardsFromGlobalIndex($dbname, $filters); + foreach($segments as $segment){ + if($targets !== null && !in_array($segment, $targets, true)){ + continue; + } + $candidates = $this->fieldIndexCandidates($dbname, $segment, $filters); + if($candidates !== null && empty($candidates)){ + continue; + } + foreach($this->readSegmentRecords($dbname, $segment, $candidates) as $key => $record){ + if(empty($filters) || $this->matchesFilters($record, $filters)){ + $result[$key] = $record; + } + } + } + return $result; } /** - * Find by key using JSONL index - O(1) - * @param string $dbname - * @param int|array $keys - * @param int|null $shardId - * @return array|null + * Group [key => record] by the segment that stores each key + * @return array [segmentId|'' => [key => record]] */ - private function findByKeyJsonl($dbname, $keys, $shardId = null){ - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null || !isset($index['o'])){ - return null; + private function groupBySegment($dbname, array $records){ + $groups = []; + $sharded = $this->isSharded($dbname); + $size = $this->shardSizeOf($dbname); + foreach($records as $key => $record){ + $groups[$sharded ? intdiv($key, $size) : ''][$key] = $record; } + return $groups; + } - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } + // ========================================== + // FIELD INDEXES + // ========================================== + // Which fields are indexed is recorded in {base}.nonedb.indexes ("manifest"), so new shards and + // field names with any characters are handled. Each segment has one file per field: + // {segment}.nonedb.fidx.{name} where name is the field itself when it matches [A-Za-z0-9_]+ and + // "~" + hex(field) otherwise. Index files of format v2 are complete and exact; older files are + // rebuilt on first use. - $keys = is_array($keys) ? $keys : [$keys]; + private function getIndexManifestPath($dbname){ + return $this->dbBase($dbname) . '.nonedb.indexes'; + } - // Collect offsets for requested keys - $offsets = []; - foreach($keys as $key){ - $key = (int)$key; - if(isset($index['o'][$key])){ - $offsets[$key] = $index['o'][$key]; - } + /** + * Indexed fields of a database: ['fields' => [...], 'spatial' => [...]] + * Databases created before v3.2 have no manifest; it is derived from their index files. + */ + private function getIndexManifest($dbname){ + $path = $this->getIndexManifestPath($dbname); + if(isset($this->fieldIndexCache[$path])){ + return $this->fieldIndexCache[$path]; } - - if(empty($offsets)){ - return []; + $manifest = $this->atomicRead($path, false); + if($manifest === false && $this->isVerified($dbname)){ + $manifest = []; // no index list: no indexes (pre-3.2 databases got theirs when verified) + } elseif(!is_array($manifest)){ + $manifest = $this->deriveIndexManifest($dbname); } + $manifest = [ + 'fields' => array_values(array_unique(array_map('strval', $manifest['fields'] ?? []))), + 'spatial' => array_values(array_unique(array_map('strval', $manifest['spatial'] ?? []))), + 'spatialOptions' => is_array($manifest['spatialOptions'] ?? null) ? $manifest['spatialOptions'] : [], + ]; + $this->fieldIndexCache[$path] = $manifest; + return $manifest; + } - // Single key: use simple read (no batch overhead) - if(count($offsets) === 1){ - $key = array_key_first($offsets); - [$offset, $length] = $offsets[$key]; - $record = $this->readJsonlRecord($path, $offset, $length); - return $record !== null ? [$record] : []; + private function saveIndexManifest($dbname, array $manifest){ + $manifest = [ + 'v' => 1, + 'fields' => array_values(array_unique($manifest['fields'] ?? [])), + 'spatial' => array_values(array_unique($manifest['spatial'] ?? [])), + 'spatialOptions' => array_intersect_key($manifest['spatialOptions'] ?? [], array_flip(array_values(array_unique($manifest['spatial'] ?? [])))), + ]; + $path = $this->getIndexManifestPath($dbname); + if(empty($manifest['fields']) && empty($manifest['spatial'])){ + @unlink($path); + } elseif(!$this->atomicWrite($path, $manifest)){ + throw new \RuntimeException("noneDB: failed to write index list of database '{$dbname}'"); } + $this->fieldIndexCache[$path] = ['fields' => $manifest['fields'], 'spatial' => $manifest['spatial'], 'spatialOptions' => $manifest['spatialOptions']]; + } - // Multiple keys: use batch read for efficiency (v3.0.0) - $records = $this->readJsonlRecordsBatch($path, $offsets); - - // Maintain original key order - $result = []; - foreach($keys as $key){ - $key = (int)$key; - if(isset($records[$key])){ - $result[] = $records[$key]; + /** + * Pre-3.2 databases: read field names from existing index files (their "field" property is exact, + * the file name may have been lossy) + */ + private function deriveIndexManifest($dbname){ + $base = $this->dbBase($dbname); + $hashBase = $this->dbDir . $this->hashDBName($dbname); + $fields = []; + foreach(array_merge(glob($base . '.nonedb.fidx.*') ?: [], glob($base . '_s*.nonedb.fidx.*') ?: []) as $file){ + $index = $this->atomicRead($file, null); + if(is_array($index) && isset($index['field']) && is_string($index['field'])){ + $fields[] = $index['field']; + } elseif(preg_match('/\.fidx\.(.+)$/', $file, $m)){ + $fields[] = $this->decodeIndexFieldName($m[1]); } } - - return $result; + $spatial = []; + foreach(array_merge(glob($hashBase . '.nonedb.sidx.*') ?: [], glob($hashBase . '_s*.nonedb.sidx.*') ?: []) as $file){ + $index = $this->atomicRead($file, null); + if(is_string($index)){ + $index = json_decode($index, true); // pre-3.2 files were JSON-encoded twice + } + if(is_array($index) && isset($index['field']) && is_string($index['field'])){ + $spatial[] = $index['field']; + } elseif(preg_match('/\.sidx\.(.+)$/', $file, $m)){ + $spatial[] = $this->decodeIndexFieldName($m[1]); + } + } + return ['fields' => array_values(array_unique($fields)), 'spatial' => array_values(array_unique($spatial))]; } /** - * Read all records from JSONL file (streaming) - * Memory efficient for large files - * @param string $path - * @param array|null $index Optional index to filter valid records - * @return array + * File-name-safe, reversible encoding of a field name */ - private function readAllJsonl($path, $index = null){ - // If index provided, use batch read for better performance (v3.0.0) - if($index !== null && isset($index['o'])){ - // Use batch read for efficiency (single file open, buffered reads) - $records = $this->readJsonlRecordsBatch($path, $index['o']); + private function encodeIndexFieldName($field){ + $field = (string)$field; + return preg_match('/^[A-Za-z0-9_]+$/', $field) ? $field : '~' . bin2hex($field); + } - // Sort by key and return as indexed array - ksort($records, SORT_NUMERIC); - return array_values($records); + private function decodeIndexFieldName($name){ + if(isset($name[0]) && $name[0] === '~' && ctype_xdigit(substr($name, 1)) && strlen($name) % 2 === 1){ + return hex2bin(substr($name, 1)); } - - // Fallback: scan all lines (no index) - $handle = fopen($path, 'rb'); - if($handle === false){ - return []; - } - - if(!flock($handle, LOCK_SH)){ - fclose($handle); - return []; - } - - $results = []; - while(($line = fgets($handle)) !== false){ - $line = rtrim($line, "\n\r"); - if(empty($line)){ - continue; - } - $record = json_decode($line, true); - if($record === null){ - continue; - } - $results[] = $record; - } - - flock($handle, LOCK_UN); - fclose($handle); - - return $results; - } + return $name; + } /** - * Append record to JSONL file - * @param string $path - * @param array $record - * @param array &$index Reference to index for updating - * @return int|false New key or false on failure - */ - private function appendJsonlRecord($path, $record, &$index){ - clearstatcache(true, $path); - $offset = file_exists($path) ? filesize($path) : 0; - - $key = $index['n']; - $record['key'] = $key; - $json = json_encode($record, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - // Append with exclusive lock - $result = file_put_contents($path, $json, FILE_APPEND | LOCK_EX); - if($result === false){ - return false; - } - - $index['o'][$key] = [$offset, $length]; - $index['n']++; - - return $key; - } - - /** - * Bulk append records to JSONL file - * @param string $path - * @param array $records - * @param array &$index Reference to index - * @return array Keys of inserted records - */ - private function bulkAppendJsonl($path, $records, &$index){ - clearstatcache(true, $path); - $offset = file_exists($path) ? filesize($path) : 0; - - $buffer = ''; - $keys = []; - - foreach($records as $record){ - $key = $index['n']; - $record['key'] = $key; - $json = json_encode($record, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $index['o'][$key] = [$offset, $length]; - $offset += strlen($json); - $index['n']++; - - $buffer .= $json; - $keys[] = $key; - } - - // Single write for all records - file_put_contents($path, $buffer, FILE_APPEND | LOCK_EX); - - return $keys; - } - - /** - * Update record in JSONL (append new version, mark old as garbage) - * @param string $dbname - * @param int $key - * @param array $newData - * @param int|null $shardId - * @param bool $skipCompaction Skip auto-compaction (for batch operations) - * @return bool - */ - private function updateJsonlRecord($dbname, $key, $newData, $shardId = null, $skipCompaction = false){ - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null || !isset($index['o'][$key])){ - return false; - } - - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - - // Read old record for field/spatial index update - $oldRecord = null; - if($this->fieldIndexEnabled || $this->spatialIndexEnabled){ - $location = $index['o'][$key]; - $oldRecord = $this->readJsonlRecord($path, $location[0], $location[1]); - } - - clearstatcache(true, $path); - $offset = filesize($path); - - $newData['key'] = $key; - $json = json_encode($newData, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $result = file_put_contents($path, $json, FILE_APPEND | LOCK_EX); - if($result === false){ - return false; - } - - // Old record becomes garbage - $index['o'][$key] = [$offset, $length]; - $index['d']++; - - $this->writeJsonlIndex($dbname, $index, $shardId); - - // Update field indexes - if($this->fieldIndexEnabled && $oldRecord !== null){ - $this->updateFieldIndexOnUpdate($dbname, $oldRecord, $newData, $key, $shardId); - } - - // Update spatial indexes (v3.1.0) - if($this->spatialIndexEnabled && $oldRecord !== null){ - $this->updateSpatialIndexOnUpdate($dbname, $oldRecord, $newData, $key, $shardId); - } - - // Check if compaction needed (skip during batch operations) - if(!$skipCompaction && $index['d'] > $index['n'] * $this->jsonlGarbageThreshold){ - $this->compactJsonl($dbname, $shardId); - } - - return true; - } - - /** - * Batch update multiple JSONL records - single index write for performance - * @param string $dbname - * @param array $updates Array of ['key' => int, 'data' => array] - * @param int|null $shardId - * @return int Number of updated records - */ - private function updateJsonlRecordsBatch($dbname, array $updates, $shardId = null){ - if(empty($updates)){ - return 0; - } - - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null){ - return 0; - } - - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - - // Build all data to append in one buffer - clearstatcache(true, $path); - $offset = file_exists($path) ? filesize($path) : 0; - $buffer = ''; - $indexUpdates = []; - $updated = 0; - - $needRecord = $this->fieldIndexEnabled || $this->spatialIndexEnabled; - - foreach($updates as $item){ - $key = $item['key']; - $newData = $item['data']; - - if(!isset($index['o'][$key])){ - continue; - } - - // Read old record for field/spatial index update - $oldRecord = null; - if($needRecord){ - $location = $index['o'][$key]; - $oldRecord = $this->readJsonlRecord($path, $location[0], $location[1]); - } - - // Update field indexes - if($this->fieldIndexEnabled && $oldRecord !== null){ - $this->updateFieldIndexOnUpdate($dbname, $oldRecord, $newData, $key, $shardId); - } - - // Update spatial indexes (v3.1.0) - if($this->spatialIndexEnabled && $oldRecord !== null){ - $this->updateSpatialIndexOnUpdate($dbname, $oldRecord, $newData, $key, $shardId); - } - - $newData['key'] = $key; - $json = json_encode($newData, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $indexUpdates[$key] = [$offset, $length]; - $buffer .= $json; - $offset += strlen($json); - $index['d']++; - $updated++; - } - - // Single file write for all records - if(!empty($buffer)){ - $result = file_put_contents($path, $buffer, FILE_APPEND | LOCK_EX); - if($result === false){ - return 0; - } - } - - // Update index with new offsets - foreach($indexUpdates as $key => $location){ - $index['o'][$key] = $location; - } - - // Single index write - $this->writeJsonlIndex($dbname, $index, $shardId); - - // Batch flush spatial indexes (single write instead of N writes) - if($this->spatialIndexEnabled && $updated > 0){ - $this->flushSpatialIndexes(); - } - - // Check if compaction needed - if($index['d'] > $index['n'] * $this->jsonlGarbageThreshold){ - $this->compactJsonl($dbname, $shardId); - } - - return $updated; - } - - /** - * Delete record from JSONL (just remove from index) - * @param string $dbname - * @param int $key - * @param int|null $shardId - * @return bool - */ - private function deleteJsonlRecord($dbname, $key, $shardId = null){ - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null || !isset($index['o'][$key])){ - return false; - } - - // Read record for field/spatial index update before deletion - $record = null; - if($this->fieldIndexEnabled || $this->spatialIndexEnabled){ - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - $location = $index['o'][$key]; - $record = $this->readJsonlRecord($path, $location[0], $location[1]); - } - - unset($index['o'][$key]); - $index['d']++; - - $this->writeJsonlIndex($dbname, $index, $shardId); - - // Update field indexes - if($this->fieldIndexEnabled && $record !== null){ - $this->updateFieldIndexOnDelete($dbname, $record, $key, $shardId); - } - - // Update spatial indexes (v3.1.0) - if($this->spatialIndexEnabled && $record !== null){ - $this->updateSpatialIndexOnDelete($dbname, $record, $key, $shardId); - } - - // Check if compaction needed - if($index['d'] > $index['n'] * $this->jsonlGarbageThreshold){ - $this->compactJsonl($dbname, $shardId); - } - - return true; - } - - /** - * Batch delete multiple JSONL records - single index write for performance - * @param string $dbname - * @param array $keys Array of keys to delete - * @param int|null $shardId - * @return int Number of deleted records - */ - private function deleteJsonlRecordsBatch($dbname, array $keys, $shardId = null){ - if(empty($keys)){ - return 0; - } - - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null){ - return 0; - } - - // Get path for field/spatial index updates - $path = null; - $needRecord = $this->fieldIndexEnabled || $this->spatialIndexEnabled; - if($needRecord){ - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - } - - $deleted = 0; - foreach($keys as $key){ - if(!isset($index['o'][$key])){ - continue; - } - - // Read record for field/spatial index update before deletion - $record = null; - if($needRecord && $path !== null){ - $location = $index['o'][$key]; - $record = $this->readJsonlRecord($path, $location[0], $location[1]); - } - - // Update field indexes - if($this->fieldIndexEnabled && $record !== null){ - $this->updateFieldIndexOnDelete($dbname, $record, $key, $shardId); - } - - // Update spatial indexes (v3.1.0) - if($this->spatialIndexEnabled && $record !== null){ - $this->updateSpatialIndexOnDelete($dbname, $record, $key, $shardId); - } - - unset($index['o'][$key]); - $index['d']++; - $deleted++; - } - - // Single index write for all deletions - $this->writeJsonlIndex($dbname, $index, $shardId); - - // Batch flush spatial indexes (single write instead of N writes) - if($this->spatialIndexEnabled && $deleted > 0){ - $this->flushSpatialIndexes(); - } - - // Check if compaction needed - if($index['d'] > $index['n'] * $this->jsonlGarbageThreshold){ - $this->compactJsonl($dbname, $shardId); - } - - return $deleted; - } - - /** - * Compact JSONL file (remove garbage) - * @param string $dbname - * @param int|null $shardId - * @return array ['compacted' => int, 'freed' => int] - */ - private function compactJsonl($dbname, $shardId = null){ - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null){ - return ['compacted' => 0, 'freed' => 0]; - } - - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - - $tempPath = $path . '.compact.tmp'; - $handle = fopen($path, 'rb'); - $tempHandle = fopen($tempPath, 'wb'); - - if($handle === false || $tempHandle === false){ - if($handle) fclose($handle); - if($tempHandle) fclose($tempHandle); - return ['compacted' => 0, 'freed' => 0]; - } - - flock($handle, LOCK_SH); - flock($tempHandle, LOCK_EX); - - $newIndex = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => $index['created'] ?? time(), - 'n' => $index['n'], // Preserve next key counter (don't reset!) - 'd' => 0, - 'o' => [] - ]; - - $offset = 0; - $compacted = 0; - - // Sort keys for sequential read - $sortedKeys = array_keys($index['o']); - sort($sortedKeys, SORT_NUMERIC); - - foreach($sortedKeys as $key){ - [$oldOffset, $length] = $index['o'][$key]; - - fseek($handle, $oldOffset); - $json = fread($handle, $length); - - fwrite($tempHandle, $json . "\n"); - - $newIndex['o'][$key] = [$offset, $length]; - $offset += $length + 1; - $compacted++; - } - - $freed = $index['d']; - - flock($handle, LOCK_UN); - flock($tempHandle, LOCK_UN); - fclose($handle); - fclose($tempHandle); - - // Atomic swap - rename($tempPath, $path); - - // Update index - $this->writeJsonlIndex($dbname, $newIndex, $shardId); - - return ['compacted' => $compacted, 'freed' => $freed]; - } - - /** - * Ensure database is in JSONL format (auto-migrate if needed) - * @param string $dbname - * @param int|null $shardId - * @return bool True if JSONL format (or migrated), false otherwise - */ - private function ensureJsonlFormat($dbname, $shardId = null){ - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - - if(!$this->cachedFileExists($path)){ - return true; // New file will be created in JSONL format - } - - if($this->isJsonlFormat($path)){ - return true; - } - - // Auto-migrate v2 format to JSONL - return $this->migrateToJsonl($path, $dbname, $shardId); - } - - /** - * Create new JSONL database file with empty index - * @param string $dbname - * @param int|null $shardId - * @return bool - */ - private function createJsonlDatabase($dbname, $shardId = null){ - if($shardId !== null){ - $path = $this->getShardPath($dbname, $shardId); - } else { - $hash = $this->hashDBName($dbname); - $path = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - } - - // Create empty file - if(!$this->cachedFileExists($path)){ - touch($path); - $this->markFileExists($path); - } - - // Create index - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - - $this->writeJsonlIndex($dbname, $index, $shardId); - $this->jsonlFormatCache[$path] = true; - - return true; - } - - // ========================================== - // WRITE BUFFER METHODS - // ========================================== - - /** - * Get buffer file path for non-sharded database - * @param string $dbname - * @return string - */ - private function getBufferPath($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - return $this->dbDir . $hash . "-" . $dbname . ".nonedb.buffer"; - } - - /** - * Get buffer file path for a specific shard - * @param string $dbname - * @param int $shardId - * @return string - */ - private function getShardBufferPath($dbname, $shardId){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - return $this->dbDir . $hash . "-" . $dbname . "_s" . $shardId . ".nonedb.buffer"; - } - - /** - * Check if buffer exists and has content - * @param string $bufferPath - * @return bool - */ - private function hasBuffer($bufferPath){ - clearstatcache(true, $bufferPath); - return file_exists($bufferPath) && filesize($bufferPath) > 0; - } - - /** - * Get buffer file size in bytes - * @param string $bufferPath - * @return int - */ - private function getBufferSize($bufferPath){ - clearstatcache(true, $bufferPath); - if(!file_exists($bufferPath)){ - return 0; - } - return (int) filesize($bufferPath); - } - - /** - * Count records in buffer file - * @param string $bufferPath - * @return int - */ - private function getBufferRecordCount($bufferPath){ - if(!$this->hasBuffer($bufferPath)){ - return 0; - } - $count = 0; - $fp = fopen($bufferPath, 'rb'); - if($fp === false){ - return 0; - } - // Lock for reading - flock($fp, LOCK_SH); - while(($line = fgets($fp)) !== false){ - $line = trim($line); - if($line !== ''){ - $count++; - } - } - flock($fp, LOCK_UN); - fclose($fp); - return $count; - } - - /** - * Atomically append records to buffer file (JSONL format) - * This is fast because it doesn't read the entire file - * - * @param string $bufferPath - * @param array $records Array of records to append - * @return array ['success' => bool, 'count' => int, 'error' => string|null] - */ - private function atomicAppendToBuffer($bufferPath, array $records){ - if(empty($records)){ - return ['success' => true, 'count' => 0, 'error' => null]; - } - - // Ensure directory exists - $dir = dirname($bufferPath); - if(!is_dir($dir)){ - mkdir($dir, 0755, true); - } - - // Open in append mode - $fp = fopen($bufferPath, 'ab'); - if($fp === false){ - return ['success' => false, 'count' => 0, 'error' => 'Failed to open buffer file']; - } - - $startTime = microtime(true); - $locked = false; - - // Try to acquire exclusive lock with timeout - while(!$locked && (microtime(true) - $startTime) < $this->lockTimeout){ - $locked = flock($fp, LOCK_EX | LOCK_NB); - if(!$locked){ - usleep($this->lockRetryDelay); - } - } - - if(!$locked){ - $locked = flock($fp, LOCK_EX); - } - - if(!$locked){ - fclose($fp); - return ['success' => false, 'count' => 0, 'error' => 'Failed to acquire lock']; - } - - try { - $written = 0; - foreach($records as $record){ - $line = json_encode($record) . "\n"; - if(fwrite($fp, $line) !== false){ - $written++; - } - } - fflush($fp); - return ['success' => true, 'count' => $written, 'error' => null]; - } finally { - flock($fp, LOCK_UN); - fclose($fp); - } - } - - /** - * Read all records from buffer file (JSONL format) - * @param string $bufferPath - * @return array Array of records - */ - private function readBufferRecords($bufferPath){ - if(!$this->hasBuffer($bufferPath)){ - return []; - } - - $fp = fopen($bufferPath, 'rb'); - if($fp === false){ - return []; - } - - $startTime = microtime(true); - $locked = false; - - while(!$locked && (microtime(true) - $startTime) < $this->lockTimeout){ - $locked = flock($fp, LOCK_SH | LOCK_NB); - if(!$locked){ - usleep($this->lockRetryDelay); - } - } - - if(!$locked){ - $locked = flock($fp, LOCK_SH); - } - - if(!$locked){ - fclose($fp); - return []; - } - - $records = []; - try { - while(($line = fgets($fp)) !== false){ - $line = trim($line); - if($line !== ''){ - $record = json_decode($line, true); - if($record !== null && json_last_error() === JSON_ERROR_NONE){ - $records[] = $record; - } - // Skip corrupted lines silently - } - } - } finally { - flock($fp, LOCK_UN); - fclose($fp); - } - - return $records; - } - - /** - * Clear buffer file (delete it) - * @param string $bufferPath - * @return bool - */ - private function clearBuffer($bufferPath){ - clearstatcache(true, $bufferPath); - if(file_exists($bufferPath)){ - return @unlink($bufferPath); - } - return true; - } - - /** - * Flush buffer to main database (non-sharded) - * @param string $dbname - * @return array ['success' => bool, 'flushed' => int, 'error' => string|null] - */ - private function flushBufferToMain($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $bufferPath = $this->getBufferPath($dbname); - - if(!$this->hasBuffer($bufferPath)){ - return ['success' => true, 'flushed' => 0, 'error' => null]; - } - - // Read buffer records - $bufferRecords = $this->readBufferRecords($bufferPath); - if(empty($bufferRecords)){ - $this->clearBuffer($bufferPath); - return ['success' => true, 'flushed' => 0, 'error' => null]; - } - - // Rename buffer to temp file (atomic on POSIX) - $tempPath = $bufferPath . '.flushing'; - if(!@rename($bufferPath, $tempPath)){ - return ['success' => false, 'flushed' => 0, 'error' => 'Failed to rename buffer']; - } - - // Get main DB path - $hash = $this->hashDBName($dbname); - $mainPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - - // Ensure JSONL format exists (auto-migrate v2 if needed) - if(!$this->ensureJsonlFormat($dbname)){ - $this->createJsonlDatabase($dbname); - } - - $index = $this->readJsonlIndex($dbname); - if($index === null){ - @rename($tempPath, $bufferPath); - return ['success' => false, 'flushed' => 0, 'error' => 'Failed to read index']; - } - - // Bulk append buffer records - $keys = $this->bulkAppendJsonl($mainPath, $bufferRecords, $index); - $this->writeJsonlIndex($dbname, $index); - - // Update field indexes for flushed records - if($this->fieldIndexEnabled){ - foreach($bufferRecords as $i => $record){ - $this->updateFieldIndexOnInsert($dbname, $record, $keys[$i], null); - } - } - - // Delete temp file - @unlink($tempPath); - $this->bufferLastFlush[$dbname] = time(); - return ['success' => true, 'flushed' => count($bufferRecords), 'error' => null]; + * Get field index file path + * @param string $dbname Database name + * @param string $field Field name + * @param int|null $shardId Shard ID or null for non-sharded + * @return string Path to field index file + */ + private function getFieldIndexPath($dbname, $field, $shardId = null){ + return $this->segDataPath($dbname, $shardId) . ".fidx." . $this->encodeIndexFieldName($field); } /** - * Flush buffer to shard - * @param string $dbname - * @param int $shardId - * @return array ['success' => bool, 'flushed' => int, 'error' => string|null] + * Read a segment's field index. Returns null when the file is missing or predates v3.2 + * (such files may be incomplete and are rebuilt by ensureSegmentFieldIndex). + * @return array|null */ - private function flushShardBuffer($dbname, $shardId){ - $dbname = $this->sanitizeDbName($dbname); - $bufferPath = $this->getShardBufferPath($dbname, $shardId); - - if(!$this->hasBuffer($bufferPath)){ - return ['success' => true, 'flushed' => 0, 'error' => null]; - } - - $bufferRecords = $this->readBufferRecords($bufferPath); - if(empty($bufferRecords)){ - $this->clearBuffer($bufferPath); - return ['success' => true, 'flushed' => 0, 'error' => null]; - } - - // Rename buffer to temp - $tempPath = $bufferPath . '.flushing'; - if(!@rename($bufferPath, $tempPath)){ - return ['success' => false, 'flushed' => 0, 'error' => 'Failed to rename buffer']; - } - - // v3.0.0: Use JSONL format for sharded writes - $shardPath = $this->getShardPath($dbname, $shardId); - - // Ensure JSONL format exists - if(!$this->cachedFileExists($shardPath)){ - $this->createJsonlDatabase($dbname, $shardId); - } else if(!$this->isJsonlFormat($shardPath)){ - // Migrate existing JSON to JSONL - $this->migrateToJsonl($shardPath, $dbname, $shardId); - } - - // Read current JSONL index - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null){ - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - } - - // Calculate base key for this shard - $baseKey = $shardId * $this->shardSize; - - // Bulk append records to JSONL file using global keys - $insertedKeys = []; - clearstatcache(true, $shardPath); - $offset = file_exists($shardPath) ? filesize($shardPath) : 0; - $buffer = ''; - - foreach($bufferRecords as $record){ - // Use global key: baseKey + local position within shard - $localKey = $index['n']; - $globalKey = $baseKey + $localKey; - $record['key'] = $globalKey; - - $json = json_encode($record, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $index['o'][$globalKey] = [$offset, $length]; - $offset += strlen($json); - $index['n']++; - - $buffer .= $json; - $insertedKeys[] = $globalKey; + private function readFieldIndex($dbname, $field, $shardId = null){ + $path = $this->getFieldIndexPath($dbname, $field, $shardId); + if(array_key_exists($path, $this->fieldIndexCache)){ + return $this->fieldIndexCache[$path]; } - - // Single write for all records - $result = file_put_contents($shardPath, $buffer, FILE_APPEND | LOCK_EX); - - if($result !== false){ - - // Write updated JSONL index - $this->writeJsonlIndex($dbname, $index, $shardId); - - // Update field indexes for flushed records (with shardId for global index) - if($this->fieldIndexEnabled){ - foreach($bufferRecords as $i => $record){ - $globalKey = $insertedKeys[$i]; - $this->updateFieldIndexOnInsert($dbname, $record, $globalKey, $shardId); - } - } - - @unlink($tempPath); - $flushKey = $dbname . '_s' . $shardId; - $this->bufferLastFlush[$flushKey] = time(); - return ['success' => true, 'flushed' => count($bufferRecords), 'error' => null]; - } else { - @rename($tempPath, $bufferPath); - return ['success' => false, 'flushed' => 0, 'error' => 'Failed to append records']; + $index = $this->atomicRead($path, null); + if(!is_array($index) || ($index['v'] ?? 1) < 2 || !isset($index['values']) || !is_array($index['values'])){ + $index = null; } + $this->fieldIndexCache[$path] = $index; + return $index; } /** - * Check if buffer needs flushing (size, count, or time based) - * @param string $bufferPath - * @param string $flushKey Key for tracking last flush time - * @return bool + * Write field index to file (caller holds the exclusive lock) + * @return bool Success */ - private function shouldFlushBuffer($bufferPath, $flushKey){ - if(!$this->hasBuffer($bufferPath)){ - return false; - } - - // Check size limit - $size = $this->getBufferSize($bufferPath); - if($size >= $this->bufferSizeLimit){ - return true; - } - - // Check count limit - $count = $this->getBufferRecordCount($bufferPath); - if($count >= $this->bufferCountLimit){ - return true; - } - - // Check time-based flush - if($this->bufferFlushInterval > 0){ - $lastFlush = $this->bufferLastFlush[$flushKey] ?? 0; - if((time() - $lastFlush) >= $this->bufferFlushInterval){ - return true; - } + private function writeFieldIndex($dbname, $field, $index, $shardId = null){ + $path = $this->getFieldIndexPath($dbname, $field, $shardId); + $index['v'] = 2; + $index['field'] = (string)$field; + $index['updated'] = time(); + if(!$this->atomicWrite($path, $index)){ + unset($this->fieldIndexCache[$path]); + throw new \RuntimeException("noneDB: failed to write field index '{$field}' of database '{$dbname}'"); } - - return false; + $this->fieldIndexCache[$path] = $index; + $this->markFileExists($path); + return true; } /** - * Register shutdown handler for flushing all buffers + * Delete field index file + * @return bool Success */ - private function registerShutdownHandler(){ - if($this->shutdownHandlerRegistered){ - return; - } - if($this->bufferAutoFlushOnShutdown){ - register_shutdown_function([$this, 'flushAllBuffers']); - $this->shutdownHandlerRegistered = true; - } + private function deleteFieldIndexFile($dbname, $field, $shardId = null){ + $path = $this->getFieldIndexPath($dbname, $field, $shardId); + unset($this->fieldIndexCache[$path]); + $this->markFileNotExists($path); + clearstatcache(true, $path); + return !file_exists($path) || @unlink($path); } /** - * Flush all shard buffers for a database - * @param string $dbname - * @param array|null $meta Optional meta data (avoids re-reading) - * @return array ['flushed' => total records flushed] + * Get list of indexed fields for a database + * @return array */ - private function flushAllShardBuffers($dbname, $meta = null){ - $dbname = $this->sanitizeDbName($dbname); - if($meta === null){ - $meta = $this->getCachedMeta($dbname); - } - if($meta === null || !isset($meta['shards'])){ - return ['flushed' => 0]; - } - - $totalFlushed = 0; - foreach($meta['shards'] as $shard){ - $result = $this->flushShardBuffer($dbname, $shard['id']); - $totalFlushed += $result['flushed']; - } - - return ['flushed' => $totalFlushed]; + private function getIndexedFields($dbname, $shardId = null){ + return $this->getIndexManifest($dbname)['fields']; } /** - * Migrate a legacy (non-sharded) database to sharded format - * @param string $dbname + * Check if a field has an index * @return bool */ - private function migrateToSharded($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $hash = $this->hashDBName($dbname); - $legacyPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - - if(!$this->cachedFileExists($legacyPath)){ - return false; - } - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $allRecords = []; - $totalRecords = 0; - $deletedCount = 0; - - $index = $this->readJsonlIndex($dbname); - if($index === null){ - return false; - } + private function isFieldIndexed($dbname, $field, $shardId = null){ + return in_array((string)$field, $this->getIndexedFields($dbname), true); + } - $allRecordsRaw = $this->readAllJsonl($legacyPath, $index); - // Convert to indexed array with key field - foreach($allRecordsRaw as $record){ - $key = $record['key'] ?? count($allRecords); - unset($record['key']); - $allRecords[$key] = $record; - $totalRecords++; - } - // Fill gaps with null for deleted records - if(!empty($allRecords)){ - $maxKey = max(array_keys($allRecords)); - for($i = 0; $i <= $maxKey; $i++){ - if(!isset($allRecords[$i])){ - $allRecords[$i] = null; - $deletedCount++; - } - } - ksort($allRecords); - $allRecords = array_values($allRecords); + /** + * Invalidate field index cache for a database + */ + private function invalidateFieldIndexCache($dbname, $field = null, $shardId = null){ + if($field !== null){ + unset($this->fieldIndexCache[$this->getFieldIndexPath($dbname, $field, $shardId)]); + return; } - - // Calculate number of shards needed - $totalEntries = count($allRecords); - $numShards = (int) ceil($totalEntries / $this->shardSize); - if($numShards === 0) $numShards = 1; - - // Create shards - $meta = array( - "version" => 1, - "shardSize" => $this->shardSize, - "totalRecords" => $totalRecords, - "deletedCount" => $deletedCount, - "nextKey" => $totalEntries, - "shards" => [] - ); - - for($shardId = 0; $shardId < $numShards; $shardId++){ - $start = $shardId * $this->shardSize; - $end = min($start + $this->shardSize, $totalEntries); - $shardRecords = array_slice($allRecords, $start, $end - $start); - - // Count records in this shard - $shardCount = 0; - $shardDeleted = 0; - foreach($shardRecords as $record){ - if($record === null){ - $shardDeleted++; - } else { - $shardCount++; - } + $prefix = $this->dbDir . $this->hashDBName($dbname); + foreach(array_keys($this->fieldIndexCache) as $key){ + if(strpos($key, $prefix) === 0){ + unset($this->fieldIndexCache[$key]); } - - $meta['shards'][] = array( - "id" => $shardId, - "file" => "_s" . $shardId, - "count" => $shardCount, - "deleted" => $shardDeleted - ); - - // v3.0.0: Write shard file in JSONL format - $shardPath = $this->getShardPath($dbname, $shardId); - $baseKey = $shardId * $this->shardSize; - - // Create JSONL file and index - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - - $buffer = ''; - $offset = 0; - - foreach($shardRecords as $localKey => $record){ - if($record === null){ - $index['d']++; - continue; - } - - $globalKey = $baseKey + $localKey; - $record['key'] = $globalKey; - - $json = json_encode($record, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $index['o'][$globalKey] = [$offset, $length]; - $offset += strlen($json); - $index['n']++; - - $buffer .= $json; - } - - // Write JSONL file - file_put_contents($shardPath, $buffer, LOCK_EX); - $this->markFileExists($shardPath); - $this->jsonlFormatCache[$shardPath] = true; - - // Write JSONL index - $this->writeJsonlIndex($dbname, $index, $shardId); - } - - // Write meta file - $this->writeMeta($dbname, $meta); - - // Backup and remove legacy file - $backupPath = $legacyPath . ".backup"; - rename($legacyPath, $backupPath); - - // Clean up JSONL index file if exists - $indexPath = $legacyPath . ".jidx"; - if(file_exists($indexPath)){ - @unlink($indexPath); } - - // Clear index cache for this database - unset($this->indexCache[$indexPath]); - - // Invalidate sharded cache - database is now sharded - $this->invalidateShardedCache($dbname); - - return true; } /** - * Insert data into sharded database with atomic locking - * @param string $dbname - * @param array $data - * @return array + * Bucket key of a field value. Values of different types may share a bucket (1, "1", 1.0); results + * are always re-checked with strict comparison, so that only costs a few extra reads. + * @param mixed $value Field value + * @return string Index key */ - private function insertSharded($dbname, $data){ - $dbname = $this->sanitizeDbName($dbname); - $main_response = array("n" => 0); - - // Validate data first - $validItems = []; - if($this->isRecordList($data)){ - foreach($data as $item){ - if(!is_array($item)) continue; - if($this->hasReservedKeyField($item)){ - $main_response['error'] = "You cannot set key name to key"; - return $main_response; - } - $validItems[] = $item; - } - if(empty($validItems)){ - return array("n" => 0); - } - } else { - if($this->hasReservedKeyField($data)){ - $main_response['error'] = "You cannot set key name to key"; - return $main_response; - } - $validItems[] = $data; + private function fieldIndexValueKey($value){ + if($value === null){ + return '__null__'; } - - // Use buffered insert if enabled - if($this->bufferEnabled){ - return $this->insertShardedBuffered($dbname, $validItems); + if(is_bool($value)){ + return $value ? '__true__' : '__false__'; } + if(is_float($value) && $value == 0.0){ + return '0'; // -0.0 === 0.0 + } + return (string)$value; + } - // Non-buffered insert (original method) - return $this->insertShardedDirect($dbname, $validItems); + /** + * True if a value can be stored in a field index + */ + private function isIndexableValue($value){ + return is_scalar($value) || is_null($value); } /** - * Buffered insert for sharded database - writes to per-shard buffers - * @param string $dbname - * @param array $validItems Pre-validated items + * Build a segment's field index from its data * @return array */ - private function insertShardedBuffered($dbname, array $validItems){ - $this->registerShutdownHandler(); - - $shardSize = $this->shardSize; - $insertedCount = 0; - $shardWrites = []; // Collect items per shard - - // Atomically update meta and calculate which shards to write - $metaResult = $this->modifyMeta($dbname, function($meta) use ($validItems, $shardSize, &$insertedCount, &$shardWrites) { - if($meta === null){ - return null; - } - - $lastShardIdx = count($meta['shards']) - 1; - $shardId = $meta['shards'][$lastShardIdx]['id']; - $currentShardCount = $meta['shards'][$lastShardIdx]['count'] + $meta['shards'][$lastShardIdx]['deleted']; - - foreach($validItems as $item){ - // Check if current shard is full - if($currentShardCount >= $shardSize){ - $shardId++; - $meta['shards'][] = array( - "id" => $shardId, - "file" => "_s" . $shardId, - "count" => 0, - "deleted" => 0 - ); - $lastShardIdx = count($meta['shards']) - 1; - $currentShardCount = 0; - } - - if(!isset($shardWrites[$shardId])){ - $shardWrites[$shardId] = ['items' => []]; - } - $shardWrites[$shardId]['items'][] = $item; - $currentShardCount++; - $insertedCount++; - - $meta['shards'][$lastShardIdx]['count']++; - $meta['totalRecords']++; - $meta['nextKey']++; - } - - return $meta; - }); - - if(!$metaResult['success'] || $metaResult['data'] === null){ - return array("n" => 0, "error" => $metaResult['error'] ?? 'Meta update failed'); + private function buildSegmentFieldIndex($dbname, $shardId, $field){ + $index = ['v' => 2, 'field' => (string)$field, 'created' => time(), 'values' => []]; + if($shardId !== null){ + $index['shardId'] = $shardId; } - - // Write to each affected shard's buffer - foreach($shardWrites as $shardId => $writeInfo){ - $bufferPath = $this->getShardBufferPath($dbname, $shardId); - $flushKey = $dbname . '_s' . $shardId; - - // Check if buffer needs flushing before write - if($this->shouldFlushBuffer($bufferPath, $flushKey)){ - $this->flushShardBuffer($dbname, $shardId); - } - - // Append to shard buffer (fast) - $this->atomicAppendToBuffer($bufferPath, $writeInfo['items']); - - // Check again after write - if($this->shouldFlushBuffer($bufferPath, $flushKey)){ - $this->flushShardBuffer($dbname, $shardId); + foreach($this->readSegmentRecords($dbname, $shardId) as $key => $record){ + if(array_key_exists($field, $record) && $this->isIndexableValue($record[$field])){ + $index['values'][$this->fieldIndexValueKey($record[$field])][] = (int)$key; } } - - return array("n" => $insertedCount); + return $index; } /** - * Direct insert for sharded database without buffer - * @param string $dbname - * @param array $validItems Pre-validated items + * A segment's field index, (re)built when missing or from an older version (exclusive lock) * @return array */ - private function insertShardedDirect($dbname, array $validItems){ - $shardSize = $this->shardSize; - $insertedCount = 0; - $shardWrites = []; - - // Atomically update meta and calculate which shards to write - $metaResult = $this->modifyMeta($dbname, function($meta) use ($validItems, $shardSize, &$insertedCount, &$shardWrites) { - if($meta === null){ - return null; - } - - $lastShardIdx = count($meta['shards']) - 1; - $shardId = $meta['shards'][$lastShardIdx]['id']; - $currentShardCount = $meta['shards'][$lastShardIdx]['count'] + $meta['shards'][$lastShardIdx]['deleted']; - - foreach($validItems as $item){ - if($currentShardCount >= $shardSize){ - $shardId++; - $meta['shards'][] = array( - "id" => $shardId, - "file" => "_s" . $shardId, - "count" => 0, - "deleted" => 0 - ); - $lastShardIdx = count($meta['shards']) - 1; - $currentShardCount = 0; - } - - if(!isset($shardWrites[$shardId])){ - $shardWrites[$shardId] = ['items' => [], 'shardIdx' => $lastShardIdx]; + private function ensureSegmentFieldIndex($dbname, $shardId, $field){ + $index = $this->readFieldIndex($dbname, $field, $shardId); + if($index !== null){ + return $index; + } + return $this->withLock($dbname, true, function() use ($dbname, $shardId, $field){ + unset($this->fieldIndexCache[$this->getFieldIndexPath($dbname, $field, $shardId)]); + $index = $this->readFieldIndex($dbname, $field, $shardId); + if($index === null){ + $index = $this->buildSegmentFieldIndex($dbname, $shardId, $field); + $this->writeFieldIndex($dbname, $field, $index, $shardId); + $index = $this->readFieldIndex($dbname, $field, $shardId); + if($shardId !== null){ + $this->rebuildGlobalFieldIndex($dbname, $field); } - $shardWrites[$shardId]['items'][] = $item; - $currentShardCount++; - $insertedCount++; - - $meta['shards'][$lastShardIdx]['count']++; - $meta['totalRecords']++; - $meta['nextKey']++; } - - return $meta; + return $index; }); + } - if(!$metaResult['success'] || $metaResult['data'] === null){ - return array("n" => 0, "error" => $metaResult['error'] ?? 'Meta update failed'); + /** + * Candidate keys of a segment for equality filters, from field indexes. + * Null means "no usable index, scan the segment". + * @return array|null + */ + private function fieldIndexCandidates($dbname, $shardId, array $filters){ + if(!$this->fieldIndexEnabled || empty($filters)){ + return null; } - - // v3.0.0: Write to each affected shard using JSONL format - foreach($shardWrites as $shardId => $writeInfo){ - $shardPath = $this->getShardPath($dbname, $shardId); - - // Ensure JSONL format exists - if(!$this->cachedFileExists($shardPath)){ - $this->createJsonlDatabase($dbname, $shardId); - } else if(!$this->isJsonlFormat($shardPath)){ - // Migrate existing JSON to JSONL - $this->migrateToJsonl($shardPath, $dbname, $shardId); - } - - // Read current JSONL index - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null){ - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - } - - // Calculate base key for this shard - $baseKey = $shardId * $shardSize; - - // Bulk append records using global keys - $insertedKeys = []; - clearstatcache(true, $shardPath); - $offset = file_exists($shardPath) ? filesize($shardPath) : 0; - $buffer = ''; - - foreach($writeInfo['items'] as $item){ - $localKey = $index['n']; - $globalKey = $baseKey + $localKey; - $item['key'] = $globalKey; - - $json = json_encode($item, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $index['o'][$globalKey] = [$offset, $length]; - $offset += strlen($json); - $index['n']++; - - $buffer .= $json; - $insertedKeys[] = $globalKey; - } - - // Single write for all records - file_put_contents($shardPath, $buffer, FILE_APPEND | LOCK_EX); - - // Write updated JSONL index - $this->writeJsonlIndex($dbname, $index, $shardId); - - // Update field indexes (with shardId for global index) - if($this->fieldIndexEnabled){ - foreach($writeInfo['items'] as $i => $record){ - $this->updateFieldIndexOnInsert($dbname, $record, $insertedKeys[$i], $shardId); - } + $indexed = $this->getIndexedFields($dbname); + if(empty($indexed)){ + return null; + } + $candidates = null; + foreach($filters as $field => $value){ + if($field === 'key' || !$this->isIndexableValue($value) || !in_array((string)$field, $indexed, true)){ + continue; } - - // Update spatial indexes (v3.1.0) - if($this->spatialIndexEnabled){ - foreach($writeInfo['items'] as $i => $record){ - $this->updateSpatialIndexOnInsert($dbname, $record, $insertedKeys[$i], $shardId); - } + $index = $this->ensureSegmentFieldIndex($dbname, $shardId, $field); + $keys = $index['values'][$this->fieldIndexValueKey($value)] ?? []; + $candidates = $candidates === null ? $keys : array_values(array_intersect($candidates, $keys)); + if(empty($candidates)){ + return []; } } - - // Batch flush spatial indexes after all shards processed (single write per index) - if($this->spatialIndexEnabled){ - $this->flushSpatialIndexes(); - } - - return array("n" => $insertedCount); + return $candidates; } /** - * Find records in sharded database - * @param string $dbname - * @param mixed $filters - * @return array|false + * Apply record changes of one segment to every field index of the database + * @param array $removed [key => old record] + * @param array $added [key => new record] */ - private function findSharded($dbname, $filters){ - $dbname = $this->sanitizeDbName($dbname); - $meta = $this->getCachedMeta($dbname); - if($meta === null){ - return false; - } - - // Flush all shard buffers before read (flush-before-read strategy) - if($this->bufferEnabled && $this->bufferFlushOnRead){ - $this->flushAllShardBuffers($dbname, $meta); + private function applyFieldIndexChanges($dbname, $shardId, array $removed, array $added){ + // Maintained even when fieldIndexEnabled is off (that flag only stops reads from using indexes), + // so turning it back on never exposes a stale index + if(empty($removed) && empty($added)){ + return; } - - // Handle key-based search - use index for O(1) lookup - if(is_array($filters) && count($filters) > 0){ - $filterKeys = array_keys($filters); - if($filterKeys[0] === "key"){ - // Try to use index for fast lookup - $index = $this->getOrBuildIndex($dbname); - if($index !== null){ - $indexResult = $this->findByKeyWithIndex($dbname, $filters['key'], $index); - if($indexResult !== null){ - return $indexResult; - } - // Index lookup failed, fall back to full scan below - } - - // Fallback: Direct shard calculation (still fast for sharded DBs) - $result = []; - $keys = is_array($filters['key']) ? $filters['key'] : array($filters['key']); - - foreach($keys as $globalKey){ - $globalKey = (int)$globalKey; - $shardId = $this->getShardIdForKey($globalKey); - - // Check if shard exists - $shardExists = false; - foreach($meta['shards'] as $shard){ - if($shard['id'] === $shardId){ - $shardExists = true; - break; - } - } - - if(!$shardExists) continue; - - // Use JSONL direct lookup for O(1) performance - // Note: JSONL shards store global keys - $records = $this->findByKeyJsonl($dbname, $globalKey, $shardId); - if($records !== null && !empty($records)){ - $result = array_merge($result, $records); - } + foreach($this->getIndexedFields($dbname) as $field){ + $index = $this->readFieldIndex($dbname, $field, $shardId); + if($index === null){ + // Missing (new shard) or pre-3.2 file: build it from the data, which already holds the change + $index = $this->buildSegmentFieldIndex($dbname, $shardId, $field); + $this->writeFieldIndex($dbname, $field, $index, $shardId); + if($shardId !== null){ + $this->rebuildGlobalFieldIndex($dbname, $field); } - return $result; + continue; } - } - - // Try to use field index for O(1) lookup in sharded database - if($this->fieldIndexEnabled && is_array($filters) && count($filters) > 0){ - $useFieldIndex = false; - $firstFilterField = array_keys($filters)[0]; - $firstFilterValue = $filters[$firstFilterField]; - // Check if first filter field has index in first shard - if($this->hasFieldIndex($dbname, $firstFilterField, $meta['shards'][0]['id'])){ - $useFieldIndex = true; + $touched = []; + $removals = []; + foreach($removed as $key => $record){ + if(array_key_exists($field, $record) && $this->isIndexableValue($record[$field])){ + $removals[$this->fieldIndexValueKey($record[$field])][(int)$key] = true; + } } - - if($useFieldIndex){ - $result = []; - - // Shard-skip optimization: Use global field index to find target shards - $targetShards = null; - if(is_scalar($firstFilterValue) || is_null($firstFilterValue)){ - $targetShards = $this->getTargetShardsFromGlobalIndex($dbname, $firstFilterField, $firstFilterValue); + foreach($removals as $valueKey => $keys){ + if(!isset($index['values'][$valueKey])){ + continue; } - - // If global index available, only scan target shards; otherwise scan all - if($targetShards !== null){ - // Shard-skip: Only iterate target shards - foreach($targetShards as $shardId){ - $this->findShardedFieldIndexScan($dbname, $shardId, $filters, $result); + $kept = []; + foreach($index['values'][$valueKey] as $k){ + if(!isset($keys[$k])){ + $kept[] = $k; } + } + if(empty($kept)){ + unset($index['values'][$valueKey]); } else { - // Fallback: Scan all shards - foreach($meta['shards'] as $shard){ - $this->findShardedFieldIndexScan($dbname, $shard['id'], $filters, $result); - } + $index['values'][$valueKey] = $kept; } - return $result; + $touched[$valueKey] = true; } - } - - // For all other searches, scan all shards - $result = []; - foreach($meta['shards'] as $shard){ - $shardData = $this->getShardData($dbname, $shard['id']); - $baseKey = $shard['id'] * $this->shardSize; - - foreach($shardData['data'] as $localKey => $record){ - if($record === null) continue; - - $globalKey = $baseKey + $localKey; - $record['key'] = $globalKey; + foreach($added as $key => $record){ + if(array_key_exists($field, $record) && $this->isIndexableValue($record[$field])){ + $valueKey = $this->fieldIndexValueKey($record[$field]); + $index['values'][$valueKey][] = (int)$key; + $touched[$valueKey] = true; + } + } + if(empty($touched)){ + continue; + } + $this->writeFieldIndex($dbname, $field, $index, $shardId); - // Return all if no filter - if(is_int($filters) || (is_array($filters) && count($filters) === 0)){ - $result[] = $record; + if($shardId !== null){ + $global = $this->readGlobalFieldIndex($dbname, $field); + if($global === null){ + $this->rebuildGlobalFieldIndex($dbname, $field); continue; } - - // Apply filter - $match = true; - foreach($filters as $field => $value){ - if(!array_key_exists($field, $record) || $record[$field] !== $value){ - $match = false; - break; + $changed = false; + foreach(array_keys($touched) as $valueKey){ + $shards = $global['shardMap'][$valueKey] ?? []; + $has = in_array($shardId, $shards, true); + if(!empty($index['values'][$valueKey]) && !$has){ + $shards[] = $shardId; + $global['shardMap'][$valueKey] = $shards; + $changed = true; + } elseif(empty($index['values'][$valueKey]) && $has){ + $shards = array_values(array_diff($shards, [$shardId])); + if(empty($shards)){ + unset($global['shardMap'][$valueKey]); + } else { + $global['shardMap'][$valueKey] = $shards; + } + $changed = true; } } - if($match){ - $result[] = $record; + if($changed){ + $this->writeGlobalFieldIndex($dbname, $field, $global); } } } - - return $result; } + // ==================== GLOBAL FIELD INDEX (shard skip) ==================== + /** - * Helper: Scan a single shard using field index - * @param string $dbname Database name - * @param int $shardId Shard ID - * @param array $filters Filter conditions - * @param array &$result Result array (passed by reference) + * Get path for global field index file */ - private function findShardedFieldIndexScan($dbname, $shardId, $filters, &$result){ - $candidateKeys = null; - - // Find intersection of keys from all indexed fields in this shard - foreach($filters as $field => $value){ - if(!is_scalar($value) && !is_null($value)) continue; + private function getGlobalFieldIndexPath($dbname, $field){ + return $this->dbBase($dbname) . ".nonedb.gfidx." . $this->encodeIndexFieldName($field); + } - if($this->hasFieldIndex($dbname, $field, $shardId)){ - $fieldKeys = $this->getKeysFromFieldIndex($dbname, $field, $value, $shardId); - if($candidateKeys === null){ - $candidateKeys = $fieldKeys; - } else { - $candidateKeys = array_intersect($candidateKeys, $fieldKeys); - } - if(empty($candidateKeys)){ - return; // No matches in this shard - } - } + /** + * Read global field index (null when missing or predating v3.2) + * @return array|null + */ + private function readGlobalFieldIndex($dbname, $field){ + $path = $this->getGlobalFieldIndexPath($dbname, $field); + if(array_key_exists($path, $this->fieldIndexCache)){ + return $this->fieldIndexCache[$path]; } + $index = $this->atomicRead($path, null); + if(!is_array($index) || ($index['v'] ?? 1) < 2 || !isset($index['shardMap']) || !is_array($index['shardMap'])){ + $index = null; + } + $this->fieldIndexCache[$path] = $index; + return $index; + } - // If we found candidate keys, read the records - if($candidateKeys !== null && !empty($candidateKeys)){ - $shardPath = $this->getShardPath($dbname, $shardId); - $jsonlIndex = $this->readJsonlIndex($dbname, $shardId); - - if($jsonlIndex !== null){ - // JSONL format - use batch read - $offsets = []; - foreach($candidateKeys as $key){ - if(isset($jsonlIndex['o'][$key])){ - $offsets[$key] = $jsonlIndex['o'][$key]; - } - } - - $records = $this->readJsonlRecordsBatch($shardPath, $offsets); - - foreach($records as $record){ - if($record === null) continue; - - // Verify all filters match - $match = true; - foreach($filters as $field => $value){ - if(!array_key_exists($field, $record) || $record[$field] !== $value){ - $match = false; - break; - } - } - if($match){ - $result[] = $record; - } - } - } else { - // Fallback to JSON format - read from shard data - $shardData = $this->getShardData($dbname, $shardId); - $baseKey = $shardId * $this->shardSize; - - foreach($candidateKeys as $localKey){ - if(!isset($shardData['data'][$localKey]) || $shardData['data'][$localKey] === null){ - continue; - } - - $record = $shardData['data'][$localKey]; + /** + * Write global field index (caller holds the exclusive lock) + */ + private function writeGlobalFieldIndex($dbname, $field, $metadata){ + $path = $this->getGlobalFieldIndexPath($dbname, $field); + $metadata['v'] = 2; + $metadata['field'] = (string)$field; + $metadata['updated'] = time(); + if(!$this->atomicWrite($path, $metadata)){ + unset($this->fieldIndexCache[$path]); + throw new \RuntimeException("noneDB: failed to write global field index '{$field}' of database '{$dbname}'"); + } + $this->fieldIndexCache[$path] = $metadata; + return true; + } - // Verify all filters match - $match = true; - foreach($filters as $field => $value){ - if(!array_key_exists($field, $record) || $record[$field] !== $value){ - $match = false; - break; - } - } - if($match){ - $record['key'] = $baseKey + $localKey; - $result[] = $record; - } + /** + * Rebuild the shard map of a field from the shard field indexes (exclusive lock) + */ + private function rebuildGlobalFieldIndex($dbname, $field){ + $global = ['v' => 2, 'field' => (string)$field, 'shardMap' => []]; + foreach($this->listSegments($dbname) as $shardId){ + $index = $this->readFieldIndex($dbname, $field, $shardId); + if($index === null){ + $index = $this->buildSegmentFieldIndex($dbname, $shardId, $field); + $this->writeFieldIndex($dbname, $field, $index, $shardId); + } + foreach($index['values'] as $valueKey => $keys){ + if(!empty($keys)){ + $global['shardMap'][(string)$valueKey][] = $shardId; } } } + $this->writeGlobalFieldIndex($dbname, $field, $global); + return $global; } /** - * Update records in sharded database with atomic locking - * @param string $dbname - * @param array $data - * @return array + * Shards that can contain matches for the filters (null = all shards / not sharded) + * @return array|null */ - private function updateSharded($dbname, $data){ - $dbname = $this->sanitizeDbName($dbname); - $main_response = array("n" => 0); - - $filters = $data[0]; - $setValues = $data[1]['set']; - $shardSize = $this->shardSize; - - $meta = $this->getCachedMeta($dbname); - if($meta === null){ - return $main_response; - } - - // Flush all shard buffers before update - if($this->bufferEnabled){ - $this->flushAllShardBuffers($dbname, $meta); + private function targetShardsFromGlobalIndex($dbname, array $filters){ + if(!$this->fieldIndexEnabled || !$this->isSharded($dbname) || empty($filters)){ + return null; } - - // v3.0.0: Update each shard using JSONL format (batch read for performance) - $totalUpdated = 0; - foreach($meta['shards'] as $shard){ - $shardId = $shard['id']; - $baseKey = $shardId * $shardSize; - $shardPath = $this->getShardPath($dbname, $shardId); - - // Ensure JSONL format (auto-migrate if needed) - $this->ensureJsonlFormat($dbname, $shardId); - - // Read JSONL index - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null || empty($index['o'])) continue; - - // Batch read all records in shard for efficient filtering - $records = $this->readJsonlRecordsBatch($shardPath, $index['o']); - - // Collect keys to update - $keysToUpdate = []; - foreach($records as $globalKey => $record){ - if($record === null) continue; - - // Check if record matches filters - $match = true; - foreach($filters as $filterKey => $filterValue){ - if($filterKey === 'key'){ - $targetKeys = is_array($filterValue) ? $filterValue : [$filterValue]; - if(!in_array($globalKey, $targetKeys)){ - $match = false; - break; - } - } else if(!isset($record[$filterKey]) || $record[$filterKey] !== $filterValue){ - $match = false; - break; - } - } - - if($match){ - $keysToUpdate[] = ['key' => $globalKey, 'record' => $record]; - } + $indexed = $this->getIndexedFields($dbname); + $targets = null; + foreach($filters as $field => $value){ + if($field === 'key' || !$this->isIndexableValue($value) || !in_array((string)$field, $indexed, true)){ + continue; } - - // Prepare batch updates - $batchUpdates = []; - foreach($keysToUpdate as $item){ - $record = $item['record']; - - // Apply updates - foreach($setValues as $field => $value){ - $record[$field] = $value; - } - - // Remove key field (will be re-added by updateJsonlRecordsBatch) - unset($record['key']); - - $batchUpdates[] = ['key' => $item['key'], 'data' => $record]; + $global = $this->readGlobalFieldIndex($dbname, $field); + if($global === null){ + $global = $this->withLock($dbname, true, function() use ($dbname, $field){ + return $this->rebuildGlobalFieldIndex($dbname, $field); + }); } - - // Apply updates using batch method (single index write per shard) - $totalUpdated += $this->updateJsonlRecordsBatch($dbname, $batchUpdates, $shardId); + $shards = array_map('intval', $global['shardMap'][$this->fieldIndexValueKey($value)] ?? []); + $targets = $targets === null ? $shards : array_values(array_intersect($targets, $shards)); } - - return array("n" => $totalUpdated); + return $targets; } /** - * Delete records from sharded database with atomic locking - * @param string $dbname - * @param array $data - * @return array + * Delete global field index file */ - private function deleteSharded($dbname, $data){ - $dbname = $this->sanitizeDbName($dbname); - $main_response = array("n" => 0); - - $filters = $data; - $shardSize = $this->shardSize; - - $meta = $this->getCachedMeta($dbname); - if($meta === null){ - return $main_response; + private function deleteGlobalFieldIndex($dbname, $field){ + $path = $this->getGlobalFieldIndexPath($dbname, $field); + clearstatcache(true, $path); + if(file_exists($path)){ + @unlink($path); } + unset($this->fieldIndexCache[$path]); + } - // Flush all shard buffers before delete - if($this->bufferEnabled){ - $this->flushAllShardBuffers($dbname, $meta); + /** + * After a migration to sharded format: rebuild every field and spatial index per shard and + * remove the non-sharded index files + */ + private function rebuildIndexesAfterMigration($dbname){ + $manifest = $this->getIndexManifest($dbname); + foreach($manifest['fields'] as $field){ + $this->deleteFieldIndexFile($dbname, $field, null); + foreach($this->listSegments($dbname) as $shardId){ + $this->writeFieldIndex($dbname, $field, $this->buildSegmentFieldIndex($dbname, $shardId, $field), $shardId); + } + $this->rebuildGlobalFieldIndex($dbname, $field); } + foreach($manifest['spatial'] as $field){ + $this->rebuildSpatialIndexForDb($dbname, $field); + } + $this->saveIndexManifest($dbname, $manifest); + } - // Track deletions per shard for meta update and index - $shardDeletions = []; - $deletedKeys = []; // Track deleted keys for index update - $totalDeleted = 0; - - // v3.0.0: Delete from each shard using JSONL format (two-phase approach) - // Phase 1: Collect all keys to delete from each shard (batch read for performance) - $keysToDeleteByShard = []; - - foreach($meta['shards'] as $shard){ - $shardId = $shard['id']; - $shardPath = $this->getShardPath($dbname, $shardId); + // ========================================== + // PUBLIC INDEX API + // ========================================== - // Ensure JSONL format (auto-migrate if needed) - $this->ensureJsonlFormat($dbname, $shardId); + /** + * Enable or disable indexing (kept for backward compatibility; key lookups always use the .jidx index) + * @param bool $enable + */ + public function enableIndexing($enable = true){ + $this->indexEnabled = (bool)$enable; + } - // Read JSONL index - $index = $this->readJsonlIndex($dbname, $shardId); - if($index === null || empty($index['o'])) continue; - - // Batch read all records in shard for efficient filtering - $records = $this->readJsonlRecordsBatch($shardPath, $index['o']); - - // Collect keys that match filters - $keysToDelete = []; - foreach($records as $globalKey => $record){ - if($record === null) continue; - - // Check if record matches filters - $match = true; - foreach($filters as $filterKey => $filterValue){ - if($filterKey === 'key'){ - $targetKeys = is_array($filterValue) ? $filterValue : [$filterValue]; - if(!in_array($globalKey, $targetKeys)){ - $match = false; - break; - } - } else if(!array_key_exists($filterKey, $record) || $record[$filterKey] !== $filterValue){ - $match = false; - break; - } - } + /** + * Check if indexing is enabled + * @return bool + */ + public function isIndexingEnabled(){ + return $this->indexEnabled; + } - if($match){ - $keysToDelete[] = $globalKey; - } - } + /** + * Rebuild the key indexes (.jidx) of a database from its data files. + * Normal operation never needs this; it is a recovery tool after files were copied or restored by hand. + * @param string $dbname + * @return array ['success' => bool, 'totalRecords' => int, 'time' => string] + */ + public function rebuildIndex($dbname){ + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname){ + return $this->rebuildIndexUnlocked($dbname); + }); + } - if(!empty($keysToDelete)){ - $keysToDeleteByShard[$shardId] = $keysToDelete; - } + private function rebuildIndexUnlocked($dbname){ + $start = microtime(true); + $this->flushPendingBuffers($dbname); + $segments = $this->listSegments($dbname); + if(empty($segments)){ + return ['success' => false, 'error' => 'Database does not exist']; } - - // Phase 2: Delete collected keys using batch delete (single index write per shard) - foreach($keysToDeleteByShard as $shardId => $keysToDelete){ - $deletedInShard = $this->deleteJsonlRecordsBatch($dbname, $keysToDelete, $shardId); - - if($deletedInShard > 0){ - $shardDeletions[$shardId] = $deletedInShard; - $deletedKeys = array_merge($deletedKeys, $keysToDelete); - $totalDeleted += $deletedInShard; - } + $total = 0; + foreach($segments as $segment){ + $this->ensureJsonlFormat($dbname, $segment); + $total += count($this->rebuildSegmentIndexFromData($dbname, $segment)['o']); } + if($this->isSharded($dbname)){ + $this->recountMeta($dbname); + } + return [ + 'success' => true, + 'totalRecords' => $total, + 'time' => round((microtime(true) - $start) * 1000, 2) . 'ms' + ]; + } - // Atomically update meta with deletion counts - if($totalDeleted > 0){ - $this->modifyMeta($dbname, function($meta) use ($shardDeletions, $totalDeleted) { - if($meta === null) return null; - - foreach($meta['shards'] as &$shard){ - if(isset($shardDeletions[$shard['id']])){ - $shard['count'] -= $shardDeletions[$shard['id']]; - $shard['deleted'] += $shardDeletions[$shard['id']]; - } - } - - $meta['totalRecords'] -= $totalDeleted; - $meta['deletedCount'] = ($meta['deletedCount'] ?? 0) + $totalDeleted; - return $meta; - }); + /** + * Get index information for a database + * @param string $dbname + * @return array|null + */ + public function getIndexInfo($dbname){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname){ + return $this->getIndexInfoUnlocked($dbname); + }); + } - // Update index with deleted keys - $this->updateIndexOnDelete($dbname, $deletedKeys); + private function getIndexInfoUnlocked($dbname){ + $segments = $this->listSegments($dbname); + if(empty($segments)){ + return null; } - - return array("n" => $totalDeleted); + $total = 0; + $created = null; + $updated = null; + foreach($segments as $segment){ + $index = $this->readJsonlIndex($dbname, $segment); + if($index === null){ + continue; + } + $total += count($index['o']); + $created = $created === null ? ($index['created'] ?? null) : min($created, $index['created'] ?? $created); + $updated = max((int)$updated, (int)($index['updated'] ?? 0)); + } + $sharded = $this->isSharded($dbname); + return [ + 'exists' => true, + 'version' => 3, + 'created' => $created, + 'updated' => $updated, + 'totalRecords' => $total, + 'sharded' => $sharded, + 'path' => $sharded ? $this->getMetaPath($dbname) : $this->getJsonlIndexPath($dbname, null) + ]; } /** @@ -3732,81 +3539,140 @@ private function deleteSharded($dbname, $data){ * if auto create db is true will be create db * if auto create db is false and is not in db dir return false * @param string $dbname + * @return bool + * @throws \InvalidArgumentException for names with characters outside [A-Za-z0-9' -] */ - function checkDB($dbname=null){ - if(!$dbname){ + public function checkDB($dbname=null){ + if($dbname === null || $dbname === '' || $dbname === false){ return false; } - $dbname = $this->sanitizeDbName($dbname); - // Sanitize sonrası boş string kontrolü - if($dbname === ''){ + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname){ + return $this->checkDBUnlocked($dbname); + }); + } + + private function checkDBUnlocked($dbname){ + $this->ensureDbDir(); + + // Existing database (non-sharded data file or sharded meta) + $fullDBPath = $this->getDataPath($dbname); + if($this->cachedFileExists($fullDBPath) || $this->cachedFileExists($this->getMetaPath($dbname))){ + return true; + } + + if($this->autoCreateDB){ + return $this->createDBUnlocked($dbname); + } + return false; + } + + /** + * create db function + * @param string $dbname + * @return bool True if created, false if it already existed + */ + public function createDB($dbname){ + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname){ + return $this->createDBUnlocked($dbname); + }); + } + + private function createDBUnlocked($dbname){ + $this->ensureDbDir(); + $fullDBPath = $this->getDataPath($dbname); + if($this->cachedFileExists($fullDBPath) || $this->cachedFileExists($this->getMetaPath($dbname))){ return false; } - /** - * if db dir is not in project folder will be create. - */ - if(!file_exists($this->dbDir)){ - mkdir($this->dbDir, 0777); + + // Create info file + @file_put_contents($fullDBPath . "info", (string)time()); + + // v3.0: Create empty JSONL format database + if(@touch($fullDBPath) === false){ + throw new \RuntimeException("noneDB: cannot create database file for '{$dbname}'"); } - $dbnameHashed=$this->hashDBName($dbname); - $fullDBPath=$this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - /** - * check db is in db folder? (use cache for existing files) - */ - if($this->cachedFileExists($fullDBPath)){ - return true; - } + // Create empty JSONL index + $index = [ + 'v' => 3, + 'format' => 'jsonl', + 'created' => time(), + 'n' => 0, + 'd' => 0, + 'o' => [] + ]; + $this->writeJsonlIndex($dbname, $index); + + // Mark file as existing in cache + $this->markFileExists($fullDBPath); + $this->jsonlFormatCache[$fullDBPath] = true; - /** - * if auto create db is true will be create db. - */ - if($this->autoCreateDB){ - return $this->createDB($dbname); - } - return false; + return true; } /** - * create db function + * Delete a database and every file that belongs to it (data, shards, indexes, buffers, metadata) * @param string $dbname - */ - public function createDB($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $dbnameHashed=$this->hashDBName($dbname); - $fullDBPath=$this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - if(!file_exists($this->dbDir)){ - mkdir($this->dbDir, 0777); - } - if(!$this->cachedFileExists($fullDBPath)){ - // Create info file - $infoDB = fopen($fullDBPath."info", "a+"); - fwrite($infoDB, time()); - fclose($infoDB); - - // v3.0: Create empty JSONL format database - touch($fullDBPath); - - // Create empty JSONL index - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - $this->writeJsonlIndex($dbname, $index); - - // Mark file as existing in cache - $this->markFileExists($fullDBPath); + * @return array ['success' => bool, 'files' => int deleted file count] + */ + public function dropDB($dbname){ + $dbname = $this->normalizeDbName($dbname); + if(!$this->dbFilesExist($dbname)){ + return ['success' => false, 'files' => 0, 'error' => 'Database does not exist']; + } + return $this->withLock($dbname, true, function() use ($dbname){ + $base = $this->dbBase($dbname); + $hashBase = $this->dbDir . $this->hashDBName($dbname); + $lockPath = $this->getLockPath($dbname); + $files = array_merge( + glob($base . '.nonedb*') ?: [], + glob($base . '_s*.nonedb*') ?: [], + glob($hashBase . '.nonedb.*') ?: [], + glob($hashBase . '_s*.nonedb.*') ?: [] + ); + $deleted = 0; + foreach(array_unique($files) as $file){ + // The lock file stays: other processes may be waiting on it, and it carries the generation counter + if($file === $lockPath || !is_file($file)){ + continue; + } + if(@unlink($file)){ + $deleted++; + } + } + $this->invalidateDbCaches($dbname); + // Nothing is left: no buffer, no journal, no pre-3.2 files + self::$lockRegistry[$lockPath]['flags'] = ['v' => true]; + $this->saveLockState($lockPath); + return ['success' => true, 'files' => $deleted]; + }); + } - return true; + /** + * Name, creation time and total size of a database (all shards included) + * @return array|false + */ + private function getDBInfo($dbname){ + $base = $this->dbBase($dbname); + $infoPath = $base . '.nonedbinfo'; + clearstatcache(); + if(!file_exists($infoPath)){ + return false; } - return false; + $created = (int)@file_get_contents($infoPath); + $size = 0; + if(file_exists($base . '.nonedb')){ + $size += (int)filesize($base . '.nonedb'); + } + foreach(glob($base . '_s*.nonedb') ?: [] as $shardFile){ + if(preg_match('/_s\d+\.nonedb$/', $shardFile)){ + $size += (int)filesize($shardFile); + } + } + return array("name" => $dbname, "createdTime" => $created, "size" => $this->fileSizeConvert($size)); } - - /** * Convert bytes to human readable format * @param float $bytes @@ -3840,53 +3706,41 @@ public function getDBs($info=false){ if(is_bool($info)){ $withMetadata = $info; }else{ - $specificDb = $this->sanitizeDbName($info); - $this->checkDB($specificDb); + $specificDb = $this->normalizeDbName($info); // reading information never creates the database } // If specific database requested if($specificDb !== null){ - $dbnameHashed=$this->hashDBName($specificDb); - $fullDBPathInfo=$this->dbDir.$dbnameHashed."-".$specificDb.".nonedbinfo"; - $fullDBPath=$this->dbDir.$dbnameHashed."-".$specificDb.".nonedb"; - if(file_exists($fullDBPathInfo)){ - $dbInfo = fopen($fullDBPathInfo, "r"); - clearstatcache(true, $fullDBPath); // Clear cache before getting file size - $db= array("name"=>$specificDb, "createdTime"=>(int)fgets($dbInfo), "size"=>$this->fileSizeConvert(filesize($fullDBPath))); - fclose($dbInfo); - return $db; - } - return false; + return $this->getDBInfo($specificDb); } - // List all databases + // List all databases: one entry per {hash}-{name}.nonedb (non-sharded) or {hash}-{name}.nonedb.meta (sharded) $dbs = []; - if(!file_exists($this->dbDir)){ + if(!is_dir($this->dbDir)){ return $dbs; } + $seen = []; foreach(new DirectoryIterator($this->dbDir) as $item) { - if(!$item->isDot() && $item->isFile()) { - $filename = $item->getFilename(); - $parts = explode('-', $filename, 2); - if(count($parts) < 2) continue; - - $dbb = explode('.', $parts[1]); - if(count($dbb) < 2 || $dbb[1] !== "nonedb") continue; - - $dbname = $dbb[0]; - - if($withMetadata){ - $dbnameHashed=$this->hashDBName($dbname); - $fullDBPathInfo=$this->dbDir.$dbnameHashed."-".$dbname.".nonedbinfo"; - $fullDBPath=$this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - if(file_exists($fullDBPathInfo)){ - $dbInfo = fopen($fullDBPathInfo, "r"); - $dbs[]= array("name"=>$dbname, "createdTime"=>(int)fgets($dbInfo), "size"=>$this->fileSizeConvert(filesize($fullDBPath))); - fclose($dbInfo); - } - }else{ - $dbs[]= $dbname; + if($item->isDot() || !$item->isFile()) { + continue; + } + if(!preg_match("/^([0-9a-f]{20})-([A-Za-z0-9' -]+)\\.nonedb(\\.meta)?$/", $item->getFilename(), $m)){ + continue; + } + $dbname = $m[2]; + // Only databases that belong to this secretKey + if($this->hashDBName($dbname) !== $m[1] || isset($seen[$dbname])){ + continue; + } + $seen[$dbname] = true; + + if($withMetadata){ + $info = $this->getDBInfo($dbname); + if($info !== false){ + $dbs[] = $info; } + }else{ + $dbs[] = $dbname; } } return $dbs; @@ -3897,12 +3751,16 @@ public function getDBs($info=false){ * limit function * @param array $array Extract a slice of the array * @param integer $limit + * @return array|false False for invalid arguments */ public function limit($array, $limit=0){ if(!is_array($array) || !is_int($limit) || $limit <= 0){ return false; } - // Multidimensional array kontrolü + if(empty($array)){ + return []; + } + // Must be a list of records if(count($array) === count($array, COUNT_RECURSIVE)) { return false; } @@ -3910,129 +3768,75 @@ public function limit($array, $limit=0){ } /** - * read db all data - * @param string $dbname - * @param mixed $filters 0 for all, array for filter + * Run a read operation: shared lock while the database exists, no lock (and no files created) + * when it doesn't. Buffered records are flushed first. */ - public function find($dbname, $filters=0){ - $dbname = $this->sanitizeDbName($dbname); - - // Check for sharded database first - if($this->isSharded($dbname)){ - return $this->findSharded($dbname, $filters); + private function withReadLock($dbname, callable $fn){ + if(!$this->hasLockFile($dbname) && !$this->dbFilesExist($dbname)){ + return $fn(); } + return $this->withLock($dbname, false, function() use ($dbname, $fn){ + $this->flushPendingBuffers($dbname); + return $fn(); + }); + } - // Flush buffer before read (flush-before-read strategy) - if($this->bufferEnabled && $this->bufferFlushOnRead){ - $bufferPath = $this->getBufferPath($dbname); - if($this->hasBuffer($bufferPath)){ - $this->flushBufferToMain($dbname); + /** + * Uncached existence check used before taking a lock + */ + private function dbFilesExist($dbname){ + $base = $this->dbBase($dbname); + foreach(['.nonedb', '.nonedb.meta', '.nonedb.buffer', '.nonedb.buffer.flushing'] as $suffix){ + clearstatcache(true, $base . $suffix); + if(file_exists($base . $suffix)){ + return true; } } + return false; + } - $dbnameHashed=$this->hashDBName($dbname); - $fullDBPath=$this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - if(!$this->checkDB($dbname)){ - return false; - } - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $jsonlIndex = $this->readJsonlIndex($dbname); - if($jsonlIndex === null){ - return []; + /** + * True when the database's lock file exists (the database may have been dropped since). Taking + * that lock creates no file, and the operation itself finds out whether the database exists. + */ + private function hasLockFile($dbname){ + $path = $this->getLockPath($dbname); + if(isset(self::$lockHandles[$path]) || isset(self::$lockRegistry[$path])){ + return true; } + clearstatcache(true, $path); + return file_exists($path); + } - // Key-based search - O(1) lookup - if(is_array($filters) && count($filters) > 0){ - $filterKeys = array_keys($filters); - if($filterKeys[0] === "key"){ - $result = $this->findByKeyJsonl($dbname, $filters['key']); - return $result !== null ? $result : []; - } + /** + * read db all data + * @param string $dbname + * @param mixed $filters 0/null/[] for all records, or an array of field => value (strict ===); + * a "key" entry selects by record key (int or list of ints) + * @return array|false False when the database doesn't exist and autoCreateDB is off, or for invalid filters + */ + public function find($dbname, $filters=0){ + $dbname = $this->normalizeDbName($dbname); + if($filters === null || is_int($filters)){ + $filters = []; } - - // Return all if no filter - if(is_int($filters) || (is_array($filters) && count($filters) === 0)){ - $allRecords = $this->readAllJsonl($fullDBPath, $jsonlIndex); - return $allRecords; + if(!is_array($filters)){ + return false; } + return $this->withReadLock($dbname, function() use ($dbname, $filters){ + return $this->findUnlocked($dbname, $filters); + }); + } - // Try to use field index for O(1) lookup - if($this->fieldIndexEnabled && is_array($filters)){ - $candidateKeys = null; - - // Find intersection of keys from all indexed fields - foreach($filters as $field => $value){ - if(!is_scalar($value) && !is_null($value)) continue; - - if($this->hasFieldIndex($dbname, $field, null)){ - $fieldKeys = $this->getKeysFromFieldIndex($dbname, $field, $value, null); - if($candidateKeys === null){ - $candidateKeys = $fieldKeys; - } else { - $candidateKeys = array_intersect($candidateKeys, $fieldKeys); - } - // Early exit if no matches - if(empty($candidateKeys)){ - return []; - } - } - } - - // If we found candidate keys from field indexes, use batch read - if($candidateKeys !== null){ - // Build offsets array for batch reading - $offsets = []; - foreach($candidateKeys as $key){ - if(isset($jsonlIndex['o'][$key])){ - $offsets[$key] = $jsonlIndex['o'][$key]; - } - } - - // Batch read all matching records at once - $records = $this->readJsonlRecordsBatch($fullDBPath, $offsets); - - // Filter and verify matches - $result = []; - foreach($records as $record){ - if($record === null) continue; - - // Verify all filters match (some fields may not have indexes) - $match = true; - foreach($filters as $field => $value){ - if(!array_key_exists($field, $record) || $record[$field] !== $value){ - $match = false; - break; - } - } - if($match){ - $result[] = $record; - } - } - return $result; - } + private function findUnlocked($dbname, $filters=0){ + if($filters === null || is_int($filters)){ + $filters = []; } - - // Fallback: Get all records and filter - $allRecords = $this->readAllJsonl($fullDBPath, $jsonlIndex); - - // Apply filters - $result = []; - foreach($allRecords as $record){ - $match = true; - foreach($filters as $field => $value){ - if(!array_key_exists($field, $record) || $record[$field] !== $value){ - $match = false; - break; - } - } - if($match){ - $result[] = $record; - } + if(!$this->dbExists($dbname)){ + // Reads never create databases + return $this->autoCreateDB ? [] : false; } - return $result; + return array_values($this->selectRecords($dbname, $filters)); } /** @@ -4045,7 +3849,7 @@ private function hasReservedKeyField($data){ } /** - * Check if array is a list of records (numeric keys with array values) + * Check if array is a list of records: sequential 0..n-1 keys, every element an array * @param array $data * @return bool */ @@ -4053,343 +3857,155 @@ private function isRecordList($data){ if(!is_array($data) || count($data) === 0){ return false; } - // Check if first key is numeric and value is array - $keys = array_keys($data); - if(!is_int($keys[0])){ - return false; + $i = 0; + foreach($data as $k => $v){ + if($k !== $i++ || !is_array($v)){ + return false; + } } - // Check if first element is an array (a record) - return is_array($data[$keys[0]]); + return true; } /** - * insert to db - * @param string $dbname - * @param array $data + * Validate insert data + * @return array [items, error|null] */ - public function insert($dbname, $data){ - $dbname = $this->sanitizeDbName($dbname); - $main_response=array("n"=>0); - + private function validateInsertData($data){ if(!is_array($data)){ - $main_response['error']="insert data must be array"; - return $main_response; - } - - // Check for sharded database first - if($this->isSharded($dbname)){ - return $this->insertSharded($dbname, $data); - } - - // Validate data before any operation - $validItems = []; - if($this->isRecordList($data)){ - foreach($data as $item){ - if(!is_array($item)){ - continue; - } - if($this->hasReservedKeyField($item)){ - $main_response['error']="You cannot set key name to key"; - return $main_response; - } - $validItems[] = $item; - } - } else { - if($this->hasReservedKeyField($data)){ - $main_response['error']="You cannot set key name to key"; - return $main_response; - } - $validItems[] = $data; - } - - if(empty($validItems)){ - return array("n"=>0); - } - - // Use buffered insert if enabled - if($this->bufferEnabled){ - return $this->insertBuffered($dbname, $validItems); + return [[], "insert data must be array"]; } - - // Non-buffered insert (original atomic method) - return $this->insertDirect($dbname, $validItems); - } - - /** - * Buffered insert - fast append-only to buffer file - * @param string $dbname - * @param array $validItems Pre-validated items - * @return array - */ - private function insertBuffered($dbname, array $validItems){ - // Ensure database metadata exists (creates .nonedbinfo file) - $this->checkDB($dbname); - - // Register shutdown handler for auto-flush - $this->registerShutdownHandler(); - - $bufferPath = $this->getBufferPath($dbname); - - // Check if buffer needs flushing before insert - if($this->shouldFlushBuffer($bufferPath, $dbname)){ - $this->flushBufferToMain($dbname); + if(empty($data)){ + return [[], "insert data must not be empty"]; } - - // Append to buffer (fast, no full file read) - $result = $this->atomicAppendToBuffer($bufferPath, $validItems); - - if(!$result['success']){ - return array("n" => 0, "error" => $result['error']); + $keys = array_keys($data); + $looksLikeList = is_int($keys[0]) && is_array($data[$keys[0]]); + if($looksLikeList && !$this->isRecordList($data)){ + return [[], "insert data must be either one record or a list of records"]; } - - // Check again after insert if we crossed threshold - if($this->shouldFlushBuffer($bufferPath, $dbname)){ - $flushResult = $this->flushBufferToMain($dbname); - - // After flush, check if main DB needs sharding - if($flushResult['success'] && $this->shardingEnabled && $this->autoMigrate){ - $index = $this->readJsonlIndex($dbname); - if($index !== null && $index['n'] >= $this->shardSize){ - $this->migrateToSharded($dbname); - } + $items = $looksLikeList ? array_values($data) : [$data]; + foreach($items as $item){ + if($this->hasReservedKeyField($item)){ + return [[], "You cannot set key name to key"]; + } + if($this->encodeJson($item) === false){ + return [[], "insert data cannot be encoded as JSON (invalid UTF-8, NAN/INF or unsupported value)"]; } } - - return array("n" => $result['count']); + return [$items, null]; } /** - * Direct insert without buffer - uses atomic modify + * insert to db * @param string $dbname - * @param array $validItems Pre-validated items - * @return array + * @param array $data One record, or a list of records + * @return array ['n' => inserted count] or ['n' => 0, 'error' => message] */ - private function insertDirect($dbname, array $validItems){ - $this->checkDB($dbname); - $dbnameHashed = $this->hashDBName($dbname); - $fullDBPath = $this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - - $countData = count($validItems); - - // Ensure JSONL format (auto-migrate v2 if needed) - if(!$this->ensureJsonlFormat($dbname)){ - // DB doesn't exist yet, create as JSONL - $this->createJsonlDatabase($dbname); - } - - $index = $this->readJsonlIndex($dbname); - if($index === null){ - return array("n" => 0, "error" => "Failed to read index"); - } - - // Use bulk append for multiple records - $keys = $this->bulkAppendJsonl($fullDBPath, $validItems, $index); - $this->writeJsonlIndex($dbname, $index); - - // Update field indexes for inserted records - if($this->fieldIndexEnabled){ - foreach($validItems as $i => $record){ - $this->updateFieldIndexOnInsert($dbname, $record, $keys[$i], null); - } + public function insert($dbname, $data){ + $dbname = $this->normalizeDbName($dbname); + list($items, $error) = $this->validateInsertData($data); + if($error !== null){ + return array("n" => 0, "error" => $error); } - - // Update spatial indexes for inserted records (v3.1.0) - if($this->spatialIndexEnabled){ - foreach($validItems as $i => $record){ - $this->updateSpatialIndexOnInsert($dbname, $record, $keys[$i], null); + return $this->withLock($dbname, true, function() use ($dbname, $items){ + if($this->bufferEnabled){ + return $this->insertBuffered($dbname, $items); } - // Batch flush spatial indexes (single write instead of N writes) - $this->flushSpatialIndexes(); - } - - // Auto-migrate to sharded format if threshold reached - if($this->shardingEnabled && $this->autoMigrate && $index['n'] >= $this->shardSize){ - $this->migrateToSharded($dbname); - } - - return array("n" => $countData); + $this->flushPendingBuffers($dbname); + return $this->insertRecords($dbname, $items); + }); } /** * delete function * @param string $dbname - * @param array $data + * @param array $data Filters (same rules as find); [] deletes every record + * @return array ['n' => deleted count] */ public function delete($dbname, $data){ - $dbname = $this->sanitizeDbName($dbname); - $main_response=array("n"=>0); + $dbname = $this->normalizeDbName($dbname); if(!is_array($data)){ - $main_response['error']="Please check your delete paramters"; - return $main_response; - } - - // Check for sharded database first - if($this->isSharded($dbname)){ - return $this->deleteSharded($dbname, $data); - } - - // Flush buffer before delete operation - if($this->bufferEnabled){ - $bufferPath = $this->getBufferPath($dbname); - if($this->hasBuffer($bufferPath)){ - $this->flushBufferToMain($dbname); - } - } - - $this->checkDB($dbname); - $dbnameHashed=$this->hashDBName($dbname); - $fullDBPath=$this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $filters = $data; - $deletedCount = 0; - - // Key-based delete - O(1) - if(isset($filters['key'])){ - $targetKeys = is_array($filters['key']) ? $filters['key'] : [$filters['key']]; - foreach($targetKeys as $key){ - if($this->deleteJsonlRecord($dbname, $key)){ - $deletedCount++; - } - } - return array("n" => $deletedCount); + return array("n" => 0, "error" => "Please check your delete parameters"); } + return $this->withLock($dbname, true, function() use ($dbname, $data){ + return $this->deleteUnlocked($dbname, $data); + }); + } - // Filter-based delete - need to scan (batch read for performance) - $index = $this->readJsonlIndex($dbname); - if($index === null || empty($index['o'])){ + private function deleteUnlocked($dbname, $data){ + $this->flushPendingBuffers($dbname); + if(!$this->dbExists($dbname)){ return array("n" => 0); } - - // Batch read all records for efficient filtering - $records = $this->readJsonlRecordsBatch($fullDBPath, $index['o']); - - // First pass: collect all keys to delete - $keysToDelete = []; - foreach($records as $key => $record){ - if($record === null) continue; - - $match = true; - foreach($filters as $filterKey => $filterValue){ - if(!isset($record[$filterKey]) || $record[$filterKey] !== $filterValue){ - $match = false; - break; - } - } - - if($match){ - $keysToDelete[] = $key; - } + $deleted = 0; + foreach($this->groupBySegment($dbname, $this->selectRecords($dbname, $data)) as $segment => $records){ + $deleted += $this->deleteSegmentRecords($dbname, $segment === '' ? null : (int)$segment, $records); } - - // Second pass: delete collected keys using batch delete (single index write) - $deletedCount = $this->deleteJsonlRecordsBatch($dbname, $keysToDelete); - - return array("n" => $deletedCount); + return array("n" => $deleted); } /** * update function * @param string $dbname - * @param array $data + * @param array $data [filters, ['set' => [field => value, ...]]] + * @return array ['n' => updated count] */ public function update($dbname, $data){ - $dbname = $this->sanitizeDbName($dbname); - $main_response=array("n"=>0); - if(!is_array($data) || count($data) === count($data, COUNT_RECURSIVE) || !isset($data[1]['set']) || array_key_exists("key", $data[1]['set'])){ - $main_response['error']="Please check your update paramters"; - return $main_response; - } - - // Check for sharded database first - if($this->isSharded($dbname)){ - return $this->updateSharded($dbname, $data); + $dbname = $this->normalizeDbName($dbname); + if(!is_array($data) || !array_key_exists(0, $data) || !is_array($data[0]) + || !isset($data[1]) || !is_array($data[1]) || !isset($data[1]['set']) || !is_array($data[1]['set']) + || array_key_exists("key", $data[1]['set'])){ + return array("n" => 0, "error" => "Please check your update parameters"); } - - // Flush buffer before update operation - if($this->bufferEnabled){ - $bufferPath = $this->getBufferPath($dbname); - if($this->hasBuffer($bufferPath)){ - $this->flushBufferToMain($dbname); - } + if($this->encodeJson($data[1]['set']) === false){ + return array("n" => 0, "error" => "update data cannot be encoded as JSON (invalid UTF-8, NAN/INF or unsupported value)"); } + return $this->withLock($dbname, true, function() use ($dbname, $data){ + return $this->updateUnlocked($dbname, $data); + }); + } - $this->checkDB($dbname); - $dbnameHashed=$this->hashDBName($dbname); - $fullDBPath=$this->dbDir.$dbnameHashed."-".$dbname.".nonedb"; - - $filters = $data[0]; - $setData = $data[1]['set']; - $updatedCount = 0; - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $index = $this->readJsonlIndex($dbname); - if($index === null){ + private function updateUnlocked($dbname, $data){ + $this->flushPendingBuffers($dbname); + if(!$this->dbExists($dbname)){ return array("n" => 0); } - - // Key-based update - O(1) lookup, batch update - if(isset($filters['key'])){ - $targetKeys = is_array($filters['key']) ? $filters['key'] : [$filters['key']]; - $batchUpdates = []; - - foreach($targetKeys as $key){ - if(!isset($index['o'][$key])) continue; - - $record = $this->readJsonlRecord($fullDBPath, $index['o'][$key][0], $index['o'][$key][1]); - if($record === null) continue; - - // Apply updates - foreach($setData as $setKey => $setValue){ - $record[$setKey] = $setValue; - } - - // Remove key field (will be re-added by updateJsonlRecordsBatch) - unset($record['key']); - - $batchUpdates[] = ['key' => $key, 'data' => $record]; - } - - $updatedCount = $this->updateJsonlRecordsBatch($dbname, $batchUpdates); - return array("n" => $updatedCount); + $set = $data[1]['set']; + if(empty($set)){ + return array("n" => count($this->selectRecords($dbname, $data[0]))); } - - // Filter-based update - need to scan (batch read for performance) - $records = $this->readJsonlRecordsBatch($fullDBPath, $index['o']); - - $batchUpdates = []; - foreach($records as $key => $record){ - if($record === null) continue; - - $match = true; - foreach($filters as $filterKey => $filterValue){ - if(!isset($record[$filterKey]) || $record[$filterKey] !== $filterValue){ - $match = false; - break; + $updated = 0; + foreach($this->groupBySegment($dbname, $this->selectRecords($dbname, $data[0])) as $segment => $records){ + $new = []; + foreach($records as $key => $record){ + foreach($set as $field => $value){ + $record[$field] = $value; } + $new[$key] = $record; } + $updated += $this->updateSegmentRecords($dbname, $segment === '' ? null : (int)$segment, $records, $new); + } + return array("n" => $updated); + } - if($match){ - // Apply updates - foreach($setData as $setKey => $setValue){ - $record[$setKey] = $setValue; - } - - // Remove key field (will be re-added by updateJsonlRecordsBatch) - unset($record['key']); - - $batchUpdates[] = ['key' => $key, 'data' => $record]; + /** + * Replace whole records by key (used by noneDBQuery::removeFields); caller holds the exclusive lock + * @param array $records [key => new record without "key"] + * @return int + */ + private function replaceRecordsByKey($dbname, array $records){ + if(empty($records) || !$this->dbExists($dbname)){ + return 0; + } + $old = $this->selectRecords($dbname, ['key' => array_keys($records)]); + $written = 0; + foreach($this->groupBySegment($dbname, $old) as $segment => $oldRecords){ + $new = []; + foreach(array_keys($oldRecords) as $key){ + $new[$key] = $records[$key]; } + $written += $this->updateSegmentRecords($dbname, $segment === '' ? null : (int)$segment, $oldRecords, $new); } - - // Apply updates using batch method (single index write) - $updatedCount = $this->updateJsonlRecordsBatch($dbname, $batchUpdates); - - return array("n" => $updatedCount); + return $written; } // ========================================== @@ -4403,6 +4019,13 @@ public function update($dbname, $data){ * @return array|false */ public function distinct($dbname, $field){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field){ + return $this->distinctUnlocked($dbname, $field); + }); + } + + private function distinctUnlocked($dbname, $field){ $all = $this->find($dbname, 0); if($all === false) return false; $values = []; @@ -4422,14 +4045,52 @@ public function distinct($dbname, $field){ * @return array|false */ public function sort($array, $field, $order = 'asc'){ - if(!is_array($array) || count($array) === 0) return false; - $order = strtolower($order); - usort($array, function($a, $b) use ($field, $order){ - if(!isset($a[$field]) || !isset($b[$field])) return 0; - $result = $a[$field] <=> $b[$field]; - return $order === 'desc' ? -$result : $result; + if(!is_array($array)) return false; + if(count($array) === 0) return []; + $desc = strtolower((string)$order) === 'desc'; + $items = []; + foreach(array_values($array) as $i => $row){ + $items[] = [self::sortKey(is_array($row) && array_key_exists($field, $row) ? $row[$field] : null), $i, $row]; + } + usort($items, function($a, $b) use ($desc){ + $result = $a[0] <=> $b[0]; + if($desc){ + $result = -$result; + } + return $result !== 0 ? $result : $a[1] <=> $b[1]; // stable }); - return $array; + return array_column($items, 2); + } + + /** + * Total order used by sort(): missing/null < booleans < numbers < strings < arrays. + * Values of the same kind compare naturally (numbers numerically, strings with <=>). + * @return int + */ + private static function compareSortValues($a, $b){ + return self::sortKey($a) <=> self::sortKey($b); + } + + /** + * Sort key of a value: [kind, comparable value]. Two keys compare with <=> in the order above + * (kind first, then the value; arrays by their JSON), so sorts compute each key once instead of + * classifying both values in every comparison. + * @return array + */ + private static function sortKey($v){ + if($v === null) return [0, 0]; + if(is_bool($v)) return [1, $v]; + if(is_int($v) || is_float($v)) return [2, $v]; + if(is_string($v)) return [3, $v]; + return [4, (string)json_encode($v)]; + } + + /** + * sortKey() for the query builder + * @return \Closure + */ + private function sortKeyFunction(){ + return \Closure::fromCallable([self::class, 'sortKey']); } /** @@ -4439,43 +4100,36 @@ public function sort($array, $field, $order = 'asc'){ * @return int */ public function count($dbname, $filter = 0){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $filter){ + return $this->countUnlocked($dbname, $filter); + }); + } - // Fast-path: No filter = use index/meta count directly (v3.0.0 optimization) - if($filter === 0 || (is_array($filter) && empty($filter))){ + private function countUnlocked($dbname, $filter = 0){ + // Fast-path: no filter = use index/meta counters + if($filter === 0 || $filter === null || (is_array($filter) && empty($filter))){ return $this->countFast($dbname); } - - // Filtered count still needs to scan - $result = $this->find($dbname, $filter); + $result = $this->findUnlocked($dbname, $filter); return $result === false ? 0 : count($result); } /** - * Fast count using index/meta data - O(1) for unfiltered count - * v3.0.0 optimization: Avoids loading all records into memory - * @param string $dbname Already sanitized + * Count without reading records: index entries (non-sharded) or meta totalRecords (sharded) + * @param string $dbname Already normalized * @return int */ private function countFast($dbname){ - // Sharded database: use metadata - // Note: totalRecords is already decremented after delete operations - // deletedCount tracks garbage records for compaction, not active count - if($this->isSharded($dbname)){ - $meta = $this->getCachedMeta($dbname); - if($meta !== null){ - return $meta['totalRecords'] ?? 0; - } + if(!$this->dbExists($dbname)){ return 0; } - - // Non-sharded: use JSONL index offset count - $index = $this->readJsonlIndex($dbname); - if($index !== null && isset($index['o'])){ - return count($index['o']); + if($this->isSharded($dbname)){ + $meta = $this->getCachedMeta($dbname); + return $meta !== null ? (int)($meta['totalRecords'] ?? 0) : 0; } - - return 0; + $index = $this->readJsonlIndex($dbname, null); + return $index !== null ? count($index['o']) : 0; } /** @@ -4486,29 +4140,46 @@ private function countFast($dbname){ * @return array|false */ public function like($dbname, $field, $pattern){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $pattern){ + return $this->likeUnlocked($dbname, $field, $pattern); + }); + } + + private function likeUnlocked($dbname, $field, $pattern){ $all = $this->find($dbname, 0); if($all === false) return false; $result = []; - // Convert simple patterns to regex - if(strpos($pattern, '^') === 0 || substr($pattern, -1) === '$'){ - $regex = '/' . $pattern . '/i'; - }else{ - $regex = '/' . preg_quote($pattern, '/') . '/i'; - } + $regex = $this->likePatternToRegex((string)$pattern); foreach($all as $record){ if(!isset($record[$field])) continue; $value = $record[$field]; // Skip arrays and objects - can't do string matching on them if(is_array($value) || is_object($value)) continue; - if(preg_match($regex, (string)$value)){ + if(preg_match($regex, (string)$value) === 1){ $result[] = $record; } } return $result; } + /** + * Turn a like() pattern into a safe regex: "^" anchors the start, a trailing "$" the end, + * everything else is matched literally and case-insensitively (UTF-8 aware). + * @param string $pattern + * @return string + */ + private function likePatternToRegex($pattern){ + $pattern = (string)$pattern; + $start = $pattern !== '' && $pattern[0] === '^'; + $end = strlen($pattern) > ($start ? 1 : 0) && substr($pattern, -1) === '$'; + $body = substr($pattern, $start ? 1 : 0, strlen($pattern) - ($start ? 1 : 0) - ($end ? 1 : 0)); + $flags = preg_match('//u', $body) === 1 ? 'iu' : 'i'; + return '~' . ($start ? '^' : '') . preg_quote($body, '~') . ($end ? '$' : '') . '~' . $flags; + } + // ========================================== // AGGREGATION METHODS // ========================================== @@ -4521,6 +4192,13 @@ public function like($dbname, $field, $pattern){ * @return float|int */ public function sum($dbname, $field, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $filter){ + return $this->sumUnlocked($dbname, $field, $filter); + }); + } + + private function sumUnlocked($dbname, $field, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false) return 0; $sum = 0; @@ -4540,6 +4218,13 @@ public function sum($dbname, $field, $filter = 0){ * @return float|int */ public function avg($dbname, $field, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $filter){ + return $this->avgUnlocked($dbname, $field, $filter); + }); + } + + private function avgUnlocked($dbname, $field, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false || count($result) === 0) return 0; $sum = 0; @@ -4561,6 +4246,13 @@ public function avg($dbname, $field, $filter = 0){ * @return mixed|null */ public function min($dbname, $field, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $filter){ + return $this->minUnlocked($dbname, $field, $filter); + }); + } + + private function minUnlocked($dbname, $field, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false || count($result) === 0) return null; $min = null; @@ -4582,6 +4274,13 @@ public function min($dbname, $field, $filter = 0){ * @return mixed|null */ public function max($dbname, $field, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $filter){ + return $this->maxUnlocked($dbname, $field, $filter); + }); + } + + private function maxUnlocked($dbname, $field, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false || count($result) === 0) return null; $max = null; @@ -4606,6 +4305,13 @@ public function max($dbname, $field, $filter = 0){ * @return array|null */ public function first($dbname, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $filter){ + return $this->firstUnlocked($dbname, $filter); + }); + } + + private function firstUnlocked($dbname, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false || count($result) === 0) return null; return $result[0]; @@ -4618,6 +4324,13 @@ public function first($dbname, $filter = 0){ * @return array|null */ public function last($dbname, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $filter){ + return $this->lastUnlocked($dbname, $filter); + }); + } + + private function lastUnlocked($dbname, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false || count($result) === 0) return null; return $result[count($result) - 1]; @@ -4630,6 +4343,13 @@ public function last($dbname, $filter = 0){ * @return bool */ public function exists($dbname, $filter){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $filter){ + return $this->existsUnlocked($dbname, $filter); + }); + } + + private function existsUnlocked($dbname, $filter){ $result = $this->find($dbname, $filter); return $result !== false && count($result) > 0; } @@ -4644,15 +4364,26 @@ public function exists($dbname, $filter){ * @return array|false */ public function between($dbname, $field, $min, $max, $filter = 0){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $min, $max, $filter){ + return $this->betweenUnlocked($dbname, $field, $min, $max, $filter); + }); + } + + private function betweenUnlocked($dbname, $field, $min, $max, $filter = 0){ $result = $this->find($dbname, $filter); if($result === false) return false; + // Same comparison as query()->between(): numbers (and numeric strings) numerically, strings + // lexically; null, booleans, arrays and number-vs-text are never in range $filtered = []; foreach($result as $record){ - if(isset($record[$field])){ - $value = $record[$field]; - if($value >= $min && $value <= $max){ - $filtered[] = $record; - } + if(!array_key_exists($field, $record)){ + continue; + } + $lo = noneDBQuery::compareValues($record[$field], $min); + $hi = noneDBQuery::compareValues($record[$field], $max); + if($lo !== null && $hi !== null && $lo >= 0 && $hi <= 0){ + $filtered[] = $record; } } return $filtered; @@ -4668,252 +4399,95 @@ public function between($dbname, $field, $min, $max, $filter = 0){ * @return array|false */ public function getShardInfo($dbname){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname){ + return $this->getShardInfoUnlocked($dbname); + }); + } + private function getShardInfoUnlocked($dbname){ + if(!$this->dbExists($dbname)){ + return false; + } if(!$this->isSharded($dbname)){ - $hash = $this->hashDBName($dbname); - $dbPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - if($this->cachedFileExists($dbPath)){ - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $index = $this->readJsonlIndex($dbname); - if($index !== null){ - return array( - "sharded" => false, - "shards" => 0, - "totalRecords" => count($index['o']), - "shardSize" => $this->shardSize - ); - } + $index = $this->readJsonlIndex($dbname, null); + if($index === null){ + return false; } - return false; + return array( + "sharded" => false, + "shards" => 0, + "totalRecords" => count($index['o']), + "shardSize" => $this->shardSize + ); } $meta = $this->getCachedMeta($dbname); if($meta === null){ return false; } - return array( "sharded" => true, "shards" => count($meta['shards']), - "totalRecords" => $meta['totalRecords'], - "deletedCount" => $meta['deletedCount'], - "shardSize" => $meta['shardSize'], - "nextKey" => $meta['nextKey'] + "totalRecords" => (int)$meta['totalRecords'], + "deletedCount" => (int)($meta['deletedCount'] ?? 0), + "shardSize" => (int)$meta['shardSize'], + "nextKey" => (int)$meta['nextKey'] ); } - // ========================================== // FIELD INDEX PUBLIC API (v3.0.0) // ========================================== /** - * Create a field index for faster filter-based queries + * Create an index on a field for faster equality lookups (find / where with field => value) * @param string $dbname Database name - * @param string $field Field name to index - * @return array ['success' => bool, 'indexed' => int, 'values' => int, 'error' => string|null] + * @param string $field Field name (any name except "key") + * @return array ['success' => bool, 'indexed' => int, 'values' => int, 'error' => ?string] */ public function createFieldIndex($dbname, $field){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname, $field){ + return $this->createFieldIndexUnlocked($dbname, $field); + }); + } - if(empty($field) || $field === 'key'){ + private function createFieldIndexUnlocked($dbname, $field){ + if(!is_string($field) || $field === '' || $field === 'key'){ return ['success' => false, 'indexed' => 0, 'values' => 0, 'error' => 'Invalid field name']; } - - // Handle sharded databases - if($this->isSharded($dbname)){ - return $this->createFieldIndexSharded($dbname, $field); - } - - // Non-sharded: build index from all records - $this->checkDB($dbname); - $hash = $this->hashDBName($dbname); - $fullPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - - // Ensure JSONL format - if(!$this->ensureJsonlFormat($dbname)){ - return ['success' => false, 'indexed' => 0, 'values' => 0, 'error' => 'JSONL format required']; - } - - $jsonlIndex = $this->readJsonlIndex($dbname); - if($jsonlIndex === null){ - return ['success' => false, 'indexed' => 0, 'values' => 0, 'error' => 'Could not read index']; - } - - // Build field index - $fieldIndex = [ - 'v' => 1, - 'field' => $field, - 'created' => time(), - 'values' => [] - ]; - - $indexedCount = 0; - foreach($jsonlIndex['o'] as $key => $location){ - $record = $this->readJsonlRecord($fullPath, $location[0], $location[1]); - if($record === null) continue; - - // Use array_key_exists to include null values - if(array_key_exists($field, $record)){ - $value = $record[$field]; - // Only index scalar values - if(is_scalar($value) || is_null($value)){ - $valueKey = $this->fieldIndexValueKey($value); - if(!isset($fieldIndex['values'][$valueKey])){ - $fieldIndex['values'][$valueKey] = []; - } - $fieldIndex['values'][$valueKey][] = (int)$key; - $indexedCount++; - } + $this->flushPendingBuffers($dbname); + if(!$this->dbExists($dbname)){ + if(!$this->autoCreateDB){ + return ['success' => false, 'indexed' => 0, 'values' => 0, 'error' => 'Database does not exist']; } + $this->createDBUnlocked($dbname); } - // Write field index - if($this->writeFieldIndex($dbname, $field, $fieldIndex)){ - return [ - 'success' => true, - 'indexed' => $indexedCount, - 'values' => count($fieldIndex['values']), - 'error' => null - ]; - } - - return ['success' => false, 'indexed' => 0, 'values' => 0, 'error' => 'Failed to write index']; - } - - /** - * Create field index for sharded database - * @param string $dbname Database name - * @param string $field Field name - * @return array Result - */ - private function createFieldIndexSharded($dbname, $field){ - $meta = $this->getCachedMeta($dbname); - if($meta === null){ - return ['success' => false, 'indexed' => 0, 'values' => 0, 'error' => 'Could not read meta']; - } - - $totalIndexed = 0; - $totalValues = 0; - - // Initialize global field index metadata for shard-skip optimization - $globalMeta = [ - 'v' => 1, - 'field' => $field, - 'shardMap' => [] - ]; - - foreach($meta['shards'] as $shard){ - $shardId = $shard['id']; - $shardPath = $this->getShardPath($dbname, $shardId); - - // Build field index for this shard - $fieldIndex = [ - 'v' => 1, - 'field' => $field, - 'shardId' => $shardId, - 'created' => time(), - 'values' => [] - ]; - - // Try JSONL format first - $jsonlIndex = $this->readJsonlIndex($dbname, $shardId); - if($jsonlIndex !== null){ - // JSONL format - use batch read - foreach($jsonlIndex['o'] as $key => $location){ - $record = $this->readJsonlRecord($shardPath, $location[0], $location[1]); - if($record === null) continue; - - if(isset($record[$field])){ - $value = $record[$field]; - if(is_scalar($value) || is_null($value)){ - $valueKey = $this->fieldIndexValueKey($value); - if(!isset($fieldIndex['values'][$valueKey])){ - $fieldIndex['values'][$valueKey] = []; - } - $fieldIndex['values'][$valueKey][] = (int)$key; - $totalIndexed++; - } - } - } - } else { - // Fallback to JSON format - $shardData = $this->getShardData($dbname, $shardId); - foreach($shardData['data'] as $key => $record){ - if($record === null) continue; - - if(isset($record[$field])){ - $value = $record[$field]; - if(is_scalar($value) || is_null($value)){ - $valueKey = $this->fieldIndexValueKey($value); - if(!isset($fieldIndex['values'][$valueKey])){ - $fieldIndex['values'][$valueKey] = []; - } - $fieldIndex['values'][$valueKey][] = (int)$key; - $totalIndexed++; - } - } - } - } - - // Add this shard to global metadata for each unique value in this shard - foreach($fieldIndex['values'] as $valueKey => $keys){ - if(!isset($globalMeta['shardMap'][$valueKey])){ - $globalMeta['shardMap'][$valueKey] = []; - } - $globalMeta['shardMap'][$valueKey][] = $shardId; + $indexed = 0; + $values = []; + $segments = $this->listSegments($dbname); + foreach($segments as $segment){ + $index = $this->buildSegmentFieldIndex($dbname, $segment, $field); + $this->writeFieldIndex($dbname, $field, $index, $segment); + foreach($index['values'] as $valueKey => $keys){ + $indexed += count($keys); + $values[(string)$valueKey] = true; } - - $totalValues += count($fieldIndex['values']); - $this->writeFieldIndex($dbname, $field, $fieldIndex, $shardId); - } - - // Write global field index metadata for shard-skip optimization - $this->writeGlobalFieldIndex($dbname, $field, $globalMeta); - - return [ - 'success' => true, - 'indexed' => $totalIndexed, - 'values' => $totalValues, - 'shards' => count($meta['shards']), - 'error' => null - ]; - } - - /** - * Convert field value to index key (handles type conversion) - * @param mixed $value Field value - * @return string Index key - */ - private function fieldIndexValueKey($value){ - if($value === null){ - return '__null__'; } - if(is_bool($value)){ - return $value ? '__true__' : '__false__'; + if($this->isSharded($dbname)){ + $this->rebuildGlobalFieldIndex($dbname, $field); } - return (string)$value; - } - /** - * Convert index key back to original value type - * @param string $key Index key - * @param mixed $originalValue Sample original value for type detection - * @return mixed Converted value - */ - private function fieldIndexKeyToValue($key){ - if($key === '__null__'){ - return null; - } - if($key === '__true__'){ - return true; - } - if($key === '__false__'){ - return false; + $manifest = $this->getIndexManifest($dbname); + $manifest['fields'][] = $field; + $this->saveIndexManifest($dbname, $manifest); + + $result = ['success' => true, 'indexed' => $indexed, 'values' => count($values), 'error' => null]; + if($this->isSharded($dbname)){ + $result['shards'] = count($segments); } - return $key; + return $result; } /** @@ -4923,61 +4497,86 @@ private function fieldIndexKeyToValue($key){ * @return array ['success' => bool, 'error' => string|null] */ public function dropFieldIndex($dbname, $field){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname, $field){ + return $this->dropFieldIndexUnlocked($dbname, $field); + }); + } - // Handle sharded databases - if($this->isSharded($dbname)){ - $meta = $this->getCachedMeta($dbname); - if($meta !== null){ - // Check if index exists in first shard - if(!$this->hasFieldIndex($dbname, $field, $meta['shards'][0]['id'])){ - return ['success' => false, 'error' => 'Index does not exist']; - } - foreach($meta['shards'] as $shard){ - $this->deleteFieldIndexFile($dbname, $field, $shard['id']); + private function dropFieldIndexUnlocked($dbname, $field){ + $manifest = $this->getIndexManifest($dbname); + if(!in_array((string)$field, $manifest['fields'], true)){ + return ['success' => false, 'error' => 'Index does not exist']; + } + $segments = $this->listSegments($dbname); + foreach($segments as $segment){ + $this->deleteFieldIndexFile($dbname, $field, $segment); + // Also remove a lossy pre-3.2 file name for this field, if any + $legacy = $this->segDataPath($dbname, $segment) . '.fidx.' . preg_replace('/[^a-zA-Z0-9_]/', '_', (string)$field); + if($legacy !== $this->getFieldIndexPath($dbname, $field, $segment) && is_file($legacy)){ + $old = $this->atomicRead($legacy, null); + if(is_array($old) && ($old['field'] ?? null) === (string)$field){ + @unlink($legacy); } - // Also delete global field index - $this->deleteGlobalFieldIndex($dbname, $field); - } - } else { - // Check if index exists - if(!$this->hasFieldIndex($dbname, $field, null)){ - return ['success' => false, 'error' => 'Index does not exist']; } - $this->deleteFieldIndexFile($dbname, $field); } - + $this->deleteGlobalFieldIndex($dbname, $field); + $manifest['fields'] = array_values(array_diff($manifest['fields'], [(string)$field])); + $this->saveIndexManifest($dbname, $manifest); $this->invalidateFieldIndexCache($dbname, $field); return ['success' => true, 'error' => null]; } + /** + * Check if a field has an index + * @param string $dbname Database name + * @param string $field Field name + * @return bool + */ + public function hasFieldIndex($dbname, $field){ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field){ + return $this->dbExists($dbname) && $this->isFieldIndexed($dbname, $field); + }); + } + /** * Get list of field indexes for a database * @param string $dbname Database name * @return array ['fields' => array, 'sharded' => bool] */ public function getFieldIndexes($dbname){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname){ + return $this->getFieldIndexesUnlocked($dbname); + }); + } - if($this->isSharded($dbname)){ - // For sharded, check shard 0 - $fields = $this->getIndexedFields($dbname, 0); - return ['fields' => $fields, 'sharded' => true]; + private function getFieldIndexesUnlocked($dbname){ + if(!$this->dbExists($dbname)){ + return ['fields' => [], 'sharded' => false]; } - - $fields = $this->getIndexedFields($dbname); - return ['fields' => $fields, 'sharded' => false]; + return ['fields' => $this->getIndexedFields($dbname), 'sharded' => $this->isSharded($dbname)]; } /** - * Rebuild a field index (useful after bulk operations) + * Rebuild a field index * @param string $dbname Database name * @param string $field Field name * @return array Result from createFieldIndex */ public function rebuildFieldIndex($dbname, $field){ - $this->dropFieldIndex($dbname, $field); - return $this->createFieldIndex($dbname, $field); + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname, $field){ + return $this->rebuildFieldIndexUnlocked($dbname, $field); + }); + } + + private function rebuildFieldIndexUnlocked($dbname, $field){ + if($this->isFieldIndexed($dbname, $field)){ + $this->dropFieldIndexUnlocked($dbname, $field); + } + return $this->createFieldIndexUnlocked($dbname, $field); } /** @@ -4986,194 +4585,33 @@ public function rebuildFieldIndex($dbname, $field){ * @param string $dbname Database name * @param string $field Field name * @param mixed $value Value to match - * @param int|null $shardId Shard ID for sharded databases + * @param int|null $shardId Shard ID for sharded databases (null = all shards) * @return array|null Array of matching keys, or null if no index */ public function getKeysFromFieldIndex($dbname, $field, $value, $shardId = null){ - $index = $this->readFieldIndex($dbname, $field, $shardId); - if($index === null){ - return null; - } - - $valueKey = $this->fieldIndexValueKey($value); - if(!isset($index['values'][$valueKey])){ - return []; // Value not in index = no matches - } - - return $index['values'][$valueKey]; - } - - /** - * Update field index when a record is inserted - * @param string $dbname Database name - * @param array $record Record data - * @param int $key Record key - * @param int|null $shardId Shard ID - */ - private function updateFieldIndexOnInsert($dbname, $record, $key, $shardId = null){ - if(!$this->fieldIndexEnabled) return; - - // For sharded databases, get indexed fields from any existing shard or global index - $indexedFields = $this->getIndexedFields($dbname, $shardId); - - // If new shard has no indexes, check if other shards have indexes - if(empty($indexedFields) && $shardId !== null){ - // Check shard 0 for indexed fields (if it exists) - $indexedFields = $this->getIndexedFields($dbname, 0); - } - - foreach($indexedFields as $field){ - // Use array_key_exists to include null values - if(!array_key_exists($field, $record)) continue; - - $value = $record[$field]; - if(!is_scalar($value) && !is_null($value)) continue; - - $index = $this->readFieldIndex($dbname, $field, $shardId); - $isNewValue = true; - - if($index === null){ - // Create new field index for this shard - $index = [ - 'v' => 1, - 'field' => $field, - 'shardId' => $shardId, - 'created' => time(), - 'values' => [] - ]; - } else { - $valueKey = $this->fieldIndexValueKey($value); - $isNewValue = !isset($index['values'][$valueKey]) || empty($index['values'][$valueKey]); - } - - $valueKey = $this->fieldIndexValueKey($value); - if(!isset($index['values'][$valueKey])){ - $index['values'][$valueKey] = []; - } - - // Add key if not already present - if(!in_array($key, $index['values'][$valueKey])){ - $index['values'][$valueKey][] = $key; - $this->writeFieldIndex($dbname, $field, $index, $shardId); - - // Update global field index for sharded databases - if($shardId !== null && $isNewValue){ - $this->addShardToGlobalIndex($dbname, $field, $value, $shardId); - } - } - } + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $value, $shardId){ + return $this->getKeysFromFieldIndexUnlocked($dbname, $field, $value, $shardId); + }); } - /** - * Update field index when a record is deleted - * @param string $dbname Database name - * @param array $record Record data (before deletion) - * @param int $key Record key - * @param int|null $shardId Shard ID - */ - private function updateFieldIndexOnDelete($dbname, $record, $key, $shardId = null){ - if(!$this->fieldIndexEnabled) return; - - $indexedFields = $this->getIndexedFields($dbname, $shardId); - foreach($indexedFields as $field){ - if(!isset($record[$field])) continue; - - $value = $record[$field]; - if(!is_scalar($value) && !is_null($value)) continue; - - $index = $this->readFieldIndex($dbname, $field, $shardId); - if($index === null) continue; - - $valueKey = $this->fieldIndexValueKey($value); - if(isset($index['values'][$valueKey])){ - $index['values'][$valueKey] = array_values( - array_filter($index['values'][$valueKey], function($k) use ($key) { - return $k != $key; - }) - ); - // Remove empty value entries and update global index - $shouldUpdateGlobalIndex = false; - if(empty($index['values'][$valueKey])){ - unset($index['values'][$valueKey]); - $shouldUpdateGlobalIndex = true; - } - - // Write field index FIRST (so global index update sees the new state) - $this->writeFieldIndex($dbname, $field, $index, $shardId); - - // Then update global field index for sharded databases - if($shouldUpdateGlobalIndex && $shardId !== null){ - $this->removeShardFromGlobalIndex($dbname, $field, $value, $shardId); - } - } + private function getKeysFromFieldIndexUnlocked($dbname, $field, $value, $shardId = null){ + if(!$this->dbExists($dbname) || !$this->isFieldIndexed($dbname, $field) || !$this->isIndexableValue($value)){ + return null; } - } - - /** - * Update field index when a record is updated - * @param string $dbname Database name - * @param array $oldRecord Old record data - * @param array $newRecord New record data - * @param int $key Record key - * @param int|null $shardId Shard ID - */ - private function updateFieldIndexOnUpdate($dbname, $oldRecord, $newRecord, $key, $shardId = null){ - if(!$this->fieldIndexEnabled) return; - - $indexedFields = $this->getIndexedFields($dbname, $shardId); - foreach($indexedFields as $field){ - $oldValue = isset($oldRecord[$field]) ? $oldRecord[$field] : null; - $newValue = isset($newRecord[$field]) ? $newRecord[$field] : null; - - // Skip if value unchanged - if($oldValue === $newValue) continue; - - // Skip non-scalar values - if((!is_scalar($oldValue) && !is_null($oldValue)) || - (!is_scalar($newValue) && !is_null($newValue))) continue; - - $index = $this->readFieldIndex($dbname, $field, $shardId); - if($index === null) continue; - - // Remove from old value - $oldKey = $this->fieldIndexValueKey($oldValue); - $oldValueBecomesEmpty = false; - if(isset($index['values'][$oldKey])){ - $index['values'][$oldKey] = array_values( - array_filter($index['values'][$oldKey], function($k) use ($key) { - return $k != $key; - }) - ); - if(empty($index['values'][$oldKey])){ - unset($index['values'][$oldKey]); - $oldValueBecomesEmpty = true; - } - } - - // Add to new value - $newValueKey = $this->fieldIndexValueKey($newValue); - $newValueWasEmpty = !isset($index['values'][$newValueKey]) || empty($index['values'][$newValueKey]); - if(!isset($index['values'][$newValueKey])){ - $index['values'][$newValueKey] = []; - } - if(!in_array($key, $index['values'][$newValueKey])){ - $index['values'][$newValueKey][] = $key; + $segments = $shardId !== null ? [(int)$shardId] : $this->listSegments($dbname); + $keys = []; + foreach($segments as $segment){ + if(!in_array($segment, $this->listSegments($dbname), true)){ + continue; } - - $this->writeFieldIndex($dbname, $field, $index, $shardId); - - // Update global field index for sharded databases - if($shardId !== null){ - // Remove shard from old value's global index if shard no longer has old value - if($oldValueBecomesEmpty){ - $this->removeShardFromGlobalIndex($dbname, $field, $oldValue, $shardId); - } - // Add shard to new value's global index if this is first record with new value - if($newValueWasEmpty){ - $this->addShardToGlobalIndex($dbname, $field, $newValue, $shardId); - } + $index = $this->ensureSegmentFieldIndex($dbname, $segment, $field); + foreach($index['values'][$this->fieldIndexValueKey($value)] ?? [] as $key){ + $keys[] = (int)$key; } } + sort($keys); + return $keys; } // ========================================== @@ -5237,8 +4675,9 @@ private function validateGeoJSONPoint(array $coords): array { if (!is_array($coords) || count($coords) < 2) { return ['valid' => false, 'error' => 'Point must have at least 2 coordinates', 'type' => null]; } - if (!is_numeric($coords[0]) || !is_numeric($coords[1])) { - return ['valid' => false, 'error' => 'Point coordinates must be numeric', 'type' => null]; + if (!isset($coords[0], $coords[1]) || !is_numeric($coords[0]) || !is_numeric($coords[1]) + || is_string($coords[0]) || is_string($coords[1]) || !is_finite((float)$coords[0]) || !is_finite((float)$coords[1])) { + return ['valid' => false, 'error' => 'Point coordinates must be finite numbers', 'type' => null]; } $lon = (float)$coords[0]; $lat = (float)$coords[1]; @@ -5286,16 +4725,58 @@ private function validateGeoJSONPolygon(array $coords): array { return ['valid' => false, 'error' => "Invalid position at ring $ringIndex, index $i: " . $result['error'], 'type' => null]; } } - // Check if ring is closed (first == last) + // Check if ring is closed (first == last, compared numerically: 29 and 29.0 are the same) $first = $ring[0]; $last = $ring[count($ring) - 1]; - if ($first[0] !== $last[0] || $first[1] !== $last[1]) { + if ((float)$first[0] != (float)$last[0] || (float)$first[1] != (float)$last[1]) { return ['valid' => false, 'error' => "Ring $ringIndex must be closed (first and last positions must be identical)", 'type' => null]; } + if ($this->ringSelfIntersects($ring)) { + return ['valid' => false, 'error' => "Ring $ringIndex must not intersect itself", 'type' => null]; + } + if ($ringIndex > 0) { + foreach ($ring as $position) { + if (!$this->pointInRing((float)$position[0], (float)$position[1], $coords[0]) && !$this->pointOnRing((float)$position[0], (float)$position[1], $coords[0])) { + return ['valid' => false, 'error' => "Hole $ringIndex must lie inside the outer ring", 'type' => null]; + } + } + } } return ['valid' => true, 'error' => null, 'type' => 'Polygon']; } + /** + * True when two non-adjacent edges of a closed ring cross + */ + private function ringSelfIntersects(array $ring): bool { + $n = count($ring) - 1; // edges + for ($i = 0; $i < $n; $i++) { + for ($j = $i + 2; $j < $n; $j++) { + if ($i === 0 && $j === $n - 1) { + continue; // first and last edge share the closing vertex + } + if ($this->lineSegmentsIntersect($ring[$i], $ring[$i + 1], $ring[$j], $ring[$j + 1])) { + return true; + } + } + } + return false; + } + + /** + * True when a point lies on a ring's boundary + */ + private function pointOnRing(float $lon, float $lat, array $ring): bool { + $n = count($ring); + for ($i = 0; $i < $n - 1; $i++) { + if ($this->crossProductDirection($ring[$i], $ring[$i + 1], [$lon, $lat]) == 0 + && $this->onSegment($ring[$i], $ring[$i + 1], [$lon, $lat])) { + return true; + } + } + return false; + } + /** * Validate MultiPoint coordinates [[lon,lat], [lon,lat], ...] */ @@ -5517,10 +4998,11 @@ public function haversineDistance(float $lon1, float $lat1, float $lon2, float $ $distance = $earthRadius * $c; - // Cache result (limit cache size to prevent memory issues) - if (count($this->distanceCache) < 10000) { - $this->distanceCache[$key] = $distance; + // Cache result; when full, the older half makes room (the cache keeps working) + if (count($this->distanceCache) >= 10000) { + $this->distanceCache = array_slice($this->distanceCache, 5000, null, true); } + $this->distanceCache[$key] = $distance; return $distance; } @@ -5544,9 +5026,9 @@ private function circleToBBox(float $lon, float $lat, float $radiusMeters): arra $latDeg = $radiusMeters / 111000.0; // 1 degree lat ~ 111 km = 111000 m $lonDeg = $radiusMeters / (111000.0 * cos(deg2rad($lat))); - // Handle edge case near poles - if (abs($lat) > 89) { - $lonDeg = 360; // Cover all longitudes near poles + // Near or over a pole every longitude is in range + if ($lat + $latDeg >= 90 || $lat - $latDeg <= -90 || $lonDeg >= 180 || !is_finite($lonDeg)) { + $lonDeg = 360; } return [ @@ -5723,8 +5205,11 @@ private function polygonsIntersect(array $poly1, array $poly2): bool { * @return array [lon, lat] */ private function getGeometryCentroid(array $geometry): array { + if (($geometry['type'] ?? '') === 'Point' && isset($geometry['coordinates'][0], $geometry['coordinates'][1])) { + return [$geometry['coordinates'][0], $geometry['coordinates'][1]]; + } // Check cache first (use JSON hash as key) - $cacheKey = md5(json_encode($geometry)); + $cacheKey = md5((string)json_encode($geometry)); if (isset($this->centroidCache[$cacheKey])) { return $this->centroidCache[$cacheKey]; } @@ -5767,10 +5252,11 @@ private function getGeometryCentroid(array $geometry): array { $result = [($mbr[0] + $mbr[2]) / 2, ($mbr[1] + $mbr[3]) / 2]; } - // Cache result (limit size) - if (count($this->centroidCache) < 5000) { - $this->centroidCache[$cacheKey] = $result; + // Cache result; when full, the older half makes room + if (count($this->centroidCache) >= 5000) { + $this->centroidCache = array_slice($this->centroidCache, 2500, null, true); } + $this->centroidCache[$cacheKey] = $result; return $result; } @@ -5785,6 +5271,10 @@ public function clearCentroidCache(): void { // ========================================== // SPATIAL INDEX - R-TREE ENGINE // ========================================== + // One R-tree per segment and indexed field: {dbDir}{hash}[_sN].nonedb.sidx.{field}. Sharded databases + // also keep {hash}.nonedb.gsidx.{field} with each shard's bounding box (shard skip). Which fields are + // indexed is recorded in the index manifest. Index files of format v3 are complete; older files + // (which could miss records) are rebuilt on first use. /** * Get path to spatial index file @@ -5795,184 +5285,106 @@ public function clearCentroidCache(): void { */ private function getSpatialIndexPath(string $dbname, string $field, ?int $shardId = null): string { $hash = $this->hashDBName($dbname); - $field = preg_replace('/[^a-zA-Z0-9_]/', '', $field); - + $name = $this->encodeIndexFieldName($field); if ($shardId !== null) { - return $this->dbDir . $hash . '_s' . $shardId . '.nonedb.sidx.' . $field; + return $this->dbDir . $hash . '_s' . $shardId . '.nonedb.sidx.' . $name; } - return $this->dbDir . $hash . '.nonedb.sidx.' . $field; + return $this->dbDir . $hash . '.nonedb.sidx.' . $name; } /** * Get path to global spatial index file (shard MBR metadata) - * @param string $dbname Database name - * @param string $field Field name - * @return string File path */ private function getGlobalSpatialIndexPath(string $dbname, string $field): string { - $hash = $this->hashDBName($dbname); - $field = preg_replace('/[^a-zA-Z0-9_]/', '', $field); - return $this->dbDir . $hash . '.nonedb.gsidx.' . $field; - } - - /** - * Get spatial index cache key - */ - private function getSpatialIndexCacheKey(string $dbname, string $field, ?int $shardId = null): string { - if ($shardId !== null) { - return $dbname . ':' . $field . ':s' . $shardId; - } - return $dbname . ':' . $field; + return $this->dbDir . $this->hashDBName($dbname) . '.nonedb.gsidx.' . $this->encodeIndexFieldName($field); } /** - * Read spatial index from cache or disk - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID - * @return array|null Index data or null if not exists + * Read a segment's R-tree. Null when missing or from a pre-3.2 version. */ private function readSpatialIndex(string $dbname, string $field, ?int $shardId = null): ?array { - $cacheKey = $this->getSpatialIndexCacheKey($dbname, $field, $shardId); - - // Check cache first - if (isset($this->spatialIndexCache[$cacheKey])) { - return $this->spatialIndexCache[$cacheKey]; - } - $path = $this->getSpatialIndexPath($dbname, $field, $shardId); - if (!file_exists($path)) { - return null; + if (array_key_exists($path, $this->spatialIndexCache)) { + return $this->spatialIndexCache[$path]; } - - $content = $this->atomicRead($path); - if ($content === false) { - return null; - } - - $index = json_decode($content, true); - if ($index === null) { - return null; + $index = $this->atomicRead($path, null); + if (!is_array($index) || ($index['v'] ?? 0) < 3 || !isset($index['nodes'], $index['rootId'], $index['leafOf'])) { + $index = null; } - - // Cache it - $this->spatialIndexCache[$cacheKey] = $index; - + $this->spatialIndexCache[$path] = $index; return $index; } /** - * Write spatial index to disk and update cache - * @param string $dbname Database name - * @param string $field Field name - * @param array $index Index data - * @param int|null $shardId Shard ID - * @return bool Success + * Write a segment's R-tree (caller holds the exclusive lock) */ private function writeSpatialIndex(string $dbname, string $field, array $index, ?int $shardId = null): bool { $path = $this->getSpatialIndexPath($dbname, $field, $shardId); $index['updated'] = time(); - - $result = $this->atomicWrite($path, json_encode($index)); - - if ($result) { - $cacheKey = $this->getSpatialIndexCacheKey($dbname, $field, $shardId); - $this->spatialIndexCache[$cacheKey] = $index; + if (!$this->atomicWrite($path, $index)) { + unset($this->spatialIndexCache[$path]); + throw new \RuntimeException("noneDB: failed to write spatial index '{$field}' of database '{$dbname}'"); } - - return $result; + $this->spatialIndexCache[$path] = $index; + unset($this->dirtySpatialIndexes[$path]); + return true; } /** - * Mark spatial index as dirty (for batch write pattern) - * Updates cache immediately but defers disk write - * @param string $dbname Database name - * @param string $field Field name - * @param array $index Updated index data - * @param int|null $shardId Shard ID + * Keep an updated R-tree in memory; it is written before the write lock is released */ private function markSpatialIndexDirty(string $dbname, string $field, array $index, ?int $shardId = null): void { - $cacheKey = $this->getSpatialIndexCacheKey($dbname, $field, $shardId); - - // Update cache immediately - $this->spatialIndexCache[$cacheKey] = $index; - - // Mark as dirty for later flush - $this->dirtySpatialIndexes[$cacheKey] = [ - 'dbname' => $dbname, - 'field' => $field, - 'shardId' => $shardId - ]; + $path = $this->getSpatialIndexPath($dbname, $field, $shardId); + $this->spatialIndexCache[$path] = $index; + $this->dirtySpatialIndexes[$path] = ['dbname' => $dbname, 'field' => $field, 'shardId' => $shardId]; } /** - * Flush all dirty spatial indexes to disk - * Call this after batch operations for optimal I/O performance + * Write all dirty spatial indexes to disk (called automatically when a write lock is released) * @return int Number of indexes flushed */ public function flushSpatialIndexes(): int { $flushed = 0; - - foreach ($this->dirtySpatialIndexes as $cacheKey => $meta) { - if (isset($this->spatialIndexCache[$cacheKey])) { - $this->writeSpatialIndex( - $meta['dbname'], - $meta['field'], - $this->spatialIndexCache[$cacheKey], - $meta['shardId'] - ); + foreach ($this->dirtySpatialIndexes as $path => $info) { + if (isset($this->spatialIndexCache[$path])) { + $this->writeSpatialIndex($info['dbname'], $info['field'], $this->spatialIndexCache[$path], $info['shardId']); $flushed++; } } - $this->dirtySpatialIndexes = []; return $flushed; } /** - * Read global spatial index (shard MBR metadata) + * Read global spatial index (shard MBR metadata); null when missing or pre-3.2 */ private function readGlobalSpatialIndex(string $dbname, string $field): ?array { - $cacheKey = 'gsidx:' . $dbname . ':' . $field; - - if (isset($this->globalSpatialIndexCache[$cacheKey])) { - return $this->globalSpatialIndexCache[$cacheKey]; - } - $path = $this->getGlobalSpatialIndexPath($dbname, $field); - if (!file_exists($path)) { - return null; - } - - $content = $this->atomicRead($path); - if ($content === false) { - return null; + if (array_key_exists($path, $this->globalSpatialIndexCache)) { + return $this->globalSpatialIndexCache[$path]; } - - $index = json_decode($content, true); - if ($index === null) { - return null; + $index = $this->atomicRead($path, null); + if (!is_array($index) || ($index['v'] ?? 0) < 3 || !isset($index['shardMBRs']) || !is_array($index['shardMBRs'])) { + $index = null; } - - $this->globalSpatialIndexCache[$cacheKey] = $index; + $this->globalSpatialIndexCache[$path] = $index; return $index; } /** - * Write global spatial index + * Write global spatial index (caller holds the exclusive lock) */ private function writeGlobalSpatialIndex(string $dbname, string $field, array $index): bool { $path = $this->getGlobalSpatialIndexPath($dbname, $field); + $index['v'] = 3; + $index['field'] = $field; $index['updated'] = time(); - - $result = $this->atomicWrite($path, json_encode($index)); - - if ($result) { - $cacheKey = 'gsidx:' . $dbname . ':' . $field; - $this->globalSpatialIndexCache[$cacheKey] = $index; + if (!$this->atomicWrite($path, $index)) { + unset($this->globalSpatialIndexCache[$path]); + throw new \RuntimeException("noneDB: failed to write global spatial index '{$field}' of database '{$dbname}'"); } - - return $result; + $this->globalSpatialIndexCache[$path] = $index; + return true; } /** @@ -5982,7 +5394,7 @@ private function writeGlobalSpatialIndex(string $dbname, string $field, array $i */ private function createEmptyRTreeIndex(string $field): array { return [ - 'v' => 2, + 'v' => 3, 'field' => $field, 'type' => 'rtree', 'nodeSize' => $this->rtreeNodeSize, @@ -5991,96 +5403,71 @@ private function createEmptyRTreeIndex(string $field): array { 'rootId' => 0, 'nextNodeId' => 1, 'nodes' => [ - '0' => [ - 'isLeaf' => true, - 'mbr' => null, - 'entries' => [], - 'parent' => null - ] - ] + '0' => ['isLeaf' => true, 'mbr' => null, 'entries' => [], 'parent' => null] + ], + 'leafOf' => [] ]; } /** - * Insert an entry into R-tree - * @param string $dbname Database name - * @param string $field Field name - * @param int $key Record key - * @param array $geometry GeoJSON geometry - * @param int|null $shardId Shard ID - * @return bool Success + * Add an entry to an in-memory R-tree */ - private function rtreeInsert(string $dbname, string $field, int $key, array $geometry, ?int $shardId = null): bool { - $index = $this->readSpatialIndex($dbname, $field, $shardId); - - if ($index === null) { - $index = $this->createEmptyRTreeIndex($field); + private function rtreeAddEntry(array &$index, int $key, array $geometry): void { + if (isset($index['leafOf'][$key])) { + $this->rtreeRemoveEntry($index, $key); } - $mbr = $this->calculateMBR($geometry); - $entry = [ - 'key' => $key, - 'mbr' => $mbr, - 'type' => $geometry['type'] - ]; - - // Find leaf node to insert $leafId = $this->rtreeChooseLeaf($index, $mbr); - $leaf = &$index['nodes'][$leafId]; - - // Add entry to leaf - $leaf['entries'][] = $entry; - - // Update leaf MBR - $leaf['mbr'] = $leaf['mbr'] === null ? $mbr : $this->mbrUnion($leaf['mbr'], $mbr); - - // Check if split is needed - if (count($leaf['entries']) > $index['nodeSize']) { + $index['nodes'][$leafId]['entries'][] = ['key' => $key, 'mbr' => $mbr, 'type' => $geometry['type']]; + $index['nodes'][$leafId]['mbr'] = $index['nodes'][$leafId]['mbr'] === null + ? $mbr + : $this->mbrUnion($index['nodes'][$leafId]['mbr'], $mbr); + $index['leafOf'][$key] = $leafId; + if (count($index['nodes'][$leafId]['entries']) > $index['nodeSize']) { $this->rtreeSplitNode($index, $leafId); } - - // Propagate MBR changes up the tree $this->rtreeAdjustTree($index, $leafId); - - // Mark dirty for batch write (instead of immediate write) - $this->markSpatialIndexDirty($dbname, $field, $index, $shardId); - return true; } /** - * Choose the best leaf node for inserting an entry - * @param array $index R-tree index - * @param array $entryMBR Entry's MBR + * Choose the best leaf node for inserting an entry (least enlargement, then smallest area) * @return string Leaf node ID */ private function rtreeChooseLeaf(array &$index, array $entryMBR): string { $nodeId = (string)$index['rootId']; while (!$index['nodes'][$nodeId]['isLeaf']) { - $node = $index['nodes'][$nodeId]; $minEnlargement = PHP_FLOAT_MAX; $bestChild = null; + $bestArea = PHP_FLOAT_MAX; - foreach ($node['children'] as $childId) { - $childMBR = $index['nodes'][$childId]['mbr']; + foreach ($index['nodes'][$nodeId]['children'] as $childId) { + $child = $index['nodes'][$childId] ?? null; + if ($child === null) { + continue; + } + $childMBR = $child['mbr']; if ($childMBR === null) { - $bestChild = $childId; - break; + if ($child['isLeaf']) { + $bestChild = $childId; // an empty leaf costs nothing + break; + } + continue; // empty internal node: never descend into it } - $enlargement = $this->mbrEnlargement($childMBR, $entryMBR); - - if ($enlargement < $minEnlargement) { + $area = $this->mbrArea($childMBR); + if ($enlargement < $minEnlargement || ($enlargement == $minEnlargement && $area < $bestArea)) { $minEnlargement = $enlargement; + $bestArea = $area; $bestChild = $childId; - } elseif ($enlargement == $minEnlargement) { - // Tie-breaker: choose smaller area - if ($this->mbrArea($childMBR) < $this->mbrArea($index['nodes'][$bestChild]['mbr'] ?? [0,0,0,0])) { - $bestChild = $childId; - } } } + if ($bestChild === null) { + // Internal node without usable children: turn it into an empty leaf + $index['nodes'][$nodeId] = ['isLeaf' => true, 'mbr' => null, 'entries' => [], 'parent' => $index['nodes'][$nodeId]['parent']]; + return $nodeId; + } $nodeId = (string)$bestChild; } @@ -6099,57 +5486,37 @@ private function rtreeSplitNode(array &$index, string $nodeId): void { $n = count($entries); if ($n <= $index['nodeSize']) { - return; // No split needed + return; } - // Linear Split: O(n) seed selection using maximum separation - // Find extremes along each dimension in single pass + $mbrOf = function ($entry) use (&$index, $isLeaf) { + return $isLeaf ? $entry['mbr'] : ($index['nodes'][$entry]['mbr'] ?? null); + }; + + // Linear split: seeds with the greatest normalized separation $minLon = $maxLon = $minLat = $maxLat = null; $minLonIdx = $maxLonIdx = $minLatIdx = $maxLatIdx = 0; - $globalMinLon = PHP_FLOAT_MAX; - $globalMaxLon = -PHP_FLOAT_MAX; - $globalMinLat = PHP_FLOAT_MAX; - $globalMaxLat = -PHP_FLOAT_MAX; + $globalMinLon = $globalMinLat = PHP_FLOAT_MAX; + $globalMaxLon = $globalMaxLat = -PHP_FLOAT_MAX; for ($i = 0; $i < $n; $i++) { - $mbr = $isLeaf ? $entries[$i]['mbr'] : $index['nodes'][$entries[$i]]['mbr']; + $mbr = $mbrOf($entries[$i]); if ($mbr === null) continue; - - // Track global bounds for normalization $globalMinLon = min($globalMinLon, $mbr[0]); $globalMaxLon = max($globalMaxLon, $mbr[2]); $globalMinLat = min($globalMinLat, $mbr[1]); $globalMaxLat = max($globalMaxLat, $mbr[3]); - - // Find entry with highest low-side (minLon, minLat) - if ($minLon === null || $mbr[0] > $minLon) { - $minLon = $mbr[0]; - $minLonIdx = $i; - } - if ($minLat === null || $mbr[1] > $minLat) { - $minLat = $mbr[1]; - $minLatIdx = $i; - } - - // Find entry with lowest high-side (maxLon, maxLat) - if ($maxLon === null || $mbr[2] < $maxLon) { - $maxLon = $mbr[2]; - $maxLonIdx = $i; - } - if ($maxLat === null || $mbr[3] < $maxLat) { - $maxLat = $mbr[3]; - $maxLatIdx = $i; - } + if ($minLon === null || $mbr[0] > $minLon) { $minLon = $mbr[0]; $minLonIdx = $i; } + if ($minLat === null || $mbr[1] > $minLat) { $minLat = $mbr[1]; $minLatIdx = $i; } + if ($maxLon === null || $mbr[2] < $maxLon) { $maxLon = $mbr[2]; $maxLonIdx = $i; } + if ($maxLat === null || $mbr[3] < $maxLat) { $maxLat = $mbr[3]; $maxLatIdx = $i; } } - // Calculate normalized separation for each dimension $lonWidth = $globalMaxLon - $globalMinLon; $latWidth = $globalMaxLat - $globalMinLat; + $lonSep = ($lonWidth > 0 && $minLon !== null) ? ($minLon - $maxLon) / $lonWidth : 0; + $latSep = ($latWidth > 0 && $minLat !== null) ? ($minLat - $maxLat) / $latWidth : 0; - $lonSep = ($lonWidth > 0) ? ($minLon - $maxLon) / $lonWidth : 0; - $latSep = ($latWidth > 0) ? ($minLat - $maxLat) / $latWidth : 0; - - // Choose seeds from dimension with greatest separation if ($lonSep >= $latSep) { $seed1 = $minLonIdx; $seed2 = $maxLonIdx; @@ -6157,39 +5524,26 @@ private function rtreeSplitNode(array &$index, string $nodeId): void { $seed1 = $minLatIdx; $seed2 = $maxLatIdx; } - - // Ensure seeds are different if ($seed1 === $seed2) { $seed2 = ($seed1 + 1) % $n; } - // Initialize two groups $group1 = [$entries[$seed1]]; $group2 = [$entries[$seed2]]; - $mbr1 = $isLeaf ? $entries[$seed1]['mbr'] : $index['nodes'][$entries[$seed1]]['mbr']; - $mbr2 = $isLeaf ? $entries[$seed2]['mbr'] : $index['nodes'][$entries[$seed2]]['mbr']; - - // Distribute remaining entries - $minEntries = max(1, (int)($index['nodeSize'] * 0.4)); // Minimum 40% fill + $mbr1 = $mbrOf($entries[$seed1]); + $mbr2 = $mbrOf($entries[$seed2]); + $minEntries = max(1, (int)($index['nodeSize'] * 0.4)); - for ($i = 0; $i < count($entries); $i++) { + for ($i = 0; $i < $n; $i++) { if ($i == $seed1 || $i == $seed2) continue; - $entry = $entries[$i]; - $entryMBR = $isLeaf ? $entry['mbr'] : $index['nodes'][$entry]['mbr']; + $entryMBR = $mbrOf($entry); + $remaining = $n - $i - 1; if ($entryMBR === null) { - // Assign to smaller group - if (count($group1) <= count($group2)) { - $group1[] = $entry; - } else { - $group2[] = $entry; - } + if (count($group1) <= count($group2)) { $group1[] = $entry; } else { $group2[] = $entry; } continue; } - - // Check if one group needs all remaining entries - $remaining = count($entries) - $i - 1; if (count($group1) + $remaining <= $minEntries) { $group1[] = $entry; $mbr1 = $mbr1 === null ? $entryMBR : $this->mbrUnion($mbr1, $entryMBR); @@ -6200,13 +5554,9 @@ private function rtreeSplitNode(array &$index, string $nodeId): void { $mbr2 = $mbr2 === null ? $entryMBR : $this->mbrUnion($mbr2, $entryMBR); continue; } - - // Choose group with least enlargement $enlargement1 = $mbr1 === null ? 0 : $this->mbrEnlargement($mbr1, $entryMBR); $enlargement2 = $mbr2 === null ? 0 : $this->mbrEnlargement($mbr2, $entryMBR); - - if ($enlargement1 < $enlargement2 || - ($enlargement1 == $enlargement2 && count($group1) <= count($group2))) { + if ($enlargement1 < $enlargement2 || ($enlargement1 == $enlargement2 && count($group1) <= count($group2))) { $group1[] = $entry; $mbr1 = $mbr1 === null ? $entryMBR : $this->mbrUnion($mbr1, $entryMBR); } else { @@ -6215,49 +5565,28 @@ private function rtreeSplitNode(array &$index, string $nodeId): void { } } - // Create new node for group2 $newNodeId = (string)$index['nextNodeId']++; - - // Get current parent (will be used for both nodes after split) $currentParent = $index['nodes'][$nodeId]['parent'] ?? null; if ($isLeaf) { - // Update original node with group1 $index['nodes'][$nodeId]['entries'] = $group1; $index['nodes'][$nodeId]['mbr'] = $mbr1; - - // Create new leaf node with group2 (same parent as original) - $index['nodes'][$newNodeId] = [ - 'isLeaf' => true, - 'mbr' => $mbr2, - 'entries' => $group2, - 'parent' => $currentParent - ]; + $index['nodes'][$newNodeId] = ['isLeaf' => true, 'mbr' => $mbr2, 'entries' => $group2, 'parent' => $currentParent]; + foreach ($group2 as $entry) { + $index['leafOf'][$entry['key']] = $newNodeId; + } } else { - // Update original node with group1 $index['nodes'][$nodeId]['children'] = $group1; $index['nodes'][$nodeId]['mbr'] = $mbr1; - - // Create new internal node with group2 (same parent as original) - $index['nodes'][$newNodeId] = [ - 'isLeaf' => false, - 'mbr' => $mbr2, - 'children' => $group2, - 'parent' => $currentParent - ]; - - // Update parent pointers of children in group2 + $index['nodes'][$newNodeId] = ['isLeaf' => false, 'mbr' => $mbr2, 'children' => $group2, 'parent' => $currentParent]; foreach ($group2 as $childId) { $index['nodes'][$childId]['parent'] = $newNodeId; } - - // Ensure children in group1 point to original node foreach ($group1 as $childId) { $index['nodes'][$childId]['parent'] = $nodeId; } } - // Handle root split if ($nodeId == (string)$index['rootId']) { $newRootId = (string)$index['nextNodeId']++; $index['nodes'][$newRootId] = [ @@ -6267,219 +5596,304 @@ private function rtreeSplitNode(array &$index, string $nodeId): void { 'parent' => null ]; $index['rootId'] = (int)$newRootId; - - // Update parent pointers of the two children $index['nodes'][$nodeId]['parent'] = $newRootId; $index['nodes'][$newNodeId]['parent'] = $newRootId; - } else { - // Use parent pointer directly (O(1) instead of O(n) lookup) - $parentId = $currentParent; - if ($parentId !== null) { - $index['nodes'][$parentId]['children'][] = $newNodeId; - // Check if parent needs split - if (count($index['nodes'][$parentId]['children']) > $index['nodeSize']) { - $this->rtreeSplitNode($index, $parentId); - } + } elseif ($currentParent !== null) { + $index['nodes'][$currentParent]['children'][] = $newNodeId; + if (count($index['nodes'][$currentParent]['children']) > $index['nodeSize']) { + $this->rtreeSplitNode($index, (string)$currentParent); } } } /** - * Find parent node of a given node - * O(1) lookup using parent pointer + * Recompute one node's MBR from its entries/children */ - private function rtreeFindParent(array &$index, string $nodeId): ?string { - return $index['nodes'][$nodeId]['parent'] ?? null; + private function rtreeRecomputeMBR(array &$index, string $nodeId): void { + $node = $index['nodes'][$nodeId]; + $mbr = null; + if ($node['isLeaf']) { + foreach ($node['entries'] as $entry) { + $mbr = $mbr === null ? $entry['mbr'] : $this->mbrUnion($mbr, $entry['mbr']); + } + } else { + foreach ($node['children'] as $childId) { + $childMBR = $index['nodes'][$childId]['mbr'] ?? null; + if ($childMBR !== null) { + $mbr = $mbr === null ? $childMBR : $this->mbrUnion($mbr, $childMBR); + } + } + } + $index['nodes'][$nodeId]['mbr'] = $mbr; } /** - * Adjust MBRs up the tree after insert/delete - * Uses parent pointers for O(1) traversal + * Adjust MBRs of all ancestors of a node */ private function rtreeAdjustTree(array &$index, string $nodeId): void { $parentId = $index['nodes'][$nodeId]['parent'] ?? null; - - while ($parentId !== null) { - $parent = &$index['nodes'][$parentId]; - $newMBR = null; - - foreach ($parent['children'] as $childId) { - $childMBR = $index['nodes'][$childId]['mbr']; - if ($childMBR !== null) { - $newMBR = $newMBR === null ? $childMBR : $this->mbrUnion($newMBR, $childMBR); - } - } - - $parent['mbr'] = $newMBR; - $parentId = $parent['parent'] ?? null; + while ($parentId !== null && isset($index['nodes'][$parentId])) { + $this->rtreeRecomputeMBR($index, (string)$parentId); + $parentId = $index['nodes'][$parentId]['parent'] ?? null; } } /** - * Delete an entry from R-tree - * @param string $dbname Database name - * @param string $field Field name - * @param int $key Record key to delete - * @param int|null $shardId Shard ID - * @return bool Success + * Remove an entry from an in-memory R-tree, removing emptied nodes bottom-up + * @return bool True if the key was indexed */ - private function rtreeDelete(string $dbname, string $field, int $key, ?int $shardId = null): bool { - $index = $this->readSpatialIndex($dbname, $field, $shardId); - - if ($index === null) { + private function rtreeRemoveEntry(array &$index, int $key): bool { + $leafId = $index['leafOf'][$key] ?? null; + if ($leafId === null || !isset($index['nodes'][$leafId])) { return false; } - - // Find and remove the entry - $found = false; - foreach ($index['nodes'] as $nodeId => &$node) { - if (!$node['isLeaf']) continue; - - foreach ($node['entries'] as $i => $entry) { - if ($entry['key'] === $key) { - array_splice($node['entries'], $i, 1); - $found = true; - - // Recalculate node MBR - if (empty($node['entries'])) { - $node['mbr'] = null; - } else { - $node['mbr'] = null; - foreach ($node['entries'] as $e) { - $node['mbr'] = $node['mbr'] === null ? $e['mbr'] : $this->mbrUnion($node['mbr'], $e['mbr']); - } - } - - // Adjust tree - $this->rtreeAdjustTree($index, $nodeId); - break 2; - } + $leafId = (string)$leafId; + $entries = $index['nodes'][$leafId]['entries']; + foreach ($entries as $i => $entry) { + if ($entry['key'] === $key) { + array_splice($entries, $i, 1); + break; } } + $index['nodes'][$leafId]['entries'] = $entries; + unset($index['leafOf'][$key]); - if (!$found) { - return false; + // Condense: drop empty non-root nodes, walking up while parents become empty too + $nodeId = $leafId; + $rootId = (string)$index['rootId']; + while ($nodeId !== $rootId) { + $node = $index['nodes'][$nodeId]; + $empty = $node['isLeaf'] ? empty($node['entries']) : empty($node['children']); + if (!$empty) { + break; + } + $parentId = (string)$node['parent']; + $index['nodes'][$parentId]['children'] = array_values(array_filter( + $index['nodes'][$parentId]['children'], + function ($c) use ($nodeId) { return (string)$c !== $nodeId; } + )); + unset($index['nodes'][$nodeId]); + $nodeId = $parentId; } - - // Condense tree (remove empty nodes) + $this->rtreeRecomputeMBR($index, $nodeId); + $this->rtreeAdjustTree($index, $nodeId); $this->rtreeCondenseTree($index); - - // Mark dirty for batch write (instead of immediate write) - $this->markSpatialIndexDirty($dbname, $field, $index, $shardId); return true; } /** - * Condense tree by removing empty nodes - * Uses parent pointers for O(1) traversal + * Collapse the root: an internal root with no children becomes an empty leaf, one with a single + * child is replaced by that child */ private function rtreeCondenseTree(array &$index): void { - // Remove empty leaf nodes (except root) - $nodesToRemove = []; - - foreach ($index['nodes'] as $nodeId => $node) { - if ($nodeId == (string)$index['rootId']) continue; + while (true) { + $rootId = (string)$index['rootId']; + $root = $index['nodes'][$rootId]; + if ($root['isLeaf']) { + return; + } + $children = $root['children'] ?? []; + if (empty($children)) { + $index['nodes'][$rootId] = ['isLeaf' => true, 'mbr' => null, 'entries' => [], 'parent' => null]; + return; + } + if (count($children) > 1) { + return; + } + $newRootId = (string)$children[0]; + unset($index['nodes'][$rootId]); + $index['rootId'] = (int)$newRootId; + $index['nodes'][$newRootId]['parent'] = null; + } + } - if ($node['isLeaf'] && empty($node['entries'])) { - $nodesToRemove[] = $nodeId; - } elseif (!$node['isLeaf'] && empty($node['children'])) { - $nodesToRemove[] = $nodeId; + /** + * Search R-tree for entries overlapping any of the given MBRs + * @param array $searchMBR One MBR [minLon, minLat, maxLon, maxLat] or a list of MBRs + * @return array Matching keys + */ + private function rtreeSearchMBR(string $dbname, string $field, array $searchMBR, ?int $shardId = null): array { + $index = $this->ensureSegmentSpatialIndex($dbname, $field, $shardId); + $boxes = is_array($searchMBR[0] ?? null) ? $searchMBR : [$searchMBR]; + $results = []; + foreach ($boxes as $box) { + $found = []; + $this->rtreeSearchNode($index, (string)$index['rootId'], $box, $found); + foreach ($found as $key) { + $results[$key] = $key; } } + return array_values($results); + } - foreach ($nodesToRemove as $nodeId) { - // Use parent pointer directly (O(1)) - $parentId = $index['nodes'][$nodeId]['parent'] ?? null; - if ($parentId !== null) { - $children = &$index['nodes'][$parentId]['children']; - $children = array_values(array_filter($children, fn($c) => $c != $nodeId)); + /** + * Recursive search helper + */ + private function rtreeSearchNode(array &$index, string $nodeId, array $searchMBR, array &$results): void { + $node = $index['nodes'][$nodeId] ?? null; + if ($node === null || $node['mbr'] === null || !$this->mbrOverlaps($node['mbr'], $searchMBR)) { + return; + } + if ($node['isLeaf']) { + foreach ($node['entries'] as $entry) { + if ($this->mbrOverlaps($entry['mbr'], $searchMBR)) { + $results[] = $entry['key']; + } + } + } else { + foreach ($node['children'] as $childId) { + $this->rtreeSearchNode($index, (string)$childId, $searchMBR, $results); } - - unset($index['nodes'][$nodeId]); } + } - // If root has only one child, make that child the new root - $root = $index['nodes'][(string)$index['rootId']]; - if (!$root['isLeaf'] && count($root['children'] ?? []) === 1) { - $newRootId = $root['children'][0]; - unset($index['nodes'][(string)$index['rootId']]); - $index['rootId'] = (int)$newRootId; - // Set new root's parent to null - $index['nodes'][$newRootId]['parent'] = null; + /** + * Build a segment's R-tree from its data + */ + private function buildSegmentSpatialIndex(string $dbname, string $field, ?int $shardId, array $options = []): array { + $index = $this->createEmptyRTreeIndex($field); + if (isset($options['nodeSize']) && (int)$options['nodeSize'] >= 4) { + $index['nodeSize'] = (int)$options['nodeSize']; + } + foreach ($this->readSegmentRecords($dbname, $shardId) as $key => $record) { + $geometry = $record[$field] ?? null; + if (is_array($geometry) && $this->validateGeoJSON($geometry)['valid']) { + $this->rtreeAddEntry($index, (int)$key, $geometry); + } } + return $index; } /** - * Search R-tree for entries overlapping with given MBR - * @param string $dbname Database name - * @param string $field Field name - * @param array $searchMBR Search MBR [minLon, minLat, maxLon, maxLat] - * @param int|null $shardId Shard ID - * @return array Array of matching entry keys + * A segment's R-tree, (re)built when missing or from an older version (exclusive lock) */ - private function rtreeSearchMBR(string $dbname, string $field, array $searchMBR, ?int $shardId = null): array { + private function ensureSegmentSpatialIndex(string $dbname, string $field, ?int $shardId): array { $index = $this->readSpatialIndex($dbname, $field, $shardId); + if ($index !== null) { + return $index; + } + return $this->withLock($dbname, true, function () use ($dbname, $field, $shardId) { + unset($this->spatialIndexCache[$this->getSpatialIndexPath($dbname, $field, $shardId)]); + $index = $this->readSpatialIndex($dbname, $field, $shardId); + if ($index === null) { + $nodeSize = $this->getIndexManifest($dbname)['spatialOptions'][$field]['nodeSize'] ?? null; + $index = $this->buildSegmentSpatialIndex($dbname, $field, $shardId, $nodeSize ? ['nodeSize' => $nodeSize] : []); + $this->writeSpatialIndex($dbname, $field, $index, $shardId); + if ($shardId !== null) { + $this->updateGlobalSpatialShard($dbname, $field, $shardId, $index); + } + } + return $index; + }); + } - if ($index === null) { - return []; + /** + * Record a shard's bounding box in the global spatial index + */ + private function updateGlobalSpatialShard(string $dbname, string $field, int $shardId, array $index): void { + $global = $this->readGlobalSpatialIndex($dbname, $field) ?? ['v' => 3, 'field' => $field, 'shardMBRs' => []]; + $mbr = $index['nodes'][(string)$index['rootId']]['mbr'] ?? null; + $current = $global['shardMBRs'][(string)$shardId] ?? null; + if ($mbr === $current) { + return; + } + if ($mbr === null) { + unset($global['shardMBRs'][(string)$shardId]); + } else { + $global['shardMBRs'][(string)$shardId] = $mbr; + } + $this->writeGlobalSpatialIndex($dbname, $field, $global); + } + + /** + * Rebuild a spatial index for the whole database (all segments + global shard boxes) + */ + private function rebuildSpatialIndexForDb($dbname, $field){ + $manifest = $this->getIndexManifest($dbname); + $nodeSize = $manifest['spatialOptions'][$field]['nodeSize'] ?? null; + $options = $nodeSize ? ['nodeSize' => $nodeSize] : []; + @unlink($this->getSpatialIndexPath($dbname, $field, null)); + unset($this->spatialIndexCache[$this->getSpatialIndexPath($dbname, $field, null)]); + $indexed = 0; + $sharded = $this->isSharded($dbname); + if ($sharded) { + @unlink($this->getGlobalSpatialIndexPath($dbname, $field)); + unset($this->globalSpatialIndexCache[$this->getGlobalSpatialIndexPath($dbname, $field)]); + } + foreach ($this->listSegments($dbname) as $segment) { + $index = $this->buildSegmentSpatialIndex($dbname, $field, $segment, $options); + $this->writeSpatialIndex($dbname, $field, $index, $segment); + $indexed += count($index['leafOf']); + if ($sharded) { + $this->updateGlobalSpatialShard($dbname, $field, $segment, $index); + } + } + if ($sharded && $this->readGlobalSpatialIndex($dbname, $field) === null) { + $this->writeGlobalSpatialIndex($dbname, $field, ['v' => 3, 'field' => $field, 'shardMBRs' => []]); } - - $results = []; - $this->rtreeSearchNode($index, (string)$index['rootId'], $searchMBR, $results); - - return $results; + return $indexed; } /** - * Recursive search helper + * Delete the spatial index files of one shard */ - private function rtreeSearchNode(array &$index, string $nodeId, array $searchMBR, array &$results): void { - $node = $index['nodes'][$nodeId]; - - // Skip if node MBR doesn't overlap search MBR - if ($node['mbr'] !== null && !$this->mbrOverlaps($node['mbr'], $searchMBR)) { - return; + private function removeSpatialShardFiles($dbname, $shardId){ + $prefix = $this->dbDir . $this->hashDBName($dbname) . '_s' . (int)$shardId . '.nonedb.sidx.'; + foreach (glob($prefix . '*') ?: [] as $file) { + @unlink($file); + unset($this->spatialIndexCache[$file], $this->dirtySpatialIndexes[$file]); } - - if ($node['isLeaf']) { - // Check each entry - foreach ($node['entries'] as $entry) { - if ($this->mbrOverlaps($entry['mbr'], $searchMBR)) { - $results[] = $entry['key']; - } - } - } else { - // Search children - foreach ($node['children'] as $childId) { - $this->rtreeSearchNode($index, (string)$childId, $searchMBR, $results); + foreach ($this->getIndexManifest($dbname)['spatial'] as $field) { + $global = $this->readGlobalSpatialIndex($dbname, $field); + if ($global !== null && isset($global['shardMBRs'][(string)$shardId])) { + unset($global['shardMBRs'][(string)$shardId]); + $this->writeGlobalSpatialIndex($dbname, $field, $global); } } } /** - * Get all entries from spatial index - * @param string $dbname Database name - * @param string $field Field name - * @param int|null $shardId Shard ID - * @return array Array of all entries + * Keep spatial indexes in sync after records of a segment changed (called by the storage core) + * @param array $removed [key => old record] + * @param array $added [key => new record] */ - private function rtreeGetAllEntries(string $dbname, string $field, ?int $shardId = null): array { - $index = $this->readSpatialIndex($dbname, $field, $shardId); - - if ($index === null) { - return []; + private function applySpatialIndexChanges($dbname, $shardId, array $removed, array $added){ + // Maintained even when spatialIndexEnabled is off; that flag only affects creating/using indexes + if (empty($removed) && empty($added)) { + return; } - - $entries = []; - foreach ($index['nodes'] as $node) { - if ($node['isLeaf'] && !empty($node['entries'])) { - foreach ($node['entries'] as $entry) { - $entries[] = $entry; + foreach ($this->getIndexManifest($dbname)['spatial'] as $field) { + $existing = $this->readSpatialIndex($dbname, $field, $shardId); + if ($existing === null) { + // Missing (new shard) or pre-3.2: built from the data, which already holds the change + $this->ensureSegmentSpatialIndex($dbname, $field, $shardId); + continue; + } + $index = $existing; + $changed = false; + foreach ($removed as $key => $old) { + if (isset($added[$key]) && ($old[$field] ?? null) === ($added[$key][$field] ?? null)) { + continue; // geometry unchanged + } + $changed = $this->rtreeRemoveEntry($index, (int)$key) || $changed; + } + foreach ($added as $key => $new) { + if (isset($removed[$key]) && ($removed[$key][$field] ?? null) === ($new[$field] ?? null)) { + continue; + } + $geometry = $new[$field] ?? null; + if (is_array($geometry) && $this->validateGeoJSON($geometry)['valid']) { + $this->rtreeAddEntry($index, (int)$key, $geometry); + $changed = true; + } + } + if ($changed) { + $this->markSpatialIndexDirty($dbname, $field, $index, $shardId); + if ($shardId !== null) { + $this->updateGlobalSpatialShard($dbname, $field, $shardId, $index); } } } - - return $entries; } /** @@ -6489,47 +5903,18 @@ private function rtreeGetAllEntries(string $dbname, string $field, ?int $shardId * @param int|null $shardId Shard ID * @return bool */ - public function hasSpatialIndex(string $dbname, string $field, ?int $shardId = null): bool { - $dbname = $this->sanitizeDbName($dbname); - $path = $this->getSpatialIndexPath($dbname, $field, $shardId); - return file_exists($path); + public function hasSpatialIndex(string $dbname, string $field, ?int $shardId = null): bool{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $shardId){ + return $this->hasSpatialIndexUnlocked($dbname, $field, $shardId); + }); } - /** - * Get list of spatial indexed fields for a database - * @param string $dbname Database name - * @param int|null $shardId Shard ID - * @return array Field names - */ - private function getSpatialIndexedFields(string $dbname, ?int $shardId = null): array { - $hash = $this->hashDBName($dbname); - - if ($shardId !== null) { - $pattern = $this->dbDir . $hash . '_s' . $shardId . '.nonedb.sidx.*'; - } else { - $pattern = $this->dbDir . $hash . '.nonedb.sidx.*'; - } - - $files = glob($pattern); - $fields = []; - - foreach ($files as $file) { - // Extract field name from filename - $parts = explode('.sidx.', basename($file)); - if (count($parts) >= 2) { - $fields[] = $parts[1]; - } + private function hasSpatialIndexUnlocked(string $dbname, string $field, ?int $shardId = null): bool{ + if (!$this->dbExists($dbname) || !in_array($field, $this->getIndexManifest($dbname)['spatial'], true)) { + return false; } - - return $fields; - } - - /** - * Invalidate spatial index cache - */ - private function invalidateSpatialIndexCache(string $dbname, string $field, ?int $shardId = null): void { - $cacheKey = $this->getSpatialIndexCacheKey($dbname, $field, $shardId); - unset($this->spatialIndexCache[$cacheKey]); + return $shardId === null || in_array($shardId, $this->listSegments($dbname), true); } // ========================================== @@ -6540,205 +5925,75 @@ private function invalidateSpatialIndexCache(string $dbname, string $field, ?int * Create a spatial index on a GeoJSON field * @param string $dbname Database name * @param string $field Field name containing GeoJSON - * @param array $options Options ['nodeSize' => 16] + * @param array $options Options ['nodeSize' => 32] * @return array ['success' => bool, 'indexed' => int, 'error' => ?string] */ - public function createSpatialIndex(string $dbname, string $field, array $options = []): array { - $dbname = $this->sanitizeDbName($dbname); + public function createSpatialIndex(string $dbname, string $field, array $options = []): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname, $field, $options){ + return $this->createSpatialIndexUnlocked($dbname, $field, $options); + }); + } + private function createSpatialIndexUnlocked(string $dbname, string $field, array $options = []): array{ if (!$this->spatialIndexEnabled) { return ['success' => false, 'indexed' => 0, 'error' => 'Spatial indexing is disabled']; } - - // Check if index already exists - if ($this->hasSpatialIndex($dbname, $field)) { - return ['success' => false, 'indexed' => 0, 'error' => "Spatial index already exists for field '$field'"]; - } - - // Handle sharded databases - if ($this->isSharded($dbname)) { - return $this->createSpatialIndexSharded($dbname, $field, $options); + if ($field === '' || $field === 'key') { + return ['success' => false, 'indexed' => 0, 'error' => 'Invalid field name']; } - - // Non-sharded: flush buffer and read all records - if ($this->bufferEnabled) { - $bufferPath = $this->getBufferPath($dbname); - if ($this->hasBuffer($bufferPath)) { - $this->flushBufferToMain($dbname); + $this->flushPendingBuffers($dbname); + if (!$this->dbExists($dbname)) { + if (!$this->autoCreateDB) { + return ['success' => false, 'indexed' => 0, 'error' => 'Database does not exist']; } + $this->createDBUnlocked($dbname); } - $records = $this->find($dbname); - if (!is_array($records)) { - return ['success' => false, 'indexed' => 0, 'error' => 'Failed to read records']; - } - - // Create new index - $index = $this->createEmptyRTreeIndex($field); - if (isset($options['nodeSize'])) { - $index['nodeSize'] = (int)$options['nodeSize']; + $manifest = $this->getIndexManifest($dbname); + if (in_array($field, $manifest['spatial'], true)) { + return ['success' => false, 'indexed' => 0, 'error' => "Spatial index already exists for field '$field'"]; } - $indexed = 0; - foreach ($records as $record) { - if (!isset($record[$field]) || !is_array($record[$field])) { - continue; - } - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) { - continue; - } - - $key = $record['key']; - $mbr = $this->calculateMBR($geometry); - - $entry = [ - 'key' => $key, - 'mbr' => $mbr, - 'type' => $geometry['type'] - ]; - - // Find leaf and insert - $leafId = $this->rtreeChooseLeaf($index, $mbr); - $index['nodes'][$leafId]['entries'][] = $entry; - $index['nodes'][$leafId]['mbr'] = $index['nodes'][$leafId]['mbr'] === null - ? $mbr - : $this->mbrUnion($index['nodes'][$leafId]['mbr'], $mbr); - - // Split if needed - if (count($index['nodes'][$leafId]['entries']) > $index['nodeSize']) { - $this->rtreeSplitNode($index, $leafId); - } - - $this->rtreeAdjustTree($index, $leafId); - $indexed++; + $manifest['spatial'][] = $field; + if (isset($options['nodeSize']) && (int)$options['nodeSize'] >= 4) { + $manifest['spatialOptions'][$field] = ['nodeSize' => (int)$options['nodeSize']]; } + $this->saveIndexManifest($dbname, $manifest); - $this->writeSpatialIndex($dbname, $field, $index, null); - + $indexed = $this->rebuildSpatialIndexForDb($dbname, $field); return ['success' => true, 'indexed' => $indexed, 'error' => null]; } - /** - * Create spatial index for sharded database - */ - private function createSpatialIndexSharded(string $dbname, string $field, array $options = []): array { - $meta = $this->getCachedMeta($dbname); - if (!isset($meta['shards'])) { - return ['success' => false, 'indexed' => 0, 'error' => 'No shards found']; - } - - $totalIndexed = 0; - $globalIndex = [ - 'v' => 1, - 'field' => $field, - 'shardMBRs' => [] - ]; - - foreach ($meta['shards'] as $shard) { - $shardId = $shard['id']; - - // Flush shard buffer if buffering enabled - if ($this->bufferEnabled) { - $this->flushShardBuffer($dbname, $shardId); - } - - // Create index for this shard - $index = $this->createEmptyRTreeIndex($field); - if (isset($options['nodeSize'])) { - $index['nodeSize'] = (int)$options['nodeSize']; - } - - $records = $this->findShard($dbname, $shardId, []); - $shardMBR = null; - - foreach ($records as $record) { - if (!isset($record[$field]) || !is_array($record[$field])) { - continue; - } - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) { - continue; - } - - $key = $record['key']; - $mbr = $this->calculateMBR($geometry); - - $entry = [ - 'key' => $key, - 'mbr' => $mbr, - 'type' => $geometry['type'] - ]; - - $leafId = $this->rtreeChooseLeaf($index, $mbr); - $index['nodes'][$leafId]['entries'][] = $entry; - $index['nodes'][$leafId]['mbr'] = $index['nodes'][$leafId]['mbr'] === null - ? $mbr - : $this->mbrUnion($index['nodes'][$leafId]['mbr'], $mbr); - - if (count($index['nodes'][$leafId]['entries']) > $index['nodeSize']) { - $this->rtreeSplitNode($index, $leafId); - } - - $this->rtreeAdjustTree($index, $leafId); - - // Track shard MBR - $shardMBR = $shardMBR === null ? $mbr : $this->mbrUnion($shardMBR, $mbr); - $totalIndexed++; - } - - $this->writeSpatialIndex($dbname, $field, $index, $shardId); - - // Add to global index - if ($shardMBR !== null) { - $globalIndex['shardMBRs'][(string)$shardId] = $shardMBR; - } - } - - // Write global spatial index - $this->writeGlobalSpatialIndex($dbname, $field, $globalIndex); - - return ['success' => true, 'indexed' => $totalIndexed, 'error' => null]; - } - /** * Drop a spatial index * @param string $dbname Database name * @param string $field Field name * @return array ['success' => bool, 'error' => ?string] */ - public function dropSpatialIndex(string $dbname, string $field): array { - $dbname = $this->sanitizeDbName($dbname); + public function dropSpatialIndex(string $dbname, string $field): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname, $field){ + return $this->dropSpatialIndexUnlocked($dbname, $field); + }); + } - if ($this->isSharded($dbname)) { - $meta = $this->getCachedMeta($dbname); - if (isset($meta['shards'])) { - foreach ($meta['shards'] as $shard) { - $path = $this->getSpatialIndexPath($dbname, $field, $shard['id']); - if (file_exists($path)) { - @unlink($path); - } - $this->invalidateSpatialIndexCache($dbname, $field, $shard['id']); - } - } - // Drop global index - $globalPath = $this->getGlobalSpatialIndexPath($dbname, $field); - if (file_exists($globalPath)) { - @unlink($globalPath); - } - unset($this->globalSpatialIndexCache['gsidx:' . $dbname . ':' . $field]); - } else { - $path = $this->getSpatialIndexPath($dbname, $field, null); - if (file_exists($path)) { - @unlink($path); - } - $this->invalidateSpatialIndexCache($dbname, $field, null); + private function dropSpatialIndexUnlocked(string $dbname, string $field): array{ + foreach (array_merge([null], $this->isSharded($dbname) ? $this->listSegments($dbname) : []) as $segment) { + $path = $this->getSpatialIndexPath($dbname, $field, $segment); + @unlink($path); + unset($this->spatialIndexCache[$path], $this->dirtySpatialIndexes[$path]); } + $globalPath = $this->getGlobalSpatialIndexPath($dbname, $field); + @unlink($globalPath); + unset($this->globalSpatialIndexCache[$globalPath]); + if ($this->dbExists($dbname)) { + $manifest = $this->getIndexManifest($dbname); + $manifest['spatial'] = array_values(array_diff($manifest['spatial'], [$field])); + unset($manifest['spatialOptions'][$field]); + $this->saveIndexManifest($dbname, $manifest); + } return ['success' => true, 'error' => null]; } @@ -6747,27 +6002,15 @@ public function dropSpatialIndex(string $dbname, string $field): array { * @param string $dbname Database name * @return array List of indexed field names */ - public function getSpatialIndexes(string $dbname): array { - $dbname = $this->sanitizeDbName($dbname); - - if ($this->isSharded($dbname)) { - // Check global indexes - $hash = $this->hashDBName($dbname); - $pattern = $this->dbDir . $hash . '.nonedb.gsidx.*'; - $files = glob($pattern); - $fields = []; - - foreach ($files as $file) { - $parts = explode('.gsidx.', basename($file)); - if (count($parts) >= 2) { - $fields[] = $parts[1]; - } - } - - return $fields; - } + public function getSpatialIndexes(string $dbname): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname){ + return $this->getSpatialIndexesUnlocked($dbname); + }); + } - return $this->getSpatialIndexedFields($dbname, null); + private function getSpatialIndexesUnlocked(string $dbname): array{ + return $this->dbExists($dbname) ? $this->getIndexManifest($dbname)['spatial'] : []; } /** @@ -6776,209 +6019,254 @@ public function getSpatialIndexes(string $dbname): array { * @param string $field Field name * @return array Result from createSpatialIndex */ - public function rebuildSpatialIndex(string $dbname, string $field): array { - $this->dropSpatialIndex($dbname, $field); - return $this->createSpatialIndex($dbname, $field); + public function rebuildSpatialIndex(string $dbname, string $field): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname, $field){ + return $this->rebuildSpatialIndexUnlocked($dbname, $field); + }); } - /** - * Find records within a distance from a point - * @param string $dbname Database name - * @param string $field GeoJSON field name - * @param float $lon Center longitude - * @param float $lat Center latitude - * @param float $distanceMeters Distance in meters - * @param array $options ['includeDistance' => false] - * @return array Matching records (with _distance in meters if includeDistance=true) - */ - public function withinDistance(string $dbname, string $field, float $lon, float $lat, float $distanceMeters, array $options = []): array { - $dbname = $this->sanitizeDbName($dbname); - $includeDistance = $options['includeDistance'] ?? false; - - // Convert circle to bounding box for R-tree search - $searchMBR = $this->circleToBBox($lon, $lat, $distanceMeters); - - // Get candidate keys from spatial index - $candidateKeys = []; - if ($this->isSharded($dbname)) { - $candidateKeys = $this->withinDistanceSharded($dbname, $field, $lon, $lat, $distanceMeters, $searchMBR); - } else { - if ($this->hasSpatialIndex($dbname, $field, null)) { - $candidateKeys = $this->rtreeSearchMBR($dbname, $field, $searchMBR, null); - } + private function rebuildSpatialIndexUnlocked(string $dbname, string $field): array{ + $options = []; + $nodeSize = $this->dbExists($dbname) ? ($this->getIndexManifest($dbname)['spatialOptions'][$field]['nodeSize'] ?? null) : null; + if ($nodeSize) { + $options['nodeSize'] = $nodeSize; } + $this->dropSpatialIndexUnlocked($dbname, $field); + return $this->createSpatialIndexUnlocked($dbname, $field, $options); + } - // Fetch and filter records - $results = []; - - if (!empty($candidateKeys)) { - // Indexed path: fetch by keys - foreach ($candidateKeys as $key) { - $record = $this->find($dbname, ['key' => $key]); - if (empty($record)) continue; - $record = $record[0] ?? $record; - - if (!isset($record[$field])) continue; + // ========================================== + // SPATIAL QUERIES + // ========================================== - $geometry = $record[$field]; - $centroid = $this->getGeometryCentroid($geometry); - $distance = $this->haversineDistance($lon, $lat, $centroid[0], $centroid[1]); + /** + * Records whose geometry may overlap the search boxes: [key => record]. + * Uses the R-tree (and shard boxes) when the field is indexed; otherwise every record. + * @param array $boxes List of MBRs + */ + private function spatialCandidates(string $dbname, string $field, array $boxes): array { + if (!$this->dbExists($dbname)) { + return []; + } + $indexed = $this->spatialIndexEnabled && in_array($field, $this->getIndexManifest($dbname)['spatial'], true); + $records = []; + $sharded = $this->isSharded($dbname); + $global = ($indexed && $sharded) ? $this->readGlobalSpatialIndex($dbname, $field) : null; - if ($distance <= $distanceMeters) { - if ($includeDistance) { - $record['_distance'] = round($distance, 2); + foreach ($this->listSegments($dbname) as $segment) { + if (!$indexed) { + $records += $this->readSegmentRecords($dbname, $segment); + continue; + } + if ($global !== null) { + $shardMBR = $global['shardMBRs'][(string)$segment] ?? null; + $overlaps = false; + foreach ($boxes as $box) { + if ($shardMBR !== null && $this->mbrOverlaps($shardMBR, $box)) { + $overlaps = true; + break; } - $results[] = $record; } - } - } else { - // Fallback: scan all records - $allRecords = $this->find($dbname); - foreach ($allRecords as $record) { - if (!isset($record[$field])) continue; - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) continue; - - $centroid = $this->getGeometryCentroid($geometry); - $distance = $this->haversineDistance($lon, $lat, $centroid[0], $centroid[1]); - - if ($distance <= $distanceMeters) { - if ($includeDistance) { - $record['_distance'] = round($distance, 2); + if (!$overlaps) { + // Verify the shard box is current before skipping: it's rebuilt with the shard index + if ($this->readSpatialIndex($dbname, $field, $segment) !== null) { + continue; } - $results[] = $record; } } + $keys = $this->rtreeSearchMBR($dbname, $field, $boxes, $segment); + if (!empty($keys)) { + $records += $this->readSegmentRecords($dbname, $segment, $keys); + } } + return $records; + } - // Sort by distance if included - if ($includeDistance) { - usort($results, fn($a, $b) => $a['_distance'] <=> $b['_distance']); + /** + * Search boxes for a circle; splits at the antimeridian and covers all longitudes near the poles + * @return array List of [minLon, minLat, maxLon, maxLat] + */ + private function circleToBBoxes(float $lon, float $lat, float $radiusMeters): array { + $latDeg = $radiusMeters / 111000.0; + $minLat = $lat - $latDeg; + $maxLat = $lat + $latDeg; + if ($minLat <= -90 || $maxLat >= 90) { + return [[-180.0, max(-90.0, $minLat), 180.0, min(90.0, $maxLat)]]; } - - return $results; + $cos = cos(deg2rad(max(abs($minLat), abs($maxLat)))); + $lonDeg = $cos > 1e-9 ? $radiusMeters / (111000.0 * $cos) : 360.0; + if ($lonDeg >= 180) { + return [[-180.0, $minLat, 180.0, $maxLat]]; + } + $minLon = $lon - $lonDeg; + $maxLon = $lon + $lonDeg; + if ($minLon < -180) { + return [[-180.0, $minLat, $maxLon, $maxLat], [$minLon + 360.0, $minLat, 180.0, $maxLat]]; + } + if ($maxLon > 180) { + return [[$minLon, $minLat, 180.0, $maxLat], [-180.0, $minLat, $maxLon - 360.0, $maxLat]]; + } + return [[$minLon, $minLat, $maxLon, $maxLat]]; } /** - * Sharded withinDistance with shard-skip optimization + * Distance in meters from a point to a geometry: 0 inside a polygon, otherwise to the nearest + * vertex or edge (edges use a local equirectangular approximation) */ - private function withinDistanceSharded(string $dbname, string $field, float $lon, float $lat, float $distanceMeters, array $searchMBR): array { - $globalIndex = $this->readGlobalSpatialIndex($dbname, $field); - $candidateKeys = []; - - if ($globalIndex !== null && isset($globalIndex['shardMBRs'])) { - // Shard-skip: only search shards whose MBR overlaps search MBR - foreach ($globalIndex['shardMBRs'] as $shardId => $shardMBR) { - if ($this->mbrOverlaps($searchMBR, $shardMBR)) { - $keys = $this->rtreeSearchMBR($dbname, $field, $searchMBR, (int)$shardId); - $candidateKeys = array_merge($candidateKeys, $keys); + private function distanceToGeometry(float $lon, float $lat, array $geometry): float { + $type = $geometry['type'] ?? ''; + $coords = $geometry['coordinates'] ?? []; + switch ($type) { + case 'Point': + return $this->haversineDistance($lon, $lat, (float)$coords[0], (float)$coords[1]); + case 'MultiPoint': + $best = INF; + foreach ($coords as $p) { + $best = min($best, $this->haversineDistance($lon, $lat, (float)$p[0], (float)$p[1])); } - } - } else { - // No global index: search all shards - $meta = $this->getCachedMeta($dbname); - if (isset($meta['shards'])) { - foreach ($meta['shards'] as $shard) { - if ($this->hasSpatialIndex($dbname, $field, $shard['id'])) { - $keys = $this->rtreeSearchMBR($dbname, $field, $searchMBR, $shard['id']); - $candidateKeys = array_merge($candidateKeys, $keys); - } + return $best; + case 'LineString': + return $this->distanceToPath($lon, $lat, $coords); + case 'MultiLineString': + $best = INF; + foreach ($coords as $line) { + $best = min($best, $this->distanceToPath($lon, $lat, $line)); } - } + return $best; + case 'Polygon': + if ($this->pointInPolygon($lon, $lat, $coords)) { + return 0.0; + } + $best = INF; + foreach ($coords as $ring) { + $best = min($best, $this->distanceToPath($lon, $lat, $ring)); + } + return $best; + case 'MultiPolygon': + $best = INF; + foreach ($coords as $polygon) { + $best = min($best, $this->distanceToGeometry($lon, $lat, ['type' => 'Polygon', 'coordinates' => $polygon])); + } + return $best; + case 'GeometryCollection': + $best = INF; + foreach ($geometry['geometries'] ?? [] as $g) { + $best = min($best, $this->distanceToGeometry($lon, $lat, $g)); + } + return $best; } + return INF; + } - return $candidateKeys; + /** + * Distance from a point to a polyline (meters) + */ + private function distanceToPath(float $lon, float $lat, array $points): float { + $n = count($points); + if ($n === 0) { + return INF; + } + $best = $this->haversineDistance($lon, $lat, (float)$points[0][0], (float)$points[0][1]); + $kx = cos(deg2rad($lat)); + for ($i = 0; $i < $n - 1; $i++) { + $ax = ((float)$points[$i][0] - $lon) * $kx; + $ay = (float)$points[$i][1] - $lat; + $bx = ((float)$points[$i + 1][0] - $lon) * $kx; + $by = (float)$points[$i + 1][1] - $lat; + $dx = $bx - $ax; + $dy = $by - $ay; + $len2 = $dx * $dx + $dy * $dy; + $t = $len2 > 0 ? max(0.0, min(1.0, -($ax * $dx + $ay * $dy) / $len2)) : 0.0; + $px = (float)$points[$i][0] + ((float)$points[$i + 1][0] - (float)$points[$i][0]) * $t; + $py = (float)$points[$i][1] + ((float)$points[$i + 1][1] - (float)$points[$i][1]) * $t; + $best = min($best, $this->haversineDistance($lon, $lat, $px, $py)); + } + return $best; } /** - * Find records within a bounding box + * Find records within a distance from a point (distance to the nearest point of the geometry) * @param string $dbname Database name * @param string $field GeoJSON field name - * @param float $minLon Minimum longitude - * @param float $minLat Minimum latitude - * @param float $maxLon Maximum longitude - * @param float $maxLat Maximum latitude - * @return array Matching records + * @param float $lon Center longitude + * @param float $lat Center latitude + * @param float $distanceMeters Distance in meters + * @param array $options ['includeDistance' => false] + * @return array Matching records (sorted by distance, with _distance in meters if includeDistance=true) */ - public function withinBBox(string $dbname, string $field, float $minLon, float $minLat, float $maxLon, float $maxLat): array { - $dbname = $this->sanitizeDbName($dbname); - $searchMBR = [$minLon, $minLat, $maxLon, $maxLat]; + public function withinDistance(string $dbname, string $field, float $lon, float $lat, float $distanceMeters, array $options = []): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $lon, $lat, $distanceMeters, $options){ + return $this->withinDistanceUnlocked($dbname, $field, $lon, $lat, $distanceMeters, $options); + }); + } - $candidateKeys = []; - if ($this->isSharded($dbname)) { - $candidateKeys = $this->withinBBoxSharded($dbname, $field, $searchMBR); - } else { - if ($this->hasSpatialIndex($dbname, $field, null)) { - $candidateKeys = $this->rtreeSearchMBR($dbname, $field, $searchMBR, null); - } + private function withinDistanceUnlocked(string $dbname, string $field, float $lon, float $lat, float $distanceMeters, array $options = []): array{ + $includeDistance = $options['includeDistance'] ?? false; + if ($distanceMeters < 0) { + return []; } - $results = []; - - if (!empty($candidateKeys)) { - foreach ($candidateKeys as $key) { - $record = $this->find($dbname, ['key' => $key]); - if (empty($record)) continue; - $record = $record[0] ?? $record; - - if (!isset($record[$field])) continue; - - // Verify MBR overlap - $geometry = $record[$field]; - $recordMBR = $this->calculateMBR($geometry); - if ($this->mbrOverlaps($recordMBR, $searchMBR)) { - $results[] = $record; - } + foreach ($this->spatialCandidates($dbname, $field, $this->circleToBBoxes($lon, $lat, $distanceMeters)) as $record) { + $geometry = $record[$field] ?? null; + if (!is_array($geometry) || !$this->validateGeoJSON($geometry)['valid']) { + continue; } - } else { - // Fallback: scan all records - $allRecords = $this->find($dbname); - foreach ($allRecords as $record) { - if (!isset($record[$field])) continue; - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) continue; - - $recordMBR = $this->calculateMBR($geometry); - if ($this->mbrOverlaps($recordMBR, $searchMBR)) { - $results[] = $record; - } + $distance = $this->distanceToGeometry($lon, $lat, $geometry); + if ($distance <= $distanceMeters) { + $record['_distance'] = round($distance, 2); + $results[] = $record; } } - + usort($results, function ($a, $b) { + return $a['_distance'] <=> $b['_distance'] ?: $a['key'] <=> $b['key']; + }); + if (!$includeDistance) { + foreach ($results as &$r) { + unset($r['_distance']); + } + unset($r); + } + $this->clearDistanceCache(); return $results; } /** - * Sharded withinBBox + * Find records within a bounding box (geometries whose bounding box overlaps it). + * minLon > maxLon means the box crosses the antimeridian. + * @return array Matching records */ - private function withinBBoxSharded(string $dbname, string $field, array $searchMBR): array { - $globalIndex = $this->readGlobalSpatialIndex($dbname, $field); - $candidateKeys = []; + public function withinBBox(string $dbname, string $field, float $minLon, float $minLat, float $maxLon, float $maxLat): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $minLon, $minLat, $maxLon, $maxLat){ + return $this->withinBBoxUnlocked($dbname, $field, $minLon, $minLat, $maxLon, $maxLat); + }); + } - if ($globalIndex !== null && isset($globalIndex['shardMBRs'])) { - foreach ($globalIndex['shardMBRs'] as $shardId => $shardMBR) { - if ($this->mbrOverlaps($searchMBR, $shardMBR)) { - $keys = $this->rtreeSearchMBR($dbname, $field, $searchMBR, (int)$shardId); - $candidateKeys = array_merge($candidateKeys, $keys); - } + private function withinBBoxUnlocked(string $dbname, string $field, float $minLon, float $minLat, float $maxLon, float $maxLat): array{ + if ($minLat > $maxLat) { + [$minLat, $maxLat] = [$maxLat, $minLat]; + } + $boxes = $minLon <= $maxLon + ? [[$minLon, $minLat, $maxLon, $maxLat]] + : [[$minLon, $minLat, 180.0, $maxLat], [-180.0, $minLat, $maxLon, $maxLat]]; + + $results = []; + foreach ($this->spatialCandidates($dbname, $field, $boxes) as $record) { + $geometry = $record[$field] ?? null; + if (!is_array($geometry) || !$this->validateGeoJSON($geometry)['valid']) { + continue; } - } else { - $meta = $this->getCachedMeta($dbname); - if (isset($meta['shards'])) { - foreach ($meta['shards'] as $shard) { - if ($this->hasSpatialIndex($dbname, $field, $shard['id'])) { - $keys = $this->rtreeSearchMBR($dbname, $field, $searchMBR, $shard['id']); - $candidateKeys = array_merge($candidateKeys, $keys); - } + $recordMBR = $this->calculateMBR($geometry); + foreach ($boxes as $box) { + if ($this->mbrOverlaps($recordMBR, $box)) { + $results[] = $record; + break; } } } - - return $candidateKeys; + return $results; } /** @@ -6988,333 +6276,202 @@ private function withinBBoxSharded(string $dbname, string $field, array $searchM * @param float $lon Center longitude * @param float $lat Center latitude * @param int $k Number of nearest records - * @param array $options ['maxDistance' => null (in meters), 'includeDistance' => true] - * @return array Nearest records sorted by distance (with _distance in meters) + * @param array $options ['maxDistance' => null (meters), 'includeDistance' => true] + * @return array Up to k records sorted by distance (fewer when maxDistance cuts them off) */ - public function nearest(string $dbname, string $field, float $lon, float $lat, int $k, array $options = []): array { - $dbname = $this->sanitizeDbName($dbname); - $maxDistance = $options['maxDistance'] ?? null; - $includeDistance = $options['includeDistance'] ?? true; - - // Adaptive expanding search with exponential growth - // Start small for dense data, expand quickly for sparse data - $results = []; - $radius = 500; // Start with 500 meters - $maxRadius = $maxDistance ?? 10000000; // Max 10000km = 10M meters (global) - $attempts = 0; - $maxAttempts = 15; // Prevents infinite loop - - while (count($results) < $k && $radius <= $maxRadius && $attempts < $maxAttempts) { - $found = $this->withinDistance($dbname, $field, $lon, $lat, $radius, ['includeDistance' => true]); + public function nearest(string $dbname, string $field, float $lon, float $lat, int $k, array $options = []): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $lon, $lat, $k, $options){ + return $this->nearestUnlocked($dbname, $field, $lon, $lat, $k, $options); + }); + } - if (count($found) >= $k) { - $results = $found; - break; - } + /** + * True when spatial queries on the field use the R-tree (used by the query builder) + */ + private function usesSpatialIndex(string $dbname, string $field): bool{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field){ + return $this->spatialIndexEnabled && $this->dbExists($dbname) + && in_array($field, $this->getIndexManifest($dbname)['spatial'], true); + }); + } - // Double the radius for next attempt (exponential growth) - $radius *= 2; - $attempts++; + private function nearestUnlocked(string $dbname, string $field, float $lon, float $lat, int $k, array $options = []): array{ + $maxDistance = isset($options['maxDistance']) ? (float)$options['maxDistance'] : null; + $includeDistance = $options['includeDistance'] ?? true; + if ($k <= 0 || !$this->dbExists($dbname)) { + return []; } - // If we still don't have enough and no max distance, do full scan - if (count($results) < $k && $maxDistance === null && $attempts >= $maxAttempts) { - $allRecords = $this->find($dbname); - $results = []; - - foreach ($allRecords as $record) { - if (!isset($record[$field])) continue; - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) continue; - - $centroid = $this->getGeometryCentroid($geometry); - $distance = $this->haversineDistance($lon, $lat, $centroid[0], $centroid[1]); + $indexed = $this->spatialIndexEnabled && in_array($field, $this->getIndexManifest($dbname)['spatial'], true); + $limit = $maxDistance ?? 20040000.0; // half the earth's circumference covers everything + $results = []; - $record['_distance'] = round($distance, 2); - $results[] = $record; + if ($indexed && $k < $this->countFast($dbname)) { + // Expanding search: nothing outside radius r is closer than r. Results are ordered by the + // rounded distance, so only records rounded below round(r) are certainly in final order + // (a record just outside r can round to the same value and has to win a key tie). + $radius = 1000.0; + while (true) { + $radius = min($radius, $limit); + $found = $this->withinDistanceUnlocked($dbname, $field, $lon, $lat, $radius, ['includeDistance' => true]); + if ($radius >= $limit || (count($found) >= $k && $found[$k - 1]['_distance'] < round($radius, 2))) { + $results = $found; + break; + } + $radius *= 4; + } + } else { + foreach ($this->listSegments($dbname) as $segment) { + foreach ($this->readSegmentRecords($dbname, $segment) as $record) { + $geometry = $record[$field] ?? null; + if (!is_array($geometry) || !$this->validateGeoJSON($geometry)['valid']) { + continue; + } + $distance = $this->distanceToGeometry($lon, $lat, $geometry); + if ($maxDistance === null || $distance <= $maxDistance) { + $record['_distance'] = round($distance, 2); + $results[] = $record; + } + } } + usort($results, function ($a, $b) { + return $a['_distance'] <=> $b['_distance'] ?: $a['key'] <=> $b['key']; + }); } - // Sort by distance and take k - usort($results, fn($a, $b) => $a['_distance'] <=> $b['_distance']); $results = array_slice($results, 0, $k); - - // Remove distance if not requested if (!$includeDistance) { foreach ($results as &$r) { unset($r['_distance']); } + unset($r); } - - // Clear distance cache after query to free memory $this->clearDistanceCache(); - return $results; } /** - * Find records within a polygon + * Find records whose geometry lies within a Polygon or MultiPolygon (holes excluded) * @param string $dbname Database name * @param string $field GeoJSON field name - * @param array $polygon GeoJSON Polygon geometry + * @param array $polygon GeoJSON Polygon or MultiPolygon geometry * @return array Matching records */ - public function withinPolygon(string $dbname, string $field, array $polygon): array { - $dbname = $this->sanitizeDbName($dbname); + public function withinPolygon(string $dbname, string $field, array $polygon): array{ + $dbname = $this->normalizeDbName($dbname); + return $this->withReadLock($dbname, function() use ($dbname, $field, $polygon){ + return $this->withinPolygonUnlocked($dbname, $field, $polygon); + }); + } - // Validate polygon + private function withinPolygonUnlocked(string $dbname, string $field, array $polygon): array{ $validation = $this->validateGeoJSON($polygon); - if (!$validation['valid'] || $validation['type'] !== 'Polygon') { + if (!$validation['valid'] || !in_array($validation['type'], ['Polygon', 'MultiPolygon'], true)) { return []; } - - $polygonCoords = $polygon['coordinates']; - $polygonMBR = $this->calculateMBR($polygon); - - // Get candidates from spatial index - $candidateKeys = []; - if ($this->isSharded($dbname)) { - $candidateKeys = $this->withinPolygonSharded($dbname, $field, $polygonMBR); - } else { - if ($this->hasSpatialIndex($dbname, $field, null)) { - $candidateKeys = $this->rtreeSearchMBR($dbname, $field, $polygonMBR, null); - } + $polygons = $validation['type'] === 'Polygon' ? [$polygon['coordinates']] : $polygon['coordinates']; + $boxes = []; + foreach ($polygons as $coords) { + $boxes[] = $this->mbrFromPoints($coords[0]); } $results = []; - - if (!empty($candidateKeys)) { - foreach ($candidateKeys as $key) { - $record = $this->find($dbname, ['key' => $key]); - if (empty($record)) continue; - $record = $record[0] ?? $record; - - if (!isset($record[$field])) continue; - - $geometry = $record[$field]; - - // Check if geometry is within polygon - if ($this->geometryWithinPolygon($geometry, $polygonCoords)) { - $results[] = $record; - } + foreach ($this->spatialCandidates($dbname, $field, $boxes) as $record) { + $geometry = $record[$field] ?? null; + if (!is_array($geometry) || !$this->validateGeoJSON($geometry)['valid']) { + continue; } - } else { - // Fallback: scan all records - $allRecords = $this->find($dbname); - foreach ($allRecords as $record) { - if (!isset($record[$field])) continue; - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) continue; - - if ($this->geometryWithinPolygon($geometry, $polygonCoords)) { + foreach ($polygons as $coords) { + if ($this->geometryWithinPolygon($geometry, $coords)) { $results[] = $record; + break; } } } - return $results; } /** - * Sharded withinPolygon - */ - private function withinPolygonSharded(string $dbname, string $field, array $polygonMBR): array { - $globalIndex = $this->readGlobalSpatialIndex($dbname, $field); - $candidateKeys = []; - - if ($globalIndex !== null && isset($globalIndex['shardMBRs'])) { - foreach ($globalIndex['shardMBRs'] as $shardId => $shardMBR) { - if ($this->mbrOverlaps($polygonMBR, $shardMBR)) { - $keys = $this->rtreeSearchMBR($dbname, $field, $polygonMBR, (int)$shardId); - $candidateKeys = array_merge($candidateKeys, $keys); - } - } - } else { - $meta = $this->getCachedMeta($dbname); - if (isset($meta['shards'])) { - foreach ($meta['shards'] as $shard) { - if ($this->hasSpatialIndex($dbname, $field, $shard['id'])) { - $keys = $this->rtreeSearchMBR($dbname, $field, $polygonMBR, $shard['id']); - $candidateKeys = array_merge($candidateKeys, $keys); - } - } - } - } - - return $candidateKeys; - } - - /** - * Check if a geometry is within a polygon + * Check if a geometry lies entirely within a polygon: every vertex inside (not in a hole) and no + * edge crossing the polygon's boundary */ private function geometryWithinPolygon(array $geometry, array $polygonCoords): bool { $type = $geometry['type'] ?? ''; + $coords = $geometry['coordinates'] ?? []; switch ($type) { case 'Point': - $coords = $geometry['coordinates']; - return $this->pointInPolygon($coords[0], $coords[1], $polygonCoords); - - case 'LineString': + return $this->pointInPolygon((float)$coords[0], (float)$coords[1], $polygonCoords); case 'MultiPoint': - // All points must be within polygon - foreach ($geometry['coordinates'] as $point) { - if (!$this->pointInPolygon($point[0], $point[1], $polygonCoords)) { + foreach ($coords as $point) { + if (!$this->pointInPolygon((float)$point[0], (float)$point[1], $polygonCoords)) { + return false; + } + } + return true; + case 'LineString': + return $this->pathWithinPolygon($coords, $polygonCoords); + case 'MultiLineString': + foreach ($coords as $line) { + if (!$this->pathWithinPolygon($line, $polygonCoords)) { return false; } } return true; - case 'Polygon': - // All vertices of outer ring must be within polygon - foreach ($geometry['coordinates'][0] as $point) { - if (!$this->pointInPolygon($point[0], $point[1], $polygonCoords)) { + return $this->pathWithinPolygon($coords[0], $polygonCoords); + case 'MultiPolygon': + foreach ($coords as $poly) { + if (!$this->pathWithinPolygon($poly[0], $polygonCoords)) { + return false; + } + } + return true; + case 'GeometryCollection': + foreach ($geometry['geometries'] ?? [] as $g) { + if (!$this->geometryWithinPolygon($g, $polygonCoords)) { return false; } } return true; - - default: - // For complex types, check centroid - $centroid = $this->getGeometryCentroid($geometry); - return $this->pointInPolygon($centroid[0], $centroid[1], $polygonCoords); - } - } - - // ========================================== - // SPATIAL INDEX - CRUD INTEGRATION - // ========================================== - - /** - * Update spatial index when a record is inserted - * @param string $dbname Database name - * @param array $record The inserted record - * @param int $key Record key - * @param int|null $shardId Shard ID - */ - private function updateSpatialIndexOnInsert(string $dbname, array $record, int $key, ?int $shardId = null): void { - if (!$this->spatialIndexEnabled) return; - - $spatialFields = $this->getSpatialIndexedFields($dbname, $shardId); - - foreach ($spatialFields as $field) { - if (!isset($record[$field]) || !is_array($record[$field])) { - continue; - } - - $geometry = $record[$field]; - $validation = $this->validateGeoJSON($geometry); - if (!$validation['valid']) { - continue; - } - - // Insert into R-tree - $this->rtreeInsert($dbname, $field, $key, $geometry, $shardId); - - // Update global spatial index for sharded databases - if ($shardId !== null) { - $this->updateGlobalSpatialMBR($dbname, $field, $shardId, $geometry); - } } + return false; } /** - * Update spatial index when a record is deleted - * @param string $dbname Database name - * @param array $record The deleted record - * @param int $key Record key - * @param int|null $shardId Shard ID + * All vertices of a path inside the polygon and no segment crossing any of its rings */ - private function updateSpatialIndexOnDelete(string $dbname, array $record, int $key, ?int $shardId = null): void { - if (!$this->spatialIndexEnabled) return; - - $spatialFields = $this->getSpatialIndexedFields($dbname, $shardId); - - foreach ($spatialFields as $field) { - if (!isset($record[$field])) { - continue; + private function pathWithinPolygon(array $points, array $polygonCoords): bool { + foreach ($points as $point) { + if (!$this->pointInPolygon((float)$point[0], (float)$point[1], $polygonCoords)) { + return false; } - - // Delete from R-tree - $this->rtreeDelete($dbname, $field, $key, $shardId); } - } - - /** - * Update spatial index when a record is updated - * @param string $dbname Database name - * @param array $oldRecord The old record - * @param array $newRecord The new record - * @param int $key Record key - * @param int|null $shardId Shard ID - */ - private function updateSpatialIndexOnUpdate(string $dbname, array $oldRecord, array $newRecord, int $key, ?int $shardId = null): void { - if (!$this->spatialIndexEnabled) return; - - $spatialFields = $this->getSpatialIndexedFields($dbname, $shardId); - - foreach ($spatialFields as $field) { - $oldGeometry = $oldRecord[$field] ?? null; - $newGeometry = $newRecord[$field] ?? null; - - // Check if geometry changed - if ($oldGeometry === $newGeometry) { - continue; - } - - // Delete old entry if it existed - if ($oldGeometry !== null && is_array($oldGeometry)) { - $this->rtreeDelete($dbname, $field, $key, $shardId); - } - - // Insert new entry if it exists - if ($newGeometry !== null && is_array($newGeometry)) { - $validation = $this->validateGeoJSON($newGeometry); - if ($validation['valid']) { - $this->rtreeInsert($dbname, $field, $key, $newGeometry, $shardId); - - // Update global MBR for sharded - if ($shardId !== null) { - $this->updateGlobalSpatialMBR($dbname, $field, $shardId, $newGeometry); + $n = count($points); + for ($i = 0; $i < $n - 1; $i++) { + foreach ($polygonCoords as $ring) { + $m = count($ring); + for ($j = 0; $j < $m - 1; $j++) { + if ($this->segmentsCrossProperly($points[$i], $points[$i + 1], $ring[$j], $ring[$j + 1])) { + return false; } } } } + return true; } /** - * Update global spatial index MBR for a shard + * True when two segments cross at a single interior point (touching endpoints don't count) */ - private function updateGlobalSpatialMBR(string $dbname, string $field, int $shardId, array $geometry): void { - $globalIndex = $this->readGlobalSpatialIndex($dbname, $field); - - if ($globalIndex === null) { - $globalIndex = [ - 'v' => 1, - 'field' => $field, - 'shardMBRs' => [] - ]; - } - - $geometryMBR = $this->calculateMBR($geometry); - $shardIdStr = (string)$shardId; - - if (isset($globalIndex['shardMBRs'][$shardIdStr])) { - // Expand existing MBR - $globalIndex['shardMBRs'][$shardIdStr] = $this->mbrUnion( - $globalIndex['shardMBRs'][$shardIdStr], - $geometryMBR - ); - } else { - // New shard MBR - $globalIndex['shardMBRs'][$shardIdStr] = $geometryMBR; - } - - $this->writeGlobalSpatialIndex($dbname, $field, $globalIndex); + private function segmentsCrossProperly(array $p1, array $p2, array $p3, array $p4): bool { + $d1 = $this->crossProductDirection($p3, $p4, $p1); + $d2 = $this->crossProductDirection($p3, $p4, $p2); + $d3 = $this->crossProductDirection($p1, $p2, $p3); + $d4 = $this->crossProductDirection($p1, $p2, $p4); + return (($d1 > 0 && $d2 < 0) || ($d1 < 0 && $d2 > 0)) && (($d3 > 0 && $d4 < 0) || ($d3 < 0 && $d4 > 0)); } // ========================================== @@ -7327,74 +6484,53 @@ private function updateGlobalSpatialMBR(string $dbname, string $field, int $shar * @return array ['success' => bool, 'flushed' => int, 'error' => string|null] */ public function flush($dbname){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname){ + return $this->flushUnlocked($dbname); + }); + } - if($this->isSharded($dbname)){ - $result = $this->flushAllShardBuffers($dbname); - return ['success' => true, 'flushed' => $result['flushed'], 'error' => null]; - } else { - return $this->flushBufferToMain($dbname); - } + private function flushUnlocked($dbname){ + $flushed = $this->flushPendingBuffers($dbname); + return ['success' => true, 'flushed' => $flushed, 'error' => null]; } /** - * Flush all buffers for all known databases - * Called automatically on shutdown if bufferAutoFlushOnShutdown is true + * Flush the write buffers of every database (of this secretKey) in dbDir * @return array ['databases' => int, 'flushed' => int] */ public function flushAllBuffers(){ - $dbDir = $this->dbDir; - $totalFlushed = 0; - $dbCount = 0; - - // Find all buffer files - $bufferFiles = glob($dbDir . '*.buffer'); - if($bufferFiles === false){ - $bufferFiles = []; - } - - // Track which databases we've processed - $processedDbs = []; - - foreach($bufferFiles as $bufferFile){ - $basename = basename($bufferFile); - - // Extract database name from buffer file name - // Format: hash-dbname.nonedb.buffer or hash-dbname_s0.nonedb.buffer - if(preg_match('/^[a-f0-9]+-(.+?)(?:_s\d+)?\.nonedb\.buffer$/', $basename, $matches)){ - $dbname = $matches[1]; - - // Avoid processing same DB multiple times - if(isset($processedDbs[$dbname])){ - continue; - } - $processedDbs[$dbname] = true; - - // Check if sharded or non-sharded - if($this->isSharded($dbname)){ - $result = $this->flushAllShardBuffers($dbname); - $totalFlushed += $result['flushed']; - } else { - $result = $this->flushBufferToMain($dbname); - if($result['success']){ - $totalFlushed += $result['flushed']; - } + $names = []; + if(is_dir($this->dbDir)){ + foreach(scandir($this->dbDir) ?: [] as $file){ + if(preg_match("/^([0-9a-f]{20})-([A-Za-z0-9' -]+)(?:_s\\d+)?\\.nonedb\\.buffer(?:\\.flushing)?$/", $file, $m) + && $this->hashDBName($m[2]) === $m[1]){ + $names[$m[2]] = true; } - $dbCount++; } } - - return ['databases' => $dbCount, 'flushed' => $totalFlushed]; + $total = 0; + foreach(array_keys($names) as $dbname){ + $result = $this->flush((string)$dbname); + $total += $result['flushed']; + } + return ['databases' => count($names), 'flushed' => $total]; } /** * Get buffer information for a database * @param string $dbname - * @return array Buffer statistics + * @return array */ public function getBufferInfo($dbname){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + $fn = function() use ($dbname){ + return $this->getBufferInfoUnlocked($dbname); + }; + return $this->dbFilesExist($dbname) ? $this->withLock($dbname, false, $fn) : $fn(); + } + private function getBufferInfoUnlocked($dbname){ $info = [ 'enabled' => $this->bufferEnabled, 'sizeLimit' => $this->bufferSizeLimit, @@ -7402,28 +6538,21 @@ public function getBufferInfo($dbname){ 'flushInterval' => $this->bufferFlushInterval, 'buffers' => [] ]; - - if($this->isSharded($dbname)){ - $meta = $this->getCachedMeta($dbname); - if($meta !== null && isset($meta['shards'])){ - foreach($meta['shards'] as $shard){ - $bufferPath = $this->getShardBufferPath($dbname, $shard['id']); - $info['buffers']['shard_' . $shard['id']] = [ - 'exists' => $this->hasBuffer($bufferPath), - 'size' => $this->getBufferSize($bufferPath), - 'records' => $this->hasBuffer($bufferPath) ? $this->getBufferRecordCount($bufferPath) : 0 - ]; - } - } - } else { - $bufferPath = $this->getBufferPath($dbname); - $info['buffers']['main'] = [ - 'exists' => $this->hasBuffer($bufferPath), - 'size' => $this->getBufferSize($bufferPath), - 'records' => $this->hasBuffer($bufferPath) ? $this->getBufferRecordCount($bufferPath) : 0 + $describe = function($path){ + clearstatcache(true, $path); + $exists = file_exists($path) && filesize($path) > 0; + return [ + 'exists' => $exists, + 'size' => $exists ? (int)filesize($path) : 0, + 'records' => $exists ? $this->getBufferRecordCount($path) : 0 ]; + }; + $info['buffers']['main'] = $describe($this->getBufferPath($dbname)); + foreach($this->legacyShardBufferFiles($dbname) as $file){ + if(preg_match('/_s(\d+)\.nonedb\.buffer$/', $file, $m)){ + $info['buffers']['shard_' . $m[1]] = $describe($file); + } } - return $info; } @@ -7468,190 +6597,128 @@ public function setBufferCountLimit($count){ } /** - * Compact a database by removing null entries - * Works for both sharded and non-sharded databases + * Compact a database: drop old record versions and deleted records from the data files. + * Keys are preserved (sharded and non-sharded alike); indexes stay valid. * @param string $dbname - * @return array + * @return array ['success' => bool, 'freedSlots' => int, 'totalRecords' => int, 'sharded' => bool] */ public function compact($dbname){ - $dbname = $this->sanitizeDbName($dbname); - $result = array("success" => false, "freedSlots" => 0); - - // Handle non-sharded database - if(!$this->isSharded($dbname)){ - $hash = $this->hashDBName($dbname); - $fullDBPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - - if(!$this->cachedFileExists($fullDBPath)){ - $result['status'] = 'database_not_found'; - return $result; - } - - // Ensure JSONL format (auto-migrate v2 if needed) - $this->ensureJsonlFormat($dbname); - - $index = $this->readJsonlIndex($dbname); - if($index === null){ - $result['status'] = 'read_error'; - return $result; - } - - $freedSlots = $index['d']; // Dirty count = freed slots - $totalRecords = count($index['o']); // Active records in index - - $compactResult = $this->compactJsonl($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname){ + return $this->compactUnlocked($dbname); + }); + } - $result['success'] = true; - $result['freedSlots'] = $freedSlots; - $result['totalRecords'] = $totalRecords; - $result['sharded'] = false; + private function compactUnlocked($dbname){ + $result = array("success" => false, "freedSlots" => 0); + $this->flushPendingBuffers($dbname); + if(!$this->dbExists($dbname)){ + $result['status'] = 'database_not_found'; return $result; } - // Handle sharded database - $meta = $this->getCachedMeta($dbname); - if($meta === null){ - $result['status'] = 'meta_read_error'; - return $result; + $sharded = $this->isSharded($dbname); + $freed = 0; + if($sharded){ + $meta = $this->getCachedMeta($dbname, true); + $freed = (int)($meta['deletedCount'] ?? 0); } - - $allRecords = []; - // Use meta's deletedCount for freedSlots (JSONL index 'd' may be 0 after auto-compaction) - $freedSlots = $meta['deletedCount'] ?? 0; - - // v3.0.0: Collect all non-null records from all shards (JSONL format) - foreach($meta['shards'] as $shard){ - $shardId = $shard['id']; - $shardPath = $this->getShardPath($dbname, $shardId); - - // Ensure JSONL format (auto-migrate if needed) - $this->ensureJsonlFormat($dbname, $shardId); - - // Read from JSONL - $jsonlIndex = $this->readJsonlIndex($dbname, $shardId); - if($jsonlIndex !== null){ - foreach($jsonlIndex['o'] as $globalKey => $location){ - $record = $this->readJsonlRecord($shardPath, $location[0], $location[1]); - if($record !== null){ - unset($record['key']); // Remove key as it will be reassigned - $allRecords[] = $record; - } - } + $total = 0; + foreach($this->listSegments($dbname) as $segment){ + $compacted = $this->compactJsonl($dbname, $segment); + if(!$sharded){ + $freed += $compacted['freed']; } - - // Delete old shard file and index - if(file_exists($shardPath)){ - unlink($shardPath); - } - $indexPath = $this->getJsonlIndexPath($dbname, $shardId); - if(file_exists($indexPath)){ - unlink($indexPath); + $total += $compacted['compacted']; + } + if($sharded){ + $this->removeEmptyShards($dbname); + $meta = $this->getCachedMeta($dbname, true); + foreach($meta['shards'] as $i => $shard){ + $meta['shards'][$i]['deleted'] = 0; } - // Clear cache - unset($this->indexCache[$indexPath]); - unset($this->jsonlFormatCache[$shardPath]); + $meta['deletedCount'] = 0; + $meta['totalRecords'] = $total; + $this->writeMeta($dbname, $meta); } - // Recalculate and rebuild shards - $totalRecords = count($allRecords); - $numShards = (int) ceil($totalRecords / $this->shardSize); - if($numShards === 0) $numShards = 1; - - $newMeta = array( - "version" => 1, - "shardSize" => $this->shardSize, - "totalRecords" => $totalRecords, - "deletedCount" => 0, - "nextKey" => $totalRecords, - "shards" => [] - ); - - // v3.0.0: Write shards in JSONL format - for($shardId = 0; $shardId < $numShards; $shardId++){ - $start = $shardId * $this->shardSize; - $shardRecords = array_slice($allRecords, $start, $this->shardSize); - - $newMeta['shards'][] = array( - "id" => $shardId, - "file" => "_s" . $shardId, - "count" => count($shardRecords), - "deleted" => 0 - ); - - // Write shard in JSONL format - $shardPath = $this->getShardPath($dbname, $shardId); - $baseKey = $shardId * $this->shardSize; - - $index = [ - 'v' => 3, - 'format' => 'jsonl', - 'created' => time(), - 'n' => 0, - 'd' => 0, - 'o' => [] - ]; - - $buffer = ''; - $offset = 0; - - foreach($shardRecords as $localKey => $record){ - $globalKey = $baseKey + $localKey; - $record['key'] = $globalKey; - - $json = json_encode($record, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES) . "\n"; - $length = strlen($json) - 1; - - $index['o'][$globalKey] = [$offset, $length]; - $offset += strlen($json); - $index['n']++; + $result['success'] = true; + $result['freedSlots'] = $freed; + $result['totalRecords'] = $total; + $result['sharded'] = $sharded; + if($sharded){ + $result['newShardCount'] = count($this->listSegments($dbname)); + } + return $result; + } - $buffer .= $json; + /** + * Remove shards without records (keys never move, so nothing else changes). The shard that holds + * the newest key is kept: new records go there. + */ + private function removeEmptyShards($dbname){ + $meta = $this->getCachedMeta($dbname, true); + if($meta === null || count($meta['shards']) < 2){ + return; + } + $lastShard = intdiv(max(0, (int)$meta['nextKey'] - 1), (int)$meta['shardSize']); + $kept = []; + $removed = []; + foreach($meta['shards'] as $shard){ + $shardId = (int)$shard['id']; + $index = $this->readJsonlIndex($dbname, $shardId); + if($shardId !== $lastShard && ($index === null || empty($index['o']))){ + $removed[] = $shardId; + } else { + $kept[] = $shard; } - - // Write JSONL file - file_put_contents($shardPath, $buffer, LOCK_EX); - $this->markFileExists($shardPath); - $this->jsonlFormatCache[$shardPath] = true; - - // Write JSONL index - $this->writeJsonlIndex($dbname, $index, $shardId); } + if(empty($removed)){ + return; + } + $meta['shards'] = $kept; + $this->writeMeta($dbname, $meta); - $this->writeMeta($dbname, $newMeta); - - // Rebuild index after compaction (keys are reassigned) - $this->invalidateIndexCache($dbname); - @unlink($this->getIndexPath($dbname)); - $this->buildIndex($dbname); - - $result['success'] = true; - $result['freedSlots'] = $freedSlots; - $result['newShardCount'] = $numShards; - $result['sharded'] = true; - return $result; + $manifest = $this->getIndexManifest($dbname); + foreach($removed as $shardId){ + $dataPath = $this->getShardPath($dbname, $shardId); + foreach($manifest['fields'] as $field){ + $this->deleteFieldIndexFile($dbname, $field, $shardId); + } + $this->removeSpatialShardFiles($dbname, $shardId); + @unlink($dataPath . '.jidx'); + @unlink($dataPath); + $this->markFileNotExists($dataPath); + unset($this->indexCache[$dataPath . '.jidx'], $this->jsonlFormatCache[$dataPath]); + } + foreach($manifest['fields'] as $field){ + $this->rebuildGlobalFieldIndex($dbname, $field); + } } - /** * Manually trigger migration to sharded format * @param string $dbname * @return array ["success" => bool, "status" => string] */ public function migrate($dbname){ - $dbname = $this->sanitizeDbName($dbname); + $dbname = $this->normalizeDbName($dbname); + return $this->withLock($dbname, true, function() use ($dbname){ + return $this->migrateUnlocked($dbname); + }); + } + private function migrateUnlocked($dbname){ if($this->isSharded($dbname)){ return array("success" => true, "status" => "already_sharded"); } - - // Check if legacy database exists - $hash = $this->hashDBName($dbname); - $legacyPath = $this->dbDir . $hash . "-" . $dbname . ".nonedb"; - if(!$this->cachedFileExists($legacyPath)){ + $this->flushPendingBuffers($dbname); + if($this->isSharded($dbname)){ + return array("success" => true, "status" => "migrated"); + } + if(!$this->cachedFileExists($this->getDataPath($dbname))){ return array("success" => false, "status" => "database_not_found"); } - - $result = $this->migrateToSharded($dbname); - if($result){ + if($this->migrateToSharded($dbname)){ return array("success" => true, "status" => "migrated"); } return array("success" => false, "status" => "migration_failed"); @@ -7683,7 +6750,7 @@ public function getShardSize(){ * @return noneDBQuery */ public function query(string $dbname): noneDBQuery { - return new noneDBQuery($this, $dbname); + return new noneDBQuery($this, $this->normalizeDbName($dbname)); } } @@ -7696,10 +6763,13 @@ public function query(string $dbname): noneDBQuery { class noneDBQuery { private noneDB $db; private string $dbname; + /** @var array[] where() groups; all of them must match (AND) */ private array $whereFilters = []; + /** @var array[] orWhere() groups; a record matches if the where groups match OR any orWhere group does */ private array $orWhereFilters = []; private array $whereInFilters = []; private array $whereNotInFilters = []; + /** @var array[] whereNot() groups */ private array $whereNotFilters = []; private array $likeFilters = []; private array $notLikeFilters = []; @@ -7711,15 +6781,20 @@ class noneDBQuery { private array $havingFilters = []; private array $searchFilters = []; private array $joinConfigs = []; - private ?string $sortField = null; - private string $sortOrder = 'asc'; + /** @var array[] sort keys in priority order: [['field' => ..., 'order' => 'asc'|'desc'], ...] */ + private array $sortKeys = []; private ?int $limitCount = null; private int $offsetCount = 0; - // Spatial query support (v3.1.0) + // Spatial query support private array $spatialFilters = []; private ?array $distanceConfig = null; + /** @var \Closure Calls noneDB internals without Reflection */ + private \Closure $internal; + + private const OPERATORS = ['$gt', '$gte', '$lt', '$lte', '$ne', '$eq', '$in', '$nin', '$exists', '$like', '$regex', '$contains']; + /** * @param noneDB $db * @param string $dbname @@ -7727,171 +6802,219 @@ class noneDBQuery { public function __construct(noneDB $db, string $dbname) { $this->db = $db; $this->dbname = $dbname; + $this->internal = \Closure::bind(function (string $method, array $args) { + return $this->$method(...$args); + }, $db, noneDB::class); + } + + /** + * Debug output without the parent instance's internals + */ + public function __debugInfo() { + return [ + 'dbname' => $this->dbname, + 'where' => $this->whereFilters, + 'orWhere' => $this->orWhereFilters, + 'sort' => $this->sortKeys, + 'limit' => $this->limitCount, + 'offset' => $this->offsetCount, + ]; } // ========================================== - // FILTER HELPER METHODS (v3.0.0) + // FILTER HELPERS // ========================================== /** - * Check if a record matches all advanced filters (single-pass optimization) - * Consolidates whereNot, whereIn, whereNotIn, like, notLike, between, notBetween, search - * @param array $record - * @return bool + * Call a private method of the parent noneDB instance + * @param string $methodName + * @param mixed ...$args + * @return mixed */ - private function matchesAdvancedFilters(array $record): bool { - // whereNot filters - foreach ($this->whereNotFilters as $field => $value) { - if (array_key_exists($field, $record) && $record[$field] === $value) { - return false; - } - } + private function callPrivateMethod(string $methodName, ...$args) { + return ($this->internal)($methodName, $args); + } - // whereIn filters - foreach ($this->whereInFilters as $filter) { - if (!array_key_exists($filter['field'], $record)) return false; - if (!in_array($record[$filter['field']], $filter['values'], true)) return false; + /** + * True for a filter value that is an operator spec: a non-empty array whose keys all start with "$" + * @throws \InvalidArgumentException when operators and plain keys are mixed + */ + private function isOperatorSpec($value): bool { + if (!is_array($value) || empty($value)) { + return false; } - - // whereNotIn filters - foreach ($this->whereNotInFilters as $filter) { - if (array_key_exists($filter['field'], $record)) { - if (in_array($record[$filter['field']], $filter['values'], true)) return false; + $operators = 0; + foreach (array_keys($value) as $key) { + if (is_string($key) && $key !== '' && $key[0] === '$') { + $operators++; } } - - // like filters - foreach ($this->likeFilters as $like) { - if (!isset($record[$like['field']])) return false; - $value = $record[$like['field']]; - if (is_array($value) || is_object($value)) return false; - $pattern = $like['pattern']; - if (strpos($pattern, '^') === 0 || substr($pattern, -1) === '$') { - $regex = '/' . $pattern . '/i'; - } else { - $regex = '/' . preg_quote($pattern, '/') . '/i'; - } - if (!preg_match($regex, (string)$value)) return false; + if ($operators > 0 && $operators !== count($value)) { + throw new \InvalidArgumentException('noneDB: a filter value cannot mix operators ($gt, $in, ...) with plain keys'); } + return $operators > 0; + } - // notLike filters - foreach ($this->notLikeFilters as $notLike) { - if (isset($record[$notLike['field']])) { - $value = $record[$notLike['field']]; - if (!is_array($value) && !is_object($value)) { - $pattern = $notLike['pattern']; - if (strpos($pattern, '^') === 0 || substr($pattern, -1) === '$') { - $regex = '/' . $pattern . '/i'; - } else { - $regex = '/' . preg_quote($pattern, '/') . '/i'; + /** + * Compile where()/orWhere() groups once per query run: operators are validated and patterns + * turned into regexes here instead of for every record. + * @return array[] per group a list of [field, operator or null for a plain value, operand] + * @throws \InvalidArgumentException for unknown operators or invalid operands + */ + private function compileGroups(array $groups): array { + $compiled = []; + foreach ($groups as $group) { + $conditions = []; + foreach ($group as $field => $value) { + if ($this->isOperatorSpec($value)) { + foreach ($value as $operator => $operand) { + $conditions[] = [(string)$field, $operator, $this->compileOperand($operator, $operand)]; } - if (preg_match($regex, (string)$value)) return false; + } else { + $conditions[] = [(string)$field, null, $value]; } } + $compiled[] = $conditions; } + return $compiled; + } - // between filters - foreach ($this->betweenFilters as $between) { - if (!isset($record[$between['field']])) return false; - $value = $record[$between['field']]; - if ($value < $between['min'] || $value > $between['max']) return false; + private function compileOperand(string $operator, $operand) { + switch ($operator) { + case '$in': + case '$nin': + if (!is_array($operand)) { + throw new \InvalidArgumentException("noneDB: {$operator} needs an array"); + } + return $operand; + case '$like': + return $this->likeRegex($operand); + case '$regex': + return self::regexOperand($operand); + case '$gt': + case '$gte': + case '$lt': + case '$lte': + case '$ne': + case '$eq': + case '$exists': + case '$contains': + return $operand; + default: + throw new \InvalidArgumentException("noneDB: unknown operator '{$operator}' (supported: " . implode(', ', self::OPERATORS) . ')'); } + } - // notBetween filters - foreach ($this->notBetweenFilters as $notBetween) { - if (isset($record[$notBetween['field']])) { - $value = $record[$notBetween['field']]; - if ($value >= $notBetween['min'] && $value <= $notBetween['max']) return false; + /** + * Check if a record matches one compiled where()/orWhere() group (AND of its conditions). + * Plain values compare with ===; operator conditions apply the operator. + */ + private function matchesGroup(array $record, array $conditions): bool { + foreach ($conditions as [$field, $operator, $operand]) { + if ($operator === null) { + if (!array_key_exists($field, $record) || $record[$field] !== $operand) { + return false; + } + } elseif (!$this->matchesOperator($record, $field, $operator, $operand)) { + return false; } } + return true; + } - // search filters - foreach ($this->searchFilters as $search) { - $term = strtolower($search['term']); - if ($term === '') continue; - $fields = $search['fields']; - $found = false; - $searchFields = empty($fields) ? array_keys($record) : $fields; - foreach ($searchFields as $field) { - if (!isset($record[$field])) continue; - $value = $record[$field]; - if (is_array($value) || is_object($value)) continue; - if (strpos(strtolower((string)$value), $term) !== false) { - $found = true; + /** + * where() groups AND-ed, OR any orWhere() group. Without where(), only orWhere() groups decide. + * @param array[] $where compiled where() groups + * @param array[] $orWhere compiled orWhere() groups + */ + private function matchesWhereLogic(array $record, array $where, array $orWhere): bool { + if (!empty($where)) { + $all = true; + foreach ($where as $conditions) { + if (!$this->matchesGroup($record, $conditions)) { + $all = false; break; } } - if (!$found) return false; + if ($all) { + return true; + } + } + foreach ($orWhere as $conditions) { + if ($this->matchesGroup($record, $conditions)) { + return true; + } } + return false; + } - return true; + /** + * Compare two values for ordering operators ($gt, between, ...). + * Numbers (and numeric strings) compare numerically, other strings lexically; anything else + * (null, booleans, arrays, number vs. non-numeric string) is not comparable. + * @internal also used by noneDB::between(); not part of the public API + * @return int|null + */ + public static function compareValues($a, $b): ?int { + if ($a === null || $b === null || is_bool($a) || is_bool($b) || is_array($a) || is_array($b) || is_object($a) || is_object($b)) { + return null; + } + $numA = is_int($a) || is_float($a) || (is_string($a) && is_numeric($a)); + $numB = is_int($b) || is_float($b) || (is_string($b) && is_numeric($b)); + if ($numA && $numB) { + return ($a + 0) <=> ($b + 0); + } + if (is_string($a) && is_string($b)) { + return strcmp($a, $b) <=> 0; + } + return null; } /** - * Check if we have any advanced filters that need single-pass processing - * @return bool + * Regex for a like()/$like pattern ("^" start anchor, trailing "$" end anchor, rest literal) */ - private function hasAdvancedFilters(): bool { - return count($this->whereNotFilters) > 0 || - count($this->whereInFilters) > 0 || - count($this->whereNotInFilters) > 0 || - count($this->likeFilters) > 0 || - count($this->notLikeFilters) > 0 || - count($this->betweenFilters) > 0 || - count($this->notBetweenFilters) > 0 || - count($this->searchFilters) > 0; + private function likeRegex($pattern): string { + if (!is_scalar($pattern)) { + throw new \InvalidArgumentException('noneDB: like pattern must be a string'); + } + return $this->callPrivateMethod('likePatternToRegex', (string)$pattern); } /** - * Check if WHERE filters contain any operator-based comparisons - * @return bool + * Run preg_match, turning compile errors and backtrack-limit failures into exceptions */ - private function hasOperatorFilters(): bool { - foreach ($this->whereFilters as $value) { - if (is_array($value)) { - foreach (array_keys($value) as $key) { - if (is_string($key) && strpos($key, '$') === 0) { - return true; - } - } + private static function regexMatch(string $regex, string $subject): bool { + $result = @preg_match($regex, $subject); + if ($result === false) { + $error = preg_last_error(); + if ($error === PREG_BAD_UTF8_ERROR || $error === PREG_BAD_UTF8_OFFSET_ERROR) { + return false; // subject isn't UTF-8: can't match a UTF-8 pattern + } + if ($error === PREG_NO_ERROR) { + throw new \InvalidArgumentException('noneDB: invalid regular expression'); } + throw new \RuntimeException('noneDB: regular expression failed (' . (function_exists('preg_last_error_msg') ? preg_last_error_msg() : 'error ' . $error) . ')'); } - return false; + return $result === 1; } /** - * Check if a record matches WHERE filters with comparison operators - * Supports: $gt, $gte, $lt, $lte, $ne, $like, $in, $nin, $exists, $regex - * @param array $record - * @param array $filters WHERE filters to apply - * @return bool + * Regex for a $regex operand: the pattern body, case-insensitive (delimiter added here) */ - private function matchesWhereFilters(array $record, array $filters): bool { - foreach ($filters as $field => $value) { - if (!is_array($value)) { - // Simple equality: ['field' => 'value'] - if (!array_key_exists($field, $record) || $record[$field] !== $value) { - return false; - } - } else { - // Operator-based comparison: ['field' => ['$gt' => 10]] - foreach ($value as $operator => $operand) { - if (!$this->matchesOperator($record, $field, $operator, $operand)) { - return false; - } - } - } + private static function regexOperand($pattern): string { + if (!is_string($pattern) || $pattern === '') { + throw new \InvalidArgumentException('noneDB: $regex needs a non-empty string pattern'); } - return true; + $body = preg_replace('/(? $operand; - + $c = $hasField ? self::compareValues($fieldValue, $operand) : null; + return $c !== null && $c > 0; case '$gte': - return $hasField && $fieldValue >= $operand; - + $c = $hasField ? self::compareValues($fieldValue, $operand) : null; + return $c !== null && $c >= 0; case '$lt': - return $hasField && $fieldValue < $operand; - + $c = $hasField ? self::compareValues($fieldValue, $operand) : null; + return $c !== null && $c < 0; case '$lte': - return $hasField && $fieldValue <= $operand; - + $c = $hasField ? self::compareValues($fieldValue, $operand) : null; + return $c !== null && $c <= 0; case '$ne': return !$hasField || $fieldValue !== $operand; - case '$eq': return $hasField && $fieldValue === $operand; - case '$in': - return $hasField && is_array($operand) && in_array($fieldValue, $operand, true); - + return $hasField && in_array($fieldValue, $operand, true); case '$nin': - return !$hasField || !is_array($operand) || !in_array($fieldValue, $operand, true); - + return !$hasField || !in_array($fieldValue, $operand, true); case '$exists': return $operand ? $hasField : !$hasField; - case '$like': - if (!$hasField || is_array($fieldValue) || is_object($fieldValue)) return false; - $pattern = $operand; - if (strpos($pattern, '^') === 0 || substr($pattern, -1) === '$') { - $regex = '/' . $pattern . '/i'; - } else { - $regex = '/' . preg_quote($pattern, '/') . '/i'; + if (!$hasField || !is_scalar($fieldValue) || is_bool($fieldValue)) { + return false; } - return preg_match($regex, (string)$fieldValue) === 1; - + return self::regexMatch($operand, (string)$fieldValue); case '$regex': - if (!$hasField || is_array($fieldValue) || is_object($fieldValue)) return false; - return preg_match('/' . $operand . '/i', (string)$fieldValue) === 1; - + if (!$hasField || !is_scalar($fieldValue) || is_bool($fieldValue)) { + return false; + } + return self::regexMatch($operand, (string)$fieldValue); case '$contains': - if (!$hasField) return false; + if (!$hasField) { + return false; + } if (is_array($fieldValue)) { return in_array($operand, $fieldValue, true); } - if (is_string($fieldValue)) { - return strpos($fieldValue, $operand) !== false; + if (is_string($fieldValue) && is_scalar($operand) && !is_bool($operand)) { + return (string)$operand === '' || strpos($fieldValue, (string)$operand) !== false; } return false; - default: - // Unknown operator - treat as nested field comparison - // e.g., ['address' => ['city' => 'Istanbul']] - if ($hasField && is_array($fieldValue) && array_key_exists($operator, $fieldValue)) { - return $fieldValue[$operator] === $operand; + throw new \InvalidArgumentException("noneDB: unknown operator '{$operator}' (supported: " . implode(', ', self::OPERATORS) . ')'); + } + } + + /** + * Check if a record matches all the other filters (whereNot, whereIn, whereNotIn, like, notLike, + * between, notBetween, search) + */ + private function matchesAdvancedFilters(array $record): bool { + foreach ($this->whereNotFilters as $group) { + foreach ($group as $field => $value) { + if (array_key_exists($field, $record) && $record[$field] === $value) { + return false; } + } + } + + foreach ($this->whereInFilters as $filter) { + if (!array_key_exists($filter['field'], $record)) return false; + if (!in_array($record[$filter['field']], $filter['values'], true)) return false; + } + + foreach ($this->whereNotInFilters as $filter) { + if (array_key_exists($filter['field'], $record) && in_array($record[$filter['field']], $filter['values'], true)) { + return false; + } + } + + foreach ($this->likeFilters as $like) { + $value = $record[$like['field']] ?? null; + if (!is_scalar($value) || is_bool($value)) return false; + if (!self::regexMatch($like['regex'], (string)$value)) return false; + } + + foreach ($this->notLikeFilters as $notLike) { + $value = $record[$notLike['field']] ?? null; + if (is_scalar($value) && !is_bool($value) && self::regexMatch($notLike['regex'], (string)$value)) { return false; + } + } + + foreach ($this->betweenFilters as $between) { + if (!array_key_exists($between['field'], $record)) return false; + $value = $record[$between['field']]; + $lo = self::compareValues($value, $between['min']); + $hi = self::compareValues($value, $between['max']); + if ($lo === null || $hi === null || $lo < 0 || $hi > 0) return false; + } + + foreach ($this->notBetweenFilters as $notBetween) { + if (!array_key_exists($notBetween['field'], $record)) continue; + $value = $record[$notBetween['field']]; + $lo = self::compareValues($value, $notBetween['min']); + $hi = self::compareValues($value, $notBetween['max']); + if ($lo !== null && $hi !== null && $lo >= 0 && $hi <= 0) return false; + } + + foreach ($this->searchFilters as $search) { + $term = (string)$search['term']; + if ($term === '') continue; + $searchFields = empty($search['fields']) ? array_keys($record) : $search['fields']; + $found = false; + foreach ($searchFields as $field) { + if ($field === 'key' && empty($search['fields'])) continue; // the record key is not content + $value = $record[$field] ?? null; + if (!is_scalar($value) || is_bool($value)) continue; + if (self::containsIgnoreCase((string)$value, $term)) { + $found = true; + break; + } + } + if (!$found) return false; + } + + return true; + } + + /** + * Case-insensitive substring test, multibyte aware when mbstring is available + */ + private static function containsIgnoreCase(string $haystack, string $needle): bool { + if (function_exists('mb_strtolower') && preg_match('//u', $haystack) === 1 && preg_match('//u', $needle) === 1) { + return strpos(self::foldCase($haystack), self::foldCase($needle)) !== false; } + return stripos($haystack, $needle) !== false; + } + + /** + * Lower-case for matching. Unicode lower-cases "İ" to "i" + combining dot (U+0307); the dot is + * dropped so "İzmir", "IZMIR" and "izmir" all match each other. + */ + private static function foldCase(string $s): string { + return str_replace("\u{0307}", '', mb_strtolower($s, 'UTF-8')); + } + + private function hasAdvancedFilters(): bool { + return !empty($this->whereNotFilters) || !empty($this->whereInFilters) || !empty($this->whereNotInFilters) + || !empty($this->likeFilters) || !empty($this->notLikeFilters) || !empty($this->betweenFilters) + || !empty($this->notBetweenFilters) || !empty($this->searchFilters); } // ========================================== @@ -7964,23 +7171,29 @@ private function matchesOperator(array $record, string $field, string $operator, // ========================================== /** - * Add where filter (AND condition) + * Add where filters. Each call adds a group; all groups must match (AND). + * Values are compared with === or, for arrays like ['$gt' => 10], with operators. * @param array $filters * @return self */ public function where(array $filters): self { - $this->whereFilters = array_merge($this->whereFilters, $filters); + if (!empty($filters)) { + foreach ($filters as $value) { + $this->isOperatorSpec($value); // validates early + } + $this->whereFilters[] = $filters; + } return $this; } /** * Add pattern matching filter * @param string $field - * @param string $pattern Use ^ for starts with, $ for ends with + * @param string $pattern "^" anchors the start, a trailing "$" the end; everything else is literal * @return self */ public function like(string $field, string $pattern): self { - $this->likeFilters[] = ['field' => $field, 'pattern' => $pattern]; + $this->likeFilters[] = ['field' => $field, 'pattern' => $pattern, 'regex' => $this->likeRegex($pattern)]; return $this; } @@ -7997,14 +7210,13 @@ public function between(string $field, $min, $max): self { } /** - * Set sort order + * Add a sort key. The first call is the primary key; later calls break ties. * @param string $field * @param string $order 'asc' or 'desc' * @return self */ public function sort(string $field, string $order = 'asc'): self { - $this->sortField = $field; - $this->sortOrder = $order; + $this->sortKeys[] = ['field' => $field, 'order' => strtolower($order) === 'desc' ? 'desc' : 'asc']; return $this; } @@ -8014,7 +7226,7 @@ public function sort(string $field, string $order = 'asc'): self { * @return self */ public function limit(int $count): self { - $this->limitCount = $count; + $this->limitCount = max(0, $count); return $this; } @@ -8024,7 +7236,7 @@ public function limit(int $count): self { * @return self */ public function offset(int $count): self { - $this->offsetCount = $count; + $this->offsetCount = max(0, $count); return $this; } @@ -8038,7 +7250,7 @@ public function skip(int $count): self { } /** - * Alias for sort() - set sort order + * Alias for sort() * @param string $field * @param string $order 'asc' or 'desc' * @return self @@ -8048,44 +7260,49 @@ public function orderBy(string $field, string $order = 'asc'): self { } /** - * Add OR where filter + * Add an OR group: a record matches if all where() groups match OR any orWhere() group does * @param array $filters * @return self */ public function orWhere(array $filters): self { + foreach ($filters as $value) { + $this->isOperatorSpec($value); + } $this->orWhereFilters[] = $filters; return $this; } /** - * Filter where field value is in given array + * Filter where field value is in given array (strict comparison) * @param string $field * @param array $values * @return self */ public function whereIn(string $field, array $values): self { - $this->whereInFilters[] = ['field' => $field, 'values' => $values]; + $this->whereInFilters[] = ['field' => $field, 'values' => array_values($values)]; return $this; } /** - * Filter where field value is NOT in given array + * Filter where field value is NOT in given array (strict comparison) * @param string $field * @param array $values * @return self */ public function whereNotIn(string $field, array $values): self { - $this->whereNotInFilters[] = ['field' => $field, 'values' => $values]; + $this->whereNotInFilters[] = ['field' => $field, 'values' => array_values($values)]; return $this; } /** - * Filter where field value does NOT match + * Exclude records where a field equals the value (strict comparison) * @param array $filters * @return self */ public function whereNot(array $filters): self { - $this->whereNotFilters = array_merge($this->whereNotFilters, $filters); + if (!empty($filters)) { + $this->whereNotFilters[] = $filters; + } return $this; } @@ -8096,7 +7313,7 @@ public function whereNot(array $filters): self { * @return self */ public function notLike(string $field, string $pattern): self { - $this->notLikeFilters[] = ['field' => $field, 'pattern' => $pattern]; + $this->notLikeFilters[] = ['field' => $field, 'pattern' => $pattern, 'regex' => $this->likeRegex($pattern)]; return $this; } @@ -8113,12 +7330,12 @@ public function notBetween(string $field, $min, $max): self { } /** - * Select only specific fields in results + * Select only specific fields in results ("key" is always included) * @param array $fields * @return self */ public function select(array $fields): self { - $this->selectFields = $fields; + $this->selectFields = array_values($fields); return $this; } @@ -8128,12 +7345,12 @@ public function select(array $fields): self { * @return self */ public function except(array $fields): self { - $this->exceptFields = $fields; + $this->exceptFields = array_values($fields); return $this; } /** - * Group results by field + * Group results by field. get() then returns groups: ['_group' => value, '_items' => [...], '_count' => n] * @param string $field * @return self */ @@ -8143,29 +7360,26 @@ public function groupBy(string $field): self { } /** - * Filter groups by aggregate condition + * Filter groups by aggregate condition. Only used together with groupBy(); without it, and for + * unknown aggregates or operators, the condition is ignored (passes) * @param string $aggregate 'count', 'sum:field', 'avg:field', 'min:field', 'max:field' * @param string $operator '>', '<', '>=', '<=', '=', '!=' * @param mixed $value * @return self */ public function having(string $aggregate, string $operator, $value): self { - $this->havingFilters[] = [ - 'aggregate' => $aggregate, - 'operator' => $operator, - 'value' => $value - ]; + $this->havingFilters[] = ['aggregate' => $aggregate, 'operator' => $operator, 'value' => $value]; return $this; } /** - * Full-text search across multiple fields + * Case-insensitive full-text search across fields * @param string $term Search term - * @param array $fields Fields to search in (empty = all string fields) + * @param array $fields Fields to search in (empty = all fields except "key") * @return self */ public function search(string $term, array $fields = []): self { - $this->searchFilters[] = ['term' => $term, 'fields' => $fields]; + $this->searchFilters[] = ['term' => $term, 'fields' => array_values($fields)]; return $this; } @@ -8174,7 +7388,8 @@ public function search(string $term, array $fields = []): self { * @param string $foreignDb Foreign database name * @param string $localKey Local key field * @param string $foreignKey Foreign key field - * @param string|null $alias Alias for joined data (default: foreignDb name) + * @param string|null $alias Field that receives the joined record (default: the foreign database name; + * it replaces a field of the same name in the result, never in storage) * @return self */ public function join(string $foreignDb, string $localKey, string $foreignKey, ?string $alias = null): self { @@ -8188,7 +7403,7 @@ public function join(string $foreignDb, string $localKey, string $foreignKey, ?s } // ========================================== - // SPATIAL QUERY METHODS (v3.1.0) + // SPATIAL QUERY METHODS // ========================================== /** @@ -8200,13 +7415,7 @@ public function join(string $foreignDb, string $localKey, string $foreignKey, ?s * @return self */ public function withinDistance(string $field, float $lon, float $lat, float $distanceMeters): self { - $this->spatialFilters[] = [ - 'type' => 'withinDistance', - 'field' => $field, - 'lon' => $lon, - 'lat' => $lat, - 'distance' => $distanceMeters - ]; + $this->spatialFilters[] = ['type' => 'withinDistance', 'field' => $field, 'lon' => $lon, 'lat' => $lat, 'distance' => $distanceMeters]; return $this; } @@ -8220,20 +7429,12 @@ public function withinDistance(string $field, float $lon, float $lat, float $dis * @return self */ public function withinBBox(string $field, float $minLon, float $minLat, float $maxLon, float $maxLat): self { - $this->spatialFilters[] = [ - 'type' => 'withinBBox', - 'field' => $field, - 'minLon' => $minLon, - 'minLat' => $minLat, - 'maxLon' => $maxLon, - 'maxLat' => $maxLat - ]; + $this->spatialFilters[] = ['type' => 'withinBBox', 'field' => $field, 'minLon' => $minLon, 'minLat' => $minLat, 'maxLon' => $maxLon, 'maxLat' => $maxLat]; return $this; } /** - * Find K nearest records to a point - * Returns records with _distance in meters + * The K records nearest to a point among those matching the other filters (adds _distance in meters) * @param string $field GeoJSON field name * @param float $lon Center longitude * @param float $lat Center latitude @@ -8241,13 +7442,7 @@ public function withinBBox(string $field, float $minLon, float $minLat, float $m * @return self */ public function nearest(string $field, float $lon, float $lat, int $k): self { - $this->spatialFilters[] = [ - 'type' => 'nearest', - 'field' => $field, - 'lon' => $lon, - 'lat' => $lat, - 'k' => $k - ]; + $this->spatialFilters[] = ['type' => 'nearest', 'field' => $field, 'lon' => $lon, 'lat' => $lat, 'k' => max(0, $k)]; return $this; } @@ -8258,455 +7453,410 @@ public function nearest(string $field, float $lon, float $lat, int $k): self { * @return self */ public function withinPolygon(string $field, array $polygon): self { - $this->spatialFilters[] = [ - 'type' => 'withinPolygon', - 'field' => $field, - 'polygon' => $polygon - ]; + $this->spatialFilters[] = ['type' => 'withinPolygon', 'field' => $field, 'polygon' => $polygon]; return $this; } /** - * Include distance in results (adds _distance field) + * Add _distance to every result: meters from the point to the geometry, measured like + * withinDistance()/nearest() (0 inside a polygon, otherwise to the nearest point of the geometry). + * It is computed before sorting, so sort('_distance') works. * @param string $field GeoJSON field name * @param float $lon Center longitude * @param float $lat Center latitude * @return self */ public function withDistance(string $field, float $lon, float $lat): self { - $this->distanceConfig = [ - 'field' => $field, - 'lon' => $lon, - 'lat' => $lat - ]; + $this->distanceConfig = ['field' => $field, 'lon' => $lon, 'lat' => $lat]; return $this; } - /** - * Check if query has spatial filters - * @return bool - */ - private function hasSpatialFilters(): bool { - return !empty($this->spatialFilters); - } + // ========================================== + // EXECUTION + // ========================================== /** - * Apply spatial filters to get candidate records - * @return array|null Records from spatial filter or null if no spatial filters + * Records the spatial filters allow ([key => record]); nearest() returns its candidates + * in distance order with _distance set. Null when there are no spatial filters. + * @param float|null $nearestRadius Only the records within this many meters for a nearest() filter + * (null: all records in distance order) + * @param float|null $nearestBoundary Set when the radius was used: records with a smaller _distance + * are exactly those of the full list (anything outside the radius is at least this far) */ - private function applySpatialFilters(): ?array { + private function spatialCandidates(?float $nearestRadius = null, ?float &$nearestBoundary = null): ?array { + $nearestBoundary = null; if (empty($this->spatialFilters)) { return null; } - - // Process each spatial filter and intersect results - $results = null; - + $result = null; foreach ($this->spatialFilters as $filter) { - $filterResults = []; - switch ($filter['type']) { case 'withinDistance': - $filterResults = $this->db->withinDistance( - $this->dbname, - $filter['field'], - $filter['lon'], - $filter['lat'], - $filter['distance'], - ['includeDistance' => true] - ); + $rows = $this->db->withinDistance($this->dbname, $filter['field'], $filter['lon'], $filter['lat'], $filter['distance'], ['includeDistance' => true]); break; - case 'withinBBox': - $filterResults = $this->db->withinBBox( - $this->dbname, - $filter['field'], - $filter['minLon'], - $filter['minLat'], - $filter['maxLon'], - $filter['maxLat'] - ); + $rows = $this->db->withinBBox($this->dbname, $filter['field'], $filter['minLon'], $filter['minLat'], $filter['maxLon'], $filter['maxLat']); break; - case 'nearest': - $filterResults = $this->db->nearest( - $this->dbname, - $filter['field'], - $filter['lon'], - $filter['lat'], - $filter['k'], - ['includeDistance' => true] - ); + // Records in distance order; the k cut happens after the other filters + if ($filter['k'] <= 0) { + $rows = []; + } elseif ($nearestRadius !== null) { + $rows = $this->db->withinDistance($this->dbname, $filter['field'], $filter['lon'], $filter['lat'], $nearestRadius, ['includeDistance' => true]); + $nearestBoundary = round($nearestRadius, 2); + } else { + $all = max(1, $this->db->count($this->dbname)); + $rows = $this->db->nearest($this->dbname, $filter['field'], $filter['lon'], $filter['lat'], $all, ['includeDistance' => true]); + } + break; + case 'withinPolygon': + $rows = $this->db->withinPolygon($this->dbname, $filter['field'], $filter['polygon']); + break; + default: + $rows = []; + } + $keyed = []; + foreach ($rows as $row) { + if (isset($row['key'])) { + $keyed[$row['key']] = $row; + } + } + if ($result === null) { + $result = $keyed; + } else { + // Intersect by key, keep the order and fields of the earlier result + $result = array_intersect_key($result, $keyed); + foreach ($result as $key => $row) { + if (isset($keyed[$key]['_distance']) && !isset($row['_distance'])) { + $result[$key]['_distance'] = $keyed[$key]['_distance']; + } + } + } + } + return $result ?? []; + } + + /** + * Base records: pushes a single plain where() group down to find() so field indexes and key + * lookups are used; otherwise all records. + * @return array [records, bool whereAlreadyApplied] + */ + private function baseRecords(): array { + if (count($this->whereFilters) === 1 && empty($this->orWhereFilters)) { + $group = $this->whereFilters[0]; + $plain = true; + foreach ($group as $value) { + if ($this->isOperatorSpec($value)) { + $plain = false; break; + } + } + if ($plain) { + $records = $this->db->find($this->dbname, $group); + return [$records === false ? [] : $records, true]; + } + } + $records = $this->db->find($this->dbname, 0); + return [$records === false ? [] : $records, false]; + } + + /** + * Filtered records (no join, grouping or projection) + * @param bool $earlyExit Stop once offset+limit records are collected (only when order can't change) + */ + private function filteredRecords(bool $earlyExit): array { + // One shared lock for every read the query makes (count, spatial rounds, records), so they + // all see the same state of the database + $dbname = $this->callPrivateMethod('normalizeDbName', $this->dbname); + return $this->callPrivateMethod('withReadLock', $dbname, function () use ($earlyExit) { + return $this->filteredRecordsLocked($earlyExit); + }); + } + + private function filteredRecordsLocked(bool $earlyExit): array { + $nearest = null; + $nearestFilters = 0; + foreach ($this->spatialFilters as $filter) { + if ($filter['type'] === 'nearest') { + $nearest = $nearest === null ? $filter['k'] : min($nearest, $filter['k']); + $nearestFilters++; + } + } + // A single nearest() as the first spatial filter (its distances are the ones kept) on an indexed + // field doesn't need every record in distance order: search a growing radius until k records match + $nearestRadius = null; + $total = 0; + if ($nearestFilters === 1 && $this->spatialFilters[0]['type'] === 'nearest' && $nearest > 0 + && $this->callPrivateMethod('usesSpatialIndex', $this->dbname, $this->spatialFilters[0]['field'])) { + $nearestRadius = 1000.0; + $total = $this->db->count($this->dbname); + } + + $needWhere = !empty($this->whereFilters) || !empty($this->orWhereFilters); + $where = $needWhere ? $this->compileGroups($this->whereFilters) : []; + $orWhere = $needWhere ? $this->compileGroups($this->orWhereFilters) : []; + $needAdvanced = $this->hasAdvancedFilters(); + $target = ($earlyExit && $nearest === null && empty($this->sortKeys) && $this->limitCount !== null) + ? $this->offsetCount + $this->limitCount + : PHP_INT_MAX; + // The usual case, one where() group: no need to go through the AND/OR logic per record + $single = ($needWhere && count($where) === 1 && empty($orWhere)) ? $where[0] : null; + + while (true) { + $boundary = null; + $spatial = $this->spatialCandidates($nearestRadius, $boundary); + if ($spatial !== null) { + $records = array_values($spatial); + $whereApplied = false; + } else { + [$records, $whereApplied] = $this->baseRecords(); + } - case 'withinPolygon': - $filterResults = $this->db->withinPolygon( - $this->dbname, - $filter['field'], - $filter['polygon'] - ); + $filtered = []; + foreach ($records as $record) { + if (!$whereApplied) { + if ($single !== null) { + if (!$this->matchesGroup($record, $single)) continue; + } elseif ($needWhere && !$this->matchesWhereLogic($record, $where, $orWhere)) continue; + } + if ($needAdvanced && !$this->matchesAdvancedFilters($record)) continue; + $filtered[] = $record; + if (count($filtered) >= $target) break; + } + + if ($boundary !== null) { + // Candidates closer than the radius are exactly the full list's (at the radius itself + // rounding may leave some out). Enough matches among them: done. + $certain = []; + foreach ($filtered as $record) { + if (($record['_distance'] ?? INF) < $boundary) { + $certain[] = $record; + } + } + if (count($certain) >= $nearest) { + $filtered = $certain; break; + } + if (count($records) >= $total) { + break; // every record lies within the radius: the list is complete + } + // Matches grow with the area (r²): aim for k from the matches so far, at least doubling + $factor = count($certain) > 0 ? min(16.0, max(2.0, sqrt($nearest / count($certain)) * 1.25)) : 4.0; + // Half the earth's circumference covers everything: then take the complete list + $nearestRadius = $nearestRadius * $factor >= 20040000.0 ? null : $nearestRadius * $factor; + continue; } + break; + } - // Intersect with previous results - if ($results === null) { - $results = $filterResults; - } else { - // Intersect by key - $resultKeys = array_column($results, 'key'); - $results = array_filter($filterResults, fn($r) => in_array($r['key'], $resultKeys)); - $results = array_values($results); + if ($nearest !== null) { + // Stable on every PHP version: equal distances keep the candidate order + $positions = array_keys($filtered); + $distances = array_map(function ($r) { return $r['_distance'] ?? INF; }, $filtered); + array_multisort($distances, SORT_ASC, SORT_NUMERIC, $positions, SORT_ASC, SORT_NUMERIC); + $sorted = []; + foreach (array_slice($positions, 0, $nearest) as $position) { + $sorted[] = $filtered[$position]; } + $filtered = $sorted; } - - return $results ?? []; + return $filtered; } /** - * Add distance to records if distanceConfig is set - * @param array $records - * @return array + * Add _distance for withDistance() */ private function applyDistanceCalculation(array $records): array { if ($this->distanceConfig === null) { return $records; } - $field = $this->distanceConfig['field']; - $lon = $this->distanceConfig['lon']; - $lat = $this->distanceConfig['lat']; - foreach ($records as &$record) { - if (!isset($record[$field]) || isset($record['_distance'])) { - continue; - } - - $geometry = $record[$field]; - $validation = $this->db->validateGeoJSON($geometry); - if (!$validation['valid']) { + $geometry = $record[$field] ?? null; + if (!is_array($geometry) || !$this->db->validateGeoJSON($geometry)['valid']) { continue; } - - // Calculate distance to centroid - $centroid = $this->getRecordCentroid($geometry); - $record['_distance'] = round( - $this->db->haversineDistance($lon, $lat, $centroid[0], $centroid[1]), - 4 - ); + $record['_distance'] = round($this->callPrivateMethod('distanceToGeometry', + (float)$this->distanceConfig['lon'], (float)$this->distanceConfig['lat'], $geometry), 4); } - + unset($record); return $records; } - /** - * Get centroid of a geometry (helper for query builder) - */ - private function getRecordCentroid(array $geometry): array { - $type = $geometry['type'] ?? ''; - $coords = $geometry['coordinates'] ?? []; - - switch ($type) { - case 'Point': - return [$coords[0], $coords[1]]; - case 'Polygon': - $ring = $coords[0]; - $sumLon = $sumLat = 0; - $n = count($ring) - 1; - for ($i = 0; $i < $n; $i++) { - $sumLon += $ring[$i][0]; - $sumLat += $ring[$i][1]; - } - return [$sumLon / $n, $sumLat / $n]; - default: - // Use MBR center - $mbr = $this->calculateSimpleMBR($geometry); - return [($mbr[0] + $mbr[2]) / 2, ($mbr[1] + $mbr[3]) / 2]; - } - } - - /** - * Simple MBR calculation for query builder - */ - private function calculateSimpleMBR(array $geometry): array { - $type = $geometry['type'] ?? ''; - $coords = $geometry['coordinates'] ?? []; - - switch ($type) { - case 'Point': - return [$coords[0], $coords[1], $coords[0], $coords[1]]; - case 'LineString': - case 'MultiPoint': - $minLon = $maxLon = $coords[0][0]; - $minLat = $maxLat = $coords[0][1]; - foreach ($coords as $p) { - $minLon = min($minLon, $p[0]); - $maxLon = max($maxLon, $p[0]); - $minLat = min($minLat, $p[1]); - $maxLat = max($maxLat, $p[1]); - } - return [$minLon, $minLat, $maxLon, $maxLat]; - case 'Polygon': - $ring = $coords[0]; - $minLon = $maxLon = $ring[0][0]; - $minLat = $maxLat = $ring[0][1]; - foreach ($ring as $p) { - $minLon = min($minLon, $p[0]); - $maxLon = max($maxLon, $p[0]); - $minLat = min($minLat, $p[1]); - $maxLat = max($maxLat, $p[1]); + private function applyJoins(array $records): array { + foreach ($this->joinConfigs as $join) { + $foreignData = $this->db->find($join['foreignDb'], 0); + if ($foreignData === false) { + $foreignData = []; + } + $foreignIndexed = []; + foreach ($foreignData as $fRecord) { + $fk = $fRecord[$join['foreignKey']] ?? null; + if (is_scalar($fk)) { + $foreignIndexed[$fk] = $fRecord; } - return [$minLon, $minLat, $maxLon, $maxLat]; - default: - return [0, 0, 0, 0]; + } + foreach ($records as &$record) { + $local = $record[$join['localKey']] ?? null; + $record[$join['alias']] = is_scalar($local) ? ($foreignIndexed[$local] ?? null) : null; + } + unset($record); } + return $records; } - // ========================================== - // TERMINAL METHODS (execute and return result) - // ========================================== - - /** - * Execute query and get all results - * @return array - */ - public function get(): array { - // 0. Apply spatial filters first if present (v3.1.0) - if ($this->hasSpatialFilters()) { - $results = $this->applySpatialFilters(); - if ($results === null) { - $results = []; - } - - // Apply WHERE filters to spatial results (with operator support) - if (count($this->whereFilters) > 0) { - $results = array_filter($results, function($record) { - return $this->matchesWhereFilters($record, $this->whereFilters); - }); - $results = array_values($results); + private function applyGroupBy(array $records): array { + $groups = []; + foreach ($records as $record) { + $value = $record[$this->groupByField] ?? null; + $groupKey = json_encode($value, JSON_PRESERVE_ZERO_FRACTION | JSON_PARTIAL_OUTPUT_ON_ERROR); + if (!isset($groups[$groupKey])) { + $groups[$groupKey] = ['_group' => $value, '_items' => [], '_count' => 0]; } + $groups[$groupKey]['_items'][] = $record; + $groups[$groupKey]['_count']++; + } - // Apply OR WHERE filters (with operator support) - if (count($this->orWhereFilters) > 0) { - $results = array_filter($results, function($record) { - foreach ($this->orWhereFilters as $orFilter) { - if ($this->matchesWhereFilters($record, $orFilter)) { - return true; + foreach ($this->havingFilters as $having) { + $groups = array_filter($groups, function ($group) use ($having) { + if ($having['aggregate'] === 'count') { + $aggValue = $group['_count']; + } elseif (!preg_match('/^(sum|avg|min|max):.+$/', $having['aggregate'])) { + return true; // unknown aggregate: ignored + } else { + [$aggType, $aggField] = explode(':', $having['aggregate'], 2); + $values = []; + foreach ($group['_items'] as $item) { + if (isset($item[$aggField]) && is_numeric($item[$aggField]) && !is_bool($item[$aggField])) { + $values[] = $item[$aggField] + 0; } } + switch ($aggType) { + case 'sum': $aggValue = array_sum($values); break; + case 'avg': $aggValue = count($values) > 0 ? array_sum($values) / count($values) : 0; break; + case 'min': $aggValue = count($values) > 0 ? min($values) : null; break; + default: $aggValue = count($values) > 0 ? max($values) : null; break; + } + } + if ($aggValue === null) { return false; - }); - $results = array_values($results); - } - - // Continue with advanced filters, joins, etc. - goto applyAdvancedFilters; - } - - // 1. Base query - get all records first if we have OR conditions or operator filters - $hasOperatorFilters = $this->hasOperatorFilters(); - - if (count($this->orWhereFilters) > 0 || $hasOperatorFilters) { - // With OR conditions or operators, we need all records first - $results = $this->db->find($this->dbname, 0); - if ($results === false) return []; - - // Apply WHERE + OR WHERE logic (with operator support) - $results = array_filter($results, function($record) { - // Check main WHERE filters (AND logic) - $matchesWhere = $this->matchesWhereFilters($record, $this->whereFilters); - - // If no OR filters, just return WHERE result - if (count($this->orWhereFilters) === 0) { - return $matchesWhere; } - - // Check OR WHERE filters - $matchesOrWhere = false; - foreach ($this->orWhereFilters as $orFilter) { - if ($this->matchesWhereFilters($record, $orFilter)) { - $matchesOrWhere = true; - break; - } + switch ($having['operator']) { + case '>': return $aggValue > $having['value']; + case '<': return $aggValue < $having['value']; + case '>=': return $aggValue >= $having['value']; + case '<=': return $aggValue <= $having['value']; + case '=': return $aggValue == $having['value']; + case '!=': return $aggValue != $having['value']; + default: return true; // unknown operator: ignored } - - // Record matches if (WHERE conditions match) OR (any OR WHERE matches) - return $matchesWhere || $matchesOrWhere; }); - $results = array_values($results); - } else { - // No OR conditions and no operators, use standard WHERE (direct pass to find) - $filters = count($this->whereFilters) > 0 ? $this->whereFilters : 0; - $results = $this->db->find($this->dbname, $filters); - if ($results === false) return []; } + return array_values($groups); + } - applyAdvancedFilters: - // 2-9. Apply all advanced filters in single pass (v3.0.0 optimization) - // Replaces multiple array_filter calls with one pass for better performance - if ($this->hasAdvancedFilters()) { - $filtered = []; - // Early exit optimization: when no join/groupBy/sort, we can stop at limit+offset - $canEarlyExit = empty($this->joinConfigs) && - $this->groupByField === null && - $this->sortField === null && - $this->limitCount !== null; - $earlyExitTarget = $canEarlyExit ? ($this->limitCount + $this->offsetCount) : PHP_INT_MAX; - - foreach ($results as $record) { - if ($this->matchesAdvancedFilters($record)) { - $filtered[] = $record; - // Early exit when we have enough records - if (count($filtered) >= $earlyExitTarget) { - break; - } - } - } - $results = $filtered; + private function applySort(array $records): array { + if (empty($this->sortKeys) || count($records) < 2) { + return $records; } - - // 10. Apply joins - foreach ($this->joinConfigs as $join) { - $foreignData = $this->db->find($join['foreignDb'], 0); - if ($foreignData === false) continue; - - $foreignIndexed = []; - foreach ($foreignData as $fRecord) { - if (isset($fRecord[$join['foreignKey']])) { - $key = $fRecord[$join['foreignKey']]; - // Skip array/object keys - they can't be used as array indices - if (is_array($key) || is_object($key)) continue; - $foreignIndexed[$key] = $fRecord; - } + // Every sort field becomes two columns, kind and value (noneDB::sortKey), sorted by + // array_multisort with the same comparison as <=>. The original position is the last column, + // so records that compare equal keep their order (a stable sort on every PHP version). + $records = array_values($records); + $sortKey = $this->callPrivateMethod('sortKeyFunction'); + $columns = []; + foreach ($this->sortKeys as $sk) { + $kinds = []; + $values = []; + foreach ($records as $record) { + [$kinds[], $values[]] = $sortKey(is_array($record) && array_key_exists($sk['field'], $record) ? $record[$sk['field']] : null); } + $order = $sk['order'] === 'desc' ? SORT_DESC : SORT_ASC; + array_push($columns, $kinds, $order, SORT_REGULAR, $values, $order, SORT_REGULAR); + } + $columns[] = array_keys($records); + array_multisort(...$columns); + $sorted = []; + foreach ($columns[count($columns) - 1] as $position) { + $sorted[] = $records[$position]; + } + return $sorted; + } - foreach ($results as &$record) { - if (isset($record[$join['localKey']])) { - $localValue = $record[$join['localKey']]; - // Skip array/object values - they can't be used as array indices - if (is_array($localValue) || is_object($localValue)) { - $record[$join['alias']] = null; - } else { - $record[$join['alias']] = $foreignIndexed[$localValue] ?? null; + private function applyProjection(array $records): array { + if (empty($this->selectFields) && empty($this->exceptFields)) { + return $records; + } + return array_map(function ($record) { + if (!empty($this->selectFields)) { + $filtered = []; + foreach ($this->selectFields as $field) { + if (array_key_exists($field, $record)) { + $filtered[$field] = $record[$field]; } - } else { - $record[$join['alias']] = null; } - } - unset($record); - } - - // 11. Apply groupBy - if ($this->groupByField !== null) { - $groups = []; - foreach ($results as $record) { - $groupKey = isset($record[$this->groupByField]) ? $record[$this->groupByField] : '__null__'; - if (!isset($groups[$groupKey])) { - $groups[$groupKey] = [ - '_group' => $groupKey === '__null__' ? null : $groupKey, - '_items' => [], - '_count' => 0 - ]; + if (isset($record['key']) && !array_key_exists('key', $filtered)) { + $filtered['key'] = $record['key']; } - $groups[$groupKey]['_items'][] = $record; - $groups[$groupKey]['_count']++; - } - - // Apply having filters - foreach ($this->havingFilters as $having) { - $groups = array_filter($groups, function($group) use ($having) { - $aggregate = $having['aggregate']; - $operator = $having['operator']; - $compareValue = $having['value']; - - // Calculate aggregate value - if ($aggregate === 'count') { - $aggValue = $group['_count']; - } elseif (strpos($aggregate, ':') !== false) { - [$aggType, $aggField] = explode(':', $aggregate, 2); - $values = array_filter( - array_column($group['_items'], $aggField), - 'is_numeric' - ); - switch ($aggType) { - case 'sum': - $aggValue = array_sum($values); - break; - case 'avg': - $aggValue = count($values) > 0 ? array_sum($values) / count($values) : 0; - break; - case 'min': - $aggValue = count($values) > 0 ? min($values) : null; - break; - case 'max': - $aggValue = count($values) > 0 ? max($values) : null; - break; - default: - return true; - } - } else { - return true; - } - - // Compare - switch ($operator) { - case '>': return $aggValue > $compareValue; - case '<': return $aggValue < $compareValue; - case '>=': return $aggValue >= $compareValue; - case '<=': return $aggValue <= $compareValue; - case '=': return $aggValue == $compareValue; - case '!=': return $aggValue != $compareValue; - default: return true; - } - }); + if ($this->distanceConfig !== null && isset($record['_distance']) && !array_key_exists('_distance', $filtered)) { + $filtered['_distance'] = $record['_distance']; + } + return $filtered; } - - $results = array_values($groups); - } - - // 12. Sort - if ($this->sortField !== null && count($results) > 0) { - $sorted = $this->db->sort($results, $this->sortField, $this->sortOrder); - if ($sorted !== false) { - $results = $sorted; + foreach ($this->exceptFields as $field) { + unset($record[$field]); } - } + return $record; + }, $records); + } - // 13. Offset + Limit + /** + * Execute query and get all results + * @return array + */ + public function get(): array { + $canEarlyExit = empty($this->joinConfigs) && $this->groupByField === null && empty($this->sortKeys); + $results = $this->filteredRecords($canEarlyExit); + $results = $this->applyDistanceCalculation($results); + $results = $this->applyJoins($results); + if ($this->groupByField !== null) { + $results = $this->applyGroupBy($results); // having() only applies to groups + } + $results = $this->applySort($results); if ($this->offsetCount > 0 || $this->limitCount !== null) { $results = array_slice($results, $this->offsetCount, $this->limitCount); } + return $this->applyProjection($results); + } - // 14. Apply select/except fields - if (count($this->selectFields) > 0 || count($this->exceptFields) > 0) { - $results = array_map(function($record) { - if (count($this->selectFields) > 0) { - // Include only selected fields (always include 'key') - $filtered = []; - foreach ($this->selectFields as $field) { - if (array_key_exists($field, $record)) { - $filtered[$field] = $record[$field]; - } - } - if (isset($record['key']) && !in_array('key', $this->selectFields)) { - $filtered['key'] = $record['key']; - } - return $filtered; - } else { - // Exclude specified fields - foreach ($this->exceptFields as $field) { - unset($record[$field]); - } - return $record; - } - }, $results); + /** + * Keys of the records a write (update/delete/removeFields) applies to: + * filters, sort, offset and limit apply; select/except/join don't matter. + * @return int[] + */ + private function targetKeys(): array { + if ($this->groupByField !== null) { + throw new \LogicException('noneDB: update(), delete() and removeFields() cannot be combined with groupBy()'); } - - // Apply distance calculation if withDistance() was called (v3.1.0) - if ($this->distanceConfig !== null) { - $results = $this->applyDistanceCalculation($results); + $records = $this->applySort($this->applyDistanceCalculation($this->filteredRecords(empty($this->sortKeys)))); + if ($this->offsetCount > 0 || $this->limitCount !== null) { + $records = array_slice($records, $this->offsetCount, $this->limitCount); + } + $keys = []; + foreach ($records as $record) { + if (isset($record['key']) && is_int($record['key'])) { + $keys[] = $record['key']; + } } + return $keys; + } - return $results; + /** + * Run a write under the database's exclusive lock, so selecting and writing are one atomic step + */ + private function withWriteLock(callable $fn) { + $dbname = $this->callPrivateMethod('normalizeDbName', $this->dbname); + return $this->callPrivateMethod('withLock', $dbname, true, $fn); } /** @@ -8716,8 +7866,11 @@ public function get(): array { public function first(): ?array { $originalLimit = $this->limitCount; $this->limitCount = 1; - $results = $this->get(); - $this->limitCount = $originalLimit; + try { + $results = $this->get(); + } finally { + $this->limitCount = $originalLimit; + } return $results[0] ?? null; } @@ -8731,34 +7884,36 @@ public function last(): ?array { } /** - * Count matching records - * v3.0.0 optimization: Uses fast-path for unfiltered count + * Count matching records (groups, with groupBy()) * @return int */ public function count(): int { - // Fast-path: No filters = use direct count from index/meta - if($this->isUnfiltered()){ + if ($this->isUnfiltered()) { return $this->db->count($this->dbname, 0); } - return count($this->get()); + if ($this->groupByField !== null) { + return count($this->get()); + } + $results = $this->filteredRecords(true); + if ($this->offsetCount > 0 || $this->limitCount !== null) { + $results = array_slice($results, $this->offsetCount, $this->limitCount); + } + return count($results); } /** - * Check if query has no filters applied - * Used for fast-path count optimization + * True when count() can use the stored counters * @return bool */ private function isUnfiltered(): bool { return empty($this->whereFilters) && empty($this->orWhereFilters) - && empty($this->whereInFilters) - && empty($this->whereNotInFilters) - && empty($this->whereNotFilters) - && empty($this->likeFilters) - && empty($this->notLikeFilters) - && empty($this->betweenFilters) - && empty($this->notBetweenFilters) - && empty($this->searchFilters); + && !$this->hasAdvancedFilters() + && empty($this->spatialFilters) + && $this->groupByField === null + && empty($this->havingFilters) + && $this->limitCount === null + && $this->offsetCount === 0; } /** @@ -8766,31 +7921,34 @@ private function isUnfiltered(): bool { * @return bool */ public function exists(): bool { - $originalLimit = $this->limitCount; - $this->limitCount = 1; - $count = count($this->get()); - $this->limitCount = $originalLimit; - return $count > 0; + return $this->first() !== null; } // ========================================== // AGGREGATION METHODS // ========================================== + /** + * Numeric values of a field in the results (booleans and non-numeric values are ignored) + */ + private function numericValues(string $field): array { + $values = []; + foreach ($this->get() as $record) { + $value = $record[$field] ?? null; + if ((is_int($value) || is_float($value) || (is_string($value) && is_numeric($value)))) { + $values[] = $value + 0; + } + } + return $values; + } + /** * Sum of field values * @param string $field * @return float */ public function sum(string $field): float { - $results = $this->get(); - $sum = 0; - foreach ($results as $record) { - if (isset($record[$field]) && is_numeric($record[$field])) { - $sum += $record[$field]; - } - } - return $sum; + return (float)array_sum($this->numericValues($field)); } /** @@ -8799,49 +7957,41 @@ public function sum(string $field): float { * @return float */ public function avg(string $field): float { - $results = $this->get(); - $sum = 0; - $count = 0; - foreach ($results as $record) { - if (isset($record[$field]) && is_numeric($record[$field])) { - $sum += $record[$field]; - $count++; - } - } - return $count > 0 ? $sum / $count : 0; + $values = $this->numericValues($field); + return count($values) > 0 ? array_sum($values) / count($values) : 0.0; } /** - * Minimum value of field + * Minimum value of field (same ordering as sort()) * @param string $field * @return mixed|null */ public function min(string $field) { - $results = $this->get(); $min = null; - foreach ($results as $record) { - if (isset($record[$field])) { - if ($min === null || $record[$field] < $min) { - $min = $record[$field]; - } + $found = false; + foreach ($this->get() as $record) { + if (!isset($record[$field])) continue; + if (!$found || $this->callPrivateMethod('compareSortValues', $record[$field], $min) < 0) { + $min = $record[$field]; + $found = true; } } return $min; } /** - * Maximum value of field + * Maximum value of field (same ordering as sort()) * @param string $field * @return mixed|null */ public function max(string $field) { - $results = $this->get(); $max = null; - foreach ($results as $record) { - if (isset($record[$field])) { - if ($max === null || $record[$field] > $max) { - $max = $record[$field]; - } + $found = false; + foreach ($this->get() as $record) { + if (!isset($record[$field])) continue; + if (!$found || $this->callPrivateMethod('compareSortValues', $record[$field], $max) > 0) { + $max = $record[$field]; + $found = true; } } return $max; @@ -8853,9 +8003,8 @@ public function max(string $field) { * @return array */ public function distinct(string $field): array { - $results = $this->get(); $values = []; - foreach ($results as $record) { + foreach ($this->get() as $record) { if (isset($record[$field]) && !in_array($record[$field], $values, true)) { $values[] = $record[$field]; } @@ -8868,151 +8017,77 @@ public function distinct(string $field): array { // ========================================== /** - * Update matching records + * Update matching records (sort/offset/limit select which ones) * @param array $set Fields to update * @return array ["n" => count] */ public function update(array $set): array { - $results = $this->get(); - if (count($results) === 0) { - return ['n' => 0]; - } - - $keys = array_map(fn($r) => $r['key'], $results); - return $this->db->update($this->dbname, [ - ['key' => $keys], - ['set' => $set] - ]); + return $this->withWriteLock(function () use ($set) { + $keys = $this->targetKeys(); + if (empty($keys)) { + return ['n' => 0]; + } + return $this->db->update($this->dbname, [['key' => $keys], ['set' => $set]]); + }); } /** - * Delete matching records + * Delete matching records (sort/offset/limit select which ones) * @return array ["n" => count] */ public function delete(): array { - $results = $this->get(); - if (count($results) === 0) { - return ['n' => 0]; - } - - $keys = array_map(fn($r) => $r['key'], $results); - return $this->db->delete($this->dbname, ['key' => $keys]); + return $this->withWriteLock(function () { + $keys = $this->targetKeys(); + if (empty($keys)) { + return ['n' => 0]; + } + return $this->db->delete($this->dbname, ['key' => $keys]); + }); } /** - * Remove specified fields from matching records - * This permanently removes fields from the database - * @param array $fields Fields to remove + * Remove fields from the stored matching records (permanent) + * @param array $fields Fields to remove ("key" can't be removed) * @return array ["n" => count of updated records, "fields_removed" => array of removed fields] */ public function removeFields(array $fields): array { if (empty($fields)) { return ['n' => 0, 'fields_removed' => [], 'error' => 'No fields specified']; } - - // Filter out reserved 'key' field - cannot be removed - $fieldsToRemove = array_values(array_filter($fields, fn($f) => $f !== 'key')); + $fieldsToRemove = array_values(array_filter($fields, function ($f) { return $f !== 'key'; })); if (empty($fieldsToRemove)) { return ['n' => 0, 'fields_removed' => [], 'error' => 'Cannot remove reserved field: key']; } - $results = $this->get(); - if (count($results) === 0) { - return ['n' => 0, 'fields_removed' => $fieldsToRemove]; - } - - $updatedCount = 0; - $actuallyRemoved = []; - - foreach ($results as $record) { - $key = $record['key']; - $fieldsRemovedFromRecord = []; - - // Check which fields exist in this record - foreach ($fieldsToRemove as $field) { - if (array_key_exists($field, $record)) { - $fieldsRemovedFromRecord[] = $field; - if (!in_array($field, $actuallyRemoved)) { - $actuallyRemoved[] = $field; - } - } + return $this->withWriteLock(function () use ($fieldsToRemove) { + $keys = $this->targetKeys(); + if (empty($keys)) { + return ['n' => 0, 'fields_removed' => $fieldsToRemove]; } - - if (!empty($fieldsRemovedFromRecord)) { - // Create new record without the removed fields - $newRecord = []; - foreach ($record as $k => $v) { - if ($k !== 'key' && !in_array($k, $fieldsRemovedFromRecord)) { - $newRecord[$k] = $v; + // Fields are removed from the stored records, never from a projected/joined result + $stored = $this->db->find($this->dbname, ['key' => $keys]); + $replacements = []; + $removed = []; + foreach ($stored as $record) { + $changed = false; + foreach ($fieldsToRemove as $field) { + if (array_key_exists($field, $record)) { + unset($record[$field]); + $removed[$field] = true; + $changed = true; } } - - // Directly update the record at its position - $this->updateRecordAtPosition($key, $newRecord); - $updatedCount++; - } - } - - return [ - 'n' => $updatedCount, - 'fields_removed' => $actuallyRemoved - ]; - } - - /** - * Helper method to update a record at a specific key position - * v3.0: Uses JSONL format via parent's updateJsonlRecord method - * @param int $key The record key - * @param array $newData The new record data - */ - private function updateRecordAtPosition(int $key, array $newData): void { - $dbname = $this->callPrivateMethod('sanitizeDbName', $this->dbname); - $hash = $this->callPrivateMethod('hashDBName', $dbname); - $dbDir = $this->getDbDir(); - $fullPath = $dbDir . $hash . "-" . $dbname . ".nonedb"; - - // Check if sharded - $metaPath = $fullPath . ".meta"; - if (file_exists($metaPath)) { - // Sharded database - find correct shard - $metaContent = file_get_contents($metaPath); - $meta = json_decode($metaContent, true); - if ($meta) { - $shardSize = $meta['shardSize']; - $shardId = (int)floor($key / $shardSize); - $localKey = $key % $shardSize; - - // Use JSONL update method for sharded data - $this->callPrivateMethod('updateJsonlRecord', $dbname, $localKey, $newData, $shardId); + if ($changed) { + $key = $record['key']; + unset($record['key']); + $replacements[$key] = $record; + } } - } else { - // Non-sharded database - use JSONL update - $this->callPrivateMethod('updateJsonlRecord', $dbname, $key, $newData, null); - } - } - - /** - * Get the database directory from parent noneDB instance - * @return string - */ - private function getDbDir(): string { - $reflector = new \ReflectionClass($this->db); - $property = $reflector->getProperty('dbDir'); - $property->setAccessible(true); - return $property->getValue($this->db); - } - - /** - * Call a private method on the parent noneDB instance - * @param string $methodName - * @param mixed ...$args - * @return mixed - */ - private function callPrivateMethod(string $methodName, ...$args) { - $reflector = new \ReflectionClass($this->db); - $method = $reflector->getMethod($methodName); - $method->setAccessible(true); - return $method->invoke($this->db, ...$args); + $dbname = $this->callPrivateMethod('normalizeDbName', $this->dbname); + return [ + 'n' => $this->callPrivateMethod('replaceRecordsByKey', $dbname, $replacements), + 'fields_removed' => array_values(array_filter($fieldsToRemove, function ($f) use ($removed) { return isset($removed[$f]); })) + ]; + }); } } -?> \ No newline at end of file diff --git a/phpunit.xml b/phpunit.xml index f9561d2..a6c1fee 100644 --- a/phpunit.xml +++ b/phpunit.xml @@ -7,7 +7,8 @@ stopOnFailure="false" convertErrorsToExceptions="true" convertNoticesToExceptions="true" - convertWarningsToExceptions="true"> + convertWarningsToExceptions="true" + convertDeprecationsToExceptions="true"> @@ -19,6 +20,9 @@ tests/Integration + + tests/Regression + diff --git a/tests/Feature/CheckDBTest.php b/tests/Feature/CheckDBTest.php index 6829584..6e22cd2 100644 --- a/tests/Feature/CheckDBTest.php +++ b/tests/Feature/CheckDBTest.php @@ -34,9 +34,9 @@ public function checkDBReturnsFalseForEmptyString(): void */ public function checkDBReturnsFalseForOnlySpecialChars(): void { - $result = $this->noneDB->checkDB('@#$%^&*()'); - - $this->assertFalse($result, 'Only special chars should return false after sanitization'); + // v3.2.0: invalid names are rejected instead of being silently stripped + $this->expectException(\InvalidArgumentException::class); + $this->noneDB->checkDB('@#$%^&*()'); } /** @@ -81,10 +81,14 @@ public function checkDBReturnsFalseWhenAutoCreateDisabled(): void */ public function checkDBSanitizesDBName(): void { - $result = $this->noneDB->checkDB('test', @@ -74,15 +68,16 @@ public function xssAttemptInData(): void */ public function pathTraversalAttemptInDBName(): void { - $maliciousName = '../../../etc/passwd'; - - // The hash function should neutralize path traversal - $result = $this->noneDB->createDB($maliciousName); - - $this->assertTrue($result); + try { + $this->noneDB->createDB('../../../etc/passwd'); + $this->fail('Expected InvalidArgumentException'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('invalid database name', $e->getMessage()); + } - // Should NOT create file outside db directory + // Nothing created outside (or inside) the db directory $this->assertFileDoesNotExist('/etc/passwd.json'); + $this->assertSame([], glob($this->testDbDir . '*passwd*')); } /** @@ -90,12 +85,8 @@ public function pathTraversalAttemptInDBName(): void */ public function nullByteInjectionAttempt(): void { - $dbName = "test\x00malicious"; - - // Should handle null bytes safely - $result = $this->noneDB->createDB($dbName); - - $this->assertTrue($result); + $this->expectException(\InvalidArgumentException::class); + $this->noneDB->createDB("test\x00malicious"); } // ========================================== @@ -107,14 +98,23 @@ public function nullByteInjectionAttempt(): void */ public function unicodeInDatabaseName(): void { - $unicodeName = 'veritabanı_测试_データ'; - - $this->noneDB->createDB($unicodeName); - $this->noneDB->insert($unicodeName, ['data' => 'test']); - - $result = $this->noneDB->find($unicodeName, 0); + // v3.2.0: non-ASCII names are rejected instead of being stripped (stripping let names collide) + try { + $this->noneDB->createDB('veritabanı_测试_データ'); + $this->fail('Expected InvalidArgumentException'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('invalid database name', $e->getMessage()); + } - $this->assertCount(1, $result); + // 'çalışan' used to be stripped to 'alan' and write into that database; now it is rejected + $this->noneDB->insert('alan', ['tip' => 'alan']); + try { + $this->noneDB->insert('çalışan', ['tip' => 'çalışan']); + $this->fail('Expected InvalidArgumentException'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('invalid database name', $e->getMessage()); + } + $this->assertSame(['alan'], array_column($this->noneDB->find('alan', 0), 'tip'), "nothing leaked into 'alan'"); } /** @@ -122,7 +122,7 @@ public function unicodeInDatabaseName(): void */ public function unicodeInData(): void { - $dbName = 'unicode_data_test'; + $dbName = 'unicode-data-test'; $unicodeData = [ 'turkish' => 'Türkçe karakterler: ğüşıöç', @@ -146,7 +146,7 @@ public function unicodeInData(): void */ public function specialCharactersInFieldNames(): void { - $dbName = 'special_fields_test'; + $dbName = 'special-fields-test'; $data = [ 'field-with-dash' => 'value1', @@ -186,7 +186,7 @@ public function reasonablyLongDatabaseNameWorks(): void */ public function veryLongFieldName(): void { - $dbName = 'long_field_test'; + $dbName = 'long-field-test'; $longFieldName = str_repeat('field', 100); $data = [$longFieldName => 'value']; @@ -202,7 +202,7 @@ public function veryLongFieldName(): void */ public function veryLongStringValue(): void { - $dbName = 'long_value_test'; + $dbName = 'long-value-test'; $longValue = str_repeat('x', 1000000); // 1MB string $data = ['content' => $longValue]; @@ -218,7 +218,7 @@ public function veryLongStringValue(): void */ public function deeplyNestedData(): void { - $dbName = 'nested_test'; + $dbName = 'nested-test'; // Create 10-level deep nested array $nested = ['level' => 10, 'data' => 'deepest']; @@ -238,10 +238,10 @@ public function deeplyNestedData(): void */ public function emptyStringDatabaseName(): void { - $result = $this->noneDB->createDB(''); - - // Empty string should still work (hash will be consistent) - $this->assertTrue($result); + // v3.2.0: an empty name is invalid; checkDB() reports false, other operations throw + $this->assertFalse($this->noneDB->checkDB('')); + $this->expectException(\InvalidArgumentException::class); + $this->noneDB->createDB(''); } /** @@ -266,7 +266,7 @@ public function numericDatabaseName(): void */ public function booleanValues(): void { - $dbName = 'boolean_test'; + $dbName = 'boolean-test'; $data = [ ['flag' => true], @@ -287,7 +287,7 @@ public function booleanValues(): void */ public function numericStringVsInteger(): void { - $dbName = 'numeric_test'; + $dbName = 'numeric-test'; $data = [ ['value' => 123], @@ -312,7 +312,7 @@ public function numericStringVsInteger(): void */ public function floatValues(): void { - $dbName = 'float_test'; + $dbName = 'float-test'; $data = [ 'price' => 19.99, @@ -332,7 +332,7 @@ public function floatValues(): void */ public function nullValues(): void { - $dbName = 'null_test'; + $dbName = 'null-test'; $data = [ 'name' => 'test', @@ -351,7 +351,7 @@ public function nullValues(): void */ public function emptyArrayValue(): void { - $dbName = 'empty_array_test'; + $dbName = 'empty-array-test'; $data = [ 'items' => [], @@ -396,7 +396,7 @@ public function mixedArrayValue(): void */ public function jsonSpecialCharacters(): void { - $dbName = 'json_special_test'; + $dbName = 'json-special-test'; $data = [ 'quotes' => 'He said "Hello"', @@ -418,7 +418,7 @@ public function jsonSpecialCharacters(): void */ public function jsonControlCharacters(): void { - $dbName = 'control_chars_test'; + $dbName = 'control-chars-test'; $data = [ 'content' => "text\r\nwith\tcontrol\x00chars", @@ -439,7 +439,7 @@ public function jsonControlCharacters(): void */ public function rapidCreateDeleteCycle(): void { - $dbName = 'rapid_cycle_test'; + $dbName = 'rapid-cycle-test'; for ($i = 0; $i < 10; $i++) { $this->noneDB->insert($dbName, ['iteration' => $i]); @@ -458,7 +458,7 @@ public function rapidCreateDeleteCycle(): void */ public function findAfterMultipleDeletes(): void { - $dbName = 'find_after_delete_test'; + $dbName = 'find-after-delete-test'; // Insert multiple records for ($i = 0; $i < 5; $i++) { @@ -480,7 +480,7 @@ public function findAfterMultipleDeletes(): void */ public function updateAfterPartialDelete(): void { - $dbName = 'update_after_delete_test'; + $dbName = 'update-after-delete-test'; $this->noneDB->insert($dbName, [ ['id' => 1, 'status' => 'active'], @@ -532,7 +532,7 @@ public function operationsOnCorruptedDataReturnsEmptyArray(): void */ public function operationsOnEmptyFile(): void { - $dbName = 'empty_file_test'; + $dbName = 'empty-file-test'; // Create empty file $this->noneDB->createDB($dbName); @@ -600,7 +600,7 @@ public function keyFieldInNestedData(): void */ public function findByKeyZero(): void { - $dbName = 'key_zero_test'; + $dbName = 'key-zero-test'; $this->noneDB->insert($dbName, [ ['name' => 'first'], @@ -619,7 +619,7 @@ public function findByKeyZero(): void */ public function deleteByKeyZero(): void { - $dbName = 'delete_key_zero_test'; + $dbName = 'delete-key-zero-test'; $this->noneDB->insert($dbName, [ ['name' => 'first'], diff --git a/tests/Regression/ConcurrencyRegressionTest.php b/tests/Regression/ConcurrencyRegressionTest.php new file mode 100644 index 0000000..ab181fc --- /dev/null +++ b/tests/Regression/ConcurrencyRegressionTest.php @@ -0,0 +1,155 @@ +childDb($config); + $writer = "\$db = $db; for (\$i = 0; \$i < 60; \$i++) { \$db->insert('c', ['w' => (int)\$argv[1], 'i' => \$i]); }"; + $this->runChild(str_replace('(int)$argv[1]', '0', $writer)); + $this->runParallel($writer, 5); + + $check = $this->db($config); + $check->flush('c'); + $all = $check->find('c', 0); + $keys = array_column($all, 'key'); + $pairs = array_unique(array_map(function ($r) { return $r['w'] . '-' . $r['i']; }, $all)); + + $this->assertCount(360, $all, "$label: every insert is visible"); + $this->assertCount(360, array_unique($keys), "$label: keys are unique"); + $this->assertCount(360, $pairs, "$label: no record lost or duplicated"); + $this->assertSame(360, $check->count('c'), "$label: count() agrees"); + } + + /** @test K-01 */ + public function concurrentInsertsLoseNothingNonSharded(): void + { + $this->assertNoLostInserts([], 'plain'); + } + + /** @test K-01 */ + public function concurrentInsertsLoseNothingSharded(): void + { + $this->assertNoLostInserts(['shardSize' => 40], 'sharded'); + } + + /** @test K-01 K-08 */ + public function concurrentBufferedInsertsLoseNothing(): void + { + $this->assertNoLostInserts(['bufferEnabled' => true], 'buffered'); + } + + /** @test K-01 */ + public function concurrentMixedWritesStayConsistent(): void + { + $db = $this->db(); + $rows = []; + for ($i = 0; $i < 100; $i++) { + $rows[] = ['n' => $i, 'v' => 0]; + } + $db->insert('mix', $rows); + + // 4 processes: each increments v of its own 25 records 5 times (read-modify-write through update) + $child = $this->childDb(); + $this->runParallel("\$db = $child; \$p = (int)\$argv[1]; + for (\$round = 0; \$round < 5; \$round++) { + for (\$n = (\$p - 1) * 25; \$n < \$p * 25; \$n++) { + \$r = \$db->find('mix', ['n' => \$n])[0]; + \$db->update('mix', [['key' => \$r['key']], ['set' => ['v' => \$r['v'] + 1]]]); + } + \$db->delete('mix', ['n' => 1000 + \$p]); // no-op deletes interleaved + }", 4); + + \noneDB::clearStaticCache(); + $all = $this->db()->find('mix', 0); + $this->assertCount(100, $all); + foreach ($all as $r) { + $this->assertSame(5, $r['v'], 'record ' . $r['n'] . ' got every increment'); + } + } + + /** @test K-07 */ + public function staleInstanceSeesCompactionByAnotherProcess(): void + { + $db = $this->db(); + $rows = []; + for ($i = 0; $i < 30; $i++) { + $rows[] = ['name' => 'user' . $i]; + } + $db->insert('u', $rows); + $this->assertSame('user10', $db->find('u', ['key' => 10])[0]['name']); // index now cached here + + // Another process deletes records and compacts: every byte offset changes + $this->runChild('$db = ' . $this->childDb() . '; for ($i = 0; $i < 10; $i++) { $db->delete("u", ["key" => $i]); } $db->compact("u");'); + + $this->assertSame('user10', $db->find('u', ['key' => 10])[0]['name']); + $this->assertSame('user17', $db->find('u', ['key' => 17])[0]['name']); + $db->update('u', [['key' => 10], ['set' => ['seen' => true]]]); + + \noneDB::clearStaticCache(); + $fresh = $this->db()->find('u', 0); + $this->assertCount(20, $fresh); + $names = array_column($fresh, 'name'); + $this->assertSame(count($names), count(array_unique($names)), 'no record duplicated or overwritten'); + $this->assertTrue($this->db()->find('u', ['key' => 10])[0]['seen']); + } + + /** @test K-01 */ + public function longLivedInstanceSeesWritesOfOtherProcesses(): void + { + $worker = $this->db(); + $worker->insert('w', ['from' => 'worker-1']); + $this->runChild('$db = ' . $this->childDb() . '; $db->insert("w", ["from" => "web"]);'); + $worker->insert('w', ['from' => 'worker-2']); + + \noneDB::clearStaticCache(); + $all = $this->db()->find('w', 0); + $this->assertSame(['worker-1', 'web', 'worker-2'], array_column($all, 'from')); + $this->assertSame([0, 1, 2], array_column($all, 'key')); + } + + /** @test Y-13 */ + public function migrationByAnotherProcessIsNotSplitBrain(): void + { + $a = $this->db(['shardSize' => 20]); + $a->insert('m', ['i' => -1]); + $this->assertSame(1, count($a->find('m', 0))); // A has "not sharded" cached + + // B pushes the database over the threshold: it migrates to shards + $this->runChild('$db = ' . $this->childDb(['shardSize' => 20]) . '; $r = []; for ($i = 0; $i < 30; $i++) { $r[] = ["i" => $i]; } $db->insert("m", $r);'); + $this->assertTrue(file_exists($this->dbBasePath('m') . '.nonedb.meta')); + + $a->insert('m', ['i' => 999]); + $this->assertFalse(file_exists($this->dbBasePath('m') . '.nonedb'), 'A must not recreate the non-sharded file'); + \noneDB::clearStaticCache(); + $this->assertCount(32, $this->db()->find('m', 0)); + } + + /** @test O-26 */ + public function lockTimeoutIsHonoured(): void + { + $db = $this->db(); + $db->insert('lk', ['a' => 1]); + $lock = $this->dbBasePath('lk') . '.nonedb.lock'; + $holder = $this->childScript('$fp = fopen(' . var_export($lock, true) . ', "c+"); flock($fp, LOCK_EX); echo "locked\n"; fflush(STDOUT); sleep(4);'); + $proc = proc_open([PHP_BINARY, $holder], [1 => ['pipe', 'w']], $pipes); // no shell: proc_terminate() reaches PHP + $this->assertSame("locked", trim(fgets($pipes[1]))); + + $start = microtime(true); + try { + $this->db(['lockTimeout' => 1])->insert('lk', ['a' => 2]); + $this->fail('expected a lock timeout'); + } catch (\RuntimeException $e) { + $this->assertStringContainsString('timed out', $e->getMessage()); + } + $this->assertLessThan(3.5, microtime(true) - $start); + fclose($pipes[1]); + proc_terminate($proc); + proc_close($proc); + } +} diff --git a/tests/Regression/CrashRecoveryTest.php b/tests/Regression/CrashRecoveryTest.php new file mode 100644 index 0000000..35339e8 --- /dev/null +++ b/tests/Regression/CrashRecoveryTest.php @@ -0,0 +1,183 @@ +markTestSkipped('needs proc_open and POSIX signals'); + } + parent::setUp(); + } + + private function crashOnce(string $name, array $config, int $killAfterAcks, int $seed): void + { + $db = $this->db($config); + if (!$db->checkDB($name) || $db->count($name) === 0) { + $db->insert($name, ['n' => -1, 'c' => 'seed', 'loc' => ['type' => 'Point', 'coordinates' => [0, 0]]]); + $db->createFieldIndex($name, 'c'); + $db->createSpatialIndex($name, 'loc'); + } + $before = []; + foreach ($db->find($name, 0) as $r) { + $before[$r['n']] = $r['c']; + } + + // The writer announces every operation ("try ...") and acknowledges it once the call returned. + // Only the single operation in flight at the moment of the kill may or may not have happened. + $writer = $this->childScript(' + mt_srand(' . $seed . '); + $db = ' . $this->childDb($config) . '; + $own = []; + for ($i = 0; ; $i++) { + $n = ' . ($seed * 100000) . ' + $i; + echo "try ins $n\n"; + $r = $db->insert(' . var_export($name, true) . ', ["n" => $n, "c" => "c" . ($n % 5), "loc" => ["type" => "Point", "coordinates" => [($n % 360) - 180 + 0.5, (($n * 7) % 170) - 85 + 0.5]]]); + if ($r["n"] !== 1) { echo "ERR ", json_encode($r), "\n"; exit(1); } + echo "ins $n\n"; + $own[] = $n; + if ($i % 7 === 3) { + $target = $own[array_rand($own)]; + echo "try upd $target\n"; + $db->update(' . var_export($name, true) . ', [["n" => $target], ["set" => ["c" => "upd"]]]); + echo "upd $target\n"; + } + if ($i % 11 === 5) { + $target = array_shift($own); + echo "try del $target\n"; + $db->delete(' . var_export($name, true) . ', ["n" => $target]); + echo "del $target\n"; + } + }'); + // Command as an array: no shell in between, so SIGKILL reaches the writer itself + // (a "sh -c" that doesn't exec, like dash, would be killed and leave the writer running) + $proc = proc_open([PHP_BINARY, $writer], [1 => ['pipe', 'w'], 2 => ['pipe', 'w']], $pipes); + $state = []; // n => 'inserted' | 'deleted' + $updated = []; // n => true (acknowledged update) + $inFlight = null; // [op, n] announced but not acknowledged + $lines = 0; + $apply = function (string $line) use (&$state, &$updated, &$inFlight) { + $line = trim($line); + if ($line === '') { + return; + } + $parts = explode(' ', $line); + if ($parts[0] === 'ERR') { + $this->fail('writer failed: ' . $line); + } + if ($parts[0] === 'try') { + $inFlight = [$parts[1], (int)$parts[2]]; + return; + } + [$op, $n] = [$parts[0], (int)$parts[1]]; + if ($op === 'ins') { $state[$n] = 'inserted'; } + if ($op === 'upd') { $updated[$n] = true; } + if ($op === 'del') { $state[$n] = 'deleted'; } + $inFlight = null; + }; + while ($lines < $killAfterAcks && ($line = fgets($pipes[1])) !== false) { + $apply($line); + $lines++; + } + usleep(mt_rand(0, 3000)); + proc_terminate($proc, 9); // SIGKILL: no shutdown handlers, no destructors + while (($line = fgets($pipes[1])) !== false) { + if (substr($line, -1) === "\n") { + $apply($line); + } + } + $stderr = stream_get_contents($pipes[2]); + fclose($pipes[1]); + fclose($pipes[2]); + proc_close($proc); + $this->assertSame('', trim((string)$stderr), 'writer printed no warnings or errors'); + + \noneDB::clearStaticCache(); + $db = $this->db($config); + $all = $db->find($name, 0); + $byN = []; + foreach ($all as $r) { + $this->assertArrayNotHasKey($r['n'], $byN, "record {$r['n']} duplicated"); + $byN[$r['n']] = $r; + } + $this->assertSame(count($all), count(array_unique(array_column($all, 'key'))), 'keys unique'); + + foreach ($state as $n => $st) { + $uncertain = $inFlight !== null && $inFlight[1] === $n; + if ($st === 'inserted' && !($uncertain && $inFlight[0] === 'del')) { + $this->assertArrayHasKey($n, $byN, "acknowledged insert $n survived the crash"); + } + if ($st === 'deleted') { + $this->assertArrayNotHasKey($n, $byN, "acknowledged delete of $n survived the crash"); + } + if (isset($updated[$n]) && isset($byN[$n])) { + $this->assertSame('upd', $byN[$n]['c'], "acknowledged update of $n survived the crash"); + } + } + foreach ($byN as $n => $r) { + $known = isset($before[$n]) || isset($state[$n]) || ($inFlight !== null && $inFlight[0] === 'ins' && $inFlight[1] === $n); + $this->assertTrue($known, "record $n appeared from nowhere"); + } + + // Field index == scan + foreach (['c0', 'c1', 'c2', 'c3', 'c4', 'upd', 'seed'] as $c) { + $scan = array_values(array_filter($all, function ($r) use ($c) { return $r['c'] === $c; })); + $this->assertSame(array_column($scan, 'key'), array_column($db->find($name, ['c' => $c]), 'key'), "field index for c=$c"); + } + // Spatial index == brute force + foreach ([[-180, -90, 180, 90], [-10, -10, 40, 50], [100, -80, 179, 0]] as $box) { + $expected = []; + foreach ($all as $r) { + [$x, $y] = $r['loc']['coordinates']; + if ($x >= $box[0] && $x <= $box[2] && $y >= $box[1] && $y <= $box[3]) { + $expected[] = $r['key']; + } + } + $got = array_column($db->withinBBox($name, 'loc', ...$box), 'key'); + sort($got); + $this->assertSame($expected, $got, 'spatial index for box ' . json_encode($box)); + } + $this->assertSame(count($all), $db->count($name), 'count() agrees with the records'); + $this->assertSame([], preg_grep('/^(j=|J$)/', $this->lockState($name)['flags']), 'journal cleared after recovery'); + $this->assertNotContains('b', $this->lockState($name)['flags'], 'buffer flag cleared after the replay'); + $this->assertSame([], glob($this->dbBasePath($name) . '*.flushing*'), 'interrupted flushes replayed'); + + // Still writable + $db->insert($name, ['n' => -2 - $seed, 'c' => 'after', 'loc' => ['type' => 'Point', 'coordinates' => [1, 1]]]); + $this->assertCount(1, $db->find($name, ['c' => 'after', 'n' => -2 - $seed])); + } + /** @test */ + public function killedWriterNonSharded(): void + { + mt_srand(11); + foreach ([5, 40, 120, 300, 77, 9, 450] as $i => $acks) { + $this->crashOnce('crash', [], $acks, 10 + $i); + } + } + + /** @test */ + public function killedWriterShardedIncludingMigration(): void + { + mt_srand(12); + foreach ([20, 60, 150, 250, 99, 33, 400] as $i => $acks) { + $this->crashOnce('crashs', ['shardSize' => 40], $acks, 20 + $i); + } + } + + /** @test */ + public function killedWriterWithBuffer(): void + { + mt_srand(13); + foreach ([15, 80, 200, 41, 333, 7] as $i => $acks) { + $this->crashOnce('crashb', ['bufferEnabled' => true], $acks, 30 + $i); + } + } +} diff --git a/tests/Regression/DifferentialFuzzTest.php b/tests/Regression/DifferentialFuzzTest.php new file mode 100644 index 0000000..7ccf3e9 --- /dev/null +++ b/tests/Regression/DifferentialFuzzTest.php @@ -0,0 +1,425 @@ + reference model: key => record (without "key") */ + private $model = []; + /** @var int */ + private $nextKey = 0; + + // ------------------------------------------------------------------ + // Reference model + // ------------------------------------------------------------------ + + private static function keyList($value): array + { + $out = []; + foreach (is_array($value) ? $value : [$value] as $v) { + if (is_int($v) && $v >= 0) { + $k = $v; + } elseif (is_string($v) && preg_match('/^(0|[1-9][0-9]*)$/', $v)) { + $k = (int)$v; + } else { + continue; + } + if (!in_array($k, $out, true)) { + $out[] = $k; + } + } + return $out; + } + + /** find()/update()/delete() filter semantics */ + private function modelSelect(array $filters): array + { + $keys = null; + if (array_key_exists('key', $filters)) { + $keys = self::keyList($filters['key']); + unset($filters['key']); + } + $result = []; + $candidates = $keys === null ? array_keys($this->model) : $keys; + if ($keys === null) { + sort($candidates); + } + foreach ($candidates as $k) { + if (!array_key_exists($k, $this->model)) { + continue; + } + $rec = $this->model[$k]; + $ok = true; + foreach ($filters as $f => $v) { + if (!array_key_exists($f, $rec) || $rec[$f] !== $v) { + $ok = false; + break; + } + } + if ($ok) { + $result[$k] = $rec; + } + } + return $result; + } + + private static function withKeys(array $records): array + { + $out = []; + foreach ($records as $k => $r) { + $r['key'] = $k; + $out[] = $r; + } + return $out; + } + + /** ordering comparison of the query builder: numbers (and numeric strings) numerically, strings lexically */ + private static function cmp($a, $b): ?int + { + foreach ([$a, $b] as $v) { + if ($v === null || is_bool($v) || is_array($v)) { + return null; + } + } + $na = is_int($a) || is_float($a) || (is_string($a) && is_numeric($a)); + $nb = is_int($b) || is_float($b) || (is_string($b) && is_numeric($b)); + if ($na && $nb) { + return ($a + 0) <=> ($b + 0); + } + if (is_string($a) && is_string($b)) { + return strcmp($a, $b) <=> 0; + } + return null; + } + + private static function opMatch(array $rec, string $field, string $op, $operand): bool + { + $has = array_key_exists($field, $rec); + $v = $has ? $rec[$field] : null; + switch ($op) { + case '$gt': $c = $has ? self::cmp($v, $operand) : null; return $c !== null && $c > 0; + case '$gte': $c = $has ? self::cmp($v, $operand) : null; return $c !== null && $c >= 0; + case '$lt': $c = $has ? self::cmp($v, $operand) : null; return $c !== null && $c < 0; + case '$lte': $c = $has ? self::cmp($v, $operand) : null; return $c !== null && $c <= 0; + case '$ne': return !$has || $v !== $operand; + case '$eq': return $has && $v === $operand; + case '$in': return $has && in_array($v, $operand, true); + case '$nin': return !$has || !in_array($v, $operand, true); + case '$exists': return $operand ? $has : !$has; + } + throw new \LogicException($op); + } + + private static function groupMatch(array $rec, array $group): bool + { + foreach ($group as $field => $value) { + if (is_array($value) && !empty($value) && is_string(array_key_first($value)) && array_key_first($value)[0] === '$') { + foreach ($value as $op => $operand) { + if (!self::opMatch($rec, (string)$field, $op, $operand)) { + return false; + } + } + } elseif (!array_key_exists($field, $rec) || $rec[$field] !== $value) { + return false; + } + } + return true; + } + + private static function sortRank($v): int + { + if ($v === null) return 0; + if (is_bool($v)) return 1; + if (is_int($v) || is_float($v)) return 2; + if (is_string($v)) return 3; + return 4; + } + + private function modelQuery(array $q): array + { + $rows = self::withKeys($this->modelSelect([])); + $rows = array_values(array_filter($rows, function ($r) use ($q) { + $hasWhere = !empty($q['where']); + $hasOr = !empty($q['or']); + $match = !$hasWhere && !$hasOr; + if ($hasWhere) { + $all = true; + foreach ($q['where'] as $g) { + if (!self::groupMatch($r, $g)) { $all = false; break; } + } + $match = $all; + } + if (!$match && $hasOr) { + foreach ($q['or'] as $g) { + if (self::groupMatch($r, $g)) { $match = true; break; } + } + } + if (!$match) return false; + foreach ($q['whereIn'] as [$f, $vals]) { + if (!array_key_exists($f, $r) || !in_array($r[$f], $vals, true)) return false; + } + foreach ($q['whereNotIn'] as [$f, $vals]) { + if (array_key_exists($f, $r) && in_array($r[$f], $vals, true)) return false; + } + foreach ($q['whereNot'] as $g) { + foreach ($g as $f => $v) { + if (array_key_exists($f, $r) && $r[$f] === $v) return false; + } + } + foreach ($q['between'] as [$f, $min, $max]) { + if (!array_key_exists($f, $r)) return false; + $lo = self::cmp($r[$f], $min); + $hi = self::cmp($r[$f], $max); + if ($lo === null || $hi === null || $lo < 0 || $hi > 0) return false; + } + return true; + })); + if ($q['sort'] !== null) { + [$field, $order] = $q['sort']; + $decorated = []; + foreach ($rows as $i => $r) { + $decorated[] = [$i, $r]; + } + usort($decorated, function ($x, $y) use ($field, $order) { + $a = $x[1][$field] ?? null; + $b = $y[1][$field] ?? null; + $ra = self::sortRank($a); + $rb = self::sortRank($b); + $c = $ra !== $rb ? $ra <=> $rb : ($ra === 4 ? strcmp(json_encode($a), json_encode($b)) : $a <=> $b); + if ($order === 'desc') $c = -$c; + return $c !== 0 ? $c : $x[0] <=> $y[0]; + }); + $rows = array_column($decorated, 1); + } + return array_slice($rows, $q['offset'], $q['limit']); + } + + // ------------------------------------------------------------------ + // Random generators + // ------------------------------------------------------------------ + + private function randValue(string $field) + { + switch ($field) { + case 'a': return [0, 1, 2, 3, 5, 8, '3', '8', 2.5, null, -0.0][mt_rand(0, 10)]; + case 'b': return ['x', 'y', 'z', 'Z', '', true, false, null][mt_rand(0, 7)]; + case 'c': return mt_rand(0, 1) ? mt_rand(-5, 5) / 2 : 'w' . mt_rand(0, 3); + default: return [['t' => mt_rand(0, 1)], [mt_rand(0, 2)], []][mt_rand(0, 2)]; + } + } + + private function randRecord(): array + { + $r = []; + foreach (['a', 'b', 'c', 'd'] as $f) { + if (mt_rand(0, 4) > 0) { + $r[$f] = $this->randValue($f); + } + } + if (empty($r)) { + $r['a'] = 0; + } + return $r; + } + + private function randFilter(): array + { + $filter = []; + $n = mt_rand(0, 2); + for ($i = 0; $i < $n; $i++) { + $f = ['a', 'b', 'c', 'd'][mt_rand(0, 3)]; + $filter[$f] = $this->randValue($f); + } + if (mt_rand(0, 5) === 0) { + $keys = array_keys($this->model); + $pick = empty($keys) ? 0 : $keys[array_rand($keys)]; + $filter['key'] = [$pick, (string)$pick, mt_rand(0, $this->nextKey + 2), [$pick, $pick + 1]][mt_rand(0, 3)]; + } + return $filter; + } + + private function randOperatorGroup(): array + { + $ops = [ + ['a', '$gt', mt_rand(0, 6)], ['a', '$lte', '3'], ['a', '$ne', 3], ['b', '$in', ['x', 'Z', true]], + ['b', '$nin', ['y', null]], ['c', '$gte', 0], ['c', '$lt', 'w2'], ['d', '$exists', (bool)mt_rand(0, 1)], + ['a', '$eq', 8], ['c', '$gt', -1.5], + ]; + $group = []; + for ($i = mt_rand(1, 2); $i > 0; $i--) { + [$f, $op, $v] = $ops[mt_rand(0, count($ops) - 1)]; + $group[$f][$op] = $v; + } + if (mt_rand(0, 2) === 0) { + $f = ['a', 'b'][mt_rand(0, 1)]; + if (!isset($group[$f])) { + $group[$f] = $this->randValue($f); + } + } + return $group; + } + + private function randQuery(): array + { + $q = ['where' => [], 'or' => [], 'whereIn' => [], 'whereNotIn' => [], 'whereNot' => [], 'between' => [], 'sort' => null, 'limit' => null, 'offset' => 0]; + for ($i = mt_rand(0, 2); $i > 0; $i--) $q['where'][] = $this->randOperatorGroup(); + for ($i = mt_rand(0, 2) === 0 ? 1 : 0; $i > 0; $i--) $q['or'][] = $this->randOperatorGroup(); + if (mt_rand(0, 3) === 0) $q['whereIn'][] = ['b', ['x', 'y', false]]; + if (mt_rand(0, 3) === 0) $q['whereNotIn'][] = ['a', [0, '3']]; + if (mt_rand(0, 3) === 0) $q['whereNot'][] = ['b' => 'z']; + if (mt_rand(0, 3) === 0) $q['between'][] = ['a', 1, '5']; + if (mt_rand(0, 1)) $q['sort'] = [['a', 'b', 'c', 'd'][mt_rand(0, 3)], mt_rand(0, 1) ? 'asc' : 'desc']; + if (mt_rand(0, 2) === 0) $q['limit'] = mt_rand(0, 5); + if (mt_rand(0, 3) === 0) $q['offset'] = mt_rand(0, 3); + return $q; + } + + private function runQuery(\noneDB $db, string $name, array $q): array + { + $b = $db->query($name); + foreach ($q['where'] as $g) $b->where($g); + foreach ($q['or'] as $g) $b->orWhere($g); + foreach ($q['whereIn'] as [$f, $v]) $b->whereIn($f, $v); + foreach ($q['whereNotIn'] as [$f, $v]) $b->whereNotIn($f, $v); + foreach ($q['whereNot'] as $g) $b->whereNot($g); + foreach ($q['between'] as [$f, $min, $max]) $b->between($f, $min, $max); + if ($q['sort'] !== null) $b->sort($q['sort'][0], $q['sort'][1]); + if ($q['limit'] !== null) $b->limit($q['limit']); + if ($q['offset'] > 0) $b->offset($q['offset']); + return $b->get(); + } + + // ------------------------------------------------------------------ + // Driver + // ------------------------------------------------------------------ + + private function fuzz(string $name, array $config, int $ops, int $seed, bool $alternateInstances = false, array $indexes = []): void + { + mt_srand($seed); + $this->model = []; + $this->nextKey = 0; + if ($alternateInstances) { + \noneDB::disableStaticCache(); + } + $instances = [$this->db($config)]; + if ($alternateInstances) { + $instances[] = $this->db($config); + } + $instances[0]->insert($name, ['seed' => 1]); + $this->model[$this->nextKey++] = ['seed' => 1]; + foreach ($indexes as $field) { + $instances[0]->createFieldIndex($name, $field); + } + + for ($op = 0; $op < $ops; $op++) { + $db = $instances[$op % count($instances)]; + $kind = mt_rand(0, 9); + $ctx = "seed=$seed op=$op kind=$kind"; + if ($kind <= 3) { + $batch = mt_rand(0, 3) === 0 ? mt_rand(2, 6) : 1; + $records = []; + for ($i = 0; $i < $batch; $i++) { + $records[] = $this->randRecord(); + } + $r = $db->insert($name, $batch === 1 ? $records[0] : $records); + $this->assertSame(['n' => $batch], $r, $ctx); + foreach ($records as $rec) { + $this->model[$this->nextKey++] = $rec; + } + } elseif ($kind <= 5) { + $filter = $this->randFilter(); + $set = []; + foreach (['a', 'b', 'c', 'd', 'e'] as $f) { + if (mt_rand(0, 2) === 0) $set[$f] = $f === 'e' ? $op : $this->randValue($f); + } + if (empty($set)) $set['e'] = $op; + $expected = $this->modelSelect($filter); + $r = $db->update($name, [$filter, ['set' => $set]]); + $this->assertSame(['n' => count($expected)], $r, "$ctx update " . json_encode($filter)); + foreach (array_keys($expected) as $k) { + $this->model[$k] = array_merge($this->model[$k], $set); + } + } elseif ($kind <= 6) { + $filter = $this->randFilter(); + if (empty($filter)) $filter = ['b' => 'z']; + $expected = $this->modelSelect($filter); + $r = $db->delete($name, $filter); + $this->assertSame(['n' => count($expected)], $r, "$ctx delete " . json_encode($filter)); + foreach (array_keys($expected) as $k) { + unset($this->model[$k]); + } + } elseif ($kind <= 8) { + $filter = $this->randFilter(); + $this->assertSame(self::withKeys($this->modelSelect($filter)), $db->find($name, $filter), "$ctx find " . json_encode($filter)); + $q = $this->randQuery(); + $this->assertSame($this->modelQuery($q), $this->runQuery($db, $name, $q), "$ctx query " . json_encode($q)); + } else { + if (mt_rand(0, 3) === 0) { + $db->compact($name); + } + $this->assertSame(count($this->model), $db->count($name), "$ctx count"); + } + + if ($op % 25 === 0 || $op === $ops - 1) { + $this->assertSame(self::withKeys($this->modelSelect([])), $db->find($name, 0), "$ctx full contents"); + } + } + + // Independent fresh instance sees exactly the model + \noneDB::enableStaticCache(); + \noneDB::clearStaticCache(); + $this->assertSame(self::withKeys($this->modelSelect([])), $this->db($config)->find($name, 0), "seed=$seed final"); + } + + /** @test */ + public function nonSharded(): void + { + $this->fuzz('fz', [], 600, 101); + } + + /** @test */ + public function shardedWithMigrationMidRun(): void + { + $this->fuzz('fzs', ['shardSize' => 7], 600, 202); + } + + /** @test */ + public function withFieldIndexes(): void + { + $this->fuzz('fzi', [], 500, 303, false, ['a', 'b', 'd']); + $this->fuzz('fzis', ['shardSize' => 9], 500, 304, false, ['a', 'b']); + } + + /** @test */ + public function withWriteBuffer(): void + { + $this->fuzz('fzb', ['bufferEnabled' => true], 500, 404); + $this->fuzz('fzbs', ['bufferEnabled' => true, 'shardSize' => 8], 500, 405); + } + + /** @test */ + public function twoInstancesWithSeparateCaches(): void + { + $this->fuzz('fzt', [], 500, 505, true); + $this->fuzz('fzts', ['shardSize' => 6], 500, 506, true, ['a']); + } + + /** @test */ + public function manySeeds(): void + { + for ($seed = 1000; $seed < 1012; $seed++) { + $config = $seed % 3 === 0 ? ['shardSize' => 5 + $seed % 4] : []; + $indexes = $seed % 2 === 0 ? ['a', 'b'] : []; + $this->fuzz('fzm' . $seed, $config, 150, $seed, $seed % 4 === 1, $indexes); + } + } +} diff --git a/tests/Regression/FieldIndexRegressionTest.php b/tests/Regression/FieldIndexRegressionTest.php new file mode 100644 index 0000000..9ade5cd --- /dev/null +++ b/tests/Regression/FieldIndexRegressionTest.php @@ -0,0 +1,188 @@ +db(); + $db->insert('fx', [['e-mail' => 'a@x', 'e_mail' => 'under', 'şehir' => 'İzmir', 'first.name' => 'Ada']]); + foreach (['e-mail', 'e_mail', 'şehir', 'first.name'] as $field) { + $this->assertTrue($db->createFieldIndex('fx', $field)['success'], $field); + } + $db->insert('fx', ['e-mail' => 'b@x', 'e_mail' => 'u2', 'şehir' => 'Bursa', 'first.name' => 'Bob']); + + \noneDB::clearStaticCache(); + $db = $this->db(); + $this->assertSame('Bob', $db->find('fx', ['e-mail' => 'b@x'])[0]['first.name']); + $this->assertSame('u2', $db->find('fx', ['e_mail' => 'u2'])[0]['e_mail']); + $this->assertSame('b@x', $db->find('fx', ['şehir' => 'Bursa'])[0]['e-mail']); + $this->assertSame('Bob', $db->find('fx', ['first.name' => 'Bob'])[0]['first.name']); + $this->assertEqualsCanonicalizing(['e-mail', 'e_mail', 'şehir', 'first.name'], $db->getFieldIndexes('fx')['fields']); + $this->assertCount(4, glob($this->dbBasePath('fx') . '.nonedb.fidx.*'), 'one file per field, no collisions'); + } + + /** @test Y-12 */ + public function nullAndTypeTransitionsAreIndexed(): void + { + $db = $this->db(); + $db->insert('nt', [['x' => 'a'], ['y' => 1], ['x' => ['arr']], ['x' => null]]); + $db->createFieldIndex('nt', 'x'); + $db->update('nt', [['y' => 1], ['set' => ['x' => null]]]); // missing -> null + $db->update('nt', [['key' => 2], ['set' => ['x' => 'b']]]); // array -> scalar + $db->update('nt', [['key' => 0], ['set' => ['x' => ['now-array']]]]); // scalar -> array + + $this->assertSame([1, 3], array_column($db->find('nt', ['x' => null]), 'key')); + $this->assertSame([2], array_column($db->find('nt', ['x' => 'b']), 'key')); + $this->assertSame([], $db->find('nt', ['x' => 'a'])); + $db->delete('nt', ['x' => null]); + $this->assertSame([], $db->find('nt', ['x' => null])); + $this->assertSame([], $db->getKeysFromFieldIndex('nt', 'x', null)); + } + + /** @test Y-12 O-28 */ + public function shardedIndexesCoverNullsAndNewShards(): void + { + $db = $this->db(['shardSize' => 4]); + $rows = []; + for ($i = 0; $i < 10; $i++) { + $rows[] = ['i' => $i, 'v' => $i % 2 ? null : 'x']; + } + $db->insert('sn', $rows); + $db->createFieldIndex('sn', 'v'); + $this->assertCount(5, $db->find('sn', ['v' => null])); + + // New shards created after the index + $more = []; + for ($i = 10; $i < 20; $i++) { + $more[] = ['i' => $i, 'v' => 'late']; + } + $db->insert('sn', $more); + $db->update('sn', [['i' => 19], ['set' => ['v' => 'Z']]]); + $this->assertSame([19], array_column($db->find('sn', ['v' => 'Z']), 'i')); + $this->assertCount(9, $db->find('sn', ['v' => 'late'])); + $this->assertSame(['v'], $db->getFieldIndexes('sn')['fields']); + $this->assertTrue($db->getFieldIndexes('sn')['sharded']); + + $this->assertTrue($db->dropFieldIndex('sn', 'v')['success']); + $this->assertSame([], glob($this->dbBasePath('sn') . '*fidx*')); + $this->assertFalse($db->dropFieldIndex('sn', 'v')['success']); + } + + /** @test O-30 */ + public function negativeZeroAndMixedTypesMatchLikeAScan(): void + { + $db = $this->db(); + $db->insert('mz', [['v' => -0.0], ['v' => 0], ['v' => '0'], ['v' => 1.0], ['v' => 1], ['v' => '1'], ['v' => true], ['v' => '__null__'], ['v' => null]]); + $values = [0.0, -0.0, 0, '0', 1, 1.0, '1', true, false, null, '__null__']; + $scan = []; + foreach ($values as $v) { + $scan[] = array_column($db->find('mz', ['v' => $v]), 'key'); + } + $db->createFieldIndex('mz', 'v'); + foreach ($values as $i => $v) { + $this->assertSame($scan[$i], array_column($db->find('mz', ['v' => $v]), 'key'), 'value ' . var_export($v, true)); + } + } + + /** @test P-02 property: index never changes results */ + public function randomizedIndexedResultsEqualScan(): void + { + mt_srand(42); + foreach ([[], ['shardSize' => 25]] as $config) { + $name = empty($config) ? 'rp' : 'rps'; + $db = $this->db($config); + $rows = []; + for ($i = 0; $i < 120; $i++) { + $rows[] = ['c' => mt_rand(0, 5), 's' => ['a', 'b', null][mt_rand(0, 2)], 'n' => $i]; + } + $db->insert($name, $rows); + $db->createFieldIndex($name, 'c'); + $db->createFieldIndex($name, 's'); + for ($op = 0; $op < 60; $op++) { + $r = mt_rand(0, 2); + if ($r === 0) { + $db->insert($name, ['c' => mt_rand(0, 5), 's' => ['a', 'b', null][mt_rand(0, 2)], 'n' => 1000 + $op]); + } elseif ($r === 1) { + $db->update($name, [['c' => mt_rand(0, 5)], ['set' => ['s' => ['a', 'b', null, 'q'][mt_rand(0, 3)]]]]); + } else { + $db->delete($name, ['c' => mt_rand(0, 5), 's' => ['a', 'b', null][mt_rand(0, 2)]]); + } + } + $all = $db->find($name, 0); + foreach ([0, 1, 2, 3, 4, 5] as $c) { + foreach (['a', 'b', null, 'q'] as $s) { + $expected = array_values(array_filter($all, function ($r) use ($c, $s) { + return $r['c'] === $c && array_key_exists('s', $r) && $r['s'] === $s; + })); + $this->assertSame(array_column($expected, 'key'), array_column($db->find($name, ['c' => $c, 's' => $s]), 'key'), "$name c=$c s=" . var_export($s, true)); + } + } + } + } + + /** @test crash recovery of secondary indexes */ + public function leftoverJournalRebuildsIndexes(): void + { + $db = $this->db(); + $db->insert('j', [['city' => 'A'], ['city' => 'B']]); + $db->createFieldIndex('j', 'city'); + // Simulate a writer that appended data and died before updating the field index: it had + // bumped the generation and journaled the segment in the lock file's state record + $state = $this->lockState('j'); + $this->assertContains('v', $state['flags']); + $this->assertSame([], preg_grep('/^(j=|J$)/', $state['flags']), 'no journal entry after a completed write'); + file_put_contents($this->dbBasePath('j') . '.nonedb.lock', str_pad(($state['gen'] + 1) . ' v j=main', 128)); + $index = json_decode(file_get_contents($this->dbBasePath('j') . '.nonedb.fidx.city'), true); + $index['values'] = ['A' => [0]]; // B lost + file_put_contents($this->dbBasePath('j') . '.nonedb.fidx.city', json_encode($index)); + + \noneDB::clearStaticCache(); + $db = $this->db(); + $this->assertSame([1], array_column($db->find('j', ['city' => 'B']), 'key')); + $this->assertSame([0], array_column($db->find('j', ['city' => 'A']), 'key')); + $this->assertSame([], preg_grep('/^(j=|J$)/', $this->lockState('j')['flags']), 'journal entry cleared after recovery'); + // "J" (too many segments to list) rebuilds every segment + file_put_contents($this->dbBasePath('j') . '.nonedb.fidx.city', json_encode($index)); + file_put_contents($this->dbBasePath('j') . '.nonedb.lock', str_pad(($this->lockState('j')['gen'] + 1) . ' v J', 128)); + \noneDB::clearStaticCache(); + $this->assertSame([1], array_column($this->db()->find('j', ['city' => 'B']), 'key')); + $this->assertSame([], preg_grep('/^(j=|J$)/', $this->lockState('j')['flags'])); + } + + /** @test pre-3.2 index files are rebuilt, not trusted */ + public function legacyIndexFilesAreRebuilt(): void + { + $db = $this->db(); + $db->insert('lv', [['city' => 'A'], ['city' => 'B'], ['city' => null]]); + // A v1 file from an older version that missed records (and, like every pre-3.2 database, no lock file) + file_put_contents($this->dbBasePath('lv') . '.nonedb.fidx.city', json_encode(['v' => 1, 'field' => 'city', 'values' => ['A' => [0]]])); + $this->removeLockFile('lv'); + $db = $this->db(); + $this->assertSame(['city'], $db->getFieldIndexes('lv')['fields'], 'derived from the old file'); + $this->assertSame([1], array_column($db->find('lv', ['city' => 'B']), 'key')); + $this->assertSame([2], array_column($db->find('lv', ['city' => null]), 'key')); + $this->assertSame(2, json_decode(file_get_contents($this->dbBasePath('lv') . '.nonedb.fidx.city'), true)['v']); + } + + /** @test P-02 a batch writes each index file once */ + public function batchInsertWritesIndexOnce(): void + { + $db = $this->db(); + $db->insert('bt', ['c' => 0]); + $db->createFieldIndex('bt', 'c'); + $rows = []; + for ($i = 0; $i < 3000; $i++) { + $rows[] = ['c' => $i % 7]; + } + $start = microtime(true); + $db->insert('bt', $rows); + $this->assertLessThan(2.0, microtime(true) - $start); + $this->assertCount(428, $db->find('bt', ['c' => 6])); // i % 7 === 6 for i < 3000 + } +} diff --git a/tests/Regression/LockStateRegressionTest.php b/tests/Regression/LockStateRegressionTest.php new file mode 100644 index 0000000..5d7b76f --- /dev/null +++ b/tests/Regression/LockStateRegressionTest.php @@ -0,0 +1,193 @@ +db(); + $db->insert('st', ['a' => 1]); + $state = $this->lockState('st'); + $this->assertSame(['v'], $state['flags'], 'a new database is verified, nothing pending'); + $this->assertSame(128, filesize($this->dbBasePath('st') . '.nonedb.lock'), 'fixed size record'); + + // Buffered records: flag set while they wait, cleared once they are flushed + $buffered = $this->db(['bufferEnabled' => true]); + $buffered->insert('st', ['a' => 2]); + $this->assertContains('b', $this->lockState('st')['flags']); + $this->assertSame([1, 2], array_column($this->db()->find('st', 0), 'a'), 'a reader without buffering flushes them'); + $this->assertSame(['v'], $this->lockState('st')['flags']); + + // Field index: the journal entry exists only during the write + $db->createFieldIndex('st', 'a'); + $db->insert('st', ['a' => 3]); + $this->assertSame(['v'], $this->lockState('st')['flags']); + + $gen = $this->lockState('st')['gen']; + $this->assertTrue($db->dropDB('st')['success']); + $this->assertFileExists($this->dbBasePath('st') . '.nonedb.lock', 'lock file survives a drop'); + $this->assertSame(['v'], $this->lockState('st')['flags']); + $this->assertGreaterThan($gen, $this->lockState('st')['gen']); + $this->assertFalse($db->checkDB('st') && $db->count('st') > 0); + $this->assertSame([], $this->db()->find('st', 0) ?: []); + } + + /** @test */ + public function generationMovesOncePerWriteAndNeverOnReads(): void + { + $db = $this->db(); + $db->insert('g', [['a' => 1], ['a' => 2]]); + $db->createFieldIndex('g', 'a'); + $gen = $this->lockState('g')['gen']; + + $db->find('g', 0); + $db->find('g', ['a' => 1]); + $db->count('g'); + $db->query('g')->where(['a' => ['$gt' => 0]])->get(); + $this->assertSame($gen, $this->lockState('g')['gen'], 'reads leave the generation alone'); + + $db->insert('g', ['a' => 3]); + $this->assertSame($gen + 1, $this->lockState('g')['gen'], 'one write session, one bump'); + $db->update('g', [['a' => 3], ['set' => ['b' => 1]]]); + $this->assertSame($gen + 2, $this->lockState('g')['gen']); + $db->delete('g', ['a' => 3]); + $this->assertSame($gen + 3, $this->lockState('g')['gen']); + + // Another process sees each change (its caches are dropped on the generation change) + $out = $this->runChild('$d = ' . $this->childDb() . '; echo json_encode(array_column($d->find("g", 0), "a"));'); + $this->assertSame('[1,2]', $out); + } + + /** @test the path is locked, not a deleted file a kept handle still points to */ + public function replacedLockFileIsNotTrusted(): void + { + $db = $this->db(['lockTimeout' => 1]); + $db->insert('rl', ['a' => 1]); // the lock handle stays open in this process + $lock = $this->dbBasePath('rl') . '.nonedb.lock'; + $this->assertTrue(unlink($lock)); // e.g. a cleanup script + + $holder = $this->childScript('$fp = fopen(' . var_export($lock, true) . ', "c+"); flock($fp, LOCK_EX); echo "locked\n"; fflush(STDOUT); sleep(3);'); + $proc = proc_open([PHP_BINARY, $holder], [1 => ['pipe', 'w']], $pipes); // no shell: proc_terminate() reaches PHP + $this->assertSame('locked', trim(fgets($pipes[1]))); + try { + $db->insert('rl', ['a' => 2]); + $this->fail('the insert must wait for the process holding the new lock file'); + } catch (\RuntimeException $e) { + $this->assertStringContainsString('timed out', $e->getMessage()); + } finally { + fclose($pipes[1]); + proc_terminate($proc); + proc_close($proc); + } + $db->insert('rl', ['a' => 3]); + $this->assertSame([1, 3], array_column($db->find('rl', 0), 'a')); + $this->assertSame(['v'], $this->lockState('rl')['flags'], 'state record rewritten in the new file'); + } + + /** @test a forked child must not share the parent's lock handle (flock locks belong to the open file) */ + public function forkedProcessesStillExcludeEachOther(): void + { + if (!function_exists('pcntl_fork')) { + $this->markTestSkipped('needs pcntl'); + } + $out = $this->runChild(' + $cfg = ' . var_export(['secretKey' => self::SECRET, 'dbDir' => $this->testDbDir], true) . '; + $db = new \noneDB($cfg); + $db->insert("fk", ["w" => "init"]); // lock handle opened before the fork + $pid = pcntl_fork(); + $who = $pid === 0 ? "c" : "p"; + for ($i = 0; $i < 300; $i++) { + $db->insert("fk", ["w" => $who, "i" => $i]); + $db->update("fk", [["w" => "init"], ["set" => ["n" => $i]]]); + } + if ($pid === 0) { exit(0); } + pcntl_waitpid($pid, $status); + \noneDB::clearStaticCache(); + $all = (new \noneDB($cfg))->find("fk", 0); + $c = count(array_filter($all, function ($r) { return $r["w"] === "c"; })); + echo count($all), " ", count(array_unique(array_column($all, "key"))), " ", $c; + '); + $this->assertSame('601 601 300', $out); + } + + /** + * @test kept lock handles are close-on-exec: a program this process starts must not inherit one + * (an inherited copy would keep a lock held after this process died in the middle of a write) + */ + public function keptLockHandlesAreNotInherited(): void + { + if (DIRECTORY_SEPARATOR === '\\') { + $this->markTestSkipped('POSIX file descriptors'); + } + $db = $this->db(); + $db->insert('ih', ['a' => 1]); // the lock handle stays open in this process + $lock = $this->dbBasePath('ih') . '.nonedb.lock'; + $out = $this->runChild('$ino = stat(' . var_export($lock, true) . ')["ino"]; $n = 0; + for ($i = 3; $i < 256; $i++) { + $f = @fopen("php://fd/$i", "r"); + if ($f) { $s = @fstat($f); if ($s && $s["ino"] === $ino) { $n++; } fclose($f); } + } + echo "inherited:", $n;'); + $this->assertSame('inherited:0', $out); + } + + /** @test */ + public function keptLockHandlesAreLimited(): void + { + $db = $this->db(); + for ($i = 0; $i < 100; $i++) { + $db->insert('h' . $i, ['i' => $i]); + } + $handles = new \ReflectionProperty(\noneDB::class, 'lockHandles'); + if (PHP_VERSION_ID < 80100) { + $handles->setAccessible(true); // no effect since 8.1, deprecated in 8.5 + } + $this->assertLessThanOrEqual(64, count($handles->getValue())); + for ($i = 0; $i < 100; $i++) { + $this->assertSame([$i], array_column($db->find('h' . $i, 0), 'i')); + } + \noneDB::clearStaticCache(); + $this->assertSame([], $handles->getValue(), 'clearStaticCache closes them'); + } + + /** @test */ + public function readingAMissingDatabaseCreatesNothing(): void + { + $db = $this->db(); + $before = glob($this->testDbDir . '*'); + $this->assertSame([], $db->find('nothere', 0) ?: []); + $this->assertSame(0, $db->count('nothere')); + $this->assertSame([], $db->query('nothere')->get()); + $this->assertSame($before, glob($this->testDbDir . '*')); + } + + /** + * @test a pre-3.2 database is checked once (when it has no state record yet), not per operation; + * deleting the lock files makes the check run again + */ + public function oldVersionLeftoversAreCheckedOnce(): void + { + $db = $this->db(); + $db->insert('ov', [['a' => 1]]); + file_put_contents($this->dbBasePath('ov') . '.nonedb.buffer', json_encode(['a' => 2]) . "\n"); + $this->removeLockFile('ov'); + + $db = $this->db(); + $this->assertSame([1, 2], array_column($db->find('ov', 0), 'a'), 'found by the one-time check'); + $this->assertSame(['v'], $this->lockState('ov')['flags']); + + // Verified: an old-version file appearing now is not looked for on every operation... + file_put_contents($this->dbBasePath('ov') . '.nonedb.buffer.flushing', json_encode(['a' => 3]) . "\n"); + $this->assertSame([1, 2], array_column($db->find('ov', 0), 'a')); + // ...until the state record is removed (documented for going back and forth between versions) + $this->removeLockFile('ov'); + $this->assertSame([1, 2, 3], array_column($this->db()->find('ov', 0), 'a')); + } +} diff --git a/tests/Regression/PublicApiCoverageTest.php b/tests/Regression/PublicApiCoverageTest.php new file mode 100644 index 0000000..cca8b81 --- /dev/null +++ b/tests/Regression/PublicApiCoverageTest.php @@ -0,0 +1,150 @@ +db(); + $this->assertFalse($db->isBufferingEnabled(), 'off by default since 3.2'); + $db->enableBuffering(true); + $this->assertTrue($db->isBufferingEnabled()); + $db->setBufferSizeLimit(10); // clamped to 1KB + $db->setBufferCountLimit(3); // clamped to 10 + $db->setBufferFlushInterval(0); + + for ($i = 0; $i < 5; $i++) { + $db->insert('bf', ['i' => $i]); + } + $info = $db->getBufferInfo('bf'); + $this->assertTrue($info['enabled']); + $this->assertSame(1024, $info['sizeLimit']); + $this->assertSame(10, $info['countLimit']); + $this->assertSame(5, $info['buffers']['main']['records']); + $this->assertTrue($info['buffers']['main']['exists']); + + for ($i = 5; $i < 12; $i++) { + $db->insert('bf', ['i' => $i]); // the 10th record triggers a flush + } + $this->assertLessThan(10, $db->getBufferInfo('bf')['buffers']['main']['records']); + $db->insert('other', ['x' => 1]); + $r = $db->flushAllBuffers(); + $this->assertSame(2, $r['databases']); + $this->assertFalse($db->getBufferInfo('bf')['buffers']['main']['exists']); + $this->assertSame(range(0, 11), array_column($db->find('bf', 0), 'i')); + $this->assertSame(['success' => true, 'flushed' => 0, 'error' => null], $db->flush('bf')); + } + + /** @test K-08 */ + public function interruptedFlushIsRecovered(): void + { + $db = $this->db(); + $db->insert('fl', ['i' => 0]); + $buffer = $this->dbBasePath('fl') . '.nonedb.buffer'; + // Left by a pre-3.2 version (".flushing" without start key; those versions had no lock file) + file_put_contents($buffer . '.flushing', json_encode(['i' => 1]) . "\n" . json_encode(['i' => 2]) . "\n"); + file_put_contents($buffer, json_encode(['i' => 3]) . "\n"); + $this->removeLockFile('fl'); + $this->assertSame([0, 1, 2, 3], array_column($this->db()->find('fl', 0), 'i')); + $this->assertNotContains('b', $this->lockState('fl')['flags'], 'buffer flag cleared after the flush'); + $this->assertFileDoesNotExist($buffer); + $this->assertFileDoesNotExist($buffer . '.flushing'); + } + + /** @test K-08 pre-3.2 shard buffers (already counted in the meta) */ + public function legacyShardBuffersAreFlushed(): void + { + $db = $this->db(['shardSize' => 5]); + $rows = []; + for ($i = 0; $i < 7; $i++) { + $rows[] = ['i' => $i]; + } + $db->insert('ls', $rows); + // An old version had buffered two records for shard 1 and counted them in the meta + $metaPath = $this->dbBasePath('ls') . '.nonedb.meta'; + $meta = json_decode(file_get_contents($metaPath), true); + $meta['nextKey'] += 2; + $meta['totalRecords'] += 2; + $meta['shards'][1]['count'] += 2; + file_put_contents($metaPath, json_encode($meta)); + file_put_contents($this->dbBasePath('ls') . '_s1.nonedb.buffer', json_encode(['i' => 7]) . "\n" . json_encode(['i' => 8]) . "\n"); + + $this->removeLockFile('ls'); // pre-3.2 versions had no lock file + $db = $this->db(); + $this->assertSame(range(0, 8), array_column($db->find('ls', 0), 'i')); + $this->assertSame(range(0, 8), array_column($db->find('ls', 0), 'key')); + $this->assertSame(9, $db->count('ls')); + $db->insert('ls', ['i' => 9]); + $this->assertSame(9, $db->find('ls', ['i' => 9])[0]['key']); + } + + /** @test */ + public function indexApi(): void + { + $db = $this->db(); + $db->enableIndexing(false); + $this->assertFalse($db->isIndexingEnabled()); + $db->enableIndexing(true); + $this->assertTrue($db->isIndexingEnabled()); + + $this->assertNull($db->getIndexInfo('ix')); + $this->assertFalse($db->rebuildIndex('ix')['success']); + $db->insert('ix', [['c' => 'a'], ['c' => 'b'], ['c' => 'a']]); + $db->delete('ix', ['key' => 1]); + $info = $db->getIndexInfo('ix'); + $this->assertTrue($info['exists']); + $this->assertSame(2, $info['totalRecords']); + $this->assertFalse($info['sharded']); + + unlink($this->dbBasePath('ix') . '.nonedb.jidx'); + $r = $db->rebuildIndex('ix'); + $this->assertTrue($r['success']); + $this->assertSame(2, $r['totalRecords']); + $this->assertSame([0, 2], array_column($db->find('ix', 0), 'key')); + + $this->assertNull($db->getKeysFromFieldIndex('ix', 'c', 'a'), 'no index yet'); + $db->createFieldIndex('ix', 'c'); + $this->assertSame([0, 2], $db->getKeysFromFieldIndex('ix', 'c', 'a')); + $this->assertSame(['success' => true, 'indexed' => 2, 'values' => 1, 'error' => null], $db->rebuildFieldIndex('ix', 'c')); + } + + /** @test */ + public function spatialHelpers(): void + { + $db = $this->db(); + $this->assertEqualsWithDelta(0, $db->haversineDistance(29, 41, 29, 41), 0.0001); + $this->assertEqualsWithDelta(111194.9, $db->haversineDistance(0, 0, 1, 0), 1); + $db->clearDistanceCache(); + $db->clearCentroidCache(); + + $db->insert('sp', ['loc' => ['type' => 'Point', 'coordinates' => [1, 1]]]); + $db->createSpatialIndex('sp', 'loc'); + $this->assertSame(0, $db->flushSpatialIndexes(), 'nothing pending outside a write'); + $r = $db->rebuildSpatialIndex('sp', 'loc'); + $this->assertTrue($r['success']); + $this->assertSame(1, $r['indexed']); + $this->assertTrue($db->hasSpatialIndex('sp', 'loc')); + $this->assertTrue($db->dropSpatialIndex('sp', 'loc')['success']); + $this->assertFalse($db->hasSpatialIndex('sp', 'loc')); + $this->assertCount(1, $db->withinDistance('sp', 'loc', 1, 1, 10), 'queries still work without an index'); + } + + /** @test */ + public function staticCacheSwitch(): void + { + \noneDB::disableStaticCache(); + $a = $this->db(); + $b = $this->db(); + $a->insert('sc', ['v' => 1]); + $this->assertCount(1, $b->find('sc', 0)); + $b->insert('sc', ['v' => 2]); + $this->assertCount(2, $a->find('sc', 0), 'separate caches still see each other\'s writes'); + \noneDB::enableStaticCache(); + $this->assertCount(2, $this->db()->find('sc', 0)); + } +} diff --git a/tests/Regression/QueryRegressionTest.php b/tests/Regression/QueryRegressionTest.php new file mode 100644 index 0000000..1d1c9c7 --- /dev/null +++ b/tests/Regression/QueryRegressionTest.php @@ -0,0 +1,445 @@ +insert($name, [ + ['name' => 'Alice', 'age' => 25, 'city' => 'Istanbul', 'score' => 85, 'tags' => ['php', 'js']], + ['name' => 'Bob', 'age' => '30', 'city' => 'Ankara', 'score' => 90], + ['name' => 'Charlie', 'age' => 35, 'city' => 'Istanbul', 'score' => null], + ['name' => 'abc', 'age' => 'abc', 'city' => 'Izmir'], + ['name' => 'AYDOĞDU', 'age' => [1], 'city' => 'İzmir'], + ]); + } + + /** @test K-04 Y-26 P-06 */ + public function removeFieldsOnlyRemovesTheGivenFields(): void + { + $db = $this->db(['shardSize' => 5]); + $rows = []; + for ($i = 0; $i < 12; $i++) { + $rows[] = ['name' => 'n' . $i, 'pw' => 'secret' . $i, 'tmp' => $i, 'loc' => ['type' => 'Point', 'coordinates' => [1, 1]]]; + } + $db->insert('rf', $rows); + $db->insert('dept', ['id' => 'x', 'title' => 'X']); + $r = $db->query('rf')->select(['name', 'tmp'])->withDistance('loc', 1, 1)->join('dept', 'name', 'id')->removeFields(['tmp']); + $this->assertSame(12, $r['n']); + $this->assertSame(['tmp'], $r['fields_removed']); + foreach ($db->find('rf', 0) as $record) { + $this->assertArrayNotHasKey('tmp', $record); + $this->assertArrayHasKey('pw', $record, 'select() must not leak into storage'); + $this->assertArrayNotHasKey('_distance', $record); + $this->assertArrayNotHasKey('dept', $record); + } + $this->assertSame(12, $db->query('rf')->except(['key'])->removeFields(['name'])['n']); + $this->assertArrayHasKey('pw', $db->find('rf', ['key' => 11])[0]); + } + + /** @test Y-15 */ + public function keyFilterIsCombinedWithOtherFilters(): void + { + $db = $this->db(); + $this->seed($db); + $this->assertSame([], $db->find('q', ['key' => 0, 'name' => 'zzz'])); + $this->assertSame(0, $db->delete('q', ['key' => 0, 'name' => 'zzz'])['n']); + $this->assertSame(0, $db->update('q', [['key' => 0, 'name' => 'zzz'], ['set' => ['x' => 1]]])['n']); + $this->assertSame('Bob', $db->find('q', ['name' => 'Bob', 'key' => '1'])[0]['name'], 'key filter in any position'); + $this->assertCount(5, $db->find('q', 0)); + } + + /** @test Y-17 */ + public function nullFiltersBehaveTheSameEverywhere(): void + { + foreach ([[], ['shardSize' => 2]] as $config) { + $name = empty($config) ? 'nu' : 'nus'; + $db = $this->db($config); + $this->seed($db, $name); + $this->assertCount(1, $db->find($name, ['score' => null])); + $this->assertSame(1, $db->update($name, [['score' => null], ['set' => ['flag' => 1]]])['n']); + $this->assertSame(1, $db->delete($name, ['score' => null])['n']); + } + } + + /** @test Y-18 */ + public function malformedUpdatesAreRejected(): void + { + $db = $this->db(); + $this->seed($db); + foreach ([[1 => ['set' => ['x' => 1]]], [['name' => 'Bob'], ['set' => 'str']], [['name' => 'Bob']], 'x', [['name' => 'Bob'], ['set' => ['key' => 5]]]] as $bad) { + $r = $db->update('q', $bad); + $this->assertSame(0, $r['n']); + $this->assertSame('Please check your update parameters', $r['error']); + } + foreach ($db->find('q', 0) as $record) { + $this->assertArrayNotHasKey('x', $record); + } + $this->assertSame(1, $db->update('q', [['name' => 'Bob'], ['set' => []]])['n'], 'empty set reports matches, writes nothing'); + } + + /** @test Y-19 */ + public function likePatternsAreLiteralAndSafe(): void + { + $db = $this->db(); + $db->insert('lk', [['url' => 'http://a.com'], ['url' => 'https://b.com'], ['url' => 'h1tp://c'], ['name' => 'john.doe'], ['name' => 'johnXdoe']]); + $this->assertSame(['http://a.com'], array_column($db->like('lk', 'url', '^http://'), 'url')); + $this->assertSame([], $db->like('lk', 'url', '^h.tp')); + $this->assertSame(['john.doe'], array_column($db->like('lk', 'name', '^john.doe$'), 'name')); + $this->assertSame(['http://a.com'], array_column($db->query('lk')->like('url', '^http://')->get(), 'url')); + $this->assertSame(['http://a.com'], array_column($db->query('lk')->where(['url' => ['$like' => '^http://']])->get(), 'url')); + $this->assertSame(['http://a.com', 'https://b.com'], array_column($db->query('lk')->where(['url' => ['$regex' => '^https?://[a-z]']])->get(), 'url')); + $this->assertCount(2, $db->query('lk')->where(['url' => ['$regex' => 'a.com|b.com']])->get(), 'slashes inside $regex are fine'); + try { + $db->query('lk')->where(['url' => ['$regex' => '(unclosed']])->get(); + $this->fail('invalid regex must be reported'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('invalid', $e->getMessage()); + } + try { + $db->query('lk')->where(['url' => ['$like' => ['not', 'a', 'string']]])->get(); + $this->fail('non-string $like must be reported'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('like', $e->getMessage()); + } + } + + /** @test Y-20 Y-21 */ + public function whereAndOrWhereLogic(): void + { + $db = $this->db(); + $this->seed($db); + $this->assertSame(['Bob'], array_column($db->query('q')->orWhere(['name' => 'Bob'])->get(), 'name')); + $this->assertSame(['Alice', 'Charlie'], array_column($db->query('q')->where(['age' => ['$gt' => 20]])->where(['age' => ['$lt' => 40]])->where(['city' => 'Istanbul'])->get(), 'name')); + $this->assertSame(['Alice', 'Bob'], array_column($db->query('q')->where(['age' => ['$gt' => 20]])->where(['age' => ['$lt' => 31]])->get(), 'name')); + $db->insert('num', [['100' => 'x'], ['100' => 'y']]); + $this->assertCount(1, $db->query('num')->where(['100' => 'x'])->where(['100' => ['$ne' => 'z']])->get(), 'numeric field names'); + + $db->insert('sp', [ + ['n' => 'a', 'loc' => ['type' => 'Point', 'coordinates' => [29, 41]]], + ['n' => 'b', 'loc' => ['type' => 'Point', 'coordinates' => [29.001, 41]]], + ['n' => 'far', 'loc' => ['type' => 'Point', 'coordinates' => [40, 41]]], + ]); + $with = $db->query('sp')->withinDistance('loc', 29, 41, 1000)->where(['n' => 'a'])->orWhere(['n' => 'b'])->get(); + $this->assertSame(['a', 'b'], array_column($with, 'n'), 'same logic with spatial filters'); + } + + /** @test Y-22 */ + public function sortIsATotalOrder(): void + { + $db = $this->db(); + $rows = [['v' => 5], ['x' => 1], ['v' => 3], ['v' => 9], ['v' => null], ['v' => 1], ['v' => 'b'], ['v' => 7]]; + $db->insert('so', $rows); + $asc = array_map(function ($r) { return $r['v'] ?? '-'; }, $db->query('so')->sort('v')->get()); + $this->assertSame(['-', '-', 1, 3, 5, 7, 9, 'b'], $asc); + $desc = array_map(function ($r) { return $r['v'] ?? '-'; }, $db->query('so')->sort('v', 'desc')->get()); + $this->assertSame(['b', 9, 7, 5, 3, 1, '-', '-'], $desc); + $multi = $db->query('q2')->get(); + $db->insert('q2', [['c' => 'b', 'n' => 2], ['c' => 'a', 'n' => 3], ['c' => 'b', 'n' => 1]]); + $this->assertSame([3, 1, 2], array_column($db->query('q2')->sort('c')->sort('n')->get(), 'n'), 'second sort() breaks ties'); + } + + /** @test Y-22: query sort, the sort() helper and an independent reference agree on mixed values */ + public function sortOrderMatchesReferenceOnMixedTypes(): void + { + $db = $this->db(); + mt_srand(77); + $pool = [null, true, false, 0, 1, -1, 2.5, -0.5, 10, '10', '9', '1e1', 'abc', 'ABC', 'b', '', ['x'], ['a' => 1], [1, 2], 3.0]; + $rows = []; + for ($i = 0; $i < 120; $i++) { + $row = ['i' => $i, 'g' => mt_rand(0, 2)]; + if (mt_rand(0, 9) > 0) { + $row['v'] = $pool[mt_rand(0, count($pool) - 1)]; + } + $rows[] = $row; + } + $db->insert('mix', $rows); + + // Reference: missing/null < booleans < numbers < strings < arrays; same kind with <=> (arrays by JSON); ties by insertion order + $kind = function ($v) { + if ($v === null) return 0; + if (is_bool($v)) return 1; + if (is_int($v) || is_float($v)) return 2; + if (is_string($v)) return 3; + return 4; + }; + $cmp = function ($a, $b) use ($kind) { + $ka = $kind($a); + $kb = $kind($b); + if ($ka !== $kb) return $ka <=> $kb; + return $ka === 4 ? strcmp(json_encode($a), json_encode($b)) <=> 0 : $a <=> $b; + }; + $reference = function (array $keys) use ($rows, $cmp) { + $sorted = $rows; + usort($sorted, function ($x, $y) use ($keys, $cmp) { + foreach ($keys as [$field, $order]) { + $c = $cmp($x[$field] ?? null, $y[$field] ?? null); + if ($c !== 0) return $order === 'desc' ? -$c : $c; + } + return $x['i'] <=> $y['i']; + }); + return array_column($sorted, 'i'); + }; + + foreach ([[['v', 'asc']], [['v', 'desc']], [['g', 'asc'], ['v', 'desc']], [['g', 'desc'], ['v', 'asc']]] as $keys) { + $q = $db->query('mix'); + foreach ($keys as [$field, $order]) { + $q->sort($field, $order); + } + $this->assertSame($reference($keys), array_column($q->get(), 'i'), json_encode($keys)); + } + $this->assertSame($reference([['v', 'desc']]), array_column($db->sort($db->find('mix', 0), 'v', 'desc'), 'i'), 'sort() helper'); + $this->assertSame(array_slice($reference([['v', 'asc']]), 5, 7), array_column($db->query('mix')->sort('v')->offset(5)->limit(7)->get(), 'i')); + } + + /** @test Y-27 */ + public function nestedValuesCompareTheSameWay(): void + { + $db = $this->db(); + $db->insert('ne', [['addr' => ['city' => 'Ist', 'zip' => 1]], ['addr' => ['city' => 'Ist']], ['addr' => ['city' => 'Ank']]]); + $filter = ['addr' => ['city' => 'Ist']]; + $viaFind = array_column($db->find('ne', $filter), 'key'); + $viaQuery = array_column($db->query('ne')->where($filter)->get(), 'key'); + $viaOr = array_column($db->query('ne')->where($filter)->orWhere(['addr' => ['city' => 'none']])->get(), 'key'); + $this->assertSame([1], $viaFind); + $this->assertSame($viaFind, $viaQuery); + $this->assertSame($viaFind, $viaOr); + } + + /** @test O-01 O-02 */ + public function countMatchesGet(): void + { + $db = $this->db(['bufferEnabled' => true]); + for ($i = 0; $i < 5; $i++) { + $db->insert('ct', ['i' => $i, 'g' => $i % 2, 'loc' => ['type' => 'Point', 'coordinates' => [$i, 0]]]); + } + $this->assertSame(5, $db->count('ct'), 'buffered records are counted'); + $this->assertSame(5, $db->query('ct')->count()); + $this->assertSame(2, $db->query('ct')->groupBy('g')->count(), 'groups'); + $this->assertSame(2, $db->query('ct')->limit(2)->count()); + $this->assertSame(3, $db->query('ct')->offset(2)->count()); + $this->assertSame(count($db->query('ct')->withinDistance('loc', 0, 0, 350000)->get()), $db->query('ct')->withinDistance('loc', 0, 0, 350000)->count()); + } + + /** @test O-03 */ + public function getDBsListsEachDatabaseOnce(): void + { + $db = $this->db(['shardSize' => 3]); + $db->insert('users', ['a' => 1]); + $db->createFieldIndex('users', 'a'); + $db->insert('sh', [['i' => 1], ['i' => 2], ['i' => 3], ['i' => 4]]); + $this->assertEqualsCanonicalizing(['users', 'sh'], $db->getDBs()); + $info = $db->getDBs('sh'); + $this->assertSame('sh', $info['name']); + $this->assertNotSame('0 B', $info['size']); + $this->assertCount(2, $db->getDBs(true)); + } + + /** @test O-04 */ + public function withDistanceIsComputedBeforeSortAndKeptBySelect(): void + { + $db = $this->db(); + $db->insert('wd', [ + ['n' => 'a', 'loc' => ['type' => 'Point', 'coordinates' => [1, 0]]], + ['n' => 'b', 'loc' => ['type' => 'Point', 'coordinates' => [0.5, 0]]], + ['n' => 'c', 'loc' => ['type' => 'Point', 'coordinates' => [0.1, 0]]], + ['n' => 'm', 'loc' => ['type' => 'MultiPolygon', 'coordinates' => [[[[2, 2], [2.001, 2], [2.001, 2.001], [2, 2.001], [2, 2]]]]]], + ]); + $r = $db->query('wd')->withDistance('loc', 0, 0)->sort('_distance')->limit(2)->select(['n'])->get(); + $this->assertSame(['c', 'b'], array_column($r, 'n')); + $this->assertArrayHasKey('_distance', $r[0]); + $m = $db->query('wd')->where(['n' => 'm'])->withDistance('loc', 2.0005, 2.0005)->first(); + $this->assertLessThan(100, $m['_distance'], 'MultiPolygon centroid'); + $own = $db->query('wd')->withinDistance('loc', 1, 0, 200000)->withDistance('loc', 0, 0)->sort('_distance')->get(); + $this->assertEqualsWithDelta(11119.49, $own[0]['_distance'], 1, 'withDistance uses its own point'); + } + + /** @test O-04: withDistance() measures like the spatial filters (nearest point of the geometry) */ + public function withDistanceMeasuresLikeTheSpatialFilters(): void + { + $db = $this->db(); + $square = [[[0.01, -0.005], [0.02, -0.005], [0.02, 0.005], [0.01, 0.005], [0.01, -0.005]]]; + $db->insert('wm', [ + ['n' => 'sq', 'loc' => ['type' => 'Polygon', 'coordinates' => $square]], + ['n' => 'pt', 'loc' => ['type' => 'Point', 'coordinates' => [0.005, 0]]], + ]); + // The square's nearest edge is ~1112 m away, its centroid ~1668 m + $r = $db->query('wm')->withinDistance('loc', 0, 0, 1200)->withDistance('loc', 0, 0)->sort('_distance')->get(); + $this->assertSame(['pt', 'sq'], array_column($r, 'n')); + foreach ($r as $row) { + $this->assertLessThanOrEqual(1200, $row['_distance'], $row['n'] . ' passed a 1200 m filter'); + } + $filterOnly = $db->query('wm')->withinDistance('loc', 0, 0, 1200)->get(); + $this->assertEqualsWithDelta(array_column($filterOnly, '_distance', 'n'), array_column($r, '_distance', 'n'), 0.01); + $inside = $db->query('wm')->where(['n' => 'sq'])->withDistance('loc', 0.015, 0)->first(); + $this->assertSame(0.0, (float)$inside['_distance'], 'a point inside the polygon is at distance 0'); + } + + /** @test O-10: the direct between() compares like query()->between() */ + public function directBetweenIsTypeAwareToo(): void + { + $db = $this->db(); + $this->seed($db); + $db->insert('q', [['name' => 'yes', 'age' => true], ['name' => 'float', 'age' => 26.5], ['name' => 'nul', 'age' => null]]); + foreach ([[26, 40], [0, 100], ['a', 'z'], [25, 25]] as [$min, $max]) { + $this->assertSame( + array_column($db->query('q')->between('age', $min, $max)->get(), 'name'), + array_column($db->between('q', 'age', $min, $max), 'name'), + "between $min and $max" + ); + } + $this->assertSame(['Bob', 'Charlie', 'float'], array_column($db->between('q', 'age', 26, 40), 'name')); + $this->assertSame(['Charlie'], array_column($db->between('q', 'age', 26, 40, ['city' => 'Istanbul']), 'name')); + } + + /** @test O-05 */ + public function nearestAppliesKAfterOtherFilters(): void + { + $db = $this->db(); + $db->insert('nw', [ + ['t' => 'x', 'loc' => ['type' => 'Point', 'coordinates' => [0, 0]]], + ['t' => 'x', 'loc' => ['type' => 'Point', 'coordinates' => [0.1, 0]]], + ['t' => 'z', 'loc' => ['type' => 'Point', 'coordinates' => [1, 0]]], + ['t' => 'z', 'loc' => ['type' => 'Point', 'coordinates' => [2, 0]]], + ]); + $this->assertSame([2, 3], array_column($db->query('nw')->nearest('loc', 0, 0, 2)->where(['t' => 'z'])->get(), 'key')); + } + + /** @test O-06 */ + public function groupByKeepsValueTypes(): void + { + $db = $this->db(); + $db->insert('gb', [['v' => 1.5], ['v' => 1.7], ['v' => [1]], ['v' => '__null__'], ['v' => null], ['x' => 1], ['v' => 1], ['v' => '1']]); + $groups = $db->query('gb')->groupBy('v')->get(); + $this->assertCount(7, $groups); + $byGroup = []; + foreach ($groups as $g) { + $byGroup[json_encode($g['_group'])] = $g['_count']; + } + $this->assertSame(2, $byGroup['null'], 'missing and null share a group'); + $this->assertSame(1, $byGroup['"__null__"']); + $this->assertSame(1, $byGroup['1.5']); + $this->assertSame(1, $byGroup['1']); + $this->assertSame(1, $byGroup['"1"']); + } + + /** @test O-07 */ + public function writesAfterProjectionWorkAndGroupByIsRejected(): void + { + $db = $this->db(); + $this->seed($db); + $this->assertSame(2, $db->query('q')->where(['city' => 'Istanbul'])->except(['key'])->update(['seen' => 1])['n']); + try { + $db->query('q')->groupBy('city')->delete(); + $this->fail('groupBy + delete must be rejected'); + } catch (\LogicException $e) { + $this->assertStringContainsString('groupBy', $e->getMessage()); + } + $this->assertCount(5, $db->find('q', 0)); + } + + /** @test O-08 */ + public function searchIgnoresKeyAndIsMultibyteAware(): void + { + $db = $this->db(); + $this->seed($db); + $this->assertSame([], $db->query('q')->search('4')->get(), '"4" only appears as a record key'); + $this->assertSame(['AYDOĞDU'], array_column($db->query('q')->search('aydoğdu')->get(), 'name')); + $this->assertCount(2, $db->query('q')->search('izmir')->get(), 'Izmir and İzmir'); + $this->assertCount(2, $db->query('q')->search('İZMİR')->get()); + $this->assertCount(2, $db->query('q')->search('ZMIR')->get()); + } + + /** @test O-09 */ + public function joinDoesNotCreateDatabases(): void + { + $db = $this->db(); + $this->seed($db); + $before = glob($this->testDbDir . '*'); + $r = $db->query('q')->join('nothere', 'city', 'id')->get(); + $this->assertNull($r[0]['nothere']); + $this->assertSame($before, glob($this->testDbDir . '*')); + } + + /** @test O-10 O-11 */ + public function operatorsAreTypeAware(): void + { + $db = $this->db(); + $this->seed($db); + $this->assertSame(['Alice', 'Bob', 'Charlie'], array_column($db->query('q')->where(['age' => ['$gt' => 3]])->get(), 'name'), '"abc" and arrays are not greater than 3; "30" is'); + $this->assertSame([], $db->query('q')->where(['score' => ['$lt' => 18]])->get(), 'null is not less than 18'); + $this->assertSame(['Bob', 'Charlie'], array_column($db->query('q')->between('age', 26, 40)->get(), 'name')); + try { + $db->query('q')->where(['age' => ['$gtee' => 1]])->get(); + $this->fail('unknown operator'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('$gtee', $e->getMessage()); + } + try { + $db->query('q')->where(['age' => ['$gt' => 1, 'plain' => 2]])->get(); + $this->fail('mixed operator spec'); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('mix', $e->getMessage()); + } + // Filters are compiled before records are scanned: an invalid query fails the same way + // whether or not there are records to look at + $db->insert('empty-q', ['a' => 1]); + $db->delete('empty-q', []); + foreach ([['$gtee' => 1], ['$in' => 'notarray'], ['$regex' => '(']] as $spec) { + foreach (['q', 'empty-q'] as $name) { + try { + $db->query($name)->where(['age' => $spec])->get(); + $this->fail('invalid spec ' . json_encode($spec) . " on $name"); + } catch (\InvalidArgumentException $e) { + $this->assertNotSame('', $e->getMessage()); + } + } + } + } + + /** @test O-13 O-14 O-29 */ + public function insertAndNameEdgeCases(): void + { + $db = $this->db(); + $r = $db->insert('ie', [0 => ['a' => 1], 'name' => 'x']); + $this->assertSame(0, $r['n']); + $this->assertArrayHasKey('error', $r); + $this->assertSame(0, $db->insert('ie', [])['n']); + $this->assertSame(['n' => 2], $db->insert('ie', [['a' => 1], ['a' => 2]])); + $this->assertSame(['n' => 1], $db->insert('0', ['z' => 1])); + $this->assertSame([['z' => 1, 'key' => 0]], $db->find('0', 0)); + $this->assertCount(2, $db->find('ie', null)); + $this->assertFalse($db->find('ie', 'not-a-filter')); + } + + /** @test D-05 D-06 */ + public function smallApiFixes(): void + { + $db = $this->db(); + $this->assertSame([], $db->limit([], 2)); + $this->assertSame([], $db->sort([], 'a')); + $this->assertSame('Please check your delete parameters', $db->delete('x', 'bad')['error']); + $this->assertSame('1,5 KB', $this->getPrivateMethod('fileSizeConvert')->invoke($this->noneDB, 1536), 'documented format'); + } + + /** @test O-33 */ + public function queryWritesRespectSortAndLimit(): void + { + $db = $this->db(); + $this->seed($db); + $this->assertSame(1, $db->query('q')->where(['city' => 'Istanbul'])->sort('name', 'desc')->limit(1)->delete()['n']); + $this->assertSame(['Alice'], array_column($db->find('q', ['city' => 'Istanbul']), 'name')); + } + + /** @test P-07 */ + public function queryBuilderDoesNotUseReflection(): void + { + $source = file_get_contents(__DIR__ . '/../../noneDB.php'); + $query = substr($source, strpos($source, 'class noneDBQuery')); + $this->assertStringNotContainsString('ReflectionClass', $query); + } +} diff --git a/tests/Regression/RegressionTestCase.php b/tests/Regression/RegressionTestCase.php new file mode 100644 index 0000000..9aad4a3 --- /dev/null +++ b/tests/Regression/RegressionTestCase.php @@ -0,0 +1,141 @@ +tempFiles as $file) { + @unlink($file); + } + $this->tempFiles = []; + \noneDB::enableStaticCache(); + parent::tearDown(); + } + + /** + * New instance on the test directory with extra config + */ + protected function db(array $config = []): \noneDB + { + return new \noneDB(array_merge([ + 'secretKey' => self::SECRET, + 'dbDir' => $this->testDbDir, + 'autoCreateDB' => true, + ], $config)); + } + + /** + * PHP expression creating a noneDB instance in a child process + */ + protected function childDb(array $config = []): string + { + $config = array_merge(['secretKey' => self::SECRET, 'dbDir' => $this->testDbDir], $config); + return 'new \noneDB(' . var_export($config, true) . ')'; + } + + /** + * Write a child script (body runs after noneDB.php is loaded) and return its path + */ + protected function childScript(string $body): string + { + $path = sys_get_temp_dir() . '/nonedb_regression_' . bin2hex(random_bytes(6)) . '.php'; + $repo = realpath(__DIR__ . '/../../noneDB.php'); + file_put_contents($path, "tempFiles[] = $path; + return $path; + } + + /** + * Run PHP code in a separate process and return its stdout (stderr appended) + */ + protected function runChild(string $body, array $args = []): string + { + $script = $this->childScript($body); + $cmd = escapeshellarg(PHP_BINARY) . ' ' . escapeshellarg($script); + foreach ($args as $arg) { + $cmd .= ' ' . escapeshellarg((string)$arg); + } + return trim((string)shell_exec($cmd . ' 2>&1')); + } + + /** + * Run the same script in $count processes at once (argv[1] = process number) and wait for all + * @return string[] outputs + */ + protected function runParallel(string $body, int $count): array + { + $script = $this->childScript($body); + $procs = []; + $pipesList = []; + for ($i = 1; $i <= $count; $i++) { + $cmd = [PHP_BINARY, $script, (string)$i]; + $pipes = []; + $proc = proc_open( + implode(' ', array_map('escapeshellarg', $cmd)) . ' 2>&1', + [1 => ['pipe', 'w']], + $pipes + ); + $procs[$i] = $proc; + $pipesList[$i] = $pipes; + } + $outputs = []; + foreach ($procs as $i => $proc) { + $outputs[$i] = trim((string)stream_get_contents($pipesList[$i][1])); + fclose($pipesList[$i][1]); + proc_close($proc); + } + return $outputs; + } + + /** + * Base path {dbDir}{hash}-{name} of a database for the test secret + */ + protected function dbBasePath(string $dbName): string + { + $hash = hash_pbkdf2('sha256', $dbName, self::SECRET, 1000, 20); + return $this->testDbDir . $hash . '-' . $dbName; + } + + /** + * State record of a database's lock file: ['gen' => int, 'flags' => string[]] + */ + protected function lockState(string $dbName): array + { + $tokens = explode(' ', trim((string)file_get_contents($this->dbBasePath($dbName) . '.nonedb.lock'))); + return ['gen' => (int)array_shift($tokens), 'flags' => array_values(array_filter($tokens, 'strlen'))]; + } + + /** + * Turn a database into what a pre-3.2 version leaves behind as far as the lock is concerned: + * those versions had no lock file (and no state record) + */ + protected function removeLockFile(string $dbName): void + { + \noneDB::clearStaticCache(); // closes the kept lock handles + $this->assertTrue(unlink($this->dbBasePath($dbName) . '.nonedb.lock')); + } + + /** + * Records sorted by key, without "key" + */ + protected function plain(array $records): array + { + usort($records, function ($a, $b) { return $a['key'] <=> $b['key']; }); + return array_map(function ($r) { unset($r['key']); return $r; }, $records); + } +} diff --git a/tests/Regression/SecurityConfigRegressionTest.php b/tests/Regression/SecurityConfigRegressionTest.php new file mode 100644 index 0000000..83a7ab3 --- /dev/null +++ b/tests/Regression/SecurityConfigRegressionTest.php @@ -0,0 +1,371 @@ +scratch('s01') . '/'; + $this->runChild('(new \noneDB(["secretKey" => "alpha-key", "dbDir" => ' . var_export($dir, true) . ']))->insert("users", ["pin" => "1234"]);'); + $wrong = $this->runChild('echo json_encode((new \noneDB(["secretKey" => "WRONG", "dbDir" => ' . var_export($dir, true) . ', "autoCreateDB" => false]))->find("users", 0));'); + $this->assertSame('false', $wrong); + foreach (['null', "''", 'false'] as $bad) { + $out = $this->runChild('try { new \noneDB(["secretKey" => ' . $bad . ', "dbDir" => ' . var_export($dir, true) . ']); echo "ACCEPTED"; } catch (\InvalidArgumentException $e) { echo "REJECTED"; }'); + $this->assertSame('REJECTED', $out, "key $bad"); + } + } + + /** @test S-02 */ + public function wrongKeyRunDoesNotPoisonTheCache(): void + { + $dir = $this->scratch('s02') . '/'; + $mk = function ($key) use ($dir) { return '(new \noneDB(["secretKey" => "' . $key . '", "dbDir" => ' . var_export($dir, true) . ']))'; }; + $this->runChild($mk('alpha-key') . '->insert("users", ["pin" => "1234"]);'); + foreach (glob($dir . '.nonedb_hash_cache*') as $f) { + unlink($f); + } + $this->runChild($mk('WRONG') . '->find("users", 0);'); + $right = $this->runChild('echo json_encode(' . $mk('alpha-key') . '->find("users", 0));'); + $this->assertSame('[{"pin":"1234","key":0}]', $right); + $this->runChild($mk('alpha-key') . '->insert("users", ["pin" => "5678"]);'); + foreach (glob($dir . '.nonedb_hash_cache*') as $f) { + unlink($f); + } + $this->assertSame('[{"pin":"1234","key":0},{"pin":"5678","key":1}]', $this->runChild('echo json_encode(' . $mk('alpha-key') . '->find("users", 0));')); + } + + /** @test S-03 S-14 */ + public function tenantsInOneProcessStayIsolated(): void + { + $base = $this->scratch('s03'); + $out = $this->runChild(' + $B = new \noneDB(["secretKey" => "tenantB-key", "dbDir" => ' . var_export("$base/B/", true) . ']); + $B->insert("orders", ["o" => 1]); + $A = new \noneDB(["secretKey" => "tenantA-key", "dbDir" => ' . var_export("$base/A/", true) . ']); + $A->insert("payroll-2026-ceo-bonus", ["x" => 1]); + $A->insert("users", ["u" => "a"]); + $B->insert("users", ["u" => "b"]); + echo json_encode($B->find("users", 0)), json_encode($A->find("users", 0));'); + $this->assertSame('[{"u":"b","key":0}][{"u":"a","key":0}]', $out); + $expectedB = hash_pbkdf2('sha256', 'users', 'tenantB-key', 1000, 20); + $this->assertFileExists("$base/B/$expectedB-users.nonedb"); + $cache = ''; + foreach (glob("$base/B/.nonedb_hash_cache_*") as $f) { + $cache .= file_get_contents($f); + $this->assertSame('0600', substr(sprintf('%o', fileperms($f)), -4)); + } + $this->assertStringNotContainsString('payroll', $cache); + $this->assertFileDoesNotExist("$base/B/.nonedb_hash_cache"); + } + + /** @test S-04 */ + public function invalidKeysAreRejected(): void + { + foreach ([null, false, '', ' ', 12345, ['a'], 'CHANGE_THIS_TO_A_SECURE_RANDOM_STRING'] as $key) { + try { + new \noneDB(['secretKey' => $key, 'dbDir' => $this->testDbDir]); + $this->fail('accepted ' . var_export($key, true)); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('secretKey', $e->getMessage()); + } + } + $this->assertInstanceOf(\noneDB::class, new \noneDB(['secretKey' => 'short', 'dbDir' => $this->testDbDir])); + } + + /** @test S-05 S-06 S-16 Y-23 D-02 D-03 */ + public function configFileResolution(): void + { + $p = $this->scratch('proj'); + mkdir("$p/public"); + mkdir("$p/bin"); + copy(__DIR__ . '/../../noneDB.php', "$p/noneDB.php"); + file_put_contents("$p/.nonedb", json_encode(['secretKey' => 's3cr3t-long-random', 'dbDir' => './db/'])); + file_put_contents("$p/public/index.php", 'insert("orders", ["from" => "web"]); echo json_encode($db->find("orders", 0));'); + file_put_contents("$p/bin/cron.php", 'find("orders", 0));'); + file_put_contents("$p/bin/mixed.php", ' __DIR__."/../db/"]))->find("orders", 0));'); + $php = escapeshellarg(PHP_BINARY); + $web = trim(shell_exec("$php " . escapeshellarg("$p/public/index.php") . ' 2>&1')); + $cron = trim(shell_exec("$php " . escapeshellarg("$p/bin/cron.php") . ' 2>&1')); + $fromParent = trim(shell_exec('cd ' . escapeshellarg($p) . " && $php bin/cron.php 2>&1")); + $mixed = trim(shell_exec("$php " . escapeshellarg("$p/bin/mixed.php") . ' 2>&1')); + $this->assertSame('[{"from":"web","key":0}]', $web); + $this->assertSame($web, $cron, 'web and cron share one database'); + $this->assertSame($web, $fromParent, 'relative script path, other cwd'); + $this->assertSame($web, $mixed, 'array config merges over the file (secretKey from file)'); + $this->assertCount(1, glob("$p/db/*-orders.nonedb")); + $this->assertFileExists("$p/db/.htaccess"); + $this->assertFileExists("$p/db/index.html"); + $this->assertSame([], glob("$p/public/db") ?: []); + } + + /** @test S-05 S-16: a relative dbDir without "./" follows the same rule */ + public function plainRelativeDbDirFollowsTheConfigFile(): void + { + $p = $this->scratch('plain'); + mkdir("$p/public"); + mkdir("$p/bin"); + copy(__DIR__ . '/../../noneDB.php', "$p/noneDB.php"); + file_put_contents("$p/.nonedb", json_encode(['secretKey' => 's3cr3t-long-random', 'dbDir' => 'data'])); + file_put_contents("$p/public/index.php", 'insert("orders", ["from" => "web"]); echo json_encode($db->find("orders", 0));'); + file_put_contents("$p/bin/cron.php", 'find("orders", 0));'); + // Array config: relative to the entry script (bin/), not to the working directory + file_put_contents("$p/bin/array.php", ' "other-key-123", "dbDir" => "store/"]); $db->insert("a", ["x" => 1]); echo "ok";'); + $php = escapeshellarg(PHP_BINARY); + $web = trim(shell_exec('cd ' . escapeshellarg("$p/public") . " && $php index.php 2>&1")); + $cron = trim(shell_exec('cd ' . escapeshellarg(sys_get_temp_dir()) . " && $php " . escapeshellarg("$p/bin/cron.php") . ' 2>&1')); + $this->assertSame('[{"from":"web","key":0}]', $web); + $this->assertSame($web, $cron, 'a cwd elsewhere does not matter'); + $this->assertCount(1, glob("$p/data/*-orders.nonedb")); + $this->assertSame([], glob("$p/public/data") ?: []); + $this->assertSame('ok', trim(shell_exec('cd ' . escapeshellarg($p) . " && $php bin/array.php 2>&1"))); + $this->assertCount(1, glob("$p/bin/store/*-a.nonedb")); + $this->assertSame([], glob("$p/store") ?: []); + } + + /** @test S-05: data in a pre-3.2 location keeps being used (with a deprecation notice) */ + public function oldDbDirLocationsKeepWorking(): void + { + foreach (['./old/' => 'script', 'old/' => 'cwd'] as $dbDir => $oldBase) { + $p = realpath($this->scratch('legacy')); + mkdir("$p/app/web", 0777, true); + mkdir("$p/cwd"); + copy(__DIR__ . '/../../noneDB.php', "$p/app/noneDB.php"); + file_put_contents("$p/app/.nonedb", json_encode(['secretKey' => 's3cr3t-long-random', 'dbDir' => $dbDir])); + $script = 'insert("t", ["v" => 1]); echo json_encode(array_column($db->find("t", 0), "v"));'; + file_put_contents("$p/app/web/run.php", $script); + // Where v3.1 put the data: next to the entry script ("./old/") or in the working directory ("old/") + $old = $oldBase === 'script' ? "$p/app/web/old" : "$p/cwd/old"; + mkdir($old); + $out = trim(shell_exec('cd ' . escapeshellarg("$p/cwd") . ' && ' . escapeshellarg(PHP_BINARY) . ' ' . escapeshellarg("$p/app/web/run.php") . ' 2>&1')); + $this->assertStringContainsString('NOTICE:noneDB: dbDir', $out, $dbDir); + $this->assertStringContainsString("using the old location $old", $out, $dbDir); + $this->assertStringEndsWith('[1]', $out, $dbDir); + $this->assertCount(1, glob("$old/*-t.nonedb"), $dbDir); + $this->assertDirectoryDoesNotExist("$p/app/old", $dbDir); + } + } + + /** @test S-04 S-11: an explicitly given invalid secretKey is rejected whatever the environment */ + public function givenInvalidKeyIsRejectedWithoutConfigFileAndInDevMode(): void + { + $dir = $this->scratch('nokey'); + copy(__DIR__ . '/../../noneDB.php', "$dir/noneDB.php"); // no .nonedb anywhere near it + $script = ' $k, "dbDir" => __DIR__ . "/db/"]); echo "ACCEPTED "; } + catch (InvalidArgumentException $e) { echo "REJECTED "; } + catch (Throwable $e) { echo get_class($e), " "; } + } + try { new noneDB(["dbDir" => __DIR__ . "/db/"]); echo "NOKEY:", noneDB::isDevMode() ? "dev" : "file"; } + catch (Throwable $e) { echo "NOKEY:", get_class($e); }'; + // Without config file: an invalid key is reported as such (not as a missing config file). + // Dev mode: an invalid key is not silently replaced by the predictable dev mode key. + foreach (['' => 'NOKEY:RuntimeException', 'define("NONEDB_DEV_MODE", true);' => 'NOKEY:dev'] as $prefix => $noKey) { + file_put_contents("$dir/run.php", sprintf($script, $prefix)); + $out = trim((string)shell_exec('cd ' . escapeshellarg($dir) . ' && env -u NONEDB_DEV_MODE ' . escapeshellarg(PHP_BINARY) . ' run.php 2>&1')); + $this->assertSame("REJECTED REJECTED REJECTED $noKey", $out, $prefix === '' ? 'no config file' : 'dev mode'); + } + } + + /** @test S-07 */ + public function invalidDatabaseNamesAreRejected(): void + { + $db = $this->db(); + $db->insert('alan', ['tip' => 'alan']); + foreach (['çalışan', 'my_db', '../../x', "a\0b", 'a/b', ''] as $name) { + try { + $db->insert($name, ['x' => 1]); + $this->fail('accepted ' . $name); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString('invalid database name', $e->getMessage()); + } + } + $this->assertCount(1, $db->find('alan', 0)); + $this->assertSame(["Bob's list", 'a b-c', '0'], array_map(function ($n) use ($db) { $db->insert($n, ['ok' => 1]); return $db->find($n, 0) ? $n : null; }, ["Bob's list", 'a b-c', '0'])); + } + + /** @test S-09 */ + public function secretNeverAppearsInDebugOutput(): void + { + $db = new \noneDB(['secretKey' => 'TOP-SECRET-123', 'dbDir' => $this->testDbDir]); + $q = $db->query('x'); + ob_start(); + var_dump($db, $q); + $dump = ob_get_clean() . print_r($db, true) . print_r($q, true) . var_export($db, true) . var_export($q, true); + $this->assertStringNotContainsString('TOP-SECRET-123', $dump); + $this->expectException(\LogicException::class); + serialize($db); + } + + /** @test S-09 */ + public function clonesKeepTheirKey(): void + { + $db = new \noneDB(['secretKey' => 'clone-key', 'dbDir' => $this->testDbDir]); + $db->insert('cl', ['a' => 1]); + $copy = clone $db; + unset($db); + gc_collect_cycles(); + $this->assertCount(1, $copy->find('cl', 0)); + } + + /** @test S-11 Y-25 */ + public function devModeInstancesAreConfigured(): void + { + $dir = $this->scratch('dev'); + copy(__DIR__ . '/../../noneDB.php', "$dir/noneDB.php"); + file_put_contents("$dir/run.php", 'insert("t", ["a" => 1]); echo count($a->find("t", 0));'); + $out = trim(shell_exec('cd ' . escapeshellarg(sys_get_temp_dir()) . ' && env -u NONEDB_DEV_MODE ' . escapeshellarg(PHP_BINARY) . ' ' . escapeshellarg("$dir/run.php") . ' 2>&1')); + $this->assertSame('1', $out); + $this->assertCount(1, glob("$dir/db/*-t.nonedb"), 'data next to noneDB.php, not in the working directory'); + } + + /** @test S-12 */ + public function readsNeverCreateFiles(): void + { + $db = $this->db(); + $before = glob($this->testDbDir . '*'); + $this->assertSame([], $db->find('ghost', 0)); + $this->assertSame(0, $db->count('ghost')); + $this->assertSame([], $db->query('ghost')->where(['a' => 1])->get()); + $this->assertSame([], $db->withinDistance('ghost', 'loc', 0, 0, 10)); + $this->assertFalse($this->db(['autoCreateDB' => false])->find('ghost', 0)); + $this->assertFalse($db->getDBs('ghost')); + $this->assertFalse($db->getShardInfo('ghost')); + $this->assertSame(['fields' => [], 'sharded' => false], $db->getFieldIndexes('ghost')); + $this->assertSame($before, glob($this->testDbDir . '*')); + } + + /** @test S-15 */ + public function dropDbRemovesEveryFile(): void + { + $db = $this->db(['shardSize' => 3]); + $db->insert('dd', [['i' => 1, 'loc' => ['type' => 'Point', 'coordinates' => [1, 1]]], ['i' => 2], ['i' => 3], ['i' => 4]]); + $db->createFieldIndex('dd', 'i'); + $db->createSpatialIndex('dd', 'loc'); + $r = $db->dropDB('dd'); + $this->assertTrue($r['success']); + $hash = hash_pbkdf2('sha256', 'dd', self::SECRET, 1000, 20); + $left = array_filter(glob($this->testDbDir . $hash . '*'), function ($f) { return substr($f, -5) !== '.lock'; }); + $this->assertSame([], array_values($left)); + $this->assertSame([], $db->find('dd', 0)); + $this->assertFalse($db->dropDB('dd')['success']); + $db->insert('dd', ['fresh' => true]); + $this->assertSame([['fresh' => true, 'key' => 0]], $db->find('dd', 0)); + } + + /** @test S-04 migrateKey */ + public function migrateKeyMovesEverything(): void + { + $dir = $this->scratch('mk') . '/'; + $old = new \noneDB(['secretKey' => 'old-key-abc', 'dbDir' => $dir, 'shardSize' => 5]); + $old->insert('plain', [['city' => 'A'], ['city' => 'B']]); + $old->createFieldIndex('plain', 'city'); + $old->insert('geo', ['loc' => ['type' => 'Point', 'coordinates' => [29.0, 41.0]]]); + $old->createSpatialIndex('geo', 'loc'); + $rows = []; + for ($i = 0; $i < 12; $i++) { + $rows[] = ['i' => $i]; + } + $old->insert('sharded', $rows); + unset($old); + \noneDB::clearStaticCache(); + + $r = \noneDB::migrateKey($dir, 'old-key-abc', 'brand-new-key-0123456789abcdef'); + sort($r['databases']); + $this->assertTrue($r['success']); + $this->assertSame(['geo', 'plain', 'sharded'], $r['databases']); + + $new = new \noneDB(['secretKey' => 'brand-new-key-0123456789abcdef', 'dbDir' => $dir, 'autoCreateDB' => false]); + $this->assertSame([1], array_column($new->find('plain', ['city' => 'B']), 'key')); + $this->assertSame(['city'], $new->getFieldIndexes('plain')['fields']); + $this->assertSame(11, $new->find('sharded', ['key' => 11])[0]['i']); + $this->assertCount(1, $new->withinDistance('geo', 'loc', 29.0, 41.0, 100)); + \noneDB::clearStaticCache(); + $this->assertFalse((new \noneDB(['secretKey' => 'old-key-abc', 'dbDir' => $dir, 'autoCreateDB' => false]))->find('plain', 0)); + $this->assertFalse(\noneDB::migrateKey($dir, 'old-key-abc', 'CHANGE_THIS_TO_A_SECURE_RANDOM_STRING')['success']); + foreach (['plain', 'geo', 'sharded'] as $name) { + $this->assertSame([], glob($dir . hash_pbkdf2('sha256', $name, 'old-key-abc', 1000, 20) . '*'), "nothing of $name left under the old key"); + } + } + + /** @test S-04: databases from before v3.2 (no lock files) leave nothing behind under the old key */ + public function migrateKeyOfPreLockDatabases(): void + { + $dir = $this->scratch('mk31') . '/'; + $old = new \noneDB(['secretKey' => 'old-key-abc', 'dbDir' => $dir, 'shardSize' => 5]); + $rows = []; + for ($i = 0; $i < 12; $i++) { + $rows[] = ['i' => $i]; + } + $old->insert('users', $rows); + unset($old); + \noneDB::clearStaticCache(); // closes kept lock handles + foreach (glob($dir . '*.nonedb.lock') as $lock) { + $this->assertTrue(unlink($lock)); // pre-3.2 versions had no lock files + } + $oldHash = hash_pbkdf2('sha256', 'users', 'old-key-abc', 1000, 20); + $before = count(glob($dir . $oldHash . '*')); + + $r = \noneDB::migrateKey($dir, 'old-key-abc', 'brand-new-key-0123456789abcdef'); + $this->assertTrue($r['success']); + $this->assertSame($before, $r['files']); + $this->assertSame([], glob($dir . $oldHash . '*'), 'no lock file (or anything else) left under the old key'); + $new = new \noneDB(['secretKey' => 'brand-new-key-0123456789abcdef', 'dbDir' => $dir, 'autoCreateDB' => false]); + $this->assertSame(range(0, 11), array_column($new->find('users', 0), 'i')); + } + + /** @test O-19 */ + public function instancesAreReleased(): void + { + $out = $this->runChild('$m = memory_get_usage(); for ($i = 0; $i < 20000; $i++) { $x = ' . $this->childDb() . '; } unset($x); gc_collect_cycles(); echo memory_get_usage() - $m;'); + $this->assertLessThan(1024 * 1024, (int)$out); + } + + /** @test O-20 */ + public function includingDoesNotChangeMemoryLimit(): void + { + $this->assertNotSame('-1', $this->runChild('echo ini_get("memory_limit");')); + } + + /** @test D-01 */ + public function invalidConfigValuesAreRejected(): void + { + foreach (['shardSize' => 0, 'lockTimeout' => -1, 'lockRetryDelay' => 0, 'autoCompactThreshold' => 0] as $key => $value) { + try { + $this->db([$key => $value]); + $this->fail("accepted $key"); + } catch (\InvalidArgumentException $e) { + $this->assertStringContainsString($key, $e->getMessage()); + } + } + } + + /** @test S-10 */ + public function noFilePathsInWarnings(): void + { + $out = $this->runChild('ini_set("display_errors", "1"); $db = ' . $this->childDb() . '; $db->insert("w", ["a" => 1]); $db->find("w", ["key" => 99]); $db->getDBs("w"); $db->delete("w", ["a" => 2]); echo "done";'); + $this->assertSame('done', $out); + } +} diff --git a/tests/Regression/SpatialRegressionTest.php b/tests/Regression/SpatialRegressionTest.php new file mode 100644 index 0000000..e4a50e1 --- /dev/null +++ b/tests/Regression/SpatialRegressionTest.php @@ -0,0 +1,345 @@ + 'Point', 'coordinates' => [$lon, $lat]]; + } + + /** @test Y-01 */ + public function bufferedInsertsAreSpatiallyIndexed(): void + { + $db = $this->db(['bufferEnabled' => true]); + $db->insert('pl', ['n' => 'seed', 'loc' => $this->point(29.0, 41.0)]); + $db->createSpatialIndex('pl', 'loc'); + $db->insert('pl', ['n' => 'A', 'loc' => $this->point(29.0001, 41.0001)]); + $db->insert('pl', ['n' => 'B', 'loc' => $this->point(29.0002, 41.0)]); + $this->assertSame(['seed', 'A', 'B'], array_column($db->withinDistance('pl', 'loc', 29.0, 41.0, 1000, ['includeDistance' => true]), 'n')); + $this->assertCount(3, $db->withinBBox('pl', 'loc', 28.99, 40.99, 29.01, 41.01)); + $this->assertCount(3, $db->query('pl')->withinDistance('loc', 29.0, 41.0, 1000)->get()); + } + + /** @test Y-02 Y-05 */ + public function shardedSpatialIndexWorksAndCoversNewShards(): void + { + $db = $this->db(['shardSize' => 50]); + $rows = []; + for ($i = 0; $i < 120; $i++) { + $rows[] = ['i' => $i, 'loc' => $this->point(29 + $i / 1000, 41)]; + } + $db->insert('sh', $rows); + $r = $db->createSpatialIndex('sh', 'loc'); + $this->assertTrue($r['success']); + $this->assertSame(120, $r['indexed']); + $this->assertFalse($db->createSpatialIndex('sh', 'loc')['success'], 'second create reports "already exists"'); + + $more = []; + for ($i = 120; $i < 200; $i++) { + $more[] = ['i' => $i, 'loc' => $this->point(29 + $i / 1000, 41)]; + } + $db->insert('sh', $more); + $this->assertCount(200, $db->withinBBox('sh', 'loc', 28.9, 40.9, 29.3, 41.1)); + $this->assertSame(199, $db->nearest('sh', 'loc', 29.199, 41.0, 1)[0]['i']); + $this->assertTrue($db->hasSpatialIndex('sh', 'loc')); + $this->assertSame(['loc'], $db->getSpatialIndexes('sh')); + } + + /** @test Y-04 */ + public function migrationKeepsTheSpatialIndex(): void + { + $db = $this->db(['shardSize' => 30]); + $db->insert('mg', ['n' => 'first', 'loc' => $this->point(10, 10)]); + $db->createSpatialIndex('mg', 'loc'); + $rows = []; + for ($i = 0; $i < 40; $i++) { + $rows[] = ['n' => 'p' . $i, 'loc' => $this->point(10 + $i / 100, 10)]; + } + $db->insert('mg', $rows); // crosses the threshold: migrates + $this->assertTrue($db->getShardInfo('mg')['sharded']); + $this->assertSame(['loc'], $db->getSpatialIndexes('mg')); + $this->assertTrue($db->hasSpatialIndex('mg', 'loc')); + $this->assertCount(41, $db->withinBBox('mg', 'loc', 9.9, 9.9, 10.5, 10.1)); + $this->assertFileDoesNotExist($this->testDbDir . hash_pbkdf2('sha256', 'mg', self::SECRET, 1000, 20) . '.nonedb.sidx.loc'); + $this->assertTrue($db->rebuildSpatialIndex('mg', 'loc')['success']); + $this->assertCount(41, $db->withinBBox('mg', 'loc', 9.9, 9.9, 10.5, 10.1)); + } + + /** @test Y-03 P-04: R-tree never hangs and agrees with brute force */ + public function randomizedRtreeMatchesBruteForce(): void + { + mt_srand(7); + $db = $this->db(); + $rows = []; + // three clusters + foreach ([[29, 41], [32.8, 39.9], [27.1, 38.4]] as $c => $center) { + for ($i = 0; $i < 700; $i++) { + $rows[] = ['c' => $c, 'loc' => $this->point($center[0] + mt_rand(-1000, 1000) / 10000, $center[1] + mt_rand(-1000, 1000) / 10000)]; + } + } + $db->insert('rt', $rows); + $db->createSpatialIndex('rt', 'loc'); + $db->delete('rt', ['c' => 1]); // remove a whole cluster + $start = microtime(true); + $db->insert('rt', ['c' => 9, 'loc' => $this->point(32.8, 39.9)]); // used to loop forever + $this->assertLessThan(5, microtime(true) - $start); + for ($i = 0; $i < 150; $i++) { + $db->update('rt', [['key' => mt_rand(0, 2099)], ['set' => ['loc' => $this->point(26 + mt_rand(0, 8000) / 1000, 37 + mt_rand(0, 5000) / 1000)]]]); + } + + $all = $db->find('rt', 0); + for ($q = 0; $q < 25; $q++) { + $lon = 26 + mt_rand(0, 8000) / 1000; + $lat = 37 + mt_rand(0, 5000) / 1000; + $radius = mt_rand(1000, 150000); + $expected = []; + foreach ($all as $r) { + if ($db->haversineDistance($lon, $lat, $r['loc']['coordinates'][0], $r['loc']['coordinates'][1]) <= $radius) { + $expected[] = $r['key']; + } + } + $got = array_column($db->withinDistance('rt', 'loc', $lon, $lat, $radius), 'key'); + sort($expected); + sort($got); + $this->assertSame($expected, $got, "withinDistance query $q"); + + $box = [$lon - 0.5, $lat - 0.3, $lon + 0.5, $lat + 0.3]; + $expectedBox = []; + foreach ($all as $r) { + [$x, $y] = $r['loc']['coordinates']; + if ($x >= $box[0] && $x <= $box[2] && $y >= $box[1] && $y <= $box[3]) { + $expectedBox[] = $r['key']; + } + } + $gotBox = array_column($db->withinBBox('rt', 'loc', ...$box), 'key'); + sort($expectedBox); + sort($gotBox); + $this->assertSame($expectedBox, $gotBox, "withinBBox query $q"); + + $dist = []; + foreach ($all as $r) { + $dist[$r['key']] = $db->haversineDistance($lon, $lat, $r['loc']['coordinates'][0], $r['loc']['coordinates'][1]); + } + asort($dist); + $nearest = $db->nearest('rt', 'loc', $lon, $lat, 5); + $this->assertEqualsWithDelta(array_slice(array_values($dist), 0, 5), array_column($nearest, '_distance'), 0.01, "nearest query $q"); + } + } + + /** @test Y-06 */ + public function antimeridianAndPolesAreSearched(): void + { + $db = $this->db(); + $db->insert('am', [ + ['n' => 'east', 'loc' => $this->point(179.99, 0)], + ['n' => 'west', 'loc' => $this->point(-179.99, 0.001)], + ['n' => 'pole-a', 'loc' => $this->point(0, 89.5)], + ['n' => 'pole-b', 'loc' => $this->point(180, 89.5)], + ]); + $db->createSpatialIndex('am', 'loc'); + $this->assertSame(['west', 'east'], array_column($db->withinDistance('am', 'loc', -179.99, 0, 5000, ['includeDistance' => true]), 'n')); + $nearPole = array_column($db->withinDistance('am', 'loc', 0, 88.5, 300000), 'n'); + sort($nearPole); + $this->assertSame(['pole-a', 'pole-b'], $nearPole); + $this->assertSame(['east', 'west'], array_column($db->withinBBox('am', 'loc', 179.9, -1, -179.9, 1), 'n'), 'box crossing the antimeridian'); + } + + /** @test Y-07 */ + public function nearestRespectsMaxDistanceAndReturnsPartialResults(): void + { + $db = $this->db(); + $db->insert('nr', [ + ['n' => 'a', 'loc' => $this->point(29.0, 41.0)], + ['n' => 'b', 'loc' => $this->point(29.002, 41.0)], + ['n' => 'c', 'loc' => $this->point(29.004, 41.0)], + ['n' => 'far', 'loc' => $this->point(35, 41)], + ]); + foreach ([false, true] as $indexed) { + if ($indexed) { + $db->createSpatialIndex('nr', 'loc'); + } + $this->assertSame(['a', 'b', 'c'], array_column($db->nearest('nr', 'loc', 29.0, 41.0, 5, ['maxDistance' => 1000]), 'n')); + $this->assertSame(['a'], array_column($db->nearest('nr', 'loc', 29.0, 41.0, 1, ['maxDistance' => 100]), 'n')); + $this->assertSame(['far'], array_column($db->nearest('nr', 'loc', 35, 41, 1), 'n')); + $this->assertSame([], $db->nearest('nr', 'loc', 29, 41, 0)); + $this->assertCount(4, $db->nearest('nr', 'loc', 29, 41, 10)); + } + } + + /** @test Y-08 */ + public function spatialFieldNamesWithAnyCharacters(): void + { + $db = $this->db(); + $db->insert('sf', ['n' => 'first', 'home-loc' => $this->point(1, 1), 'homeloc' => $this->point(50, 50)]); + $db->createSpatialIndex('sf', 'home-loc'); + $db->createSpatialIndex('sf', 'homeloc'); + $db->insert('sf', ['n' => 'second', 'home-loc' => $this->point(1.001, 1), 'homeloc' => $this->point(50, 50)]); + $this->assertSame(['first', 'second'], array_column($db->withinDistance('sf', 'home-loc', 1, 1, 1000), 'n')); + $this->assertCount(2, $db->withinDistance('sf', 'homeloc', 50, 50, 10)); + $this->assertEqualsCanonicalizing(['home-loc', 'homeloc'], $db->getSpatialIndexes('sf')); + } + + /** @test O-24 */ + public function geometrySemantics(): void + { + $db = $this->db(); + // concave "C" shaped polygon: the notch is outside + $c = ['type' => 'Polygon', 'coordinates' => [[[0, 0], [10, 0], [10, 3], [3, 3], [3, 7], [10, 7], [10, 10], [0, 10], [0, 0]]]]; + $db->insert('gs', [ + ['n' => 'crossing-line', 'g' => ['type' => 'LineString', 'coordinates' => [[5, 2], [5, 8]]]], + ['n' => 'inside-line', 'g' => ['type' => 'LineString', 'coordinates' => [[1, 1], [1, 9]]]], + ['n' => 'big-poly', 'g' => ['type' => 'Polygon', 'coordinates' => [[[20, 20], [30, 20], [30, 30], [20, 30], [20, 20]]]]], + ]); + $this->assertSame(['inside-line'], array_column($db->withinPolygon('gs', 'g', $c), 'n')); + $multi = ['type' => 'MultiPolygon', 'coordinates' => [$c['coordinates'], [[[19, 19], [31, 19], [31, 31], [19, 31], [19, 19]]]]]; + $this->assertSame(['inside-line', 'big-poly'], array_column($db->withinPolygon('gs', 'g', $multi), 'n')); + // a point inside a large polygon is at distance 0 from it + $this->assertSame(['big-poly'], array_column($db->withinDistance('gs', 'g', 25, 25, 10), 'n')); + + $this->assertTrue($db->validateGeoJSON(['type' => 'Polygon', 'coordinates' => [[[29, 41], [30, 41], [30, 42], [29.0, 41.0]]]])['valid'], '29 == 29.0'); + $this->assertFalse($db->validateGeoJSON(['type' => 'Polygon', 'coordinates' => [[[0, 0], [10, 10], [10, 0], [0, 10], [0, 0]]]])['valid'], 'bow-tie'); + $this->assertFalse($db->validateGeoJSON(['type' => 'Polygon', 'coordinates' => [[[0, 0], [5, 0], [5, 5], [0, 5], [0, 0]], [[6, 6], [7, 6], [7, 7], [6, 6]]]])['valid'], 'hole outside'); + $this->assertFalse($db->validateGeoJSON($this->point(NAN, 1))['valid']); + $this->assertFalse($db->validateGeoJSON(['type' => 'Point', 'coordinates' => ['1', '2']])['valid'], 'strings are not coordinates'); + } + + /** @test O-34 P-05 */ + public function queriesWithoutHitsDoNotScan(): void + { + $db = $this->db(); + $rows = []; + for ($i = 0; $i < 3000; $i++) { + $rows[] = ['loc' => $this->point(29 + ($i % 100) / 1000, 41 + intdiv($i, 100) / 1000)]; + } + $db->insert('zh', $rows); + $db->createSpatialIndex('zh', 'loc'); + $start = microtime(true); + for ($i = 0; $i < 20; $i++) { + $this->assertSame([], $db->withinDistance('zh', 'loc', -70, -30, 1000)); + $this->assertSame([], $db->withinBBox('zh', 'loc', -71, -31, -69, -29)); + } + $this->assertLessThan(0.5, microtime(true) - $start); + } + + /** @test O-25 */ + public function spatialIndexFilesAreEncodedOnce(): void + { + $db = $this->db(); + $db->insert('en', ['loc' => $this->point(1, 1)]); + $db->createSpatialIndex('en', 'loc'); + $file = $this->testDbDir . hash_pbkdf2('sha256', 'en', self::SECRET, 1000, 20) . '.nonedb.sidx.loc'; + $decoded = json_decode(file_get_contents($file), true); + $this->assertIsArray($decoded); + $this->assertSame(3, $decoded['v']); + } + + /** @test D-08 */ + public function centroidOfPointNeedsNoHashing(): void + { + $method = $this->getPrivateMethod('getGeometryCentroid'); + $this->assertSame([28.9803, 41.0086], $method->invoke($this->noneDB, $this->point(28.9803, 41.0086))); + } + + /** @test D-08 the memo caches stay bounded and keep caching once full */ + public function memoCachesKeepWorkingWhenFull(): void + { + $db = $this->noneDB; + for ($i = 0; $i < 12000; $i++) { + $db->haversineDistance(0, 0, $i / 1000, 1); + } + $before = $this->getPrivateProperty('distanceCache'); + $this->assertLessThanOrEqual(10000, count($before)); + $d = $db->haversineDistance(10, 10, 20, 20); + $after = $this->getPrivateProperty('distanceCache'); + $this->assertLessThanOrEqual(10000, count($after)); + $this->assertSame([$d], array_values(array_diff_key($after, $before)), 'a new result is still cached'); + $this->assertSame($d, $db->haversineDistance(10, 10, 20, 20)); + + $centroid = $this->getPrivateMethod('getGeometryCentroid'); + for ($i = 0; $i < 6000; $i++) { + $centroid->invoke($db, ['type' => 'LineString', 'coordinates' => [[0, 0], [$i / 100, 1]]]); + } + $before = $this->getPrivateProperty('centroidCache'); + $this->assertLessThanOrEqual(5000, count($before)); + $line = ['type' => 'LineString', 'coordinates' => [[5, 5], [7, 9]]]; + $c = $centroid->invoke($db, $line); + $after = $this->getPrivateProperty('centroidCache'); + $this->assertLessThanOrEqual(5000, count($after)); + $this->assertSame([$c], array_values(array_diff_key($after, $before)), 'a new centroid is still cached'); + } + + /** + * @test Y-07: the expanding radius search must not cut through a rounding tie. A record just outside + * the first radius (1000.003 m) rounds to the same 1000.00 m as one just inside (999.998 m) and has the + * smaller key, so it comes first in distance order. + */ + public function nearestDoesNotCutThroughRoundingTies(): void + { + $at = function (float $meters) { + return ['type' => 'Point', 'coordinates' => [29.0, 41.0 + rad2deg($meters / 6371000.0)]]; + }; + foreach ([true, false] as $indexed) { + $name = $indexed ? 'tie-idx' : 'tie-scan'; + $db = $this->db(); + $db->insert($name, [['loc' => $at(1000.003)], ['loc' => $at(999.998)], ['loc' => $at(500)]]); + if ($indexed) { + $db->createSpatialIndex($name, 'loc'); + } + $distances = array_column($db->withinDistance($name, 'loc', 29.0, 41.0, 2000, ['includeDistance' => true]), '_distance', 'key'); + ksort($distances); + $this->assertSame([0 => 1000.0, 1 => 1000.0, 2 => 500.0], array_map('floatval', $distances), 'setup: both round to 1000.00'); + $this->assertCount(1, array_filter($db->withinDistance($name, 'loc', 29.0, 41.0, 1000), function ($r) { return $r['key'] === 1; }), 'setup: only key 1 lies within 1000 m'); + $this->assertSame([2, 0], array_column($db->nearest($name, 'loc', 29.0, 41.0, 2), 'key'), "$name direct"); + $this->assertSame([2, 0], array_column($db->query($name)->nearest('loc', 29.0, 41.0, 2)->get(), 'key'), "$name builder"); + $this->assertSame([2, 0], array_column($db->query($name)->nearest('loc', 29.0, 41.0, 2)->where(['loc' => ['$exists' => true]])->get(), 'key'), "$name builder + where"); + } + } + + /** @test O-05: query()->nearest(k) with other filters equals a brute-force answer (ties included) */ + public function nearestWithFiltersMatchesBruteForce(): void + { + mt_srand(4242); + $center = [29.0, 41.0]; + foreach ([['idx' => true, 'cfg' => []], ['idx' => false, 'cfg' => []], ['idx' => true, 'cfg' => ['shardSize' => 150]]] as $n => $case) { + $name = 'nb' . $n; + $db = $this->db($case['cfg']); + $rows = []; + for ($i = 0; $i < 600; $i++) { + if ($i % 10 === 0 && $i > 0) { + $p = $rows[$i - 1]['loc']['coordinates']; // same point again: equal distances + } elseif ($i % 10 === 1) { + $p = [2 * $center[0] - 29.01, 41.0]; // mirrored pairs at the same distance + $p = mt_rand(0, 1) ? $p : [29.01, 41.0]; + } else { + $p = [$center[0] + mt_rand(-20000, 20000) / 100000, $center[1] + mt_rand(-20000, 20000) / 100000]; + } + $rows[] = ['t' => mt_rand(0, 19), 'loc' => ['type' => 'Point', 'coordinates' => $p]]; + } + $db->insert($name, $rows); + if ($case['idx']) { + $db->createSpatialIndex($name, 'loc'); + } + $all = $db->find($name, 0); + foreach ([[1, 0], [5, 3], [10, 19], [7, 100], [40, 0]] as [$k, $tMax]) { + // Reference: distance rounded like the library, ties by key, filter, then the k cut + $expected = []; + foreach ($all as $r) { + if ($r['t'] <= $tMax) { + $d = round($db->haversineDistance($center[0], $center[1], $r['loc']['coordinates'][0], $r['loc']['coordinates'][1]), 2); + $expected[] = [$d, $r['key']]; + } + } + usort($expected, function ($a, $b) { return $a[0] <=> $b[0] ?: $a[1] <=> $b[1]; }); + $expected = array_column(array_slice($expected, 0, $k), 1); + $got = $db->query($name)->nearest('loc', $center[0], $center[1], $k)->where(['t' => ['$lte' => $tMax]])->get(); + $this->assertSame($expected, array_column($got, 'key'), "$name k=$k t<=$tMax"); + $this->assertSame($expected, array_column($db->query($name)->where(['t' => ['$lte' => $tMax]])->nearest('loc', $center[0], $center[1], $k)->get(), 'key'), "$name order of calls"); + } + } + } +} diff --git a/tests/Regression/StorageRegressionTest.php b/tests/Regression/StorageRegressionTest.php new file mode 100644 index 0000000..bb1ee94 --- /dev/null +++ b/tests/Regression/StorageRegressionTest.php @@ -0,0 +1,307 @@ +db()->insert('p', ['data' => [['title' => 'A'], ['title' => 'B']], 'owner' => 'orhan']); + $this->db()->insert('p', ['data' => 'v1', 'x' => 1]); + $out = $this->runChild('echo json_encode((' . $this->childDb() . ')->find("p", 0));'); + $this->assertSame( + '[{"data":[{"title":"A"},{"title":"B"}],"owner":"orhan","key":0},{"data":"v1","x":1,"key":1}]', + $out + ); + } + + /** @test K-02 */ + public function preJsonlFormatIsStillMigrated(): void + { + $base = $this->dbBasePath('legacy'); + file_put_contents($base . '.nonedb', json_encode(['data' => [['n' => 'a'], null, ['n' => 'c']]])); + file_put_contents($base . '.nonedbinfo', (string)time()); + $all = $this->db()->find('legacy', 0); + $this->assertSame([['n' => 'a', 'key' => 0], ['n' => 'c', 'key' => 2]], $all); + $this->db()->insert('legacy', ['n' => 'd']); + $this->assertSame(3, $this->db()->find('legacy', ['n' => 'd'])[0]['key'], 'next key continues after the old slots'); + } + + /** @test K-03 */ + public function unencodableDataIsRejectedWithoutTouchingFiles(): void + { + $db = $this->db(); + $db->insert('c', ['city' => 'Istanbul']); + $db->createFieldIndex('c', 'city'); + $files = glob($this->dbBasePath('c') . '*'); + $before = array_map('md5_file', $files); + + $r1 = $db->insert('c', ['city' => "Bad\xB1Latin1"]); + $r2 = $db->insert('c', ['v' => NAN]); + $r3 = $db->update('c', [['city' => 'Istanbul'], ['set' => ['city' => "x\xB1"]]]); + foreach ([$r1, $r2, $r3] as $r) { + $this->assertSame(0, $r['n']); + $this->assertArrayHasKey('error', $r); + } + clearstatcache(); + $this->assertSame($before, array_map('md5_file', $files), 'no file changed'); + + \noneDB::clearStaticCache(); + $this->assertSame([['city' => 'Istanbul', 'key' => 0]], $this->db()->find('c', ['city' => 'Istanbul'])); + $this->assertSame([['city' => 'Istanbul', 'key' => 0]], $this->db()->find('c', ['key' => 0])); + $this->assertTrue($this->db()->compact('c')['success']); + } + + /** @test Y-14 */ + public function missingOrCorruptIndexIsRebuiltFromData(): void + { + $db = $this->db(); + $db->insert('r', [['n' => 1], ['n' => 2], ['n' => 3], ['n' => 4]]); + $db->delete('r', ['n' => 2]); + $db->update('r', [['n' => 3], ['set' => ['n' => 33]]]); + $expected = $this->plain($db->find('r', 0)); + + foreach (['missing', 'corrupt', 'truncated'] as $mode) { + $idx = $this->dbBasePath('r') . '.nonedb.jidx'; + if ($mode === 'missing') { + unlink($idx); + } elseif ($mode === 'corrupt') { + file_put_contents($idx, '{"v":3,"o":{"0":[0,'); + } else { + file_put_contents($idx, ''); + } + \noneDB::clearStaticCache(); + $this->assertSame($expected, $this->plain($this->db()->find('r', 0)), "$mode index: deleted record stays deleted, update kept"); + $this->assertSame(3, $this->db()->count('r')); + } + $this->db()->insert('r', ['n' => 5]); + $this->assertSame(4, $this->db()->find('r', ['n' => 5])[0]['key'], 'key counter survives a rebuild'); + } + + /** @test Y-14 */ + public function tornLastLineIsCutOff(): void + { + $db = $this->db(); + $db->insert('t', [['n' => 1], ['n' => 2]]); + $data = $this->dbBasePath('t') . '.nonedb'; + file_put_contents($data, '{"n":3,"ke', FILE_APPEND); // crash in the middle of an append + unlink($data . '.jidx'); + \noneDB::clearStaticCache(); + $this->assertSame([['n' => 1], ['n' => 2]], $this->plain($this->db()->find('t', 0))); + $this->db()->insert('t', ['n' => 3]); + $this->assertSame([['n' => 1], ['n' => 2], ['n' => 3]], $this->plain($this->db()->find('t', 0))); + } + + /** @test Y-14 O-21 */ + public function crashBetweenCompactionRenamesIsDetected(): void + { + $db = $this->db(); + $rows = []; + for ($i = 0; $i < 20; $i++) { + $rows[] = ['n' => $i]; + } + $db->insert('k', $rows); + $db->delete('k', ['n' => 3]); + $oldIndex = file_get_contents($this->dbBasePath('k') . '.nonedb.jidx'); + $db->compact('k'); + // Simulate: new data file renamed into place, old index still on disk + file_put_contents($this->dbBasePath('k') . '.nonedb.jidx', $oldIndex); + \noneDB::clearStaticCache(); + $all = $this->db()->find('k', 0); + $this->assertCount(19, $all); + $this->assertSame(19, $this->db()->find('k', ['n' => 19])[0]['n']); + $this->assertSame([], $this->db()->find('k', ['n' => 3])); + } + + /** @test K-05 O-35 */ + public function shardedCompactKeepsKeysAndIndexes(): void + { + $db = $this->db(['shardSize' => 5]); + $rows = []; + for ($i = 0; $i < 15; $i++) { + $rows[] = ['i' => $i, 'city' => $i % 2 ? 'A' : 'B']; + } + $db->insert('s', $rows); + $db->createFieldIndex('s', 'city'); + for ($i = 0; $i < 10; $i++) { + $db->delete('s', ['i' => $i]); + } + $r = $db->compact('s'); + $this->assertTrue($r['success']); + $this->assertSame(5, $r['totalRecords']); + $this->assertSame(1, $db->getShardInfo('s')['shards'], 'empty shards are dropped'); + $this->assertSame(12, $db->find('s', ['key' => 12])[0]['i'], 'keys did not move'); + $this->assertSame([10, 12, 14], array_column($db->find('s', ['city' => 'B']), 'i')); + $db->insert('s', ['i' => 15, 'city' => 'B']); + $this->assertSame(15, $db->find('s', ['i' => 15])[0]['key']); + } + + /** @test K-05 */ + public function shardedCompactFlushesBuffersFirst(): void + { + $db = $this->db(['shardSize' => 5, 'bufferEnabled' => true]); + $rows = []; + for ($i = 0; $i < 12; $i++) { + $rows[] = ['i' => $i]; + } + $db->insert('sb', $rows); + $db->flush('sb'); + $db->insert('sb', ['i' => 12]); + $db->insert('sb', ['i' => 13]); + $db->compact('sb'); + $this->assertSame(14, $db->count('sb')); + $this->assertSame([], glob($this->dbBasePath('sb') . '*.buffer')); + } + + /** @test O-15 a buffered database switches to shards when its flush crosses shardSize */ + public function bufferedDatabaseShardsWhenFlushed(): void + { + $db = $this->db(['shardSize' => 10, 'bufferEnabled' => true]); + for ($i = 0; $i < 25; $i++) { + $db->insert('bs', ['i' => $i]); + } + $this->assertFileDoesNotExist($this->dbBasePath('bs') . '.nonedb.meta', 'still buffered'); + $this->assertSame(range(0, 24), array_column($db->find('bs', 0), 'i'), 'the read flushes'); + $this->assertFileExists($this->dbBasePath('bs') . '.nonedb.meta', 'and the flush migrated to shards'); + $this->assertSame(range(0, 24), array_column($db->find('bs', 0), 'key')); + $this->assertSame(25, $db->count('bs')); + $this->assertSame(3, $db->getShardInfo('bs')['shards']); + } + + /** @test O-16 manual migrate() takes buffered records along */ + public function manualMigrateFlushesTheBufferFirst(): void + { + $db = $this->db(['bufferEnabled' => true, 'shardingEnabled' => false]); + $db->insert('mb', ['i' => 0]); + $db->flush('mb'); + $db->insert('mb', [['i' => 1], ['i' => 2]]); + $this->assertSame('migrated', $db->migrate('mb')['status']); + $this->assertSame([], glob($this->dbBasePath('mb') . '*.buffer*')); + $this->assertSame([0, 1, 2], array_column($this->db()->find('mb', 0), 'i')); + $this->assertSame(3, $this->db()->count('mb')); + } + + /** @test K-06 D-10 */ + public function migrationNeverReusesKeys(): void + { + $db = $this->db(['shardSize' => 10, 'autoMigrate' => false]); + $rows = []; + for ($i = 0; $i < 15; $i++) { + $rows[] = ['i' => $i]; + } + $db->insert('mg', $rows); + $db->delete('mg', ['i' => 3]); + $db->delete('mg', ['i' => 13]); + $db->delete('mg', ['i' => 14]); // trailing keys deleted too + $this->assertSame('migrated', $db->migrate('mg')['status']); + $this->assertFalse(file_exists($this->dbBasePath('mg') . '.nonedb'), 'non-sharded file removed'); + $this->assertSame([], glob($this->dbBasePath('mg') . '.nonedb.backup')); + + $db->insert('mg', ['i' => 'NEW']); + $new = $db->find('mg', ['i' => 'NEW'])[0]; + $this->assertSame(15, $new['key'], 'next key continues after the highest ever used'); + $this->assertSame(12, $db->find('mg', ['key' => 12])[0]['i'], 'existing record untouched'); + $this->assertSame(13, $db->count('mg')); + } + + /** @test Y-09 */ + public function shardedKeyLookupSeesNewRecords(): void + { + $db = $this->db(['shardSize' => 5]); + $rows = []; + for ($i = 0; $i < 12; $i++) { + $rows[] = ['i' => $i]; + } + $db->insert('sk', $rows); + $this->assertSame(3, $db->find('sk', ['key' => 3])[0]['i']); + $db->insert('sk', ['i' => 12]); + $db->insert('sk', ['i' => 13]); + $this->assertSame(13, $db->find('sk', ['key' => 13])[0]['i']); + $this->assertSame(13, $db->query('sk')->where(['key' => 13])->first()['i']); + \noneDB::clearStaticCache(); + $this->assertSame(12, $this->db()->find('sk', ['key' => 12])[0]['i']); + } + + /** @test Y-10 */ + public function reopeningWithAnotherShardSizeKeepsKeys(): void + { + $a = $this->db(['shardSize' => 5]); + $rows = []; + for ($i = 0; $i < 12; $i++) { + $rows[] = ['i' => $i]; + } + $a->insert('rs', $rows); + \noneDB::clearStaticCache(); + $b = $this->db(['shardSize' => 10000]); + $this->assertSame(range(0, 11), array_column($b->find('rs', 0), 'key')); + $this->assertSame(7, $b->find('rs', ['key' => 7])[0]['i']); + $b->insert('rs', ['i' => 12]); + $this->assertSame(12, $b->find('rs', ['i' => 12])[0]['key']); + $this->assertSame(5, $b->getShardInfo('rs')['shardSize']); + } + + /** @test Y-16 */ + public function keyFiltersAreNotCoerced(): void + { + $db = $this->db(); + $db->insert('kc', [['a' => 'zero'], ['a' => 'one']]); + foreach ([null, 'abc', true, 1.9, -1, '1.0', [], ['x']] as $bad) { + $this->assertSame([], $db->find('kc', ['key' => $bad]), 'key ' . var_export($bad, true)); + } + $this->assertSame('one', $db->find('kc', ['key' => '1'])[0]['a'], 'digit strings are keys'); + $db->update('kc', [['key' => '1'], ['set' => ['b' => 2]]]); + $this->assertSame(1, $db->find('kc', ['a' => 'one', 'key' => 1])[0]['key']); + $this->assertIsInt($db->find('kc', ['key' => 1])[0]['key']); + } + + /** @test O-22 */ + public function compactionTriggersOnGarbageRatio(): void + { + $db = $this->db(); + $rows = []; + for ($i = 0; $i < 10; $i++) { + $rows[] = ['i' => $i]; + } + $db->insert('g', $rows); + $db->delete('g', ['i' => 0]); + $db->delete('g', ['i' => 1]); + $db->delete('g', ['i' => 2]); + $index = json_decode(file_get_contents($this->dbBasePath('g') . '.nonedb.jidx'), true); + $this->assertSame(3, $index['d'], '3 of 10 deleted: not above 30% yet'); + $db->delete('g', ['i' => 3]); + $index = json_decode(file_get_contents($this->dbBasePath('g') . '.nonedb.jidx'), true); + $this->assertSame(0, $index['d'], '4 of 10: compacted'); + $this->assertSame(6, $db->count('g')); + } + + /** @test O-23 */ + public function legacyNullSlotsAreNotCountedAsGarbage(): void + { + $base = $this->dbBasePath('lg'); + file_put_contents($base . '.nonedb', json_encode(['data' => [null, null, ['n' => 'a'], 'not-a-record']])); + file_put_contents($base . '.nonedbinfo', (string)time()); + $this->assertSame([['n' => 'a', 'key' => 2]], $this->db()->find('lg', 0)); + $index = json_decode(file_get_contents($base . '.nonedb.jidx'), true); + $this->assertSame(0, $index['d']); + } + + /** @test K-03 */ + public function zeroLengthIndexEntryDoesNotBreakTheDatabase(): void + { + $db = $this->db(); + $db->insert('z', [['n' => 1], ['n' => 2]]); + $idxPath = $this->dbBasePath('z') . '.nonedb.jidx'; + $index = json_decode(file_get_contents($idxPath), true); + $index['o'][1] = [$index['o'][1][0], 0]; // what pre-3.2 wrote after a failed json_encode + file_put_contents($idxPath, json_encode($index)); + \noneDB::clearStaticCache(); + $db = $this->db(); + $this->assertSame([['n' => 1], ['n' => 2]], $this->plain($db->find('z', 0))); + $this->assertSame(1, $db->delete('z', ['key' => 1])['n']); + $this->assertTrue($db->compact('z')['success']); + } +} diff --git a/tests/Unit/LimitTest.php b/tests/Unit/LimitTest.php index 8ff671e..d938225 100644 --- a/tests/Unit/LimitTest.php +++ b/tests/Unit/LimitTest.php @@ -101,9 +101,8 @@ public function flatArrayReturnsFalse(): void */ public function emptyArrayReturnsFalse(): void { - $result = $this->noneDB->limit([], 3); - - $this->assertFalse($result, 'Empty array should return false'); + // v3.2.0: an empty result set limited is still an empty result set (it used to be false) + $this->assertSame([], $this->noneDB->limit([], 3)); } /** diff --git a/tests/noneDBTestCase.php b/tests/noneDBTestCase.php index c53eddf..db066ae 100644 --- a/tests/noneDBTestCase.php +++ b/tests/noneDBTestCase.php @@ -26,7 +26,7 @@ abstract class noneDBTestCase extends TestCase /** * @var string Test database name */ - protected $testDbName = 'phpunit_test_db'; + protected $testDbName = 'phpunit-test-db'; /** * Set up test environment before each test @@ -109,7 +109,9 @@ protected function getPrivateMethod(string $methodName): ReflectionMethod { $reflector = new ReflectionClass(\noneDB::class); $method = $reflector->getMethod($methodName); - $method->setAccessible(true); + if (PHP_VERSION_ID < 80100) { + $method->setAccessible(true); // no effect since 8.1, deprecated in 8.5 + } return $method; } @@ -124,7 +126,9 @@ protected function getPrivateProperty(string $propertyName) { $reflector = new ReflectionClass(\noneDB::class); $property = $reflector->getProperty($propertyName); - $property->setAccessible(true); + if (PHP_VERSION_ID < 80100) { + $property->setAccessible(true); // no effect since 8.1, deprecated in 8.5 + } return $property->getValue($this->noneDB); } @@ -139,7 +143,9 @@ protected function setPrivateProperty(string $propertyName, $value): void { $reflector = new ReflectionClass(\noneDB::class); $property = $reflector->getProperty($propertyName); - $property->setAccessible(true); + if (PHP_VERSION_ID < 80100) { + $property->setAccessible(true); // no effect since 8.1, deprecated in 8.5 + } $property->setValue($this->noneDB, $value); } diff --git a/tests/operation_benchmark.php b/tests/operation_benchmark.php new file mode 100644 index 0000000..0c4d426 --- /dev/null +++ b/tests/operation_benchmark.php @@ -0,0 +1,80 @@ + 'nonedb-benchmark-key-not-for-production', 'dbDir' => $dir]; + +function wipe($dir) { + if (!is_dir($dir)) { + return; + } + foreach (array_diff(scandir($dir) ?: [], ['.', '..']) as $file) { + if (is_file($dir . $file)) { + unlink($dir . $file); + } + } +} + +function runOnce($cfg, $dir, $records) { + wipe($dir); + noneDB::clearStaticCache(); + $db = new noneDB($cfg); + if (method_exists($db, 'enableBuffering')) { + $db->enableBuffering(false); // direct writes (the default since 3.2) + } + $rows = []; + for ($i = 0; $i < $records; $i++) { + $rows[] = ['name' => 'user' . $i, 'city' => ['Istanbul', 'Ankara', 'Izmir', 'Bursa'][$i % 4], 'age' => 18 + $i % 50, 'active' => $i % 3 === 0]; + } + $results = []; + $t = function ($name, $ops, callable $fn) use (&$results) { + $start = microtime(true); + $fn(); + $results[$name] = (microtime(true) - $start) * 1000 / $ops; + }; + $t('insert (5,000 records, one call)', 1, function () use ($db, $rows) { $db->insert('bench', $rows); }); + $t('insert (one record)', 500, function () use ($db) { for ($i = 0; $i < 500; $i++) { $db->insert('single', ['i' => $i, 'v' => 'x']); } }); + $t('find (all records)', 20, function () use ($db) { for ($i = 0; $i < 20; $i++) { $db->find('bench', 0); } }); + $t('find (by key)', 2000, function () use ($db, $records) { for ($i = 0; $i < 2000; $i++) { $db->find('bench', ['key' => ($i * 7) % $records]); } }); + $t('find (filter)', 50, function () use ($db) { for ($i = 0; $i < 50; $i++) { $db->find('bench', ['city' => 'Ankara']); } }); + $t('count', 2000, function () use ($db) { for ($i = 0; $i < 2000; $i++) { $db->count('bench'); } }); + $t('query (where range + sort + limit)', 20, function () use ($db) { for ($i = 0; $i < 20; $i++) { $db->query('bench')->where(['age' => ['$gte' => 30, '$lt' => 40]])->sort('age', 'desc')->limit(10)->get(); } }); + $t('update (by key)', 300, function () use ($db, $records) { for ($i = 0; $i < 300; $i++) { $db->update('bench', [['key' => ($i * 13) % $records], ['set' => ['touched' => $i]]]); } }); + $t('update (filter, 1,250 records)', 5, function () use ($db) { for ($i = 0; $i < 5; $i++) { $db->update('bench', [['city' => 'Bursa'], ['set' => ['flag' => $i]]]); } }); + $t('delete (by key)', 300, function () use ($db, $records) { for ($i = 0; $i < 300; $i++) { $db->delete('bench', ['key' => ($i * 17) % $records]); } }); + $db->createFieldIndex('bench', 'city'); + $t('find (filter on indexed field)', 200, function () use ($db) { for ($i = 0; $i < 200; $i++) { $db->find('bench', ['city' => 'Izmir']); } }); + $t('insert (one record, indexed field)', 300, function () use ($db) { for ($i = 0; $i < 300; $i++) { $db->insert('bench', ['name' => 'n' . $i, 'city' => 'Izmir', 'age' => 30]); } }); + $t('new instance + find (by key)', 500, function () use ($cfg, $records) { for ($i = 0; $i < 500; $i++) { $d = new noneDB($cfg); $d->find('bench', ['key' => $i % $records]); } }); + return $results; +} + +$best = []; +for ($run = 0; $run < $runs; $run++) { + foreach (runOnce($cfg, $dir, $records) as $name => $ms) { + $best[$name] = isset($best[$name]) ? min($best[$name], $ms) : $ms; + } +} +wipe($dir); + +echo "noneDB single-operation benchmark ({$records} records, best of {$runs}, ms per operation)\n"; +echo "Library: " . realpath($lib) . "\nPHP " . PHP_VERSION . "\n\n"; +echo "| Operation | ms/op |\n|-----------|-------|\n"; +foreach ($best as $name => $ms) { + printf("| %s | %s |\n", $name, $ms < 1 ? number_format($ms, 3) : number_format($ms, 1)); +} diff --git a/tests/operator_benchmark.php b/tests/operator_benchmark.php index 68f0750..ede104f 100644 --- a/tests/operator_benchmark.php +++ b/tests/operator_benchmark.php @@ -247,9 +247,9 @@ echo "\n"; } -// Cleanup -foreach (glob($testDir . '*') as $file) { - if (is_file($file)) unlink($file); +// Cleanup (including dot files such as .htaccess and the hash cache) +foreach (array_diff(scandir($testDir) ?: [], ['.', '..']) as $file) { + if (is_file($testDir . $file)) unlink($testDir . $file); } rmdir($testDir); diff --git a/tests/performance_benchmark.php b/tests/performance_benchmark.php index 159f7cb..4999d8d 100644 --- a/tests/performance_benchmark.php +++ b/tests/performance_benchmark.php @@ -9,10 +9,15 @@ ini_set('memory_limit', '-1'); set_time_limit(0); -$db = new noneDB(); +// Own directory and key: the benchmark wipes its directory between sizes, never your databases +$benchDir = sys_get_temp_dir() . '/nonedb_performance_benchmark/'; +if (!is_dir($benchDir)) { + mkdir($benchDir, 0777, true); +} +$db = new noneDB(['secretKey' => 'nonedb-benchmark-key-not-for-production', 'dbDir' => $benchDir]); -// Test sizes -$sizes = [100, 1000, 10000, 50000, 100000, 500000]; +// Test sizes (override: php tests/performance_benchmark.php 100,1000) +$sizes = isset($argv[1]) ? array_map('intval', explode(',', $argv[1])) : [100, 1000, 10000, 50000, 100000, 500000]; // Colors function green($t) { return "\033[32m{$t}\033[0m"; } @@ -27,6 +32,11 @@ function formatTime($ms) { return round($ms) . " ms"; } +// Sub-millisecond times with 3 decimals +function formatFine($ms) { + return $ms < 1 ? number_format($ms, 3) . " ms" : formatTime($ms); +} + // Generate test record function generateRecord($i) { $cities = ['Istanbul', 'Ankara', 'Izmir', 'Bursa', 'Antalya']; @@ -43,7 +53,7 @@ function generateRecord($i) { } echo blue("╔════════════════════════════════════════════════════════════════════╗\n"); -echo blue("║ noneDB Performance Benchmark v3.0 ║\n"); +echo blue("║ noneDB Performance Benchmark v3.2 ║\n"); echo blue("║ JSONL Engine + Static Cache + Batch Read + Single-Pass Filter ║\n"); echo blue("╚════════════════════════════════════════════════════════════════════╝\n\n"); @@ -54,23 +64,28 @@ function generateRecord($i) { 'write' => ['insert' => [], 'update' => [], 'delete' => []], 'read' => ['find_all' => [], 'find_key' => [], 'find_filter' => []], 'query' => ['count' => [], 'distinct' => [], 'sum' => [], 'like' => [], 'between' => [], 'sort' => [], 'first' => [], 'exists' => []], - 'chain' => ['whereIn' => [], 'orWhere' => [], 'search' => [], 'groupBy' => [], 'select' => [], 'complex' => []] + 'chain' => ['whereIn' => [], 'orWhere' => [], 'search' => [], 'groupBy' => [], 'select' => [], 'complex' => []], + 'key' => ['cold' => [], 'warm' => []], + 'storage' => ['size' => [], 'memory' => []] ]; foreach ($sizes as $size) { - $dbName = "perftest_" . $size; + $dbName = "perftest-" . $size; echo yellow("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n"); echo yellow(" Testing with " . number_format($size) . " records\n"); echo yellow("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n"); - // Clean up ENTIRE db folder for fair benchmarking - $files = glob(__DIR__ . '/../db/*'); + // Clean up the benchmark folder for fair benchmarking + $files = glob($benchDir . '*'); foreach ($files as $f) { if (is_file($f)) @unlink($f); } noneDB::clearStaticCache(); // Clear static cache for accurate benchmarks clearstatcache(true); + if (function_exists('memory_reset_peak_usage')) { + memory_reset_peak_usage(); // PHP 8.2+: peak memory per size + } // ===== WRITE OPERATIONS ===== echo "\n" . cyan(" Write Operations:\n"); @@ -104,8 +119,8 @@ function generateRecord($i) { $results['write']['delete'][$size] = $deleteTime; echo " delete(): " . green(formatTime($deleteTime)) . "\n"; - // Re-insert for read tests (clean entire db folder) - $files = glob(__DIR__ . '/../db/*'); + // Re-insert for read tests (clean the benchmark folder) + $files = glob($benchDir . '*'); foreach ($files as $f) { if (is_file($f)) @unlink($f); } @@ -138,6 +153,28 @@ function generateRecord($i) { $results['read']['find_filter'][$size] = $findFilterTime; echo " find(filter): " . green(formatTime($findFilterTime)) . "\n"; + // KEY LOOKUP: cold (nothing cached, like the first lookup of a new process) and warm (average of 200) + noneDB::clearStaticCache(); + $start = microtime(true); + $db->find($dbName, ['key' => $testKey]); + $results['key']['cold'][$size] = (microtime(true) - $start) * 1000; + $keys = []; + for ($k = 0; $k < 200; $k++) { + $keys[] = ($k * 7919) % $size; // spread over every shard + } + foreach ($keys as $key) { + $db->find($dbName, ['key' => $key]); // warm-up: loads each shard's index once + } + $start = microtime(true); + foreach ($keys as $key) { + $db->find($dbName, ['key' => $key]); + } + $results['key']['warm'][$size] = (microtime(true) - $start) * 1000 / count($keys); + echo " key lookup: " . green(formatFine($results['key']['cold'][$size]) . " cold, " . formatFine($results['key']['warm'][$size]) . " warm") . "\n"; + + $info = $db->getDBs($dbName); + $results['storage']['size'][$size] = is_array($info) ? $info['size'] : '-'; + // ===== QUERY & AGGREGATION ===== echo "\n" . cyan(" Query & Aggregation:\n"); @@ -249,8 +286,10 @@ function generateRecord($i) { $results['chain']['complex'][$size] = $complexTime; echo " complex: " . green(formatTime($complexTime)) . "\n"; + $results['storage']['memory'][$size] = memory_get_peak_usage(true); + // Cleanup - $files = glob(__DIR__ . '/../db/*' . $dbName . '*'); + $files = glob($benchDir . '*' . $dbName . '*'); foreach ($files as $f) @unlink($f); echo "\n"; @@ -288,4 +327,14 @@ function printTable($title, $data, $sizes) { printTable("Query & Aggregation", $results['query'], $sizes); printTable("Method Chaining (v2.1+)", $results['chain'], $sizes); +echo "### Key Lookup\n| Records | Cold (first access) | Warm (cached) |\n|---------|-----|-----|\n"; +foreach ($sizes as $s) { + echo "| " . number_format($s) . " | " . formatFine($results['key']['cold'][$s]) . " | " . formatFine($results['key']['warm'][$s]) . " |\n"; +} +echo "\n### Storage & Memory\n| Records | Data Size | Peak Memory (incl. generated data) |\n|---------|-----|-----|\n"; +foreach ($sizes as $s) { + echo "| " . number_format($s) . " | " . $results['storage']['size'][$s] . " | " . round($results['storage']['memory'][$s] / 1048576) . " MB |\n"; +} +echo "\n"; + echo green("\nBenchmark completed!\n"); diff --git a/tests/sleekdb_comparison.php b/tests/sleekdb_comparison.php index c4c4138..359768b 100644 --- a/tests/sleekdb_comparison.php +++ b/tests/sleekdb_comparison.php @@ -12,8 +12,14 @@ ini_set('memory_limit', '-1'); set_time_limit(0); -// Test sizes -$sizes = [100, 1000, 10000, 50000, 100000]; +// Test sizes (override: php tests/sleekdb_comparison.php 100,1000) +$sizes = isset($argv[1]) ? array_map('intval', explode(',', $argv[1])) : [100, 1000, 10000, 50000, 100000]; + +// Own directory and key: the benchmark wipes its directory, never your databases +$benchDir = sys_get_temp_dir() . '/nonedb_sleekdb_comparison/'; +if (!is_dir($benchDir)) { + mkdir($benchDir, 0777, true); +} // Colors function green($t) { return "\033[32m{$t}\033[0m"; } @@ -61,11 +67,11 @@ function ratio($nonedb, $sleekdb) { } echo blue("╔══════════════════════════════════════════════════════════════════════════╗\n"); -echo blue("║ noneDB v3.0 vs SleekDB Comprehensive Benchmark ║\n"); +echo blue("║ noneDB v3.2 vs SleekDB Comprehensive Benchmark ║\n"); echo blue("╚══════════════════════════════════════════════════════════════════════════╝\n\n"); echo "PHP Version: " . PHP_VERSION . "\n"; -echo "noneDB: v3.0.0 (JSONL + Static Cache + Batch Read)\n"; +echo "noneDB: v3.2.0 (JSONL + database locks + static cache)\n"; echo "SleekDB: v2.x\n\n"; $results = []; @@ -76,11 +82,11 @@ function ratio($nonedb, $sleekdb) { echo yellow("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n"); // Cleanup - Clean ALL databases for fair benchmarking - $nonedbName = "benchmark_nonedb_" . $size; - $sleekdbDir = __DIR__ . "/sleekdb_benchmark_" . $size; + $nonedbName = "benchmark-nonedb-" . $size; + $sleekdbDir = sys_get_temp_dir() . "/sleekdb_benchmark_" . $size; - // Clean ENTIRE noneDB db folder - $files = glob(__DIR__ . '/../db/*'); + // Clean the noneDB benchmark folder + $files = glob($benchDir . '*'); foreach ($files as $f) { if (is_file($f)) @unlink($f); } @@ -88,7 +94,7 @@ function ratio($nonedb, $sleekdb) { clearstatcache(true); // Clean ALL SleekDB benchmark folders - $sleekDirs = glob(__DIR__ . '/sleekdb_benchmark_*'); + $sleekDirs = glob(sys_get_temp_dir() . '/sleekdb_benchmark_*'); foreach ($sleekDirs as $dir) { if (is_dir($dir)) { $files = glob($dir . '/*'); @@ -111,7 +117,7 @@ function ratio($nonedb, $sleekdb) { $data[] = generateRecord($i); } - $nonedb = new noneDB(); + $nonedb = new noneDB(['secretKey' => 'nonedb-benchmark-key-not-for-production', 'dbDir' => $benchDir]); // ===== BULK INSERT ===== echo cyan(" Bulk Insert ($size records):\n"); @@ -286,7 +292,7 @@ function ratio($nonedb, $sleekdb) { $results[$size]['complex'] = ['nonedb' => $nonedbComplex, 'sleekdb' => $sleekdbComplex]; // Cleanup - $files = glob(__DIR__ . '/../db/*benchmark_nonedb_' . $size . '*'); + $files = glob($benchDir . '*benchmark-nonedb-' . $size . '*'); foreach ($files as $f) @unlink($f); if (is_dir($sleekdbDir)) { diff --git a/tests/sleekdb_vs_nonedb_benchmark.php b/tests/sleekdb_vs_nonedb_benchmark.php index 6dcd170..14233b1 100644 --- a/tests/sleekdb_vs_nonedb_benchmark.php +++ b/tests/sleekdb_vs_nonedb_benchmark.php @@ -52,8 +52,8 @@ function generateRecord($i) { } // Test directories -$sleekDbDir = __DIR__ . '/sleekdb_bench/'; -$noneDbDir = __DIR__ . '/nonedb_bench/'; +$sleekDbDir = sys_get_temp_dir() . '/sleekdb_bench/'; +$noneDbDir = sys_get_temp_dir() . '/nonedb_bench/'; // Cleanup function function cleanup($sleekDbDir, $noneDbDir) { @@ -90,7 +90,7 @@ function cleanup($sleekDbDir, $noneDbDir) { echo "noneDB: v2.3.0 (sharding ON, buffer ON/OFF)\n\n"; // Test sizes -$sizes = [100, 1000, 10000, 50000, 100000]; +$sizes = isset($argv[1]) ? array_map('intval', explode(',', $argv[1])) : [100, 1000, 10000, 50000, 100000]; // Results storage $results = []; @@ -212,15 +212,11 @@ function cleanup($sleekDbDir, $noneDbDir) { if (!is_dir($noneDbDir)) mkdir($noneDbDir, 0777, true); // ===================================================================== - // NONEDB (DEFAULT - Buffer ON, Sharding ON) + // NONEDB (Buffer ON, Sharding ON; the default is buffer OFF since v3.2) // ===================================================================== - echo magenta(" ┌─ noneDB (default: buffer ON, sharding ON) ───────────────────────┐\n"); + echo magenta(" ┌─ noneDB (buffer ON, sharding ON) ────────────────────────────────┐\n"); - $nonedb = new noneDB(); - $ref = new ReflectionClass($nonedb); - $prop = $ref->getProperty('dbDir'); - $prop->setAccessible(true); - $prop->setValue($nonedb, $noneDbDir); + $nonedb = new noneDB(['secretKey' => 'nonedb-benchmark-key-not-for-production', 'dbDir' => $noneDbDir, 'bufferEnabled' => true]); // Bulk Insert gc_collect_cycles(); @@ -305,12 +301,7 @@ function cleanup($sleekDbDir, $noneDbDir) { // ===================================================================== echo magenta(" ┌─ noneDB (buffer OFF, sharding ON) ────────────────────────────────┐\n"); - $nonedb2 = new noneDB(); - $ref2 = new ReflectionClass($nonedb2); - $prop2 = $ref2->getProperty('dbDir'); - $prop2->setAccessible(true); - $prop2->setValue($nonedb2, $noneDbDir); - $nonedb2->enableBuffering(false); + $nonedb2 = new noneDB(['secretKey' => 'nonedb-benchmark-key-not-for-production', 'dbDir' => $noneDbDir, 'bufferEnabled' => false]); // Bulk Insert (no buffer) gc_collect_cycles(); diff --git a/tests/spatial_benchmark.php b/tests/spatial_benchmark.php index 69c9473..9130738 100644 --- a/tests/spatial_benchmark.php +++ b/tests/spatial_benchmark.php @@ -54,39 +54,39 @@ // 1. Batch Insert (without spatial index) $start = microtime(true); - $db->insert('benchmark_locations', $data); + $db->insert('benchmark-locations', $data); $insertTime = (microtime(true) - $start) * 1000; echo "Insert $count records: " . round($insertTime, 2) . " ms\n"; // 2. Create Spatial Index $start = microtime(true); - $db->createSpatialIndex('benchmark_locations', 'location'); + $db->createSpatialIndex('benchmark-locations', 'location'); $indexTime = (microtime(true) - $start) * 1000; echo "Create spatial index: " . round($indexTime, 2) . " ms\n"; // 3. withinDistance query $start = microtime(true); - $results = $db->withinDistance('benchmark_locations', 'location', 28.9, 41.0, 5000); + $results = $db->withinDistance('benchmark-locations', 'location', 28.9, 41.0, 5000); $queryTime = (microtime(true) - $start) * 1000; echo "withinDistance (5km): " . round($queryTime, 2) . " ms (" . count($results) . " results)\n"; // 4. nearest() query $start = microtime(true); - $results = $db->nearest('benchmark_locations', 'location', 28.9, 41.0, 10); + $results = $db->nearest('benchmark-locations', 'location', 28.9, 41.0, 10); $nearestTime = (microtime(true) - $start) * 1000; echo "nearest(10): " . round($nearestTime, 2) . " ms\n"; // 5. withinBBox query $start = microtime(true); - $results = $db->withinBBox('benchmark_locations', 'location', 28.85, 40.95, 28.95, 41.05); + $results = $db->withinBBox('benchmark-locations', 'location', 28.85, 40.95, 28.95, 41.05); $bboxTime = (microtime(true) - $start) * 1000; echo "withinBBox: " . round($bboxTime, 2) . " ms (" . count($results) . " results)\n"; // 6. Query Builder with spatial + where filter $start = microtime(true); - $results = $db->query('benchmark_locations') + $results = $db->query('benchmark-locations') ->withinDistance('location', 28.9, 41.0, 10000) - ->where(['name' => ['$like' => 'Location 1%']]) + ->where(['name' => ['$like' => '^Location 1']]) // names starting with "Location 1" ($like is literal except ^ and $) ->get(); $combinedTime = (microtime(true) - $start) * 1000; echo "Combined query (spatial+where): " . round($combinedTime, 2) . " ms (" . count($results) . " results)\n"; From eea09f3af2a5a8aff8d4e2063d2bad140301f23b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Orhan=20AYDO=C4=9EDU?= Date: Sat, 24 Oct 2026 01:47:14 +0300 Subject: [PATCH 3/3] Fix environment-dependent memory_limit test includingDoesNotChangeMemoryLimit asserted that memory_limit is not -1 after including noneDB.php. GitHub's setup-php already sets memory_limit=-1 for the CLI, so the test failed on every PHP version although noneDB.php changes nothing. The child process now starts with -d memory_limit=123M and the test checks the value is unchanged after the include. --- tests/Regression/SecurityConfigRegressionTest.php | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/Regression/SecurityConfigRegressionTest.php b/tests/Regression/SecurityConfigRegressionTest.php index 83a7ab3..dab8d5e 100644 --- a/tests/Regression/SecurityConfigRegressionTest.php +++ b/tests/Regression/SecurityConfigRegressionTest.php @@ -346,7 +346,11 @@ public function instancesAreReleased(): void /** @test O-20 */ public function includingDoesNotChangeMemoryLimit(): void { - $this->assertNotSame('-1', $this->runChild('echo ini_get("memory_limit");')); + // Start the child with a known limit: the environment's own php.ini may already say -1 + // (GitHub's setup-php does), which says nothing about what including noneDB.php does + $script = $this->childScript('echo ini_get("memory_limit");'); // noneDB.php is required first + $out = trim((string)shell_exec(escapeshellarg(PHP_BINARY) . ' -d memory_limit=123M ' . escapeshellarg($script) . ' 2>&1')); + $this->assertSame('123M', $out); } /** @test D-01 */