perf: reduce per-item allocation churn in Arius.Core

docs/design/core/shared/streaming.md

@@ -21,14 +21,15 @@ flowchart LR
   tee -. "tee for inline verify" .-> ver[RoundTripVerifier]
 ```
 
-- **`ProgressStream(inner, IProgress<long>)`** wraps the *source* at the top of the chain so progress tracks logical bytes read off disk, not compressed bytes on the wire. `Length` is delegated to the inner `FileStream`, so the consumer knows the total up front and can compute a percentage. It reports after each read; a zero-length read reports nothing. The download path reuses it the same way, wrapping the blob's read stream.
+- **`ProgressStream(inner, IProgress<long>)`** wraps the *source* at the top of the chain so progress tracks logical bytes read off disk, not compressed bytes on the wire. `Length` is delegated to the inner `FileStream`, so the consumer knows the total up front and can compute a percentage. Reports are **coalesced to at most one per 500 ms**: the first read reports immediately, later reads wait for the interval, and the true total is emitted at EOF or disposal. Empty sources report nothing. The download path reuses it the same way, wrapping the blob's read stream.
 - **`CountingStream(inner)`** sits at the *bottom* of the chain, directly above `OpenWriteAsync`, and increments `BytesWritten` on every write. It is read *after* the chain is disposed to capture the final compressed-and-encrypted blob size, which `UploadChunkAsync` then writes into blob metadata (`chunk-size`).
 
 The note in `UploadChunkAsync` is load-bearing here: the encryption stream is disposed *explicitly* before reading `BytesWritten`, because GCM flushes its final auth tag on dispose — reading the count earlier would undercount by the tag bytes.
 
 ## Key invariants
 
-- **No buffering proportional to file size.** Both wrappers hold only a `long` counter; the chain streams a multi-GB file without an O(file-size) allocation. See [memory-boundedness](../../cross-cutting/memory-boundedness.md).
+- **No buffering proportional to file size.** Both wrappers hold only a few `long` counters; the chain streams a multi-GB file without an O(file-size) allocation. See [memory-boundedness](../../cross-cutting/memory-boundedness.md).
+- **A coalesced `ProgressStream` still ends on the true total.** Throttling may drop intermediate reports, but the EOF and dispose flushes must stay, or a consumer is left short of 100% by up to one interval's worth of bytes.
 - **`CountingStream.BytesWritten` is valid only after the whole chain is finalized.** Every layer above it (compression frame, GCM tag) must be flushed/disposed first, or the recorded `chunk-size` is short.
 - **`ProgressStream` is read-only, `CountingStream` is write-only.** They guard their unsupported direction with `NotSupportedException` rather than silently no-op'ing, so a misuse fails loudly.
 - **Counting reflects what was actually persisted.** `CountingStream` wraps the Azure write stream, so its total is the blob's real stored size, not a pre-computed estimate.
@@ -39,5 +40,5 @@ Splitting "report read progress" and "count written bytes" into separate one-lin
 
 ## Open seams / future
 
-- `ProgressStream` reports raw read counts; smoothing/throttling of the `IProgress<long>` callback is left to the consumer (`UploadChunkAsync` already de-dupes non-increasing reports via a `CallbackProgress`).
+- `ProgressStream` throttles reports at the source to one per 500 ms. `UploadChunkAsync`'s `CallbackProgress` still de-dupes non-increasing reports on top of it. The interval is a fixed constant, not a per-consumer setting.
 - Any future upload layer (e.g. a second integrity tee) slots into the same push chain in `UploadChunkAsync` between source and `OpenWriteAsync`; these two wrappers stay unchanged as the progress/size endpoints.

docs/design/cross-cutting/memory-boundedness.md

@@ -46,7 +46,7 @@ Earlier the [chunk index](../../glossary.md#chunk-index) kept an in-memory LRU o
 
 ### 4. Streaming up/download — bound the single-file byte axis
 
-A single multi-GB file must not be buffered. `ChunkStorageService.UploadChunkAsync` assembles a push chain (source → `ProgressStream` → zstd → encryption → `CountingStream` → `OpenWriteAsync`) with no `MemoryStream` and no intermediate temp file (the tar bundle being the one streamed-in exception). The `Streaming` decorators hold only a `long` counter, so the chain moves a file of any size with a working set of roughly one buffer. This is the byte-scale complement to the file-count techniques above — see [streaming](../core/shared/streaming.md).
+A single multi-GB file must not be buffered. `ChunkStorageService.UploadChunkAsync` assembles a push chain (source → `ProgressStream` → zstd → encryption → `CountingStream` → `OpenWriteAsync`) with no `MemoryStream` and no intermediate temp file (the tar bundle being the one streamed-in exception). The `Streaming` decorators hold only a few counters, so the chain moves a file of any size with a working set of roughly one buffer. This is the byte-scale complement to the file-count techniques above — see [streaming](../core/shared/streaming.md).
 
 ## Key invariants
 

src/Arius.Api.FakeTestHost/CanonicalScenarios.cs

@@ -18,7 +18,7 @@ public static class CanonicalScenarios
             new FileHashingEvent(RelativePath.Parse("big.bin"), 100_000_000),
             new FileHashedEvent(RelativePath.Parse("big.bin"), ContentHash.Parse(new string('a', 64)), FastHashReused: false, FastHashRehashed: true, FileSize: 100_000_000),
             new ChunkUploadedEvent(ChunkHash.Parse(new string('a', 64)), StoredSize: 60_000_000, OriginalSize: 100_000_000),
-            new FileDedupedEvent(ContentHash.Parse(new string('b', 64)), OriginalSize: 48_000_000),
+            new FileDedupedEvent(RelativePath.Parse("deduped.bin"), ContentHash.Parse(new string('b', 64)), OriginalSize: 48_000_000),
             new RoutingCompleteEvent(NewByteTotal: 100_000_000),   // dedup/route drained → exact new-byte total
             new SnapshotCreatedEvent(default, DateTimeOffset.UnixEpoch, 3122),
         ],

src/Arius.Api.Integration.Tests/RepresentationScenarioTests.cs

@@ -25,7 +25,7 @@ public async Task Pointer_heavy_archive_reports_additive_new_bytes_not_underflow
             Events:
             [
                 new ScanCompleteEvent(TotalFiles: 1001, TotalBytes: 100_000_000),           // pointer-only files scanned as 0
-                new FileDedupedEvent(ContentHash.Parse(new string('b', 64)), OriginalSize: 1_000_000_000), // pointer-only dedup, full size
+                new FileDedupedEvent(RelativePath.Parse("deduped.bin"), ContentHash.Parse(new string('b', 64)), OriginalSize: 1_000_000_000), // pointer-only dedup, full size
                 new ChunkUploadingEvent(ChunkHash.Parse(new string('c', 64)), 100_000_000), // one new chunk queued
                 new ChunkUploadedEvent(ChunkHash.Parse(new string('c', 64)), StoredSize: 60_000_000, OriginalSize: 100_000_000),
             ],

src/Arius.Api.Tests/Jobs/JobSinkAggregateTests.cs

@@ -112,7 +112,7 @@ public async Task Archive_forwarders_populate_byte_layers()
         await new ScanCompleteForwarder(s).Handle(new ScanCompleteEvent(2, 3000), default);
         await new FileScannedForwarder(s).Handle(new FileScannedEvent(RelativePath.Parse("a"), 2000), default);
         await new FileHashingForwarder(s).Handle(new FileHashingEvent(RelativePath.Parse("a"), 2000), default);
-        await new FileDedupedForwarder(s).Handle(new FileDedupedEvent(ContentHash.Parse(new string('b', 64)), 1000), default);
+        await new FileDedupedForwarder(s).Handle(new FileDedupedEvent(RelativePath.Parse("deduped.bin"), ContentHash.Parse(new string('b', 64)), 1000), default);
         await new ChunkUploadingForwarder(s).Handle(new ChunkUploadingEvent(ChunkHash.Parse(new string('d', 64)), 2000), default);
         await new ChunkUploadedForwarder(s).Handle(new ChunkUploadedEvent(ChunkHash.Parse(new string('c', 64)), 300, 2000), default);
 

src/Arius.Benchmarks/AllocationBenchmarks.cs

@@ -0,0 +1,269 @@
+using Arius.Core.Features.ArchiveCommand;
+using Arius.Core.Shared.ChunkIndex;
+using Arius.Core.Shared.Encryption;
+using Arius.Core.Shared.FileSystem;
+using Arius.Core.Shared.FileTree;
+using Arius.Core.Shared.HashCache;
+using Arius.Core.Shared.Hashes;
+using Arius.Core.Shared.Storage;
+using Arius.Tests.Shared;
+using BenchmarkDotNet.Attributes;
+
+namespace Arius.Benchmarks;
+
+/// <summary>
+/// In-process allocation benchmarks for selected Arius.Core components.
+/// Run with <c>micro</c> (e.g. <c>micro --filter '*TarBuilder*'</c>); no Azurite or Docker is required.
+/// </summary>
+[MemoryDiagnoser]
+public class AllocationBenchmarks
+{
+    private const int EntryCount = 1_000;
+
+    /// <summary>
+    /// Digest containing hexadecimal letters, ensuring the lowercase conversion path is exercised.
+    /// </summary>
+    private readonly byte[] _digest = CreateHighNibbleDigest();
+
+    private const string CanonicalHex   = "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff";
+    private const string UppercaseHex   = "00112233445566778899AABBCCDDEEFF00112233445566778899AABBCCDDEEFF";
+
+    [Benchmark(Description = "HashCodec.ToLowerHex")]
+    public string HashCodec_ToLowerHex() => HashCodec.ToLowerHex(_digest);
+
+    /// <summary>Parses an already canonical lowercase value.</summary>
+    [Benchmark(Description = "HashCodec.NormalizeHex (already canonical)")]
+    public string HashCodec_NormalizeHex_Canonical() => HashCodec.NormalizeHex(CanonicalHex);
+
+    [Benchmark(Description = "HashCodec.NormalizeHex (uppercase input)")]
+    public string HashCodec_NormalizeHex_Uppercase() => HashCodec.NormalizeHex(UppercaseHex);
+
+    // ── Sparse fingerprint ───────────────────────────────────────────────────────
+
+    private const long SmallFileSize = 200L * 1024;              // one whole-file region
+    private const long LargeFileSize = 64L * 1024 * 1024 * 1024; // k = MaxBlocks = 64 regions
+
+    private byte[] _readBuffer = null!;
+
+    /// <summary>Exercises the single-region sampler path for a small file.</summary>
+    [Benchmark(Description = "SparseFingerprint.Sampler small file (200 KB)")]
+    public byte[] SparseFingerprint_Sampler_SmallFile()
+    {
+        using var sampler = new SparseFingerprint.Sampler(SmallFileSize);
+        var position      = 0L;
+
+        while (position < SmallFileSize)
+        {
+            var length = (int)Math.Min(_readBuffer.Length, SmallFileSize - position);
+            sampler.Capture(position, _readBuffer.AsSpan(0, length));
+            position += length;
+        }
+
+        return sampler.Finish();
+    }
+
+    /// <summary>Exercises the maximum sampled-region buffer size.</summary>
+    [Benchmark(Description = "SparseFingerprint.Sampler large file (64 GB logical)")]
+    public byte[] SparseFingerprint_Sampler_LargeFile()
+    {
+        using var sampler = new SparseFingerprint.Sampler(LargeFileSize);
+
+        foreach (var (offset, length) in SparseFingerprint.Regions(LargeFileSize))
+            sampler.Capture(offset, _readBuffer.AsSpan(0, length));
+
+        return sampler.Finish();
+    }
+
+    // ── Filetree serialization ───────────────────────────────────────────────────
+
+    private IReadOnlyList<FileTreeEntry> _fileTreeEntries = null!;
+    private byte[]                       _fileTreeBytes   = null!;
+
+    [Benchmark(Description = "FileTreeSerializer.Serialize (1000 entries)")]
+    public byte[] FileTreeSerializer_Serialize() => FileTreeSerializer.Serialize(_fileTreeEntries);
+
+    [Benchmark(Description = "FileTreeSerializer.Deserialize (1000 entries)")]
+    public IReadOnlyList<FileTreeEntry> FileTreeSerializer_Deserialize() => FileTreeSerializer.Deserialize(_fileTreeBytes);
+
+    // ── Tar builder ──────────────────────────────────────────────────────────────
+
+    private const long TarTargetSize  = 64L * 1024 * 1024;
+    private const int  TarEntrySize   = 64 * 1024;
+
+    private byte[]             _tarEntryPayload = null!;
+    private byte[]             _smallEntryPayload = null!;
+    private IEncryptionService _encryption      = null!;
+
+    /// <summary>Builds and seals one 64 MB TAR bundle.</summary>
+    [Benchmark(Description = "TarBuilder seal one 64 MB bundle")]
+    public async Task<int> TarBuilder_Seal_64MB()
+    {
+        await using var builder = new TarBuilder(TarTargetSize, _encryption);
+
+        var sealedCount = 0;
+        for (var i = 0; i < TarTargetSize / TarEntrySize; i++)
+        {
+            var source = new MemoryStream(_tarEntryPayload, writable: false);
+            if (await builder.AddAsync(CreateUpload(i, TarEntrySize), source, CancellationToken.None) is not null)
+                sealedCount++;
+        }
+
+        return sealedCount;
+    }
+
+    /// <summary>A handful of small files: the shape of a tiny repository or a small tail bundle.</summary>
+    [Benchmark(Description = "TarBuilder seal one small bundle (5 x 1 KB)")]
+    public async Task<int> TarBuilder_Seal_SmallBundle()
+    {
+        await using var builder = new TarBuilder(TarTargetSize, _encryption);
+
+        for (var i = 0; i < 5; i++)
+        {
+            var source = new MemoryStream(_smallEntryPayload, writable: false);
+            await builder.AddAsync(CreateUpload(i, _smallEntryPayload.Length), source, CancellationToken.None);
+        }
+
+        return (await builder.SealAsync(CancellationToken.None))!.Entries.Count;
+    }
+
+    // ── Chunk-index local store ──────────────────────────────────────────────────
+
+    private LocalDirectory       _storeRoot     = default;
+    private ChunkIndexLocalStore _store         = null!;
+    private ShardEntry[]         _shardEntries  = null!;
+    private ContentHash[]        _lookupHashes  = null!;
+    private PathSegment          _rangePrefix   = default;
+
+    [Benchmark(Description = "ChunkIndexLocalStore.UpsertRemoteBacked (1000 rows)")]
+    public void ChunkIndexLocalStore_UpsertRemoteBacked() => _store.UpsertRemoteBacked(_shardEntries);
+
+    [Benchmark(Description = "ChunkIndexLocalStore.ReadRangeEntries (1000 rows)")]
+    public int ChunkIndexLocalStore_ReadRangeEntries()
+    {
+        var count = 0;
+        _store.ReadRangeEntries(_rangePrefix, _ => count++);
+        return count;
+    }
+
+    /// <summary>Measures the per-hash lookup shape for a 256-hash deduplication batch.</summary>
+    [Benchmark(Description = "ChunkIndexLocalStore.FindEntry x256 (one dedup batch)")]
+    public int ChunkIndexLocalStore_FindEntry_256()
+    {
+        var found = 0;
+        foreach (var hash in _lookupHashes)
+            if (_store.FindEntry(hash) is not null)
+                found++;
+
+        return found;
+    }
+
+    /// <summary>Measures the batched lookup for a 256-hash deduplication batch.</summary>
+    [Benchmark(Description = "ChunkIndexLocalStore.FindEntries x1 (one dedup batch)")]
+    public int ChunkIndexLocalStore_FindEntries_Batch() => _store.FindEntries(_lookupHashes).Count;
+
+    // ── Setup ────────────────────────────────────────────────────────────────────
+
+    [GlobalSetup]
+    public void Setup()
+    {
+        _readBuffer      = new byte[256 * 1024];
+        _tarEntryPayload = new byte[TarEntrySize];
+        _smallEntryPayload = new byte[1024];
+        Random.Shared.NextBytes(_readBuffer);
+        Random.Shared.NextBytes(_tarEntryPayload);
+        Random.Shared.NextBytes(_smallEntryPayload);
+
+        _encryption = IEncryptionService.EncryptedInstance;
+
+        _fileTreeEntries = BuildFileTreeEntries(EntryCount);
+        _fileTreeBytes   = FileTreeSerializer.Serialize(_fileTreeEntries);
+
+        _shardEntries = BuildShardEntries(EntryCount);
+        _lookupHashes = _shardEntries.Take(256).Select(e => e.ContentHash).ToArray();
+        _rangePrefix  = PathSegment.Parse("00");
+
+        _storeRoot = TestTempRoots.CreateDirectory("benchmark-chunkindex");
+        _store     = new ChunkIndexLocalStore(_storeRoot);
+        _store.UpsertRemoteBacked(_shardEntries);
+    }
+
+    [GlobalCleanup]
+    public void Cleanup()
+    {
+        try
+        {
+            RelativeFileSystem.DeleteDirectory(_storeRoot, RelativePath.Root, recursive: true);
+        }
+        catch (IOException)
+        {
+            // Best effort: the SQLite connection pool may still hold the file. TestTempRoots sweeps stale dirs.
+        }
+    }
+
+    // ── Deterministic fixtures ───────────────────────────────────────────────────
+
+    /// <summary>
+    /// Creates distinct digests under the <c>"00"</c> range prefix used by the range benchmark.
+    /// </summary>
+    private static byte[] CreateDigest(int seed)
+    {
+        var digest = new byte[32];
+        BitConverter.TryWriteBytes(digest.AsSpan(1), seed);
+        return digest;
+    }
+
+    private static ContentHash CreateContentHash(int seed) => ContentHash.FromDigest(CreateDigest(seed));
+
+    /// <summary>Creates a digest containing hexadecimal letters.</summary>
+    private static byte[] CreateHighNibbleDigest()
+    {
+        var digest = new byte[32];
+        for (var i = 0; i < digest.Length; i++)
+            digest[i] = (byte)(0xA0 | (i & 0x0F));
+
+        return digest;
+    }
+
+    private static IReadOnlyList<FileTreeEntry> BuildFileTreeEntries(int count)
+    {
+        var entries = new List<FileTreeEntry>(count);
+        var created = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero);
+
+        for (var i = 0; i < count; i++)
+        {
+            entries.Add(new FileEntry
+            {
+                Name        = PathSegment.Parse($"file-{i:D6}.bin"),
+                ContentHash = CreateContentHash(i),
+                Created     = created.AddSeconds(i),
+                Modified    = created.AddSeconds(i * 2),
+            });
+        }
+
+        return entries;
+    }
+
+    private static ShardEntry[] BuildShardEntries(int count)
+    {
+        var entries = new ShardEntry[count];
+        for (var i = 0; i < count; i++)
+        {
+            var contentHash = CreateContentHash(i);
+            entries[i] = new ShardEntry(
+                ContentHash:     contentHash,
+                ChunkHash:       ChunkHash.Parse(contentHash), // large chunk: chunk hash == content hash
+                OriginalSize:    4096 + i,
+                ChunkSize:       2048 + i,
+                StorageTierHint: BlobTier.Cool);
+        }
+
+        return entries;
+    }
+
+    private static FileToUpload CreateUpload(int seed, long size)
+    {
+        var filePair = new FilePair { RelativePath = RelativePath.Parse($"file-{seed:D6}.bin") };
+        var hashed   = new HashedFilePair(filePair, CreateContentHash(seed), DateTimeOffset.UnixEpoch, DateTimeOffset.UnixEpoch);
+        return new FileToUpload(hashed, size);
+    }
+}

src/Arius.Benchmarks/BenchmarkRunOptions.cs

@@ -71,6 +71,7 @@ static string FindRepositoryRoot()
     static void PrintHelp()
     {
         Console.WriteLine("Runs the canonical representative workflow benchmark on Azurite.");
+        Console.WriteLine("Pass 'micro [BenchmarkDotNet options]' instead for the in-process allocation benchmarks.");
         Console.WriteLine();
         Console.WriteLine("Options:");
         Console.WriteLine("  --raw-output <path>  Folder where per-run raw BenchmarkDotNet output is saved.");