Multi-process: two processes creating a new database at the same time crash one of them on open #216

Activity

1. github-actions

   added

   triage

   on Oct 13, 2026
2. CaffeinatedCoder commented on Oct 13, 2026

   

   CaffeinatedCoder

   on Oct 13, 2026

   Contributor

   More actions

   Here is a new reproduction harness; the one from #134 is gone. It's a ~230-line console app that runs N copies of itself against one database path per round.

   How it works

   • Each round, the coordinator starts N child processes on a fresh path and releases them together through a "go" file.
   • Each child opens the database (a typed DocumentDbContext, or BLiteEngine with --dynamic) and writes B explicit transactions of K documents.
   • A crashed child is classified by its exit code and stderr. An AccessViolationException kills the process, so it shows up as [fatal].
   • Afterwards, a separate verify process reopens the database and checks that every batch a child reported as committed is present.

   dotnet run -c Release -p:BLiteSrc=<BLite>/src -- --rounds 30 --procs 2      # or -p:BLiteVersion=5.1.1
      --batches 0      open only
      --writers 1      one writer; the other processes only open and read
      --precreate      control: the database exists before the processes start
   

   Repro216.csproj + Program.cs

   <Project Sdk="Microsoft.NET.Sdk">
     <!-- Default: builds against the BLite sources in $(BLiteSrc) (override with -p:BLiteSrc=...).
          With -p:BLiteVersion=5.1.1 it builds against that NuGet package instead. -->
     <PropertyGroup>
       <OutputType>Exe</OutputType>
       <TargetFramework>net10.0</TargetFramework>
       <Nullable>enable</Nullable>
       <ImplicitUsings>enable</ImplicitUsings>
       <BLiteSrc Condition="'$(BLiteSrc)' == ''">$(MSBuildThisFileDirectory)../wt216/src</BLiteSrc>
     </PropertyGroup>
     <ItemGroup Condition="'$(BLiteVersion)' == ''">
       <ProjectReference Include="$(BLiteSrc)/BLite.Core/BLite.Core.csproj" />
       <ProjectReference Include="$(BLiteSrc)/BLite.SourceGenerators/BLite.SourceGenerators.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
     </ItemGroup>
     <ItemGroup Condition="'$(BLiteVersion)' != ''">
       <PackageReference Include="BLite" Version="$(BLiteVersion)" />
     </ItemGroup>
   </Project>
   

   // Repro for EntglDb/BLite#216: N processes open the same, not yet existing database with
   // AllowMultiProcessAccess = true at the same moment; one of them crashes while opening.
   //
   //   dotnet run -c Release -- [--rounds 30] [--procs 2] [--batches 20] [--size 50] [--writers N]
   //                            [--dynamic] [--precreate] [--keep] [--timeout 120]
   //
   //   --writers    only the first N children write (default: all); the others open the database
   //                and read it, the "N readers / 1 writer" model of the multi-process WAL
   //   --dynamic    use BLiteEngine + DynamicCollection instead of a typed DocumentDbContext
   //   --precreate  control: the coordinator creates the database before the children start
   //   --keep       keep the database directory of rounds that failed
   //   --timeout    seconds after which a child still running is killed and reported as a hang
   //
   // Each round: spawn N children of this executable on a fresh temp path, wait until all of
   // them are ready, release them at once through a "go" file, then reopen the database in one
   // more child ("verify", so a hang can be killed) and check that every batch a child reported
   // as committed is present. --batches 0 only opens the database (and creates the collection).
   using System.Diagnostics;
   using System.Runtime.InteropServices;
   using System.Text.RegularExpressions;
   using BLite.Bson;
   using BLite.Core;
   using BLite.Core.Collections;
   using BLite.Core.Storage;
   
   namespace Repro216;
   
   public class Item
   {
       public ObjectId Id { get; set; }
       public string Tag { get; set; } = "";
   }
   
   public partial class ReproDbContext : DocumentDbContext
   {
       public DocumentCollection<ObjectId, Item> Items { get; set; } = null!;
   
       public ReproDbContext(string path, PageFileConfig config) : base(path, config) => InitializeCollections();
   }
   
   public static class Program
   {
       private static readonly PageFileConfig Config = PageFileConfig.Default with { AllowMultiProcessAccess = true };
       private static TimeSpan _timeout = TimeSpan.FromMinutes(2);
   
       public static Task<int> Main(string[] args) => args switch
       {
           ["child", ..] => Child(args),
           ["verify", var db, var api] => Verify(db, api == "dynamic"),
           _ => Coordinator(args),
       };
   
       // ── Child: open, write `batches` explicit transactions of `size` documents, report each commit ──
       private static async Task<int> Child(string[] a)
       {
           string db = a[1], go = a[2], who = "p" + a[3];
           int batches = int.Parse(a[4]), size = int.Parse(a[5]);
           if (batches < 0) return await Verify(db, a[6] == "dynamic", waitFor: go); // a reader
           Console.WriteLine("READY");
           while (!File.Exists(go)) Thread.SpinWait(50);
           try
           {
               await WriteBatches(db, a[6] == "dynamic", who, batches, size, b => Console.WriteLine($"COMMITTED {who}-{b}"));
               return 0;
           }
           catch (Exception ex)
           {
               Console.Error.WriteLine("EXC " + ex);
               return 2;
           }
       }
   
       private static async Task WriteBatches(string db, bool dynamic, string who, int batches, int size, Action<int> onCommit)
       {
           if (dynamic)
           {
               using var engine = new BLiteEngine(db, Config);
               var items = engine.GetOrCreateCollection("items");
               for (int b = 0; b < batches; b++)
               {
                   using var txn = engine.BeginTransaction();
                   for (int i = 0; i < size; i++)
                       await items.InsertAsync(items.CreateDocument(["tag"], d => d.AddString("tag", $"{who}-{b}-{i}")), txn);
                   await txn.CommitAsync();
                   onCommit(b);
               }
           }
           else
           {
               using var ctx = new ReproDbContext(db, Config);
               for (int b = 0; b < batches; b++)
               {
                   using var txn = ctx.BeginTransaction();
                   for (int i = 0; i < size; i++)
                       await ctx.Items.InsertAsync(new Item { Tag = $"{who}-{b}-{i}" }, txn);
                   await txn.CommitAsync();
                   onCommit(b);
               }
           }
       }
   
       // ── Verify: reopen the database and print the distinct tags of its documents ──
       // (collected first: a scan of a corrupt page chain can loop forever, and is killed then)
       private static async Task<int> Verify(string db, bool dynamic, string? waitFor = null)
       {
           if (waitFor != null)
           {
               Console.WriteLine("READY");
               while (!File.Exists(waitFor)) Thread.SpinWait(50);
           }
           try
           {
               var tags = new HashSet<string>();
               if (dynamic)
               {
                   using var engine = new BLiteEngine(db, Config);
                   await foreach (var d in engine.GetOrCreateCollection("items").FindAllAsync())
                       tags.Add(d.TryGetString("tag", out var t) ? t! : "?");
               }
               else
               {
                   using var ctx = new ReproDbContext(db, Config);
                   await foreach (var item in ctx.Items.FindAllAsync()) tags.Add(item.Tag);
               }
               Console.WriteLine(string.Join("\n", tags));
               return 0;
           }
           catch (Exception ex)
           {
               Console.Error.WriteLine("EXC " + ex);
               return 2;
           }
       }
   
       // ── Coordinator ──────────────────────────────────────────────────────────────────────────
       private static async Task<int> Coordinator(string[] args)
       {
           int Opt(string name, int fallback) => Array.IndexOf(args, name) is var i and >= 0 ? int.Parse(args[i + 1]) : fallback;
           int rounds = Opt("--rounds", 30), procs = Opt("--procs", 2), batches = Opt("--batches", 20), size = Opt("--size", 50);
           int writers = Opt("--writers", procs);
           bool dynamic = args.Contains("--dynamic"), precreate = args.Contains("--precreate"), keep = args.Contains("--keep");
           _timeout = TimeSpan.FromSeconds(Opt("--timeout", 120));
   
           Console.WriteLine($"BLite {typeof(StorageEngine).Assembly.GetName().Version} | {RuntimeInformation.OSDescription} " +
                             $"{RuntimeInformation.OSArchitecture} | .NET {Environment.Version}");
           Console.WriteLine($"rounds={rounds} procs={procs} writers={writers} batches={batches}x{size} api={(dynamic ? "dynamic" : "typed")} " +
                             $"db={(precreate ? "pre-created" : "new")}\n");
   
           var forms = new SortedDictionary<string, int>();
           int crashRounds = 0, lossRounds = 0;
           for (int round = 1; round <= rounds; round++)
           {
               var dir = Directory.CreateTempSubdirectory("blite216_").FullName;
               string db = Path.Combine(dir, "repro.db"), go = Path.Combine(dir, "go");
               if (precreate) await WriteBatches(db, dynamic, "seed", 1, 1, _ => { });
   
               string api = dynamic ? "dynamic" : "typed";
               var children = Enumerable.Range(0, procs)
                   .Select(i => Spawn("child", db, go, $"{i}", $"{(i < writers ? batches : -1)}", $"{size}", api)).ToArray();
               var stderr = children.Select(p => p.StandardError.ReadToEndAsync()).ToArray();
               foreach (var p in children) await p.StandardOutput.ReadLineAsync(); // READY
               File.WriteAllText(go, "");
               var stdout = children.Select(p => p.StandardOutput.ReadToEndAsync()).ToArray();
   
               var outcomes = new string[procs];
               var committed = new HashSet<string>();
               for (int i = 0; i < procs; i++)
               {
                   bool hung = await Exited(children[i]) == false;
                   string err = await stderr[i];
                   outcomes[i] = Classify(children[i].ExitCode, err, hung);
                   if (outcomes[i] != "ok" && forms.TryAdd(outcomes[i], 0))
                       Console.WriteLine($"  first '{outcomes[i]}' (p{i}, exit {children[i].ExitCode}):\n    " +
                                         string.Join("\n    ", err.Split('\n').Take(14)).TrimEnd());
                   if (outcomes[i] != "ok") forms[outcomes[i]]++;
                   foreach (Match m in Regex.Matches(await stdout[i], @"COMMITTED (\S+)"))
                       for (int k = 0; k < size; k++) committed.Add($"{m.Groups[1].Value}-{k}");
               }
   
               var verifier = Spawn("verify", db, api);
               var (tags, verifyErr) = (verifier.StandardOutput.ReadToEndAsync(), verifier.StandardError.ReadToEndAsync());
               bool verifyHung = await Exited(verifier) == false;
               var present = (await tags).Split('\n', StringSplitOptions.TrimEntries).ToHashSet();
               int lost = committed.Count(t => !present.Contains(t));
               string verify = verifier.ExitCode == 0 && !verifyHung
                   ? $"committed {committed.Count}, lost {lost}"
                   : $"VERIFY FAILED: {Classify(verifier.ExitCode, await verifyErr, verifyHung)}";
               if (lost > 0 || verify.StartsWith("VERIFY")) lossRounds++;
   
               bool crashed = outcomes.Any(o => o != "ok");
               if (crashed) crashRounds++;
               Console.WriteLine($"round {round,3}: " + string.Join("  ", outcomes.Select((o, i) => $"p{i}={o}")) + $" | {verify}");
               if (keep && (crashed || verify.Contains("FAILED") || !verify.EndsWith("lost 0"))) Console.WriteLine($"  kept {dir}");
               else try { Directory.Delete(dir, recursive: true); } catch { /* best-effort */ }
           }
   
           Console.WriteLine($"\nrounds with a crashed child: {crashRounds}/{rounds}   rounds with lost/unreadable commits: {lossRounds}/{rounds}");
           Console.WriteLine($"| {"crash form",-70} | count |\n|{new string('-', 72)}|-------|");
           foreach (var (form, count) in forms) Console.WriteLine($"| {form,-70} | {count,5} |");
           return crashRounds + lossRounds == 0 ? 0 : 1;
       }
   
       private static Process Spawn(params string[] args)
       {
           var psi = new ProcessStartInfo(Environment.ProcessPath!) { RedirectStandardOutput = true, RedirectStandardError = true };
           if (Path.GetFileNameWithoutExtension(psi.FileName) == "dotnet") psi.ArgumentList.Add(typeof(Program).Assembly.Location);
           foreach (var arg in args) psi.ArgumentList.Add(arg);
           return Process.Start(psi)!;
       }
   
       // Waits up to --timeout; a process still running then is killed and reported as a hang.
       private static async Task<bool> Exited(Process p)
       {
           try { await p.WaitForExitAsync(new CancellationTokenSource(_timeout).Token); return true; }
           catch (OperationCanceledException) { p.Kill(entireProcessTree: true); await p.WaitForExitAsync(); return false; }
       }
   
       // "<ExceptionType> in <innermost BLite.Core frame>", e.g. "AccessViolationException in PageFile.ReadPageCore".
       // An AccessViolationException cannot be caught: the runtime prints "Fatal error." and aborts the process.
       private static string Classify(int exitCode, string stderr, bool hung)
       {
           if (hung) return "hang (killed)";
           if (exitCode == 0) return "ok";
           var type = Regex.Match(stderr, @"\b(\w+Exception)\b") is { Success: true } t ? t.Groups[1].Value : $"exit code {exitCode}";
           var frame = Regex.Match(stderr, @"at BLite\.Core\.(?:[\w`]+\.)*?([\w`]+\.\w+)\(");
           return (stderr.Contains("Fatal error.") ? "[fatal] " : "") + type + (frame.Success ? " in " + frame.Groups[1].Value : "");
       }
   }

   Results on current main (f8f437b), macOS 27 arm64 (M4 Max), .NET 10.0.12. Each cell shows rounds with a crashed process / rounds with lost commits:

   Variant	main	with the fix (#221)
   2 processes, both write 20×50 (the #134 setup)	27/30, 3/30	no crashes on open; 27/30 lost (see below)
   2 processes, open only	24/30, 0	0/30, 0
   4 processes, open only	10/10, 0	0/10, 0
   1 writer + 1 reader	30/30, 0	0/30, 0/30
   1 writer + 3 readers	10/10, 0	0/10, 0/10
   DB already exists, both write (control)	7/10 (hangs), 10/10 lost	1/10, 9/10 lost

   Crash forms on main:

   • UnauthorizedAccessException in PageFile.WritePageCore;
   • InvalidOperationException "Page size mismatch … 32 byte pages" in PageFile.Open;
   • with a reader process: [fatal] AccessViolationException in PageFile.ReadPageCore and InvalidDataException in PageFile.CheckedPageOffset.

   NuGet 5.1.1: 30/30 rounds. A Linux container (Ubuntu 24.04 arm64) shows the same, including the AccessViolation and IndexOutOfRangeException in DictionaryPage.GetAll, so this is not macOS-specific.

   Root cause. Your hypothesis holds, with two additions (items 3 and 4):

   1. StorageEngine.cs:160-162 creates and opens the page file, and :246-247 initialise the dictionary and KV roots, all before and outside any cross-process lock. The SHM is only opened at :213.
   2. PageFile.Open sizes a new file (PageFile.cs:415) before writing its header (:490), so a second opener reads a zero header (:450).
   3. Each process maps the file at its open-time length (:473), but the bounds checks use the live file length (:605, :733, :875). After another process grows the file, a write past the map fails in CreateViewAccessor (:696). A read past it is a raw memcpy (:627), which ends in an AccessViolation.
   4. Collection creation (EnsureSchema, AppendSchema, the catalog) and new dictionary keys write the page file directly. They use a per-process _nextPageId (:963) and a dictionary cache loaded once (StorageEngine.Dictionary.cs:170), so concurrent creators reuse page ids and dictionary ids.

   Two processes writing at the same time. With the fix, nothing crashes on open any more, but the #134 setup (two writers) now loses committed documents in most rounds. Main does the same when the database already exists (control row), and in the 3 of 30 rounds where neither process crashed. The cause is per-process state: the free-space index, B-tree roots and data-page allocation. MULTI_PROCESS_WAL.md lists simultaneous writers as out of scope ("N readers / 1 writer"), but nothing enforces that. I'd suggest a follow-up that makes a second writer process fail fast, and I can open an issue for it.

   #221 has the regression tests (2 and 4 concurrent creators, a half-initialised file, remap, dictionary ids) and the fix. The fix adds a cross-process open lock (a lock slot next to the WAL writer lock, <shm>.open.lock on macOS) around create/open/init and collection creation, and remaps before touching pages past the map.