Improve caching behavior.

README.md

@@ -21,9 +21,9 @@ Get started at https://featureflags.app, or if you want details first and vibes
 ## What This Library Does
 
 - Registers feature management services in ASP.NET Core via a single `AddFeatureFlags()` call.
-- Fetches feature definitions from our API using an API key header (`x-api-key`).
+- Fetches feature definitions from our API using an API key header (`x-api-key`), in a background service, and keeps the last-known-good copy if a refresh fails.
 - Exposes `IFeatureManager`/`IFeatureManagerSnapshot` usage patterns you already know from `Microsoft.FeatureManagement`.
-- Includes a deterministic percentage filter (`Acmi.FeatureFlags.ConsistentPercentage`) and targeting support.
+- Includes a deterministic percentage filter (`FeatureFlags.ConsistentPercentage`) and targeting support.
 
 ## Package And Runtime
 
@@ -108,12 +108,15 @@ Also works with the normal ASP.NET Core feature management integrations:
 
 ## Cache Behavior
 
-`HttpFeatureFlagClient` caches all retrieved definitions in memory under a single cache entry.
+`AddFeatureFlags()` registers a hosted background service (`FeatureDefinitionRefreshService`) that keeps an in-memory snapshot of all feature definitions.
 
-- First request fetches from remote API.
-- Subsequent requests read from cache until expiration.
-- Expiration defaults to 15 minutes.
-- You can evict cache manually via `IFeatureFlagClient.ClearCache()`.
+- Definitions are fetched at startup. Startup waits up to 5 seconds for the first fetch, then continues without it.
+- After each refresh completes, the next periodic refresh starts after `CacheExpirationInMinutes` (default: `15`); requests time out after 30 seconds.
+- Flag checks read the current snapshot. Evaluation never waits on an HTTP call.
+- The snapshot is swapped atomically after each successful refresh.
+- If a refresh fails (timeout, network error, 5xx, 401/403), the last-known-good snapshot stays in place and the next refresh is retried on the next tick.
+- `IFeatureFlagClient.ClearCache()` requests an immediate background refresh. The old snapshot stays in place until that refresh succeeds. The method name is kept for compatibility.
+- Flag changes typically reach your app within the configured interval plus the time taken by the next refresh (up to 30 seconds), assuming the API responds successfully. Use a shorter interval for apps that rely on kill-switch flags.
 
 Example:
 
@@ -128,7 +131,7 @@ public class AdminController : Controller {
 	[HttpPost]
 	public IActionResult RefreshFlags() {
 		_featureFlagClient.ClearCache();
-		return Ok(new { message = "Feature flag cache cleared." });
+		return Accepted(new { message = "Feature flag refresh requested." });
 	}
 }
 ```
@@ -155,13 +158,13 @@ Use this filter when you want stable rollout behavior for authenticated users in
 
 ## Failure Semantics
 
-When remote API calls fail:
+When a refresh fails:
 
-- Client logs an error.
-- `GetAllFeatureDefinitionsAsync()` returns an empty list.
-- `GetFeatureDefinitionByNameAsync()` returns `null`.
+- The client logs a warning with the reason. During a long outage it logs the first failure and then at most once an hour.
+- The last-known-good definitions keep being served, and the refresh is retried on the next tick.
+- A log message is written when refreshes recover.
 
-In practice this means feature checks degrade to "off" unless your app defines alternate behavior. This is generally safer than throwing exceptions into request pipelines and setting your pager on fire.
+**Cold start with the API down:** if the app starts while FeatureFlags.app is unreachable (or the API key is invalid), no snapshot exists yet. `GetAllFeatureDefinitionsAsync()` returns an empty list, `GetFeatureDefinitionByNameAsync()` returns `null`, and all flags evaluate off until the first successful fetch. Nothing is thrown into the request pipeline.
 
 ## Common Issues And Fixes
 
@@ -182,13 +185,13 @@ Fix:
 Possible causes:
 
 - API key invalid or missing permissions.
-- API unavailable (client degrades to empty definitions).
+- The API was unreachable the whole time since the app started, so no definitions have been loaded (see "Cold start" above).
 - Flag name mismatch (`"NewDashboard"` vs `"NewDashbaord"`, yes this typo happens a lot).
 
 Fix:
 
 - Verify API key and endpoint.
-- Check app logs for "Error fetching feature definitions".
+- Check app logs for "Failed to refresh feature definitions".
 - Centralize flag names in constants to avoid string-literal drift.
 
 ### 3. Rollout percentages look random per request
@@ -202,16 +205,16 @@ Fix:
 - Ensure authenticated identity with stable `User.Identity.Name`.
 - If anonymous traffic dominates, choose filter strategy accordingly.
 
-### 4. Flag updates not visible immediately
+### 4. Flag updates are not visible right away
 
 Cause:
 
-- Cached definitions not yet expired.
+- After each refresh completes, the next periodic refresh starts after `CacheExpirationInMinutes` (15 by default). A change may take that interval plus the time taken by the next refresh (up to 30 seconds) to show up.
 
 Fix:
 
-- Lower `CacheExpirationInMinutes` for development.
-- Call `IFeatureFlagClient.ClearCache()` after admin updates when immediate refresh is required.
+- Lower `CacheExpirationInMinutes` for development, or for apps that rely on kill-switch flags.
+- Call `IFeatureFlagClient.ClearCache()` to request an immediate background refresh. It returns right away, and the new values appear once the refresh completes.
 
 ## Local Validation
 

src/FeatureFlags.Client.Tests/ExtensionsTests.cs

@@ -1,4 +1,3 @@
-using Microsoft.Extensions.Caching.Memory;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Hosting;
@@ -60,7 +59,8 @@ public void AddFeatureFlags_RegistersServicesAndReturnsBuilder() {
         Assert.Same(builderMock.Object, result);
         Assert.Contains(services, s => s.ServiceType == typeof(IFeatureFlagClient));
         Assert.Contains(services, s => s.ServiceType == typeof(IFeatureDefinitionProvider));
-        Assert.Contains(services, s => s.ServiceType == typeof(IMemoryCache));
+        Assert.Contains(services, s => s.ServiceType == typeof(FeatureDefinitionRefreshService));
+        Assert.Contains(services, s => s.ServiceType == typeof(IHostedService) && s.ImplementationFactory is not null);
         Assert.Contains(services, s => s.ServiceType == typeof(IHttpClientFactory));
 
         // Build the service provider and get the factory

src/FeatureFlags.Client.Tests/FeatureDefinitionRefreshServiceTests.cs

@@ -0,0 +1,223 @@
+using System.Net;
+using System.Net.Http.Json;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.Logging;
+using Moq;
+
+namespace Acmi.FeatureFlags.Client.Tests;
+
+public class FeatureDefinitionRefreshServiceTests {
+    private readonly Mock<ILogger<FeatureDefinitionRefreshService>> _LoggerMock = new();
+
+    private sealed class StubHandler(Func<CancellationToken, Task<HttpResponseMessage>> respond) : HttpMessageHandler {
+        public int Calls;
+
+        protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) {
+            Interlocked.Increment(ref Calls);
+            return respond(cancellationToken);
+        }
+    }
+
+    private sealed class ManualTimeProvider : TimeProvider {
+        private long _Timestamp;
+        public override long GetTimestamp() => Interlocked.Read(ref _Timestamp);
+        public override long TimestampFrequency => TimeSpan.TicksPerSecond;
+        public void Advance(TimeSpan by) => Interlocked.Add(ref _Timestamp, by.Ticks);
+    }
+
+    private static HttpResponseMessage Ok(params string[] names)
+        => new(HttpStatusCode.OK) { Content = JsonContent.Create(names.Select(n => new CustomFeatureDefinition { Name = n }).ToList()) };
+
+    private static HttpResponseMessage Status(HttpStatusCode code) => new(code);
+
+    private FeatureDefinitionRefreshService CreateService(StubHandler handler, TimeProvider? timeProvider = null, string minutes = "15") {
+        var factory = new Mock<IHttpClientFactory>();
+        factory.Setup(f => f.CreateClient(Constants.HttpClientName)).Returns(() => new HttpClient(handler, disposeHandler: false) { BaseAddress = new Uri("http://localhost/") });
+        var configuration = new ConfigurationBuilder()
+            .AddInMemoryCollection(new Dictionary<string, string?> { { "FeatureFlags:CacheExpirationInMinutes", minutes } })
+            .Build();
+        return new FeatureDefinitionRefreshService(factory.Object, configuration, _LoggerMock.Object, timeProvider);
+    }
+
+    private static async Task WaitUntilAsync(Func<bool> condition) {
+        var deadline = DateTime.UtcNow.AddSeconds(10);
+        while (!condition()) {
+            Assert.True(DateTime.UtcNow < deadline, "Timed out waiting for condition");
+            await Task.Delay(10, TestContext.Current.CancellationToken);
+        }
+    }
+
+    private void VerifyWarnings(Times times)
+        => _LoggerMock.Verify(l => l.Log(LogLevel.Warning, It.IsAny<EventId>(), It.IsAny<It.IsAnyType>(), It.IsAny<Exception?>(), It.IsAny<Func<It.IsAnyType, Exception?, string>>()), times);
+
+    [Fact]
+    public async Task RefreshAsync_Success_ReplacesSnapshot() {
+        var responses = new Queue<HttpResponseMessage>([Ok("A"), Ok("B", "C")]);
+        var service = CreateService(new StubHandler(_ => Task.FromResult(responses.Dequeue())));
+
+        Assert.True(await service.RefreshAsync(TestContext.Current.CancellationToken));
+        Assert.Equal(["A"], service.GetDefinitions().Select(d => d.Name));
+
+        Assert.True(await service.RefreshAsync(TestContext.Current.CancellationToken));
+        Assert.Equal(["B", "C"], service.GetDefinitions().Select(d => d.Name));
+        Assert.Null(service.GetDefinition("A"));
+        Assert.NotNull(service.GetDefinition("c"));
+    }
+
+    [Theory]
+    [InlineData(HttpStatusCode.InternalServerError)]
+    [InlineData(HttpStatusCode.Unauthorized)]
+    [InlineData(HttpStatusCode.Forbidden)]
+    public async Task RefreshAsync_BadStatus_KeepsPreviousSnapshot(HttpStatusCode failure) {
+        var responses = new Queue<HttpResponseMessage>([Ok("A"), Status(failure)]);
+        var service = CreateService(new StubHandler(_ => Task.FromResult(responses.Dequeue())));
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+
+        Assert.False(await service.RefreshAsync(TestContext.Current.CancellationToken));
+
+        Assert.Equal(["A"], service.GetDefinitions().Select(d => d.Name));
+        VerifyWarnings(Times.Once());
+    }
+
+    [Fact]
+    public async Task RefreshAsync_NetworkError_KeepsPreviousSnapshot() {
+        var calls = 0;
+        var service = CreateService(new StubHandler(_ => ++calls == 1 ? Task.FromResult(Ok("A")) : throw new HttpRequestException("boom")));
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+
+        Assert.False(await service.RefreshAsync(TestContext.Current.CancellationToken));
+
+        Assert.Equal(["A"], service.GetDefinitions().Select(d => d.Name));
+    }
+
+    [Fact]
+    public async Task RefreshAsync_RecoversAfterOutage() {
+        var responses = new Queue<HttpResponseMessage>([Ok("A"), Status(HttpStatusCode.ServiceUnavailable), Status(HttpStatusCode.ServiceUnavailable), Ok("A", "B")]);
+        var service = CreateService(new StubHandler(_ => Task.FromResult(responses.Dequeue())));
+
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        Assert.Single(service.GetDefinitions());
+
+        Assert.True(await service.RefreshAsync(TestContext.Current.CancellationToken));
+        Assert.Equal(["A", "B"], service.GetDefinitions().Select(d => d.Name));
+    }
+
+    [Fact]
+    public async Task RefreshAsync_RepeatedFailures_LogsFirstThenThrottles() {
+        var time = new ManualTimeProvider();
+        var service = CreateService(new StubHandler(_ => Task.FromResult(Status(HttpStatusCode.BadGateway))), time);
+
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        VerifyWarnings(Times.Once());
+
+        time.Advance(TimeSpan.FromHours(1));
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+        VerifyWarnings(Times.Exactly(2));
+    }
+
+    [Fact]
+    public async Task ColdStart_ApiDown_EvaluatesOffWithoutThrowing() {
+        var service = CreateService(new StubHandler(_ => throw new HttpRequestException("down")));
+        var client = new HttpFeatureFlagClient(service);
+
+        Assert.False(await service.RefreshAsync(TestContext.Current.CancellationToken));
+
+        Assert.False(service.HasSnapshot);
+        Assert.Empty(await client.GetAllFeatureDefinitionsAsync(TestContext.Current.CancellationToken));
+        Assert.Null(await client.GetFeatureDefinitionByNameAsync("A", TestContext.Current.CancellationToken));
+    }
+
+    [Fact]
+    public async Task Reads_DoNotBlockOnInFlightRefresh() {
+        var gate = new TaskCompletionSource<HttpResponseMessage>(TaskCreationOptions.RunContinuationsAsynchronously);
+        var calls = 0;
+        var handler = new StubHandler(_ => ++calls == 1 ? Task.FromResult(Ok("A")) : gate.Task);
+        var service = CreateService(handler);
+        var client = new HttpFeatureFlagClient(service);
+        await service.RefreshAsync(TestContext.Current.CancellationToken);
+
+        var inFlight = service.RefreshAsync(TestContext.Current.CancellationToken);
+        await WaitUntilAsync(() => handler.Calls == 2);
+
+        var read = client.GetAllFeatureDefinitionsAsync(TestContext.Current.CancellationToken);
+        Assert.True(read.IsCompletedSuccessfully);
+        Assert.Equal(["A"], (await read).Select(d => d.Name));
+        Assert.False(inFlight.IsCompleted);
+
+        gate.SetResult(Ok("B"));
+        await inFlight;
+        Assert.Equal(["B"], service.GetDefinitions().Select(d => d.Name));
+    }
+
+    [Fact]
+    public async Task HostedService_RefreshesAtStartup_AndStopsCleanly() {
+        var handler = new StubHandler(_ => Task.FromResult(Ok("A")));
+        var service = CreateService(handler);
+
+        await service.StartAsync(TestContext.Current.CancellationToken);
+
+        Assert.True(service.HasSnapshot);
+        await service.StopAsync(TestContext.Current.CancellationToken);
+        Assert.Equal(1, handler.Calls);
+    }
+
+    [Fact]
+    public async Task HostedService_RefreshesOnInterval() {
+        var handler = new StubHandler(_ => Task.FromResult(Ok("A")));
+        var service = CreateService(handler, minutes: "0.0002");
+
+        await service.StartAsync(TestContext.Current.CancellationToken);
+        await WaitUntilAsync(() => handler.Calls >= 3);
+        await service.StopAsync(TestContext.Current.CancellationToken);
+    }
+
+    [Fact]
+    public async Task ClearCache_TriggersImmediateRefresh() {
+        var calls = 0;
+        var handler = new StubHandler(_ => Task.FromResult(++calls == 1 ? Ok("A") : Ok("B")));
+        var service = CreateService(handler);
+        var client = new HttpFeatureFlagClient(service);
+        await service.StartAsync(TestContext.Current.CancellationToken);
+
+        Assert.True(client.ClearCache());
+        await WaitUntilAsync(() => service.GetDefinition("B") is not null);
+
+        Assert.Equal(["B"], service.GetDefinitions().Select(d => d.Name));
+        await service.StopAsync(TestContext.Current.CancellationToken);
+    }
+
+    [Fact]
+    public async Task ClearCache_DuringOutage_KeepsOldValues() {
+        var calls = 0;
+        var handler = new StubHandler(_ => Task.FromResult(++calls == 1 ? Ok("A") : Status(HttpStatusCode.InternalServerError)));
+        var service = CreateService(handler);
+        var client = new HttpFeatureFlagClient(service);
+        await service.StartAsync(TestContext.Current.CancellationToken);
+
+        Assert.True(client.ClearCache());
+        await WaitUntilAsync(() => handler.Calls >= 2);
+
+        var definitions = await client.GetAllFeatureDefinitionsAsync(TestContext.Current.CancellationToken);
+        Assert.Equal(["A"], definitions.Select(d => d.Name));
+        await service.StopAsync(TestContext.Current.CancellationToken);
+    }
+
+    [Fact]
+    public async Task StartAsync_DoesNotHangWhenApiNeverResponds() {
+        var handler = new StubHandler(async ct => {
+            await Task.Delay(Timeout.Infinite, ct);
+            return Ok();
+        });
+        var service = CreateService(handler);
+
+        var started = service.StartAsync(TestContext.Current.CancellationToken);
+        await started.WaitAsync(TimeSpan.FromSeconds(15), TestContext.Current.CancellationToken);
+
+        Assert.False(service.HasSnapshot);
+        await service.StopAsync(TestContext.Current.CancellationToken);
+    }
+}