test(ffe): use checked-in canonical fixture snapshot by leoromanovsky · Pull Request #7511 · DataDog/system-tests

leoromanovsky · 2026-08-14T06:07:07Z

Motivation

Attach the canonical DataDog/ffe-system-test-data corpus to system-tests without requiring submodules.

This follows the checked-in snapshot and automated updater model merged in dd-trace-dotnet#8616. Local and CI tests work from a normal clone, while upstream fixture changes still arrive through an automated draft PR.

This supersedes the fixture transport proposed in #6594 and #6901. Those drafts used a git submodule and required initialization changes throughout CI.

Changes

Replace the copied root-level parametric fixtures with a canonical snapshot from ffe-system-test-data@ea8b5cc.
Load all 37 fixture files and 266 evaluation cases from the snapshot.
Assert canonical values, reasons, and the 15 fixture cases that specify errorCode.
Point the Remote Configuration and mocked agentless-backend consumers at the same canonical UFC.
Add a repository-owned updater plus a weekly/manual workflow that opens a signed draft PR only when fixture contents change.
Document the local refresh command and upstream-first contribution flow.

Decisions

Keep the fixture snapshot checked in so ordinary clones, local unit tests, and CI all work without additional git setup or network access.
Keep copy policy in system-tests through a client-owned disallow list. The updater also validates refs and fixture structure and rejects symbolic links.
Compare allowed contents independently of generated source metadata, so documentation-only upstream commits do not create empty update PRs.
Let automated fixture PRs go red when a new case catches a downstream evaluator bug. Those PRs are drafts, so the failure is a useful conformance signal rather than a blocker on changing the shared fixture repository.
Use the repository's existing Octo STS plus peter-evans/create-pull-request signed-commit pattern.

Verification

python3 utils/scripts/update_ffe_fixtures.py --ref ea8b5cc5ce335109f11f3efbc5fd608f98a3ca54
- loaded 266 canonical cases;
- a second run reported changed: false.
./run.sh TEST_THE_TEST tests/test_the_test/test_update_ffe_fixtures.py: 13 passed.
./run.sh TEST_THE_TEST tests/test_the_test/test_mock_ffe_agentless_backend.py: 7 passed.
Ruff format/lint, mypy, yamllint, and git diff --check: passed.

The initial snapshot exactly matches the merged .NET snapshot except for its generated repository-specific SOURCE.md.

Workflow

Add or change shared behavior in ffe-system-test-data.
The weekly workflow, or a manual dispatch for a branch/commit, refreshes this snapshot.
If allowed fixture contents changed, the workflow opens a signed draft PR.
CI runs the updated cases against downstream tracers; fix or classify any newly exposed evaluator bugs in that update PR.
Merge the update only after the expected downstream state is understood.

github-actions · 2026-08-14T06:07:40Z

CODEOWNERS have been resolved as:

.github/workflows/update-ffe-fixtures.yml                               @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/SOURCE.md                @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-invalid-condition-operands.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-invalid-regex-isolation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-invalid-shard-bounds-isolation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-malformed-flag-isolation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-microsecond-date-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-missing-split-shards-isolation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-null-shard-range-isolation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-null-targeting-key.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-numeric-one-of-default.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-regex-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-semver-comparison-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-semver-validation-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-unknown-fields-tolerance.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-unknown-operator-isolation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-variant-type-mismatch.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-flag-that-does-not-exist.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/test_the_test/test_update_ffe_fixtures.py                         @DataDog/system-tests-core
utils/scripts/update_ffe_fixtures.py                                    @DataDog/system-tests-core
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/mocked_backend/ffe.py                                             @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-boolean-false-assignment.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-boolean-one-of-matches.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-comparator-operator-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-disabled-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-empty-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-empty-string-variation.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-falsy-value-assignments.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-flag-with-empty-string.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-integer-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-kill-switch-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-new-user-onboarding-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-no-allocations-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-null-operator-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-numeric-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-numeric-one-of.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-of-7-empty-targeting-key.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-start-and-end-date-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-json-config-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-no-allocations-flag.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-special-characters.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-string-with-special-characters.json  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/ffe-system-test-data/ufc-config.json          @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-case-regex-flag.json                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test-flag-that-does-not-exist.json            @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-08-14T06:14:00Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 17 Pipeline jobs failed

Testing the test | System Tests (dotnet, prod) / parametric / parametric (1)

🔧 Fix in code. This looks caused by changes in this PR.

1 failed test. AssertionError: No RemoteConfig apply status found, got requests [] at utils/docker_fixtures/_test_agent.py:909.

🧪 1 Test failed

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_remote_config[library_env0, parametric-dotnet]

from system_tests_suite

AssertionError: No RemoteConfig apply status found, got requests [{&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw0-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-81256&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.1.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;adb9a4ee-7a0b-4e85-9ed4-0db6110c6e13&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;ed4eb7de-5047-4c90-913f-132311933c81&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.1&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;tracer_version:3.51.1&#39;, &#39;host_name:34015c96b42f&#39;, &#39;git.commit.sha:&#39;, &#39;git.repository_url:&#39;]}, &#39;capabilities&#39;: &#39;Y8CgCPf&#43;&#39;}, &#39;cached_target_files&#39;: []}, &#39;url&#39;: &#39;http://ddapm-test-agent-worker-gw0-5740354900026072187:8126/v0.7/config&#39;, &#39;method&#39;: &#39;POST&#39;}, {&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw0-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-81256&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.1.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;traceparent&#39;: &#39;00-027a6371a5c6a52544e9f093cfdd9477-617f22a18c27ea50-00&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;adb9a4ee-7a0b-4e85-9ed4-0db6110c6e13&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;, &#39;LIVE_DEBUGGING_SYMBOL_DB&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;ed4eb7de-5047-4c90-913f-132311933c81&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.1&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;tracer_version:3.51.1&#39;, &#39;host_name:34015c96b42f&#39;, &#39;git.commit.sha:&#39;, &#39;git.repository_url:&#39;]}, &#39;capabilities&#39;: &#39;Y8CgCPf&#43;&#39;}, &#39;cached_target_files&#39;: []}, &#39;url&#39;: &#39;http://ddapm-test-agent-worker-gw0-5740354900026072187:8126/v0.7/config&#39;, &#39;method&#39;: &#39;POST&#39;}, {&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw0-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-81256&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.1.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;traceparent&#39;: &#39;00-341dd49e398eca0feb11f629143bc5ef-81e79720ac3e0224-00&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;adb9a4ee-7a0b-4e85-9ed4-0db6110c6e13&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;, &#39;LIVE_DEBUGGING_SYMBOL_DB&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;ed4eb7de-5047-4c90-913f-132311933c81&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.1&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;tracer_version:3.51.1&#39;, &#39;host_name:34015c96b42f&#39;, &#39;git.commit.sha:&#39;, &#39;git.repository_url:&#39;]}, &#39;capabilities&#39;: &#39;Y8CgCPf&#43;&#39;}, &#39;cached_target_files&#39;: []}, &#39;url&#39;: &#39;http://ddapm-test-agent-worker-gw0-5740354900026072187:8126/v0.7/config&#39;, &#39;method&#39;: &#39;POST&#39;}, {&#39;headers&#39;: {&#39;Host&#39;: &#39;ddapm-test-agent-worker-gw0-5740354900026072187:8126&#39;, &#39;Datadog-Entity-ID&#39;: &#39;in-81256&#39;, &#39;Datadog-Meta-Lang&#39;: &#39;.NET&#39;, &#39;Datadog-Meta-Tracer-Version&#39;: &#39;3.51.1.0&#39;, &#39;x-datadog-tracing-enabled&#39;: &#39;false&#39;, &#39;Connection&#39;: &#39;close&#39;, &#39;Transfer-Encoding&#39;: &#39;chunked&#39;, &#39;traceparent&#39;: &#39;00-e4e1b3dfd08af49bef8925a8180327c9-06ae30bbe0300c96-00&#39;, &#39;Content-Type&#39;: &#39;application/json&#39;}, &#39;body&#39;: {&#39;client&#39;: {&#39;state&#39;: {&#39;root_version&#39;: 1, &#39;targets_version&#39;: 0, &#39;config_states&#39;: [], &#39;has_error&#39;: False, &#39;error&#39;: None, &#39;backend_client_state&#39;: None}, &#39;id&#39;: &#39;adb9a4ee-7a0b-4e85-9ed4-0db6110c6e13&#39;, &#39;products&#39;: [&#39;ASM_FEATURES&#39;, &#39;ASM_DD&#39;, &#39;FFE_FLAGS&#39;, &#39;APM_TRACING&#39;, &#39;AGENT_CONFIG&#39;, &#39;AGENT_TASK&#39;, &#39;LIVE_DEBUGGING_SYMBOL_DB&#39;], &#39;is_tracer&#39;: True, &#39;client_tracer&#39;: {&#39;runtime_id&#39;: &#39;ed4eb7de-5047-4c90-913f-132311933c81&#39;, &#39;language&#39;: &#39;dotnet&#39;, &#39;tracer_version&#39;: &#39;3.51.1&#39;, &#39;service&#39;: &#39;apmtestapi&#39;, &#39;process_tags&#39;: [&#39;entrypoint.basedir:app&#39;, &#39;entrypoint.name:program&#39;, &#39;entrypoint.workdir:app&#39;, &#39;svc.auto:apmtestapi&#39;], &#39;extra_services&#39;: None, &#39;env&#39;: None, &#39;app_version&#39;: None, &#39;tags&#39;: [&#39;

Testing the test | System Tests (golang, dev) / parametric / parametric (2)

🔧 Fix in code. This looks caused by changes in this PR.

Test run failed. AssertionError: Test case 0 in test-case-start-and-end-date-flag.json failed: flag='start-and-end-date-test', targetingKey='alice', expected reason=DEFAULT, actual reason=None

🧪 37 Tests failed · 🎯 related to the job error above

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-start-and-end-date-flag.json-library_env0, parametric-golang]

from system_tests_suite

AssertionError: Test case 0 in test-case-start-and-end-date-flag.json failed: flag=&#39;start-and-end-date-test&#39;, targetingKey=&#39;alice&#39;, expected reason=DEFAULT, actual reason=None
assert None == &#39;DEFAULT&#39;

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fb5195f85f0&gt;
test_case_file = &#39;test-case-start-and-end-date-flag.json&#39;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fb518a5ade0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb51892a180&gt;

    @parametrize(&#34;library_env&#34;, [{**DEFAULT_ENVVARS}])
    @parametrize(&#34;test_case_file&#34;, ALL_TEST_CASE_FILES)
...

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-boolean-false-assignment.json-library_env0, parametric-golang]

from system_tests_suite

AssertionError: Test case 0 in test-case-boolean-false-assignment.json failed: flag=&#39;boolean-false-assignment&#39;, targetingKey=&#39;alice&#39;, expected reason=TARGETING_MATCH, actual reason=None
assert None == &#39;TARGETING_MATCH&#39;

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fb519296f30&gt;
test_case_file = &#39;test-case-boolean-false-assignment.json&#39;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fb518a5ade0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb5182665a0&gt;

    @parametrize(&#34;library_env&#34;, [{**DEFAULT_ENVVARS}])
    @parametrize(&#34;test_case_file&#34;, ALL_TEST_CASE_FILES)
...

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-boolean-one-of-matches.json-library_env0, parametric-golang]

from system_tests_suite

AssertionError: Test case 0 in test-case-boolean-one-of-matches.json failed: flag=&#39;boolean-one-of-matches&#39;, targetingKey=&#39;alice&#39;, expected reason=TARGETING_MATCH, actual reason=None
assert None == &#39;TARGETING_MATCH&#39;

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fb519296f00&gt;
test_case_file = &#39;test-case-boolean-one-of-matches.json&#39;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fb518a5ade0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb518175bb0&gt;

    @parametrize(&#34;library_env&#34;, [{**DEFAULT_ENVVARS}])
    @parametrize(&#34;test_case_file&#34;, ALL_TEST_CASE_FILES)
...

View all failed tests

Testing the test | System Tests (java, prod) / parametric / parametric (1)

🔧 Fix in code. This looks caused by changes in this PR.

Assertion Error: Expected reason='SPLIT', actual reason='DEFAULT' in test case 0 of test-json-config-flag.json

🧪 31 Tests failed · 🎯 related to the job error above

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-missing-split-shards-isolation.json-library_env0, parametric-java]

from system_tests_suite

AssertionError: Test case 0 in test-case-missing-split-shards-isolation.json failed: flag=&#39;missing-split-shards-flag&#39;, targetingKey=&#39;alice&#39;, expected=default, actual=trap
assert &#39;trap&#39; == &#39;default&#39;
  - default
  &#43; trap

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fc4690b7980&gt;
test_case_file = &#39;test-case-missing-split-shards-isolation.json&#39;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fc4686127e0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fc4681d8440&gt;

...

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-json-config-flag.json-library_env0, parametric-java]

from system_tests_suite

AssertionError: Test case 0 in test-json-config-flag.json failed: flag=&#39;json-config-flag&#39;, targetingKey=&#39;alice&#39;, expected reason=SPLIT, actual reason=DEFAULT
assert &#39;DEFAULT&#39; == &#39;SPLIT&#39;
  - SPLIT
  &#43; DEFAULT

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f1137fd8800&gt;
test_case_file = &#39;test-json-config-flag.json&#39;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f11075c4ce0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f11077d68a0&gt;

...

🎯

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-special-characters.json-library_env0, parametric-java]

from system_tests_suite

AssertionError: Test case 0 in test-special-characters.json failed: flag=&#39;special-characters&#39;, targetingKey=&#39;ash&#39;, expected reason=SPLIT, actual reason=DEFAULT
assert &#39;DEFAULT&#39; == &#39;SPLIT&#39;
  - SPLIT
  &#43; DEFAULT

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fa169938230&gt;
test_case_file = &#39;test-special-characters.json&#39;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fa1687d8c20&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fa1689afb90&gt;

...

View all failed tests

View all 17 failed jobs.

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: agent/ffe-fixture-snapshot

Testing the test | System Tests (dotnet, prod) / parametric / parametric (1)
Commit: 3afbe522b2462494f30a5b65caf908e4a117fa13
Error (code / test):
1 failed test. AssertionError: No RemoteConfig apply status found, got requests [] at utils/docker_fixtures/_test_agent.py:909.
CI job: https://github.com/DataDog/system-tests/actions/runs/31775208508/job/94705145646

Testing the test | System Tests (golang, dev) / parametric / parametric (2)
Commit: 3afbe522b2462494f30a5b65caf908e4a117fa13
Error (code / test):
5 failed tests. AssertionError: Test case 0 in test-case-start-and-end-date-flag.json failed: flag='start-and-end-date-test', targetingKey='alice', expected reason=DEFAULT, actual reason=None
CI job: https://github.com/DataDog/system-tests/actions/runs/31775208508/job/94705144910

Testing the test | System Tests (java, prod) / parametric / parametric (1)
Commit: 3afbe522b2462494f30a5b65caf908e4a117fa13
Error (code / test):
Assertion Error: Expected reason='SPLIT', actual reason='DEFAULT' in test case 0 of test-json-config-flag.json
CI job: https://github.com/DataDog/system-tests/actions/runs/31775208508/job/94705144020

Plus 9 more failing jobs not shown here.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 32 jobs - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: fd2ebcf | Docs | Datadog PR Page | Give us feedback!}

test(ffe): sync canonical fixture snapshot

fd2ebcf

Navigation Menu

Uh oh!

test(ffe): use checked-in canonical fixture snapshot - #7511

test(ffe): use checked-in canonical fixture snapshot#7511
leoromanovsky wants to merge 1 commit into
mainfrom
agent/ffe-fixture-snapshot

leoromanovsky commented Aug 14, 2026

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 14, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

leoromanovsky commented Aug 14, 2026

Motivation

Changes

Decisions

Verification

Workflow

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 14, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

datadog-datadog-prod-us1 Bot commented Aug 14, 2026 •

edited by datadog-prod-us1-5 Bot

Loading