feat!: only provide the Juju default databag keys while the charm runs

[tonyandrewmeyer]

Relation databags (local_unit_data, remote_units_data, remote_unit_data) now default to empty dicts. While the charm is running, Scenario adds the keys that Juju manages itself (egress-subnets, ingress-address, and on Juju 3 private-address) to every unit databag, and removes them again before the output state is returned, so that the output state has the same shape as the input one. A key that the charm wrote to while it was running is left in place.

The values come from the Network for the relation's endpoint, the way Juju does it: ingress-address (and private-address) is the first ingress address, and egress-subnets is the comma-separated list of egress subnets. With the default network that means egress-subnets is now 192.0.2.0/24 rather than 192.0.2.0. I checked this against 3.6.27 and 4.0.12 on LXD, and in both the databag values match network-get exactly (egress-subnets: 10.5.87.59/32, ingress-address: 10.5.87.59), with 4.0 not setting private-address at all.

This is the backwards-incompatible alternative to #2618: instead of stripping private-address from the construction default when the mocked Juju version is 4+, Scenario provides the keys that the real Juju would provide, sort-of for as long as it would provide them.

Fixes #2185.

[tonyandrewmeyer]

I ran hyrum over the curated charm list to see what this actually breaks, comparing ops @ canonical:main against this branch. Unit tests only, on Ubuntu 24.04 with Python 3.12. 342 of the 748 rows ran: 315 are legacy/reactive charms that hyrum skips, 84 have no unit target, and 7 hit patcher errors.

main: 200 passed, 142 failed. This branch: 199 passed, 143 failed.

The one status flip is karapace-operator, and it isn't this PR. It dies in poetry install with Link.__init__() got an unexpected keyword argument 'size' before collecting a single test. It reproduced across two runs, but on a wiped .tox this branch passes karapace's tests, and main passes on top of that too, so whatever the trigger is (something about poetry moving an already-pinned git commit, I think) it isn't a behaviour change from here.

More usefully than the pass/fail counts, I diffed the individual failing tests across all 433 per-charm logs in both directions: zero tests newly failing, and zero newly passing.

The reason nothing moves is that the charms that care about these keys have already coded defensively, in one of two ways. Either they set the keys explicitly (zookeeper-operator, karapace-operator, and spark-integration-hub-k8s-operator all pass things like local_unit_data={"private-address": "treebeard"}), which this PR leaves untouched, or they strip them before asserting (pyroscope-operators has _purge_default_juju_keys, cos-coordinated-workers has BUILTIN_JUJU_KEYS), which is insensitive in both directions. Nothing hard-codes 192.0.2.0 as an expected egress-subnets.

I checked separately that the changes are observable at all, rather than the fleet simply not exercising them. Running a probe under both versions confirms all five: empty construction defaults, the keys gone from the output state, egress-subnets moving to 192.0.2.0/24, no private-address on Juju 4, and the keys being injected during the run into peer databags and into remote databags the test set explicitly. That last one is the only change that adds content rather than removing it, and it's the one I'd expect to hit someone eventually: a test that sets remote_units_data={0: {"mine": "yes"}} and then compares dict(rel.data[unit]) from inside the charm now sees three extra keys.

Caveats: unit tests only, so no lint. 22 charm environments kept a pinned ops-scenario that the patcher didn't swap, but none of them reference these keys, so I don't think anything is hiding in there; 176 environments were confirmed to have the new code. And traefik-k8s-operator has assert ipa_out.local_unit_data == {"host": ..., "ip": ...}, which I think this PR actually fixes, but it fails dependency resolution on both branches so I couldn't confirm that.

[tonyandrewmeyer]

A thought about a third approach: we could overlay the Juju keys on the read path: in mocking.py's _relation_get, return the databag with the missing Juju keys filled in, and never put them in the state. Then the output state is unchanged by construction, charm writes are preserved for free because they go to the underlying dict, and per-unit values fall out naturally because _relation_get already knows which unit it is answering for. It also means no mutation of State objects mid-run. It feels less close to what Juju does, but then Juju leaves the values in the databag, so this PR is not really accurate either.

[james-garner-canonical]

A thought about a third approach: we could overlay the Juju keys on the read path: in mocking.py's _relation_get, return the databag with the missing Juju keys filled in, and never put them in the state. Then the output state is unchanged by construction, charm writes are preserved for free because they go to the underlying dict, and per-unit values fall out naturally because _relation_get already knows which unit it is answering for. It also means no mutation of State objects mid-run.

This sounds quite elegant.

It feels less close to what Juju does, but then Juju leaves the values in the databag, so this PR is not really accurate either.

I'm not sure I understand this bit. I do think we should be trying to match the semantics of when Juju adds or restores these keys to the databag and when that's visible -- that seems like it will set us up best for when Juju eventually makes another change to these keys.