Fix traefik tls certs during refresh

[Thanhphan1147]

The release of traefik revision 281 introduces managing certificates in "APP" mode. However, due to an issue with the tls-certiicates library, the corresponding private_key did not had the correct scope, which introduce a risk of service disruption . This PR proposes a fix so that deployments that are in a "degraded" state can self-heal after running the sunbeam refresh command.

The PR adds a custom logic towards the end of the refresh command to find outstanding CSRs, check that:

1. The subject matches the certificates provided by the operator via sunbeam tls ca unit_certs
2. The public key of the certificate matches the outstanding CSR
3. The certificate has not expired

Then rerun provide-certifcate action on manual-tls-certificates using a certificate that matched from the previous step.

Assisted-By: copilot

Related-PR: https://github.com/canonical/traefik-k8s-operator/pull/729/changes
Related-bug: canonical/traefik-k8s-operator#709

QA steps

These steps need to be performed on a sunbeam deployment after enabling tls ca for the public endpoints

1. Save the current private key

TLS_KEY_SECRET_ID=$(juju secrets --format yaml | python3 -c "
import sys,yaml
s=yaml.safe_load(sys.stdin)
[print(k) for k,v in s.items() if v.get(label)==tls-key]")

juju show-secret ${TLS_KEY_SECRET_ID} --reveal --format yaml \
    | python3 -c "
import sys,yaml,json
d=list(yaml.safe_load(sys.stdin).values())[0]
yaml.dump({private-keys:d[content][private-keys]},open(/tmp/orig.yaml,w))"

2. Replace the current private key with a random one ( we'll throw it away later, it's just to simulate a replacement of the CSR )

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/new_tls_key.yaml
jhack fire traefik-public/0 config-changed
sleep 15

3. Restore the original key

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/orig.yaml
jhack fire traefik-public/0 config-changed
sleep 15

4. Re-provide certificate

openstack.sunbeam cluster refresh

[hmlanigan]

Question: how do we test this charnge?

[Raven-182]

Thanks for the patch. Some comments:

• ce0361a and f8b7da7 seem like fixups to 268ebdd so should be squashed.

[Thanhphan1147]

@Raven-182 I squashed all commits into 1

[Thanhphan1147]

@hmlanigan Here are the steps to test the changes ( I tested it on a 1-node sunbeam deployment after enabling tls ca for the public endpoints). The idea is to simulate a replacement of CSR after a leader change for example.

1. Save the current private key

TLS_KEY_SECRET_ID=$(juju secrets --format yaml | python3 -c "
import sys,yaml
s=yaml.safe_load(sys.stdin)
[print(k) for k,v in s.items() if v.get('label')=='tls-key']")

juju show-secret ${TLS_KEY_SECRET_ID} --reveal --format yaml \
    | python3 -c "
import sys,yaml,json
d=list(yaml.safe_load(sys.stdin).values())[0]
yaml.dump({'private-keys':d['content']['private-keys']},open('/tmp/orig.yaml','w'))"

2. Replace the current private key with a random one ( we'll throw it away later, it's just to simulate a replacement of the CSR )

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/new_tls_key.yaml
jhack fire traefik-public/0 config-changed
sleep 15

3. Restore the original key

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/orig.yaml
jhack fire traefik-public/0 config-changed
sleep 15

4. Re-provide certificate

openstack.sunbeam cluster refresh

or

openstack.sunbeam cluster refresh certificates

To only trigger the certificate refresh.

Do you think I need to turn this into a doc or integration test?

[hmlanigan]

@Thanhphan1147, the QA steps are for someone to be able to verify the changes in the PR. The tests added should be appropriate for regression.

[Thanhphan1147]

Hi @gboutry / @hmlanigan, any updated on this?

[hmlanigan]

Hi @gboutry / @hmlanigan, any updated on this?

The code looks okay to me, we have someone verifying the change now.

[hemanthnakkina]

Hi @gboutry / @hmlanigan, any updated on this?

The code looks okay to me, we have someone verifying the change now.

If my understanding is correct, this is an issue when upgrading from speciifc traefik-k8s revision or less. There are already process in place to avoid these kind of issues in future like SolQA tests before traefik-k8s promotion.

Adding new command sunbeam cluster refresh certificates does not make sense.
I think it is enough to handle this in sunbeam cluster refresh ONLY for stable/2024.1 branch

2025.1 is SLURP release, the chances someone uses sunbeam with 2025.1 is very less
2026.1 is not yet released

Thoughts?

[gboutry]

I agree, this should be maintained on the stable version that has a chance to encounter this issue, rather than being maintained on version that will not.

[hemanthnakkina]

@Thanhphan1147 Please see the latest comments in conversation

[Thanhphan1147]

@hemanthnakkina @gboutry I have removed the sunbeam cluster refresh certificates command and kept the logic only in sunbeam cluster refresh. I have also rebased the branch on top of stable/2024.1 and squashed the commits into a single commit.