Update configuration on the fly

[eloekset]

I was looking for an issue about changing configuration without having to redeploy the service. I didn't find any existing issue about that, but I found this TODO comment in ServiceFabricDiagnosticPipelineFactory:

diagnostics-eventflow/src/Microsoft.Diagnostics.EventFlow.ServiceFabric/ServiceFabricDiagnosticPipelineFactory.cs

Line 32 in 3628217

// TODO: dynamically re-configure the pipeline when configuration changes, without stopping the service

I'm thinking about some kind of integration with our custom ConfigurationService, so it would be of great help if the official solution had support for updating configuration on the fly, to get an idea of what's needed.

[karolz-ms]

Eivind Gussiås Løkseth (@eloekset) what kind of EventFlow settings would you like to change on the fly? Can you give us an example?

Dynamic re-configuration is not an easy thing to implement generically because it might involve changing the pipeline composition on the fly and ensuring that no events are lost. Honestly I do not have an idea how to do it. Next best thing is to assume that the pipeline structure cannot change, and it is just that the pipeline elements (inputs, filters, outputs, health reporter) will be notified about configuration change and do their best to react accordingly. That is probably sufficient in most practical scenarios.

[eloekset]

I don’t have an idea how to support it either. What’s most important, is to be able to adjust the filters, and outputs. If filters can be set invidually for each output - that probably needs som restructure of EventFlow - it would be sufficient to be able to adjust filters.

Right now, I don’t even know a way to restart single micro services in Service Fabric to have them load a new configuration, so that makes it very hard to increase logging in a period and then tuning it down again after a while.

All our micro services are implemented in ASP.NET Core, which by design seems to require a restart to be reconfigured.

[karolz-ms]

As described in the docs, filters can be assigned to individual outputs (as opposed to be applied to all outputs)

Technically ASP.NET Core supports dynamic configuration changes via change tokens We do not take advantage of this capability in our SF configuration provider though, this would be something to implement. And then EventFlow pipeline elements would have to react to configuration changes. Once that is done, you would be able to make config changes through config-only SF service upgrade without a full restart.

If the main scenario you are after is to adjust logging level, that can be done without pipeline rebuild, and my proposed solution involving notifying pipeline elements about config change would work. Is there something else you would like to change about EventFlow config dynamically?

[eloekset]

The way you describe updated configuration is just what I was looking for.

And I simply missed the filters block inside the output block in the documentation. That's also what I was looking for. 👍