add metrics for log counts

[jkarneges]

This adds a log_messages_total prometheus metric with a level label, for counting produced log messages. Only messages actually logged (i.e. not filtered out by a level filter) are counted.

Example:

# HELP log_messages_total Total number of log events emitted by level
# TYPE log_messages_total counter
log_messages_total{level="debug"} 19
log_messages_total{level="info"} 2

These metrics should make it possible to monitor for error-level logs by checking counters rather than log scanning, which can be cheaper and allow for more powerful alerting conditions.

[clintjedwards]

Love it.

My only criticism is I probably wouldn't tally all logs, since its pretty rare for me to care about alerting on Info or Debug or even Warn logs.

But, I doubt the cardinality increase by tracking each distinct level matters at all and it might be good to look at in a pinch for extra data. So full steam ahead.

[jkarneges]

Yeah, I agree only error and warn levels are interesting here, so we could consider limiting the counts to those levels.

Note that since only levels within the configured log level are counted, a production system in practice might only produce error+warn level counts anyway.