Better docs

.dockerignore

@@ -0,0 +1,30 @@
+# Version-control and local tooling
+.git
+.github
+.venv
+venv
+.ruff_cache
+.mypy_cache
+.pytest_cache
+.tox
+.nox
+.idea
+.vscode
+
+# Python build and test artifacts
+**/__pycache__
+**/*.py[cod]
+**/*.egg-info
+build
+dist
+site
+htmlcov
+.coverage
+.coverage.*
+
+# Local configuration, credentials, and logs
+.env
+.env.*
+.pypi_token
+*.log
+uv.lock

.gitignore

@@ -3,6 +3,7 @@ site/
 docs/img/
 
 /.*
+!/.dockerignore
 !/.github
 !/.pre-commit-config.yaml
 

config/event_logging.yml

@@ -18,10 +18,16 @@ groups:
     # Two intervals - 5 min and 2 hours for longer-term history in Munin/Icinga
     intervals: ["5m", "2h"]
     # Cache counts locally, push to Redis every second
-    sync-interval: 1
+    sync_interval: 1
   # Number of processed tasks by their "src" attribute
   tasks_by_src:
     events: []
     auto_declare_events: true
     intervals: ["5m", "2h"]
-    sync-interval: 1
+    sync_interval: 1
+  # Execution statistics for callbacks registered with the task executor
+  secondary_hooks:
+    events: []
+    auto_declare_events: true
+    intervals: ["5m", "2h"]
+    sync_interval: 1

docker/python/Dockerfile

@@ -1,15 +1,24 @@
 # syntax=docker/dockerfile:1
 
-# Base interpreter with installed requirements
 FROM python:3.11-slim AS base
-RUN apt-get update; apt-get install -y  \
-    gcc \
-    git
 
-WORKDIR /dp3/
-COPY requirements.txt requirements.txt
-RUN pip install --upgrade pip; \
-    pip install -r requirements.txt
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    UV_LINK_MODE=copy \
+    UV_SYSTEM_PYTHON=1
 
-COPY . /dp3/
-RUN SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0 pip install -e /dp3
+RUN pip install --no-cache-dir "uv==0.9.7"
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends git && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /dp3
+
+COPY requirements.txt ./
+RUN uv pip install --no-cache "setuptools>=61" "setuptools_scm>=6.2" wheel && \
+    uv pip install --no-cache -r requirements.txt
+
+COPY . .
+RUN SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0 \
+    uv pip install --no-cache --no-deps --no-build-isolation -e .

docker/rabbitmq/.dockerignore

@@ -0,0 +1,3 @@
+*
+!Dockerfile
+!init-rmq.sh

docker/rabbitmq/Dockerfile

@@ -5,7 +5,6 @@ FROM rabbitmq:3-management
 ENV RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq
 
 # Add custom configuration script
-ADD init-rmq.sh /init-rmq.sh
-RUN chmod +x /init-rmq.sh
+COPY --chmod=755 init-rmq.sh /init-rmq.sh
 
 CMD ["/init-rmq.sh"]

docs/api.md

@@ -5,7 +5,7 @@ As the API is made using FastAPI, there is also an interactive documentation ava
 
 If you are wiring a new producer into a DP³ application, start with [How to add an input module](howto/add-input.md), then use this page as the endpoint reference.
 
-For routine same-host reads and writes, prefer `dp3 sh` or the generated `<APPNAME>sh` wrapper. They provide a shell-oriented interface for the common API workflows documented here. Use raw HTTP requests when you need to exercise the underlying endpoint behavior directly.
+For routine same-host reads and writes, prefer [`dp3 sh`](cli.md) or the generated `<APPNAME>sh` wrapper. They provide a shell-oriented interface for the common API workflows documented here. Use raw HTTP requests when you need to exercise the underlying endpoint behavior directly. For an operational walkthrough, see [How to inspect DP³ telemetry](howto/telemetry.md).
 
 There are several API endpoints:
 

docs/cli.md

@@ -0,0 +1,7 @@
+# `dp3 sh` command reference
+
+`dp3 sh` is a shell-oriented client for reading from and writing to a running DP³ API. It formats API responses as JSON or newline-delimited JSON and provides commands for datapoints, entities, control actions, telemetry, and shell completion.
+
+On a deployment host, you can use the generated `<APPNAME>sh` wrapper in place of `dp3 sh`. The wrapper supplies the application's configuration directory. Otherwise, select a configuration with `--config` or `DP3_CONFIG_DIR`, and use `--url` when the API is not available through the automatically probed localhost URLs.
+
+{{ dp3_sh_help() }}

docs/configuration/event_logging.md

@@ -21,13 +21,19 @@ groups:
       - task_processed
       - task_processing_error
     intervals: [ "5m", "2h" ] # (1)!
-    sync-interval: 1 # (2)!
+    sync_interval: 1 # (2)!
   # Number of processed tasks by their "src" attribute
   tasks_by_src:
     events: [ ]
     auto_declare_events: true
     intervals: [ "5s", "5m" ]
-    sync-interval: 1
+    sync_interval: 1
+  # Task-executor hook execution statistics
+  secondary_hooks:
+    events: [ ]
+    auto_declare_events: true
+    intervals: [ "5m", "2h" ]
+    sync_interval: 1
 ```
 
 1. Two intervals - 5 min and 2 hours for longer-term history in Munin/Icinga
@@ -45,8 +51,15 @@ This section describes Redis connection details:
 
 # Groups
 
-The default configuration groups enables logging of events in task execution, namely
-`task_processed` and `task_processing_error`.
+The default groups record task execution, processed tasks by source, and secondary-module hook
+statistics. Hook event names are declared dynamically and use the namespace
+`<hook-family>/<callback>/<context>/<metric>`. The callback is module-qualified but omits bound
+`partial` arguments, while the single context component identifies the hook's entity, attribute,
+or snapshot scope.
+EventCountLogger buffers each group's increments in memory and flushes them according to its
+`sync_interval` or `sync_limit` setting. See the
+[telemetry guide](../howto/telemetry.md#5-check-secondary-module-hooks) for the hook metrics and
+their interpretation.
 
-To learn more about the group configuration for EventCountLogger, 
+To learn more about the group configuration for EventCountLogger,
 please refer to the official [documentation](https://github.com/CESNET/EventCountLogger#configuration).

docs/hooks.md

@@ -26,6 +26,10 @@ The most important split is:
 - **Ingestion-time** hooks see the incoming task or datapoint.
 - **Periodic / already-in-system** hooks see persisted data (`master_record`) or snapshot-time derived values.
 
+A hook can be registered only once in the same hook context. Registering the same callback again
+for the same hook type and entity or attribute raises `ValueError`. The same callback can still be
+bound to different entities, attributes, hook types, or dependency contexts.
+
 ## Quick hook placement guide
 
 Use this as a fast way to choose the right hook family.

docs/howto/add-input.md

@@ -108,7 +108,7 @@ Before running the real input module, send one datapoint manually. This narrows
 === "HTTP (`curl`)"
 
     ```shell
-    curl -X POST 'http://localhost:5000/datapoints' \
+    curl --fail-with-body -X POST 'http://localhost:5000/datapoints' \
       -H 'Content-Type: application/json' \
       --data '[
         {
@@ -178,11 +178,31 @@ If the attribute shows up for the manually sent datapoint but not for the real i
 
 ### Check the API response first
 
+The input module must check the HTTP status code and response body for every
+`POST /datapoints` request. A completed HTTP call only proves that the server responded; it
+does not prove that DP³ accepted the datapoints. Treat every non-2xx response as a failed
+delivery and log enough of its response body to diagnose it.
+
 If `POST /datapoints` returns an error, fix that before inspecting workers or the database. Validation errors are often enough to tell you whether:
 
-- the attribute id is wrong
-- the value type does not match
-- timestamps are missing
+- the entity type or attribute id is wrong
+- the entity id or value has the wrong type
+- required timestamps are missing
+
+### Check the bad datapoint API log
+
+A producer that ignores response codes can appear healthy while every payload is rejected.
+Check `api.datapoint_logger.bad_log` in [`api.yml`](../configuration/api.md). If it contains a
+path, inspect that file for the rejected input and its validation error:
+
+```shell
+grep -n 'bad_log' /path/to/config/api.yml
+tail -n 100 /path/to/configured/bad_dp.json.log
+```
+
+Datapoints in this log failed API validation and never reached RabbitMQ or a worker. If
+`bad_log` is `false`, logging is disabled. To enable it, configure an absolute path whose
+parent directory exists and is writable by the API process, then restart the API.
 
 ### Check API and worker logs
 
@@ -248,6 +268,7 @@ worker logs, and attribute definition again.
 ## Common failure modes
 
 - The producer sends to the wrong API URL.
+- The producer ignores a non-2xx API response, and rejected datapoints are only visible in the configured bad datapoint log.
 - The payload shape does not match the configured attribute type.
 - The attribute was not added to `db_entities` before the producer started sending it.
 - The API accepted the request, but workers are not running or are using old configuration.

docs/howto/extending.md

@@ -76,6 +76,10 @@ There are additional options that can be specified, which affect the way the doc
 
 Even if you create a duplicate code reference description, the mkdocstring-style link still leads to the code reference, as you can see [here][dp3.snapshots.snapshot_hooks.SnapshotTimeseriesHookContainer.register].
 
+## CLI reference generation
+
+The [`dp3 sh` command reference](../cli.md) is generated from the CLI's argparse parsers during each MkDocs build. The `dp3_sh_help` macro in `macros.py` traverses the parser returned by `dp3.bin.sh.build_parser` and renders each command's `format_help()` output. Update command descriptions and argument help in the parser definitions under `dp3/bin/shcmd/`; do not copy command sections into `docs/cli.md` manually.
+
 ## Deployment
 
 The documentation is updated and deployed automatically with each push to selected branches thanks to the configured GitHub Action, which can be found in: `.github/workflows/deploy.yml`.

docs/howto/index.md

@@ -8,6 +8,7 @@ These guides walk through common DP³ application-development tasks step by step
 - [How to add an attribute](add-attribute.md) - Define a new attribute in `db_entities`, roll the configuration out, send test data, and verify that DP³ accepts it.
 - [How to add an input module](add-input.md) - Connect an external producer to the DP³ API and verify that the incoming data is accepted.
 - [How to add a secondary module](add-module.md) - Add worker-side logic that reacts to incoming or stored data and emits derived results.
+- [How to inspect DP³ telemetry](telemetry.md) - Check input activity, stored data, queues, snapshots, periodic processes, and secondary modules in a running application.
 - [How to deploy a DP³ application](deploy-app.md) - Install the backing services, generate the process-manager setup, and operate a live deployment.
 - [How to set up for DP³ platform development](develop-dp3.md) - Prepare the repository for working on DP³ itself, including tests and docs.
 - [How to extend the documentation](extending.md) - Build the docs locally, preview changes, and work with the documentation toolchain.