diff --git a/.security/architecture.md b/.security/architecture.md new file mode 100644 index 0000000..0959c79 --- /dev/null +++ b/.security/architecture.md @@ -0,0 +1,55 @@ +> Derived-from: snowflakedb/snowflake-telemetry-python@8b2709d · generated 2026-04-08 +> Regenerate when: a new top-level package is added under `src/snowflake/telemetry/`, or a socket/HTTP/file-write call is added anywhere in `src/` + +## What this is + +`snowflake-telemetry-python` is a Python library, not a service: it has no listener, +no ingress, and no deployment topology of its own. It is installed into, and runs +inside, whatever Python process imports it — per `README.md`, that process is a +Python UDF, UDTF, or Stored Procedure executing inside Snowflake's Python runtime, or +any other Python program a developer chooses to add the dependency to. + +## Components + +- **`snowflake.telemetry`** (`src/snowflake/telemetry/__init__.py`) — the public API: + `add_event()` and `set_span_attribute()`, both thin wrappers around + `opentelemetry.trace.get_current_span()`. +- **`snowflake.telemetry.logs`** (`src/snowflake/telemetry/logs/__init__.py`) — + `SnowflakeLogFormatter`, a `logging.Formatter` that renders a Python `LogRecord` as + the JSON shape Snowflake's log ingestion expects. +- **`snowflake.telemetry.trace`** (`src/snowflake/telemetry/trace/__init__.py`) — + `SnowflakeTraceIdGenerator`, an OpenTelemetry `RandomIdGenerator` subclass. +- **`snowflake.telemetry._internal.serialize`** + (`src/snowflake/telemetry/_internal/serialize/__init__.py`) — a hand-written, + dependency-free protobuf wire-format encoder (`MessageMarshaler`, `Varint`) that + every generated marshaler class below is built on. This exists so the package does + not need to depend on the `protobuf` runtime package (`CHANGELOG.md`, 0.6.0 entry). +- **`snowflake.telemetry._internal.opentelemetry.proto.*`** — marshaler classes for + the OTLP trace/log/metric/resource/common protobuf messages, generated by + `scripts/proto_codegen.sh` from the `open-telemetry/opentelemetry-proto` schema + (see `threat-model.md` and `secure-coding.md` for how that generation is kept + verifiable). +- **`snowflake.telemetry._internal.opentelemetry.exporter.otlp.proto.common`** — + encoder functions (`encode_spans`, `encode_logs`, `encode_metrics`) vendored from + `open-telemetry/opentelemetry-python` by `scripts/vendor_otlp_proto_common.sh`, + converting OpenTelemetry SDK objects into the generated marshaler types above. +- **`snowflake.telemetry._internal.exporter.otlp.proto.{traces,metrics}`** — + `ProtoSpanExporter`/`ProtoMetricExporter` + (`src/snowflake/telemetry/_internal/exporter/otlp/proto/traces/__init__.py`, + `src/snowflake/telemetry/_internal/exporter/otlp/proto/metrics/__init__.py`), + OpenTelemetry SDK `SpanExporter`/`MetricExporter` + implementations that serialize to OTLP protobuf bytes and then call an + application-supplied `SpanWriter.write_span()` / `MetricWriter.write_metrics()` — + both declared `abc.abstractmethod` with no implementation shipped in this repo. + +## Nothing here is internet-facing + +There is no code in `src/` that opens a socket, makes an HTTP/gRPC call, spawns a +subprocess, or writes a file. A search of `src/` for the regex +`socket|urllib|requests|http\.client|grpc|subprocess|os\.system|open\(` (via both +`rg` and Python's `re` module) returns zero matches anywhere under `src/`. A +broader search (`import socket|import subprocess|import requests|urllib|grpc|os\.system|\bopen\(`) +likewise finds nothing under `src/`, corroborating the "no socket/HTTP/gRPC/subprocess/ +file-write code in `src/`" conclusion above. +See `trust-boundaries.md` for what this means for where data this library serializes +actually goes. diff --git a/.security/secure-coding.md b/.security/secure-coding.md new file mode 100644 index 0000000..956f860 --- /dev/null +++ b/.security/secure-coding.md @@ -0,0 +1,37 @@ +> Derived-from: snowflakedb/snowflake-telemetry-python@8b2709d · generated 2026-04-08 +> Regenerate when: a third vendoring/codegen script is added, or an existing +> `check-*.yml` workflow is removed without its script being removed too + +## Pin the upstream ref in the script, and add a `check-*.yml` workflow that regenerates and diffs + +This repository has two places where code is not hand-written but produced from an +upstream OpenTelemetry repository, and both follow the same pattern: + +- `scripts/proto_codegen.sh` pins the source repo/tag in a shell variable + (`PROTO_REPO_BRANCH_OR_COMMIT="v1.7.0"`, `scripts/proto_codegen.sh:15`), clones + `open-telemetry/opentelemetry-proto` at that ref, and regenerates the marshaler + code under `src/snowflake/telemetry/_internal/opentelemetry/proto/`. + `.github/workflows/check-codegen.yml` runs this script from a clean checkout on + every push/PR that touches `scripts/**` or that generated-code directory, and + fails the build if the regeneration differs from what is committed. +- `scripts/vendor_otlp_proto_common.sh` does the same for + `open-telemetry/opentelemetry-python` (`REPO_BRANCH_OR_COMMIT="v1.38.0"`, + `scripts/vendor_otlp_proto_common.sh:12`), regenerating + `src/snowflake/telemetry/_internal/opentelemetry/exporter/`. + `.github/workflows/check-vendor.yml` is the equivalent CI check. + +**When adding a third source of vendored or generated code, follow the same shape:** +pin the upstream ref in the script itself (not only in a comment), and add a +`check-*.yml` workflow, scoped by `paths:` to the script and its output directory, +that deletes the existing output, regenerates it, and fails on `git diff`. + +**What this convention proves, and what it does not** — see `trust-boundaries.md` +and `threat-model.md` (INV-2, INV-3): it proves the committed code matches what the +pinned ref currently produces. It does not prove the pinned ref's content is safe, +and none of the three refs pinned in this repository's scripts or workflows +(`v1.7.0`, `v1.38.0`, and the third-party GitHub Actions used in the workflow +files under `.github/workflows/`) are pinned to an immutable commit SHA — all are tags or +tag-like refs, which a repository owner could in principle move. Do not describe +this convention as verifying the safety of the upstream source; describe it as +verifying that this repository's copy is in sync with what that source currently +contains. diff --git a/.security/security.yaml b/.security/security.yaml new file mode 100644 index 0000000..1783cb4 --- /dev/null +++ b/.security/security.yaml @@ -0,0 +1,20 @@ +apiVersion: snowflake.com/security/v1alpha1 +kind: SecurityContext +metadata: + name: snowflake-telemetry-python + annotations: + owner: "@snowflakedb/telemetry" +spec: + documents: + - path: architecture.md + kind: context + description: "Package layout of this OTel-extension library and why nothing in it is internet-facing" + - path: trust-boundaries.md + kind: context + description: "Why this is a same-process library (no auth boundary), that it performs no network/file I/O itself, and the one caller-input guard it does enforce (log-formatter reserved attributes)" + - path: threat-model.md + kind: threat-model + description: "Systems this package's own code and build scripts talk to, and the invariants (log-attribute protection, vendored/generated-code sync, publish-token scoping) with their controls and status" + - path: secure-coding.md + kind: secure-coding + description: "The adopted convention for vendoring or regenerating code from an upstream OpenTelemetry repository" diff --git a/.security/threat-model.md b/.security/threat-model.md new file mode 100644 index 0000000..6de0030 --- /dev/null +++ b/.security/threat-model.md @@ -0,0 +1,52 @@ +> Derived-from: snowflakedb/snowflake-telemetry-python@8b2709d · generated 2026-04-08 +> Regenerate when: `setup.py`'s `opentelemetry-api`/`opentelemetry-sdk` pin changes, +> either upstream ref pinned in `scripts/proto_codegen.sh` or +> `scripts/vendor_otlp_proto_common.sh` changes, a CI workflow under +> `.github/workflows/` is added or removed, or an invariant's control changes + +## The system + +`snowflake-telemetry-python` is a **library** (see `architecture.md`), not a service. + +- **Driven by** whatever Python process imports it and calls its API. Per + `README.md`, that is a Python UDF, UDTF, or Stored Procedure running inside + Snowflake's Python execution environment, or any other Python program that adds + the dependency. +- **Changed by** anyone able to merge to this repository; `.github/CODEOWNERS:1` + designates `@snowflakedb/telemetry` as the owner of every path in the tree. +- **Trusted with** no credentials and no secrets — none are read, held, or + transmitted anywhere in `src/`. What it is trusted with is *correctness of + serialization*: every process that calls its API, or wires up + `SnowflakeTraceIdGenerator` / `ProtoSpanExporter` / `ProtoMetricExporter`, relies + on it to produce well-formed OTLP protobuf bytes and to hand them only to the + writer the embedding application supplied (`trust-boundaries.md`) — never to an + implicit destination of its own choosing. Because the package is published to the + public PyPI registry and, via a separate Jenkins-triggered build, to a private + Snowflake-operated conda channel (`trust-boundaries.md`, "Two independent + build/publish pipelines"), a change that broke either property would ship to every + process that installs the package from either channel. +- **Reads** nothing beyond its own Python-level function arguments and, at + maintainer-invoked dev time only, two upstream OpenTelemetry git repositories + cloned by `scripts/proto_codegen.sh` and `scripts/vendor_otlp_proto_common.sh` (see + below). It holds no runtime configuration, no credentials, and no per-tenant state. + +## Systems and trust boundaries + +| System | Role here | Crosses the boundary | Trusted for | +|---|---|---|---| +| `opentelemetry-api` / `opentelemetry-sdk` (PyPI, pinned `==1.38.0` in `setup.py`) | runtime dependency this package extends (`get_current_span`, `RandomIdGenerator`, `SpanExporter`, `MetricExporter`) | installed into, and imported by, every process using this package | that the pinned version behaves as documented — a defect or a compromised release at that exact version would run in every embedding process | +| `open-telemetry/opentelemetry-proto` (git, tag `v1.7.0` pinned at `scripts/proto_codegen.sh:15`) | schema source for the generated marshaler code under `_internal/opentelemetry/proto/` | cloned by a maintainer running `scripts/proto_codegen.sh`; never fetched at package-install or runtime | that the content at the pinned tag, when regenerated, is what CI (`.github/workflows/check-codegen.yml`) sees committed to this repo | +| `open-telemetry/opentelemetry-python` (git, tag `v1.38.0` pinned at `scripts/vendor_otlp_proto_common.sh:12`) | source of the vendored encoder code under `_internal/opentelemetry/exporter/otlp/proto/common/` | cloned by a maintainer running `scripts/vendor_otlp_proto_common.sh`; never fetched at package-install or runtime | that the content at the pinned tag, when regenerated, is what CI (`.github/workflows/check-vendor.yml`) sees committed to this repo | +| GitHub Actions (workflow files under `.github/workflows/`) + third-party actions (`actions/checkout@v3/v4`, `actions/setup-python@v3`, `pypa/gh-action-pypi-publish@release/v1`) | builds, tests, and (on `release: published`) publishes this package | source in, package artifact out; `PYPI_API_TOKEN` secret out to the publish step only | that only this repository's reviewed source is built, and that `PYPI_API_TOKEN` is used by, and visible to, no step other than `python-publish.yml`'s publish step | +| PyPI (public registry) | public distribution channel for the package built by `python-publish.yml` | release artifact out | serving exactly the artifact this repo's workflow uploaded, unaltered, to every `pip install snowflake-telemetry-python` | +| Jenkins job "SnowflakeTelemetryPythonPackageBuilder" + `repo.anaconda.com/pkgs/snowflake/` conda channel (`build.sh:4`, `build.sh:38`) | second, non-GitHub-Actions build/publish path, producing a conda package | source in (checked out by Jenkins, not visible from this repo), package artifact out to the private channel | building the published conda package only from this repository's own reviewed source, the same way `python-publish.yml` does for the PyPI artifact | +| The embedding application's `SpanWriter`/`MetricWriter` subclass | sole consumer of the OTLP protobuf bytes this repo's exporters produce | serialized bytes out, in-process | not silently discarding, corrupting, or misrouting the telemetry data this library hands it | + +## Invariants + +| ID | Invariant | Control | Status | +|---|---|---|---| +| INV-1 | A caller-supplied `logging` `extra=` value cannot overwrite the `code.lineno`, `code.function`, `code.filepath`, `exception.type`, `exception.message`, or `exception.stacktrace` attributes `SnowflakeLogFormatter` derives from the actual call site | `src/snowflake/telemetry/logs/__init__.py` (`_RESERVED_ATTRS`, `SnowflakeLogFormatter.format`) — mechanism detailed in `trust-boundaries.md` | Holding (`tests/test_snowflake_log_formatter.py#test_normal_log`) | +| INV-2 | The vendored encoder code under `_internal/opentelemetry/exporter/otlp/proto/common/` matches what a fresh regeneration from the pinned `opentelemetry-python` tag (`scripts/vendor_otlp_proto_common.sh:12`) produces | `.github/workflows/check-vendor.yml` (regenerates from scratch and fails on `git diff`) | Holding — covers only "the committed code matches the pinned tag's current content", not "that tag's content is safe"; see the systems-table row above and the gap this leaves open | +| INV-3 | The generated marshaler code under `_internal/opentelemetry/proto/` matches what a fresh regeneration from the pinned `opentelemetry-proto` tag (`scripts/proto_codegen.sh:15`) produces | `.github/workflows/check-codegen.yml` (regenerates from scratch and fails on `git diff`) | Holding — same partial-coverage caveat as INV-2 | +| INV-4 | `PYPI_API_TOKEN` is used only inside the publish step of `.github/workflows/python-publish.yml`, and that workflow runs only when a GitHub release is published | `.github/workflows/python-publish.yml` (trigger `release: types: [published]` at `:11-13`; token read only at `:39`, inside the `pypa/gh-action-pypi-publish` step at `:36`) | Holding | diff --git a/.security/trust-boundaries.md b/.security/trust-boundaries.md new file mode 100644 index 0000000..4890b43 --- /dev/null +++ b/.security/trust-boundaries.md @@ -0,0 +1,84 @@ +> Derived-from: snowflakedb/snowflake-telemetry-python@8b2709d · generated 2026-04-08 +> Regenerate when: `SpanWriter`/`MetricWriter` gain a default (non-abstract) +> implementation, `_RESERVED_ATTRS` changes, or any network/file I/O is added to `src/` + +## The caller is the trust boundary, not this code + +This is a library called in-process. Its only "input" is whatever arguments the +embedding Python process passes to `snowflake.telemetry.add_event()`, +`snowflake.telemetry.set_span_attribute()`, or a `logging` call that flows through a +`logging.Handler` configured with `SnowflakeLogFormatter` +(`src/snowflake/telemetry/logs/__init__.py`). The caller and the code executing this +library's functions run as the same principal, in the same process — there is no +network hop, no serialization boundary, and no authentication step inside this +repository between "whoever can execute Python here" and "whoever can call this +API." Any code that can import `snowflake.telemetry` can call every function in it +with arbitrary argument values; this repository enforces no authorization on top of +that. + +## This library never transmits or persists the data it serializes + +`ProtoSpanExporter._serialize_traces_data()` +(`src/snowflake/telemetry/_internal/exporter/otlp/proto/traces/__init__.py`) and +`ProtoMetricExporter._serialize_metrics_data()` +(`src/snowflake/telemetry/_internal/exporter/otlp/proto/metrics/__init__.py`) produce +`bytes` and immediately hand them to `self.span_writer.write_span(...)` / +`self.metric_writer.write_metrics(...)`. `SpanWriter.write_span` and +`MetricWriter.write_metrics` are declared with `@abc.abstractmethod` and have no +body in either class — the embedding application must subclass `SpanWriter` or +`MetricWriter` and implement the method itself before any serialized data goes +anywhere. This repository ships no such subclass (the closest thing, an +`InMemorySpanWriter`/`InMemoryMetricWriter`, exists only under `tests/`, per the +module docstrings in both files). + +Combined with the absence of any socket/HTTP/file-write call anywhere in `src/` +(see `architecture.md`), a finding that assumes this repository's code sends +telemetry data to a specific network destination, an S3 bucket, or a database is +describing the *embedding application's* writer implementation — which lives +outside this repository — not code that exists here. + +## The one caller-input guard this repo does enforce: reserved log attributes + +`SnowflakeLogFormatter.format()` (`src/snowflake/telemetry/logs/__init__.py`) builds +its output `attributes` dict in two steps: it first sets `code.lineno`, +`code.function`, `code.filepath` from the real `LogRecord` fields, and — when an +exception is attached — `exception.type`, `exception.message`, +`exception.stacktrace`. It then iterates `record.__dict__.items()` (which includes +whatever a caller passed via `logging`'s `extra=` argument) and skips any key that +appears in `_RESERVED_ATTRS`, a frozenset that includes exactly those six +`code.*`/`exception.*` keys plus the standard `logging.LogRecord` attribute names +(`src/snowflake/telemetry/logs/__init__.py:11-42`). The effect: a caller cannot use +`extra={"code.lineno": ...}` (or any of the other reserved keys) to overwrite the +values this formatter derives from the actual call site. This is exercised by +`tests/test_snowflake_log_formatter.py#test_normal_log`, which passes +`extra={"body": 123, "code.lineno": 35}` and asserts the emitted `code.lineno` is +the real line number (21 in that test), not the caller-supplied 35, while the +non-reserved `body` key is passed through unchanged. + +No other caller-supplied value (event names, span/log attribute values, metric +data points) is validated, size-capped, or filtered by this repository before being +handed to the OpenTelemetry SDK or serialized to protobuf bytes. + +## `SnowflakeTraceIdGenerator`'s trace-ID generation + +`SnowflakeTraceIdGenerator.generate_trace_id()` +(`src/snowflake/telemetry/trace/__init__.py`) builds a 16-byte trace ID from a +4-byte big-endian encoding of "minutes since the epoch" followed by 12 bytes from +`random.getrandbits(96)` — the stdlib's non-cryptographic PRNG, not the `secrets` +module. + +## Two independent build/publish pipelines exist for this package + +- `.github/workflows/python-publish.yml` runs on `release: types: [published]` + (`:11-13`) and publishes to the public PyPI registry using + `pypa/gh-action-pypi-publish@release/v1` (`:36`) authenticated with the + `PYPI_API_TOKEN` repository secret (`:39`) — a stored token, not PyPI's OIDC + "trusted publishing." +- `build.sh` and `pypi-build.sh` are, per their own header comments, "called from a + Jenkins job called SnowflakeTelemetryPythonPackageBuilder" (`build.sh:4`, + `pypi-build.sh:4`) rather than from any GitHub Actions workflow in this repo. + `build.sh` additionally adds a private conda channel, + `https://repo.anaconda.com/pkgs/snowflake/` (`build.sh:38`), before building the + conda-format package and removes it afterward (`build.sh:56`). Neither the + Jenkins job's configuration nor what consumes packages from that channel is + visible from this repository.