Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,9 @@ line.

The markdown is committed to your repo. The agent can grep and read it
without warehouse credentials. Credentials are only needed when a query
runs.
runs. They can come from environment variables or a lazy
[`credential_process`](docs/wlens-yml-reference.md#credential-process), which
keeps them out of the agent's ambient environment.

### Why markdown

Expand Down
49 changes: 49 additions & 0 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,61 @@ executor:
# database: ${WLENS_DB_NAME}
# user: ${WLENS_DB_USER}
# password: ${WLENS_DB_PASSWORD}
# Or load those fields lazily from JSON instead of exporting them:
# credential_process: [./scripts/wlens-credentials]
```

Env-var substitution (`${SOME_VAR}`) happens at load time, so secrets
stay out of the committed file. Set them however you prefer (`.env`,
direnv, your platform's secret manager).

To avoid putting warehouse credentials in the caller's environment, configure
a process that returns the connection fields as JSON instead:

```yaml
executor:
kind: redshift
credential_process: [./scripts/wlens-credentials]
```

The command must write exactly one JSON object to stdout and nothing else:

```json
{
"host": "warehouse.example.com",
"port": 5439,
"database": "analytics",
"user": "reader",
"password": "secret"
}
```

It may return any subset of `host`, `port`, `database`, `user`, `password`, and
`path`. Returned values override the corresponding fields written directly in
`wlens.yml`, so a common setup keeps `host`, `port`, and `database` static while
the process returns only `user` and `password`.

The process can be any executable that produces that JSON. For example, if an
AWS Secrets Manager secret named `analytics/wlens` contains the object above:

```yaml
credential_process:
- aws
- secretsmanager
- get-secret-value
- --secret-id
- analytics/wlens
- --query
- SecretString
- --output
- text
```

The command runs directly without a shell, relative to the directory containing
`wlens.yml`, and only when wlens needs an executor. See the
[`wlens.yml` reference](wlens-yml-reference.md#credential-process) for complete
validation and security behavior.

## 4. Compile dbt

wlens reads `<project_dir>/target/manifest.json`, so make sure it's fresh.
Expand Down
42 changes: 42 additions & 0 deletions docs/wlens-yml-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,54 @@ queries entirely — you'll still get schema docs.
| `database` | — | Server engines only. |
| `user` | — | Server engines only. |
| `password` | — | Server engines only. |
| `credential_process` | — | Command that returns connection fields as JSON. Accepts an executable string or an argument list. |

**Environment-variable expansion.** Any string value of the form
`${SOME_NAME}` is expanded from the process environment when `wlens.yml`
is loaded. Missing variables become empty strings. This keeps secrets out
of the committed file.

### Credential process

Use `credential_process` when credentials should be loaded only for warehouse
operations instead of exported into the shell running wlens:

```yaml
executor:
kind: redshift
credential_process:
- ./scripts/wlens-credentials
```

The command must write one JSON object to stdout. It may return any subset of
`host`, `port`, `database`, `user`, `password`, and `path`; returned values
override the corresponding static executor fields.

```json
{
"host": "warehouse.example.com",
"port": 5439,
"database": "analytics",
"user": "reader",
"password": "..."
}
```

The process runs only when wlens builds an executor for a query, sample-row
fetch, or MCP server. Relative executable paths and arguments resolve from the
directory containing `wlens.yml`. Wlens captures the output in memory and does
not add the returned values to its environment or the caller's environment.

Commands run directly, without a shell. Use a YAML list when passing arguments:

```yaml
credential_process: [./scripts/wlens-credentials, --profile, analytics]
```

Do not put secrets directly in command arguments because other local processes
may be able to inspect process arguments. Stdout must contain only the JSON
object. For safety, wlens does not reproduce stdout or stderr in errors.

### DuckDB

```yaml
Expand Down
20 changes: 20 additions & 0 deletions src/wlens/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
database: ${WLENS_DB_NAME}
user: ${WLENS_DB_USER}
password: ${WLENS_DB_PASSWORD}
# Or load connection fields from a subprocess when the executor is built:
# credential_process: [./scripts/wlens-credentials]

output:
dir: .claude/schema
Expand Down Expand Up @@ -69,6 +71,7 @@ class ExecutorConfig:
database: str | None = None
user: str | None = None
password: str | None = None
credential_process: list[str] = field(default_factory=list)
# File path (or `:memory:`) for file-based engines like DuckDB.
path: str | None = None

Expand Down Expand Up @@ -162,6 +165,7 @@ def _build_config(raw: dict, repo_root: Path) -> Config:
database=_str_or_none(executor_raw.get("database")),
user=_str_or_none(executor_raw.get("user")),
password=_str_or_none(executor_raw.get("password")),
credential_process=_parse_credential_process(executor_raw.get("credential_process")),
path=_str_or_none(executor_raw.get("path")),
)
output = OutputConfig(
Expand Down Expand Up @@ -199,6 +203,22 @@ def _build_config(raw: dict, repo_root: Path) -> Config:
)


def _parse_credential_process(value: Any) -> list[str]:
if isinstance(value, str) and value.strip():
return [value]
if (
isinstance(value, list)
and value
and all(isinstance(argument, str) and argument for argument in value)
):
return list(value)
if value is None:
return []
raise ValueError(
"`executor.credential_process` must be a command string or a non-empty list of strings."
)


def _str_or_none(value: Any) -> str | None:
if value is None:
return None
Expand Down
6 changes: 5 additions & 1 deletion src/wlens/executor/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,13 @@
__all__ = ["Executor", "ReadOnlyViolation", "build_executor", "format_markdown_table"]


def build_executor(config: "Config") -> Executor:
def build_executor(config: Config) -> Executor:
"""Instantiate the executor indicated by `wlens.yml`."""
from .credentials import resolve_credentials

kind = (config.executor.kind or "").lower()
if kind in {"redshift", "postgres", "postgresql", "duckdb"}:
config = resolve_credentials(config)
if kind in ("redshift",):
from .redshift import RedshiftExecutor
return RedshiftExecutor(config)
Expand Down
85 changes: 85 additions & 0 deletions src/wlens/executor/credentials.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
"""Resolve executor connection fields from an external credential process."""

from __future__ import annotations

import json
import subprocess
from dataclasses import replace
from typing import Any

from ..config import Config

CREDENTIAL_PROCESS_TIMEOUT_SECONDS = 30
CONNECTION_FIELDS = frozenset({"host", "port", "database", "user", "password", "path"})


def resolve_credentials(config: Config) -> Config:
"""Return a config with credential-process output merged into its executor.

The subprocess runs relative to the directory containing ``wlens.yml``.
Its output is kept in memory and never copied into ``os.environ``. The
original config is not mutated.
"""
command = config.executor.credential_process
if not command:
return config

try:
completed = subprocess.run(
command,
cwd=config.repo_root,
capture_output=True,
text=True,
timeout=CREDENTIAL_PROCESS_TIMEOUT_SECONDS,
check=False,
)
except FileNotFoundError:
raise ValueError("executor.credential_process executable was not found") from None
except subprocess.TimeoutExpired:
raise ValueError(
"executor.credential_process timed out after "
f"{CREDENTIAL_PROCESS_TIMEOUT_SECONDS} seconds"
) from None
except OSError:
raise ValueError("executor.credential_process could not be started") from None

if completed.returncode != 0:
raise ValueError(
f"executor.credential_process exited with status {completed.returncode}"
)

try:
payload = json.loads(completed.stdout)
except (json.JSONDecodeError, TypeError):
raise ValueError("executor.credential_process must return valid JSON") from None

if not isinstance(payload, dict):
raise ValueError("executor.credential_process must return a JSON object")

unsupported_fields = sorted(set(payload) - CONNECTION_FIELDS)
if unsupported_fields:
raise ValueError(
"executor.credential_process returned unsupported fields: "
+ ", ".join(unsupported_fields)
)

updates = {name: _normalize_field(name, value) for name, value in payload.items()}
executor = replace(config.executor, credential_process=[], **updates)
return replace(config, executor=executor)


def _normalize_field(name: str, value: Any) -> str | int | None:
if name == "port":
if value in (None, ""):
return None
try:
return int(value)
except (TypeError, ValueError):
raise ValueError(
"executor.credential_process returned an invalid value for port"
) from None

if value is None:
return None
string_value = str(value)
return string_value if string_value else None
2 changes: 2 additions & 0 deletions src/wlens/templates/wlens.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ executor:
# database: ${WLENS_DB_NAME}
# user: ${WLENS_DB_USER}
# password: ${WLENS_DB_PASSWORD}
# Or load those fields lazily from JSON instead of exporting them:
# credential_process: [./scripts/wlens-credentials]

# Value-level scrubbing of sample-row cells.
# Defaults always on: email, IPv4, phone → <email> / <ip> / <phone>.
Expand Down
Loading
Loading