TaskCE studies how fine-tuning changes the internal representations of language models. It compares a base model with task-specific LoRA variants by capturing neuron activations, fitting Boolean compositional formulas, and measuring which internal features stay stable, change behavior, or recur across layers.
Eight independent top-level folders own the analytical methods, and taskce/ owns the shared CLI executor, study profiles, and data-root resolution:
capture/records final-token activation matrices for base and PEFT LoRA checkpoints.activation_analysis/measures threshold coverage, neuron fate, and PCA task separation over saved captures.formula_search/builds binary lexical features and fits Boolean formulas with GPU beam search, including the pinnedvectorized_search2.a Triton backend.formula_comparison/computes tree-edit formula diffs, heatmaps, and standalone HTML AST viewers from formula CSVs.joint_search/compares symbolic conjunctions of parent neuron formulas against formulas searched directly on measured joint firing.cross_layer_recurrence/indexes recurring compound subformulas across layers, extracts nested multi-stage chains, and matches compounds to earlier single neurons.graph_analysis/builds Pearson correlation and cosine similarity neuron graphs, detects communities, and attaches compositional formulas.ablation/ranks explained neurons and measures task accuracy deltas under zero-ablation forward hooks.
Standalone task evaluation lives outside TaskCE. Formula-fit IoU remains inside formula_search/ and joint_search/, and ablation/ retains the single-token logit hook inference required to measure interventions.
flowchart LR
GH["GitHub · pinned code revision"] -->|Fetch at startup| CODE
HF["Hugging Face · pinned model revision"] -->|Fetch when needed| CACHE
subgraph POD["Analysis pod"]
CODE["Local checkout"] --> EXEC["uv executor"]
EXEC --> RUN["Selected analysis"]
CACHE["Local model cache"] --> RUN
RUN <--> SCRATCH["Local scratch"]
end
PVC[("PVC · TASKCE_DATA_DIR")] -->|Read inputs or resume| RUN
RUN -->|Save checkpoints and results| PVC
Code, model caches, and durable research data stay on separate paths:
- Code and Python environment. Keep the git checkout and
.venvin the container image or on pod-local storage so Python imports avoid network-volume latency. - Durable data (
TASKCE_DATA_DIR). Every analysis resolves captures, prepared.npzbundles, trained tokenizers, checkpoints, and per-run outputs underTASKCE_DATA_DIR. WhenTASKCE_DATA_DIRis unset locally, it defaults to./data. On a pod, pointTASKCE_DATA_DIRat the mounted PVC directory. - Frozen baselines (
TASKCE_BASELINE_DIR). Read-only reference archives default to<TASKCE_DATA_DIR>/../baselines/57a1e74when present or can be set explicitly throughTASKCE_BASELINE_DIRor--baseline-dir. - On-demand model cache (
HF_HOME). Stages that do not load weights (joint-search,cross-layer-recurrence,formula-comparison,graph-analysis,activation-analysis, andablation --stage select) never importtransformers,peft, orhuggingface_huband never trigger model downloads. When a stage requires a model or dataset,taskce/hf.pykeepsHF_HOMEon pod-local storage outsideTASKCE_DATA_DIR.
Full dependency installation requires Linux x86_64 with an NVIDIA GPU and CUDA 12.8 driver because pyproject.toml pins a Linux FlashAttention wheel and NVIDIA cuGraph.
Install mise or uv, then sync the environment on pod-local disk:
uv sync --python 3.12Verify the environment, pinned SHA-256 hashes, and mounted data paths without downloading models:
export TASKCE_DATA_DIR="/~ /TaskCE/data"
uv run taskce list
uv run taskce doctor --profile snliEvery analysis runs through uv run taskce <analysis> --profile <profile>. Each fresh run writes into an isolated <TASKCE_DATA_DIR>/runs/<analysis>/<profile>/<run_id>/ directory with atomic checkpoints and per-stage wall-clock timing in run_meta.json.
On a GPU pod with the ggong1-dev-main PVC mounted at /~ /:
export TASKCE_DATA_DIR="/~ /TaskCE/data"
uv run taskce joint-search --profile snli --check-only
uv run taskce joint-search --profile snli --verify-baselineThe validate stage checks the SHA-256 digests of features.npz, layer.8.npz, layer.8.csv, layer.8.csv.manifest.json, formula_eval.py, and the three vectorized_search modules before running any GPU kernel. The sweep stage executes all 24 (pair, length) search cells, writes sweep.csv and results.json, and --verify-baseline compares baselines.json, masks.npz, sweep.csv, and all 24 cell files against TaskCE/baselines/57a1e74 in reproduction_report.json.
To resume an interrupted run directory after preemption, pass --run-id <run_id> --resume.
export TASKCE_DATA_DIR="/~ /TaskCE/data"
uv run taskce cross-layer-recurrence --profile snliBy default this runs the CPU-only index, chains, and match stages over saved formula CSVs and prepared .npz features. Pass --stage full_sweep on a GPU pod to refit all configured layers first.
export TASKCE_DATA_DIR="/~ /TaskCE/data"
export HF_HOME="/tmp/hf-cache"
uv run taskce capture --profile snli --batch-size 32
uv run taskce activation-analysis --profile snli --layer model.layers.8.feed_forward
uv run taskce formula-search --profile snli --layer layer.8
uv run taskce formula-comparison --profile snli --stage ast_viewer --csv path/to/layer.8.csv
uv run taskce graph-analysis --profile snli --formula-path path/to/snli_beam_results.csv
uv run taskce ablation --profile snli --formula-path path/to/snli_beam_results.csvBuilt-in TOML profiles live in taskce/profiles/ (snli, claim, fallacy). To add a new study such as bowman, place bowman.toml in $TASKCE_DATA_DIR/profiles/bowman.toml or pass --profile ./bowman.toml. Profiles support single-level inheritance with extends:
name = "bowman"
description = "Bowman CoEx single-neuron and joint-search study"
extends = "snli"
[model]
lora_task = "bowman"
[dataset]
name = "bowman"
hf_id = "stanfordnlp/snli"
split = "validation"
tokenizer_name = "spacy-pos-bowman-features"
[artifacts]
prepared_inputs_relpaths = ["bowman/inputs"]Newly trained feature tokenizers save to $TASKCE_DATA_DIR/tokenizers/ so the repository checkout stays clean.
The frozen reference dataset from commit 57a1e74 lives on the Nautilus PVC ggong1-dev-main under TaskCE/baselines/57a1e74/ and is indexed by taskce/resources/baseline_57a1e74.json. See formula_comparison/formula_ast/README.md to build and verify offline HTML AST viewers.