diff --git a/Makefile b/Makefile index 39a9f2e..b13d500 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ PY := .venv/bin/python -.PHONY: all venv lint test test-ollama measure sweep bench check-numbers run clean +.PHONY: all venv lint test test-ollama measure sweep bench loops check-numbers run clean # One gate. Lint, tests and the documented numbers pass together or the build is # not green -- keeping the numbers in a separate optional target is how a README @@ -44,3 +44,7 @@ run: clean: rm -rf .pytest_cache .ruff_cache __pycache__ tests/__pycache__ embeddings.db embeddings.ann + +# Is repetition actually the problem, or does the model drift instead? +loops: + $(PY) scripts/measure_loops.py diff --git a/README.md b/README.md index 0b1e14c..0de0792 100644 --- a/README.md +++ b/README.md @@ -10,12 +10,12 @@ pypi.org, where a relative path resolves against pypi.org and 404s. --> ![Example](https://raw.githubusercontent.com/punnerud/Local_Knowledge_Graph/main/docs/example.png) -Ask a local Llama model a question, watch it reason step by step, and see the steps drawn as a -knowledge graph where the edges are the semantic similarity between them. +Ask a local model a question, watch it reason step by step, and see the steps drawn as a graph +where the edges are how similar the steps are to each other. Everything runs on your machine. Nothing is uploaded anywhere. -## Install +## Run it ```bash pip install mpe-lkg @@ -24,219 +24,43 @@ mpe-lkg Then open . -The models are chosen for you from whatever Ollama has installed, and the page has a dropdown -for each so you can change them. If Ollama has no chat model at all, the page lists a few with -their download sizes and can fetch one. +It needs a local model, which it reaches through [Ollama](https://ollama.com). **You do not +need to work that out from here** — start it and it will tell you what it found, what is +missing, and the one command that fixes it. `mpe-lkg doctor` reports the same thing without +starting the server, and exits non-zero, so it works in a script. -`mpe-lkg doctor` reports the same thing from the terminal, exiting non-zero when something is -wrong so it works in a script. +Python 3.10 or newer. The wheel is `py3-none-any`, so nothing is compiled and the same +artefact serves Linux, macOS and Windows — all three tested on every push.
-Install from source instead +From a clone, or from Python ```bash git clone https://github.com/punnerud/Local_Knowledge_Graph cd Local_Knowledge_Graph -python3 -m venv .venv -.venv/bin/pip install -e . +python3 -m venv .venv && .venv/bin/pip install -e . .venv/bin/mpe-lkg ``` -`python app.py` still works from a clone as it always has — it is a shim over the same code. - -
- -
-Use it from Python +`python app.py` still works from a clone as it always has. ```python from mpe_lkg import create_app, health -print(health()) # {'ok': True, 'models': [...], ...} +print(health()) create_app().run(port=5100) ```
-### You also need Ollama - -[Ollama](https://ollama.com) running locally, with one chat model and, ideally, one embedding -model: - -```bash -ollama pull llama3.2:3b # or llama3.1:8b, or any chat model you already have -ollama pull nomic-embed-text # optional but recommended, see "Which embedding model" below -``` - -Python 3.10 or newer. The wheel is `py3-none-any`, so there is nothing to compile and the same -artefact serves Linux, macOS and Windows — all three are tested on every push. - -## Configuration - -Everything is an environment variable, and the defaults work unchanged. - -| Variable | Default | Meaning | -|---|---|---| -| `OLLAMA_URL` | `http://localhost:11434` | Where Ollama is listening | -| `LKG_CHAT_MODEL` | *(auto)* | The model that does the reasoning. Empty means: use an installed chat model. This is an override, not a default | -| `LKG_EMBED_MODEL` | *(auto)* | Embedding model. Empty means: use an installed embedding model if there is one, otherwise fall back to the chat model | -| `LKG_HOST` / `LKG_PORT` | `127.0.0.1` / `5100` | Where the app listens | -| `LKG_DEBUG` | off | Set to `1` for the Flask debugger. Do not do this on a shared network | - -## Which embedding model, and why it matters - -The edges in the graph are cosine similarities, so how much they vary decides whether the -picture tells you anything. Measured over four unrelated six-step reasoning chains -(`make measure`, recorded in `docs/claims/edge_spread.json`): - -| Model | Dimensions | Mean edge weight | Coefficient of variation | -|---|---|---|---| -| `all-minilm` | 384 | 0.48 ± 0.07 | **0.28 ± 0.11** | -| `nomic-embed-text` | 768 | 0.67 ± 0.05 | **0.13 ± 0.03** | - -The uncertainties are the spread across the four topics. The larger model produces the -*less* discriminative graph here: under `nomic-embed-text` almost every pair of reasoning -steps scores around 0.67, so the edge labels stop distinguishing anything. This is ordinary -distance concentration, and it is a good reason to look at the spread rather than trusting -that a better retrieval model draws a better graph. `all-minilm` is the better default for -the *drawing* even though it is the weaker retriever. - -Both work. Any embedding size works — nothing in the code assumes a dimension. - -## Embeddings from inside a model - -An embedding endpoint gives you one pooled vector from the top of the stack. You can instead -tap a chosen point *inside* a local model — which also makes models with no embedding API -usable, since a forward pass is all that is required: - -```bash -pip install torch transformers - -LKG_EMBED_BACKEND=hf \ -LKG_HF_MODEL=HuggingFaceTB/SmolLM2-135M \ -LKG_HF_LAYER=blocks.-1 \ -mpe-lkg -``` - -Layers are addressed structurally, not by a per-architecture path: `blocks.0`, `blocks.12`, -`blocks.-1`, `blocks.-1.mlp`, or any explicit dotted module path. The block stack is found by -looking for the longest `nn.ModuleList` whose children share one class, which covers Llama, -Qwen, Mistral, Gemma, Phi, GPT-2, GPT-NeoX, Falcon, BERT, ViT and CLIP without a lookup table. -`LKG_HF_POOLING` selects `last` (default, and the only architecturally correct choice for a -decoder under a causal mask), `mean`, or `cls`. - -### Does the depth matter? - -`make sweep` runs the same four-topic corpus through several layers and reports how far each -one puts steps of the same topic from steps of a different topic. On `SmolLM2-135M`: - -| Layer | Within topic | Across topics | Separation | -|---|---|---|---| -| `blocks.0` | 0.998 | 0.996 | **0.002** | -| `blocks.7` | 0.895 | 0.834 | 0.062 | -| `blocks.15` | 0.914 | 0.863 | 0.051 | -| `blocks.22` | 0.893 | 0.780 | 0.113 | -| `blocks.29` | 0.926 | 0.779 | **0.148** | - -The first block cannot tell the topics apart at all — it sees each token before any context -has been mixed in — and that near-zero is the control that says the separation deeper in is -real rather than an artefact of the metric. Separation grows roughly seventyfold with depth. - -Two details that quietly ruin a layer comparison if you skip them, and which this handles: -intermediate blocks emit the raw residual stream while the model's own last hidden state has -already been through the final norm, so that norm is applied to every layer to put them in one -space; and the states are captured with forward hooks that pool inside the hook rather than -with `output_hidden_states=True`, which would materialise every layer at once — several -gigabytes on an 8B model before any pooling happens. - -## Development - -```bash -make venv # uv-based environment, including a headless browser for the render tests -make all # ruff + pytest + the documented numbers, in one gate -make test # unit, stream and browser tests; no model needed -make test-ollama # the tests that need a live Ollama -make measure # re-measure the embedding-model table into docs/claims/ -make sweep # re-measure the per-layer separation table (needs torch) -make bench # exact scan vs an approximate index, at several store sizes -``` - -`make all` runs `scripts/check_numbers.py`, which resolves every measurable claim in this -README to a value in `docs/claims/`. If a number here stops being true, the build fails -instead of the README quietly becoming wrong. Some of its checks are ground truths computed -from arithmetic rather than from a previous run, because a consistency gate cannot detect a -consistent error. - -## Troubleshooting - -**The page stays blank when I submit.** -Open . It reports whether Ollama answered, which models are -installed, and what to pull. Errors are now shown in the page itself rather than only in the -browser console. - -**It says a model is not found.** -It should not: the app picks whichever chat and embedding models Ollama actually reports, and -the page has a dropdown for each. If Ollama has no chat model at all, the page lists a few with -their download sizes and can fetch one for you. - -**Ollama runs in Docker or on another machine.** -Set `OLLAMA_URL`, and make sure Ollama binds beyond localhost (`OLLAMA_HOST=0.0.0.0`). - -**It seemed to hang and never printed anything.** -That was a real bug: two retry paths could loop forever without ever sending anything to the -browser. Both are bounded now, and the stream sends a heartbeat while the model is thinking. - -## How it works +## More -| File | Responsibility | +| | | |---|---| -| `src/mpe_lkg/app.py` | Flask routes and server-sent-event framing | -| `src/mpe_lkg/backends.py` | Chat and embedding backends, model discovery, health checks | -| `src/mpe_lkg/reasoning.py` | The step-by-step loop | -| `src/mpe_lkg/graph.py` | Similarity, graph construction, strongest path | -| `src/mpe_lkg/store.py` | SQLite storage and exact nearest-neighbour search | -| `src/mpe_lkg/layers.py` | Embeddings read from inside a model | - -The strongest path maximises the product of the similarities along it, which is the same as -minimising a sum of `-log(similarity)`. Those costs are non-negative, so Dijkstra gives the -exactly optimal path, and the number reported is the geometric mean of the edges on it. - -## Why the similarity search has no approximate index - -The store keeps growing — it is no longer wiped between questions, so "Related Questions and -Answers" can actually surface earlier ones — which makes it fair to ask whether it needs an -ANN index. Measured with `scripts/bench_search.py` at 768 dimensions: - -| Vectors | Exact scan (numpy) | Annoy query | Annoy build, per insert | -|---|---|---|---| -| 100 | 0.007 ms | 0.031 ms | 3.5 ms | -| 1 000 | 0.017 ms | 0.032 ms | 36 ms | -| 10 000 | 0.30 ms | 0.031 ms | 366 ms | -| 100 000 | 3.3 ms | 0.032 ms | 4 020 ms | - -Three things follow. - -**The exact scan is already fast enough at any plausible size.** Hundreds of vectors cost -about 0.02 ms, against an LLM call that takes seconds. Even a hundred thousand costs 3 ms. - -**An Annoy index cannot be appended to.** It is immutable once built, and this app inserts -after every reasoning step, so the whole index has to be rebuilt on each one. That is the -last column, and it is worse than the exact scan at every size measured. - -**On a current numpy the index returns wrong answers.** With `annoy` 1.17.3 and numpy 2.5.2 on -Python 3.12, `get_nns_by_item(7, 5)` returns `[1]` — one result instead of five, and not the -vector itself, which must always be its own nearest neighbour at distance zero. That is -reproducible in a clean environment built from the old `requirements.txt`, which means the -"Related Questions" panel was silently returning a single arbitrary row. - -The last point is pinned as a check that fails if a future build ever starts behaving, so the -decision can be revisited rather than inherited. `tests/test_store.py` asserts exactness -directly: a vector is its own nearest neighbour, and the ranking matches a full brute-force -sort. - -If the store ever does grow past a few hundred thousand vectors, the argument that changes -first is memory, not speed — 100 000 × 768 × 4 bytes is about 300 MB held in RAM — and the -answer then is a memory-mapped index, not a faster query. +| [Models and configuration](docs/models.md) | Choosing models, every environment variable, troubleshooting | +| [Embeddings from inside a model](docs/internal-layers.md) | Reading a chosen layer instead of an embedding endpoint | +| [How it works](docs/design.md) | The modules, the strongest-path search, why there is no ANN index | +| [Development](docs/development.md) | Tests, and the gate that checks this documentation against measured data | ## Licence diff --git a/docs/claims/loops.json b/docs/claims/loops.json new file mode 100644 index 0000000..f61dd12 --- /dev/null +++ b/docs/claims/loops.json @@ -0,0 +1,94 @@ +{ + "provenance": "measured", + "measured_at": "2026-08-10T13:50:56+00:00", + "chat_model": "(from saved transcripts)", + "embedding_model": "nomic-embed-text:latest", + "runs": 24, + "steps": 153, + "harness_audit": { + "raw_duplicates": 2, + "normalised_duplicates": 2, + "manufactured_by_normalisation": 0, + "empty_after_normalisation": 0, + "shortest_normalised_length": 24, + "trustworthy": true + }, + "repeat_steps": 7, + "repeat_rate": 0.0458, + "repeat_by_detector": { + "restatement": 6, + "move": 3, + "exact": 2 + }, + "drift_steps": 6, + "drift_rate": 0.0392, + "clean_rate": 0.915, + "per_group": { + "simple": { + "steps": 37, + "repeat": 2, + "drift": 1, + "runs": 6 + }, + "multi_step": { + "steps": 39, + "repeat": 4, + "drift": 5, + "runs": 6 + }, + "malformed": { + "steps": 39, + "repeat": 1, + "drift": 0, + "runs": 6 + }, + "ambiguous": { + "steps": 38, + "repeat": 0, + "drift": 0, + "runs": 6 + } + }, + "mutual_knn": [ + { + "k": 2, + "n": 153, + "components": 66, + "largest": 7, + "largest_share": 0.0458, + "singletons": 31 + }, + { + "k": 3, + "n": 153, + "components": 48, + "largest": 7, + "largest_share": 0.0458, + "singletons": 21 + }, + { + "k": 4, + "n": 153, + "components": 35, + "largest": 8, + "largest_share": 0.0523, + "singletons": 11 + }, + { + "k": 6, + "n": 153, + "components": 13, + "largest": 40, + "largest_share": 0.2614, + "singletons": 0 + } + ], + "mean_steps_per_run": 6.38, + "runs_with_a_repeat": 6, + "runs_with_a_repeat_rate": 0.25, + "thresholds": { + "restatement": 0.96, + "repeated_move": 0.95 + }, + "errors": 0 +} diff --git a/docs/claims/loops_transcripts.json b/docs/claims/loops_transcripts.json new file mode 100644 index 0000000..be340de --- /dev/null +++ b/docs/claims/loops_transcripts.json @@ -0,0 +1,959 @@ +[ + { + "question": "What is the capital of France?", + "steps": [ + { + "title": "Initial Answer", + "content": "The capital of France is Paris.", + "question": "What is the capital of France?" + }, + { + "title": "Re-examine Initial Answer", + "content": "Paris is widely known as the capital of France, but I should verify this information to ensure accuracy.", + "question": "What is the capital of France?" + }, + { + "title": "Verify Information", + "content": "According to various reliable sources, including Wikipedia and official government websites, Paris is indeed the capital of France.", + "question": "What is the capital of France?" + }, + { + "title": "Consider Alternative Answers", + "content": "However, I should also consider other possibilities, such as Lyon or Bordeaux, which are also major cities in France. But based on my research, Paris is the most widely recognized and accepted capital.", + "question": "What is the capital of France?" + }, + { + "title": "Final Answer", + "content": "I should also check if there are any exceptions or special cases where a different city might be considered the capital, such as for a specific region or territory. After verifying, I found no such exceptions.", + "question": "What is the capital of France?" + } + ], + "error": null, + "seconds": 5.37, + "group": "simple" + }, + { + "question": "How many days are in a leap year?", + "steps": [ + { + "title": "Initial Assessment", + "content": "A leap year is defined as a year that is divisible by 4, except for years that are also divisible by 100 but not by 400.", + "question": "How many days are in a leap year?" + }, + { + "title": "Exploring Alternatives", + "content": "Consideration of alternative definitions or exceptions to the standard rule.", + "question": "How many days are in a leap year?" + }, + { + "title": "Verifying Leap Year Rule", + "content": "Re-examining the definition: years divisible by 4, except for those divisible by 100 but not 400.", + "question": "How many days are in a leap year?" + }, + { + "title": "Testing Edge Cases", + "content": "Checking if years like 2000 are indeed leap years (divisible by 400).", + "question": "How many days are in a leap year?" + }, + { + "title": "Re-examining Leap Year Rule", + "content": "Further consideration: years divisible by 100 but not 400 are exceptions to the rule. However, some sources may consider only years divisible by 400 as exceptions.", + "question": "How many days are in a leap year?" + }, + { + "title": "Alternative Definitions", + "content": "Exploring alternative definitions of a leap year, such as those used in Islamic or Hebrew calendars.", + "question": "How many days are in a leap year?" + }, + { + "title": "Final Answer", + "content": "Considering the International Organization for Standardization (ISO) definition: years divisible by 4 are leap years, except for years divisible by 100 but not 400.", + "question": "How many days are in a leap year?" + } + ], + "error": null, + "seconds": 7.59, + "group": "simple" + }, + { + "question": "What is the chemical symbol for gold?", + "steps": [ + { + "title": "Initial Research", + "content": "The chemical symbol for gold is Au.", + "question": "What is the chemical symbol for gold?" + }, + { + "title": "Verification", + "content": "Au comes from the Latin word 'Aurum', which means gold.", + "question": "What is the chemical symbol for gold?" + }, + { + "title": "Alternative Symbol", + "content": "However, some sources may use Ag instead of Au to represent gold, but this is less common.", + "question": "What is the chemical symbol for gold?" + }, + { + "title": "Final Answer Confirmation", + "content": "Based on the verification and exploration of alternative symbols, Au is the most widely accepted chemical symbol for gold.", + "question": "What is the chemical symbol for gold?" + }, + { + "title": "Re-examination of Alternative Symbol", + "content": "Upon further review, I found that Ag is actually the symbol for silver, not gold.", + "question": "What is the chemical symbol for gold?" + }, + { + "title": "Verification of Au Origin", + "content": "Au indeed comes from the Latin word 'Aurum', which means gold, and has been widely accepted as the standard symbol for gold since ancient times.", + "question": "What is the chemical symbol for gold?" + }, + { + "title": "Final Answer", + "content": "The International Union of Pure and Applied Chemistry (IUPAC) also recognizes Au as the official symbol for gold.", + "question": "What is the chemical symbol for gold?" + } + ], + "error": null, + "seconds": 6.78, + "group": "simple" + }, + { + "question": "Who wrote the play Hamlet?", + "steps": [ + { + "title": "Initial Research", + "content": "The authorship of Hamlet is widely attributed to William Shakespeare.", + "question": "Who wrote the play Hamlet?" + }, + { + "title": "Verification", + "content": "Historical records and scholarly consensus confirm that Shakespeare was the playwright behind Hamlet.", + "question": "Who wrote the play Hamlet?" + }, + { + "title": "Alternative Perspectives", + "content": "Some scholars argue that Edward de Vere, Earl of Oxford, may have written Hamlet instead. However, this theory is not widely accepted due to lack of concrete evidence.", + "question": "Who wrote the play Hamlet?" + }, + { + "title": "Re-examination of Oxford Theory", + "content": "Further research on de Vere's life and works reveals no clear connections to the play's themes or style, making it unlikely that he was the author.", + "question": "Who wrote the play Hamlet?" + }, + { + "title": "Final Answer", + "content": "The overwhelming consensus among scholars and literary experts confirms William Shakespeare as the writer of Hamlet.", + "question": "Who wrote the play Hamlet?" + } + ], + "error": null, + "seconds": 4.93, + "group": "simple" + }, + { + "question": "What is the largest ocean on Earth?", + "steps": [ + { + "title": "Initial Research", + "content": "The five oceans are: Pacific, Atlantic, Indian, Arctic, and Southern.", + "question": "What is the largest ocean on Earth?" + }, + { + "title": "Gather Information", + "content": "According to various sources, including the National Oceanic and Atmospheric Administration (NOAA), the Pacific Ocean is often considered the largest of the five oceans.", + "question": "What is the largest ocean on Earth?" + }, + { + "title": "Verify Information", + "content": "A quick check reveals that the Pacific Ocean covers approximately 155.6 million square kilometers, which is roughly 46% of the Earth's total ocean area.", + "question": "What is the largest ocean on Earth?" + }, + { + "title": "Re-Examine Initial Research", + "content": "Upon re-examination, I realize that the initial research only mentioned the five oceans without providing any specific information about their sizes.", + "question": "What is the largest ocean on Earth?" + }, + { + "title": "Alternative Ocean Size Measurements", + "content": "I found that some sources use the total ocean area (approximately 361 million square kilometers) to measure size, while others consider the Pacific Ocean's surface area (over 155.6 million square kilometers). This raises questions about which measurement is more accurate.", + "question": "What is the largest ocean on Earth?" + }, + { + "title": "Consider Alternative Definitions", + "content": "Another possibility is that the question asks for the largest ocean by volume, not surface area. In this case, the Southern Ocean might be considered the largest due to its unique shape and surrounding landmasses.", + "question": "What is the largest ocean on Earth?" + }, + { + "title": "Final Answer", + "content": "A quick check reveals that the Southern Ocean's volume is approximately 20% of the total ocean volume, which could make it the largest if considering volume instead of surface area.", + "question": "What is the largest ocean on Earth?" + } + ], + "error": null, + "seconds": 8.75, + "group": "simple" + }, + { + "question": "In what year did the Berlin Wall fall?", + "steps": [ + { + "title": "Initial Research", + "content": "The Berlin Wall fell on November 9, 1989.", + "question": "In what year did the Berlin Wall fall?" + }, + { + "title": "Verification of Date", + "content": "This date is widely recognized and documented in historical records. It marks the beginning of the end of the division of Germany.", + "question": "In what year did the Berlin Wall fall?" + }, + { + "title": "Alternative Answer Verification", + "content": "I have checked for alternative dates, but November 9, 1989, is the most commonly cited year for the fall of the Berlin Wall.", + "question": "In what year did the Berlin Wall fall?" + }, + { + "title": "Alternative Answer Verification", + "content": "I have checked for alternative dates, but November 9, 1989, is the most commonly cited year for the fall of the Berlin Wall. However, some sources suggest that the wall's opening began earlier, on November 9, 1989, in East Berlin, and the actual dismantling process took several weeks to complete.", + "question": "In what year did the Berlin Wall fall?" + }, + { + "title": "Re-examination of Event Timeline", + "content": "Upon further review, I realize that the fall of the Berlin Wall was a complex event with multiple stages. The initial opening of the border on November 9, 1989, marked a significant turning point, but it's essential to consider the broader context and the subsequent events that led to the wall's eventual dismantling.", + "question": "In what year did the Berlin Wall fall?" + }, + { + "title": "Final Answer", + "content": "I have re-checked historical records and found that the Berlin Wall was largely dismantled by December 1990, with the official reunification of Germany taking place on October 3, 1990. November 9, 1989, remains a crucial date in this process.", + "question": "In what year did the Berlin Wall fall?" + } + ], + "error": null, + "seconds": 8.55, + "group": "simple" + }, + { + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?", + "steps": [ + { + "title": "Step 1: Analyze given information", + "content": "Train distance = 240 km, time = 3 hours. We need to find the time for a new distance of 400 km.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + }, + { + "title": "Step 2: Calculate speed", + "content": "Speed = Distance / Time. Speed = 240 km / 3 hours = 80 km/h.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + }, + { + "title": "Step 3: Use speed to find new time", + "content": "Time = Distance / Speed. Time = 400 km / 80 km/h = 5 hours.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + }, + { + "title": "Step 4: Re-examine speed calculation", + "content": "Speed is calculated as distance divided by time. However, the train's speed remains constant. We should consider if there are any other factors that could affect the time, such as acceleration or deceleration.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + }, + { + "title": "Step 5: Consider alternative methods for finding new time", + "content": "Another approach is to use the concept of proportionality. If we know the train's speed and distance traveled in one instance, we can set up a proportion to find the time for a different distance.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + }, + { + "title": "Step 6: Set up proportion", + "content": "Let x be the new time. We can set up the proportion: (240 km / 3 hours) = (400 km / x hours). Solving for x, we get x = (400 km * 3 hours) / 240 km.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + }, + { + "title": "Final Answer", + "content": "x = (400 km * 3 hours) / 240 km = 5 hours. This confirms our initial answer, but we should also consider if there are any other possible solutions or alternative methods.", + "question": "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?" + } + ], + "error": null, + "seconds": 9.71, + "group": "multi_step" + }, + { + "question": "What is 17 percent of 250?", + "steps": [ + { + "title": "Understanding the Problem", + "content": "To find 17% of 250, we need to calculate a percentage of a given number.", + "question": "What is 17 percent of 250?" + }, + { + "title": "Exploring Alternatives", + "content": "We can use the formula: (percentage / 100) * number. In this case, it would be (17 / 100) * 250.", + "question": "What is 17 percent of 250?" + }, + { + "title": "Calculating the Value", + "content": "Using the formula, we get: (0.17) * 250 = 42.5.", + "question": "What is 17 percent of 250?" + }, + { + "title": "Re-Examining the Formula", + "content": "The formula (percentage / 100) * number is correct, but we should also consider using a decimal multiplier instead of converting to a percentage. This could provide an alternative solution.", + "question": "What is 17 percent of 250?" + }, + { + "title": "Exploring Decimal Multiplier", + "content": "Using the decimal multiplier method: 0.17 * 250 = 42.5. This confirms our initial answer is correct, but we should also consider if there are any other methods to arrive at this solution.", + "question": "What is 17 percent of 250?" + }, + { + "title": "Alternative Method - Percentage Conversion", + "content": "Converting the percentage to a decimal: 17% = 0.17. Then multiplying by 250: 0.17 * 250 = 42.5. This confirms our initial answer is correct, but we should also consider if there are any other methods to arrive at this solution.", + "question": "What is 17 percent of 250?" + }, + { + "title": "Final Answer", + "content": "We should check if rounding errors could affect the result. In this case, 42.5 is a precise calculation, so no rounding errors apply.", + "question": "What is 17 percent of 250?" + } + ], + "error": null, + "seconds": 9.24, + "group": "multi_step" + }, + { + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?", + "steps": [ + { + "title": "Step 1: Define variables", + "content": "Let's denote the width as 'w' and the length as '2w', since the rectangle is twice as long as it is wide.", + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?" + }, + { + "title": "Step 2: Write an equation for the perimeter", + "content": "The formula for the perimeter of a rectangle is P = 2(l + w), where l is the length and w is the width. We know that the perimeter is 36 cm, so we can write the equation as 36 = 2(2w + w).", + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?" + }, + { + "title": "Step 3: Simplify the equation", + "content": "Simplifying the equation, we get 36 = 6w. Dividing both sides by 6 gives us w = 6.", + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?" + }, + { + "title": "Re-examine Step 2", + "content": "Upon re-examination, I realize that I made a mistake by not considering the fact that the perimeter is given as 36 cm. This means that the sum of all sides is 36 cm, so we can write the equation as 36 = 2(l + w), where l is the length and w is the width.", + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?" + }, + { + "title": "Step 4: Write an alternative equation", + "content": "Since the rectangle is twice as long as it is wide, we know that l = 2w. Substituting this into the equation from Step 3, we get 36 = 2(2w + w). However, I realize now that this was a mistake in my previous response. The correct equation should be 36 = 2(l) + 2(w), since there are two lengths and two widths.", + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?" + }, + { + "title": "Final Answer", + "content": "Simplifying the equation, we get 36 = 4w. Dividing both sides by 4 gives us w = 9.", + "question": "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?" + } + ], + "error": null, + "seconds": 10.01, + "group": "multi_step" + }, + { + "question": "Which is larger, the area of a circle of radius 3 or a square of side 5?", + "steps": [ + { + "title": "Step 1: Calculate Circle Area", + "content": "The formula for the area of a circle is A = \u03c0r^2. Substituting r = 3, we get A \u2248 3.14159 * 9 \u2248 28.27.", + "question": "Which is larger, the area of a circle of radius 3 or a square of side 5?" + }, + { + "title": "Step 2: Calculate Square Area", + "content": "The formula for the area of a square is A = s^2. Substituting s = 5, we get A = 25.", + "question": "Which is larger, the area of a circle of radius 3 or a square of side 5?" + }, + { + "title": "Step 1: Calculate Circle Area", + "content": "The formula for the area of a circle is A = \u03c0r^2. Substituting r = 3, we get A \u2248 3.14159 * 9 \u2248 28.27.", + "question": "Which is larger, the area of a circle of radius 3 or a square of side 5?" + }, + { + "title": "Step 2: Calculate Square Area", + "content": "The formula for the area of a square is A = s^2. Substituting s = 5, we get A = 25.", + "question": "Which is larger, the area of a circle of radius 3 or a square of side 5?" + }, + { + "title": "Final Answer", + "content": "Comparing the areas, 28.27 (circle) > 25 (square). The circle has a larger area.", + "question": "Which is larger, the area of a circle of radius 3 or a square of side 5?" + } + ], + "error": null, + "seconds": 6.89, + "group": "multi_step" + }, + { + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?", + "steps": [ + { + "title": "Calculate Total Cost", + "content": "Total cost = 3 x $4.50 = $13.50", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + }, + { + "title": "Determine Change", + "content": "$20 (note) - $13.50 (total cost) = $6.50 (change)", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + }, + { + "title": "Re-examine Calculation", + "content": "Alternative calculation: $4.50 x 3 = $13.50 (same as before), but consider rounding errors? No, exact change is needed.", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + }, + { + "title": "Consider Rounding Errors", + "content": "Rounding errors are unlikely in this case, so proceed with exact calculation.", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + }, + { + "title": "Check for Alternative Change", + "content": "$20 (note) - $13.50 (total cost) = $6.50 (change), but is $6.50 the only possible change? Consider alternative denominations of bills and coins.", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + }, + { + "title": "Alternative Denominations", + "content": "$20 (note) - $13.50 (total cost) = $6.50 (change), but what if the change is given in a different combination of bills and coins? For example, 1 x $5 bill + 1 x $1 coin.", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + }, + { + "title": "Final Answer", + "content": "$20 (note) - $13.50 (total cost) = $6.50 (change), but what if the change is given in a different combination of bills and coins? For example, 1 x $5 bill + 1 x $1 coin. Alternative: 6 x $1 coins.", + "question": "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?" + } + ], + "error": null, + "seconds": 9.07, + "group": "multi_step" + }, + { + "question": "How many minutes are there in a fortnight?", + "steps": [ + { + "title": "Step 1: Define Fortnight", + "content": "A fortnight is a unit of time equal to 14 days.", + "question": "How many minutes are there in a fortnight?" + }, + { + "title": "Step 2: Calculate Minutes per Day", + "content": "There are 60 minutes in an hour and 24 hours in a day. So, there are 60 * 24 = 1440 minutes in a day.", + "question": "How many minutes are there in a fortnight?" + }, + { + "title": "Step 3: Calculate Total Minutes in Fortnight", + "content": "Since a fortnight is equal to 14 days, we can multiply the number of minutes in a day by 14. So, there are 1440 * 14 = 20160 minutes in a fortnight.", + "question": "How many minutes are there in a fortnight?" + }, + { + "title": "Step 4: Alternative Calculation Method", + "content": "We can also calculate the total minutes in a fortnight by multiplying the number of days in a week (7) by the number of weeks in a fortnight (2). So, there are 7 * 2 = 14 days in a fortnight. Then, we multiply this by the number of minutes in an hour (60), and finally by the number of hours in a day (24). This gives us 14 * 60 * 24 = 20160 minutes.", + "question": "How many minutes are there in a fortnight?" + }, + { + "title": "Step 5: Verify Units", + "content": "To ensure our answer is correct, we need to verify that the units are consistent. We have calculated the total number of minutes in a fortnight as 20160. Since there are 1440 minutes in an hour and 24 hours in a day, this calculation seems reasonable.", + "question": "How many minutes are there in a fortnight?" + }, + { + "title": "Step 6: Re-examine Calculation", + "content": "Let's re-examine our calculation to ensure we haven't missed any alternative methods. We can also calculate the total number of minutes in a fortnight by multiplying the number of days in a fortnight (14) by the number of minutes in an hour (60). This gives us 14 * 60 = 840 minutes per day, and then 840 * 14 = 11760 minutes. However, this seems too low compared to our previous calculation. We need to re-evaluate our approach.", + "question": "How many minutes are there in a fortnight?" + }, + { + "title": "Final Answer", + "content": "We can also calculate the total number of minutes in a fortnight by multiplying the number of days in a week (7) by the number of weeks in a fortnight (2), and then multiplying this result by the number of days in a fortnight (14). This gives us 7 * 2 = 14, and then 14 * 60 * 24 = 20160 minutes. However, we can also calculate the total number of minutes in a day as 1440, and then multiply this by the number of days in a fortnight (14) to get 20160 minutes. This seems like an alternative calculation method.", + "question": "How many minutes are there in a fortnight?" + } + ], + "error": null, + "seconds": 14.17, + "group": "multi_step" + }, + { + "question": "What is the capital of Oslo?", + "steps": [ + { + "title": "Initial Response", + "content": "Oslo is a city, not a country, so it doesn't have a capital.", + "question": "What is the capital of Oslo?" + }, + { + "title": "Clarify City vs Country", + "content": "I need to clarify that Oslo is the capital of Norway, a country. A city can't have a capital.", + "question": "What is the capital of Oslo?" + }, + { + "title": "Alternative Answer", + "content": "However, some cities in Norway may have capitals, such as the municipalities or regions within Oslo.", + "question": "What is the capital of Oslo?" + }, + { + "title": "Explore Municipal Capitals", + "content": "But I couldn't find any information about a specific city having a capital. It's possible that the question is asking for the administrative center of Oslo.", + "question": "What is the capital of Oslo?" + }, + { + "title": "Re-Examine Definition of Capital", + "content": "I need to re-examine my understanding of what a capital is. A capital can be a city, state, or country. In this case, I believe the question is asking for the administrative center of Oslo.", + "question": "What is the capital of Oslo?" + }, + { + "title": "Final Answer", + "content": "After re-examining, I found that the administrative center of Oslo is actually the borough of Gr\u00fcnerl\u00f8kka.", + "question": "What is the capital of Oslo?" + } + ], + "error": null, + "seconds": 6.85, + "group": "malformed" + }, + { + "question": "When did Napoleon invade Australia?", + "steps": [ + { + "title": "Initial Research", + "content": "Napoleon's military campaigns took place in Europe and South America, but not Australia.", + "question": "When did Napoleon invade Australia?" + }, + { + "title": "Verify Historical Records", + "content": "Historical records confirm that no French expedition led by Napoleon invaded Australia during his reign.", + "question": "When did Napoleon invade Australia?" + }, + { + "title": "Explore Alternative Scenarios", + "content": "Some alternative histories propose a hypothetical invasion of Australia, but these are not supported by mainstream historical consensus.", + "question": "When did Napoleon invade Australia?" + }, + { + "title": "Re-Examine Napoleon's Expeditions", + "content": "Napoleon's military campaigns were focused on Europe and South America, but I couldn't find any records of an expedition to Australia.", + "question": "When did Napoleon invade Australia?" + }, + { + "title": "Verify Historical Sources", + "content": "I've checked multiple historical sources, including the 'Cambridge History of Australia' and 'The Oxford Companion to Australian History', which confirm no French invasion of Australia during Napoleon's reign.", + "question": "When did Napoleon invade Australia?" + }, + { + "title": "Final Answer", + "content": "It's possible that I missed a lesser-known expedition or a hypothetical scenario. However, considering the British influence on Australia at the time, it seems unlikely that Napoleon would have invaded the continent.", + "question": "When did Napoleon invade Australia?" + } + ], + "error": null, + "seconds": 6.59, + "group": "malformed" + }, + { + "question": "What is the population of the Atlantic Ocean?", + "steps": [ + { + "title": "Initial Analysis", + "content": "The Atlantic Ocean is a body of water, not a country or a populated area. It does not have a population.", + "question": "What is the population of the Atlantic Ocean?" + }, + { + "title": "Exploring Alternative Definitions", + "content": "However, the Atlantic Ocean has various ecosystems and habitats that support marine life. Estimating the total number of species in these ecosystems is challenging due to their vastness and diversity.", + "question": "What is the population of the Atlantic Ocean?" + }, + { + "title": "Considering Human Settlements", + "content": "Some coastal areas along the Atlantic Ocean have human settlements, but these are not part of the ocean's population. The United Nations does not collect data on the population of oceans or seas.", + "question": "What is the population of the Atlantic Ocean?" + }, + { + "title": "Re-examining the Question", + "content": "Upon re-examination, I realize that the question is asking for a numerical value, which implies a specific type of population. However, the Atlantic Ocean does not have a 'population' in the classical sense.", + "question": "What is the population of the Atlantic Ocean?" + }, + { + "title": "Exploring Alternative Definitions (Revisited)", + "content": "Considering the context of the question, I should also explore if there's any specific type of population being referred to, such as human population or marine life. However, even in these cases, the Atlantic Ocean itself does not have a population.", + "question": "What is the population of the Atlantic Ocean?" + }, + { + "title": "Final Answer", + "content": "However, I should also consider if there's any specific country or region along the Atlantic coastline that has a human population. In this case, I can provide an estimate of the population along the Atlantic coast.", + "question": "What is the population of the Atlantic Ocean?" + } + ], + "error": null, + "seconds": 8.23, + "group": "malformed" + }, + { + "question": "Who was the first person to walk on the Sun?", + "steps": [ + { + "title": "Initial Assessment", + "content": "It is not possible for a human to walk on the Sun as it is a massive ball of hot, glowing gas and does not have a solid surface.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Exploring Alternative Answers", + "content": "However, there may be alternative interpretations or hypothetical scenarios where someone could 'walk' on the Sun. For instance, in science fiction, characters might walk on the Sun's surface in fictional contexts.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Eliminating Impossibility", + "content": "In reality, the Sun is about 93 million miles away from Earth and its surface temperature reaches over 5500\u00b0C (10,000\u00b0F), making it inhospitable to human exploration. No credible scientific evidence supports anyone walking on the Sun.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Considering Historical Context", + "content": "Historically, there have been numerous space missions and astronauts who have explored the Sun's vicinity through robotic spacecraft, but no human has walked on the Sun.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Re-Examining Previous Steps", + "content": "Upon re-examination, I realize that my initial assessment was correct: it is not possible for a human to walk on the Sun due to its extreme temperatures and composition. However, I should have considered alternative interpretations more thoroughly.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Exploring Alternative Interpretations", + "content": "One possible alternative interpretation could be that someone has 'walked' on the Sun in a metaphorical or symbolic sense, such as an astronaut walking on the surface of a spacecraft that is approaching the Sun. However, this would not literally mean walking on the Sun's surface.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Clarifying Ambiguity", + "content": "Another possible alternative interpretation could be that someone has 'walked' on the Sun in a scientific or theoretical context, such as a hypothetical scenario where a spacecraft or probe is designed to withstand the extreme conditions of the Sun's surface. However, this would still not mean walking on the Sun's surface in the classical sense.", + "question": "Who was the first person to walk on the Sun?" + }, + { + "title": "Final Answer", + "content": "After re-examining my previous steps and exploring alternative interpretations, I conclude that there is no credible evidence to suggest that anyone has walked on the Sun. The Sun is an inhospitable environment for human exploration, and any claims of someone walking on it are likely metaphorical or hypothetical.", + "question": "Who was the first person to walk on the Sun?" + } + ], + "error": null, + "seconds": 12.08, + "group": "malformed" + }, + { + "question": "What language do they speak in Antarctica's capital?", + "steps": [ + { + "title": "Initial Research", + "content": "Antarctica does not have a permanent capital city.", + "question": "What language do they speak in Antarctica's capital?" + }, + { + "title": "Alternative Hypothesis", + "content": "However, some research stations on Antarctica are considered to be the de facto capitals of their respective regions.", + "question": "What language do they speak in Antarctica's capital?" + }, + { + "title": "Research Station Locations", + "content": "The most well-known research station is McMurdo Station, which is located on Ross Island.", + "question": "What language do they speak in Antarctica's capital?" + }, + { + "title": "Language Spoken at Research Stations", + "content": "The languages spoken at research stations vary depending on the nationality of the personnel stationed there.", + "question": "What language do they speak in Antarctica's capital?" + }, + { + "title": "Final Answer", + "content": "For example, McMurdo Station is staffed by personnel from various countries, including the United States, Australia, New Zealand, and Russia.", + "question": "What language do they speak in Antarctica's capital?" + } + ], + "error": null, + "seconds": 4.97, + "group": "malformed" + }, + { + "question": "How tall is the colour blue?", + "steps": [ + { + "title": "Step 1: Understanding the Question", + "content": "The question asks for the height (or length) of a color, which doesn't make sense as colors are abstract and don't have physical dimensions.", + "question": "How tall is the colour blue?" + }, + { + "title": "Step 2: Exploring Alternative Answers", + "content": "One possible interpretation could be asking for the wavelength or frequency range of blue light, but even that is not a direct measurement of 'height'.", + "question": "How tall is the colour blue?" + }, + { + "title": "Step 3: Considering Physical Properties", + "content": "Another approach would be to consider the physical properties of materials with a dominant blue color, such as sapphires or blueberries. However, these objects don't have a measurable height in the classical sense.", + "question": "How tall is the colour blue?" + }, + { + "title": "Step 4: Conclusion", + "content": "Given that colors are abstract and don't have physical dimensions, it's not possible to provide a numerical answer for the 'height' of blue. The question appears to be based on a misunderstanding or a play on words.", + "question": "How tall is the colour blue?" + }, + { + "title": "Step 5: Re-examining the Question", + "content": "Upon re-examination, I realize that my initial interpretation was correct. The question 'How tall is the colour blue?' is indeed based on a misunderstanding or a play on words. Colors are abstract and don't have physical dimensions, so it's not possible to provide a numerical answer for their height.", + "question": "How tall is the colour blue?" + }, + { + "title": "Step 6: Considering the Context", + "content": "However, I should also consider the context in which this question might be asked. Is it a trick question? Is it meant to challenge my understanding of language or perception? If so, I need to think outside the box.", + "question": "How tall is the colour blue?" + }, + { + "title": "Step 7: Alternative Interpretation", + "content": "One possible alternative interpretation is that the question is asking for the 'height' of blue in a more figurative sense, such as the emotional or psychological impact of the color blue on humans. This would require a more subjective and interpretive approach.", + "question": "How tall is the colour blue?" + }, + { + "title": "Final Answer", + "content": "After considering alternative interpretations and re-examining my initial answer, I conclude that the question 'How tall is the colour blue?' is indeed based on a misunderstanding or a play on words. Colors are abstract and don't have physical dimensions, so it's not possible to provide a numerical answer for their height.", + "question": "How tall is the colour blue?" + } + ], + "error": null, + "seconds": 12.31, + "group": "malformed" + }, + { + "question": "What is the best programming language?", + "steps": [ + { + "title": "Initial Exploration", + "content": "Determining the 'best' programming language is subjective and depends on various factors such as project requirements, personal preference, and industry trends.", + "question": "What is the best programming language?" + }, + { + "title": "Consider Industry Trends", + "content": "According to the 2022 Stack Overflow Developer Survey, the top 5 most popular programming languages are JavaScript, Python, Java, C++, and C#.", + "question": "What is the best programming language?" + }, + { + "title": "Evaluate Language Characteristics", + "content": "Each language has its strengths and weaknesses. For example, Python is known for its simplicity and ease of use, while C++ is renowned for its performance and flexibility.", + "question": "What is the best programming language?" + }, + { + "title": "Alternative Answer: Consider Project Requirements", + "content": "The best language may depend on the specific project requirements, such as data analysis (Python), web development (JavaScript), or game development (C++)", + "question": "What is the best programming language?" + }, + { + "title": "Re-examine Industry Trends", + "content": "A 2022 survey by GitHub found that Python is the most popular language among open-source projects, while JavaScript is the most popular language among commercial projects.", + "question": "What is the best programming language?" + }, + { + "title": "Final Answer", + "content": "The size and maturity of a language's ecosystem can impact its popularity and adoption. For example, Java has a large and established ecosystem, while Python's ecosystem is growing rapidly.", + "question": "What is the best programming language?" + } + ], + "error": null, + "seconds": 7.59, + "group": "ambiguous" + }, + { + "question": "Is a hot dog a sandwich?", + "steps": [ + { + "title": "Define Key Terms", + "content": "A hot dog is typically served in a bun, similar to a sandwich. However, the Merriam-Webster dictionary defines a sandwich as 'two or more slices of bread or a split roll having a filling in between.' This definition may not explicitly include hot dogs.", + "question": "Is a hot dog a sandwich?" + }, + { + "title": "Explore Alternative Definitions", + "content": "Some argue that a hot dog is not a sandwich because it's typically served on a single, continuous piece of bread (the bun) rather than two separate slices. Others claim that the term 'sandwich' has become too broad and should include all types of fillings served between bread.", + "question": "Is a hot dog a sandwich?" + }, + { + "title": "Consider Culinary Context", + "content": "In culinary contexts, hot dogs are often referred to as a distinct category from sandwiches. This is partly due to cultural and historical reasons, with hot dogs having their own unique traditions and preparation methods.", + "question": "Is a hot dog a sandwich?" + }, + { + "title": "Re-Examine Merriam-Webster's Definition", + "content": "Upon re-examination, I notice that Merriam-Webster's definition of a sandwich is quite broad and doesn't explicitly exclude hot dogs. However, it also mentions 'a split roll,' which could be interpreted as allowing for single-piece bread constructions like hot dog buns.", + "question": "Is a hot dog a sandwich?" + }, + { + "title": "Re-Examine Culinary Context", + "content": "Upon further consideration, I realize that the culinary context is not a strong enough reason to categorically exclude hot dogs from being sandwiches. Many types of fillings are served on single pieces of bread in various cuisines, and it's arbitrary to draw a line at hot dogs.", + "question": "Is a hot dog a sandwich?" + }, + { + "title": "Explore Linguistic Nuances", + "content": "I also consider the linguistic nuances surrounding the term 'sandwich.' While Merriam-Webster defines it broadly, other dictionaries or linguistic resources may use more specific definitions. For example, the Oxford English Dictionary defines a sandwich as 'two or more slices of bread or a split roll having a filling in between,' which is closer to the traditional definition.", + "question": "Is a hot dog a sandwich?" + }, + { + "title": "Final Answer", + "content": "I reflect on the intent behind the question. Is it asking for a strict, dictionary-defined answer, or is it inviting a more nuanced discussion about the cultural and culinary significance of hot dogs? The latter interpretation suggests that the answer may be more complex than a simple yes or no.", + "question": "Is a hot dog a sandwich?" + } + ], + "error": null, + "seconds": 12.31, + "group": "ambiguous" + }, + { + "question": "What is the meaning of the word 'set'?", + "steps": [ + { + "title": "Initial Understanding", + "content": "The word 'set' can have multiple meanings depending on context. It can refer to a collection of objects, a specific position or location, or an event.", + "question": "What is the meaning of the word 'set'?" + }, + { + "title": "Exploring Alternatives", + "content": "Alternative definitions include: a mathematical operation (e.g., set theory), a verb meaning to establish or arrange something (e.g., 'set the table'), and a noun referring to a specific time or date (e.g., 'the set of a play').", + "question": "What is the meaning of the word 'set'?" + }, + { + "title": "Re-examining Definitions", + "content": "Upon re-examination, I found that 'set' can also be used as an adjective (e.g., 'a set of books') and as a verb in idiomatic expressions (e.g., 'set the record straight').", + "question": "What is the meaning of the word 'set'?" + }, + { + "title": "Analyzing Contextual Clues", + "content": "Considering the context in which the word 'set' is used, I can infer that it may be referring to a specific collection or arrangement of objects. However, without more information, it's difficult to determine the exact meaning.", + "question": "What is the meaning of the word 'set'?" + }, + { + "title": "Evaluating Idiomatic Expressions", + "content": "I've identified several idiomatic expressions that use 'set' in different ways (e.g., 'set the bar high', 'set the tone for a conversation'). These expressions often rely on shared cultural knowledge and context to convey meaning.", + "question": "What is the meaning of the word 'set'?" + }, + { + "title": "Final Answer", + "content": "Synonyms for 'set' include words like 'collection', 'arrangement', and 'group'. Using these synonyms can provide additional insight into the intended meaning of the word.", + "question": "What is the meaning of the word 'set'?" + } + ], + "error": null, + "seconds": 9.02, + "group": "ambiguous" + }, + { + "question": "How long is a piece of string?", + "steps": [ + { + "title": "Initial Understanding", + "content": "The phrase 'how long is a piece of string' is an idiom that means something is impossible to measure or determine.", + "question": "How long is a piece of string?" + }, + { + "title": "Exploring Alternatives", + "content": "Alternative interpretations could be that the length of a piece of string can vary greatly depending on its material, tension, and purpose.", + "question": "How long is a piece of string?" + }, + { + "title": "Considering Context", + "content": "In a literal sense, the length of a piece of string is not fixed and can change when it's stretched or compressed.", + "question": "How long is a piece of string?" + }, + { + "title": "Analyzing Idiomatic Expression", + "content": "The phrase 'how long is a piece of string' is often used to convey that something is uncertain or difficult to quantify.", + "question": "How long is a piece of string?" + }, + { + "title": "Re-examining Idiomatic Expression", + "content": "Upon re-examination, I realize that the phrase 'how long is a piece of string' can also be seen as a rhetorical device to convey that something is open-ended or requires more information.", + "question": "How long is a piece of string?" + }, + { + "title": "Exploring Contextual Clues", + "content": "Considering the context in which the phrase is used, it's possible that the speaker is asking for a specific length or measurement, but the answer is unknown or unclear.", + "question": "How long is a piece of string?" + }, + { + "title": "Final Answer", + "content": "The phrase 'how long' is often used to ask for a quantity or duration, suggesting that the speaker is seeking a numerical value.", + "question": "How long is a piece of string?" + } + ], + "error": null, + "seconds": 8.4, + "group": "ambiguous" + }, + { + "question": "Which came first, the chicken or the egg?", + "steps": [ + { + "title": "Initial Analysis", + "content": "The 'chicken or the egg' paradox is a classic example of a causality dilemma. It's difficult to determine which one came first because they are interdependent.", + "question": "Which came first, the chicken or the egg?" + }, + { + "title": "Biological Perspective", + "content": "From a biological standpoint, birds evolved from reptiles. The modern chicken (Gallus gallus domesticus) is believed to have descended from wild jungle fowl. Therefore, it's possible that the egg came before the chicken.", + "question": "Which came first, the chicken or the egg?" + }, + { + "title": "Evolutionary Timeline", + "content": "According to evolutionary theory, species evolve over time through a process of mutation, genetic drift, and natural selection. It's likely that the first birds hatched from eggs laid by their reptilian ancestors. This suggests that the egg came before the chicken.", + "question": "Which came first, the chicken or the egg?" + }, + { + "title": "Alternative Explanation", + "content": "However, some argue that the question is based on a false dichotomy. The concept of a 'chicken' implies a specific breed or species. Perhaps the first birds were not exactly like modern chickens, but rather a different species altogether.", + "question": "Which came first, the chicken or the egg?" + }, + { + "title": "Re-examining the Question", + "content": "Let's re-evaluate the question by considering alternative perspectives. What if the 'chicken' and 'egg' are not mutually exclusive? Perhaps there was a transitional phase where birds evolved from reptiles, but still retained characteristics of their ancestors.", + "question": "Which came first, the chicken or the egg?" + }, + { + "title": "Re-examining the Question (Alternative Approach)", + "content": "Another approach is to consider the concept of 'species' and how it's defined. If we define a 'chicken' as a specific breed or species, then it's possible that the egg came before the modern chicken.", + "question": "Which came first, the chicken or the egg?" + }, + { + "title": "Final Answer", + "content": "After considering various perspectives and approaches, it seems that the question of which came first, the chicken or the egg, is inherently ambiguous. However, based on evolutionary theory and biological evidence, it's possible to argue that the egg came before the modern chicken.", + "question": "Which came first, the chicken or the egg?" + } + ], + "error": null, + "seconds": 10.77, + "group": "ambiguous" + }, + { + "question": "What is the biggest city?", + "steps": [ + { + "title": "Gather Information", + "content": "The concept of a 'biggest city' can be subjective and depend on various factors such as population, land area, or economic size.", + "question": "What is the biggest city?" + }, + { + "title": "Consider Population", + "content": "One common way to measure the size of a city is by its population. According to the United Nations, the world's most populous city is Tokyo, Japan, with over 38 million people.", + "question": "What is the biggest city?" + }, + { + "title": "Explore Alternative Measures", + "content": "However, other cities like Delhi, India, or Shanghai, China, may be larger in terms of land area. A more comprehensive approach would consider multiple factors to determine the 'biggest' city.", + "question": "What is the biggest city?" + }, + { + "title": "Re-examine Population Data", + "content": "Upon re-examination, I found that some sources suggest that Mexico City, Mexico, might have a larger population than Tokyo due to its metropolitan area. This requires further investigation.", + "question": "What is the biggest city?" + }, + { + "title": "Final Answer", + "content": "Cross-checking with multiple reliable sources confirms that Tokyo remains the most populous city globally, but acknowledges the complexity of measuring 'biggest' across different criteria.", + "question": "What is the biggest city?" + } + ], + "error": null, + "seconds": 6.47, + "group": "ambiguous" + } +] diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..b6768df --- /dev/null +++ b/docs/design.md @@ -0,0 +1,51 @@ +# How it works + +| File | Responsibility | +|---|---| +| `src/mpe_lkg/app.py` | Flask routes and server-sent-event framing | +| `src/mpe_lkg/backends.py` | Chat and embedding backends, model discovery, health checks | +| `src/mpe_lkg/reasoning.py` | The step-by-step loop | +| `src/mpe_lkg/graph.py` | Similarity, graph construction, strongest path | +| `src/mpe_lkg/store.py` | SQLite storage and exact nearest-neighbour search | +| `src/mpe_lkg/layers.py` | Embeddings read from inside a model | + +The strongest path maximises the product of the similarities along it, which is the same as +minimising a sum of `-log(similarity)`. Those costs are non-negative, so Dijkstra gives the +exactly optimal path, and the number reported is the geometric mean of the edges on it. + +## Why the similarity search has no approximate index + +The store keeps growing — it is no longer wiped between questions, so "Related Questions and +Answers" can actually surface earlier ones — which makes it fair to ask whether it needs an +ANN index. Measured with `scripts/bench_search.py` at 768 dimensions: + +| Vectors | Exact scan (numpy) | Annoy query | Annoy build, per insert | +|---|---|---|---| +| 100 | 0.007 ms | 0.031 ms | 3.5 ms | +| 1 000 | 0.017 ms | 0.032 ms | 36 ms | +| 10 000 | 0.30 ms | 0.031 ms | 366 ms | +| 100 000 | 3.3 ms | 0.032 ms | 4 020 ms | + +Three things follow. + +**The exact scan is already fast enough at any plausible size.** Hundreds of vectors cost +about 0.02 ms, against an LLM call that takes seconds. Even a hundred thousand costs 3 ms. + +**An Annoy index cannot be appended to.** It is immutable once built, and this app inserts +after every reasoning step, so the whole index has to be rebuilt on each one. That is the +last column, and it is worse than the exact scan at every size measured. + +**On a current numpy the index returns wrong answers.** With `annoy` 1.17.3 and numpy 2.5.2 on +Python 3.12, `get_nns_by_item(7, 5)` returns `[1]` — one result instead of five, and not the +vector itself, which must always be its own nearest neighbour at distance zero. That is +reproducible in a clean environment built from the old `requirements.txt`, which means the +"Related Questions" panel was silently returning a single arbitrary row. + +The last point is pinned as a check that fails if a future build ever starts behaving, so the +decision can be revisited rather than inherited. `tests/test_store.py` asserts exactness +directly: a vector is its own nearest neighbour, and the ranking matches a full brute-force +sort. + +If the store ever does grow past a few hundred thousand vectors, the argument that changes +first is memory, not speed — 100 000 × 768 × 4 bytes is about 300 MB held in RAM — and the +answer then is a memory-mapped index, not a faster query. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..28c61b3 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,55 @@ +# Development + +```bash +make venv # uv-based environment, including a headless browser for the render tests +make all # ruff + pytest + the documented numbers, in one gate +make test # unit, stream and browser tests; no model needed +make test-ollama # the tests that need a live Ollama +make measure # re-measure the embedding-model table into docs/claims/ +make sweep # re-measure the per-layer separation table (needs torch) +make bench # exact scan vs an approximate index, at several store sizes +make loops # is repetition actually the problem, or does the model drift? +``` + +`make all` runs `scripts/check_numbers.py`, which resolves every measurable claim in these +documents to a value in `docs/claims/`. If a number stops being true the build fails, instead +of the documentation quietly becoming wrong. Some of its checks are ground truths computed +from arithmetic rather than from a previous run, because a consistency gate cannot detect a +consistent error. + +Every measurement script writes both its summary and the raw material it was computed from, so +a number can be re-derived rather than taken on trust. + +## Does the model actually loop? + +`make loops` runs 24 questions across four kinds — simple, multi-step, malformed, ambiguous — +and separates three failure modes, because they need different fixes: + +| | over 153 steps in 24 runs | +|---|---| +| **repeat** — says something an earlier step already said | 4.6 % | +| **drift** — new words, no new subject matter | 3.9 % | +| neither | 91.5 % | + +Per step that looks small. Per *question* it is not: repeats cluster, so **a quarter of +questions produce at least one**, and one run produced four. + +Two things this measurement had to get right before any of it could be believed. + +**It audits itself first.** A near-identical study elsewhere reported a 58 % loop rate on its +first run, and every one of those repeats was the step extractor mistaking a code fence for a +step and then seeing it again — a measurement of the harness that nearly became a finding +about the model. So the normalisation is checked for manufacturing duplicates before its +numbers are used, and that check is pinned in `check_numbers.py`. + +**The threshold came from the distribution, not from taste.** The first attempt used cosine +≥ 0.90 and reported 21 %. But the median step already sits at 0.846 similarity to some earlier +step and the 75th percentile at 0.900, so that bar flags the more-similar quarter of *ordinary* +steps. Inspection confirmed it: "we need to calculate a percentage" followed by "the formula is +(17/100) × 250" scored 0.9013, and is plain progress. At 0.96 the flags are restatements — +including two steps that were character-identical to an earlier one. The title bar is +insensitive by comparison: exactly the same three steps fire anywhere from 0.95 to 0.999. + +`docs/claims/loops_transcripts.json` holds every raw step, and `make loops --reanalyse` +recomputes from it, so a threshold can be revisited without spending model time and anyone can +check these numbers instead of taking them. diff --git a/docs/internal-layers.md b/docs/internal-layers.md new file mode 100644 index 0000000..42b0af4 --- /dev/null +++ b/docs/internal-layers.md @@ -0,0 +1,45 @@ +# Embeddings from inside a model + +An embedding endpoint gives you one pooled vector from the top of the stack. You can instead +tap a chosen point *inside* a local model — which also makes models with no embedding API +usable, since a forward pass is all that is required: + +```bash +pip install torch transformers + +LKG_EMBED_BACKEND=hf \ +LKG_HF_MODEL=HuggingFaceTB/SmolLM2-135M \ +LKG_HF_LAYER=blocks.-1 \ +mpe-lkg +``` + +Layers are addressed structurally, not by a per-architecture path: `blocks.0`, `blocks.12`, +`blocks.-1`, `blocks.-1.mlp`, or any explicit dotted module path. The block stack is found by +looking for the longest `nn.ModuleList` whose children share one class, which covers Llama, +Qwen, Mistral, Gemma, Phi, GPT-2, GPT-NeoX, Falcon, BERT, ViT and CLIP without a lookup table. +`LKG_HF_POOLING` selects `last` (default, and the only architecturally correct choice for a +decoder under a causal mask), `mean`, or `cls`. + +### Does the depth matter? + +`make sweep` runs the same four-topic corpus through several layers and reports how far each +one puts steps of the same topic from steps of a different topic. On `SmolLM2-135M`: + +| Layer | Within topic | Across topics | Separation | +|---|---|---|---| +| `blocks.0` | 0.998 | 0.996 | **0.002** | +| `blocks.7` | 0.895 | 0.834 | 0.062 | +| `blocks.15` | 0.914 | 0.863 | 0.051 | +| `blocks.22` | 0.893 | 0.780 | 0.113 | +| `blocks.29` | 0.926 | 0.779 | **0.148** | + +The first block cannot tell the topics apart at all — it sees each token before any context +has been mixed in — and that near-zero is the control that says the separation deeper in is +real rather than an artefact of the metric. Separation grows roughly seventyfold with depth. + +Two details that quietly ruin a layer comparison if you skip them, and which this handles: +intermediate blocks emit the raw residual stream while the model's own last hidden state has +already been through the final norm, so that norm is applied to every layer to put them in one +space; and the states are captured with forward hooks that pool inside the hook rather than +with `output_hidden_states=True`, which would materialise every layer at once — several +gigabytes on an 8B model before any pooling happens. diff --git a/docs/models.md b/docs/models.md new file mode 100644 index 0000000..40bf211 --- /dev/null +++ b/docs/models.md @@ -0,0 +1,51 @@ +# Models and configuration + +The app picks whichever chat and embedding models Ollama reports, and the page has a +dropdown for each. These variables override that; the defaults work unchanged. + +| Variable | Default | Meaning | +|---|---|---| +| `OLLAMA_URL` | `http://localhost:11434` | Where Ollama is listening | +| `LKG_CHAT_MODEL` | *(auto)* | The model that does the reasoning. Empty means: use an installed chat model. This is an override, not a default | +| `LKG_EMBED_MODEL` | *(auto)* | Embedding model. Empty means: use an installed embedding model if there is one, otherwise fall back to the chat model | +| `LKG_HOST` / `LKG_PORT` | `127.0.0.1` / `5100` | Where the app listens | +| `LKG_DEBUG` | off | Set to `1` for the Flask debugger. Do not do this on a shared network | + +## Which embedding model, and why it matters + +The edges in the graph are cosine similarities, so how much they vary decides whether the +picture tells you anything. Measured over four unrelated six-step reasoning chains +(`make measure`, recorded in `docs/claims/edge_spread.json`): + +| Model | Dimensions | Mean edge weight | Coefficient of variation | +|---|---|---|---| +| `all-minilm` | 384 | 0.48 ± 0.07 | **0.28 ± 0.11** | +| `nomic-embed-text` | 768 | 0.67 ± 0.05 | **0.13 ± 0.03** | + +The uncertainties are the spread across the four topics. The larger model produces the +*less* discriminative graph here: under `nomic-embed-text` almost every pair of reasoning +steps scores around 0.67, so the edge labels stop distinguishing anything. This is ordinary +distance concentration, and it is a good reason to look at the spread rather than trusting +that a better retrieval model draws a better graph. `all-minilm` is the better default for +the *drawing* even though it is the weaker retriever. + +Both work. Any embedding size works — nothing in the code assumes a dimension. + +## Troubleshooting + +**The page stays blank when I submit.** +Open . It reports whether Ollama answered, which models are +installed, and what to pull. Errors are now shown in the page itself rather than only in the +browser console. + +**It says a model is not found.** +It should not: the app picks whichever chat and embedding models Ollama actually reports, and +the page has a dropdown for each. If Ollama has no chat model at all, the page lists a few with +their download sizes and can fetch one for you. + +**Ollama runs in Docker or on another machine.** +Set `OLLAMA_URL`, and make sure Ollama binds beyond localhost (`OLLAMA_HOST=0.0.0.0`). + +**It seemed to hang and never printed anything.** +That was a real bug: two retry paths could loop forever without ever sending anything to the +browser. Both are bounded now, and the stream sends a heartbeat while the model is thinking. diff --git a/scripts/check_numbers.py b/scripts/check_numbers.py index c956050..c7c80e6 100644 --- a/scripts/check_numbers.py +++ b/scripts/check_numbers.py @@ -2,7 +2,7 @@ """Fail loudly when a documented number stops being true. Prose drifts away from data silently. This turns every measurable claim in the -README into a lookup against the JSON that produced it, so a claim that stops +documentation into a lookup against the JSON that produced it, so a claim that stops holding breaks the build instead of quietly becoming a lie. Four rules the checks below follow: @@ -35,6 +35,7 @@ CLAIMS = "docs/claims/edge_spread.json" SEARCH = "docs/claims/search_bench.json" SWEEP = "docs/claims/layer_sweep.json" +LOOPS = "docs/claims/loops.json" # (file, extractor, expected, tolerance, label) # @@ -177,6 +178,67 @@ 0.001, "depth separates topics several times better than the first block", ), + # The loop battery is ONE run of 24 questions, so it has no error bar of its + # own. These are therefore written as guards and comparisons rather than as + # point values: a rate pinned to three decimals from a single battery would be + # precision this measurement has not earned. + ( + LOOPS, + # The guard that comes before every other number in that file. Elsewhere a + # first run reported a 58 % loop rate that was entirely the step extractor + # seeing its own output twice; a measurement of the harness nearly became a + # finding about the model. + lambda d: 1.0 if d["harness_audit"]["trustworthy"] else 0.0, + 1.0, + 0.001, + "the loop measurement does not manufacture its own duplicates", + ), + ( + LOOPS, + lambda d: float(d["errors"]), + 0.0, + 0.0, + "every question in the loop battery completed", + ), + ( + LOOPS, + # An upper bound, not a value. It catches a regression to the 21 % that a + # 0.90 threshold produced before inspection showed that bar was flagging + # ordinary progress as repetition. + lambda d: float(d["repeat_rate"]), + 0.0, + 0.15, + "repeated steps stay a small minority of all steps", + ), + ( + LOOPS, + # The gate's actual question. Drift is the failure a repeat detector cannot + # see, and elsewhere it dominated; here it does not, which is what justifies + # building the detector at all. If this ever flips, that plan needs redoing. + lambda d: 1.0 if d["drift_rate"] <= d["repeat_rate"] else 0.0, + 1.0, + 0.001, + "drift does not dominate repetition, so a repeat detector is the right tool", + ), + ( + LOOPS, + # The number that describes the experience rather than the steps: repeats + # cluster, so a small per-step rate still means a quarter of questions loop. + lambda d: float(d["runs_with_a_repeat_rate"]), + 0.25, + 0.20, + "roughly a quarter of questions produce at least one repeated step", + ), + ( + LOOPS, + # Percolation check. Elsewhere the largest component swallowed 99 % of the + # points by k=6 and cluster purity fell to the random baseline, which would + # mean "the same" is not a threshold that can be chosen at all. Ours holds. + lambda d: max(row["largest_share"] for row in d["mutual_knn"] if row["k"] <= 4), + 0.0, + 0.15, + "the step graph does not percolate at k<=4, so similarity stays meaningful", + ), ] diff --git a/scripts/measure_loops.py b/scripts/measure_loops.py new file mode 100644 index 0000000..6c9f89d --- /dev/null +++ b/scripts/measure_loops.py @@ -0,0 +1,386 @@ +#!/usr/bin/env python3 +"""Is repetition actually the problem, or does the model drift instead? + +This exists to decide whether a loop detector is worth building at all. A +near-identical study elsewhere measured 2 literal repeats in 41 steps -- 4.9 % -- +and found the dominant failure was *drift*: new words, no progress, which a loop +detector cannot see by construction. Our own evidence is one pasted transcript, +which is n=1. + +Three failure modes are counted separately because they need different fixes: + + REPEAT the step says something an earlier step already said + DRIFT the step says something new-sounding that adds no new content + NEITHER a step that actually moves + +The first thing this script does is check *itself*. In that other study the first +run reported a 58 % loop rate, and every one of those repeats was the step +extractor mistaking a code fence for a step and then seeing it again. A +measurement of the harness nearly became a finding about the model, so the +normalisation is audited before any of its numbers are believed. + + .venv/bin/python scripts/measure_loops.py [--questions N] [--out FILE] +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import re +import sys +import time +from datetime import datetime, timezone + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / "src")) +sys.path.insert(0, str(ROOT)) + +import numpy as np # noqa: E402 + +from mpe_lkg import backends # noqa: E402 +from mpe_lkg.reasoning import reason # noqa: E402 + +# Four kinds, because a loop rate averaged over one kind says nothing about the +# others. The malformed group is where the reported transcript came from. +QUESTIONS = { + "simple": [ + "What is the capital of France?", + "How many days are in a leap year?", + "What is the chemical symbol for gold?", + "Who wrote the play Hamlet?", + "What is the largest ocean on Earth?", + "In what year did the Berlin Wall fall?", + ], + "multi_step": [ + "If a train travels 240 km in 3 hours, how long does it take to travel 400 km?", + "What is 17 percent of 250?", + "A rectangle is twice as long as it is wide and has a perimeter of 36 cm. What are its sides?", + "Which is larger, the area of a circle of radius 3 or a square of side 5?", + "If I buy 3 items at 4.50 each and pay with a 20 note, what is my change?", + "How many minutes are there in a fortnight?", + ], + "malformed": [ + "What is the capital of Oslo?", + "When did Napoleon invade Australia?", + "What is the population of the Atlantic Ocean?", + "Who was the first person to walk on the Sun?", + "What language do they speak in Antarctica's capital?", + "How tall is the colour blue?", + ], + "ambiguous": [ + "What is the best programming language?", + "Is a hot dog a sandwich?", + "What is the meaning of the word 'set'?", + "How long is a piece of string?", + "Which came first, the chicken or the egg?", + "What is the biggest city?", + ], +} + +# noqa: SIM905 -- a 70-word list is readable as prose and unreadable as a literal. +STOPWORDS = frozenset( + """a an and are as at be but by can could do does for from had has have how i if in into is it + its may might must not of on or should so than that the their them then there these they this + to was we were what when where which who why will with would you your about also just more most + other some such only own same very don now""".split() # noqa: SIM905 +) + + +def normalise(text: str) -> str: + """Canonical form for exact-duplicate detection: whitespace and case only. + + Deliberately conservative. Every extra transformation is another way to make + two distinct steps look identical, which is the failure this script audits for. + """ + return re.sub(r"\s+", "", text).lower() + + +def content_words(text: str) -> set[str]: + return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOPWORDS and len(w) > 2} + + +def audit_normalisation(steps: list[dict]) -> dict: + """Does the normalisation itself manufacture duplicates? + + Compares duplicate counts on the raw text against the normalised text. A large + gap means the canonical form is collapsing things that differ, and every number + downstream of it is a measurement of this script rather than of the model. + """ + raw = [s["content"] for s in steps] + norm = [normalise(s["content"]) for s in steps] + raw_dupes = len(raw) - len(set(raw)) + norm_dupes = len(norm) - len(set(norm)) + manufactured = norm_dupes - raw_dupes + + empties = sum(1 for n in norm if not n) + shortest = min((len(n) for n in norm), default=0) + return { + "raw_duplicates": raw_dupes, + "normalised_duplicates": norm_dupes, + "manufactured_by_normalisation": manufactured, + "empty_after_normalisation": empties, + "shortest_normalised_length": shortest, + # If normalisation invents duplicates, or empties steps out, the rest of + # this file is about the harness and not about the model. + "trustworthy": manufactured == 0 and empties == 0 and shortest > 20, + } + + +# Set from the measured distribution over 24 runs, not guessed. The median step +# already sits at 0.846 cosine to some earlier step and the 75th percentile at +# 0.900, so a 0.90 bar flags the more-similar quarter of ordinary steps rather +# than duplicates -- inspection confirmed it: "we need to calculate a percentage" +# followed by "the formula is (17/100)*250" scored 0.9013 and is plain progress. +# At 0.96 the flags are restatements on inspection. The title bar is insensitive: +# exactly the same 3 steps fire anywhere from 0.95 to 0.999, so nothing borders it. +RESTATEMENT_THRESHOLD = 0.96 +REPEATED_MOVE_THRESHOLD = 0.95 + + +def classify(steps: list[dict], title_vecs, content_vecs, *, + move_thr=REPEATED_MOVE_THRESHOLD, restate_thr=RESTATEMENT_THRESHOLD) -> list[dict]: + """Label every step REPEAT / DRIFT / NEITHER, with the reason.""" + seen_norm: dict[str, int] = {} + verdicts = [] + + for i, step in enumerate(steps): + reasons = [] + key = normalise(step["content"]) + if key in seen_norm: + reasons.append(("exact", seen_norm[key], 1.0)) + seen_norm.setdefault(key, i) + + if i: + tsim = title_vecs[i] @ title_vecs[:i].T + csim = content_vecs[i] @ content_vecs[:i].T + if float(tsim.max()) >= move_thr: + reasons.append(("move", int(tsim.argmax()), float(tsim.max()))) + if float(csim.max()) >= restate_thr: + reasons.append(("restatement", int(csim.argmax()), float(csim.max()))) + + # Drift: new words, no new subject matter. Measured against everything said + # before, including the question. + earlier = set().union(*(content_words(s["content"]) for s in steps[:i])) if i else set() + earlier |= content_words(step.get("question", "")) + mine = content_words(step["content"]) + novel = mine - earlier + novel_ratio = len(novel) / max(len(mine), 1) + + kind = "repeat" if reasons else ("drift" if novel_ratio <= 0.15 else "neither") + verdicts.append({ + "index": i, + "kind": kind, + "reasons": reasons, + "novel_ratio": round(novel_ratio, 4), + "novel_words": len(novel), + }) + return verdicts + + +def mutual_knn_components(vectors: np.ndarray, k: int) -> dict: + """Component sizes of the mutual-kNN graph. + + If this percolates -- one component swallowing nearly everything as k rises -- + then "the same" is not a threshold that can be chosen, and the detector has to + be an exact canonical form instead of a similarity. + """ + n = len(vectors) + if n <= k: + return {"k": k, "n": n, "components": 0, "largest": n, "note": "too few points"} + + sims = vectors @ vectors.T + np.fill_diagonal(sims, -np.inf) + neighbours = [set(np.argsort(-sims[i])[:k].tolist()) for i in range(n)] + + parent = list(range(n)) + + def find(x): + while parent[x] != x: + parent[x] = parent[parent[x]] + x = parent[x] + return x + + for i in range(n): + for j in neighbours[i]: + if i in neighbours[j]: # mutual, not one-way + a, b = find(i), find(j) + if a != b: + parent[a] = b + + sizes: dict[int, int] = {} + for i in range(n): + sizes[find(i)] = sizes.get(find(i), 0) + 1 + counts = sorted(sizes.values(), reverse=True) + return { + "k": k, + "n": n, + "components": len(counts), + "largest": counts[0], + "largest_share": round(counts[0] / n, 4), + "singletons": sum(1 for c in counts if c == 1), + } + + +def run_one(question: str, chat, embedder) -> dict: + started = time.time() + steps, error = [], None + for event in reason(question, chat=chat, embedder=embedder): + if event["type"] == "step": + steps.append({"title": event["title"], "content": event["content"], "question": question}) + elif event["type"] == "final": + steps.append({"title": "Final Answer", "content": event["content"], "question": question}) + elif event["type"] == "error": + error = event["message"] + return {"question": question, "steps": steps, "error": error, "seconds": round(time.time() - started, 2)} + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--questions", type=int, default=0, help="cap the number of questions (0 = all)") + parser.add_argument("--out", default="docs/claims/loops.json") + parser.add_argument( + "--reanalyse", action="store_true", + help="recompute from the saved transcripts instead of asking the model again", + ) + args = parser.parse_args() + + out = ROOT / args.out + transcripts = out.with_name("loops_transcripts.json") + embedder = backends.OllamaEmbedding("") + + if args.reanalyse: + # The raw steps are committed, so a threshold can be revisited without + # spending another twenty minutes of model time -- and so anyone can check + # these numbers rather than taking them. + if not transcripts.exists(): + print(f"{transcripts} does not exist; run without --reanalyse first.") + return 1 + runs = json.loads(transcripts.read_text()) + chat_model = "(from saved transcripts)" + print(f"re-analysing {len(runs)} saved runs at " + f"content>={RESTATEMENT_THRESHOLD} title>={REPEATED_MOVE_THRESHOLD}\n") + else: + chat_model = backends.pick_chat_model() + if not backends.chat_models(): + print("No chat model installed. Try: ollama pull llama3.2:3b") + return 1 + + chat = backends.OllamaChat(chat_model) + flat = [(group, q) for group, qs in QUESTIONS.items() for q in qs] + if args.questions: + flat = flat[: args.questions] + + print(f"chat={chat_model} embeddings={embedder.model} questions={len(flat)}\n") + runs = [] + for n, (group, question) in enumerate(flat, 1): + print(f" [{n}/{len(flat)}] {group:<11} {question[:56]}", flush=True) + run = run_one(question, chat, embedder) + run["group"] = group + runs.append(run) + if run["error"]: + print(f" error: {run['error'][:70]}") + + all_steps = [s for r in runs for s in r["steps"]] + if not all_steps: + print("No steps produced; nothing to measure.") + return 1 + + print("\naudit: is the normalisation itself creating duplicates?") + audit = audit_normalisation(all_steps) + for key, value in audit.items(): + print(f" {key:<34} {value}") + if not audit["trustworthy"]: + print("\n THE NORMALISATION IS SUSPECT. Every number below measures this script,") + print(" not the model. Fix the extraction before believing any of it.") + + title_vecs = embedder.embed([s["title"] for s in all_steps]) + content_vecs = embedder.embed([s["content"] for s in all_steps]) + + verdicts, offset = [], 0 + for run in runs: + n = len(run["steps"]) + if n: + verdicts.extend(classify( + run["steps"], title_vecs[offset:offset + n], content_vecs[offset:offset + n] + )) + offset += n + + total = len(verdicts) + repeats = [v for v in verdicts if v["kind"] == "repeat"] + + # How often a *question* loops, which is what a user experiences. Repeats + # cluster -- one run here produced four -- so the per-step rate understates it. + runs_with_repeat, cursor = 0, 0 + for run in runs: + n = len(run["steps"]) + runs_with_repeat += any(v["kind"] == "repeat" for v in verdicts[cursor:cursor + n]) + cursor += n + drifts = [v for v in verdicts if v["kind"] == "drift"] + by_reason: dict[str, int] = {} + for v in repeats: + for kind, _, _ in v["reasons"]: + by_reason[kind] = by_reason.get(kind, 0) + 1 + + print(f"\nfailure modes over {total} steps in {len(runs)} runs") + print(f" repeat {len(repeats):>4} ({len(repeats)/total:.1%}) by detector: {by_reason}") + print(f" drift {len(drifts):>4} ({len(drifts)/total:.1%})") + print(f" neither {total-len(repeats)-len(drifts):>4} ({(total-len(repeats)-len(drifts))/total:.1%})") + + print("\nper group") + per_group = {} + idx = 0 + for run in runs: + n = len(run["steps"]) + group_v = verdicts[idx:idx + n] + bucket = per_group.setdefault(run["group"], {"steps": 0, "repeat": 0, "drift": 0, "runs": 0}) + bucket["runs"] += 1 + bucket["steps"] += n + bucket["repeat"] += sum(1 for v in group_v if v["kind"] == "repeat") + bucket["drift"] += sum(1 for v in group_v if v["kind"] == "drift") + idx += n + for group, b in per_group.items(): + avg = b["steps"] / max(b["runs"], 1) + print(f" {group:<11} {b['steps']:>3} steps ({avg:.1f}/run) " + f"repeat {b['repeat']:>3} drift {b['drift']:>3}") + + print("\nmutual-kNN over all steps: does 'the same' percolate?") + knn = [mutual_knn_components(content_vecs, k) for k in (2, 3, 4, 6)] + for row in knn: + print(f" k={row['k']} components {row['components']:>4} " + f"largest {row['largest']:>4} ({row.get('largest_share', 0):.1%})") + + payload = { + "provenance": "measured", + "measured_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "chat_model": chat_model, + "embedding_model": embedder.model, + "runs": len(runs), + "steps": total, + "harness_audit": audit, + "repeat_steps": len(repeats), + "repeat_rate": round(len(repeats) / total, 4), + "repeat_by_detector": by_reason, + "drift_steps": len(drifts), + "drift_rate": round(len(drifts) / total, 4), + "clean_rate": round((total - len(repeats) - len(drifts)) / total, 4), + "per_group": per_group, + "mutual_knn": knn, + "mean_steps_per_run": round(total / len(runs), 2), + "runs_with_a_repeat": runs_with_repeat, + "runs_with_a_repeat_rate": round(runs_with_repeat / len(runs), 4), + "thresholds": {"restatement": RESTATEMENT_THRESHOLD, "repeated_move": REPEATED_MOVE_THRESHOLD}, + "errors": sum(1 for r in runs if r["error"]), + } + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(payload, indent=2) + "\n") + print(f"\nwrote {out.relative_to(ROOT)}") + + if not args.reanalyse: + transcripts.write_text(json.dumps(runs, indent=2) + "\n") + print(f"wrote {transcripts.relative_to(ROOT)} (the raw steps, so this is re-checkable)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/mpe_lkg/app.py b/src/mpe_lkg/app.py index cfa0c13..67e56d4 100644 --- a/src/mpe_lkg/app.py +++ b/src/mpe_lkg/app.py @@ -266,13 +266,55 @@ def find_free_port(host: str, first: int, window: int = PORT_SEARCH_WINDOW) -> i ) +def banner(status: dict) -> str: + """What the app found, and what to do about it. + + Written so the README does not have to explain the setup: the program that + knows what is installed is the one best placed to say what is missing. + """ + lines = ["", " Local Knowledge Graph"] + + if not status["models"]: + lines += [ + "", + f" Ollama did not answer at {status['base_url']}.", + "", + " This app runs a language model on your own machine through Ollama.", + " 1. Install it from https://ollama.com", + " 2. Pull a model: ollama pull llama3.2:3b", + " 3. Start this again.", + "", + " If Ollama runs elsewhere, set OLLAMA_URL to point at it.", + ] + return "\n".join(lines) + "\n" + + if not status["ok"]: + lines += [ + "", + f" Ollama is running at {status['base_url']}, with: {', '.join(status['models'])}", + f" {status['problem']}", + "", + " The page will offer to download one, or:", + " ollama pull llama3.2:3b", + ] + return "\n".join(lines) + "\n" + + chat = status.get("chat_model") or "?" + embed = status.get("embedding_model") or f"{chat} (no embedding model installed)" + lines += [ + "", + f" chat model {chat}", + f" embeddings {embed}", + f" ollama {status['base_url']}", + "", + " Both are changeable in the page. 'mpe-lkg doctor' reports this without starting.", + ] + return "\n".join(lines) + "\n" + + def run(host: str | None = None, port: int | None = None, debug: bool | None = None) -> None: """Start the server, after saying whether the model backend is actually there.""" - status = backends.health(backends.DEFAULT_BASE_URL) - if not status["ok"]: - print(f"\n {status['problem']}\n {status['hint']}\n") - else: - print(f"\n Ollama at {status['base_url']} — models: {', '.join(status['models'])}\n") + print(banner(backends.health(backends.DEFAULT_BASE_URL))) host = host or os.environ.get("LKG_HOST", "127.0.0.1") wanted = port or int(os.environ.get("LKG_PORT", "5100"))