Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,10 @@
# Gemini: LLM_API_KEY=AIza...
# DeepSeek: LLM_API_KEY=sk-...
LLM_API_KEY=your-key-here

# Optional base URL override for OpenAI-compatible gateways
# OPENAI_API_BASE=https://api.deepseek.com

# --- REST API server variables (read by `openkb-api` / `python -m openkb.api`) ---
# OPENKB_API_TOKEN=... # required bearer token clients send as Authorization: Bearer <token>
# OPENKB_KB_ROOT=... # optional; where REST /init creates KBs (default: ~/.config/openkb/kbs)
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@ raw/
wiki/
.openkb/
output/
kbs/

# Local only
docs/
.claude/

# Built Workbench web bundle (regenerate with `cd frontend && npm install && npm run build`)
web/
102 changes: 60 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

# OpenKB: Open LLM Knowledge Base

<p align="center"><i>Scale to long documents&nbsp; • &nbsp;Reasoning-based retrieval&nbsp; • &nbsp;Native multi-modality&nbsp; • &nbsp;No Vector DB</i></p>
<p align="center"><i>Scale to long documentsReasoning-based retrievalNative multi-modalityNo Vector DB</i></p>

</div>

Expand Down Expand Up @@ -49,6 +49,7 @@ OpenKB has two layers: a **wiki foundation** that compiles and maintains your kn
- **Skill Factory:** Distills redistributable agent skills from your wiki.
- **OKF-ready:** Wiki pages follow the [Google OKF](https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing) specification for knowledge sharing.
- **Obsidian-compatible:** The wiki is plain `.md` files with cross-links. Opens in Obsidian for graph view.
- **Knowledge Workbench (Web UI):** A bundled React SPA served at `/` turns the REST API into a full dark-themed three-pane workbench — browse stats, upload & compile documents, stream queries/chats with a live reasoning timeline, and run maintenance, all in the browser. No separate frontend server needed.

# 🚀 Getting Started

Expand All @@ -62,13 +63,13 @@ pip install openkb
<summary><b><i>Other install options:</i></b></summary>

- **Latest from GitHub:**

```bash
pip install git+https://github.com/VectifyAI/OpenKB.git
```

- **Install from source** (editable, for development):

```bash
git clone https://github.com/VectifyAI/OpenKB.git
cd OpenKB
Expand Down Expand Up @@ -115,6 +116,17 @@ Create a `.env` file with your LLM API key:
LLM_API_KEY=your_llm_api_key
```

### Knowledge Workbench (Web UI)

OpenKB ships a bundled React SPA served by the REST API at `/`. Build it with `cd frontend && npm install && npm run build`, then start the server:

```bash
pip install -e ".[api]"
OPENKB_API_TOKEN=test-token python -m openkb.api --host 127.0.0.1 --port 8000
```

Open `http://127.0.0.1:8000/` for the Workbench, or see the [full Web UI guide](examples/rest-api/README.md#knowledge-workbench-web-ui).

# 🧩 How OpenKB Works

### Architecture
Expand All @@ -125,12 +137,12 @@ LLM_API_KEY=your_llm_api_key

### Short vs Long Document Handling

| | Short documents | Long documents (PDF ≥ 20 pages) |
|---|---|---|
| **Convert** | markitdown → Markdown | PageIndex → tree index + summaries |
| **Images** | Extracted inline (pymupdf) | Extracted by PageIndex |
| **LLM reads** | Full text | Document trees |
| **Result** | summary + concepts | summary + concepts |
| | Short documents | Long documents (PDF ≥ 20 pages) |
| ------------- | -------------------------- | ---------------------------------- |
| **Convert** | markitdown → Markdown | PageIndex → tree index + summaries |
| **Images** | Extracted inline (pymupdf) | Extracted by PageIndex |
| **LLM reads** | Full text | Document trees |
| **Result** | summary + concepts | summary + concepts |

Short documents are read in full by the LLM. Long PDFs are processed by [PageIndex](https://github.com/VectifyAI/PageIndex) into a hierarchical tree index. The LLM reads the tree instead of the full text, enabling accurate and scalable retrieval for long documents.

Expand All @@ -152,24 +164,24 @@ OpenKB commands fall into two layers: the **wiki foundation** (compile + manage

## Layer 1: 🧱 Wiki Foundation — compile and maintain

| Command | Description |
|---|---|
| `openkb init` | Initialize a new knowledge base (interactive) |
| Command | Description |
| ------------------------------------------------------------ | --------------------------------------------------------------------------------------- |
| `openkb init` | Initialize a new knowledge base (interactive) |
| <code>openkb&nbsp;add&nbsp;&lt;file_or_dir_or_URL&gt;</code> | Add files, directories, or URLs and compile to wiki (URL content type is auto-detected) |
| `openkb list` | List indexed documents and concepts |
| `openkb status` | Show knowledge base stats |
| `openkb watch` | Watch `raw/` and auto-compile new files |
| `openkb lint` | Run structural and knowledge health checks |
| `openkb list` | List indexed documents and concepts |
| `openkb status` | Show knowledge base stats |
| `openkb watch` | Watch `raw/` and auto-compile new files |
| `openkb lint` | Run structural and knowledge health checks |

<details>
<summary><i>More wiki commands:</i></summary>
<br>

| Command | Description |
|---|---|
| <code>openkb&nbsp;remove&nbsp;&lt;doc&gt;</code> | Remove a document and clean up its wiki pages, images, registry, and PageIndex state (`--dry-run` to preview, `--keep-raw` / `--keep-empty` to retain artifacts) |
| Command | Description |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <code>openkb&nbsp;remove&nbsp;&lt;doc&gt;</code> | Remove a document and clean up its wiki pages, images, registry, and PageIndex state (`--dry-run` to preview, `--keep-raw` / `--keep-empty` to retain artifacts) |
| <code>openkb&nbsp;recompile&nbsp;[&lt;doc&gt;]&nbsp;[--all]</code> | Re-run the compile pipeline on already-indexed docs without re-indexing. Regenerates summaries and rewrites concept pages; manual edits are overwritten (`--dry-run` to preview, `--refresh-schema` to also update `wiki/AGENTS.md`) |
| <code>openkb&nbsp;feedback&nbsp;["msg"]</code> | File feedback by opening a prefilled GitHub issue (`--type bug/feature/question` to tag it) |
| <code>openkb&nbsp;feedback&nbsp;["msg"]</code> | File feedback by opening a prefilled GitHub issue (`--type bug/feature/question` to tag it) |

</details>

Expand All @@ -179,23 +191,23 @@ OpenKB commands fall into two layers: the **wiki foundation** (compile + manage

A "generator" reads from the compiled wiki and produces something usable: an answer, a conversation, a skill folder. The wiki is the substrate; generators are the surfaces.

| Command | Output | Example |
|---|---|---|
| <code>openkb&nbsp;query&nbsp;"question"</code> | A grounded answer with citations (`--save` to persist to `wiki/explorations/`) | [query & save](examples/commands/) |
| <code>openkb&nbsp;chat</code> | Interactive multi-turn session over the wiki (`--resume`, `--list`, `--delete` to manage sessions) | [chat](examples/chat/) |
| <code>openkb&nbsp;visualize</code> | A self-contained interactive knowledge graph at `output/visualize/graph.html` — 3D, mind-map, and radial views | [visualize](examples/visualize/) |
| <code>openkb&nbsp;skill&nbsp;new&nbsp;&lt;skill-name&gt;&nbsp;"&lt;intent&gt;"</code> | Distill a redistributable agent skill from your wiki (see [Skill Factory](#skill-factory) below) | [skills](examples/skills/) |
| <code>openkb&nbsp;deck&nbsp;new&nbsp;&lt;name&gt;&nbsp;"&lt;intent&gt;"</code> | Generate a single-file HTML slide deck (`--skill` picks a theme, `--critique` runs a quality pass) | [slides](examples/slides/) |
| Command | Output | Example |
| ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| <code>openkb&nbsp;query&nbsp;"question"</code> | A grounded answer with citations (`--save` to persist to `wiki/explorations/`) | [query & save](examples/commands/) |
| <code>openkb&nbsp;chat</code> | Interactive multi-turn session over the wiki (`--resume`, `--list`, `--delete` to manage sessions) | [chat](examples/chat/) |
| <code>openkb&nbsp;visualize</code> | A self-contained interactive knowledge graph at `output/visualize/graph.html` — 3D, mind-map, and radial views | [visualize](examples/visualize/) |
| <code>openkb&nbsp;skill&nbsp;new&nbsp;&lt;skill-name&gt;&nbsp;"&lt;intent&gt;"</code> | Distill a redistributable agent skill from your wiki (see [Skill Factory](#skill-factory) below) | [skills](examples/skills/) |
| <code>openkb&nbsp;deck&nbsp;new&nbsp;&lt;name&gt;&nbsp;"&lt;intent&gt;"</code> | Generate a single-file HTML slide deck (`--skill` picks a theme, `--critique` runs a quality pass) | [slides](examples/slides/) |

<details>
<summary><i>More skill commands:</i></summary>
<br>

| Command | Output |
|---|---|
| <code>openkb&nbsp;skill&nbsp;validate&nbsp;[name]</code> | Validate compiled skills (auto-runs after `skill new`) |
| <code>openkb&nbsp;skill&nbsp;eval&nbsp;&lt;name&gt;</code> | Check the skill triggers on the right prompts |
| <code>openkb&nbsp;skill&nbsp;history&nbsp;&lt;name&gt;</code> / <code>openkb&nbsp;skill&nbsp;rollback&nbsp;&lt;name&gt;</code> | Version history + rollback for skills |
| Command | Output |
| ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| <code>openkb&nbsp;skill&nbsp;validate&nbsp;[name]</code> | Validate compiled skills (auto-runs after `skill new`) |
| <code>openkb&nbsp;skill&nbsp;eval&nbsp;&lt;name&gt;</code> | Check the skill triggers on the right prompts |
| <code>openkb&nbsp;skill&nbsp;history&nbsp;&lt;name&gt;</code> / <code>openkb&nbsp;skill&nbsp;rollback&nbsp;&lt;name&gt;</code> | Version history + rollback for skills |

</details>

Expand Down Expand Up @@ -299,19 +311,25 @@ gemini skills install https://github.com/VectifyAI/OpenKB.git --path skills/open

The skill is read-only. It won't run `openkb add`, `remove`, or `lint --fix` without you asking. See [`skills/openkb/SKILL.md`](skills/openkb/SKILL.md) for the full instruction set.

# REST API

OpenKB ships a FastAPI service for HTTP clients. Install with `pip install -e ".[api]"`, then start with `python -m openkb.api`. The interactive API reference is at [`/docs`](http://127.0.0.1:8000/docs) (importable into Postman).

See the [full REST API reference](examples/rest-api/README.md#rest-api) for endpoints, auth, and SSE streaming.

# 🧭 Learn More

### Compared to Karpathy's Approach

| | Karpathy's workflow | OpenKB |
|---|---|---|
| Short documents | LLM reads directly | markitdown → LLM reads |
| Long documents | Context limits, context rot | PageIndex tree index |
| Input sources | Web clipper → .md | PDF, Word, PPT, Excel, HTML, text, CSV, .md, URLs |
| Wiki compilation | LLM agent | LLM agent (same) |
| Entity extraction | Manual | Automatic (people, orgs, places, products) |
| Q&A | Query over wiki | Wiki + PageIndex retrieval |
| Output | Wiki only | Wiki + Skill Factory + agent CLI integration |
| | Karpathy's workflow | OpenKB |
| ----------------- | --------------------------- | ------------------------------------------------- |
| Short documents | LLM reads directly | markitdown → LLM reads |
| Long documents | Context limits, context rot | PageIndex tree index |
| Input sources | Web clipper → .md | PDF, Word, PPT, Excel, HTML, text, CSV, .md, URLs |
| Wiki compilation | LLM agent | LLM agent (same) |
| Entity extraction | Manual | Automatic (people, orgs, places, products) |
| Q&A | Query over wiki | Wiki + PageIndex retrieval |
| Output | Wiki only | Wiki + Skill Factory + agent CLI integration |

### The Stack

Expand All @@ -328,7 +346,7 @@ The skill is read-only. It won't run `openkb add`, `remove`, or `lint --fix` wit
- [ ] Scale to large document collections with nested folder support
- [ ] Hierarchical concept (topic) indexing for massive knowledge bases
- [ ] Database-backed storage engine
- [ ] Web UI for browsing and managing wikis
- [x] Web UI for browsing and managing wikis (Knowledge Workbench, served at `/`)

### Contributing

Expand Down
Loading