Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,9 @@ __pycache__/

# Seeded design canvases: build products of design/*.dc.html.
design/build/

# `scripts/social_card.py` output, the repo's social preview. It is uploaded by hand
# (Settings ▸ General ▸ Social preview), GitHub keeps its own copy, and the script
# rebuilds it from docs/flow.gif in about half a minute. Ignored here, it also stays out
# of the sdist, which it alone would put over its size guard.
docs/social-preview.gif
230 changes: 139 additions & 91 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,140 +1,188 @@
# Flow
<p align="center">
<img src="flow/assets/flow.svg" width="88" alt="">
</p>

<h1 align="center">Flow</h1>

<p align="center">
Local voice dictation for Windows, made for developers who speak English with an accent.
</p>

<p align="center">
<a href="https://github.com/samartomar/flow/releases/latest"><img alt="Latest release" src="https://img.shields.io/github/v/release/samartomar/flow"></a>
<img alt="Windows 10 and 11" src="https://img.shields.io/badge/Windows-10%20%7C%2011-0078d4">
<a href="#macos-and-linux"><img alt="macOS and Linux: Lite, in progress" src="https://img.shields.io/badge/macOS%20%7C%20Linux-Lite%2C%20in%20progress-e8a33d"></a>
<a href="https://github.com/samartomar/flow/actions/workflows/ci.yml"><img alt="Tests" src="https://github.com/samartomar/flow/actions/workflows/ci.yml/badge.svg?branch=main"></a>
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/github/license/samartomar/flow"></a>
</p>

<p align="center">
<a href="#install">Install</a> ·
<a href="docs/guide.md">Guide</a> ·
<a href="https://github.com/samartomar/flow/issues/new">Report a mishearing</a>
</p>

Hold a key, talk, let go: the words paste into the window you were working in. Or tap the
pill to hand them to the agent CLI you already use — shaped into a prompt for your
project, or asked as a question and answered above the pill.

Speech recognition runs on your machine. No API key.
![Flow Home page by page: Home, History, Voice, then a question dictated with the pill into Conversations and answered by the agent, then Models and Settings](docs/flow.gif)

**English only · three dependencies · Windows in full, macOS and Linux in
[Lite](#flow-lite-macos-and-linux)**
Hold <kbd>Ctrl</kbd>+<kbd>Win</kbd>, say what you want to write, and let go. Flow pastes the
words wherever you were typing: your editor, a terminal, Slack, a browser tab. If it heard
something wrong, hold the keys again and say *"change Tuesday to Thursday"* or *"scratch
that"*, and it fixes the text it just pasted.

![Flow Home page by page: Home, History, Voice, then a question dictated with the pill into Conversations and answered by the agent, then Models and Settings](docs/flow.gif)
Flow sits in a small pill at the bottom of the screen, and the pill has two more modes. In
Refine, what you say is rewritten into a proper prompt for your project. In Ask, your question goes to the agent
CLI you already use (`codex` or `claude`) and the answer is read back to you.

*Flow Home and the real pill against a scripted session, recorded by [`scripts/home_reel.py`](scripts/home_reel.py).*
I made Flow for people whose English has an accent that speech recognition keeps tripping
over, often enough that they give up and type. Spanish, Indian, Russian and Japanese
speakers are who it's designed and benchmarked for. Recognition runs on your own machine,
there's no account or API key, and your audio never leaves your PC.

## Install

```bash
uv tool install git+https://github.com/samartomar/flow
flow
```
### Windows

You need Windows 10 or 11 and a microphone. Download
[`flow-windows-x64.zip`](https://github.com/samartomar/flow/releases/latest/download/flow-windows-x64.zip),
unzip it anywhere and run `flow.exe`. No Python required.

You need [`uv`](https://docs.astral.sh/uv/), which fetches Python and the three
dependencies itself, and a microphone. No Python? On Windows, [Scoop](https://scoop.sh)
installs the release build:
> [!NOTE]
> The zip isn't code-signed, so the first launch shows a SmartScreen warning. Click
> **More info**, then **Run anyway**.

Or get the same build through [Scoop](https://scoop.sh), which makes updating one command:

```powershell
scoop bucket add flow https://github.com/samartomar/scoop-flow
scoop install flow/flow # flow/ matters: Scoop's main bucket has a different "flow"
```

Or download
[`flow-windows-x64.zip`](https://github.com/samartomar/flow/releases/latest/download/flow-windows-x64.zip),
unzip it and run `flow.exe`. It is unsigned, so the first launch shows SmartScreen:
**More info → Run anyway**.
If you already use [uv](https://docs.astral.sh/uv/), you can install from source instead.
It fetches Python and Flow's three dependencies (faster-whisper, sounddevice and numpy):

An agent CLI — `codex` or `claude` on PATH, signed in — is optional. It adds Refine, Ask
and rewrites; dictation and spoken corrections work without it.
[The guide](docs/guide.md#install) has the rest.
```bash
uv tool install git+https://github.com/samartomar/flow
flow
```

The speech models aren't part of any of these. Flow downloads them on first run and shows
you the progress.

### macOS and Linux

## Flow Lite (macOS and Linux)
Flow Lite, the version for macOS and Linux, is still in progress. Install it with uv, as
above. It has the same recognition, corrections and agent features, but it can't paste into
other apps or listen for global hotkeys. You hold the pill to talk, and Send copies the
text for you to paste. The microphone is the only permission it needs. There's no Mac or
Linux download yet. [More about Lite](docs/product.md#flow-lite--the-portable-body).

Off Windows, Flow runs as **Lite**: the same recognition, corrections and agent loop, but
Send copies the draft and you paste it yourself. No injection and no global hotkeys, so
there is nothing to grant but the microphone — hold the pill to talk. There is no Mac or
Linux download; install with uv. [What Lite is, and is not](docs/product.md#flow-lite--the-portable-body).
### The agent CLI (optional)

## The loop
Refine and Ask need `codex` or `claude` on your PATH, already signed in. Dictation and
spoken corrections work without either. [The guide](docs/guide.md#install) has the details.

1. **Hold `ctrl+win`** (or the pill) and talk.
2. **Let go.** The words paste where you were typing.
3. **Said it wrong?** Hold again and say *"change Tuesday to Thursday"* or *"scratch
that"*. Flow fixes its last paste until you type or click there.
4. **Wrong window?** Click the right one and press **Paste last** (`alt+shift+Z`).
5. **Want an answer?** Hold **`ctrl+alt+win`** and ask.
## Using it

On first run, Flow Home walks you through the microphone, the model download, a
45-second tuning to your voice, and whether to keep a history. Every step can be skipped.
1. Hold <kbd>Ctrl</kbd>+<kbd>Win</kbd> (or hold the pill) and talk.
2. Let go, and the text is pasted where your cursor was.
3. Heard you wrong? Hold again and say *"change Tuesday to Thursday"* or *"scratch that"*.
Flow can fix its last paste until you type or click there.
4. Pasted into the wrong window? Click the right one and press
<kbd>Alt</kbd>+<kbd>Shift</kbd>+<kbd>Z</kbd>.
5. Want an answer rather than text? Hold <kbd>Ctrl</kbd>+<kbd>Alt</kbd>+<kbd>Win</kbd> and
ask.

## Three modes, one tap
Those last two are Wispr Flow's keys on Windows as well, so if you've used it, your hands
already know them.

- **Type** (white): speak, let go, it pastes.
- **Refine** (gold): `codex` or `claude` turns what you said into a prompt for your
project. Nothing pastes until you press Send.
- **Ask** (violet): the question goes to your agent CLI, and the answer appears above the
pill and is read aloud. **Continue in Flow** opens the conversation in a window.
The first run walks you through the microphone, the model download, a 45-second tuning to
your voice, and whether to keep a history. You can skip any step.

Point Refine and Ask at a project (Flow Home ▸ Settings ▸ Workspaces, or `--cwd`) and the
answers are about your code. The Classic pill — a draft you correct by voice before
sending — is one switch away in Settings.
## Type, Refine, Ask

Tap the pill to cycle through its three modes. Its colour tells you which one it's in.

| Mode | Pill | When you let go |
|:---|:---|:---|
| Type | white | The words are pasted. |
| Refine | gold | `codex` or `claude` turns them into a prompt for your project. Nothing is pasted until you press Send. |
| Ask | violet | The question goes to your agent CLI. The answer appears above the pill and is read aloud, and **Continue in Flow** opens the conversation in a window. |

Point Refine and Ask at a project folder (Flow Home ▸ Settings ▸ Workspaces, or `--cwd`)
and the answers are about your code. If you'd rather correct a draft by voice before
anything is sent, switch to the Classic pill in Settings.

## Flow Home

Right-click the pill ▸ **Open Flow**, or run `flow --home`: one window for everything that
isn't talking. Speech models download with progress and swap without a restart; the
microphone, shortcuts, workspaces, agent CLI and reading voice are all set there.
**Voice** tunes Flow to you, **History** shows what you dictated (if you keep one), and
**Conversations** is Ask in a window. The pill never grows a setting.
Everything that isn't talking happens in one window, the one in the GIF above. Right-click
the pill and choose **Open Flow**, or run `flow --home`. Speech models download there with
a progress bar and swap in without a restart, and it's where you set the microphone,
shortcuts, workspaces, agent CLI and the voice that reads answers. Voice tunes Flow to how
you speak, History shows what you've dictated if you chose to keep it, and Conversations
is Ask in a full window.

## Who it is for
## Privacy

Developers who speak English as a second language, with a strong accent. Spanish, Indian,
Russian and Japanese L1 speakers anchor the design and the benchmarks
([why](docs/product.md)).
By default nothing leaves your PC. Audio, recognition, corrections, your word list and your
voice profile all stay local, and no API key is read, stored or passed anywhere in the code.

## What leaves your machine
A few things do go out, and only when you use them:

- **Nothing, by default.** Audio, recognition, corrections, your lexicon and your profile
stay local. No API key is read, stored or passed anywhere in this codebase.
- **Whatever you hand an agent CLI** goes to its cloud: the draft or the question, plus
your workspace path if you set one.
- **History** is kept only if you choose to, in `~/.flow/history.jsonl`.
- **Pasting** goes through the Windows clipboard for a moment, where clipboard managers
- A draft you refine or a question you ask goes to that agent CLI's cloud, along with your
workspace path if you set one.
- Pasting puts the text on the Windows clipboard for a moment, where clipboard managers
and cloud sync can see it.
- **A Microsoft voice**, if you choose one, sends each spoken answer's text to Microsoft.
Every other voice is local.
- If you choose one of Microsoft's natural voices to read answers, the text of each answer
is sent to Microsoft. Every other voice runs locally.

Flow Home is served on `127.0.0.1`, only to the window Flow opens, with a new token each
launch. [The exact boundary](docs/architecture.md#what-leaves-the-machine).
History is kept only if you say so, in `~/.flow/history.jsonl`. Flow Home is served on
`127.0.0.1` to the window Flow opens and nothing else, with a new token every launch.
[The exact boundary](docs/architecture.md#what-leaves-the-machine) is in the architecture
doc.

## Known limits

- **Corrections must be phrased as commands.** *"delete the bit about the standup"* works;
*"I feel it should not contain the summary"* is typed as words.
- **A personal lexicon cuts both ways.** It recovers 27–34% of the rare words the model
missed, and makes WER 14–38% relatively worse on speech without them (EdAcc,
`small.en`). Add the words you say often.
- **Live text refreshes about once a second**, is shown dimmed, and is always replaced by
the final text.
- **Accuracy depends on the voice.** Flow Home ▸ Voice ▸ **How well Flow hears you**
measures yours in five sentences.
- Flow does English only.
- Corrections have to be phrased as commands. *"delete the bit about the standup"* works;
*"I feel it should not contain the summary"* gets typed as words.
- Your personal word list cuts both ways. It recovers 27–34% of the rare words the model
missed, but raises the error rate by 14–38% on speech that doesn't contain them
(measured on EdAcc with `small.en`). Add the words you say often, not every word you know.
- The live preview refreshes about once a second and is shown dimmed. The final text
always replaces it.
- How accurate it is depends on your voice. Flow Home ▸ Voice ▸ **How well Flow hears you**
measures yours with five sentences.

All of them are in [the guide](docs/guide.md#known-limitations).
The rest are in [the guide](docs/guide.md#known-limitations).

## Docs

| | |
|---|---|
| [guide](docs/guide.md) | the manual: every flag, spoken command, mode and setting |
| [product](docs/product.md) | who Flow is for, the P1–P9 requirements, the non-goals |
| [architecture](docs/architecture.md) | data flow, threads, events, and the tuned constants with their measurements |
| [roadmap](docs/roadmap.md) | what is left, and the accent benchmark that measures it |
| [development](docs/development.md) | layout, tests, harnesses, and building a release |
| [decisions](docs/decisions.md) | why things are the way they are |
[The guide](docs/guide.md) is the manual: every flag, spoken command, mode and setting.
[product.md](docs/product.md) says who Flow is for and what it deliberately won't do,
[architecture.md](docs/architecture.md) how it fits together and the measurements behind
its tuned numbers, [roadmap.md](docs/roadmap.md) what's left and the accent benchmark that
tracks it, [development.md](docs/development.md) how to work on it and cut a release, and
[decisions.md](docs/decisions.md) why things are the way they are.

## Contributing

Especially welcome: anyone who speaks accented English and can tell me where Flow
mishears them. That is the one thing I cannot measure alone.
The most useful thing you can send me is a mishearing. If you speak English with an
accent, [open an issue](https://github.com/samartomar/flow/issues/new) with what you said
and what Flow typed. That's the one thing I can't measure on my own.

Pull requests are welcome for anything else. To run it from a clone:

```bash
git clone https://github.com/samartomar/flow && cd flow
uv sync && uv run flow # run it
uv run python -m unittest discover -s tests # ~3,200 tests, ~90 s, no mic needed
uv run python -m unittest discover -s tests # ~3,200 tests, about 90 s, no mic needed
```

Open an [issue](https://github.com/samartomar/flow/issues) for a mishearing (what you
said, and what appeared) or a pull request for anything else.
The GIF at the top is recorded by [`scripts/home_reel.py`](scripts/home_reel.py) from a
scripted session, so it can be re-shot whenever the app changes.

## License

MIT licensed.
MIT. Flow is built on [faster-whisper](https://github.com/SYSTRAN/faster-whisper) and
OpenAI's [Whisper](https://github.com/openai/whisper) models, and reads answers aloud with
[Piper](https://github.com/OHF-Voice/piper1-gpl) if you add it.
Binary file modified docs/flow.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ asking and Mac hardware to measure on.
**Off Windows, Flow starts in Lite** rather than refusing. Everything that does not need
hands is there — both decoder tiers, the correction loop, the lexicon, the profile, the
polish, the thread, converse mode — and Send copies the draft for you to paste. See
[README](../README.md#flow-lite-macos-and-linux) for the short version and
[README](../README.md#macos-and-linux) for the short version and
[product.md](product.md#flow-lite--the-portable-body) for the definition and the fence.
`--lite` runs the same code on Windows.

Expand Down
7 changes: 3 additions & 4 deletions flow/home/static/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -360,17 +360,16 @@
<section class="card tight"><div class="row wrap pc">
${icon("chip", onGpu ? C.green : C.muted, 22)}
<div class="col grow">
<div class="row wrap"><b>${esc(gpu ? gpu.name : "No NVIDIA GPU found")}</b>${gpu && gpu.memory_mb ? `<span class="note">${Math.round(gpu.memory_mb / 1024)} GB</span>` : ""}
<div class="row wrap"><b>${esc(gpu ? gpu.name : "No NVIDIA GPU found")}</b>${gpu && gpu.memory_mb ? `<span class="note">${Math.round(gpu.memory_mb / 1024)} GB video memory</span>` : ""}
<span class="badge ${onGpu ? "green" : ""}">${onGpu ? '<span class="dot green"></span>speech runs on the GPU' : "speech runs on the CPU"}</span>
${sp.compute_types.includes("int8") && onGpu && !sp.compute_types.includes("float16") ? '<span class="badge">int8</span>' : ""}</div>
<p class="note">${onGpu ? (sp.compute_types.includes("float16") ? "This card has fast half-precision; Flow runs int8 on it." : "This card has no fast half-precision, so int8 is its fast path. Flow picks it for you.")
: esc(sp.why_cpu || "The CPU runs the smaller models in time; the large ones need a GPU.")}</p>
</div>
<div class="col"><b>${esc(sp.cache.text)}</b><span class="fine">of speech models on disk</span></div>
<button type="button" class="btn" data-act="open" data-what="models">${icon("folder", C.text, 15)}Open folder</button>
</div></section>
<section class="card">
<div class="row"><h2 class="grow">Speech recognition</h2>${sp.loading ? '<span class="badge"><span class="dot blue"></span>loading a model</span>' : ""}</div>
<div class="row wrap"><h2 class="grow">Speech recognition</h2>${sp.loading ? '<span class="badge"><span class="dot blue"></span>loading a model</span>' : ""}
<span class="note">${esc(sp.cache.text)} on this PC</span><button type="button" class="btn sm" data-act="open" data-what="models">${icon("folder", C.text, 14)}Open folder</button></div>
<p class="note">Now: <span class="mono">${esc(finalNow)}</span> for the words that get pasted, <span class="mono">${esc(partialNow)}</span> for the live preview.</p>
${sp.swappable ? `<div class="choose">
<label>Words that get pasted<select id="final-model">${option("final", sp.chosen.final, sp.automatic.final)}</select></label>
Expand Down
9 changes: 7 additions & 2 deletions scripts/home_reel.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@
uv run --no-sync --with pillow --with playwright python scripts/home_reel.py --frames out/

Playwright drives the Edge that Windows ships (`channel="msedge"`), so it downloads no
browser of its own.
browser of its own. After a new take, `scripts/social_card.py` rebuilds the repo's social
preview from it.
"""

from __future__ import annotations
Expand Down Expand Up @@ -335,7 +336,11 @@ def tour(self) -> None:
self.dictate(QUESTION, 2.8) # hold ctrl+win, talk, let go
self.wait(1.2)
self.press("Enter") # and Enter asks it
self.wait(ANSWER_SEC + 2.8)
# The composer drops below the answer once the question posts, so a hand left
# where it clicked would sit on the answer. It rests under the conversation list.
box = self.page.locator(".convo-list").bounding_box()
self.move((box["x"] + box["width"] / 2, box["y"] + box["height"] + 110), 0.6)
self.wait(ANSWER_SEC + 2.2)
self.session._recent += [(RECENT_ASKED, QUESTION),
(RECENT_ANSWERED, reel.ANSWER)]
self.to(NAV.format("models"), 0.7)
Expand Down
Loading
Loading