Skip to content

FlyCoder 0.3: real bench numbers, pro rebuilt on the best-measured prompt - #12

Merged
Tromset merged 5 commits into
mainfrom
claude-code/peaceful-hamilton-89ge2k
Oct 10, 2026
Merged

Tromset merged 5 commits into
mainfrom
claude-code/peaceful-hamilton-89ge2k

Conversation

@Tromset

@Tromset Tromset commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Why

The README and Documentation.md still said the 0.3 bench was "being recorded". flycoder0.3pro also used a six-step workflow with a worked example, which had never been measured.

Measurements

All runs: 20 hidden-test problems, 4-core Linux server without a GPU, thinking off, one attempt per problem, at most 2,048 tokens per answer.

Profile Passed Generation
flycoder0.3 (Qwen3.5 9B) 9/20 3.9 tok/s
flycoder0.3fast (Qwen3.5 4B) 5/20 6.1 tok/s
0.2 beta (Gemma 4 12B), recorded earlier 14/20 3.4 tok/s

The 0.2 row was measured on a slower CPU, so the speeds compare only roughly.

System-prompt A/B on the same 9B base:

Prompt Passed Cut-off answers
FlyCoder 0.3 (shipped) 9/20 3
Same, without the final-check line 8/20 3
First pro recipe (six-step workflow + worked example) 6/20 2

What changes

  • Modelfile.pro: the pro model now uses the shared prompt, plus the coding-agent rules. "Never claim you ran tests" becomes "unless a tool actually ran them". The six-step workflow and the example conversation are removed. Qwen3.8 27B, the 64K context, multi-token prediction and REQUIRES 0.32.12 stay.
  • tests/modelfile.test.mjs: checks that pro keeps every shared rule and the agent rules, and has no example conversation.
  • readme.md and Documentation.md: real bench tables, the prompt comparison and the reason pro was rebuilt. Limitations are updated.
  • docs/bench/cpu-nothink-0.3.json: the full 0.3 run.
  • docs/bench/cpu-nothink-0.3-prompts.json: the merged A/B runs.
  • docs/ollama/flycoder0.3pro.md: no longer advertises the dropped workflow.

Verification

  • npm test: 65 tests pass.
  • Every number above comes from the committed JSON files.
  • The 27B itself was not benchmarked: the server cannot hold it, so the prompt choices were measured on the 9B base.

🤖 Generated with Claude Code

https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK


Generated by Claude Code

claude added 5 commits October 8, 2026 20:20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
A/B on the same Qwen3.5 9B base (20 bench problems, thinking off, one
attempt each):
- FlyCoder 0.3 prompt: 9/20
- same prompt without the final-check line: 8/20 (keeps the line)
- first pro recipe, six-step workflow plus a worked example: 6/20

flycoder0.3pro now uses the shared prompt plus the coding-agent rules,
with Qwen3.8 27B, 64K context and multi-token prediction. The README and
Documentation.md get the real 0.3 bench numbers and the comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
@Tromset
Tromset marked this pull request as ready for review October 10, 2026 10:53
@Tromset
Tromset merged commit 6fde498 into main Oct 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants