Repository navigation
FlyCoder 0.3: real bench numbers, pro rebuilt on the best-measured prompt - #12
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
… 5/20) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
A/B on the same Qwen3.5 9B base (20 bench problems, thinking off, one attempt each): - FlyCoder 0.3 prompt: 9/20 - same prompt without the final-check line: 8/20 (keeps the line) - first pro recipe, six-step workflow plus a worked example: 6/20 flycoder0.3pro now uses the shared prompt plus the coding-agent rules, with Qwen3.8 27B, 64K context and multi-token prediction. The README and Documentation.md get the real 0.3 bench numbers and the comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Tromset
marked this pull request as ready for review
October 10, 2026 10:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README and Documentation.md still said the 0.3 bench was "being recorded".
flycoder0.3proalso used a six-step workflow with a worked example, which had never been measured.Measurements
All runs: 20 hidden-test problems, 4-core Linux server without a GPU, thinking off, one attempt per problem, at most 2,048 tokens per answer.
flycoder0.3(Qwen3.5 9B)flycoder0.3fast(Qwen3.5 4B)The 0.2 row was measured on a slower CPU, so the speeds compare only roughly.
System-prompt A/B on the same 9B base:
What changes
Modelfile.pro: the pro model now uses the shared prompt, plus the coding-agent rules. "Never claim you ran tests" becomes "unless a tool actually ran them". The six-step workflow and the example conversation are removed. Qwen3.8 27B, the 64K context, multi-token prediction andREQUIRES 0.32.12stay.tests/modelfile.test.mjs: checks that pro keeps every shared rule and the agent rules, and has no example conversation.readme.mdandDocumentation.md: real bench tables, the prompt comparison and the reason pro was rebuilt. Limitations are updated.docs/bench/cpu-nothink-0.3.json: the full 0.3 run.docs/bench/cpu-nothink-0.3-prompts.json: the merged A/B runs.docs/ollama/flycoder0.3pro.md: no longer advertises the dropped workflow.Verification
npm test: 65 tests pass.🤖 Generated with Claude Code
https://claude.ai/code/session_013QgiejZTjqQAUv4N4EFskK
Generated by Claude Code