[Grok 4.6] Plan: minimal instructional Rust port

[AuthorOfTheSurf]

Goal

Port KobraScript to Rust correctly and minimally, as an instructional rewrite. The original JavaScript should stay in the repo so a reader can compare the two implementations side by side.

This is not a redesign of the language, a performance contest, or a chance to add KS 3.0 features. The Rust compiler should implement the same pipeline the JS compiler already has:

.ks source  →  tokens  →  AST  →  (light analysis)  →  JavaScript

The JS compiler is the behavior spec. The existing fixtures under test/kobra-code/ are the acceptance tests. The README macrosyntax / microsyntax is the language spec.

Author: Grok 4.6

---

Why this port is a good Rust lesson

KobraScript is a small, real compiler. Almost every JS module maps onto a Rust concept a newcomer should actually learn:

JavaScript today	Rust we should write	Lesson
stringly-typed token.kind ('$', 'ID', 'fn')	enum TokenKind	make illegal states unrepresentable
33 constructor+prototype entity files	enum Stmt / enum Expr	algebraic data types instead of class hierarchies
error.count global + console.log	Result<T, Diagnostic> (or Vec<Diagnostic>)	recoverable errors, no hidden mutable process state
byline + callback scanner	fn scan(source: &str) -> Result<Vec<Token>>	ownership of the source string; no I/O inside the lexer
tokens.shift() mutating a shared array	parser borrows &[Token], advances an index	borrowing vs mutation
AnalysisContext with parent pointer + object hashmap	struct Scope { parent: Option<usize>, symbols: HashMap<String, Symbol> }	lifetimes / indices instead of JS object graphs
prototype.generateJavaScript on every class	fn emit(node: &Stmt, w: &mut impl Write)	match-based codegen, Write instead of string-concat classes
minimist + js-beautify + hashmap + byline	mostly std, tiny CLI	Rust’s stdlib is the default toolkit
Mocha + should	cargo test against the same .ks fixtures	testing without a parallel JS test stack

Keep the mapping obvious. Name Rust modules after the JS files (scanner, parser, analyzer, codegen) and put a short “corresponds to parser.js” comment at the top of each.

---

Non-goals (keep the port small)

Do not:

• Delete or rewrite the JavaScript compiler.
• Implement the stub Program.optimize() (JS only prints “not yet implemented”).
• Invent a static type system. entities/type.js is barely used; KS is a JS transpiler.
• Port unused JS (entities/math-change-assignment.js is never constructed by the parser).
• Bug-for-bug clone internal JS mistakes when they are not language behavior.
• Pull in parser combinators, LLVM, wasm, a custom GC, or async.
• Replace js-beautify with a heavy formatter crate. Emit readable JS with a small indent helper.

Preserve language behavior from the fixtures and README. If JS internals are buggy but the tests still pass, implement the intended semantics and note the JS quirk in a comment.

Known JS quirk worth fixing in Rust, not cloning:

• Name.prototype.analyze calls context.addVariable(this.value) with one argument, but addVariable is defined as (name, entity). Name lookup is effectively broken. Rust should have distinct declare vs lookup operations.

---

Current JS architecture (what we are porting)

kobra-cli.js / kobra
        │
        ▼
   compiler.js
        │
        ├── scanner.js   → Token { kind, lexeme, line, col }
        ├── parser.js    → Program / Block / entity tree
        ├── analyzer.js  → AnalysisContext (parent, symbolTable)
        └── entity.generateJavaScript() + code-gen/js-beautifier.js

CLI flags from kobra-cli.js (keep these):

Flag	Meaning
(none)	print usage
file.ks	scan → parse → analyze → print JS
-t	print tokens and exit
-a	print AST and exit
-o	“optimize” (no-op today; keep as a no-op flag)
-i	analyze, print semantic graph, exit

kobra compiles then pipes JS to node. kobrac is the compiler CLI.

Language surface to support (already in the parser):

• $ / .. / , declarations, fn, close {args}, if / else if / else, only, for, while
• return, leave, break, continue, say / loge
• operators including :=:, **, -**, is, ~=, #, ~?, ~!
• object / array / function / closure literals, dotted and index access, calls
• // comments, banned JS tokens (var, function, …)

Tests that define “correct”:

• ~83 good programs in test/kobra-code/good-programs/
• 21 syntax-error programs
• 4 semantic-error programs (bad-break, bad-continue, bad-return, bad-leave)
• scanner unit tests in test/test-scanner.js

Good-program tests today only assert error.count does not increase after parse+analyze. Rust should keep that bar, then add a few codegen smoke tests (hello-world, exchange, a function) so emit is not untested.

---

Proposed crate layout

Keep npm at the repo root. Put Rust in rust/ so both compilers coexist:

rust/
  Cargo.toml
  src/
    lib.rs              # public API: compile_source, compile_file
    error.rs            # Diagnostic, Result alias
    token.rs            # Token, TokenKind
    scanner.rs          # corresponds to scanner.js
    ast.rs              # Stmt / Expr / Program  (replaces entities/*)
    parser.rs           # corresponds to parser.js
    analyzer.rs         # corresponds to analyzer.js + entity.analyze
    codegen.rs          # corresponds to entity.generateJavaScript
    compiler.rs         # corresponds to compiler.js
  src/bin/
    kobrac.rs           # corresponds to kobra-cli.js
    kobra.rs            # corresponds to ./kobra (compile + run with node)
  tests/
    scanner.rs
    syntax_errors.rs
    semantic_errors.rs
    good_programs.rs

kobrascript is a library with thin bins. That is the usual Rust shape: logic in lib.rs, CLI in src/bin. Tests can call the library without spawning processes.

Cargo.toml should stay lean:

• edition 2021 (or 2024 if the toolchain is current)
• no regex crate unless the scanner becomes painful without it (prefer a hand-written scanner; that is more instructional and closer to how Rust compilers are written)
• no hashmap / byline / js-beautify equivalents
• CLI: std::env::args is enough for five boolean flags. clap is idiomatic but optional; skip it for a first port.

---

Design, module by module

1. Errors (error.rs)

JS:

error('Illegal character: ' + line[pos], {line, col})
error.count++

Rust:

#[derive(Debug, Clone)]
pub struct Diagnostic {
    pub message: String,
    pub line: Option<u32>,
    pub col: Option<u32>,
}

pub type Result<T> = std::result::Result<T, Diagnostic>;

Implement Display + std::error::Error. Do not use panic! / unwrap() for user-facing syntax errors.

The JS compiler is error-tolerant (it keeps going and counts). A minimal Rust port can be fail-fast: return the first Diagnostic. That is simpler and more idiomatic. If we want closer JS parity later, collect Vec<Diagnostic> in the parser. Start fail-fast.

Library code returns Result. Only the bins print to stderr and set the process exit code.

2. Tokens (token.rs)

JS Token is { kind, lexeme, line, col } where kind is a string.

Rust:

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum TokenKind {
    Dollar,           // $
    DotDot,           // ..
    Exchange,         // :=:
    Id, StrLit, NumLit, BoolLit, NullLit, UndefinedLit,
    Fn, Close, If, Else, Only, For, While, End,
    Return, Leave, Break, Continue, Say, Loge,
    // ...one variant per operator / keyword
    Eof,
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Token {
    pub kind: TokenKind,
    pub lexeme: String,   // or intern keywords and only allocate for Id/lits
    pub line: u32,
    pub col: u32,
}

This is the first “Rust as intended” win: match token.kind is exhaustive. Forgetting a keyword is a compile error, not a silent at('fn') miss.

Keywords that are their own kinds in JS (fn, say, end, …) should be variants, not Id. Same as the current scanner.

3. Scanner (scanner.rs)

JS reads the file with fs + byline, scans line by line, then callbacks.

Rust scanner should be pure:

pub fn scan(source: &str) -> Result<Vec<Token>>;

File I/O lives in the compiler/bin. That makes the lexer testable with string literals.

Implementation notes (instructional, not clever):

• Walk source.char_indices() with a peekable cursor (pos, line, col).
• Longest-match operators the same way JS does: 3-char, then 2-char, then 1-char.
• // comments consume the rest of the line.
• String literals keep the JS escape handling (\n, \uXXXX, \xHH, \cX → hex).
• Banned tokens (var, function, try, …) are diagnostics, as today.
• Append TokenKind::Eof.

Do not stream tokens. The JS parser wants a full vector anyway; Vec<Token> is the honest model.

Preserve observable tokenization from test/test-scanner.js (hello-world, exchange, increments, object). Those tests are the scanner spec.

4. AST (ast.rs) — the big lesson

Do not port one struct per entities/*.js file. That would fight the language.

JS is a class per node, with toString / analyze / generateJavaScript on the prototype. Rust should be data plus functions:

pub struct Program {
    pub block: Block,
}

pub struct Block {
    pub statements: Vec<Stmt>,
}

pub enum Stmt {
    Declaration { name: Name, initializer: Expr },
    FnDecl { name: Name, params: Vec<Name>, body: Block },
    If { arms: Vec<Conditional>, else_block: Option<Block> },
    OnlyIf { body: Block, condition: Expr, else_block: Option<Block> },
    For { init: Vec<Stmt>, condition: Expr, after: Vec<Expr>, body: Block },
    While { condition: Expr, body: Block },
    Return(Expr),
    Leave,
    Break,
    Continue,
    Say(Expr),
    Expr(Expr),
}

pub enum Expr {
    Assign { op: BinOp, left: Box<Expr>, right: Box<Expr> },
    Binary { op: BinOp, left: Box<Expr>, right: Box<Expr>, wrapped: bool },
    Unary { op: UnOp, operand: Box<Expr> },
    Postfix { op: UnOp, operand: Box<Expr> },
    Call { callee: Box<Expr>, args: Vec<Expr> },
    Dotted { object: Box<Expr>, field: Name },
    Index { object: Box<Expr>, index: Box<Expr> },
    Fn { name: Option<Name>, params: Vec<Name>, body: Block },
    Closure { args: Vec<Name>, body: Block },
    Array(Vec<Expr>),
    Object(Vec<Property>),
    Name(Name),
    Number(String),
    String(String),
    Bool(bool),
    Null,
    Undefined,
}

Box<Expr> is the recursive-type lesson. Vec is the list lesson. Option replaces JS null / missing fields.

toString for -a can be a Display impl that matches the existing S-expression style ((Program (Block ...))) so AST dumps stay comparable to JS.

BinOp / UnOp should be enums (Star, EqEq, Exchange, Pow, Is, …), not leftover token lexemes. Codegen then maps them to JS (== → ===, ** → Math.pow, :=: → swap, # → ||, is → typeof).

5. Parser (parser.rs)

Keep the same recursive-descent shape as parser.js so a reader can open the two files together:

• parse_program, parse_block, parse_statement, parse_declaration
• parse_exp … parse_exp9 / parse_exp_root with the same precedence
• at / next / match_token helpers

Implementation:

struct Parser<'a> {
    tokens: &'a [Token],
    pos: usize,
}

impl<'a> Parser<'a> {
    fn at(&self, kind: TokenKind) -> bool { ... }
    fn bump(&mut self) -> Result<&'a Token> { ... }
}

This is the borrowing lesson: the parser does not own tokens; it borrows them. AST nodes copy the data they need (String names, operator enums) so the token buffer can die after parse.

Keep JS parser quirks that are language:

• continuing for $ a, b .. c multi-declarations
• -> single-statement blocks vs : … end/..
• return must be on the same line as its expression
• parens only allowed around BinaryExpression (“There is no need to wrap X in parenthesis”)
• only: stmt .. if (cond)

Return Result<Program>. No global tokens array.

6. Analyzer (analyzer.rs)

JS analysis is small. Port that, not a typechecker.

What the tests actually require:

• return / leave only inside a function/closure (isSubroutine)
• break / continue only inside a loop (looped)

What we should also do, because that is what AnalysisContext was for:

• nested scopes (createChildContext)
• declare names on $ / fn / params
• lookup names on use (fix the JS addVariable arity bug)

struct Scope {
    parent: Option<usize>,
    symbols: HashMap<String, SymbolId>,
    in_fn: bool,
    in_loop: bool,
}

struct Analyzer {
    scopes: Vec<Scope>,
    current: usize,
}

Use scope indices rather than &mut parent + another &mut child. That avoids fighting the borrow checker and is a standard compiler pattern (see rustc’s hir_id / arena style, scaled way down).

Walk the AST with match (or inherent methods). Do not introduce a visitor trait unless the walk gets copy-pasted three times.

-i semantic graph: JS dumps tagged objects via hashmap. A simple recursive debug dump ({:?} plus assigned scope ids) is enough.

7. Codegen (codegen.rs)

Match JS emit rules. These are load-bearing:

KS	JS
say x / loge x	console.log(x)
$ a = 1	var a = 1
a :=: b	b = [a, a = b][0]
x ** y	Math.pow(x, y)
x -** y	((1.0)/Math.pow(x, y))
x is y	(typeof x === y)
== / !=	=== / !==
~=	==
#	||
close{x}: ... end	(function(x) { ... }(x))
leave	return
fn name(a): ... end	function name ( a ) { ... }

Pretty-print with indentation and newlines. Do not take a beautifier dependency. Tests should not assert exact whitespace against js-beautify.

continuingDeclaration in JS exists so $ a = 1 .. b = 2 becomes var a = 1, b = 2. Recreate that with a small EmitState { continuing_declaration: bool } passed down, or by emitting declaration lists as one var statement. Either is fine; the flag is the closer JS analog.

8. Compiler + CLI

pub fn compile(source: &str) -> Result<String> {
    let tokens = scanner::scan(source)?;
    let program = parser::parse(&tokens)?;
    analyzer::analyze(&program)?;
    Ok(codegen::emit(&program))
}

Bins:

• kobrac mirrors kobra-cli.js: extension check (.ks), flags -t -a -o -i, usage text.
• kobra runs kobrac logic then std::process::Command::new("node") on stdin, matching ./kobra.

Keep the usage string recognizable so the two CLIs feel like the same tool.

---

Instructional style rules

These are the “use Rust as intended” constraints for the implementation:

1. Let the compiler work. Prefer enums, match, and #[derive(Debug, PartialEq)] over boolean flags and string modes.
2. Own or borrow on purpose. Source is borrowed by the scanner. Tokens are owned. Parser borrows tokens. AST owns its strings. Analyzer borrows the AST. Codegen borrows the AST and writes to a String.
3. No clone() as a reflex. If a clone is needed, a one-line comment should say why.
4. No unwrap() in library paths. Tests may unwrap() / expect() with a message.
5. Standard library first. HashMap, Write, format!, char_indices. Extra crates need a one-sentence justification.
6. Keep functions boring. Recursive descent and match node beat clever macros. Do not write a parse_rules! DSL.
7. Mirror JS control flow in the parser/scanner so the two files can be read in two panes. Use Rust types in the AST/analyzer, where JS classes were the wrong model.
8. Comments teach the mapping, not the syntax. Good: // JS scanner.js emits kind: ':=:'. Bad: // This is a for loop.

A few targeted “JS vs Rust” comments in token.rs, ast.rs, and error.rs are enough. Do not annotate every line.

---

Implementation phases

Ship in this order so each phase is testable and readable:

Phase 0 — crate skeleton

• rust/Cargo.toml, lib.rs, empty modules, kobrac that prints usage.
• Confirm cargo test / cargo build work.

Phase 1 — tokens + scanner

• TokenKind, scan(&str).
• Port test/test-scanner.js cases (hello-world, exchange, increments, object).
• Table-driven tests: input snippet → expected [(kind, lexeme)].

Phase 2 — AST + parser

• Stmt / Expr.
• Port parseProgram through parseExpRoot.
• Run all 21 test/kobra-code/syntax-errors/*.ks (each must Err).
• Dump AST for a couple of good programs and eyeball against kobrac -a.

Phase 3 — analyzer

• Scopes, in_fn, in_loop.
• Semantic fixtures must Err; good programs must Ok.

Phase 4 — codegen + CLI

• Emit JS for the operator table above.
• kobrac flags -t -a -i.
• Smoke: hello-world.ks, exchangestatement.ks, function.ks.
• Optional: kobra bin that shells out to node.

Phase 5 — fixture sweep

• Drive cargo test over all 83 good programs (parse + analyze succeeds).
• Add 3–5 codegen equality or “contains” tests, not 83 golden files.
• Document how to run JS tests vs Rust tests in rust/README.md (short).

Do not start Phase 4 until Phase 2 passes syntax-error fixtures. That is the same order the JS compiler itself is structured.

---

Testing strategy

Reuse the existing .ks files. Do not duplicate them into rust/. Path from Rust tests: ../test/kobra-code/....

#[test]
fn good_programs_compile() {
    for entry in fs::read_dir("../test/kobra-code/good-programs").unwrap() {
        let path = entry.unwrap().path();
        let src = fs::read_to_string(&path).unwrap();
        compile(&src).unwrap_or_else(|e| panic!("{}: {e}", path.display()));
    }
}

Same idea for syntax-errors (must fail) and semantic-errors (parse ok, analyze err).

Scanner tests should stay explicit (token-by-token) because they document the lexer.

---

What “done” looks like

• cargo test in rust/ passes.
• cargo run --bin kobrac -- ../test/kobra-code/good-programs/hello-world.ks prints JS equivalent to var hello = "Hello, world!"; console.log(hello); (whitespace flexible).
• kobrac -t / -a work.
• A reader can open parser.js next to rust/src/parser.rs and follow the same functions, then open ast.rs and see why the entity folder collapsed into two enums.
• No new KS syntax. No deleted JS.

---

Suggested first PR slices (if implemented later)

1. Phase 0–1 (scanner) — smallest useful Rust, already teaches enums + Result.
2. Phase 2 (parser + AST) — the main instructional payoff.
3. Phase 3–5 (analyze, emit, CLI, fixture sweep).

This issue is the plan only. Implementation should follow these constraints unless a later review finds a fixture the plan missed.