Phase 5: NAIVE schedule, K/V overlap, performance baselines

CONTEXT.md

@@ -28,7 +28,15 @@ On a fresh machine delete the symlinks and run `.tools/setup.sh` and `.tools/ems
 - Phase 4 (memory + FLASH controller + top): DONE. `sram_bank, hbm_model, dma, flash_ctrl, perf_counters, tiny_attn_top`
   and the 32-row attn_tile. End to end: bit-exact O, HBM bytes, tiles, K/V loads and busy cycles all equal the golden /
   closed-form models for every N, causal, cfg_qrows and latency tested (`SEEDS=200 bash sim/run_e2e.sh`).
-- Next: Phase 5 (cfg_overlap_en ping-pong K/V prefetch, NAIVE schedule, latency sweep, perf baseline pins).
+- Phase 5 (overlap, NAIVE, perf pins): DONE. `naive_ctrl` (4 passes, block-by-block), FLASH double-buffered K/V prefetch
+  (`cfg_overlap`), tile operations OP_NZ / OP_NM / OP_NP / OP_NPV, HBM 2048 words. Full e2e (200 seeds) and unit suites
+  pass; perf baseline in `sim/perf_baseline.json` (re-record with `PERF_RECORD=1 COCOTB_TEST_FILTER=perf_grid bash sim/run_e2e.sh`).
+- Next: Phase 6 (WASM harness: dual models, bindings, state schema, Node test against the golden fixtures).
+
+## Gotchas found in Phase 5
+- The NAIVE tile operations reuse the online-softmax unit: OP_NM uses its m path, OP_NP uses its P and l path with alpha = 1
+  (exp2(0) = 32768), OP_NPV uses o_accumulator with alpha forced to 32768. Golden equality holds, but that identity is the thing to re-check first if NAIVE ever breaks.
+- A wrong alpha in the NAIVE O update fails the e2e test (checked by injecting it).
 
 ## Gotchas found in Phase 4
 - Disk: the root filesystem was 100% full during Phase 4. The Bash tool writes its output to /tmp, so with no space every

README.md

@@ -8,7 +8,7 @@ Fourth project in the series after [tiny-gpu](https://github.com/deaneeth/tiny-g
 [tiny-tpu](https://github.com/deaneeth/tiny-tpu) and tiny-cpu. Same rule as the others: nothing on screen is
 faked; every value comes from a `dbg_*` port on the compiled RTL.
 
-**Status:** Phases 0-4 done: toolchain spike, bit-exact golden models and numerics study, the RTL arithmetic blocks, the tile core, and the full FLASH engine (HBM model, DMA, controller) matching the golden model bit for bit, with byte counts and cycle counts pinned. See [`docs/PLAN.md`](docs/PLAN.md) for the roadmap.
+**Status:** Phases 0-5 done: the FLASH engine, the NAIVE baseline and K/V overlap are bit-exact against the golden model, with byte counts, tile counts and cycle counts verified (FLASH sequential exactly against a closed-form model; overlap and NAIVE against pinned baselines). See [`docs/PLAN.md`](docs/PLAN.md) for the roadmap.
 
 ## Quick start
 

docs/ARCHITECTURE.md

@@ -258,3 +258,84 @@ N = 32, non-causal (`traffic.py`):
   numbers in 9.3 are the Phase 4 baseline that Phase 5 will be measured against.
 - NAIVE mode is Phase 5.
 - The HBM model has no bandwidth limit beyond one word per cycle, and writes are not latency-modelled.
+
+## 10. Phase 5: overlap, NAIVE schedule, perf baselines (design, written before RTL)
+
+### 10.1 Overlap (`cfg_overlap_en`, FLASH only)
+
+- K and V are double-buffered in `tiny_attn_top` (`kbuf[2]`, `vbuf[2]`, 4 words each). `cur` selects the buffer in use.
+- A prefetch sequencer, separate from the main FSM, starts when the group loop of K/V tile `kj` begins and `kj` is not the
+  block's last tile. It loads K then V of tile `kj+1` into buffer `!cur` over the one DMA engine.
+- The main FSM, at the end of tile `kj`'s group loop, waits for the prefetch to finish (`C_WAITPF`, zero cycles if it has),
+  swaps `cur`, then runs the weight load (4 cycles, serial: PE weights are stationary) and the next group loop.
+- Not overlapped: the Q load, the first K/V tile of each block, and the O store.
+- Only one DMA burst is in flight at a time, so the prefetch and the main FSM never issue DMA together. The protocol
+  guarantees the prefetch is done before any block-boundary DMA.
+
+### 10.2 NAIVE schedule (`cfg_mode = NAIVE`)
+
+Same arithmetic as FLASH, with the normalization deferred as in the golden model (`fixed.naive`). Blocks are always
+4 rows. Off-chip layout: Z tile `(qi,kj)` at `Z_BASE + 64·qi + 8·kj` (8 words, 2 int16 per word); P tile at `P_BASE + …`.
+HBM grows to 2048 words (`HBM_AW = 11`).
+
+| Pass | Per tile pair `(qi, kj)` | Block-level |
+|---|---|---|
+| 1 | load K tile, weight load, `OP_NZ`: MM1 + scale/mask → Z rows out; DMA write Z tile | Q block loaded once |
+| 2 | DMA read Z tile → `OP_NM`: row max into `m` | `m` is final after the last kj |
+| 3 | DMA read Z tile → `OP_NP`: `P = exp2(Z − m)`, `l += ΣP`; DMA write P tile | |
+| 4 | DMA read P tile, load V tile, weight load, `OP_NPV`: MM2 with P, O += P·V with α = 1 | normalize, store O |
+
+Masking, the `MASK` sentinel, and the causal diagonal are the same as the golden model. Pass 2 and pass 3 visit the
+same tiles, so pass 2 finishes a block before pass 3 starts on it.
+
+Traffic follows `traffic.py` NAIVE term by term: pass 1 reads `Q` once per block and `K` per tile; passes 2 and 3 each
+read `Z` per tile; pass 3 writes `P` per tile; pass 4 reads `P` and `V` per tile; O is written once per block.
+
+NAIVE has no overlap in Phase 5.
+
+### 10.3 Tile-core operations
+
+`tile_start` carries an operation:
+
+| Op | Input | Output | State effect |
+|---|---|---|---|
+| `OP_FLASH` | Q (streamed), K/V weights | O rows (normalized at block end) | m, l, α, O (online) |
+| `OP_NZ` | Q (streamed), K weights | Z rows on the output port | none |
+| `OP_NM` | Z tile (`tin`) | none | m = max(m, rowmax) |
+| `OP_NP` | Z tile (`tin`) | P rows on the output port | l += ΣP (α = 1 by construction) |
+| `OP_NPV` | P tile (`tin`), V weights | none | O += P·V (α forced to 1) |
+
+The output port (`out_*`) carries normalized O, Z or P rows; they go to the staging buffer, and the DMA writes them out.
+
+### 10.4 Performance baselines
+
+Phase 5 pins measured busy cycles for a grid of `(N, causal, qrows, L, overlap, mode)` in `sim/perf_baseline.json`.
+The FLASH non-overlapped case stays exact against the closed-form model from Phase 4. The overlapped and NAIVE cases
+are regression baselines measured from the RTL. Each one is checked by `sim/test_perf.py`; a change fails the test and
+must be justified in review.
+
+### 10.5 Measured results (as built)
+
+From `sim/perf_baseline.json` (N = 32, non-causal, L = 10):
+
+| Configuration | Busy cycles | vs sequential FLASH |
+|---|---|---|
+| FLASH, `cfg_qrows = 4`, no overlap | 4113 | 1.00 |
+| FLASH, `cfg_qrows = 4`, overlap | 2937 | 0.71 |
+| FLASH, `cfg_qrows = 16`, no overlap | 2133 | 0.52 |
+| FLASH, `cfg_qrows = 16`, overlap | 1657 | 0.40 |
+| NAIVE | 10961 | 2.66 |
+
+Overlap saves about 29% at `cfg_qrows = 4` (the K/V loads are most of the time there) and about 22% at `cfg_qrows = 16`.
+Causal runs follow the same pattern (2433 → 1845 at `cfg_qrows = 4`). NAIVE is 2.7× slower than FLASH at this size. Phase 5 does not break that gap down by pass; the pass
+breakdown is a candidate for Phase 9 profiling.
+
+The FLASH sequential case equals the cycle model in `sim/golden/schedule.py` exactly. The overlap and NAIVE cases are
+pinned baselines; `test_e2e.py::perf_grid` also checks that overlap is never slower than sequential for the same
+configuration.
+
+Deviations from the first plan, recorded here so they are not forgotten:
+- HBM grew from 256 to 2048 words to hold the NAIVE Z and P tiles.
+- NAIVE runs block by block, not pass by pass over all blocks. Values and traffic totals are identical (the
+  per-row computations are independent), but the sequencing differs from `fixed.naive`'s loop order.
+- Overlap applies to FLASH only. NAIVE is sequential.

docs/PLAN.md

@@ -438,7 +438,7 @@ The estimates assume one person part-time, at tiny-cpu's pace.
 - Write the controller timing diagram in `ARCHITECTURE.md` **before** coding it.
 - **Exit:** full attention is bit-exact for 200 seeds × N × causal; HBM bytes equal `traffic.py`.
 
-### Phase 5: NAIVE mode + overlap + perf pinning (1 week)
+### Phase 5: NAIVE mode + overlap + perf pinning (1 week) ✅ DONE
 - NAIVE schedule (Z and P written to and read back from HBM), `cfg_overlap_en` ping-pong prefetch, and the `cfg_hbm_latency` sweep.
 - **Exit:** the whole random matrix (§5 layer 3) is green; `perf_baseline.json` is pinned; the accuracy test is green.
   *At this point the engine is fully verified hardware and worth publishing on its own.*

rtl/attn_tile.sv

@@ -1,14 +1,17 @@
-// Tile core for one Q block of up to 32 rows (8 row groups of 4), driven by flash_ctrl.
+// Tile core for one Q block of up to 32 rows (8 row groups of 4), driven by flash_ctrl or naive_ctrl.
 //
-// Per K/V tile: kv_load (4 cycles) puts Kᵀ into MM1's weights and V into MM2's weights. Then every unmasked
-// row group of the block runs once through the pipeline with tile_start(tile_grp):
-//   MM1 (A = Q rows, W = Kᵀ) -> scale/mask -> online softmax -> MM2 (A = P rows, W = V) -> O update.
-// Row state (m, l, alpha, O) is kept per row in 32-entry arrays; the row index is {group, row-in-group}.
-// After the last K/V tile, blk_norm normalizes the first blk_nrows rows and emits them on out_*.
+// Operations (tile_op, see tiny_attn_pkg):
+//   OP_FLASH  Per K/V tile: kv_load (4 cycles) puts Kᵀ into MM1 and V into MM2. Then each unmasked row group runs
+//             MM1 (A = Q rows, W = Kᵀ) -> scale/mask -> online softmax -> MM2 (A = P, W = V) -> O update.
+//             Row state (m, l, alpha, O) is per row, index {group, row-in-group}. After the last tile, blk_norm.
+//   OP_NZ     MM1 + scale/mask only: each Z row goes to the output port. No state change.
+//   OP_NM     tin (4 Z rows) -> m = max(m, rowmax) per row.
+//   OP_NP     tin (4 Z rows) -> P = exp2(Z - m) per row (alpha = 1 by construction), l += sum(P); P rows to the output port.
+//   OP_NPV    tin (4 P rows, 9 bit in 16-bit lanes) through MM2 with the loaded V weights; O += P.V with alpha = 1.
 //
-// Timing of one group tile (c0 = first cycle after tile_start): Q rows enter MM1 at c0..c0+3, S rows leave MM1
-// at c0+7..c0+10, the softmax stage runs on each, P rows enter MM2 at c0+8..c0+11, PV rows leave MM2 at
-// c0+15..c0+18, tile_done at c0+19. tile_start to tile_done is 20 cycles (see sim/test_attn_tile.py).
+// Timing of an OP_FLASH group tile (c0 = first cycle after tile_start): Q rows enter MM1 at c0..c0+3, S rows leave
+// MM1 at c0+7..c0+10, P rows enter MM2 at c0+8..c0+11, PV rows leave at c0+15..c0+18, tile_done at c0+19.
+// tile_start to tile_done is 20 cycles (sim/test_attn_tile.py).
 /* verilator lint_off PINCONNECTEMPTY */  // mm_engine.busy is unused: tile_busy comes from the FSM
 module attn_tile
     import tiny_attn_pkg::*;
@@ -28,17 +31,19 @@ module attn_tile
     input  logic [4*Q_W-1:0]       v_row,           // row r of V
 
     input  logic                   tile_start,
+    input  logic [2:0]             tile_op,
     input  logic [2:0]             tile_grp,        // row group within the block
     input  logic                   tile_diag,       // this group's diagonal K/V tile (causal mask inside)
+    input  logic [4*4*16-1:0]      tin,             // NAIVE input tile: 4 rows x 4 lanes x 16 bit, row-major
     output logic                   tile_busy,
     output logic                   tile_done,
 
     input  logic                   blk_norm,
     input  logic [5:0]             blk_nrows,       // rows to normalize (4 per group in use)
     output logic                   blk_done,
     output logic                   out_valid,
-    output logic [4:0]             out_idx,         // row within the block
-    output logic [4*OUT_W-1:0]     out_row,         // Q8.8 V-LSB values, lane c in bits [c*16 +: 16]
+    output logic [4:0]             out_idx,         // row within the block (or tile row for NZ / NP)
+    output logic [4*OUT_W-1:0]     out_row,         // Q8.8 V-LSB values (O), or Z / P lanes
 
     // ---- debug: real registers and wires. Tile values are for the group of the most recent tile.
     output logic [2:0]             dbg_cur_grp,
@@ -57,7 +62,7 @@ module attn_tile
     output logic [16*P_W-1:0]      dbg_mm2_a,
     output logic [16*PV_W-1:0]     dbg_mm2_psum
 );
-    typedef enum logic [1:0] {S_IDLE, S_RUN, S_NORM} state_t;
+    typedef enum logic [1:0] {S_IDLE, S_RUN, S_ROWS, S_NORM} state_t;
     state_t state;
 
     // ---- per-row state (row = {group, row-in-group})
@@ -67,12 +72,13 @@ module attn_tile
     logic [E_W-1:0]         alpha_q [32];
     logic [4*O_W-1:0]       o_q     [32];
 
-    // ---- per-tile values (4 rows of the current group)
+    // ---- per-tile values (4 rows of the current group), FLASH only
     logic [4*S_W-1:0]       s_t     [4];
     logic [4*Z_W-1:0]       z_t     [4];
     logic [4*P_W-1:0]       p_t     [4];
 
     logic [2:0]             cur_grp;
+    logic [2:0]             op_q;
     logic                   diag_q, streaming, p_valid;
     logic [1:0]             qcnt, srow, orow;
     logic [4*P_W-1:0]       p_reg;
@@ -83,13 +89,33 @@ module attn_tile
     assign sidx = {cur_grp, srow};
     assign oidx = {cur_grp, orow};
 
+    logic is_flash, is_nz, is_nm, is_np, is_npv, mm1_run;
+    assign is_flash = (op_q == OP_FLASH);
+    assign is_nz    = (op_q == OP_NZ);
+    assign is_nm    = (op_q == OP_NM);
+    assign is_np    = (op_q == OP_NP);
+    assign is_npv   = (op_q == OP_NPV);
+    assign mm1_run  = is_flash || is_nz;
+
+    // ---- tin rows (NAIVE): row i = tin[i*64 +: 64], lane j = bits [j*16 +: 16]
+    logic [63:0] tin_row;
+    assign tin_row = tin[{srow, 6'b0} +: 64];
+
+    // NPV streams P rows from tin: low 9 bits of each lane
+    logic [4*P_W-1:0] npv_row;
+    logic [7:0]       npv_base;
+    assign npv_base = {qcnt, 6'b0};
+    for (genvar j = 0; j < 4; j++) begin : g_npv
+        assign npv_row[j*P_W +: P_W] = tin[npv_base + 8'(j*16) +: P_W];
+    end
+
     // ---- MM1: S = Q . Kᵀ
     logic             mm1_c_valid;
     logic [4*S_W-1:0] mm1_c_row;
     mm_engine #(.N(4), .A_W(Q_W), .A_SIGNED(1'b1), .W_W(Q_W), .ACC_W(S_W)) u_mm1 (
         .clk(clk), .rst_n(rst_n),
         .w_row_valid(kv_load_valid), .w_row_idx(kv_load_idx), .w_row(kt_row),
-        .a_valid(streaming), .a_row(q_buf[{cur_grp, qcnt}]),
+        .a_valid(streaming && mm1_run), .a_row(q_buf[{cur_grp, qcnt}]),
         .c_valid(mm1_c_valid), .c_row(mm1_c_row), .busy(),
         .dbg_w(dbg_mm1_w), .dbg_a(dbg_mm1_a), .dbg_psum(dbg_mm1_psum)
     );
@@ -105,30 +131,40 @@ module attn_tile
         );
     end
 
-    // ---- online softmax for the row leaving MM1
+    // ---- online softmax: FLASH takes Z from MM1; NM and NP take Z from tin
+    logic [4*Z_W-1:0]      z_src;
     logic signed [Z_W-1:0] m_new;
     logic [E_W-1:0]        alpha_new;
     logic [4*P_W-1:0]      p_row;
     logic [L_W-1:0]        l_new;
+    assign z_src = (is_nm || is_np) ? tin_row : z_row;
     online_softmax u_softmax (
-        .z(z_row), .m_old(m_q[sidx]), .l_old(l_q[sidx]),
+        .z(z_src), .m_old(m_q[sidx]), .l_old(l_q[sidx]),
         .m_new(m_new), .alpha(alpha_new), .p(p_row), .l_new(l_new)
     );
 
-    // ---- MM2: PV = P . V
+    // P row as 16-bit lanes (NAIVE stores P in HBM as 16-bit lanes; the value is at most 256)
+    logic [4*OUT_W-1:0] p_lanes;
+    for (genvar j = 0; j < 4; j++) begin : g_plane
+        assign p_lanes[j*OUT_W +: OUT_W] = OUT_W'(p_row[j*P_W +: P_W]);
+    end
+
+    // ---- MM2: PV = P . V (FLASH: P from the softmax; NPV: P from tin)
     logic                mm2_c_valid;
     logic [4*PV_W-1:0]   mm2_c_row;
     mm_engine #(.N(4), .A_W(P_W), .A_SIGNED(1'b0), .W_W(Q_W), .ACC_W(PV_W)) u_mm2 (
         .clk(clk), .rst_n(rst_n),
         .w_row_valid(kv_load_valid), .w_row_idx(kv_load_idx), .w_row(v_row),
-        .a_valid(p_valid), .a_row(p_reg),
+        .a_valid(is_npv ? streaming : p_valid), .a_row(is_npv ? npv_row : p_reg),
         .c_valid(mm2_c_valid), .c_row(mm2_c_row), .busy(),
         .dbg_w(dbg_mm2_w), .dbg_a(dbg_mm2_a), .dbg_psum(dbg_mm2_psum)
     );
 
-    // ---- O update for the row leaving MM2 (alpha of that row was stored by the softmax stage)
+    // ---- O update for the row leaving MM2. NPV has alpha = 1 (UQ1.15 1.0 = 32768).
     logic [4*O_W-1:0] o_new;
-    o_accumulator u_oacc (.o_old(o_q[oidx]), .alpha(alpha_q[oidx]), .pv(mm2_c_row), .o_new(o_new));
+    logic [E_W-1:0]   alpha_o;
+    assign alpha_o = is_npv ? E_W'(32768) : alpha_q[oidx];
+    o_accumulator u_oacc (.o_old(o_q[oidx]), .alpha(alpha_o), .pv(mm2_c_row), .o_new(o_new));
 
     // ---- normalization: one row per cycle, shared reciprocal
     logic [R_W-1:0]     r_n;
@@ -142,7 +178,7 @@ module attn_tile
         );
     end
 
-    assign tile_busy = (state != S_IDLE);
+    assign tile_busy   = (state != S_IDLE);
     assign dbg_cur_grp = cur_grp;
     assign dbg_srow    = srow;
 
@@ -159,6 +195,7 @@ module attn_tile
     always_ff @(posedge clk or negedge rst_n) begin
         if (!rst_n) begin
             state     <= S_IDLE;
+            op_q      <= '0;
             streaming <= 1'b0;
             p_valid   <= 1'b0;
             p_reg     <= '0;
@@ -190,7 +227,7 @@ module attn_tile
             tile_done <= 1'b0;
             blk_done  <= 1'b0;
             out_valid <= 1'b0;
-            p_valid   <= mm1_c_valid;
+            p_valid   <= mm1_c_valid && is_flash;
 
             if (q_load_valid) q_buf[q_load_idx] <= q_load_row;
 
@@ -205,13 +242,14 @@ module attn_tile
             case (state)
                 S_IDLE: begin
                     if (tile_start) begin
-                        state     <= S_RUN;
+                        op_q      <= tile_op;
                         cur_grp   <= tile_grp;
                         diag_q    <= tile_diag;
-                        streaming <= 1'b1;
+                        streaming <= (tile_op != OP_NM) && (tile_op != OP_NP);
                         qcnt      <= '0;
                         srow      <= '0;
                         orow      <= '0;
+                        state     <= ((tile_op == OP_NM) || (tile_op == OP_NP)) ? S_ROWS : S_RUN;
                     end else if (blk_norm) begin
                         state   <= S_NORM;
                         nrows_q <= blk_nrows;
@@ -225,7 +263,7 @@ module attn_tile
                         if (qcnt == 2'd3) streaming <= 1'b0;
                     end
 
-                    if (mm1_c_valid) begin
+                    if (mm1_c_valid && is_flash) begin
                         s_t[srow]     <= mm1_c_row;
                         z_t[srow]     <= z_row;
                         p_t[srow]     <= p_row;
@@ -236,6 +274,17 @@ module attn_tile
                         srow          <= srow + 2'd1;
                     end
 
+                    if (mm1_c_valid && is_nz) begin
+                        out_valid <= 1'b1;
+                        out_idx   <= {3'b0, srow};
+                        out_row   <= z_row;
+                        srow      <= srow + 2'd1;
+                        if (srow == 2'd3) begin
+                            tile_done <= 1'b1;
+                            state     <= S_IDLE;
+                        end
+                    end
+
                     if (mm2_c_valid) begin
                         o_q[oidx] <= o_new;
                         orow      <= orow + 2'd1;
@@ -246,6 +295,22 @@ module attn_tile
                     end
                 end
 
+                S_ROWS: begin
+                    // one tile row per cycle: NM updates m, NP updates l and emits P
+                    if (is_nm) m_q[sidx] <= m_new;
+                    if (is_np) begin
+                        l_q[sidx]  <= l_new;
+                        out_valid  <= 1'b1;
+                        out_idx    <= {3'b0, srow};
+                        out_row    <= p_lanes;
+                    end
+                    srow <= srow + 2'd1;
+                    if (srow == 2'd3) begin
+                        tile_done <= 1'b1;
+                        state     <= S_IDLE;
+                    end
+                end
+
                 S_NORM: begin
                     out_valid <= 1'b1;
                     out_idx   <= nrow;