t27o: iterator chains with a single-expression closure (map, filter, sum, count, collect) lowered to loops -- measure first #11145

Activity

1. gHashTag

   added

   enhancementNew feature or requestNew feature or request

   on Oct 19, 2026
2. gHashTag commented on Oct 19, 2026

   

   gHashTag

   on Oct 19, 2026

   Last edited by gHashTag

   OwnerAuthor

   More actions

   Step 1 measured (lane A6): 17 fns convert end to end when the iterator chains of this issue's shape are lowered, 26 counting the fns that call them. The threshold is 100, so no rule PR is opened. With find and position added the figures are 23 and 32. Numbers first, then how they were made.

   1. Population (1,388 tracked files of the five repositories)

   	fns
   fn seen by the converter on master (sweep)	19,392
   converted today	1,824
   fn items my scanner lists (it also sees nested and cfg-off fns the converter does not count)	19,513
   fns that hold at least one iterator chain	2,697 (13.8 %)
   - refused on master (first refusal known)	2,546
   - inside a refused item (impl of a refused type, async, trait impl)	78
   - no refusal line: 6 convert, the others are fns the converter does not list (cfg-off variants, nested fns) or refuses at another line	73
   iterator chains in all of them	4,587 (4,387 in the refused fns)
   fns that are tests	949 of the 2,697 (919 refused)

   Chain sources, all 4,587: .iter() / .into_iter() of a slice, Vec or array 2,818; a parenthesised range 565; text (lines 410, split 221, chars 220, matches 81, split_whitespace 67, bytes 19, other 38) 1,056; other (iter_mut 61, values 40, keys 26, windows 9, chunks 9, drain 3) 148. First refusals of the 2,546 refused fns: type Vec 765, method iter 244, type Path 142, type BTreeSet 119, type String 93.

   2. (a) Consumer of the chain (the 4,387 chains of the refused fns)

   consumer	chains	fns holding one
   collect	1,805	1,328
   none: the chain is a for header or an adaptor passed on	730	548
   any	387	277
   count	350	258
   find	217	167
   sum	217	155
   all	211	183
   next	174	146
   nth 68, position 65, fold 33, max 29, find_map 27, for_each 17, last 13, min 10, partition 9, product 7, other 18	296

   Where the collect goes (1,805): Vec 1,357, String 101, a set or map 116, Result / Option 22, inferred from a later use 201, other 8. A chain with no consumer is a loop already (for (i, x) in v.iter().enumerate(), zip, take); the converter reads some of those forms today.

   3. (b) Shape of the closure (the 2,928 chains with a closure, refused fns)

   closure	chains	fns
   one pure expression, std table only	1,516	1,048
   one pure expression, calls a fn of the same file	354	325
   block body with statements	276	264
   early return, ?, break	7	7
   assigns, or calls a mutating method or a fn taking &mut (a captured mutable)	39	32
   macro call (format!, json!, matches!)	190	155
   nested closure	83	77
   call outside the std table and the file (to_string 133, powi 26, join 21, is_alphanumeric 20, is_file 16, contains_key 10 ..)	463	372

   64 % of the closures are one pure expression. That is not the wall: the tables below are.

   4. (c) Element type (chains over .iter() / .into_iter() or a range, 3,226 chains of the refused fns)

   element	chains	fns
   integer	692	628
   float	97	60
   bool	2	2
   struct or enum	712	527
   tuple	178	101
   map, array, char, other	243	164
   not declared where the scanner can read it (a local bound from a call, a closure parameter)	1,302	848

   Types are read from parameter, typed let and struct field declarations; an undeclared receiver is "unknown" and was left to the converter to judge in (d). Of the 676 fns at the "hoistable" step below, 261 hold a chain over integers, 13 over floats, 119 over structs or enums, 270 over an undeclared receiver.

   5. (d) What is left after the chain is lowered

   The next first refusal was not guessed. Every refused fn whose chains are all of the inlinable shape was rewritten at the Rust level (each chain becomes a counted loop with an accumulator, hoisted before its statement, the line count kept), and the real converter judged the result.

   step	fns
   refused fns that hold a chain	2,546
   every chain is over a slice / Vec / array .iter() or a range (not a map or set)	1,751
   ... and every chain has a consumer	1,377
   ... and the consumer is sum count any all fold collect, the adaptors only map filter copied cloned enumerate, nothing called on the result	1,112
   ... and every closure is one pure expression	711
   ... and the chain sits where a loop can be hoisted (not right of an and / or operator, not an else-if or while condition, not in a match arm, not inside another closure)	676
   ... and every collect is the returned list (the out-buffer form)	323
   the other 353 collect into a LOCAL Vec or set: the converter has no form for it (only the returned out-buffer)	(blocked)
   lowered (the rest of the 323: 38 closures over a tuple element, 35 returned lists of non-scalar elements)	250
   convert end to end after lowering	17
   the same plus the fns that call them and now convert	26
   with find, position and a call on the result added: lowered / convert / with callers	310 / 23 / 32

   The 17: 7 are any / all over a slice, Vec or constant array (first refusal on master: method iter), 10 are collect into a returned Vec of scalars (Vec<u8> 6, Vec<f32> 2, Vec<u32> 1, Vec<u64> 1; first refusal type Vec, the out-buffer form of #11109). No sum, count or fold fn converts (81 of the 250 were lowered): 36 are methods of refused structs (uses refused), 16 name an enum variant of another module, 10 stop at a type, 8 at a method outside the table. No fn went from converted to refused in any of the 119 files rewritten.

   What refuses the other 233 lowered fns first:

   first refusal after lowering	fns
   uses refused (a callee, or self, of a refused type)	70
   method outside the std table	48
   path (an enum variant or fn of another module)	25
   type String / Path / anyhow / std / a collection / a tuple	17
   vec! or another macro	14
   uses external	11
   tuple	10
   syntax	9
   ! of an operand of unknown type	7
   type Vec (a local)	6
   10 other reasons, 1 to 4 each	16

   The method iter first refusals (244 chain fns; 246 in the whole corpus, the number in #10602): 147 hold a chain of the core shape, 39 a closure that is not one pure expression, 58 a consumer or adaptor outside the core set (find 28, max 8, position 7 ..). Lowering the 147 converts 7 fns (13 with find and position). Teaching the std table iter alone would move none of them: the wall is what follows it.

   6. Against the success criteria

   • At least 100 fns would convert end to end: 17 (26 with callers; 23 and 32 with find / position). The best figure is a third of the threshold. Lowering min, max, last, nth, next, for_each, zip, rev, take, skip, filter_map too would add 44 fns that meet every other condition; at the end-to-end rate measured here (about 7 % of the lowered fns) that is about 3 more.
   • Converted fns up by 1 % of the 19,392 seen (194): the best figure is 0.17 % (32 fns).
   • The gap is not a classification error. The converter is the judge of the "after"; the end-to-end figure moved between 13 and 17 across the scanner fixes below (a receiver broken over lines was the largest), never near 100.

   7. What the data point at (observation, not a plan)

   • The largest group left is 353 fns (286 of them tests) that collect a lowerable chain into a local list: 212 over a range (82 with literal bounds, the rest constants or variables), 141 over a slice whose length is a run-time value. Closure inlining cannot convert them; they need a local array of a stated length, which is the scratch-list work of t27o: Vec<T> and vec![..] without a heap -- read-only literals, then the out-buffer form (16 % of first refusals) #10546 step 3.
   • The out-buffer form of t27o: a fn returning Vec<T> built by push converts in the out-buffer form -- a last out: []T, the result the count, the oracle checks the written prefix (Closes #11073, Closes #11112) #11109 already takes a returned Vec of scalars built by push; a returned collect over scalars is the same form once the chain is a loop. That is the 10 fns above and about as far as scalar collect reaches.
   • The most common next refusal is uses refused (70 of 233): the fn is a method of a struct the converter refuses, the cascade the sweep already counts under its root.

   How it was measured

   • Converter: gen/rust/tools/t27o/rust_convert.rs at master 98b57db (the t27o spec and generated file are the same at a1780a1, the head when I posted, and unchanged since ca90be1), built with rustc into a scratch driver that makes the same calls as cmd_t27o; nothing was added to the repo. I re-ran the whole sweep with it (the A4 / A5 sweep.zsh, one process per file, 60 s cap): 19,392 fn seen, 1,824 converted, 16,057 first refusals, type Vec 1,289, the same totals as t27o: Vec parameters -- measure them, then convert a read-only &Vec<T> to a slice and a push-only &mut Vec<T> to the out-buffer form #11135. 15,988 of the 16,057 refused fns match my scanner's fn list by name and line.
   • Scanner: scratch Python, not committed. It masks comments and strings, matches brackets, finds fn items, parses every chain forward from its source (.iter(), .into_iter(), a parenthesised range, the text and map sources), reads each call's closure, and classifies the closure body with regexes over the masked text (assignment, mutating method, &mut, return / ? / break, macro, nested closure, calls checked against the std table of rust_std.t27 and the fns of the file). A chain's place is read by walking back to the start of its statement.
   • Precision, five hand checks of 30 random chains while the scanner was being fixed: 27, 29, 28, 28 and 29 right (a chain counted right when source, adaptors, consumer, call on the result, closure shape, place and the lowerable verdict were all right), and two checks of 30 chains the scanner calls lowerable: 30 and 29 right. Defects found and fixed, every table re-run each time: a .contains on a range counted as a chain; a statement start taken from the block before it; a receiver broken over lines; a callee taking &mut read as pure; the comma of |acc, x| splitting a fold; a fn path read as a closure; ..= read as an assignment; a std method name that the file also defines (collect) read as the file's own; a let inside a nested closure read as a statement of the outer one; a map receiver read as a slice. The last two checks were taken before the last two fixes, which changed a label and removed 4 fns from the candidates, not the result.
   • The lowering used in (d) is the form the converter already converts (checked: for x in v.iter() { acc = acc + x * 2; }, if *x > k { n = n + 1; }, the push form into a returned Vec), so a rule would have to emit exactly that. The experiment is generous to the rule in that it gets the converter's whole loop machinery for free, and strict in that it never lowers a chain the scanner could not read.

   Not run

   • tri t27o-sweep itself (no cargo build: low disk); the replay of oracle harnesses, t27b test, the own-language budget, dupe_scan and the seal checks (no rule was written, no spec or generated file was touched).
   • A lowering of min, max, last, nth, next, for_each, zip, rev, take, skip, filter_map: static count only (44 fns, above).
   • Text sources (chars, bytes, lines, split: 1,056 chains), which the issue leaves out.

   Suggestion: close as not planned on the measurement, as #11135 was, and take the local-list form as the next measurement. I have left the issue open.
3. gHashTag commented on Oct 19, 2026

   

   gHashTag

   on Oct 19, 2026

   OwnerAuthor

   More actions

   Closing as not planned, on the measurement above (lane A6): 2,697 fns hold an iterator chain, but of the 676 fns where a chain can be lowered at all, 250 could be rewritten as a counted loop and only 17 convert end to end (32 counting callers, 0.17% of 19,392 against the 1% target of this issue). The rest stop at the next refusal: uses refused (70), a method outside the std table (48), a path (25), and the largest group, 353 fns that collect into a local Vec or set, has no form in t27. No sum, count or fold fn converts. The rule threshold of this issue (at least 100 fns) is not met, so no rule PR was written.

   Taken together with #11135 (Vec parameters: ceiling 1 fn) this is the second measured no in a row. The converted share of Rust is bounded by language features t27 does not have (heap collections, owned strings and paths, iterators with closures, tuples), not by missing converter rules. The new lever is the file: 85 files convert whole today, which is what the only-t27 rule and #9000 care about (hand-written Rust deleted).
4. gHashTag

   closed this as not plannedon Oct 19, 2026
5. gHashTag

   added a commit that references this issue on Oct 20, 2026

   docs(t27o): RFC for heap-free collections, owned strings and inlined …

   ...

   876d3c2