Skip to content

Add support for BEGINFILE and ENDFILE rules#515

Open
bertysentry wants to merge 2 commits into
mainfrom
457-add-support-for-beginfile-and-endfile-rules
Open

Add support for BEGINFILE and ENDFILE rules#515
bertysentry wants to merge 2 commits into
mainfrom
457-add-support-for-beginfile-and-endfile-rules

Conversation

@bertysentry

Copy link
Copy Markdown
Contributor

Closes #457.

What changed

Jawk now parses and executes gawk's BEGINFILE / ENDFILE special patterns, together with the nextfile statement and the ERRNO / ARGIND special variables.

  • Parser (AwkParser): new BEGINFILE / ENDFILE / nextfile keywords and rule ASTs. When any of these features is used, the main input loop is compiled with the per-file tuple layout from the issue (NEXT_FILE before begin_file, CONSUME_FILE_INPUT → end_file at each file's EOF, ENDFILE rules, then NEXT_FILE → no_more_input falling into the END blocks). Programs that don't use these features keep the exact previous flat layout, so there is zero overhead for existing scripts.
  • Runtime (AVM, JRT, StreamInputSource): per-file stepping with gawk-compatible FILENAME, FNR = 0, cleared $0, ARGIND, and ERRNO on open failure (No such file or directory / Is a directory, synthesized deterministically cross-platform). A file that cannot be opened is non-fatal only if a BEGINFILE rule skips it with nextfile; otherwise the usual gawk-style fatal error fires. nextfile works in ordinary rules and BEGINFILE — including from user-defined functions (it unwinds the runtime/operand stacks the same way exit does) — and runs ENDFILE for successfully opened files while skipping it for failed ones, matching the vendored .ok fixtures. exit inside the hooks jumps to the END blocks.
  • Restrictions, matching gawk: next is rejected inside BEGINFILE/ENDFILE and nextfile inside BEGIN/END/ENDFILE (compile time when direct, runtime when reached through a function); non-redirected getline is a fatal error inside both hooks, redirected forms are allowed.
  • Opcodes: NEXT_FILE, CONSUME_FILE_INPUT, EXEC_NEXTFILE, SET_ENDFILE_ADDRESS, SET_NEXTFILE_ADDRESS, PUSH/ASSIGN_ERRNO, PUSH/ASSIGN_ARGIND — appended at the end of the Opcode enum because fromId resolves by ordinal, preserving precompiled tuple (-L) compatibility.
  • POSIX mode: as in gawk, BEGINFILE and ENDFILE are not special under --posix / setPosix(true) and lex as ordinary identifiers. nextfile stays available in both modes (POSIX.1-2024 standardizes it).
  • Lexer fix: a / inside a regexp bracket expression (/.*[/]/, used by beginfile1.awk) no longer terminates the literal. This also flipped the vendored gawk regexpbrack IT from error to pass.
  • FNR fix (found while testing the hooks on stdin): FNR now counts records read from standard input (POSIX defines FNR as the record number in the current input file, which includes stdin) — printf 'a\nb\n' | jawk '{print FNR}' now prints 1 2 instead of 0 0. Custom InputSource implementations keep the documented isFromFilenameList() contract.

Tests

  • New BeginFileEndFileTest (25 cases): execution order across files, empty files, stdin, ERRNO + nextfile skip, fatal open failures, nextfile from functions and without hooks, all restriction diagnostics, ARGIND, $0 clearing, exit in hooks, POSIX-mode non-specialness, bracket-slash regexp.
  • Re-enabled the previously-skipped POSIX nextfile conformance test; added stdin-FNR conformance tests and an InputSourceTest guard for the custom-source FNR contract.
  • GawkExtensionIT#test_beginfile1 now runs the vendored fixture for real and matches beginfile1.ok byte for byte (including the ERRNO lines for . and a missing file). Six beginfile2 sub-tests (2, 5, 9a, 9b, 12, 15) extract their programs and expected transcripts from the vendored beginfile2.in / .ok and pass. The remaining sub-tests rely on non-redirected getline in BEGIN/END driving the BEGINFILE/ENDFILE hooks mid-statement (or compare non-zero gawk CLI transcripts) — that reentrancy isn't reproducible in the tuple VM; the umbrella skip documents it. getfile / readfile2 need gawk's dynamic extension API (@load, get_file()), which Jawk doesn't provide; their skip reasons now say so.

Verification

  • mvn verify site: BUILD SUCCESS; checkstyle / PMD / SpotBugs at 0 findings; 543 unit tests pass.
  • Full compatibility-IT regression check: I ran the entire failsafe suite on a clean baseline worktree and diffed all per-test outcomes — the only changes are the 7 new beginfile tests passing and regexpbrack turning green; zero regressions across the 939 pre-existing cases, including after the FNR change.

Notes for reviewers

  • Known deviation (documented in the umbrella test_beginfile2 skip): non-redirected getline in BEGIN/END reads ahead in the main input without firing BEGINFILE/ENDFILE; the loop then adopts the already-open file instead of reopening it.
  • gawk 5.1.1+ semantics were chosen where versions differ: $0 and the fields are cleared in BEGINFILE.

🤖 Generated with Claude Code

Implement gawk's BEGINFILE/ENDFILE special patterns, the nextfile
statement, and the ERRNO and ARGIND special variables:

- Parser: BEGINFILE/ENDFILE/nextfile keywords, per-file input-loop
  tuple layout (NEXT_FILE / CONSUME_FILE_INPUT scaffolding emitted only
  when these features are used, so existing programs keep the flat
  layout), and gawk-compatible compile-time restrictions (next rejected
  in BEGINFILE/ENDFILE, nextfile rejected in BEGIN/END/ENDFILE).
- Runtime: per-file stepping in StreamInputSource with FILENAME, FNR=0,
  cleared $0, ARGIND, and ERRNO on open failure; open failures become
  non-fatal only when a BEGINFILE rule skips the file with nextfile;
  nextfile unwinds user-defined function calls like exit does;
  non-redirected getline is fatal inside BEGINFILE/ENDFILE.
- New opcodes appended at the end of the Opcode enum to preserve
  precompiled tuple (-L) compatibility.
- BEGINFILE/ENDFILE are not special in POSIX mode (--posix), as in gawk.
- Lexer: a slash inside a regexp bracket expression (/[/]/) no longer
  terminates the literal, which beginfile1.awk requires; this also makes
  the vendored gawk regexpbrack test pass.
- FNR now counts records read from standard input (POSIX: FNR is the
  record number in the current input file); custom InputSource
  implementations keep the documented isFromFilenameList() contract.
- Tests: BeginFileEndFileTest (25 cases), re-enabled the POSIX nextfile
  conformance test, new stdin-FNR conformance tests, and wired the
  vendored beginfile1 fixture plus six beginfile2 sub-tests into
  GawkExtensionIT (beginfile1 matches beginfile1.ok byte for byte).
- Docs: README, cli.md, cli-reference.md, extensions.md, java.md,
  java-input.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bertysentry bertysentry linked an issue Jul 15, 2026 that may be closed by this pull request

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 697ebee809

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/main/java/io/jawk/jrt/StreamInputSource.java
gawk resets ERRNO to the empty string whenever the next main input
(file operand or stdin) is selected successfully, not only on the
BEGINFILE per-file path. Mirror that in the ordinary input flow and pin
the behavior with unit tests.

Addresses review feedback on #515.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for BEGINFILE and ENDFILE rules

1 participant