Skip to content

fix: two SIGFPE crashes, an unbounded retry loop, and a sub-minimum GNFS abort - #98

Merged
bbuhrow merged 5 commits into
bbuhrow:masterfrom
KevinLee1231:pr/defect-fixes-batch1
Oct 5, 2026
Merged

bbuhrow merged 5 commits into
bbuhrow:masterfrom
KevinLee1231:pr/defect-fixes-batch1

Conversation

@KevinLee1231

Copy link
Copy Markdown
Contributor

Four independent input-handling defects, each of which crashes or hangs yafu. They are unrelated to each other, so they are easy to review one at a time.

1. pp1 and trial read the wrong operand slots

Arguments are placed according to the declared arity in function_nargs[]: with arity na and k arguments supplied, the first argument lands in operands[na-k] and the last in operands[na-1]. Two cases disagreed with that table.

pp1 was declared with arity 3 while its case handles at most two arguments, so the shift was one slot off for every call shape. The case reads operands[0] and operands[1], so new_factorization() was handed an uninitialized mpz (0), and print_factors() then divided by the zero product of the resulting factor list. GMP raises SIGFPE, so every pp1 call died with exit 136:

printf 'pp1(18446744073709551617)\n' | yafu     -> SIGFPE

Declaring arity 2 puts both call shapes into the slots the case reads. trial had the mirror-image bug in its one-argument branch only: arity is 2, so a single argument sits in operands[1], but that branch passed operands[0] to new_factorization():

printf 'trial(15)\n' | yafu                    -> SIGFPE

The placement rule is now documented above the table so the next case does not have to rediscover it.

2. print_factors divides by a zero factor product

print_factors() builds prod from the factor list and divides flist->N by it without checking. A zero-valued factor makes prod zero, and GMP raises SIGFPE from mpz_tdiv_r, taking the program down with no diagnostic at all. That is exactly how the pp1/trial misplacement above presented itself - an uninitialized operand reached new_factorization(), was recorded as a factor, and the crash surfaced here:

#5 __gmpz_tdiv_r
#6 print_factors (fobj=...) at factor/factor_common.c:1431

The bogus list is now reported and the remainder bookkeeping is skipped, so a bad factor list becomes a visible error message rather than a signal. An empty list is unaffected (its product is 1), which is what the surrounding check already relies on.

3. Unbounded tiny-ecm retry in the siqs special-case path

check_specialcase() finishes inputs below 115 bits with tiny-qs, and when that leaves a composite behind it retries tiny-ecm in an unbounded loop. Tiny-ecm cannot always succeed: when every factor of the remaining composite is small enough that its curve group order divides the stage-1 exponent, each curve collapses to gcd == n, which the caller correctly rejects as trivial. 15 is such a case, and the loop spun on it at 100% CPU forever:

printf 'siqs(15)\n' | yafu      -> never returns

The retry is now bounded at 100 attempts, after which the job finishes with deterministic methods: trial division to 1e5 via zTrial(), then Pollard rho via brent_loop() for anything still composite. Both append to fobj->factors and reduce their own gmp_n the same way tiny-ecm did, so the surrounding code needs no change. A prime remainder is added to the factor list directly.

4. Inputs below the GNFS polynomial-search minimum

The polynomial-search parameter tables begin at 80 digits (params_deg4 in nfs_poly.c) and the search compares log10(n) against that bound, so an input needs at least 81 decimal digits. Below that get_default_params() returns without filling anything in, leaving stage2_norm at zero, and the msieve code underneath answers with error: stage 2 size bound not provided and exit(-1) - killing the program from an explicit nfs() call on a number that SIQS factors in a fraction of a second:

printf 'nfs(<60 digits>)\n' | yafu    -> exit 255

nfs() now checks the size against NFS_GNFS_POLY_MIN_DIGITS before starting a GNFS job and hands anything smaller to SIQS, the same fallback the crossover check below it already used. The crossover is about which method is faster; this is about which method is possible at all, so it is checked first:

nfs: input of size 60 is below the GNFS polynomial-search minimum of 81 digits, using siqs
... = 618970019642690137449562111 * 162259276829213363391578010288127

do_msieve_polyselect() additionally zero-initializes its parameter struct and returns with a message if the table left it empty. That keeps the same abort from coming back through any other route into poly search, and covers the far end of the range too - the struct was uninitialized stack memory before, so a guard on it could not have been relied on.

Testing

Built from source on Linux (GCC 16, make yafu) and checked with:

pp1(19327352823000000088046829527, 2) -> 2147483647 * 9000000000000000041
trial(15)                             -> 15 = 3 * 5
trial(3000018000027, 3000000)         -> 3 * 1000003 * 1000003
trial(18446744073709551617)           -> reports C20, no longer dies
siqs(15)                              -> 15 = 3 * 5
siqs(21)                              -> 21 = 3 * 7
siqs(18446744073709551617)            -> 274177 * 67280421310721

pp1 output stays probabilistic - a p+1 curve can degenerate to gcd == n - so a single-base call may legitimately report nothing.

GNFS itself is unaffected: an 83-digit job still completes end to end (polynomial search -> lattice sieving -> msieve filtering -> LA -> sqrt):

nfs(30000000000000000000000000000000000000047800000000000000000000000000000000000016459)
  = 100000000000000000000000000000000000000109 * 300000000000000000000000000000000000000151

I am putting together a separate change that adds a regression harness, so the reproductions above are manual for now. Happy to split this into four PRs if that is easier to review.

KevinLee1231 and others added 5 commits October 3, 2026 19:03
print_factors() builds prod from the factor list and divides flist->N by it
without checking.  A zero-valued factor in the list makes prod zero, and GMP
raises SIGFPE from mpz_tdiv_r: the program dies with no diagnostic at all.
That is exactly how the pp1/trial operand misplacement presented itself - an
uninitialized operand reached new_factorization(), which recorded it as a
factor, and the crash surfaced here:

  bbuhrow#5 __gmpz_tdiv_r
  bbuhrow#6 print_factors (fobj=...) at factor/factor_common.c:1431

Report the bogus list and skip the remainder bookkeeping instead, so a bad
factor list becomes a visible error message rather than a signal.  An empty
list is not affected (its product is 1), which is what the surrounding
"product of factors doesn't equal input" check below already relies on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Arguments are placed according to the declared arity in function_nargs[]: with
arity na and k arguments supplied, the first argument lands in operands[na-k]
and the last in operands[na-1].  Two cases disagreed with the table.

pp1 was declared with arity 3 while its case handles at most two arguments, so
the shift was one slot off for every call shape:

  pp1(n, bases) -> arguments in operands[1], operands[2]; operands[0] untouched
  pp1(n)        -> argument in operands[2]; operands[0] and [1] untouched

The case reads operands[0] and operands[1], so new_factorization() was handed
an uninitialized mpz (0) and print_factors() then divided by the zero product
of the resulting factor list.  GMP raises SIGFPE, so every pp1 call died with
exit 136 before doing any work:

  printf 'pp1(18446744073709551617)\n' | yafu     -> SIGFPE

Declaring arity 2 puts both call shapes into the slots the case reads: a
two-argument call now fills operands[0] and operands[1], and a one-argument
call fills operands[1], which is what the one-argument branch already used.

trial had the mirror-image bug in its one-argument branch only: arity is 2, so
a single argument sits in operands[1], but that branch passed operands[0] to
new_factorization() (while correctly taking operands[1] for div_obj).  Use
operands[1] for both.  The crash was the same:

  printf 'trial(15)\n' | yafu                    -> SIGFPE

Verified:

  pp1(19327352823000000088046829527, 2) -> 2147483647 * 9000000000000000041
  trial(15)                             -> 15 = 3 * 5
  trial(3000018000027, 3000000)         -> 3 * 1000003 * 1000003
  trial(18446744073709551617)           -> reports C20, no longer dies
                                           (274177 is above the default
                                           trial bound of 10000)

pp1 output stays probabilistic - a p+1 curve can degenerate to gcd == n - so a
single-base call may legitimately report nothing.

Document the placement rule above the table so the next case does not have to
rediscover it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check_specialcase() finishes inputs below 115 bits with tiny-qs, and when that
leaves a composite behind it retries tiny-ecm in an unbounded loop:

    do { found = getfactor_tecm(fobj->qs_obj.gmp_n, f1, 0, &lcg); ... }
    while (!found);

Tiny-ecm cannot always succeed.  When every factor of the remaining composite
is small enough that its curve group order divides the stage-1 exponent, each
curve collapses to gcd == n, which the caller correctly rejects as trivial, and
no amount of retrying will do better.  15 is such a case, and the loop spun on
it at 100% CPU forever - gdb sampling put it in ecm_stage2 -> mulmod128 with a
modulus of 15:

  printf 'siqs(15)\n' | yafu      -> never returns

Bound the retry at 100 attempts and finish with deterministic methods when
tiny-ecm gives up: trial division to 1e5 via zTrial(), then Pollard rho via
brent_loop() for anything still composite.  Both append to fobj->factors and
reduce their own gmp_n the same way tiny-ecm did, so the surrounding code needs
no change.  A prime remainder is added to the factor list directly.

Verified:

  siqs(15)              -> 15 = 3 * 5
  siqs(21)              -> 21 = 3 * 7
  siqs(18446744073709551617) -> 274177 * 67280421310721

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The polynomial-search parameter tables begin at 80 digits (params_deg4 in
nfs_poly.c) and the search compares log10(n) against that bound, so an input
needs at least 81 decimal digits.  Below that get_default_params() returns
without filling anything in, leaving stage2_norm at zero, and the msieve code
underneath answers with

    error: stage 2 size bound not provided

and exit(-1), killing the program - from an explicit nfs() call on a number
that SIQS factors in a fraction of a second:

  printf 'nfs(<60 digits>)\n' | yafu    -> exit 255

nfs() now checks the size against NFS_GNFS_POLY_MIN_DIGITS before starting a
GNFS job and hands anything smaller to SIQS, the same fallback the crossover
check below it already used.  The crossover is about which method is faster;
this is about which method is possible at all, so it is checked first:

  nfs: input of size 60 is below the GNFS polynomial-search minimum of 81
  digits, using siqs
  ... = 618970019642690137449562111 * 162259276829213363391578010288127

do_msieve_polyselect() additionally zero-initializes its parameter struct and
returns with a message if the table left it empty.  That keeps the same abort
from coming back through any other route into poly search, and covers the far
end of the range too - the struct was uninitialized stack memory before, so the
guard could not have been relied on.

GNFS itself is unaffected: an 83-digit job still completes end to end
(polynomial search -> lattice sieving -> msieve filtering -> LA -> sqrt),
verified in 69-114 seconds depending on which parameter table is in use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard prints "degree-%d table starts at %.0f digits" using pref_degree,
which can be 5 or 6, but always takes the bound from params_deg4.  Select
the table by pref_degree so the message matches the degree it names.
@bbuhrow

bbuhrow commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Thank you for these fixes!

@bbuhrow
bbuhrow merged commit dca9162 into bbuhrow:master Oct 5, 2026
12 checks passed
@KevinLee1231
KevinLee1231 deleted the pr/defect-fixes-batch1 branch October 5, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants