Skip to content

fix: evaluate the Lanczos sum in a well-conditioned form#416

Draft
agene0001 wants to merge 2 commits into
statrs-dev:mainfrom
agene0001:feat/lanczos-single-fraction
Draft

fix: evaluate the Lanczos sum in a well-conditioned form#416
agene0001 wants to merge 2 commits into
statrs-dev:mainfrom
agene0001:feat/lanczos-single-fraction

Conversation

@agene0001

Copy link
Copy Markdown

Stacked on #410 (needs its tightened ulps_eq! for the digamma pole test); review only the last commit until that merges.

ln_gamma/gamma evaluate Pugh's Lanczos approximation as the partial-fraction sum d₀ + Σ dₖ/(z+k−1). The residues alternate in sign, so the sum cancels badly — condition number ~3600 around z = 50, i.e. ~440 eps of relative error in the sum alone.

The same rational function written as a single fraction N(z)/D(z), derived from those residues in exact rational arithmetic, has all-positive coefficients in both polynomials (D(z) = z(z+1)…(z+9), whose expanded coefficients are unsigned Stirling numbers of the first kind, exact in f64). For z > 0 both Horner evaluations then have condition number 1. No new external constants — they are derived from the residues already in the file, and agree with the partial-fraction form to 3.2e-17 over [0.5, 3000]. A reversed form in 1/z covers z ≥ 1e29 where z¹⁰ would overflow.

Also in the same evaluation:

  • lanczos_power compensates the base of ((p+g)/e)^ppowf amplifies base error by the exponent, the dominant remaining term (~190 ulp by x = 122). The correction is first-order and gated on staying small, since applied blindly it can flip the sign of an overflowing result.
  • Integer arguments ≤ 171 come from the exact factorial table (Γ(2) == 1 exactly, was 1.0000000000000002).
  • gamma above x ≈ 169.7 halves the exponent and squares: the power alone overflows there while the product is representable to x ≈ 171.61. gamma(171.6) was inf, now 1.5858969096673e308 (mpmath: 1.5858969096673029e308).
  • sin_pi/tan_pi reduce by the period before sin/tanx − round(x) is exact, so reflection-formula error stays relative to the fractional part instead of ulp(π)/dist_to_pole.

Measured against mpmath at 45 dps:

before after
ln_gamma p99 45.9 ulp 8.2 ulp
gamma (positive), median 285 ulp 19 ulp
gamma (negative), median/max 109 / 1173 ulp 3.4 / 22.8 ulp
digamma (negative), median/max 19.5 / 1.5e10 ulp 1.6 / 538 ulp

The remaining ~19–35 ulp in gamma is the approximation floor of the f64-rounded Pugh coefficient set itself (~3.7e-15 — confirmed by evaluating the formula in exact arithmetic), so evaluation error is now below the fit.

Four test expectations move as a consequence, each verified against mpmath: one Binomial::sf and two FisherSnedecor literals were fitted to the old output, and the Dirichlet/MultivariateStudent doctest values were 34 and 22 ulp from truth (new outputs are within 2).

🤖 Generated with Claude Code

`approx`'s `ulps_eq` short-circuits on `abs_diff_eq(epsilon)` before it
consults the ULPs distance. The crate's wrapper paired it with
`DEFAULT_EPS = 1e-9`, which made the ULPs bound unreachable, so every internal
`ulps_eq!(x, y)` was really a 1e-9 absolute comparison.

That matters because the macro is used to recognise *exact* parameter values, so
anything within 1e-9 of them silently took a degenerate branch:

    Binomial(p = 1 - 2^-33, n = 100).pmf(99)   0    (true 1.16e-8)
    Binomial(p = 1 - 2^-33, n = 100).ln_pmf(99) -inf
    Geometric(p = 1 - 2^-33).max()             1    (true u64::MAX)
    Geometric(p = 1 - 2^-33).skewness()        inf  (true 92681.9)
    Beta(1 + 2^-33, 1 + 2^-33).pdf(0.0)        1    (true 0)
    digamma(-1 + 2^-33)                        -inf (true -8.59e9)

Adds `DEFAULT_ULPS_EPS = f64::EPSILON` and uses it for `ulps_eq!` and
`assert_ulps_eq!`. The exact values still take the degenerate branches -
`Binomial(1.0, 5).pmf(5) == 1`, `Geometric(1.0).max() == 1`, `digamma` at the
poles is still -inf - all covered by existing tests.

Tests use `1 - 2^-33` rather than `1 - 1e-10` so that `1 - p` is exact and the
references are not limited by the representation of `p`.

Also fixes `Binomial::entropy`, which summed `-p * ln(p)` unguarded and so
returned `NaN` once any mass underflowed to zero (`0 * -inf`) - for `p = 0.5`
that is every `n` past about 1100. `Categorical::entropy` already filtered zero
terms; this brings `Binomial` in line.
`ln_gamma`/`gamma` evaluate Pugh's 10-term Lanczos approximation as the
partial-fraction sum `d_0 + sum_k d_k / (z + k - 1)`. The residues alternate in
sign, so that sum cancels badly - condition number ~3600 around z = 50, i.e.
~440 eps of relative error in the sum alone.

The same quantity as a single fraction `N(z) / D(z)`, derived from those residues
in exact rational arithmetic, has *all-positive* coefficients in both numerator
and denominator (`D(z) = z (z+1) ... (z+9)`, whose expanded coefficients are
unsigned Stirling numbers of the first kind and exact in f64). For z > 0 both
Horner evaluations then have condition number 1. This needs no new external
constants - they are derived from the residues already in the file - and agrees
with the partial-fraction form to 3.2e-17 over [0.5, 3000]. A reversed form in
1/z covers z >= 1e29, where z^10 would overflow.

Two further fixes in the same evaluation:

  * `lanczos_power` compensates the base of `((p + g) / e)^p`. `powf` amplifies a
    relative error in its base by the exponent, so the two roundings in
    `(p + g) / e` were the dominant error (~190 ulp by x = 122). The residuals of
    the addition and the division - the latter against a double-double `e` - are
    recovered exactly and applied as a first-order correction, which is only
    valid while it stays small, so it is gated on that.
  * `gamma`/`ln_gamma` at the positive integers up to 171 come from the exact
    factorial table, which also makes `Gamma(2) == 1` rather than
    1.0000000000000002.
  * `gamma` for x above ~169.7 halves the exponent and squares, because the power
    alone overflows there while the full product stays representable to
    x ~ 171.61. `gamma(171.6)` was `inf`; it is now 1.5858969096673e308.

`sin_pi`/`tan_pi` reduce by the period before calling `sin`/`tan`. `x - round(x)`
is exact, so the error stays relative to the fractional part instead of being
`ulp(PI) / dist_to_pole`, which near the poles cost several decimal digits.

Measured against mpmath at 45 dps:

                          before                after
    ln_gamma p99          45.9 ulp              8.2 ulp
    gamma (positive)      285 median            19 median
    gamma (negative)      109 median, 1173 max  3.4 median, 22.8 max
    digamma (negative)    19.5 median, 1.5e10   1.6 median, 538 max

`gamma`'s remaining ~19-35 ulp is the approximation floor of the f64-rounded Pugh
coefficient set itself (~3.7e-15, confirmed by evaluating the formula in exact
arithmetic), so the evaluation is now compensated to well below the fit.

Four test expectations move as a consequence, each verified against mpmath: one
`Binomial::sf` and two `FisherSnedecor` pdf/ln_pdf literals were fitted to the
old output, and the `Dirichlet`/`MultivariateStudent` doctest values were 34 and
22 ulp from truth (the new outputs are within 2).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant