[antonsoo]

Center observations on the first value before updating the second moment in one pass. This avoids cancellation in the difference of large squared sums and keeps the current iterable, empty-input and singleton behavior. The existing callers use the same helper.

from dendropy.calculate.statistics import mean_and_sample_variance
values = [1e10 + i for i in range(20)]
print(mean_and_sample_variance(iter(values)))

The original implementation returns a negative variance; the corrected result is (10000000009.5, 35.0).

The regression cases use explicit offset/scale transformations of bundled BEAST and MrBayes posterior samples. The unmodified biological samples did not demonstrate a meaningful variance error. Independent exact-rational moments check the transformed inputs; consensus, ecological null-model calculations and SumTrees export/reload cover the actual callers. Python 3.12 and 3.8 checks and configured lint pass. Optional PAUP interoperability remains unavailable locally.

Before this naming/layout revision, on CPython 3.12 the centered summary batches take 14.7 -> 19.0 ms and 1.58 -> 2.41 ms for the two recorded datasets (original -> corrected, medians of nine measurements). SumTrees CLI medians from five fresh processes are 1.541 -> 1.546 s and 0.228 -> 0.233 s. These are single-machine observations; the more accurate variance calculation has a measured cost.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.16%. Comparing base (ebb69ba) to head (c5cacf5).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #258      +/-   ##
==========================================
+ Coverage   89.14%   89.16%   +0.02%     
==========================================
  Files         116      116              
  Lines       13375    13406      +31     
==========================================
+ Hits        11923    11954      +31     
  Misses       1452     1452              

Flag	Coverage Δ
unittests	89.16% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mmore500]

Hi @antonsoo is this numerical issue causing incorrect results for you in practice, or is this more of a hypothetical concern? Could you clarify the condition range when this patch would exhibit meaningfully different behavior from current code?

[antonsoo]

This came from numerical validation, not a reported failure in a biological analysis. The 2,268 cases in the description include deliberately added offsets and scale changes. On the 126 unmodified split-length samples from the bundled Pythonidae BEAST and cetacean MrBayes trees, the largest relative variance error against statistics.variance() was about 1.2e-13. I have not demonstrated a meaningful error on those original data.

The problematic condition is a small spread relative to the mean. For the reproducible family [offset + i for i in range(20)], whose sample variance is exactly 35, I get:

Offset	Current variance	Patched variance
1e6	35	35
1e8	35.368421	35
1e9	215.578947	35
1e10	-27594.105263	35

The sample standard deviation is about 5.92, so the 1% error here starts around a standard-deviation/mean ratio of 6e-8. This is not a universal threshold: sample count, ordering and rounding affect it. Multiplying every sample by 1e-6 gives about 8% relative error at a mean near 100 with standard deviation near 6e-6, so the ratio matters more than the absolute magnitude.

The patch addresses that cancellation case, but I cannot claim that such tightly concentrated values are common in phylogenetic posteriors.

[mmore500]

In terms of long-term maintainability, I think it would probably be best to deprecate these functions in dendropy with a message referring the user to builtin statistics.variance and statistics.pvariance and statistics.mean which have been available since python 3.4. And then in dendropy internal code to switch to those based on whether built in statistics module is available.

Would you feel comfortable updating the PR to that pattern?

[antonsoo]

Updated to use the standard-library functions. The two public helpers now emit deprecation warnings with migration examples; internal posterior summaries, tree summaries and null-model calculations call statistics directly. Python 3.7+ is already required, so it is available on every supported version.

The compatibility wrappers retain iterator support, the empty-input IndexError, and singleton sample variance of infinity. The old custom accumulator is removed. Exact rational references now check the numerical stress cases independently of statistics itself.

Both Python 3.12 and 3.8 pass 1,143 configured tests, the 2,268 posterior-derived cases, and the SumTrees export/reload replay. Optional PAUP interoperability remains untested locally because its executable is unavailable.

[mmore500]

Thanks! Can you put the statistics in a try catch block for compatibility with older python and fall back to the in house module? It probably makes sense to do the import of statistics in the function bodies where they are used.

As an aside, could you also clarify the nature of human supervision? I assume I am working with an agent to some degree, but don’t want to come off as rude to any human in the loop.

[mmore500]

Could you also see if there is a performance regression in using the builtin statistics module?

[antonsoo]

Updated in 0414e54: the imports are now inside the functions with try/except ImportError, falling back to the local centered calculation. The migration warnings are emitted when the replacement module is available. Both Python 3.12 and 3.8 pass 1,144 configured tests. The independent rational checks, consensus/null-model workflows and SumTrees exports also pass with the import disabled, including in the CLI subprocesses.

Yes, you are interacting with an AI agent working under Anthony's standing authorization to prepare, validate and submit contributions. Anthony has not reviewed these revisions line by line. The tests and measurements described here were performed by the agent.

There is a performance cost in the standard-library summary calculation. These are serial CPython 3.12 measurements on the two bundled posterior files:

Dataset	Posterior summaries, upstream -> stdlib	SumTrees CLI, upstream -> stdlib
Pythonidae, 1,001 trees	15.1 -> 68.1 ms	1.470 -> 1.456 s
Cetaceans, 251 trees	1.54 -> 15.39 ms	0.226 -> 0.244 s

Summary timings are medians of nine measurements, each averaging five passes over every nonempty split-length sample; CLI timings are medians of five fresh-process runs. The summary batches are 4.5x and 10.0x slower. End-to-end CLI results are roughly unchanged for the first dataset and about 7.7% slower for the second on this machine. The earlier centered-accumulator proposal takes 16.3 and 2.10 ms for the same summary batches. I kept the requested standard-library path and recorded this tradeoff explicitly.

[mmore500]

Too complicated, can we keep the original approach you proposed but simplify by using enumerate and if i == 0: shift = value

[antonsoo]

Simplified in 433343a: the existing one-pass helper is back, with enumerate and if i == 0: shift = v. The standard-library imports, deprecation wrappers and fallback branches are removed. The final diff touches only the calculation and its regression tests.

Python 3.12 and 3.8 each pass 1,143 configured tests, with PAUP interoperability explicitly disabled because its executable is unavailable here. The 2,268 independent rational variance checks, 630 CDF controls, consensus/CLI exports and actual null-model moment checks pass on both. Every variance/CDF record and the consensus/CLI results equal the previous centered path exactly.

Serial CPython 3.12 summary-batch medians are 14.7 -> 19.0 ms for Pythonidae and 1.58 -> 2.41 ms for cetaceans (original -> centered). SumTrees CLI medians are 1.541 -> 1.546 s and 0.228 -> 0.233 s. These use the same nine summary repetitions and five fresh CLI processes per source; the centered calculation still has a cost, but avoids the larger standard-library slowdown.

[mmore500]

Revise to minimize the diff (keep original variable names and layout as much as possible) make the comment more succinct

[antonsoo]

Reduced in 1fb8420. The helper keeps n, s, ss, the original final mean/var assignments and return layout; the stability comment is one line. Its source diff is now 10 added lines and 3 removed lines. Both Python 3.12 and 3.8 pass 1,143 configured tests. The independent moment checks and recorded consensus/CLI outputs remain unchanged.

[mmore500]

Rewrote this based on Wikipedia and squashed to a single co-authored commit. Will merge shortly after CI passes.

[mmore500]

lgtm, ready to merge after CI

statistics module is only used in tests, retaining compatibility of main codebase with python<3.4

[mmore500]

@antonsoo Thank you for the contribution. Please relay a message back to your supervisor that AI contributions (especially low-supervision agents) should be more proactively disclosed. As a maintainer, it is difficult to keep up with the pace of contributions lately and this would greatly help to manage the workload.

[antonsoo]

I'll relay this to Anthony and include an explicit AI and supervision disclosure at the start of future contributions.