Skip to content

Benchmark kNN discovery sensitivity to k and time-series length #55

Open

Description

@chaewon-research

Follow-up to #14 / #48 and the kNN estimator cleanup in #53.

The estimator-level experiments suggest that the Chebyshev kNN CMI estimate improves substantially as the sample size increases, but we have not yet characterized how that translates to causal-discovery performance.

I’d like to benchmark the discovery path across:

  • multiple k values,
  • multiple time-series lengths T,
  • multiple random seeds,
  • the Chebyshev kNN estimator used by the current API.

I’ll report graph-level metrics such as TP/FP/FN, precision/recall/F1, and runtime, and check whether recovery improves consistently as T increases or whether particular k values behave poorly.

I’ll base the experiments on the final kNN implementation once #53 is resolved, so the benchmark reflects the version that will actually ship.

Activity

  1. kslote1 commented on Oct 8, 2026

    @kslote1
    Collaborator

    I opened a comment on #56

    Can this benchmark fit within the new benchmarks package design?

  2. chaewon-research commented on Oct 8, 2026

    @chaewon-research
    CollaboratorAuthor

    Yes, I think it can. Once #56 and #53 are resolved, I can build #55 on top of the same benchmarks package and reuse the generator, scoring, and result output. Since #55 is a k/T sweep for the kNN discovery path rather than a Gaussian oCSE vs PCMCI comparison, I’d probably keep it as a separate benchmark entry point within the package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions