From 1944a041ddd6dcc3e8c26803b2ce8bf3905a5196 Mon Sep 17 00:00:00 2001 From: oisin-m Date: Thu, 30 Jul 2026 13:03:25 +0200 Subject: [PATCH 1/7] feat: add developer instructions --- docs/development/contributing.rst | 51 +++++++++++ docs/development/docs.rst | 34 ------- docs/development/documentation.rst | 88 +++++++++++++++++++ docs/development/guidelines.rst | 7 -- docs/development/index.rst | 22 ++--- .../principles/functional-first.rst | 30 +++++++ docs/development/principles/index.rst | 7 ++ .../principles/maximum-interoperability.rst | 62 +++++++++++++ docs/development/principles/naming.rst | 29 ++++++ docs/development/principles/which-package.rst | 56 ++++++++++++ docs/development/setup.rst | 45 ---------- docs/development/tests.rst | 83 ----------------- docs/index.rst | 13 ++- docs/why.rst | 4 - 14 files changed, 339 insertions(+), 192 deletions(-) create mode 100644 docs/development/contributing.rst delete mode 100644 docs/development/docs.rst create mode 100644 docs/development/documentation.rst delete mode 100644 docs/development/guidelines.rst create mode 100644 docs/development/principles/functional-first.rst create mode 100644 docs/development/principles/index.rst create mode 100644 docs/development/principles/maximum-interoperability.rst create mode 100644 docs/development/principles/naming.rst create mode 100644 docs/development/principles/which-package.rst delete mode 100644 docs/development/setup.rst delete mode 100644 docs/development/tests.rst diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst new file mode 100644 index 0000000..63dfe62 --- /dev/null +++ b/docs/development/contributing.rst @@ -0,0 +1,51 @@ +Contributing +============ + +**earthkit** is an open-source project, and contributions are highly welcomed and appreciated for all components. + +The code is hosted on GitHub: +- `earthkit `_ +- `earthkit-data `_ +- `earthkit-plots `_ +- `earthkit-meteo `_ +- `earthkit-geo `_ +- `earthkit-transforms `_ +- `earthkit-hydro `_ +- `earthkit-utils `_ + + +Development workflow +-------------------- + +1. Fork the repository on GitHub +2. Clone the fork to your local machine +3. Create a virtual environment and install the package in development mode +4. Create a new branch for your changes +5. Make your changes and commit them with a clear message +6. Run tests to ensure everything is working correctly +7. Push your changes to your fork on GitHub +8. Open a pull request against the develop branch of the repository + +Code style +---------- +This project uses ruff for code styling and formatting. To handle these automatically, you can use pre-commit hooks. To set them up, run: + +.. code-block:: bash + + pip install pre-commit + pre-commit install + +Testing +------- +To run the tests, you can use pytest. Make sure you have all dependencies installed, then simply run: + +.. code-block:: bash + + pytest + +Documentation +------------- +To contribute to the documentation, see the developer +resources in the docs: + +- :doc:`documentation` — short guidance for authors (Diátaxis) diff --git a/docs/development/docs.rst b/docs/development/docs.rst deleted file mode 100644 index e4c711a..0000000 --- a/docs/development/docs.rst +++ /dev/null @@ -1,34 +0,0 @@ -.. _dev_docs: - - -Documentation -------------------- - -Building the documentation locally -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To build the documentation locally, please install the Python dependencies first: - -.. code-block:: shell - - cd docs - pip install -r requirements.txt - -Then the documentation can be built by running the following command from the ``docs`` folder: - -.. code-block:: shell - - make html - -To see the generated HTML documentation open the ``docs/_build/html/index.html`` file in your browser. - -Notebook examples -~~~~~~~~~~~~~~~~~~~~~~ - -The notebook examples are located in the ``docs/examples`` folder and are listed in ``docs/examples/index.rst``. - - -If you add a new notebook example please: - -- also add it to ``docs/examples/index.rst`` so that it appears in the documentation. -- ensure the title and the subchapter headings have the same size as in the other notebook. Title with ``##``, subchapter with ``###``. diff --git a/docs/development/documentation.rst b/docs/development/documentation.rst new file mode 100644 index 0000000..b96e18a --- /dev/null +++ b/docs/development/documentation.rst @@ -0,0 +1,88 @@ +Documentation Guidelines +======================== + +earthkit uses the Diátaxis framework, which organises documentation into four complementary types: + +- Tutorials — learning-oriented, step-by-step examples for newcomers. +- How-to Guides — short recipes that solve a specific problem. +- Concepts — conceptual background and rationale. +- API Reference — factual API documentation and exhaustively-listed behaviour. + +In addition to these standard Diátaxis types, earthkit also two additional sections in the User Guide. + +- Installation and Getting Started - a quick way to get up and running. +- Frequently Asked Questions - common questions with short answers + +Where to place content +---------------------- + +- Frequently Asked Questions: only for questions that often come up and have short answers (1-5 lines). If the answer is longer, it should be written elsewhere and linked to from the Frequently Asked Questions. + +- Tutorials: put hands-on, example-driven content. + Preface with a clear goal, required inputs, and a short worked example. + +- How-to Guides: add focused recipes`. Keep them + concise and task-oriented; show the recipe first, then explain options. + +- Concepts: add design rationale and conceptual material. These pages are for readers who want + "why" and "how it works" rather than "what to click". + +- API Reference: API docs should be + generated from docstrings as much as possible. Keep docstrings authoritative and minimal + narrative in the reference pages. + +Writing tips +------------ + +- Title your pages for intent (e.g. "Delineating catchments", not "Notes"). +- Start tutorials with "What you will learn" and a short, copy-pastable + example that runs quickly. +- For how-to guides, lead with the exact commands or code that solves the + task; follow with explanation of options and common pitfalls. +- Use short paragraphs and clear headings; aim for a single idea per + paragraph. +- Prefer concrete examples over abstract descriptions in tutorials and + how-tos. Put conceptual material in Concepts pages. + +Writing examples +---------------- + +- Tutorial (first lines): + + "This tutorial shows how to load a precomputed EFAS river network and + compute catchment areas. By the end you'll have a CSV of catchment stats." + +- How-to (first lines): + + "How to compute upstream accumulation for a field of ones to get upstream + cell counts. Code: + + .. code-block:: python + + import numpy as np + import earthkit.hydro as ekh + + network = ekh.river_network.load('efas', '5') + counts = ekh.upstream.sum(network, np.ones(network.n_nodes))" + +- Concepts (first lines): + + "Distance vs length: distances are edge costs; lengths are node extents. + This difference matters at confluences where multiple edges meet a node." + +Keeping documentation high quality +---------------------------------- + +- Link to a single canonical location for each topic. Avoid duplicate + content across pages. +- When adding API examples, keep them small and runnable and prefer + example snippets that do not require external datasets. +- Submit documentation changes via pull requests and include a short + description that states what changed and why. + +For more detail, read the Diátaxis guide: https://diataxis.fr/ + +Notebooks +--------- + +Ensure the title and the subchapter headings have the same size as in the other notebooks and are recognised and navigable in the documentation. diff --git a/docs/development/guidelines.rst b/docs/development/guidelines.rst deleted file mode 100644 index b180b25..0000000 --- a/docs/development/guidelines.rst +++ /dev/null @@ -1,7 +0,0 @@ -.. _developer_guidelines: - -Development guidelines ------------------------------- - -Guidelines for developers to contribute to ECMWF software are explained in our -`Codex `_. diff --git a/docs/development/index.rst b/docs/development/index.rst index dcfb490..f30bf84 100644 --- a/docs/development/index.rst +++ b/docs/development/index.rst @@ -1,19 +1,11 @@ -Development -=========== +Developer Guide +=============== -The code repository is hosted on `Github`_, testing, bug reports and contributions are highly welcomed and appreciated. Feel free to fork it and submit your PRs against the **develop** branch. - - -Development guide -~~~~~~~~~~~~~~~~~ - .. toctree:: - :maxdepth: 1 - - guidelines - - - + :maxdepth: 2 + :hidden: -.. _`Github`: https://github.com/ecmwf/earthkit + contributing + principle/index + documentation diff --git a/docs/development/principles/functional-first.rst b/docs/development/principles/functional-first.rst new file mode 100644 index 0000000..b10eca8 --- /dev/null +++ b/docs/development/principles/functional-first.rst @@ -0,0 +1,30 @@ +Functional-first design +======================= + +earthkit follows a functional-first design approach. Functionality should +primarily be expressed through functions operating on data, rather than through +object-oriented hierarchies with complex inheritance structures. + +Prefer small, composable functions with clear inputs and outputs:: + + result = earthkit.foo.bar(data, options) + +over stateful objects that hide operations behind methods:: + + result = data.bar(options) + +Functions should: + +* have explicit inputs and outputs +* avoid unnecessary mutable state +* be easy to compose with other functions +* work naturally with different supported data types + +Object-oriented patterns may still be used where they provide a clear benefit, +for example for representing stateful resources, configuration, or complex +lifecycle management. However, new APIs should default to a functional design +unless there is a strong reason to introduce an object abstraction. + +A functional design also supports interoperability by allowing the same +operation to be dispatched across different data backends while keeping the +public API consistent. diff --git a/docs/development/principles/index.rst b/docs/development/principles/index.rst new file mode 100644 index 0000000..a973b22 --- /dev/null +++ b/docs/development/principles/index.rst @@ -0,0 +1,7 @@ +Principles +========== + +General principles/guidelines for developers to contribute to ECMWF software are publicly visible in our +`Codex `_. + +earthkit-specific principles are laid out here and take precedence. diff --git a/docs/development/principles/maximum-interoperability.rst b/docs/development/principles/maximum-interoperability.rst new file mode 100644 index 0000000..5c2ec3a --- /dev/null +++ b/docs/development/principles/maximum-interoperability.rst @@ -0,0 +1,62 @@ +Maximum Interoperability +======================== + +earthkit is not an attempt to reinvent the wheel and retaining interoperability is a key goal. This means both interoperability between earthkit packages, and interoperability with the rest of the Python ecosystem. + +1. Interoperability with the Python ecosystem +--------------------------------------------- + +earthkit aims to integrate naturally with the wider Scientific Python ecosystem. The primary supported data types are: +- xarray +- Array API-compatible arrays (e.g. NumPy, CuPy, PyTorch) +- pandas +- earthkit-data objects + +The public function should contain only dispatch logic. Each backend should +implement the same function with the same signature and semantics. + +A minimal made-up example implementing MSE as a function earthkit.foo.bar with xarray and array implementations:: + + # earthkit.foo + + from earthkit.utils.dispatch import dispatch + + def bar(a, b): + """ + Doc for toplevel implementation. + Links to backend implementations. + """ + dispatched_function = dispatch(bar, fieldlist=False) + return dispatched_function(a, b) + +The backend implementations live in the corresponding submodules:: + + # earthkit.foo.array + + from earthkit.utils.array import array_namespace + + def bar(a, b): + """ + Doc for array implementation. + """ + xp = array_namespace(a, b) + # array-api compat logic + return xp.vector_norm((a-b), ord=2) + + # earthkit.foo.xarray + + def bar(a, b): + """ + Doc for xarray implementation. + """ + # xarray logic + return ((a-b)**2).mean(skipna=False) + +This structure keeps the public API independent of the supported input types, +avoids unnecessary data conversion, and makes it straightforward to add support +for additional backends. + +2. Interoperability between earthkit packages +--------------------------------------------- + +earthkit should work seamlessly as an ecosystem and therefore packages should be easily interoperable between each other by supporting the same data formats, APIs, naming etc. as much as possible. diff --git a/docs/development/principles/naming.rst b/docs/development/principles/naming.rst new file mode 100644 index 0000000..35643f7 --- /dev/null +++ b/docs/development/principles/naming.rst @@ -0,0 +1,29 @@ +Naming conventions +================== + +Consistent naming is important for making earthkit APIs predictable and +easy to discover. Names should follow these principles: + +* Use British English spelling. +* Prefer descriptive names that clearly communicate the purpose of a function, + class, or module. Variables or function arguments can be shorted. +* Follow existing earthkit naming conventions rather than introducing new + patterns. If a convention is problematic, suggest changing it. +* Use terminology that is consistent with the wider scientific Python + ecosystem where appropriate/possible e.g. earthkit-plots follows closely + matplotlib conventions. + +Function names should favour clarity over brevity. Avoid abbreviations unless they are +well established and unambiguous. + +Function arguments on the other hand can be shorter. + +Before introducing a new name, check existing ``earthkit`` APIs and related +packages for similar concepts. Similar operations should use the same names +across modules. + +earthkit vs Earthkit +-------------------- + +In general, earthkit is lower caps when mentioning the repositories and software packages. +It is capitalised when mentioning the project. diff --git a/docs/development/principles/which-package.rst b/docs/development/principles/which-package.rst new file mode 100644 index 0000000..47b4583 --- /dev/null +++ b/docs/development/principles/which-package.rst @@ -0,0 +1,56 @@ +Where should my function go? +============================ + +Functionality in the ``earthkit`` ecosystem is split across a small number of +packages, each with a well-defined scope. Before adding a new function, consider +which package is the most appropriate home. + +* ``earthkit-data`` + + Reading, writing, indexing and manipulating Earth science datasets. This + includes access to local and remote data, file formats, metadata, and data + containers. + +* ``earthkit-geo`` + + Geospatial functionality, including coordinate systems, grids, spatial + operations, interpolation, and geographic utilities. + +* ``earthkit-meteo`` + + Meteorological algorithms and calculations. + +* ``earthkit-hydro`` + + Hydrological algorithms and calculations. + +* ``earthkit-plots`` + + Visualisation and plotting functionality for Earth science data. + +* ``earthkit-transforms`` + + General data transformations that are not specific to a particular scientific + domain e.g. temporal aggregations (climatologies) etc. + +* ``earthkit-utils`` + + Shared utilities used across the ``earthkit`` ecosystem. This package should + contain generic infrastructure (for example dispatch mechanisms, common + decorators and helper utilities) rather than user-facing scientific + functionality. + +General guidelines +------------------ + +* Domain-specific algorithms belong in the relevant domain package (for example, + meteorological calculations in ``earthkit-meteo`` and hydrological + calculations in ``earthkit-hydro``). +* Infrastructure and reusable implementation helpers belong in + ``earthkit-utils``. +* Avoid introducing duplicate functionality across packages. +* If functionality could reasonably fit in more than one package, prefer the + package with the narrower, more natural scope. + +If you are still unsure after reading this guide, please open an issue on +GitHub. diff --git a/docs/development/setup.rst b/docs/development/setup.rst deleted file mode 100644 index 7413361..0000000 --- a/docs/development/setup.rst +++ /dev/null @@ -1,45 +0,0 @@ -.. _dev_setup: - -Development setup with virtualenv ------------------------------------ - -The recommended development environment is a **Python virtual environment**. - -First, clone the repository locally. You can use the following command: - -.. code-block:: shell - - git clone --branch develop git@github.com:ecmwf/earthkit.git - - -Next, create your Python virtual environment and activate it. E.g. assuming your virtual environment is called `earthkit-dev` you can do: - -.. code-block:: shell - - cd YOUR_VENVS_DIR - python -m venv earthkit-dev - source earthkit-dev/bin/activate - -If you use `conda`, the equivalent would be: - -.. code-block:: shell - - conda create -n earthkit-dev python=3.12 - conda activate earthkit-dev - - -Next, install your repo with the development dependencies in editable mode by running the following commands from the root of the repository: - -.. code-block:: shell - - pip install -e . - - -We strongly recommend using the `pre-commit`_ hooks for the developments. These hooks perform a series of quality control checks on every commit. If any of these checks fails the commit will be rejected. To install the hooks run the following commands in the root of the repository: - -.. code-block:: shell - - pip install pre-commit - pre-commit install - -.. _`pre-commit`: https://pre-commit.com/ diff --git a/docs/development/tests.rst b/docs/development/tests.rst deleted file mode 100644 index ec2d7b7..0000000 --- a/docs/development/tests.rst +++ /dev/null @@ -1,83 +0,0 @@ -.. _testing: - -Testing ------------------------ - -To run the test suite, you can use the following command: - -.. code-block:: shell - - pytest - - -Long tests -~~~~~~~~~~~~~ - -Please note that by default all the tests based on remote services e.g. :ref:`data-sources-mars` are skipped. This is done because they can take a very long time to complete or just hang. To enable all these tests you need to run: - -.. code-block:: shell - - pytest -E long -v - -If just want to run e.g. the :ref:`data-sources-mars` tests you can use: - -.. code-block:: shell - - pytest -E long -v -k mars - - -Timeout for CDS tests -~~~~~~~~~~~~~~~~~~~~~~ - -Some :ref:`data-sources-cds` retrieval tests used to hang so an execution timeout was added to all the tests in ``tests/sources/test_cds.py``. The default value is 30 seconds and it can be controlled via the ``--cds-timeout`` custom option to ``pytest``. Please note that some tests have a custom hardcoded timeout value that cannot be changed via this option. - -E.g. to set the timeout to 60 seconds for all CDS tests run (supposing their names contain "test_cds"): - -.. code-block:: shell - - pytest -E long --cds-timeout=60 -v -k test_cds - - -Credentials -~~~~~~~~~~~~~~~ - -Some tests require credentials to access remote services. The existence of credentials are checked and the related tests are skipped accordingly. The logic can be found in ``src/earthkit/data/testing.py``. - -FDB tests -~~~~~~~~~~~~ - -The :ref:`data-sources-fdb` tests are only run if ``pyfdb`` can be imported and the ``FDB_HOME`` environment variable is set. See the logic in ``src/earthkit/data/testing.py``. - - -Optional dependencies -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Tests generally require all optional dependencies to be installed. However, the availability of some optional dependencies are checked and the related tests are skipped accordingly. This is typically used for dependencies which are not part of the ``[all]`` install option (see :ref:`install`). The logic can be found in ``src/earthkit/data/testing.py``. - - -no_cache_init tests -~~~~~~~~~~~~~~~~~~~ - -Tests marked with ``pytest.mark.no_cache_init`` use the ``pytest-forked`` plugin to run in a separate process. They must be run as: - -.. code-block:: shell - - pytest --forked -v -m no_cache_init - - -Notebooks -~~~~~~~~~~~~~ - -The notebook examples from the ``docs/examples`` folder are part of the test suite and are run automatically. However, some notebooks are skipped mainly because they use remote data sources. This is controlled via the ``SKIP`` list in ``tests/documentation/test_notebooks.py``. You can modify this list to add or remove notebooks to be skipped. - -To run only the notebooks tests you can use: - -.. code-block:: shell - - pytest -v -m notebook - - -Documentation code snippets -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -All ``.py`` files in the ``docs`` folder can potentially contain example code snippets and are part of the test suite and run automatically. Many of these are actually not examples and skipped. This is controlled via the ``SKIP`` list in ``tests/documentation/test_examples.py``. You can modify this list to add or remove snippets to be skipped. diff --git a/docs/index.rst b/docs/index.rst index 8c41b31..d29fed5 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -41,8 +41,8 @@ and climate science workflows by simplifying data access, processing, analysis, Step-by-step guides to learn earthkit. -We plan to add many more examples of using Earthkit as a complete package to these pages. In the meantime, we invite you to look at -the documentation for the component Earthkit packages via the navigation drop-down on the left. +We plan to add many more examples of using earthkit as a complete package to these pages. In the meantime, we invite you to look at +the documentation for the component earthkit packages via the navigation drop-down on the left. **Support** @@ -50,7 +50,11 @@ the documentation for the component Earthkit packages via the navigation drop-do Have a feature request or found a bug? Feel free to open an `issue `_. +.. toctree:: + :maxdepth: 2 + :hidden: + why .. toctree:: :caption: User guide @@ -67,8 +71,9 @@ Have a feature request or found a bug? Feel free to open an :caption: Developer guide :hidden: - development/index - + development/contributing + development/principles + development/documentation .. toctree:: diff --git a/docs/why.rst b/docs/why.rst index 5ae518f..0e98b30 100644 --- a/docs/why.rst +++ b/docs/why.rst @@ -1,13 +1,9 @@ Why earthkit? ============= - **earthkit** offers **multiple interoperable** :ref:`software components ` built on top of well-established open-source Python libraries like numpy, pandas and matplotlib. earthkit also integrates and leverages the robust and operations-ready software stack that is familiar to ECMWF production systems (e.g. ecCodes, FDB, etc). The interface of :ref:`earthkit components ` is designed to be high-level in order to provide common tools to support activities across ECMWF and beyond. The design of the components also takes **scalability** into account, so researchers can enjoy efficiency whilst providing easier transfer to operations. - -**earthkit** is still under development but a selection of core components are approaching -a version 1.0.0 release and have release candidates available on PyPi and their GitHub repositories. From 375468b84a3e220afb26cad7613067b445f16d79 Mon Sep 17 00:00:00 2001 From: oisin-m Date: Thu, 30 Jul 2026 13:28:33 +0200 Subject: [PATCH 2/7] feat: refine dev instructions --- docs/development/contributing.rst | 1 + docs/development/documentation.rst | 18 +++++++++--------- docs/development/index.rst | 11 ----------- .../principles/functional-first.rst | 2 +- docs/development/principles/index.rst | 10 +++++++++- .../principles/maximum-interoperability.rst | 7 +++++++ docs/development/principles/naming.rst | 2 +- docs/index.rst | 2 +- 8 files changed, 29 insertions(+), 24 deletions(-) delete mode 100644 docs/development/index.rst diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst index 63dfe62..11084e5 100644 --- a/docs/development/contributing.rst +++ b/docs/development/contributing.rst @@ -4,6 +4,7 @@ Contributing **earthkit** is an open-source project, and contributions are highly welcomed and appreciated for all components. The code is hosted on GitHub: + - `earthkit `_ - `earthkit-data `_ - `earthkit-plots `_ diff --git a/docs/development/documentation.rst b/docs/development/documentation.rst index b96e18a..b8dd716 100644 --- a/docs/development/documentation.rst +++ b/docs/development/documentation.rst @@ -50,25 +50,25 @@ Writing examples - Tutorial (first lines): "This tutorial shows how to load a precomputed EFAS river network and - compute catchment areas. By the end you'll have a CSV of catchment stats." + compute catchment statistics." - How-to (first lines): - "How to compute upstream accumulation for a field of ones to get upstream - cell counts. Code: + "How to load a GRIB file as an xarray Dataset. Code:" .. code-block:: python - import numpy as np - import earthkit.hydro as ekh + import earthkit.data as ekd - network = ekh.river_network.load('efas', '5') - counts = ekh.upstream.sum(network, np.ones(network.n_nodes))" + data = ekd.from_source("file", "temperature.grib") + ds = data.to_xarray() - Concepts (first lines): - "Distance vs length: distances are edge costs; lengths are node extents. - This difference matters at confluences where multiple edges meet a node." + "What is a forecast step? Forecast data has both a reference time and a +forecast step. The valid time is the sum of these two quantities. This +page explains the relationship between reference time, step and valid +time, and why they are represented separately." Keeping documentation high quality ---------------------------------- diff --git a/docs/development/index.rst b/docs/development/index.rst deleted file mode 100644 index f30bf84..0000000 --- a/docs/development/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -Developer Guide -=============== - - -.. toctree:: - :maxdepth: 2 - :hidden: - - contributing - principle/index - documentation diff --git a/docs/development/principles/functional-first.rst b/docs/development/principles/functional-first.rst index b10eca8..87ea629 100644 --- a/docs/development/principles/functional-first.rst +++ b/docs/development/principles/functional-first.rst @@ -1,4 +1,4 @@ -Functional-first design +Functional-first Design ======================= earthkit follows a functional-first design approach. Functionality should diff --git a/docs/development/principles/index.rst b/docs/development/principles/index.rst index a973b22..fdfa839 100644 --- a/docs/development/principles/index.rst +++ b/docs/development/principles/index.rst @@ -4,4 +4,12 @@ Principles General principles/guidelines for developers to contribute to ECMWF software are publicly visible in our `Codex `_. -earthkit-specific principles are laid out here and take precedence. +earthkit-specific principles are also laid out here and take precedence. + +.. toctree:: + :maxdepth: 2 + + maximum-interoperability + naming + which-package + functional-first diff --git a/docs/development/principles/maximum-interoperability.rst b/docs/development/principles/maximum-interoperability.rst index 5c2ec3a..a02a32c 100644 --- a/docs/development/principles/maximum-interoperability.rst +++ b/docs/development/principles/maximum-interoperability.rst @@ -7,6 +7,7 @@ earthkit is not an attempt to reinvent the wheel and retaining interoperability --------------------------------------------- earthkit aims to integrate naturally with the wider Scientific Python ecosystem. The primary supported data types are: + - xarray - Array API-compatible arrays (e.g. NumPy, CuPy, PyTorch) - pandas @@ -43,6 +44,8 @@ The backend implementations live in the corresponding submodules:: # array-api compat logic return xp.vector_norm((a-b), ord=2) + + # earthkit.foo.xarray def bar(a, b): @@ -56,6 +59,10 @@ This structure keeps the public API independent of the supported input types, avoids unnecessary data conversion, and makes it straightforward to add support for additional backends. +.. important:: + + Dispatching and array-api compat both rely on being able to detect the desired backend from inputs. This is not always possible. Numpy is preferred when array-api compat is infeasible, and xarray is preferred for the toplevel function when dispatching is infeasible. + 2. Interoperability between earthkit packages --------------------------------------------- diff --git a/docs/development/principles/naming.rst b/docs/development/principles/naming.rst index 35643f7..6f5b566 100644 --- a/docs/development/principles/naming.rst +++ b/docs/development/principles/naming.rst @@ -1,4 +1,4 @@ -Naming conventions +Naming Conventions ================== Consistent naming is important for making earthkit APIs predictable and diff --git a/docs/index.rst b/docs/index.rst index d29fed5..d502757 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -72,7 +72,7 @@ Have a feature request or found a bug? Feel free to open an :hidden: development/contributing - development/principles + development/principles/index development/documentation From 1069e32b950fea7caa6754f015a21c1dbe39e980 Mon Sep 17 00:00:00 2001 From: oisin-m Date: Tue, 18 Aug 2026 13:35:06 +0200 Subject: [PATCH 3/7] feat: add dependency guidance to dev instructions --- docs/development/principles/dependencies.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 docs/development/principles/dependencies.rst diff --git a/docs/development/principles/dependencies.rst b/docs/development/principles/dependencies.rst new file mode 100644 index 0000000..40f9009 --- /dev/null +++ b/docs/development/principles/dependencies.rst @@ -0,0 +1,13 @@ +Dependencies +============ + +earthkit aims to be interoperable with a wide range of Python libraries. +This means that adding hard dependencies for every supported data type will lead to a growing list of dependencies, despite the fact that most users will have no need for most of them. + + +Each component should therefore keep its core dependencies to a minimum, depending only on libraries that are fundamental to its own functionality. +In particular, NumPy should be considered a core dependency, while other third-party libraries should be added to a component's default dependency set only when they are required by its core functionality. +Support for specific data formats, libraries, or integrations should generally be provided through optional dependencies. + +At the top-level earthkit package, however, we should favour a convenient out-of-the-box experience and include dependencies needed to support the most common use cases. +Thus, while individual components should minimise their default dependencies, the top-level package may aggregate a broader set of optional functionality so that most users can install earthkit and have support for common use cases without installing additional dependencies. From ddeb893527bbc1b10179377f074d1dc6e00e8ba8 Mon Sep 17 00:00:00 2001 From: oisin-m Date: Tue, 18 Aug 2026 13:35:56 +0200 Subject: [PATCH 4/7] fix: add dependencies to index --- docs/development/principles/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/development/principles/index.rst b/docs/development/principles/index.rst index fdfa839..bd10bb5 100644 --- a/docs/development/principles/index.rst +++ b/docs/development/principles/index.rst @@ -13,3 +13,4 @@ earthkit-specific principles are also laid out here and take precedence. naming which-package functional-first + dependencies From e6a515c5f94c5bfe639b0cca28be9c6e6b81a491 Mon Sep 17 00:00:00 2001 From: oisin-m Date: Mon, 24 Aug 2026 14:49:39 +0200 Subject: [PATCH 5/7] docs: wrong example import for dispatch --- docs/development/principles/maximum-interoperability.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/principles/maximum-interoperability.rst b/docs/development/principles/maximum-interoperability.rst index a02a32c..7edb4f6 100644 --- a/docs/development/principles/maximum-interoperability.rst +++ b/docs/development/principles/maximum-interoperability.rst @@ -20,7 +20,7 @@ A minimal made-up example implementing MSE as a function earthkit.foo.bar with x # earthkit.foo - from earthkit.utils.dispatch import dispatch + from earthkit.utils.decorators import dispatch def bar(a, b): """ From 21b981153c71d88b846bb233f067704f67ba4cbd Mon Sep 17 00:00:00 2001 From: oisin-m Date: Mon, 24 Aug 2026 16:01:18 +0200 Subject: [PATCH 6/7] feat: add one ground truth principle --- .../principles/one-ground-truth.rst | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 docs/development/principles/one-ground-truth.rst diff --git a/docs/development/principles/one-ground-truth.rst b/docs/development/principles/one-ground-truth.rst new file mode 100644 index 0000000..db9a6eb --- /dev/null +++ b/docs/development/principles/one-ground-truth.rst @@ -0,0 +1,32 @@ +One Ground Truth +================ + +Where earthkit supports several data types for the same operation, that operation should +have a single canonical implementation. The other interfaces — xarray, FieldList, pandas +— are thin wrappers around it, not re-implementations of the same algorithm. This +complements :doc:`maximum-interoperability`: the maths should only be written once, or +the copies drift apart until their results diverge. + +Choosing the ground truth +------------------------- + +Implement at the lowest natural level so the other interfaces derive from it. For most +numerical operations this is the array implementation, written against the Array API (via +``earthkit.utils.array.array_namespace``) so it covers NumPy, CuPy and PyTorch. Where an +operation only makes sense with labelled dimensions, xarray is a more natural ground +truth and the others wrap that instead. + +Wrapping outwards +----------------- + +``earthkit-utils`` provides the wrappers, so this glue rarely needs writing by hand:: + + from earthkit.utils.decorators import xarray_ufunc, fieldlist_ufunc + +``xarray_ufunc`` runs the array function through ``xarray.apply_ufunc`` for the xarray +interface; ``fieldlist_ufunc`` applies it to ``Field``/``FieldList`` values and +reattaches metadata. Dispatch (see :doc:`maximum-interoperability`) routes each caller to +the right wrapper. + +A separate backend implementation is warranted only when a backend needs genuinely +different logic, not merely a different container. From 02528f3d186d9e682ec4a7e08538aba58a067fc8 Mon Sep 17 00:00:00 2001 From: oisin-m Date: Mon, 24 Aug 2026 16:16:08 +0200 Subject: [PATCH 7/7] fix: add to index and add links --- docs/development/principles/functional-first.rst | 2 +- docs/development/principles/index.rst | 2 ++ .../principles/maximum-interoperability.rst | 4 +++- docs/development/principles/missing-values.rst | 14 ++++++++++++++ 4 files changed, 20 insertions(+), 2 deletions(-) create mode 100644 docs/development/principles/missing-values.rst diff --git a/docs/development/principles/functional-first.rst b/docs/development/principles/functional-first.rst index 87ea629..cb398e0 100644 --- a/docs/development/principles/functional-first.rst +++ b/docs/development/principles/functional-first.rst @@ -15,7 +15,7 @@ over stateful objects that hide operations behind methods:: Functions should: -* have explicit inputs and outputs +* have explicit inputs and outputs i.e. avoid `**kwargs` * avoid unnecessary mutable state * be easy to compose with other functions * work naturally with different supported data types diff --git a/docs/development/principles/index.rst b/docs/development/principles/index.rst index bd10bb5..1b97581 100644 --- a/docs/development/principles/index.rst +++ b/docs/development/principles/index.rst @@ -10,7 +10,9 @@ earthkit-specific principles are also laid out here and take precedence. :maxdepth: 2 maximum-interoperability + one-ground-truth naming which-package functional-first dependencies + missing-values diff --git a/docs/development/principles/maximum-interoperability.rst b/docs/development/principles/maximum-interoperability.rst index 7edb4f6..56aa9da 100644 --- a/docs/development/principles/maximum-interoperability.rst +++ b/docs/development/principles/maximum-interoperability.rst @@ -57,7 +57,9 @@ The backend implementations live in the corresponding submodules:: This structure keeps the public API independent of the supported input types, avoids unnecessary data conversion, and makes it straightforward to add support -for additional backends. +for additional backends. Read :doc:`one-ground-truth` for guidance regarding how to +handle multiple implementations with minimal maintenance burden e.g. xarray and array +above. .. important:: diff --git a/docs/development/principles/missing-values.rst b/docs/development/principles/missing-values.rst new file mode 100644 index 0000000..a5d39c1 --- /dev/null +++ b/docs/development/principles/missing-values.rst @@ -0,0 +1,14 @@ +Missing Value Handling +====================== + +The default policy for missing values should be to propagate i.e. not to omit. +Note in particular that this is the opposite of xarray conventions. +If missing value handling needs to be configurable, it should follow scipy conventions. + + nan_policy: {‘propagate’, ‘omit’, ‘raise’} + Defines how to handle input NaNs. + * propagate: if a NaN is present in the axis slice (e.g. row) along which the statistic is computed, the corresponding entry of the output will be NaN. + * omit: NaNs will be omitted when performing the calculation. If insufficient data remains in the axis slice along which the statistic is computed, the corresponding entry of the output will be NaN. + * raise: if a NaN is present, a ValueError will be raised. + +In complex multidimensional-data cases, it may be unclear to what axes propagation/omission etc. should apply to. Such cases warrant wider discussion.