Skip to content

ENH: Portable state persistence across machines #735

New issue
New issue
Closed
#743
Closed
ENH: Portable state persistence across machines#735
#743
Labels
enhancementNew feature or request

Description

@DrorSh
DrorSh
opened on Dec 23, 2025

Research workflows often involve costly builds (large data pulls, preprocessing, model fitting). In collaborative or multi-machine setups, it would be extremely useful to persist task state across machines in a way that is portable, explicit, and git-friendly.

At the moment, (to my understanding) the available options are:

  1. Track the .pytask/ directory (including the SQLite database) - not great over git, requires bypassing pytask's own gitignore rules.
  2. Mark individual tasks as persisting - This avoids recomputation but can introduce breaks in the DAG and undermines some of pytask’s core strengths around dependency tracking and reproducibility.

Neither option works well for collaborative research projects where multiple contributors need to share task state reliably.

Conceptually, a solution could look like one of the following (or something similar):

  • A structured, human-readable state backend (e.g. JSON) that can be explicitly stored outside .pytask/
  • An explicit pytask state export and import workflow
  • A configurable state backend (local, file-based, cloud-based) with a stable public format
  • Support for scoping persistence (e.g. by module, task group, or tag)

Key properties that would make this useful:

  • Portable across machines and OSs
  • Safe to commit to git (or other VCS)
  • Preserves the integrity of the DAG
  • Explicit and opt-in

Activity

  1. DrorSh
    added
    enhancementNew feature or request
    on Dec 23, 2025

  2. tobiasraabe commented on Dec 26, 2025

    @tobiasraabe
    tobiasraabe
    on Dec 26, 2025
    Member

    Hi @DrorSh! Thanks for issue! I always wanted to go in this direction, but never really had the use case so development stopped. But, I am interested to see what is necessary and what can be done.

    I like the idea of making pytask projects more portable which means syncing state and outputs between multiple machines. I am unsure whether it means that the state should be persisted in git.

    Syncing outputs should work already using universal_pathlib. You can store files in S3 or other storages which would make them platform-independent.

    The default for the state is an SQLite database in .pytask/pytask.sqlite3, but the database location is configurable via database_url config/CLI option. Any SQLAlchemy-supported backend should work (PostgreSQL, MySQL, etc.) - untested but code is backend-agnostic. So, you could store state in a PostgreSQL hosted with AWS.

    Additionally, there are file hashes for detecting which tasks to run are stored separately in .pytask/file_hashes.json (JSON), not in database, but could be moved into the database for better portability. This is not strictly necessary for portability.

    One blocker for full portability are the task and node signatures/ids which contain absolute paths and are therefore inherently non-portable. It should be possible to express all files relative to the project's root.

    Another feature probably necessary for portability are locks to prevent that a task in the workflow is executed from two different machines.

    Maybe you can shed more light on your setup and we can flesh out what is needed.

  3. tobiasraabe commented on Dec 30, 2025

    @tobiasraabe
    tobiasraabe
    on Dec 30, 2025
    Member

    Let's tackle portability and synchronization separately and not mix them up like I did. Same about the file hash cache.

    We would still need to switch from absolute to relative paths for portability. (It might even help with pytask-dev/pytask-parallel#102)

    Here is a draft.

    The Lock File

    pytask.lock is the default state backend. It stores task state in a portable,
    git-friendly format so runs can be resumed or shared across machines.

    Example

    # This file is automatically @generated by pytask.
# It is not intended for manual editing.

lock-version = "1.0"
created-by = "pytask 0.6.0"

[[task]]
id = "src/tasks/data.py::task_clean_data"

[task.state]
value = "f9e8d7c6..."

[[task.depends_on]]
id = "data/raw/input.csv"

[task.depends_on.state]
value = "e5f6g7h8..."

[[task.produces]]
id = "data/processed/clean.parquet"

[task.produces.state]
value = "m3n4o5p6..."

    How It Works

    On each run, pytask:

    1. Reads pytask.lock (if present).
    2. Compares current dependency/product/task state() to the stored value.
    3. Skips tasks whose states match; runs the rest.
    4. Updates pytask.lock after each completed task (atomic write).

    pytask-parallel uses a single coordinator to write the lock file, so writes are
    serialized even when tasks execute in parallel.

    What Gets Stored

    • Task ID: portable identifier (module::function_name), relative to the project
      root.
    • Dependency/Product ID: typically a relative path or node name.
    • State: an opaque string from PNode.state() / PTask.state() stored as
      state.value.

    Signatures are internal identifiers and are not stored in the lock file.

    State Semantics

    The lock file does not interpret state.value. Portability depends on the node’s
    state() implementation (content hashes are portable; timestamps are not). For
    PythonNode, state.value is whatever state() returns; use a custom hash function
    for stable results on complex objects.

    Local Cache (Separate)

    The lock file is the source of truth. Local performance optimizations live in
    .pytask/ (e.g., file hash caching) and are safe to delete. See local-cache.md.

    File Format Reference

    Top-Level

    Field Required Description
    lock-version Yes Schema version (currently "1.0")
    created-by Yes pytask version that created the file

    Task Entry

    Field Required Description
    id Yes Portable task identifier
    state Yes State dictionary with a single value field

    Dependency/Product Entry

    Field Required Description
    id Yes Node identifier
    state Yes State dictionary with a single value field

    State Dictionary

    Field Required Description
    value Yes Opaque state string

    Version Compatibility

    • Upgrade: newer pytask upgrades old lock files in memory and writes the new format
      on the next update.
    • Downgrade: older pytask errors with a clear upgrade message.
    • Guarantee: breaking changes require a major lock-version bump.

    Best Practices

    1. Commit pytask.lock.
    2. Do not edit it manually.
    3. Add .pytask/ to .gitignore.

  4. DrorSh commented on Dec 30, 2025

    @DrorSh
    DrorSh
    on Dec 30, 2025
    ContributorAuthor

    Hi Tobias,

    Thank you for your reply!

    As requested, below are additional details about my setup.

    I adopted pytask because it solves a central issue in data analysis reproducibility. I develop code iteratively, with multiple steps that can be expensive or slow. I track the bld/ folder in git, and pytask verifies the DAG across these steps. This is critical because, in the end, the code (or at least large parts of it) needs to run end-to-end with a single command. This is increasingly a requirement from data editors and journals, and more generally a good practice for reducing errors along the way.

    I explored many alternatives, but they either imposed substantial overhead (e.g. Kedro, Nextflow) or were not flexible enough for my use case (e.g. R targets, Snakemake). Pytask strikes a very good balance: it is lightweight, yet flexible enough to accommodate a wide range of task types and execution environments (BigQuery, VMs with GPUs, Stata, R, etc.).

    In my view, simplicity is a key consideration. It is already a big ask to get coauthors to move from shared Dropbox folders to version-controlled GitHub repositories. I can usually explain the value pytask provides, but requiring a remote server or database solely to maintain state adds a layer of complexity that I would prefer to avoid.

    I think the lock file you outline above addresses the core challenge I am facing. What I want is for coauthors not to re-run steps that have already been completed, unless upstream data or code has changed, regardless of the machine they are using. This is why the current persist flag is problematic: it avoids recomputation, but at the cost of weakening the DAG that make pytask valuable in the first place.

  5. tobiasraabe commented on Feb 1, 2026

    @tobiasraabe
    tobiasraabe
    on Feb 1, 2026
    Member

    Hi @DrorSh,

    I am quite happy with the current state of the implementation. It would be great if you can help me testing it. The idea is

    • you take one of your current projects that is already built and copy it to another folder/another machine
    • install pytask from the branch (below are configs for uv and pixi) into this projet
    • run pytask and all of the tasks that were already built should be skipped during the execution
    • now, move this project to the machine of your coworkers and see whether everything is working there as well

    There is also some documentation in this how-to guide which might help.

    Let me know if you can test it and whether it works.

    Cheers!

    To install pytask from the branch, add the following lines to your pyproject.toml

    [project]
dependencies = [
  "pytask @ git+https://github.com/pytask-dev/pytask@lockfile",
]

    uv (same config in pyproject.toml, or add via command)

    uv add "pytask @ git+https://github.com/pytask-dev/pytask@lockfile"

    pixi (pixi.toml)

    [project]
name = "your-project"
version = "0.1.0"

[dependencies]
pytask = { git = "https://github.com/pytask-dev/pytask", branch = "lockfile" }

  6. DrorSh commented on Feb 2, 2026

    @DrorSh
    DrorSh
    on Feb 2, 2026
    ContributorAuthor

    Hi @tobiasraabe
    Thanks so much! I'm excited to test this out.
    I'll report back soon
    Best
    Dror

  7. DrorSh commented on Feb 12, 2026

    @DrorSh
    DrorSh
    on Feb 12, 2026
    ContributorAuthor

    @tobiasraabe

    This is exactly what I had in mind. works beautifully.
    I tested it across several scenarios and everything behaved as expected.

    Thanks a lot for taking the time to implement this!

    I have a couple additional ideas as follow-ups, I’ll open separate issues for those.

  8. raphaeljolivet commented on Mar 15, 2026

    @raphaeljolivet
    raphaeljolivet
    on Mar 15, 2026

    Hi

    I have a similar need :

    I have a workflow that I am currently migrating to PyTask (which is great BTW ! Thanks !).

    I would liek to be able to "bootstrap" from existing data and say to PyTask to initialize the hashed from existing files, as if it had produced it himself.

    It would take an additionnal comment.
    Something like "pytask init"

  9. tobiasraabe commented on Mar 15, 2026

    @tobiasraabe
    tobiasraabe
    on Mar 15, 2026
    Member

    Hi @raphaeljolivet, you can do this already with @pytask.mark.persist (doc). The API is not nice because you are required to keep the decorator indefinitely or recompute it at some point, but it should unblock you for now. I will think about whether it should be an init step or whether it could be tackled in #403.

  10. raphaeljolivet commented on Mar 15, 2026

    @raphaeljolivet
    raphaeljolivet
    on Mar 15, 2026

    Thanks, for your quick answer.
    I didn't notice this.

    I'll try it.

  11. tobiasraabe
    closed this as completedin #743on Mar 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions