Skip to content

Allow job resubmission from within remotely run tasks - #216

Merged
riga merged 2 commits into
masterfrom
feat/request_job_resub
Oct 9, 2026
Merged

riga merged 2 commits into
masterfrom
feat/request_job_resub

Conversation

@riga

@riga riga commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Adds a way for a remote job to ask to be resubmitted without counting as failed. For example, a task can do this

  • when it finds it landed on an unsuitable node, or
  • to gracefully shutdown a training once the walltime is close being reaches.

Example:

def run(self):

    ...

    if some_walltime_condition:
        self.request_job_resubmission(
            reason="graceful termination",
            info={"host": socket.gethostname()},
            abort=True,
        )

    ...
  • Task.request_job_resubmission() (or law.job.base.request_job_resubmission()) writes the request to $LAW_JOB_RESUBMIT_FILE.
  • law_job.sh checks for this file after the setup and after each task branch. If it exists, the job stops and exits with code 100.
  • Remote workflows always resubmit these jobs without counting them towards --retries, and list them in their own resubmission summary.

@riga riga self-assigned this Oct 9, 2026
@riga riga added this to law v1.0.0 Oct 9, 2026
@riga
riga merged commit 7eaec17 into master Oct 9, 2026
28 checks passed
@riga
riga deleted the feat/request_job_resub branch October 9, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant