Repository navigation
fix(optimizer): report Running from the Experiment Running condition - #893
Open
SahilKumar75 wants to merge 1 commit into
Open
SahilKumar75 wants to merge 1 commit into
SahilKumar75 wants to merge 1 commit into
Conversation
get_job() and list_jobs() only reported Running when a trial TrainJob was running at that moment, so between trial batches or while trial pods were pending the status fell back to Created even though Katib reported the Experiment as Running. Fixes kubeflow#892 Signed-off-by: Sahil Kumar Singh <60318530+SahilKumar75@users.noreply.github.com>
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Right now the OptimizationJob status only becomes
Runningif one of the trial TrainJobs is running at the momentget_job()is called. The Experiment's ownRunningcondition is never checked, so the status falls back toCreatedin two normal cases:CreatedOn a long experiment this makes the status go
Created -> Running -> Created -> Running, andwait_for_job_status(status={"Running"})can miss the short windows where a trial happens to be running.Changes:
backend.py: in__get_optimization_job_from_cr, aRunning=Truecondition on the Experiment now maps toOPTIMIZATION_JOB_RUNNING. The existing TrainJob check is still there as a fallback.constants.py: addedEXPERIMENT_RUNNING, next toEXPERIMENT_SUCCEEDED, and updated the comment onOPTIMIZATION_JOB_RUNNING.backend_test.py: added two cases totest_get_job_status_conditions, both withRunning=Trueon the Experiment: one where all trials areCompleteand one where trials are stillCreated.Succeeded and Failed handling is unchanged. Katib sets
Runningto False when the Experiment finishes, so a completed or failed job won't be reported as Running.Testing:
uv run pytest kubeflow: 699 passedmake verifypassesRepro script from the issue: https://gist.github.com/SahilKumar75/9e37fc12cf6cebd299203b35e9dbcdbf
Which issue(s) this PR fixes (optional, in
Fixes #<issue number>, #<issue number>, ...format, will close the issue(s) when PR gets merged):Fixes #892
Checklist: