2.13.2

ClusterFailurePolicy

Package: flyte.clustered

Failure and restart policy for the JobSet as a whole.

Parameters

class ClusterFailurePolicy(
    max_restarts: int = 0,
    restart_on_host_maintenance: bool = False,
    resume_after_preemption: bool = False,
)
Parameter Type Description
max_restarts int Number of times the entire JobSet may be restarted in place before the attempt fails. These restarts do not use the task’s retries; the failed attempt does.
restart_on_host_maintenance bool When True, node evictions (DisruptionTarget condition) trigger a free restart that does not consume the max_restarts budget. Free restarts still increment flyte.ctx().restart_attempt.
resume_after_preemption bool When True, a gang that Kueue preempts after it started keeps its attempt and JobSet and resumes when Kueue re-admits it, so preemptions do not use the task’s retries. Other evictions still end the attempt. max_runtime keeps counting while the gang waits. Save checkpoints with flyte.ctx().checkpoint so the work resumes where it stopped. Needs a Flyte backend v2.0.54 or later; older ones ignore it.