Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(backend): Add TTL Strategy in Workflow Spec for Post-Completion Cleanup #11352

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(backend): Add TTL Strategy in Workflow Spec for Post-Completion Cleanup #11352

wants to merge 1 commit into from

Conversation

VaniHaripriya
Copy link
Contributor

@VaniHaripriya VaniHaripriya commented Nov 4, 2024 •
edited by HumairAK
Loading

Description of your changes:

  • Added the TTLStrategy field in Workflow Spec, configured to use SecondsAfterCompletion.
  • Uses the pipeline_ttlseconds variable to define the TTL duration.
  • This field automates cleanup of completed workflows.

This PR should be merged only after #11269 gets merged.

Testing Instruxtions:

  • Build the API Server image and push to an image registry
  • Upload main.yaml file from here
  • Check in KFP UI Pipeline Spec tab if the following snippet is present:
platform_spec:
  platforms:
    kubernetes:
      pipelineConfig:
        pipelineTtl: 60
  • Create a run and check if TTL configuration is present in Workflow CR:
$ oc get workflow -o yaml $(oc get workflow --no-headers | awk '{print $1}') | yq .spec.ttlStrategy
secondsAfterCompletion: 60

Checklist:

@google-oss-prow Google OSS Prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
@@ -77,9 +77,19 @@ func (t *V2Spec) ScheduledWorkflow(modelJob *model.Job) (*scheduledworkflow.Sche
}
}

var pipeline_options argocompiler.Options
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Camel casing required to stay consistent with golang variable definition standards - PipelineOptions

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
var obj interface{}
if util.CurrentExecutionType() == util.ArgoWorkflow {
obj, err = argocompiler.Compile(job, kubernetesSpec, nil)
obj, err = argocompiler.Compile(job, kubernetesSpec, &pipeline_options)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as https://github.com/kubeflow/pipelines/pull/11352/files#r1845247170

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
var obj interface{}
if util.CurrentExecutionType() == util.ArgoWorkflow {
obj, err = argocompiler.Compile(job, kubernetesSpec, nil)
obj, err = argocompiler.Compile(job, kubernetesSpec, pipeline_options)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as https://github.com/kubeflow/pipelines/pull/11352/files#r1845247170

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/v2/compiler/argocompiler/argo.go Outdated
}

const (
pipeline_default_ttlSeconds = int32(30)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Camel casing :)

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/v2/compiler/argocompiler/argo.go Outdated
@@ -86,6 +91,11 @@ func Compile(jobArg *pipelinespec.PipelineJob, kubernetesSpecArg *pipelinespec.S
}
}

pipeline_ttlseconds := pipeline_default_ttlSeconds
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Camel casing here too

HumairAK
HumairAK reviewed Nov 19, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
@@ -300,9 +310,19 @@ func (t *V2Spec) RunWorkflow(modelRun *model.Run, options RunWorkflowOptions) (u
}
}

var pipeline_options *argocompiler.Options
for _, platform := range t.platformSpec.Platforms {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be doing a nil check on Platforms cc @rimolive

@VaniHaripriya
Add TTL Strategy in Workflow Spec
f0befb9 
Signed-off-by: VaniHaripriya <[email protected]>

Please enter the commit message for your changes. Lines starting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DharmitD DharmitD DharmitD left review comments

@HumairAK HumairAK HumairAK left review comments

@hbelmiro hbelmiro Awaiting requested review from hbelmiro

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@VaniHaripriya @HumairAK @DharmitD