Add Pipeline design by Yancey0623 · Pull Request #21 · sql-machine-learning/fluid

Yancey0623 · 2020-03-23T12:10:41Z

Add API design for Tekton Pipeline.

Yancey0623 · 2020-03-23T14:52:46Z

typhoonzero · 2020-03-24T04:59:26Z

doc/design.md


 ```python
-skaffold_git = fluid.Git(
+skaffold_git = fluid.git_resource(


Please also update below descriptions: Please be aware that the call to fluid.Git doesn't include the name

Why not use fluid.git for simple?

Why not use fluid.git for simple?

Just tracing the current implementation. I think we can update the API and design in another PR if need.

I remember that Tekton has a limited number of pre-defined resource types and git is one of them. I would suggest we keep it Git other than git_resource, because git_resouce is not a fullname; git_pipeline_resource is. But git_pipeline_resource is too long. It seems reasonable to use a short name Git for one of a few pre-defined types.

doc/design.md

typhoonzero · 2020-03-24T05:04:17Z

doc/design.md

+
+``` python
+@fluid.pipeline
+def tutorial(source_repo:"resource,git", web_image="resource,image"):


source_repo="resource,git"

I think it's okay, this is the parameter annotation instead of the default value, ref: https://www.python.org/dev/peps/pep-3107/#id29

typhoonzero · 2020-03-24T05:09:07Z

doc/design.md

+    build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image)
+
+    deploy_web = deploy_using_kubectl(source_repo, web_image)
+    deploy_web.web_image.from(build_skaffold_web)


How to define dependency by not using input/output?

Can use the runAfter keyword, and I added a section PiepeLine with DAG to introduce how to construct the DAG.

tonyyang-svail · 2020-03-25T00:10:57Z

doc/design.md

+
+``` yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline


What is the lifecycle of a pipeline object?

Pipeline object is like function definition in Python which includes Pipeline Tasks
A PipelineRun object would invoke the Pipeline Tasks as the dependency, can find some information from Task Status

doc/design.md

wangkuiyi · 2020-03-28T23:11:11Z

doc/design.md

+
+This will result in the following execution graph:
+
+``` text


No whitespace between ``` and text

wangkuiyi · 2020-03-29T04:29:01Z

doc/design.md


 ```python
-skaffold_git = fluid.Git(
+skaffold_git = fluid.git_resource(


I remember that Tekton has a limited number of pre-defined resource types and git is one of them. I would suggest we keep it Git other than git_resource, because git_resouce is not a fullname; git_pipeline_resource is. But git_pipeline_resource is too long. It seems reasonable to use a short name Git for one of a few pre-defined types.

wangkuiyi · 2020-03-29T04:29:38Z

doc/design.md


 ```python
-skaffold_image_leeroy_web = fluid.Image(
+skaffold_image_leeroy_web = fluid.image_resource(


Similarly, "image" is one of a few pre-defined resource types. How about we keep the name Image.

wangkuiyi · 2020-03-29T04:29:49Z

doc/design.md


 ```yaml
-goapiVersion: tekton.dev/v1alpha1
+apiVersion: tekton.dev/v1alpha1


Thanks for pointing this out!

wangkuiyi · 2020-03-29T04:35:15Z

doc/design.md

+
+### Pipeline
+
+A `Pipeline` object is like function declaration, according to the [definition](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md).


In the above text, we stated that a Task is like a function. Here we state the same with Pipeline. What is the difference between these two types of "functions"?

wangkuiyi · 2020-03-29T04:36:31Z

doc/design.md

+          value: "spec.template.spec.containers[0].image"
+```
+
+The above `Pipeline` is referencing a `Task` called `deploy-using-kubectl` defined as follows:


is referencing => refers to

wangkuiyi · 2020-03-29T04:38:38Z

doc/design.md

+    build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image)
+
+    deploy_web = deploy_using_kubectl(source_repo, web_image)
+    deploy_web.web_image.from(build_skaffold_web)


This deploy_web.web_image.from syntax looks confusing. Python programmer do not do this with function parameters.

I am afraid that this weird design might come from the fact that a Pipeline is NOT similar to a function definition.

After reading more about Pipeline, I see it describes a DAG of tasks, where edges are data dependencies between tasks.

Thinking about the following example from the Tekton tutorial https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#from:

- name: build-app taskRef: name: build-push resources: outputs: - name: image resource: my-image - name: deploy-app taskRef: name: deploy-kubectl resources: inputs: - name: image resource: my-image from: - build-app

Using programming language idiom, it is simply function calls

deploy_kubectl(image=build_push(my_image))

It seems that what we expect users to write is

@fluid.pipeline def build_and_deploy(image): deploy_kubectl(image=build_push(my_image))

where @fluid.pipeline should dry-run the function build_and_deploy to analysis the function dependencies, which is deploy_kubectl.image <- build_push, and generate the YAML definition of the Pipeline object.

I am not sure if the above suggestion is correct, or how reasonable it is. It has been a while I haven't use Tekton.

wangkuiyi · 2020-03-29T04:39:06Z

doc/design.md

+As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials:
+
+``` yaml
+- name: lint-repo


This doesn't look like a complete Kubernetes YAML. What is its kind?

wangkuiyi · 2020-03-29T04:39:53Z

doc/design.md

+- `from`:  clauses on the PipelineResources needed by a Task.
+- `runAfter`: clauses on the Pipeline Tasks.
+
+As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials:


Which tutorial? We need a URL here.

Yancey0623 added 2 commits March 23, 2020 20:09

add pipline design

4f55d49

add pipeline design

e52bd0a

Yancey0623 force-pushed the pipeline_design branch from b6daece to e52bd0a Compare March 23, 2020 14:50

Yancey0623 changed the title ~~[wip]Add Pipeline design~~ Add Pipeline design Mar 23, 2020

Yancey0623 requested review from tonyyang-svail and wangkuiyi March 23, 2020 14:51

typhoonzero reviewed Mar 24, 2020

View reviewed changes

tonyyang-svail reviewed Mar 25, 2020

View reviewed changes

update

bcd2f8e

wangkuiyi reviewed Mar 29, 2020

View reviewed changes


		### Pipeline

		A `Pipeline` object is like function declaration, according to the [definition](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md).

Conversation

Yancey0623 commented Mar 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yancey0623 commented Mar 23, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 Mar 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 Mar 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Yancey0623 commented Mar 23, 2020 •

edited

Loading

Yancey0623 Mar 27, 2020 •

edited

Loading

Yancey0623 Mar 27, 2020 •

edited

Loading