This article is based on Argo v2.7.6, Kubernetes v1.18

Argo is a workflow engine based on Kubernetes CRD implementation, providing container-native workflows for Kubernetes, i.e. each workflow node is running a task as a container.

Installation

Since it is based on the Kubernetes CRD implementation, it is composed of CRD + Controller.

Installing Argo is as simple as creating an argo namespace and applying an official yaml.

1
2

kubectl create namespace argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml

Once installed, you can see several CRDs and a Controller and its RBAC resources installed in the K8s cluster.

In addition to using kubectl to use argo workflow, you can also use argo cli.

To install on a Mac environment.

1

brew install argoproj/tap/argo

Installation in a Linux environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Download the binary
curl -sLO https://github.com/argoproj/argo/releases/download/v2.7.6/argo-linux-amd64

# Make binary executable
chmod +x argo-linux-amd64

# Move binary to path
mv ./argo-linux-amd64 /usr/local/bin/argo

# Test installation
argo version

How it works

Then we’ll start by looking at how Argo works.

Architecture

The official Argo website provides a diagram of the Argo architecture after version v2.5, as follows.

Argo Architecture

The argo server provides api services to argo cli. When receiving requests from argo cli, argo server calls K8s Api to manipulate resources; and Controller to respond to resource changes from CRD.

argo server

There are two modes of argo server: Hosted and Local. The difference between the two is that the Hosted mode server runs inside the K8s cluster, while the Local mode server runs outside the K8s cluster (mostly locally).

To start the argo server locally.

1
2
3
4
5

$ argo server
INFO[0000]                                               authMode=server baseHRef=/ managedNamespace= namespace=default
INFO[0000] config map                                    name=workflow-controller-configmap
INFO[0000] Starting Argo Server                          version=v2.7.6+70facdb.dirty
INFO[0000] Argo Server started successfully on address :2746

Once started, access the local port 2746 and you will see a ui interface provided by the Argo server.

argo ui

Workflow

One of Argo’s most important CRD’s is Workflow, first look at its Spec definition.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

type WorkflowSpec struct {
	// Templates is a list of workflow templates used in a workflow
	// +patchStrategy=merge
	// +patchMergeKey=name
	Templates []Template `json:"templates" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,1,opt,name=templates"`

	// Entrypoint is a template reference to the starting point of the workflow.
	Entrypoint string `json:"entrypoint,omitempty" protobuf:"bytes,2,opt,name=entrypoint"`

	// Arguments contain the parameters and artifacts sent to the workflow entrypoint
	// Parameters are referencable globally using the 'workflow' variable prefix.
	// e.g. {{workflow.parameters.myparam}}
	Arguments Arguments `json:"arguments,omitempty" protobuf:"bytes,3,opt,name=arguments"`

	...
}

Three of the most important parameters.

  • Templates: all workflow template definitions.
  • Entrypoint: the entry point of the workflow template, i.e. specifying a template name from which the workflow starts.
  • Arguments: you can define the entry and product information of the workflow, if the arguments are prefixed with ‘workflow’, it means the whole workflow is available globally.

Template

Let’s look at the definition of Template.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Template is a reusable and composable unit of execution in a workflow
type Template struct {
	// Name is the name of the template
	Name string `json:"name" protobuf:"bytes,1,opt,name=name"`

	// Inputs describe what inputs parameters and artifacts are supplied to this template
	Inputs Inputs `json:"inputs,omitempty" protobuf:"bytes,5,opt,name=inputs"`

	// Outputs describe the parameters and artifacts that this template produces
	Outputs Outputs `json:"outputs,omitempty" protobuf:"bytes,6,opt,name=outputs"`

	// Metdata sets the pods's metadata, i.e. annotations and labels
	Metadata Metadata `json:"metadata,omitempty" protobuf:"bytes,9,opt,name=metadata"`

	// Steps define a series of sequential/parallel workflow steps
	Steps []ParallelSteps `json:"steps,omitempty" protobuf:"bytes,11,opt,name=steps"`

	// Container is the main container image to run in the pod
	Container *apiv1.Container `json:"container,omitempty" protobuf:"bytes,12,opt,name=container"`

	...
}

As you can see, Template is reusable throughout the workflow. Several important parameters are Inputs and Outputs; Metadata of the pod on which the template runs; definition of the Container on which the template runs; parallel Steps defined in the template, etc.

A workflow can define an entry template (specified by Entrypoint). The relationship between Template and Step is shown in the figure below, and multiple parallel Steps can be defined in a template to form a workflow; and Step can refer to other templates, each running a pod for work.

argo workflow tempalte

Step

Look again at the definition of Step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

type ParallelSteps struct {
	Steps []WorkflowStep `protobuf:"bytes,1,rep,name=steps"`
}

// WorkflowStep is a reference to a template to execute in a series of step
type WorkflowStep struct {
	// Name of the step
	Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`

	// Template is the name of the template to execute as the step
	Template string `json:"template,omitempty" protobuf:"bytes,2,opt,name=template"`

	// Arguments hold arguments to the template
	Arguments Arguments `json:"arguments,omitempty" protobuf:"bytes,3,opt,name=arguments"`

	// WithItems expands a step into multiple parallel steps from the items in the list
	WithItems []Item `json:"withItems,omitempty" protobuf:"bytes,5,rep,name=withItems"`

	...
}

ParallelSteps contains a set of WorkflowSteps, forming a parallel Step; WorkflowStep references a template, which can then specify ParallelSteps, with this string of parallel Steps, Argo can achieve any form of workflow customization.

The Arguments in the Step are also used to set the input and product information in the Step; WithItems extends a Step to multiple parallel Steps.

Getting Started

After understanding how Argo works, let’s put it into practice, using an example provided by the Argo website.

1

argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/loops-maps.yaml

First, let’s look at the yaml definition of Workflow.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-maps-
spec:
  entrypoint: loop-map-example
  templates:
  - name: loop-map-example
    steps:
    - - name: test-linux
        template: cat-os-release
        arguments:
          parameters:
          - name: image
            value: "{{item.image}}"
          - name: tag
            value: "{{item.tag}}"
        withItems:
        - { image: 'debian', tag: '9.1' }
        - { image: 'debian', tag: '8.9' }
        - { image: 'alpine', tag: '3.6' }
        - { image: 'ubuntu', tag: '17.10' }

  - name: cat-os-release
    inputs:
      parameters:
      - name: image
      - name: tag
    container:
      image: "{{inputs.parameters.image}}:{{inputs.parameters.tag}}"
      command: [cat]
      args: [/etc/os-release]

The entry template for Worflow is loop-map-example; this template defines four Step executions in parallel, whose execution parameters are entered separately.

Since the watch interface is open, we can see the workflow running in the terminal.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

Name:                loops-maps-qzv56
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Created:             Tue May 05 03:42:58 +0800 (4 seconds ago)
Started:             Tue May 05 03:42:58 +0800 (4 seconds ago)
Finished:            Tue May 05 03:43:02 +0800 (now)
Duration:            4 seconds

STEP                                         TEMPLATE          PODNAME                      DURATION  MESSAGE
 ✔ loops-maps-qzv56                          loop-map-example
 └-·-✔ test-linux(0:image:debian,tag:9.1)    cat-os-release    loops-maps-qzv56-2106556403  2s
   ├-✔ test-linux(1:image:debian,tag:8.9)    cat-os-release    loops-maps-qzv56-2252793177  3s
   ├-✔ test-linux(2:image:alpine,tag:3.6)    cat-os-release    loops-maps-qzv56-3723288758  3s
   └-✔ test-linux(3:image:ubuntu,tag:17.10)  cat-os-release    loops-maps-qzv56-3710784351  2s

You can visualize the workflow running in Argo UI.

workflow running in Argo UI