Eventline is now open source and available on GitHub !

If you are still using the old Eventline platform, contact us to migrate to the new service, and head to the new documentation website for more information.

Pipelines

Introduction

Pipelines are instantiated either when triggers are activated or when commands are executed. In this section, we will examine how it happens and how we can interact with the pipelines.

Instantiation

Triggers

Triggers are activated on new events. For each event, each pipeline defined in the trigger resource is instantiated.

Instantiating a pipeline is simply instantiating every one of its task. Once created, the Eventline scheduler will use the list of dependencies in the pipeline resource to determine which tasks can be executed.

When all tasks have been either terminated (successfully or not) or aborted, the pipeline will also be considered terminated.

Commands

Commands are an alternative way to execute commands. Users can execute them at any moment, using the web interface or the evcli command line tool.

Pipelines associated with a command are immediately instantiated when the command is executed, and they are then handled the exact same way as they are for triggers.

Pipelines instantiated from triggers are associated with the event which activated the trigger. For commands, we generate an event of type command.execution containing the set of parameters used among other things. Pipelines instantiated from the command will have access to this event during execution.

Lifecycle

A pipeline can have one of the following statuses:

created
The pipeline and its tasks have just been created and are waiting for scheduling.
started
At least one task of the pipeline was started and is currently being executed.
aborted
The pipeline was stopped, usually by a user, before its tasks have been completed.
successful
All tasks in the pipeline completed successfully.
failed
At least one task in the pipeline failed.

Abortion

Created or started pipelines can be manually aborted, using either the web interface of the evcli command line tool. When that happens, any running task is interrupted, and remaining task are immediately marked as aborted.

Note that due to the concurrent nature of task execution, it is possible for a task to complete (successfully or not) after you act to abort a pipeline, but before the pipeline is aborted. In that case, the abortion could fail.

Restart

Any successful, aborted or failed pipeline can be manually restarted, using the web interface or the evcli command line tool. On restart, the state of the pipeline and of all of its tasks are reset, and they are all marked as created. Once scheduled, each task will use the exact same resource data as the previous execution(s), even if the associated resources have been modified in the mean time.

Note that identities are not resources: each identity used by a task is read at the moment the task is executed. Therefore a pipeline running a second time after a restart could still act differently if identities have been modified.

Warning

Eventline is a scheduling platform: tasks execute code which usually affect external systems. Restarting a pipeline could have unexpected consequences regarding these systems.

In general, writing tasks in an idempotent way will help a lot in keeping your technical processes robust.

Restart from failure

In addition to the normal restart procedure, aborted and failed pipelines can restart from failure. Tasks which where not successful and tasks which depend on them will be reset, and will therefore be marked as created. Successful tasks will not be affected.

Scratchpad

The scratchpad is a data storage system made available to each instantiated pipeline. Its main purpose is to allow data sharing between tasks.

A scratchpad contains multiple entries, each one identified by a key. The key itself is an arbitrary character string (with a limit of 100 bytes). Each entry contains a value, which is also a character string of up to 4000 bytes.

Info
Since a scratchpad entry is a simple character string, it is possible to store structured data by encoding them to text using the data format of your choice, for example JSON.

Lifecycle

A scratchpad is created when a pipeline is instantiated, and initially does not contain any entry.

Users can interact with the scratchpad either through the API or using the evcli client. It is possible to create new entries, update existing ones, and delete them at any moment.

When a pipeline is restarted from the start, all entries in the scratchpad are deleted.

When a pipeline is retarted from failure, the scratchpad is kept as it was when the pipeline failed.