Execution

Execution tracks a pipeline status while its being processed. It contains useful information regarding your pipeline processing:

• status - queued, started, abort requested, aborted, failed, completed
• pipeline - the pipeline that have been or that is going to be executed
• template - the template if referenced on your pipeline and/or trigger
• twin - the twin on which the pipeline is run - while a Trigger can specifies multiple twins, it creates one execution for each Twin
• queued, started, created, updated date and execution and waiting time - useful information on timing and duration.

To run a pipeline from User Interface you should create an Experiment. Pipelines are run on Engine

Run a Pipeline

When triggering an execution, the DSAPI will put your pipeline execution task on a queue. Depending on your choice your pipeline will be executed on experiment or production mode.

Experiment mode

Using Python or UI, you can test your pipeline and execute it as an Experiment. Check how you can Experiment within Wizata in the following article.

Production mode

To run a pipeline in production mode, we can use the run method:

execution = wizata_dsapi.api().run(
    pipeline='my_pipeline',
    twin='my_twin',
    properties={
        'start_q': 'now-7d',
        'end_q': 'now'
    }
)

By default, as opposite to experiment mode, no model are trained and no plots are produced. You can change this behavior using the same parameter as in experiment mode. For example, if you want to continue training model:

execution = wizata_dsapi.api().run(
    pipeline='my_pipeline',
    twin='my_twin',
    properties={
        'start_q': 'now-7d',
        'end_q': 'now'
    },
    train=True,
    plot=False,
    write=True
)

Additionally, if you would like to execute a production pipeline on multiple asset at once, you can do so efficiently by using the multi_run request. When using multi_run you need to use Technical UUID and put all execution_options directly inside properties. To modify execution_options, make the changes directly within the properties.

execution = wizata_dsapi.api().multi_run(
    pipeline='my_pipeline',
    twin_ids=[
        '770a81f8-c6d9-438c-a897-448b2ad33398',
        '09c440c1-9d75-4970-ac61-33bd3f442f24',
        'f10da4d1-7d05-4bb3-99d4-7184f089a2e6'
    ]
    properties={
        'start_q': 'now-1d',
        'end_q': 'now',
        'execution_options' : {
            "train" : false,
            "plot" : true,
            "write" : true
        }
    }
)

📘

The multi_run is intended to be replaced by the triggering system in future versions. It is intended to be more performant removing multi API calls.

Results

As a results, the platform produce an Execution object that is stored in the platform linked to your template and your experiment.

Using the user interface, you can have a look at the Execution Logs page, where you will find all the information regarding the results of your experiments, just like in the image below.

In a synchronous system, the execution will be returned directly when completed. If you have an asynchronous system using a queue, the execution will be retrieved directly but not completed. To await the result you can use the following logic:

while execution.status not in (wizata_dsapi.ExecutionStatus.COMPLETED, wizata_dsapi.ExecutionStatus.FAILED):  
    print(wizata_dsapi.api().get(execution).status)  
    time.sleep(10)

If you hover under the Warnings column response, you can find more information about the failed executions. Additionally, you can click on the eye icon to see a JSON formatted text with all the execution details, including the status and the warning variables.

If you are producing plots, you can retrieve them with the following method:

plots = wizata_dsapi.api().plots(execution)

Abort

In some case you might want to abort queued executions, especially if systems can’t process them due to performance or logical issues. To abort an execution you can use the abort endpoint in combination to the search_executions.

You can only abort an execution that is still queued, not if they have already started. If you would like to kill a stuck execution already started please contact your system administrator to kill current processes.

Here is an example on how to abort all queued executions of a specific pipeline:

pipeline_id = uuid.UUID("your_id")
total = None
page = 1
processed = 0
while total is None or total > processed:
    response_paged = wizata_dsapi.api().search_executions(page=page,
                                                          pipeline_id=pipeline_id,
                                                          status=wizata_dsapi.ExecutionStatus.QUEUED)
    total = response_paged.total
    print(wizata_dsapi.api().abort(response_paged.results))
    processed += len(response_paged.results)