With the new updates in v11.1, admins can import custom packages into the platform and use them within their Pipeline steps logic. In the following article we will outline all the steps required to import custom packages from Github or PyPI into the platform.

Custom packages

A Python package is an structured collection of Python modules grouped within a single directory for easy distribution and reuse. By importing created packages from PyPI and GitHub, users can extend the platform's functionality, reuse existing code, and improve the modularity of their solutions.

Conventionally, a custom package follows a structured format like this:

custom-package/
│── custom-package-v1/
│   ├── __init__.py
│   ├── samples.py
│── setup.py

• setup.py – Defines the package metadata, dependencies, and installation instructions, enabling its distribution via PyPI or direct installation.
• Modules (e.g., samples.py) – Contain the actual Python code, such as functions, classes, and utilities.
• __init__.py – Marks the directory as a Python package, allowing its modules to be imported as a regular package. It can also contain initialization code.

To get more information on how you can create and upload your own Python packages, you can refer to the PyPA's official documentation

Installing Custom Packages

Overview

With release 11.1 we introduce full control for your pipeline installation directly from the AI Lab. Navigate to the Overview, and click on Installed Packages button to consult which python packages are currently installed on the runners.

Furthermore, in this page you can install additional Python librairies not included in Wizata runners by yourself, including the one you have developed.

Note:Custom requirements lists differ between the Engine (production) and the Experimentation Lab environments. Each runner maintains its own requirements.txt file.
We recommend installing packages first in Experimentation Lab runners to safely test their functionality before promoting them to production.

Installing from PyPI

If the package is available on PyPI, simply click on the Custom Requirements button and add it to the list with its version:

custom_package==0.0.1

Installing Private Packages from GitHub

For private packages hosted on GitHub, add the following structure to your Custom requirements:

git+https://github.com/{YOUR_GITHUB_USER}/{YOUR_GITHUB_REPOSITORY}.git@v{VERSION}

This ensures the package is pulled from the specified GitHub repository and version.

📘

Please remember to restart your runners after you have added and saved your package on the list.

Verifying the custom package import on the platform

To ensure that your custom package was imported correctly, restart your runners, wait a few seconds, and check the list of Installed packages. You should see your package listed along with its current installed version.

Verify without saving the script in the platform

If you prefer not to store your script inside the platform, you can reference it directly within the pipeline by adding a simple script step in the pipeline configuration, like this:

...
    {
      "type": "script",
      "config": {
        "library": "my_package",
        "function" : "my_function"
      },
      "inputs": [
        { "dataframe" : "input_df" }
      ],
      "outputs": [
        { "dataframe" : "output_df" }
      ]
    }
...

Note: Don't forget to reference your function in init file of your package.