With the new updates in v11.1, admins can import custom packages into the platform and use them within their Pipeline steps logic. In the following article we will outline all the steps required to import custom packages from Github or PyPI into the platform.

Custom packages

A Python package is an structured collection of Python modules grouped within a single directory for easy distribution and reuse. By importing created packages from PyPI and GitHub, users can extend the platform's functionality, reuse existing code, and improve the modularity of their solutions.

Conventionally, a custom package follows a structured format like this:

custom-package/
│── custom-package-v1/
│   ├── __init__.py
│   ├── samples.py
│── setup.py

• setup.py – Defines the package metadata, dependencies, and installation instructions, enabling its distribution via PyPI or direct installation.
• Modules (e.g., samples.py) – Contain the actual Python code, such as functions, classes, and utilities.
• __init__.py – Marks the directory as a Python package, allowing its modules to be imported as a regular package. It can also contain initialization code.

To get more information on how you can create and upload your own Python packages, you can refer to the PyPA's official documentation

Installing Custom Packages

Installing from PyPI

If the package is available on PyPI, simply add it to a requirements.txt file file with its version:

custom_package==0.0.1

Installing Private Packages from GitHub

For private packages hosted on GitHub, add the following structure to your requirements.txt file:

git+https://github.com/{YOUR_GITHUB_USER}/{YOUR_GITHUB_REPOSITORY}.git@v{VERSION}

This ensures the package is pulled from the specified GitHub repository and version.

Upload requirements.txt file to the platform

Once you have successfully added your packages to the requirements.txt file, you can import them into the platform by clicking on Preferences>Pipeline Settings>Upload file ...

If the upload is successful, a green pop-up notification will confirm that the custom settings have been applied.

Verifying the custom package import

To ensure that our custom package was imported correctly, we will create a simple pipeline Scriptthat returns its version upon execution.

We will use [wizata_dsapi.AbortedException](doc:pipeline-error) to manually interrupt the pipeline execution within the code without raising an actual error.

def check_version(context: wizata_dsapi.Context):
    import custom_package
    raise wizata_dsapi.AbortedException("package working - version: ", custom_package.__version__)


wizata_dsapi.api().upsert(check_version)

pipeline = wizata_dsapi.Pipeline(
    key="test_package_pipeline"
)

pipeline.add_transformation(
    script="check_version"
)

wizata_dsapi.api().upsert(pipeline)

Finally, by navigating to Experiments page, we can view the execution results of our experiment, confirming that our package has been successfully imported into the platform.

Verify without saving the script in the platform

If you prefer not to store your script inside the platform, you can reference it directly within the pipeline by adding a simple script step in the pipeline configuration, like this:

...
{
    "type": "script",
    "config": {
        "library": "custom_package",
        "function": "my_function"
    },
    "inputs": [],
    "outputs": []
}
...

Note: Ensure that you have previously uploaded your requirements.txt file to the platform, including the library and its version.