Skip to main content

Pipelines

Orchestrate asynchronous, long-running, multi-step AI processes

Clarifai Pipelines let you design and run asynchronous, long-running, multi-step processes for complex AI and MLOps tasks. They act as the orchestration backbone for building advanced AI agents and managing end-to-end machine learning operations.

Pipelines enable you to define, execute, and monitor container-based jobs directly on the Clarifai platform, giving you fine-grained control over each step.

With Clarifai Pipelines, you get a scalable, end-to-end automation engine that simplifies and unifies your entire MLOps lifecycle.

Use Cases for Pipelines​

Clarifai Pipelines support a wide range of AI and MLOps workflows. Here are some common use cases:

  • Automated MLOps workflows — Build reliable, repeatable pipelines for data preparation, model training, evaluation, deployment, and monitoring. Pipelines integrate seamlessly with version control and CI/CD systems to support continuous experimentation and delivery.

  • AI agent orchestration — Design, test, and deploy advanced AI agents that execute multi-step logic autonomously. Pipelines orchestrate tool usage, LLM calls, and decision logic while managing state, external APIs, and long-running tasks that can execute for hours or even days.

  • Asynchronous, long-running application tasks — Add powerful backend capabilities to your applications without operational complexity. Pipelines make it easy to trigger and monitor tasks like large-scale batch processing, model fine-tuning, or complex data transformations.

  • Complex workflow automation — Orchestrate sophisticated, multi-stage AI workflows across multiple components and services. Pipelines provide built-in support for state management, fault tolerance, and scalable execution across distributed systems.

Quick Start​

You can follow these steps to quickly get started with pipelines via the API.

Get Credentials​

1. Go to the Clarifai platform and get your user ID, app ID, and Personal Access Token (PAT).

Then, set your PAT as an environment variable:

export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE

2. You’ll also need to create a compute cluster and nodepool and get their IDs.

Initialize a Pipeline Project​

Run the following command to create a new pipeline project in your current directory. Follow the prompts to provide the required details.

clarifai pipeline init

Upload the Pipeline​

Run the following command to upload the pipeline with associated pipeline steps to Clarifai.

clarifai pipeline upload

Run the Pipeline​

Run the following command to start the pipeline and monitor its progress until completion or timeout.

clarifai pipeline run --compute_cluster_id cluster_id_here --nodepool_id nodepool_id_here
© 2026 Clarifai, Inc. All rights reserved