Skip to main content

Pipelines

Orchestrate asynchronous, long-running, multi-step AI processes

Clarifai Pipelines let you design and run asynchronous, long-running, multi-step processes for complex AI and MLOps tasks. They act as the orchestration backbone for building advanced AI agents and managing end-to-end machine learning operations.

Pipelines enable you to define, execute, and monitor container-based jobs directly on the Clarifai platform, giving you fine-grained control over each step.

With Clarifai Pipelines, you get a scalable, end-to-end automation engine that simplifies and unifies your entire MLOps lifecycle.

info

Explore a collection of ready-to-use ML pipeline templates for training, evaluating, and running models on the Clarifai platform in our Examples repository. See the Training Models documentation to learn how to use the pipeline templates to train different types of models.

Why Clarifai Pipelines​

  • Schedule training on your inference clusters. Training, evaluation, and inference workloads share the same Compute Orchestration clusters — no separate cluster to provision and pay for. Run a training job overnight on the same nodepool that serves your daytime inference traffic.
  • Outputs land in the model registry. Trained and quantized models from pipeline runs are registered directly into Clarifai's model registry, ready to serve or evaluate further without an export step.
  • Python or YAML — both first-class. Author pipelines as decorated Python functions with the DSL, or as YAML config with scaffold templates. Same engine, same templates, same upload path — pick whichever fits your team.

Use Cases for Pipelines​

Clarifai Pipelines support a wide range of AI and MLOps workflows. Here are some common use cases:

  • Model training and fine-tuning — Fine-tune LLMs and train custom image classifiers, object detectors, and other models on your own data, with the full training loop running on GPU compute and the resulting model registered in the Clarifai model registry.

  • Model evaluation and benchmarking — Measure model performance with standardized benchmarks like IFBench, SWE-bench, or MMLU, or with task-specific evaluations that score trained models against ground-truth labels, with per-task metrics stored alongside the model version under test.

  • Model quantization — Shrink a model's memory and compute footprint with post-training quantization, orchestrating calibration, conversion, and accuracy evaluation end-to-end and registering the quantized model alongside the original.

  • Draft model training for speculative decoding — Train smaller "draft" models that accelerate inference on larger target models via speculative decoding, including dataset preparation, draft training, and alignment evaluation against the target.

  • AI agent orchestration — Design, test, and deploy AI agents that execute multi-step logic autonomously — orchestrating tool usage, LLM calls, and decision logic across long-running tasks that can take hours or days.

  • Complex workflow automation — Orchestrate sophisticated, multi-stage AI workflows across components and services — large-scale batch processing, data transformations, and other distributed jobs — with built-in state management, fault tolerance, and scalable execution.

Quick Start​

The fastest way to get a working pipeline is to scaffold one from a template, upload it, and run it. The lora-pipeline-unsloth-quick-start template fine-tunes an LLM with LoRA — all parameters are pre-set to sensible defaults so the first run works without edits.

Install and Authenticate​

Install the latest clarifai SDK (which also installs the CLI) and set your Personal Access Token (PAT):

pip install --upgrade clarifai
export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE

Scaffold a Pipeline From a Template​

clarifai pipeline init --template lora-pipeline-unsloth-quick-start
cd lora-pipeline-unsloth-quick-start

This creates a working LoRA fine-tuning project that trains unsloth/Qwen3-0.6B on the mlabonne/FineTome-100k dataset using Unsloth — with num_epochs, learning_rate, LoRA rank, sequence length, and other hyperparameters all pre-configured. No edits required to run the first time.

Tip: Browse all available starter templates with clarifai pipelinetemplate list. See the Templates page for the full catalog.

Upload and Run​

clarifai pipeline upload
clarifai pipeline run --instance=g5.xlarge

The pipeline fine-tunes the model on a single-GPU instance (the template's compute step requests num_accelerators: 1 with a NVIDIA-* GPU). --instance=g5.xlarge auto-provisions a compute cluster and nodepool — no separate cluster setup required, and the same nodepool can later serve inference for the fine-tuned model.

That's it — you have a fine-tuned model registered in your Clarifai model registry, ready to serve, evaluate, or refine further.

Two upload paths

The LoRA template ships with both a YAML scaffold (config.yaml) and a Python DSL form (dsl.py). The default clarifai pipeline upload uses the YAML scaffold. To upload from the DSL Python file instead:

clarifai pipeline upload dsl.py

Both paths produce the same pipeline. The YAML config is convenient for explicit, version-controlled configuration. The Pipeline DSL is convenient for code-first authoring with type hints, @step decorators, and >> DAG composition. Use whichever fits your team.

Customize Before Running​

To override defaults at init time, pass --set key=value flags. For example, to fine-tune a different base model and change the number of epochs:

clarifai pipeline init --template lora-pipeline-unsloth-quick-start \
--set base_model_name="unsloth/Llama-3.2-1B-Instruct" \
--set num_epochs=3

Or cd into the scaffolded directory and edit config.yaml directly. See Templates for the full reference on --template, --set, and template discovery.

Author a Pipeline From Scratch in Python​

For custom workflows that don't fit a template, use the Pipeline DSL — write @step-decorated Python functions, compose them into a DAG, and upload directly from a .py file. See the DSL reference for the full API.

Where to Go Next​

  • Templates — full catalog of starter pipelines for training, fine-tuning, evaluation, and benchmarking.
  • Pipeline DSL reference — @step, step_ref, >> composition, ComputeInfo, secrets, codegen.
  • Manage Pipelines — list, validate, and inspect pipelines on the platform.
  • Manage Pipeline Runs — monitor, pause, resume, and cancel runs.
  • YAML / config-based pipelines — the YAML config authoring path, useful for explicit version-controlled configuration, YAML-first workflows, or as a reference for the scaffold-directory file structure.

Pipeline outputs → Artifacts​

Pipeline runs produce Artifacts — versioned, addressable binary outputs such as model checkpoints, training logs, embeddings, or datasets. Once a pipeline run completes, its outputs can be managed, downloaded, and reused via the artifacts CLI and SDK without re-running the pipeline. See the Artifacts documentation for managing pipeline-produced outputs.

© 2026 Clarifai, Inc. All rights reserved