Test Models Locally

Learn how to test your custom models locally

Before uploading a custom model to the Clarifai platform, always test and debug it locally. It ensures smooth performance, verifies dependency compatibility, and streamlines the deployment process.

This step helps you detect problems like setup file errors, typos, code misconfigurations, or incorrect model implementations. This saves you time and avoids upload failures by validating the model’s behavior on the target hardware you plan to deploy to.

note

You should ensure your local environment has sufficient memory and compute resources to handle model loading and execution during the testing process.

Prerequisites

Build a Model

You can either build a custom model from scratch or leverage pre-trained models from external repositories like Hugging Face.

If you're developing your own model, our step-by-step guide provides detailed instructions to get started. You can also explore this examples repository to learn how to build models compatible with the Clarifai platform.

Install Clarifai CLI

Install the latest version of the Clarifai CLI (Command Line Interface) tool. We'll use this tool to test models in the local development environment.

Bash

 pip install --upgrade clarifai 

Develop and Test with Local Dev Runner

The local-dev method allows allows you to develop, test, and run your model as a local development runner.

This approach lets you run your model using your own compute resources, be it a laptop or server you manage, while still allowing it to securely receive and respond to requests through the Clarifai API, just like it would in production.

Here is how to run a model as a local development runner:

Bash

 clarifai model local-dev [OPTIONS] [MODEL_PATH] 

MODEL_PATH is an optional path to the model directory. If omitted, the current directory is used by default.

note

We'll use this method to test the model we created here.

Use Cases for Local Dev Runners

Local Dev Runners are a powerful capability of the Clarifai platform, enabling you to run models anywhere with Python support and an internet connection — not just in the cloud.

This flexibility unlocks several key use cases:

Streamlined model development — Local Dev Runners make it easy to build and test new models directly within your local environment. You can spin up a runner on your machine, route API calls through our public cloud endpoint, and watch requests hit your model in real time. This allows you to debug, set breakpoints, return results, and validate outputs.
Leverage your own compute resources — If you have powerful hardware — whether a laptop, desktop, or on-prem server — you can take advantage of that local compute without relying on Clarifai's autoscaling infrastructure. Your model remains accessible through our API with full authentication, even though it runs locally.
Locally connect agents — Because Local Dev Runners execute on your chosen hardware, they can interact with local file systems, make OS-level calls, or access private data stores. With our MCP (Model Context Protocol) model type, you can give your cloud-hosted agents or any MCP-enabled clients authenticated access to your locally controlled information — regardless of their deployment location.

Required Environment Variables

Running the local development runner relies on certain environment variables defined in your current context. The context refers to the active environment settings that determine how your commands interact with the Clarifai platform.

You can create this context using the provided default values when you run the local-dev command. Any configurations you create locally — such as the computer cluster and app — will also be created on the Clarifai platform, making them reusable whenever you test your model with the local development runner.

These are the environment variables required to create a runner:

Variable	Description
CLARIFAI_PAT	Personal Access Token for authentication
CLARIFAI_USER_ID (`user_id`)	User ID of the account owning the model
CLARIFAI_APP_ID (`app_id`)	App ID containing the model
CLARIFAI_MODEL_ID (`model_id`)	The model ID for the model to be run locally
CLARIFAI_COMPUTE_CLUSTER_ID (`compute_cluster_id`)	Compute cluster where the Local Dev Runner will reside. Note that the `user_id` of the compute cluster must match the `user_id` of the model.
CLARIFAI_NODEPOOL_ID (`nodepool_id`)	Nodepool within the compute cluster
CLARIFAI_DEPLOYMENT (`deployment_id`)	Deployment for a model into the cluster and nodepool
CLARIFAI_RUNNER_ID (`runner_id`)	Auto-generated runner ID, created by the API and stored in the context

tip

Click here to learn how to manage various aspects of your Clarifai context, including switching contexts and editing your configuration information.

Run the Test

To run your model with the local development runner, first navigate to the directory where your custom model is located.

Then, follow these steps.

Log In

Run the following command to log in to the Clarifai platform and establish a connection.

Bash

 clarifai login

After running the login command, you'll be prompted to enter the following details to authenticate your connection:

Bash

context name (default: "default"):
user id:
personal access token value (default: "ENVVAR" to get our env var rather than config):

Start the Runner

Next, start a local development runner.

Bash

 clarifai model local-dev

If the runner doesn't detect the necessary context configurations in your environment, it will prompt you to create them using default values. This ensures that all essential components required for Local Dev Runners are properly set up or included in your configuration context, including:

A compute cluster and nodepool configured for Local Dev Runners.

Note: This cluster is created exclusively for Local Dev Runners. It is not designed to support other tasks and lacks features like autoscaling to handle variable traffic demands, among other cloud-specific capabilities. You also cannot use other types of clusters for Local Dev Runners — only the special cluster created for this purpose is supported.

An app with a model and model version representing the local runner.
A deployment that places the model version into the designated nodepool.

Example Output

C:\#############>clarifai login
context name (default: "default"):
user id: alfrick
personal access token value (default: "ENVVAR" to get our of env var rather than config): c02f72c***************

C:\#############>clarifai model local-dev
[INFO] 14:00:01.177806 Checking setup for local development runner... |  thread=3524
[INFO] 14:00:01.179203 Current context: default |  thread=3524
[INFO] 14:00:01.189128 Current user_id: alfrick |  thread=3524
[INFO] 14:00:01.189128 Current compute_cluster_id: local-dev-compute-cluster |  thread=3524
[INFO] 14:00:02.394761 Failed to get compute cluster with ID local-dev-compute-cluster: code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "ComputeCluster with ID \'local-dev-compute-cluster\' not found. Check your request fields."
req_id: "sdk-python-11.4.1-c8a4b5bab7a84ca7a91cd10fbc620c0f"
 |  thread=3524
Compute cluster not found. Do you want to create a new compute cluster alfrick/local-dev-compute-cluster? (y/n): y
[INFO] 14:00:20.755481
Compute Cluster created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-31b3b07f08f04cbb81a50edfba30454f"
 |  thread=3524
[INFO] 14:00:20.768179 Current nodepool_id: local-dev-nodepool |  thread=3524
[INFO] 14:00:21.948659 Failed to get nodepool with ID local-dev-nodepool: code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Nodepool not found. Check your request fields."
req_id: "sdk-python-11.4.1-2e792b00b7d64521942e09f990db7e84"
 |  thread=3524
Nodepool not found. Do you want to create a new nodepool alfrick/local-dev-compute-cluster/local-dev-nodepool? (y/n): y
[INFO] 14:00:25.698797
Nodepool created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-65d15f07e7534d44aa79a0d1a8607350"
 |  thread=3524
[INFO] 14:00:25.720799 Current app_id: local-dev-runner-app |  thread=3524
[INFO] 14:00:26.010459 Failed to get app with ID local-dev-runner-app: code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "app identified by path /users/alfrick/apps/local-dev-runner-app not found"
req_id: "sdk-python-11.4.1-b3c6fcd7b4ad4c2e91f742dc31705a5b"
 |  thread=3524
App not found. Do you want to create a new app alfrick/local-dev-runner-app? (y/n): y
[INFO] 14:00:31.354436
App created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-425accc65e1b451f9f6909d9dac25ecb"
 |  thread=3524
[INFO] 14:00:31.363481 Current model_id: local-dev-model |  thread=3524
[INFO] 14:00:33.396312 Failed to get model with ID local-dev-model: code: MODEL_DOES_NOT_EXIST
description: "Model does not exist"
details: "Model \'local-dev-model\' does not exist."
req_id: "sdk-python-11.4.1-024501ac37044ed4a4ed5a397f860cf8"
 |  thread=3524
Model not found. Do you want to create a new model alfrick/local-dev-runner-app/models/local-dev-model? (y/n): y
[INFO] 14:00:43.972640
Model created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-052d3566144440bc83d3e5b66826a396"
 |  thread=3524
[INFO] 14:00:45.317992 No model versions found. Creating a new version for local dev runner. |  thread=3524
[INFO] 14:00:45.695709
Model Version created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-cc9130981bea437895364a1a0debb079"
 |  thread=3524
[INFO] 14:00:45.698444 Created model version 5aaae29b6551422ea9f78c84e5f19205 |  thread=3524
[INFO] 14:00:45.698444 Current model version 5aaae29b6551422ea9f78c84e5f19205 |  thread=3524
[INFO] 14:00:45.698444 Create the local dev runner tying this
  alfrick/local-dev-runner-app/models/local-dev-model model (version: 5aaae29b6551422ea9f78c84e5f19205) to the
  alfrick/local-dev-compute-cluster/local-dev-nodepool nodepool. |  thread=3524
[INFO] 14:00:46.922695
Runner created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-c1176211bd5e4cceadb35040f8d8d2a7"
 with id: 9dffc801b4904095a987b5c2b8508edf |  thread=3524
[INFO] 14:00:46.936701 Current runner_id: 9dffc801b4904095a987b5c2b8508edf |  thread=3524
[INFO] 14:00:47.239289 Current deployment_id: local-dev-deployment |  thread=3524
[INFO] 14:00:47.241289 Current model section of config.yaml: {'id': 'my-uploaded-model', 'user_id': 'alfrick', 'app_id': 'docs-demos', 'model_type_id': 'text-to-text'} |  thread=3524
Do you want to backup config.yaml to config.yaml.bk then update the config.yaml with the new model information? (y/n): y
[INFO] 14:00:51.122777

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# About to start up the local dev runner in this terminal...
# Here is a code snippet to call this model once it start from another terminal:
 |  thread=3524
[INFO] 14:00:51.123778
# Clarifai Model Client Script
# Example usage:
import os

from clarifai.client import Model
from clarifai.runners.utils import data_types

model = Model.from_current_context()

# Example model prediction from different model methods:

response = model.predict(text1='""')
print(response)

response = model.generate(text1='""')
for res in response:
    print(res)

 |  thread=3524
[INFO] 14:00:51.123778 Now starting the local dev runner... |  thread=3524

Note: If the config.yaml file does not contain model information that matches the user_id, app_id, and model_id defined in your current context, it will be automatically updated to include the new model details. This ensures that the model started by the local development runner is the same one you intend to call via the API. If needed, you can back up the existing config.yaml file as config.yaml.bk.

Test with Snippet

Once the local development runner starts in your terminal, an example client code snippet is automatically generated based on the model's signature to help you test it.

Example Code Snippet

# Clarifai Model Client Script
# Example usage:
import os

from clarifai.client import Model
from clarifai.runners.utils import data_types

model = Model.from_current_context()

# Example model prediction from different model methods:

response = model.predict(text1='""')
print(response)

response = model.generate(text1='""')
for res in response:
    print(res)

If you run the generated snippet in a separate terminal, but within the same directory, you’ll receive the model’s response output.

After you're done testing, simply close the terminal running the local development runner to shut it down.

Test by Running Locally

Set up Docker or a Virtual Environment

Set up either a Docker container (recommended) or a Python virtual local development environment for testing the model locally. This ensures proper dependency management and prevents conflicts in your project.

CLI Flags

These are the key CLI flags available for local testing and running your models:

--mode — Specify how to run the model: env for virtual environment or container for Docker container. Defaults to env.
-p or --port — The port to host the gRPC server for running the model locally. Defaults to 8000.
--keep_env — Retain the virtual environment after testing the model locally (applicable for env mode). Defaults to False.
--keep_image — Retain the Docker image built after testing the model locally (applicable for container mode). Defaults to False.
--skip_dockerfile — Flag to skip generating a dockerfile so that you can manually edit an already created dockerfile.

The test-locally method allows you to test your model with a single CLI command. It runs the model locally and sends a sample request to verify that the model responds successfully.

The results of the request are displayed directly in the console.

Note that to test your model locally, you need to implement a test method in the model.py file. This method should internally call other model methods to perform validation.

When you run the test-locally CLI command shown below, it will automatically invoke the test method to carry out the testing process.

Below is a sample model.py file with an example implementation of the test method:

Python

from clarifai.runners.models.model_class import ModelClass
from typing import Iterator


class MyModel(ModelClass):
    """A custom runner that adds "Hello World" to the end of the text."""

    def load_model(self):
        """Load the model here."""

    @ModelClass.method
    def predict(self, text1: str = "") -> str:
        output_text = text1 + "Hello World"
        return output_text

    @ModelClass.method
    def generate(self, text1: str = "") -> Iterator[str]:
        """Example yielding a whole batch of streamed stuff back."""

        for i in range(10):  # fake something iterating generating 10 times.
            output_text = text1 + f"Generate Hello World {i}"
            yield output_text

    def test(self):
        res = self.predict("test")
        assert res == "testHello World"

        res = self.generate("test")
        for i, r in enumerate(res):
            assert r == f"testGenerate Hello World {i}"

How to specify Local Model Path

You can specify the path to the directory containing the custom model you want to test. For example, if your model's files are stored in ./examples/models/clarifai_llama, use the following command:

clarifai model test-locally ./examples/models/clarifai_llama --mode container

If you don’t specify a path, the current directory is used by default. In that case, simply navigate to the directory and run:

clarifai model test-locally --mode container

Here is how to test a model in a Docker Container:

Bash

 clarifai model test-locally --mode container 

Here is how to test a model in a virtual environment:

Bash

 clarifai model test-locally --mode env 

Test by Starting a gRPC Server

The run-locally method starts a local gRPC server at https://localhost:{port}/ for running the model. Once the server is running, you can perform inference on the model via the Clarifai Python SDK.

Here is how to test a model in a Docker Container:

Bash

 clarifai model run-locally --mode container --port 8000 

Here is how to test a model in a virtual environment:

Bash

 clarifai model run-locally --mode env --port 8000  

Make Inference Requests

Once the model is running locally, you need to configure the CLARIFAI_API_BASE environment variable to point to the localhost and port where the gRPC server is running.

Unix-Like Systems
Windows

 export CLARIFAI_API_BASE="localhost:add-port-here" 

 set CLARIFAI_API_BASE="localhost:add-port-here" 

You can then make inference requests using the model.

Prerequisites​

Build a Model​

Install Clarifai CLI​

Develop and Test with Local Dev Runner​

Use Cases for Local Dev Runners​

Required Environment Variables​

Run the Test​

Log In​

Start the Runner​

Test with Snippet​

Test by Running Locally​

Test by Starting a gRPC Server​

Make Inference Requests​

What did you think of this doc?