Test Models Locally
Learn how to test your custom models locally
Before uploading a custom model to the Clarifai platform, always test and debug it locally. It ensures smooth performance, verifies dependency compatibility, and streamlines the deployment process.
This step helps you detect problems like setup file errors, typos, code misconfigurations, or incorrect model implementations. This saves you time and avoids upload failures by validating the model’s behavior on the target hardware you plan to deploy to.
You should ensure your local environment has sufficient memory and compute resources to handle model loading and execution during the testing process.
Prerequisites
Build a Model
You can either build a custom model from scratch or leverage pre-trained models from external repositories like Hugging Face.
If you're developing your own model, our step-by-step guide provides detailed instructions to get started. You can also explore this examples repository to learn how to build models compatible with the Clarifai platform.
Install Clarifai CLI
Install the latest version of the Clarifai CLI (Command Line Interface) tool. We'll use this tool to test models in the local development environment.
- Bash
pip install --upgrade clarifai
Develop and Test with Local Dev Runner
The local-dev
method allows allows you to develop, test, and run your model as a local development runner.
This approach lets you run your model using your own compute resources, be it a laptop or server you manage, while still allowing it to securely receive and respond to requests through the Clarifai API, just like it would in production.
Here is how to run a model as a local development runner:
- Bash
clarifai model local-dev [OPTIONS] [MODEL_PATH]
MODEL_PATH
is an optional path to the model directory. If omitted, the current directory is used by default.
We'll use this method to test the model we created here.
Use Cases for Local Dev Runners
Local Dev Runners are a powerful capability of the Clarifai platform, enabling you to run models anywhere with Python support and an internet connection — not just in the cloud.
This flexibility unlocks several key use cases:
-
Streamlined model development — Local Dev Runners make it easy to build and test new models directly within your local environment. You can spin up a runner on your machine, route API calls through our public cloud endpoint, and watch requests hit your model in real time. This allows you to debug, set breakpoints, return results, and validate outputs.
-
Leverage your own compute resources — If you have powerful hardware — whether a laptop, desktop, or on-prem server — you can take advantage of that local compute without relying on Clarifai's autoscaling infrastructure. Your model remains accessible through our API with full authentication, even though it runs locally.
-
Locally connect agents — Because Local Dev Runners execute on your chosen hardware, they can interact with local file systems, make OS-level calls, or access private data stores. With our MCP (Model Context Protocol) model type, you can give your cloud-hosted agents or any MCP-enabled clients authenticated access to your locally controlled information — regardless of their deployment location.
Required Environment Variables
Running the local development runner relies on certain environment variables defined in your current context. The context refers to the active environment settings that determine how your commands interact with the Clarifai platform.
You can create this context using the provided default values when you run the local-dev
command. Any configurations you create locally — such as the computer cluster and app — will also be created on the Clarifai platform, making them reusable whenever you test your model with the local development runner.
These are the environment variables required to create a runner:
Variable | Description |
---|---|
CLARIFAI_PAT | Personal Access Token for authentication |
CLARIFAI_USER_ID (user_id ) | User ID of the account owning the model |
CLARIFAI_APP_ID (app_id ) | App ID containing the model |
CLARIFAI_MODEL_ID (model_id ) | The model ID for the model to be run locally |
CLARIFAI_COMPUTE_CLUSTER_ID (compute_cluster_id ) | Compute cluster where the Local Dev Runner will reside. Note that the user_id of the compute cluster must match the user_id of the model. |
CLARIFAI_NODEPOOL_ID (nodepool_id ) | Nodepool within the compute cluster |
CLARIFAI_DEPLOYMENT (deployment_id ) | Deployment for a model into the cluster and nodepool |
CLARIFAI_RUNNER_ID (runner_id ) | Auto-generated runner ID, created by the API and stored in the context |
Click here to learn how to manage various aspects of your Clarifai context, including switching contexts and editing your configuration information.
Run the Test
To run your model with the local development runner, first navigate to the directory where your custom model is located.
Then, follow these steps.
Log In
Run the following command to log in to the Clarifai platform and establish a connection.
- Bash
clarifai login
After running the login
command, you'll be prompted to enter the following details to authenticate your connection:
- Bash
context name (default: "default"):
user id:
personal access token value (default: "ENVVAR" to get our env var rather than config):
Start the Runner
Next, start a local development runner.
- Bash
clarifai model local-dev
If the runner doesn't detect the necessary context configurations in your environment, it will prompt you to create them using default values. This ensures that all essential components required for Local Dev Runners are properly set up or included in your configuration context, including:
- A compute cluster and nodepool configured for Local Dev Runners.
Note: This cluster is created exclusively for Local Dev Runners. It is not designed to support other tasks and lacks features like autoscaling to handle variable traffic demands, among other cloud-specific capabilities. You also cannot use other types of clusters for Local Dev Runners — only the special cluster created for this purpose is supported.
-
An app with a model and model version representing the local runner.
-
A deployment that places the model version into the designated nodepool.
Example Output
C:\#############>clarifai login
context name (default: "default"):
user id: alfrick
personal access token value (default: "ENVVAR" to get our of env var rather than config): c02f72c***************
C:\#############>clarifai model local-dev
[INFO] 14:00:01.177806 Checking setup for local development runner... | thread=3524
[INFO] 14:00:01.179203 Current context: default | thread=3524
[INFO] 14:00:01.189128 Current user_id: alfrick | thread=3524
[INFO] 14:00:01.189128 Current compute_cluster_id: local-dev-compute-cluster | thread=3524
[INFO] 14:00:02.394761 Failed to get compute cluster with ID local-dev-compute-cluster: code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "ComputeCluster with ID \'local-dev-compute-cluster\' not found. Check your request fields."
req_id: "sdk-python-11.4.1-c8a4b5bab7a84ca7a91cd10fbc620c0f"
| thread=3524
Compute cluster not found. Do you want to create a new compute cluster alfrick/local-dev-compute-cluster? (y/n): y
[INFO] 14:00:20.755481
Compute Cluster created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-31b3b07f08f04cbb81a50edfba30454f"
| thread=3524
[INFO] 14:00:20.768179 Current nodepool_id: local-dev-nodepool | thread=3524
[INFO] 14:00:21.948659 Failed to get nodepool with ID local-dev-nodepool: code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Nodepool not found. Check your request fields."
req_id: "sdk-python-11.4.1-2e792b00b7d64521942e09f990db7e84"
| thread=3524
Nodepool not found. Do you want to create a new nodepool alfrick/local-dev-compute-cluster/local-dev-nodepool? (y/n): y
[INFO] 14:00:25.698797
Nodepool created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-65d15f07e7534d44aa79a0d1a8607350"
| thread=3524
[INFO] 14:00:25.720799 Current app_id: local-dev-runner-app | thread=3524
[INFO] 14:00:26.010459 Failed to get app with ID local-dev-runner-app: code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "app identified by path /users/alfrick/apps/local-dev-runner-app not found"
req_id: "sdk-python-11.4.1-b3c6fcd7b4ad4c2e91f742dc31705a5b"
| thread=3524
App not found. Do you want to create a new app alfrick/local-dev-runner-app? (y/n): y
[INFO] 14:00:31.354436
App created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-425accc65e1b451f9f6909d9dac25ecb"
| thread=3524
[INFO] 14:00:31.363481 Current model_id: local-dev-model | thread=3524
[INFO] 14:00:33.396312 Failed to get model with ID local-dev-model: code: MODEL_DOES_NOT_EXIST
description: "Model does not exist"
details: "Model \'local-dev-model\' does not exist."
req_id: "sdk-python-11.4.1-024501ac37044ed4a4ed5a397f860cf8"
| thread=3524
Model not found. Do you want to create a new model alfrick/local-dev-runner-app/models/local-dev-model? (y/n): y
[INFO] 14:00:43.972640
Model created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-052d3566144440bc83d3e5b66826a396"
| thread=3524
[INFO] 14:00:45.317992 No model versions found. Creating a new version for local dev runner. | thread=3524
[INFO] 14:00:45.695709
Model Version created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-cc9130981bea437895364a1a0debb079"
| thread=3524
[INFO] 14:00:45.698444 Created model version 5aaae29b6551422ea9f78c84e5f19205 | thread=3524
[INFO] 14:00:45.698444 Current model version 5aaae29b6551422ea9f78c84e5f19205 | thread=3524
[INFO] 14:00:45.698444 Create the local dev runner tying this
alfrick/local-dev-runner-app/models/local-dev-model model (version: 5aaae29b6551422ea9f78c84e5f19205) to the
alfrick/local-dev-compute-cluster/local-dev-nodepool nodepool. | thread=3524
[INFO] 14:00:46.922695
Runner created
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.4.1-c1176211bd5e4cceadb35040f8d8d2a7"
with id: 9dffc801b4904095a987b5c2b8508edf | thread=3524
[INFO] 14:00:46.936701 Current runner_id: 9dffc801b4904095a987b5c2b8508edf | thread=3524
[INFO] 14:00:47.239289 Current deployment_id: local-dev-deployment | thread=3524
[INFO] 14:00:47.241289 Current model section of config.yaml: {'id': 'my-uploaded-model', 'user_id': 'alfrick', 'app_id': 'docs-demos', 'model_type_id': 'text-to-text'} | thread=3524
Do you want to backup config.yaml to config.yaml.bk then update the config.yaml with the new model information? (y/n): y
[INFO] 14:00:51.122777
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# About to start up the local dev runner in this terminal...
# Here is a code snippet to call this model once it start from another terminal:
| thread=3524
[INFO] 14:00:51.123778
# Clarifai Model Client Script
# Example usage:
import os
from clarifai.client import Model
from clarifai.runners.utils import data_types
model = Model.from_current_context()
# Example model prediction from different model methods:
response = model.predict(text1='""')
print(response)
response = model.generate(text1='""')
for res in response:
print(res)
| thread=3524
[INFO] 14:00:51.123778 Now starting the local dev runner... | thread=3524
Note: If the
config.yaml
file does not contain model information that matches theuser_id
,app_id
, andmodel_id
defined in your current context, it will be automatically updated to include the new model details. This ensures that the model started by the local development runner is the same one you intend to call via the API. If needed, you can back up the existingconfig.yaml
file asconfig.yaml.bk
.
Test with Snippet
Once the local development runner starts in your terminal, an example client code snippet is automatically generated based on the model's signature to help you test it.
Example Code Snippet
# Clarifai Model Client Script
# Example usage:
import os
from clarifai.client import Model
from clarifai.runners.utils import data_types
model = Model.from_current_context()
# Example model prediction from different model methods:
response = model.predict(text1='""')
print(response)
response = model.generate(text1='""')
for res in response:
print(res)
If you run the generated snippet in a separate terminal, but within the same directory, you’ll receive the model’s response output.
After you're done testing, simply close the terminal running the local development runner to shut it down.
Test by Running Locally
Set up either a Docker container (recommended) or a Python virtual local development environment for testing the model locally. This ensures proper dependency management and prevents conflicts in your project.
CLI Flags
These are the key CLI flags available for local testing and running your models:
--mode
— Specify how to run the model:env
for virtual environment orcontainer
for Docker container. Defaults toenv
.-p
or--port
— The port to host the gRPC server for running the model locally. Defaults to8000
.--keep_env
— Retain the virtual environment after testing the model locally (applicable forenv
mode). Defaults toFalse
.--keep_image
— Retain the Docker image built after testing the model locally (applicable forcontainer
mode). Defaults toFalse
.--skip_dockerfile
— Flag to skip generating a dockerfile so that you can manually edit an already created dockerfile.
The test-locally
method allows you to test your model with a single CLI command. It runs the model locally and sends a sample request to verify that the model responds successfully.
The results of the request are displayed directly in the console.
Note that to test your model locally, you need to implement a test
method in the model.py
file. This method should internally call other model methods to perform validation.
When you run the test-locally
CLI command shown below, it will automatically invoke the test
method to carry out the testing process.
Below is a sample model.py
file with an example implementation of the test
method:
- Python
from clarifai.runners.models.model_class import ModelClass
from typing import Iterator
class MyModel(ModelClass):
"""A custom runner that adds "Hello World" to the end of the text."""
def load_model(self):
"""Load the model here."""
@ModelClass.method
def predict(self, text1: str = "") -> str:
output_text = text1 + "Hello World"
return output_text
@ModelClass.method
def generate(self, text1: str = "") -> Iterator[str]:
"""Example yielding a whole batch of streamed stuff back."""
for i in range(10): # fake something iterating generating 10 times.
output_text = text1 + f"Generate Hello World {i}"
yield output_text
def test(self):
res = self.predict("test")
assert res == "testHello World"
res = self.generate("test")
for i, r in enumerate(res):
assert r == f"testGenerate Hello World {i}"
You can specify the path to the directory containing the custom model you want to test. For example, if your model's files are stored in ./examples/models/clarifai_llama
, use the following command:
clarifai model test-locally ./examples/models/clarifai_llama --mode container
If you don’t specify a path, the current directory is used by default. In that case, simply navigate to the directory and run:
clarifai model test-locally --mode container
Here is how to test a model in a Docker Container:
- Bash
clarifai model test-locally --mode container
Here is how to test a model in a virtual environment:
- Bash
clarifai model test-locally --mode env
Test by Starting a gRPC Server
The run-locally
method starts a local gRPC server at https://localhost:{port}/
for running the model. Once the server is running, you can perform inference on the model via the Clarifai Python SDK.
Here is how to test a model in a Docker Container:
- Bash
clarifai model run-locally --mode container --port 8000
Here is how to test a model in a virtual environment:
- Bash
clarifai model run-locally --mode env --port 8000
Make Inference Requests
Once the model is running locally, you need to configure the CLARIFAI_API_BASE
environment variable to point to the localhost and port where the gRPC server is running.
- Unix-Like Systems
- Windows
export CLARIFAI_API_BASE="localhost:add-port-here"
set CLARIFAI_API_BASE="localhost:add-port-here"
You can then make inference requests using the model.