# Clarifai Docs > Clarifai Docs This file contains all documentation content in a single document following the llmstxt.org standard. ## Agents # Agents **Build powerful AI agents with Clarifai** Artificial intelligence (AI) agents refer to autonomous systems that often leverage large language models (LLMs) as their core. Given high-level instructions, these agents possess the remarkable ability to understand, plan, reason, and execute complex tasks to achieve specific goals. They go beyond simple automation by demonstrating: - **Intelligent planning** — Breaking down ambiguous instructions into actionable steps. - **Tools utilization** — A tool equips an agent with specific capabilities, enabling it to perform actions and interact with the external world beyond its inherent text generation and reasoning. _What sets effective agents apart from basic language models is their ability to properly use tools._ - **Adaptability & learning** — Learning from interactions, maintaining context, and improving performance over time. - **Collaboration** — Working with other agents to achieve complex, multi-faceted outcomes. ## Build Agents With Our API Clarifai empowers developers to build powerful AI agents that can tackle real-world problems by interacting intelligently with diverse data and tools, all within a cohesive and manageable environment. We provide a comprehensive suite of features, including: - **Versatile AI models** — Access to a wide range of Clarifai models, including both text and vision capabilities, allowing agents to process and understand diverse forms of information. - **Flexible interaction modes** — Seamlessly engage with base models, a single AI agent, or orchestrate complex interactions between multiple agents for intricate workflows. - **Out-of-the-box tool connectors** — Instantly equip agents with essential functionalities, such as code execution, web search, image generation, and document library access. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Agent Skills # Agent Skills **Supercharge AI coding assistants with deep Clarifai knowledge** Clarifai Skills are specialized prompt templates that transform AI coding assistants — Claude Code, Cursor, Codex, and more — into Clarifai platform experts. You don't need to explain APIs from scratch. Just describe what you want in plain language — such as _"list my apps"_ — and your assistant finds the right skill and gets to work. Built on the open [Agent Skills](https://agentskills.io) standard, Clarifai Skills work out of the box across 30+ agent platforms. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## Available Skills | Skill | What It Does | |-------|-------------| | [`clarifai-cli`](https://github.com/Clarifai/skills/tree/main/.github/skills/clarifai-cli) | Full CLI reference: `init`, `serve`, `deploy`, `status`, `logs`, `undeploy`, `predict`, `list-instances` | | [`clarifai-model-upload`](https://github.com/Clarifai/skills/tree/main/.github/skills/clarifai-model-upload) | Deploy models with vLLM, SGLang, HuggingFace, Ollama — `ModelClass`, `config.yaml`, GPU config | | [`clarifai-inference`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-inference) | Model discovery, method signatures, code generation, OpenAI-compatible API, agentic models| | [`clarifai-mcp`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-mcp) | Build MCP servers with `MCPModelClass`, `StdioMCPModelClass`, FastMCP tools| | [`clarifai-deployment-lifecycle`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-deployment-lifecycle) | Deploy/status/logs/undeploy lifecycle, version patching, state monitoring | | [`clarifai-observability`](https://github.com/Clarifai/skills/tree/main/.github/skills/clarifai-observability) | Debug stuck or crashing deployments: CLI logs, K8s events, common resolutions | | [`clarifai-agentic-flows`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-agentic-flows) | Multi-step orchestration: "train THEN deploy", client scripts, server orchestrators | | [`clarifai-datasets`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-datasets) | Upload datasets with annotations (classification, detection, segmentation), format conversion | | [`clarifai-pipelines`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-pipelines) | Batch processing pipelines with containerized steps and Artifacts API | | [`clarifai-training-pipelines`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-training-pipelines) | Train classifiers (ResNet-50) and detectors (YOLOF) using built-in templates | | [`clarifai-grpc`](https://github.com/Clarifai/skills/blob/main/.github/skills/clarifai-grpc) | Low-level gRPC API with protobufs for advanced platform operations | ## Prerequisites ### Set Up an AI Coding Assistant You need an AI coding assistant — such as [Claude Code](https://claude.ai/code), [Cursor](https://www.cursor.com/), [GitHub Copilot](https://github.com/features/copilot), [OpenAI Codex](https://openai.com/codex), or [Gemini](https://gemini.google.com/) — installed and configured on your machine. ### Install the Clarifai Python SDK ```bash pip install --upgrade clarifai ``` #### Log in via the CLI Log in to Clarifai and authenticate your connection with your [PAT](https://docs.clarifai.com/control/authentication/pat) (Personal Access Token). ```bash clarifai login ``` ### Install Skills #### Option A: Clarifai CLI (Recommended) Run the skills installer. If no agent flag is specified, it auto-detects installed agents. If no skill name is given, it installs all available skills. ```bash clarifai skills install # all skills, auto-detect agents clarifai skills install --claude # all skills for Claude Code clarifai skills install clarifai-cli --claude # one skill for Claude Code clarifai skills install --claude --cursor # all skills, multi-agent clarifai skills install --local # project-level install ``` :::note Scope flags - `--claude` — Target Claude Code - `--codex` — Target OpenAI Codex - `--cursor` — Target Cursor - `--copilot` — Target GitHub Copilot - `--gemini` — Target Gemini - `--all-agents` — Target all supported agents - `--global` — Install globally (`~/`) (default) - `--local` — Install at project level (`./`) - `--force` — Overwrite existing skills - `--source PATH` — Install or update from a local skills repo clone instead of GitHub ::: #### Option B: Claude Code Plugin If you use Claude Code, you can install Clarifai skills directly via the plugin marketplace: ```bash claude plugin marketplace add Clarifai/skills # Register the Clarifai skills repo claude plugin install clarifai-skills # Install all skills as a single plugin ``` ## Managing Skills ```bash clarifai skills list --remote # browse available skills (shorthand: ls) clarifai skills list --installed # see what's installed clarifai skills update # update all (auto-detect agents) clarifai skills update --claude # update for Claude only clarifai skills remove clarifai-cli --claude # remove one skill from Claude Code clarifai skills remove --all # remove all Clarifai skills ``` ## How It Works Each skill is a folder containing a `SKILL.md` file with YAML frontmatter and markdown documentation, plus optional `references/` and `examples/` subdirectories. ```bash clarifai-cli/ SKILL.md ← instructions loaded by the agent references/ ← detailed reference docs examples/ ← working code examples ``` When you install skills, they are placed in a central location and symlinked per agent: ```bash ~/.agents/skills/ # one copy of each skill clarifai-cli/ clarifai-model-upload/ ... ~/.claude/skills/ # symlinks for Claude Code clarifai-cli -> ~/.agents/skills/clarifai-cli ... ``` Skills activate when your request matches their description. Depending on your AI assistant, you can trigger them: - **By asking naturally** — describe what you want and the assistant picks the right skill: > "How do I run inference on a Clarifai model?" > "Deploy a vLLM model to Clarifai" > "Train an image classifier on my dataset" - **Via slash commands** (supported in assistants like Claude Code): ``` /clarifai-inference /clarifai-cli /clarifai-model-upload ``` - **By pasting skill context** — in assistants like Cursor or Codex, you can paste a skill's prompt directly into your system prompt or context window to activate its guidance. ## Example Usage ### Run Inference Tell your assistant: _"Run inference on Claude Sonnet 4 with the prompt: Hello world"_. The `clarifai-inference` skill guides the assistant to generate code like: ```python import os from openai import OpenAI client = OpenAI( base_url="https://api.clarifai.com/v2/ext/openai/v1", api_key=os.environ['CLARIFAI_PAT'], ) response = client.chat.completions.create( model="https://clarifai.com/anthropic/completion/models/claude-sonnet-4", messages=[{"role": "user", "content": "Hello world"}], ) print(response.choices[0].message.content) ``` Or, run it directly from the terminal: ```bash clarifai model predict anthropic/completion/models/claude-sonnet-4 "Hello world" ``` ### Deploy a Model Tell your assistant: _"Deploy Qwen3-0.6B with vLLM"_. The `clarifai-cli` and `clarifai-model-upload` skills guide the assistant to run: ```bash clarifai model init ./qwen --toolkit vllm --model-name "Qwen/Qwen3-0.6B" clarifai model serve ./qwen # test locally clarifai model deploy ./qwen # deploy to cloud ``` ### More Examples Here are more prompts you can use with your assistant: - _"Create a FastMCP server with a web search tool and deploy it to Clarifai"_ - _"Upload my COCO-format dataset to Clarifai with detection annotations"_ - _"My deployment is stuck — help me debug it"_ - _"Train an image classifier on my dataset"_ - _"Show me how to run inference on a Clarifai-hosted LLM using the OpenAI-compatible API"_ --- ## Build Agents # Build Agents **Learn how to build agents with Clarifai** Clarifai provides a streamlined developer experience that allows you to quickly prototype, build, and deploy agentic AI applications. We provide an [OpenAI-compatible API endpoint](https://docs.clarifai.com/compute/inference/#predict-with-openai-compatible-format), which allows you to seamlessly integrate with popular agent development toolkits that support the OpenAI's API standard. This empowers you to create powerful, flexible, and tool-using AI agents with minimal configuration. Let’s illustrate how you can build agents with the various toolkits we support: - [OpenAI](#openai) - [Google ADK](#google-adk) - [CrewAI](#crewai) - [Vercel](#vercel) - [Agno](#agno) :::warning LiteLLM [LiteLLM](https://docs.litellm.ai/docs/providers/clarifai) is a library that offers a unified API for working with various LLM providers. When using an agent toolkit that supports LiteLLM to access Clarifai's models, specify the model using the `openai/` prefix followed by the Clarifai model URL — for example: `openai/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-7B`. ::: :::tip Install Clarifai Package Install the latest version of the Clarifai Python SDK package by running `pip install --upgrade clarifai`. Also, go to the **[Secrets](https://docs.clarifai.com/control/authentication/pat)** section in your personal settings page and generate a Personal Access Token (PAT) to authenticate with the Clarifai platform. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import OpenAI from "!!raw-loader!../../../code_snippets/python-sdk/agents/openai.py"; import OpenAIOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/openai-output.txt"; import GoogleADK from "!!raw-loader!../../../code_snippets/python-sdk/agents/googleadk.py"; import GoogleADKOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/googleadk-output.txt"; import CrewAI from "!!raw-loader!../../../code_snippets/python-sdk/agents/crewai.py"; import CrewAIOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/crewai-output.txt"; import Vercel from "!!raw-loader!../../../code_snippets/python-sdk/agents/vercel.js"; import VercelOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/vercel-output.txt"; import Agno from "!!raw-loader!../../../code_snippets/python-sdk/agents/agno.py"; import AgnoOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/agno-output.txt"; import AgnoTools from "!!raw-loader!../../../code_snippets/python-sdk/agents/agno-tools.py"; import AgnoToolsOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/agno-tools-output.txt"; import AgnoTeam from "!!raw-loader!../../../code_snippets/python-sdk/agents/agno-team.py"; import AgnoTeamOutput from "!!raw-loader!../../../code_snippets/python-sdk/agents/agno-team-output.txt"; ## OpenAI You can use the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) to build agentic AI apps that make use of Clarifai models. ### Installation The following command will install the `openai-agents` package and the optional `litellm` dependency group. pip install "openai-agents[litellm]" After the installation, you can use the [LitellmModel](https://openai.github.io/openai-agents-python/models/litellm/) class in the OpenAI Agents SDK to access Clarifai models via LiteLLM. ### Build an Agent When building an agent with the OpenAI Agents SDK, the most common properties you'll configure are: - `instructions` — Also known as the system prompt or developer message, this defines the agent’s behavior and tone. - `model` — Specifies which LLM to use. In this case, it points to a Clarifai-hosted model. - `tools` — Define the actions your agent can take, such as calling APIs or fetching data. These can be implemented as regular Python functions. You'll also need the `Runner` class to run your agent. ### Example Here is an example of an AI agent powered by a Clarifai-hosted model and equipped with a simple weather lookup tool. When asked about the weather, the agent will recognize the need to use its tool, get the information, and then summarize it in the form of a haiku. {OpenAI} Example Output {OpenAIOutput} ## Google ADK You can use the [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs/) to build agentic AI apps that make use of Clarifai models. ### Installation You can install the following necessary packages: - `google-adk` — This is the Google ADK itself, which provides a wide range of components, such as `Agent`, `Runner`, `LiteLlm`, and `InMemorySessionService`. - `litellm` — The ADK's `LiteLlm` model class relies on the `litellm` library under the hood to handle multi-model support. After the installation, you can use the `LiteLlm` class to access Clarifai models via LiteLLM. - `google-generativeai` — The ADK uses it for handling `types`; that is, structuring message inputs and outputs. This is the combined commands for installing them: pip install google-adk litellm google-generativeai ### Build an Agent When building an agent with the Google ADK, the most common properties you'll configure are: - `instruction` — Also known as the system prompt or developer message, this defines the agent’s behavior and tone. - `model` — Specifies which LLM to use. In this case, it points to a Clarifai-hosted model. - `tools` — Define the actions your agent can take, such as calling APIs or fetching data. These can be implemented as regular Python functions. You'll also need the `Runner` class to run your agent. ### Example Here is an example of an interactive AI agent that is designed to act as a helpful weather assistant, leveraging a Clarifai-hosted LLM and a custom tool to fetch weather information. {GoogleADK} Example Output {GoogleADKOutput} ### Additional Examples To learn more about building AI agents with the Google ADK, see the following examples: - [Repository for Clarifai-Powered Google ADK Agents](https://github.com/Clarifai/examples/tree/main/agents/Google-ADK) ## CrewAI You can use [CrewAI](https://docs.crewai.com/introduction) to build agentic AI apps that make use of Clarifai models. CrewAI is a Python framework that empowers developers to create autonomous AI agents tailored to a wide range of use cases. ### Installation The following command will install the core CrewAI package. pip install crewai The `crewai` package provides a wide range of components for interacting with the CrewAI ecosystem, such as `Agent`, `Task`, `Crew`, `Process`, and `LLM`. CrewAI uses LiteLLM to integrate with a wide range of [LLMs](https://docs.crewai.com/learn/llm-connections). With the `LLM` class, you can easily connect to any OpenAI-compatible model, including those hosted by providers like Clarifai. ### Build an Agent When building an agent with CrewAI, the most common properties you'll configure are: - `Crew` — The top-level organization that orchestrates how agents work together and manages the overall workflow and process. - `AI Agents` — These are specialized team members, each with distinct personalities and expertise. An AI agent can: - Have a specific `role` (like _"Senior Research Analyst"_). - Work towards defined a `goal` (like _"Uncover cutting-edge developments and facts on a given topic"_). - Have a `backstory` that shape its approach and personality. - Use designated `tools` to accomplish its work. - Delegate tasks. - Specify which `llm` to use. In this case, it points to a Clarifai-hosted model. - `Process` — The workflow management system that defines how agents collaborate. For example, the `sequential` process ensures agents work one after another. - `Tasks` — The individual assignments that drive the work forward. Each task can: - Have a clear `description` of what needs to be done. - Define `expected_output` format and content. - Be assigned to a specific `agent`. - Feed into the larger process. ### Example Here is an example of a specialized AI agent powered by a Clarifai-hosted model. When you provide a topic, the agent performs a detailed analysis and then generates responses. {CrewAI} Example Output {CrewAIOutput} ### Additional Examples To learn more about building AI agents with CrewAI, see the following examples: - [YouTube Video: Building an AI Blog Writing Agent With Clarifai and CrewAI](https://www.youtube.com/watch?v=1XZT2wD4UzQ&t=25s) - [Blog Post: Build an AI Agent From Scratch With CrewAI and Clarifai](https://www.clarifai.com/blog/build-an-ai-agent-from-scratch-with-crewai-and-clarifai) - [Repository for Clarifai-Powered CrewAI Agents](https://github.com/Clarifai/examples/tree/main/agents/CrewAI) ## Vercel You can use the [Vercel](https://vercel.com/docs/agents) platform to build agentic AI apps that make use of Clarifai models. Using the [Vercel AI SDK](https://ai-sdk.dev/) — a TypeScript toolkit for working with OpenAI-compatible APIs — you can easily connect to and interact with Clarifai's hosted AI models. ### Installation You can install the following necessary packages: - `ai` — This is the Vercel AI SDK, the main library. - `@ai-sdk/openai-compatible`— The [OpenAI Compatible Provider ](https://ai-sdk.dev/providers/openai-compatible-providers) package for the AI SDK, which allows you to connect to OpenAI-compatible APIs (like Clarifai's). - `zod`— For schema validation of tool parameters. This is the combined commands for installing them: npm install ai @ai-sdk/openai-compatible zod ### Build an Agent When building an agent with the Vercel AI SDK, the core function you can use to interact with the language model is `generateText`. It serves as the main entry point for generating responses and orchestrating tool use. The most common properties you'll configure in a `generateText` call are: - `model` — Specifies which LLM to use. In this case, it points to a Clarifai-hosted model. - `maxSteps` — Sets the maximum number of reasoning or tool-use steps the agent can take before halting. - `tools` — Provides the agent with a set of callable tools. Tools are defined using the `tool` function, which requires a description, a parameter schema (using `zod`), and an `execute` function that defines the tool's behavior. - `prompt` — The user's input or query that the agent will respond to. ### Example Here is an example that demonstrates how to build an agent using the Vercel AI SDK that interacts with a Clarifai-hosted language model. {Vercel} Example Output {VercelOutput} ### Additional Examples To learn more about building AI agents with the Vercel AI SDK, see the following examples: - [Repository for Clarifai-Powered Vercel AI SDK Agents](https://github.com/Clarifai/examples/tree/main/nodejs/vercel-ai-sdk) ## Agno You can use [Agno](https://github.com/agno-agi/agno) to build agentic AI apps that make use of Clarifai models. Agno is a framework for building AI agents that can use tools, maintain memory, and work in teams. ### Installation You can install the following necessary packages: - `agno` — The Agno framework itself, which provides components such as `Agent`, `Team`, and `OpenAILike`. - `openai` — The OpenAI client library used for the OpenAI-compatible connection to Clarifai. - `ddgs` — Provides the `DuckDuckGoTools` integration for web search capabilities. - `yfinance` — Provides the `YFinanceTools` integration for financial data lookups. This is the combined command for installing them: pip install agno openai ddgs yfinance After the installation, you can use the `OpenAILike` model class in Agno to connect to Clarifai's OpenAI-compatible endpoint. ### Build an Agent When building an agent with Agno, the most common properties you'll configure are: - `model` — Specifies which LLM to use via the `OpenAILike` class, pointing to a Clarifai-hosted model. Set `api_key` to your `CLARIFAI_PAT` and `base_url` to Clarifai's OpenAI-compatible endpoint. - `tools` — Define the actions your agent can take, such as web search or financial data lookups. Agno provides ready-made tool integrations (e.g., `DuckDuckGoTools`, `YFinanceTools`). - `description` — A short description that shapes the agent's persona and behavior. - `markdown` — When set to `True`, the agent formats its responses in Markdown. You can also compose multiple agents into a `Team` to tackle complex, multi-step tasks collaboratively. ### Examples #### Simple Agent Here is an example of a simple AI assistant powered by a Clarifai-hosted model. When asked a question, the agent generates a direct response without using any external tools. {Agno} Example Output {AgnoOutput} #### Agent with Tools Here is an example of a research assistant powered by a Clarifai-hosted model and equipped with a web search tool. When given a query, the agent searches the web and returns a summarized response. {AgnoTools} Example Output {AgnoToolsOutput} #### Multi-Agent Team Here is an example of a multi-agent team where a web research agent and a financial analysis agent collaborate to answer a complex query. A shared Clarifai-hosted model coordinates the team. {AgnoTeam} Example Output {AgnoTeamOutput} ### Additional Examples To learn more about building AI agents with Agno, see the following examples: - [Blog: Building AI Agents with Agno and GPT-OSS 120B](https://www.clarifai.com/blog/building-ai-agents-with-agno-and-gpt-oss-120b) - [Repository for Clarifai-Powered Agno Agents](https://github.com/Clarifai/examples/tree/main/agents/Agno) ## Additional Examples To learn more about building AI agents with other toolkits, see the following examples: - [Notebook: Conditional Routing Agent for selecting a suitable model for a given task](https://github.com/Clarifai/examples/blob/main/agents/conditional_agent_router.ipynb) --- ## Deploy Open-Source MCP Servers # Deploy Open-Source MCP Servers **Upload, deploy, and interact with open-source MCP servers on Clarifai** Besides building a [custom MCP](mcp.md#build-an-mcp-server) (Model Context Protocol) server for the Clarifai platform, you can also upload any open-source MCP server and expose it as a managed API endpoint — just like any model in the platform. You can easily integrate third-party MCP servers with Clarifai by simply adding the `mcp_server` configuration to your [`config.yaml`](#step-3-prepare-the-configyaml-file) file. This allows you to: - Expose MCP servers as HTTP APIs accessible through Clarifai - Use the FastMCP client to interact with deployed MCP servers - Seamlessly integrate MCP tools with LLMs to extend model capabilities import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import BuildLogs from "!!raw-loader!../../../code_snippets/python-sdk/inference/deploy_mcp_logs.txt"; import FastMCPClient from "!!raw-loader!../../../code_snippets/python-sdk/inference/deploy_mcp_fastmcp.py"; import IntegrateLLM from "!!raw-loader!../../../code_snippets/python-sdk/inference/deploy_mcp_integrate_llm.py"; import FastMCPOutput from "!!raw-loader!../../../code_snippets/python-sdk/inference/deploy_mcp_fastmcp_output.txt"; ## Step 1: Perform Prerequisites ### Get an MCP Server You can get open-source MCP servers from third-party repositories, such as [mcpservers.org](https://mcpservers.org/) or [mcp.so](https://mcp.so/). :::note objective For this example, let's use the [DuckDuckGo MCP server](https://github.com/nickclyde/duckduckgo-mcp-server) to demonstrate how to upload and deploy an open-source MCP server on Clarifai. This server provides tools for web search, browsing, and information retrieval, and requires no authentication tokens or secrets — making it easy to deploy and use. You can also follow its tutorial [here](https://github.com/Clarifai/runners-examples/tree/main/mcp/browser-mcp-server). ::: ### Get an Agentic Model Integrating large language models (LLMs) with MCP servers enables agentic capabilities, allowing models to discover and use external tools autonomously to complete tasks. MCP servers expose functionalities that models can invoke as function-calling tools during conversations. With MCP server integration, an agentic model can iteratively discover tools, execute them, and reason over the results to produce more capable and context-aware responses. > **Note:** > For a model to support agentic behavior through MCP servers on the Clarifai platform, it must extend the standard `OpenAIModelClass` with the `AgenticModelClass`. This enables: > > * Tool discovery and execution handled by the agentic model class > * Iterative tool calling within a single predict or generate request > * Compatibility with the OpenAI-compatible API and Clarifai SDKs > * Support for both streaming and non-streaming modes > > The `AgenticModelClass` manages the full agentic loop: it discovers available MCP tools at model load time, injects them into the LLM context, and iteratively calls tools and feeds results back to the model until a final response is produced. > > You can see an example implementation of `AgenticModelClass` in [this `1/model.py`](https://github.com/Clarifai/runners-examples/blob/main/llm/agentic-gpt-oss-20b/1/model.py) file. :::tip To upload a model with agentic capabilities, simply use the `AgenticModelClass` — all other functionalities and steps remain the [same as uploading](https://docs.clarifai.com/compute/upload/) a standard model on Clarifai. You can follow [this example](https://github.com/Clarifai/runners-examples/tree/main/llm/agentic-gpt-oss-20b). ::: These are some example models with agentic capabilities enabled: * [Qwen3-30B-A3B-Instruct-2507](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-Instruct-2507) * [Qwen3-30B-A3B-Thinking-2507](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-Thinking-2507) * [Qwen3-Coder-30B-A3B-Instruct](https://clarifai.com/qwen/qwenCoder/models/Qwen3-Coder-30B-A3B-Instruct) ### Install Packages Install the following Python packages to work with the DuckDuckGo Browser MCP server: * `clarifai` — The latest version of the Clarifai Python SDK, required to integrate your MCP server with the Clarifai platform. This package also comes with the Clarifai [Command Line Interface (CLI)](https://docs.clarifai.com/additional-resources/api-overview/cli), which you’ll use to upload the server. * `fastmcp` — The core framework for interacting with MCP servers. * `openai` — This leverage Clarifai’s [OpenAI-compatible endpoint](https://docs.clarifai.com/compute/inference/#predict-with-openai-compatible-format) endpoint to run inferences using the OpenAI client library * `anyio` — An asynchronous I/O library used by FastMCP. * `requests` — A lightweight HTTP client for making HTTP requests. * `mcp` — The Model Context Protocol library. You can run the following command to install them: pip install --upgrade clarifai fastmcp openai anyio requests mcp ### Get Credentials You need to have the following Clarifai credentials: - **App ID** — Create a Clarifai [application](https://docs.clarifai.com/create/applications/create) and get its ID. This is where your MCP server will reside on the Clarifai platform. - **User ID** — In the collapsible left sidebar, select **Settings** and choose **Account** from the dropdown list. Then, locate your user ID. - **[Personal Access Token](https://docs.clarifai.com/control/authentication/pat)** (PAT) — From the same **Settings** option, choose **Secrets** to generate or copy your PAT. This token is used to authenticate your connection with the Clarifai platform. Then, set the `CLARIFAI_PAT` as an environment variable. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Create Files On the Clarifai platform, MCP servers are treated just like models and follow the same underlying architecture. To upload an MCP server, you need to create a project directory and organize your files according to Clarifai’s custom model requirements, as shown below: ```text your_model_directory/ ├── 1/ │ └── model.py ├── requirements.txt ├── config.yaml ├── Dockerfile └── client.py ``` * **your_model_directory/** — The root directory containing all files related to your MCP server. * **1/** — A required subdirectory that contains the model implementation (*note that the folder name is **1***). * **model.py** — Implements the core logic of the MCP server. * **requirements.txt** — Specifies the Python dependencies required to run the server. * **config.yaml** — Defines metadata and configuration settings used when uploading the MCP server to Clarifai. * **Dockerfile** — Defines the runtime environment used to build and run your MCP server on the Clarifai platform. * **client.py** — An example client script you can use to interact with the MCP server after it has been uploaded. ### Create a Cluster and Nodepool You'll need to deploy your MCP server to a dedicated compute cluster and nodepool. This action provisions the necessary resources to run your server and handle requests efficiently. Learn how to create a cluster and nodepool [here](https://docs.clarifai.com/compute/deployments/clusters-nodepools). > **Note:** Ensure that your cluster and nodepool meet the compute resource requirements specified in your [`config.yaml`](#step-3-prepare-the-configyaml-file) file. ## Step 2: Prepare `model.py` File When building a custom MCP server from scratch, `model.py` is where you implement the server’s core logic, including defining and exposing tools. However, when uploading an open-source MCP server, you do not need to reimplement this logic. Instead, you only need to define a class that inherits from `StdioMCPModelClass`, which is designed to run and manage stdio-based MCP servers, such as the Browser MCP server. Here is the `model.py` file that defines a `StdioMCPModelClass` for the Browser MCP server: ```python from clarifai.runners.models.stdio_mcp_class import StdioMCPModelClass class BrowserMCPServerClass(StdioMCPModelClass): pass ``` The `StdioMCPModelClass` abstracts away the complexity of managing stdio-based MCP servers. By inheriting from it and configuring the `mcp_server` section in `config.yaml`, your server is ready to be uploaded. Specifically, `StdioMCPModelClass` automatically: * Starts the MCP server as a single long-lived stdio process (e.g., a Node.js or Python subprocess) during `load_model()` * Opens a persistent FastMCP client session that is reused for all subsequent requests, reducing per-request overhead * Discovers all available MCP tools at startup * Exposes the tools through an HTTP API * Handles tool execution and response formatting * Supports configuration via YAML, including environment variables and injected secrets This makes it easy to deploy open-source MCP servers on Clarifai with minimal code. ## Step 3: Prepare the `config.yaml` File The `config.yaml` file defines the build, deployment, and runtime configuration for a custom model — or, in this case, an MCP server — on the Clarifai platform. It tells Clarifai how to build the execution environment and where the server should live within your account. When integrating an open-source MCP server, this file is also where you specify the server’s `mcp_server` configuration, which you can obtain from the repository where it's hosted. Here is the `config.yaml` file for the Browser MCP server: ```yaml model: id: browser-mcp-server app_id: app_id user_id: user_id model_type_id: mcp build_info: python_version: "3.11" inference_compute_info: cpu_limit: 1000m cpu_memory: 1Gi num_accelerators: 0 mcp_server: command: "uvx" args: ["duckduckgo-mcp-server"] ``` Let’s break down what each part of the configuration does. - `model` — Defines where the MCP server will be uploaded on the Clarifai platform: - `id` — Provide a unique identifier for the model (server) - `app_id` — Provide your Clarifai app where the server will reside - `user_id` — Provide your Clarifai user ID - `model_type_id` — Specifies the [model type](https://docs.clarifai.com/create/models/#model-types); use `mcp` for MCP servers - `build_info` — Specifies the Python version used to build the runtime environment. Note that Clarifai currently supports Python 3.11 and 3.12 (default). - `inference_compute_info` — Defines the compute resources allocated when the MCP server is running: * `cpu_limit: 1000m` — Allocates 1 CPU core * `cpu_memory: 1Gi` — Allocates 1 GB of RAM * `num_accelerators: 0` — No hardware accelerators (e.g., GPUs), which is typical for MCP servers > **Note:** Ensure your Clarifai compute [cluster and nodepool](#create-a-cluster-and-nodepool) meet these requirements before deploying the model. - `mcp_server` — Specifies how the MCP server process is started: * `command` — The executable used to launch the server (e.g., `npx`, `uvx`, `python`) * `args` — Arguments passed to the command :::note Examples of other MCP servers To deploy a different MCP server, simply update the `mcp_server` section in `config.yaml`. - [GitHub MCP Server](https://github.com/modelcontextprotocol/servers?tab=readme-ov-file#using-an-mcp-client) > **Note:** Learn how to deploy it on Clarifai [here](https://github.com/Clarifai/runners-examples/blob/main/mcp/github-mcp-server/README.md). ```yaml mcp_server: command: "npx" args: ["-y", "@modelcontextprotocol/server-github"] ``` - Custom Python MCP Server (a custom, user-implemented MCP server) ```yaml mcp_server: command: "python" args: ["-m", "my_mcp_server"] ``` ::: ## Step 4: Define Dependencies in `requirements.txt` The `requirements.txt` file specifies all Python packages required by your MCP server. During deployment, Clarifai uses this file to automatically install the necessary dependencies, ensuring the server runs correctly in its runtime environment. Here is the `requirements.txt` file for the custom model — or, in this case, the MCP server — we want to upload: ```text clarifai==12.1.7 anyio mcp==1.26.0 fastmcp==2.14.5 requests>=2.31.0 ``` ## Step 5: Define Dockerfile The `Dockerfile` defines the container environment used to build and run the MCP server on the Clarifai platform. When you upload an MCP server, Clarifai builds a Docker image from this file and uses it to execute your server in a secure, isolated container — exactly the same way custom models are deployed. The Dockerfile is responsible for: - Selecting the base Python image - Installing system-level dependencies (if any) - Installing Python dependencies from `requirements.txt` - Copying your MCP server code into the container - Defining the entry point that starts the MCP server Here is the `Dockerfile` file for the Browser MCP server: ```dockerfile # syntax=docker/dockerfile:1.13-labs FROM --platform=$TARGETPLATFORM python:3.12-slim COPY --link requirements.txt /home/nonroot/requirements.txt # Update clarifai package so we always have latest protocol to the API. Everything should land in /venv RUN ["pip", "install", "--no-cache-dir", "-r", "/home/nonroot/requirements.txt"] RUN ["pip", "show", "--no-cache-dir", "clarifai"] # Set the NUMBA cache dir to /tmp # Set the TORCHINDUCTOR cache dir to /tmp # The CLARIFAI* will be set by the templaing system. ENV NUMBA_CACHE_DIR=/tmp/numba_cache \ TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_cache \ HOME=/tmp \ DEBIAN_FRONTEND=noninteractive ##### # Download checkpoints if config.yaml has checkpoints.when = "build" COPY --link=true config.yaml /home/nonroot/main/ # RUN ["python", "-m", "clarifai.cli", "model", "download-checkpoints", "/home/nonroot/main", "--out_path", "/home/nonroot/main/1/checkpoints", "--stage", "build"] ##### # Copy in the actual files like config.yaml, requirements.txt, and most importantly 1/model.py # for the actual model. # If checkpoints aren't downloaded since a checkpoints: block is not provided, then they will # be in the build context and copied here as well. COPY --link=true 1 /home/nonroot/main/1 # At this point we only need these for validation in the SDK. COPY --link=true requirements.txt config.yaml /home/nonroot/main/ # Add the model directory to the python path. ENV PYTHONPATH=${PYTHONPATH}:/home/nonroot/main \ CLARIFAI_PAT=${CLARIFAI_PAT} \ CLARIFAI_USER_ID=${CLARIFAI_USER_ID} \ CLARIFAI_RUNNER_ID=${CLARIFAI_RUNNER_ID} \ CLARIFAI_NODEPOOL_ID=${CLARIFAI_NODEPOOL_ID} \ CLARIFAI_COMPUTE_CLUSTER_ID=${CLARIFAI_COMPUTE_CLUSTER_ID} \ CLARIFAI_API_BASE=${CLARIFAI_API_BASE:-https://api.clarifai.com} WORKDIR /home/nonroot/main # Finally run the clarifai entrypoint to start the runner loop and local runner server. # Note(zeiler): we may want to make this a clarifai CLI call. ENTRYPOINT ["python", "-m", "clarifai.runners.server"] CMD ["--model_path", "/home/nonroot/main"] ############################# ``` ## Step 6: Upload to Clarifai To upload your open-source MCP server to the Clarifai platform, navigate to the server’s root directory and run: clarifai model upload . --skip_dockerfile The [`--skip_dockerfile`](https://docs.clarifai.com/compute/upload/#skip-dockerfile) flag prevents the CLI from generating a default Dockerfile and instructs it to use the Dockerfile provided in your project directory. This command will: - Stream Docker build logs directly to your terminal for real-time monitoring and troubleshooting - Build the Docker image defined in your Dockerfile - Upload the MCP server to your Clarifai account - Make the server available for inference via the Clarifai HTTP API Build Logs Example {BuildLogs} :::note Once the upload completes, the build logs will include an example code snippet that you can copy into your `client.py` script. This snippet contains the URL of your deployed MCP server, which your AI agents or client applications will use to communicate with the server. The MCP server URL is constructed using the following format: `https://api.clarifai.com/v2/ext/mcp/v1/users/{user-id}/apps/{app-id}/models/{model-id}`. ::: ## Step 7: Deploy the Model After uploading the MCP server, you must deploy it to a dedicated compute cluster and nodepool. Deployment provisions the compute resources required to run the server and handle incoming requests. Learn how to perform deployments [here](https://docs.clarifai.com/compute/deployments/deploy-model). > **Note:** You can also deploy the server by following the interactive prompts shown when uploading it to Clarifai from the terminal. Once deployed, the server is automatically referenced during inference. You do not need to specify the deployment explicitly in your client code. ## Step 8: Interact with the Server Once the open-source MCP server is deployed, you can create a client script to communicate with it and invoke the exposed MCP tools. This allows agentic models to discover and use the server’s capabilities programmatically. ### Use with FastMCP Client Here is an example of interacting with the MCP server directly using the FastMCP client without going through an LLM. This is typically the first step before invoking tools, executing actions, or wiring the MCP server into an agentic LLM workflow. > **Note:** Remember to replace the placeholder values with the corresponding identifiers for your MCP server. You may also use any compatible agentic LLM. {FastMCPClient} Example Output {FastMCPOutput} The above snippet demonstrates how to: - Authenticate to an MCP server hosted on Clarifai - Establish an asynchronous, stream-capable MCP connection - Discover the tools and their capabilities as exposed by that MCP server using the `client.list_tools()` method. This is a key MCP concept: models and agents can dynamically discover what tools are available at runtime, rather than hard-coding them. > **Note:** The DuckDuckGo MCP server provides various tools for web search and information retrieval. For example, `ddg_search` is used for searching the web using DuckDuckGo. :::note If you encounter a `Server error '503 Service Unavailable'` while calling the server, it typically indicates that the model is in a cold state and still warming up. You may wait a moment before trying the request again. ::: ### Integrate with LLMs Here is an example of how to bridge an MCP server with an agentic LLM on Clarifai so the model can discover tools, decide when to use them, and call them during inference — all through Clarifai’s OpenAI-compatible endpoint. {IntegrateLLM} The above snippet demonstrates how to: - Connect to a Clarifai-hosted MCP server and discover its available tools - Convert MCP tools into OpenAI function-calling–compatible definitions - Send a chat completion request to an agentic LLM using Clarifai’s OpenAI-compatible API - Allow the model to autonomously decide whether to invoke a tool (`tool_choice="auto"`) - Execute any requested tool calls via MCP and return the results to the model for final response generation :::tip [Click here](https://docs.clarifai.com/compute/inference/mcp-servers) to learn more about integrating MCP servers with LLMs and running inferences on the Clarifai platform. ::: --- ## Build Custom MCP Servers # Build Custom MCP Servers **Build performant MCP servers with FastMCP for Clarifai** [The Model Context Protocol (MCP)](https://modelcontextprotocol.io/introduction) is an open standard developed by Anthropic that acts as a universal language for AI models, particularly large language models (LLMs), to interact with external data sources (like GitHub, Slack, or databases) and extend their capabilities. With MCP, you can build intelligent agents and complex workflows on top of LLMs, enabling secure and efficient access to contextual information. ## How it Works At its core, MCP follows a [client-server architecture](https://www.clarifai.com/blog/mcp-vs-a2a-clearly-explained) where a host application (like an AI chatbot or an IDE with AI features) can connect to an MCP server. The server acts as a gateway to specific external data, tools, or functionalities. An MCP server can expose several key capabilities — most notably, tools that function as callable actions for LLMs, such as updating records or interacting with external systems. While MCP provides the specification, implementing clients and servers that adhere to it can involve substantial boilerplate code and intricate protocol handling. This is exactly where [FastMCP](https://github.com/jlowin/fastmcp) excels. FastMCP is a high-level framework that significantly simplifies the development of MCP servers and clients. Clarifai allows you to build performant MCP servers with FastMCP by providing the necessary infrastructure needed to define, deploy, and manage custom MCP servers at scale. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import MCPModelPyFile from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/mcp_1.py"; import MCPConfigFile from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/mcp_2.yaml"; import MCPRequirementsFile from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/mcp_3.txt"; import MCPClientFile from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/mcp_4.py"; import MCPOutputExample from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/mcp_5.txt"; import BuildLogsExample from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/mcp_6.txt"; ## Build an MCP Server Building an MCP server with FastMCP using Clarifai follows the same intuitive pattern as [building and uploading models](https://docs.clarifai.com/compute/models/upload/) on the platform. Let's demonstrate how you can build a simple MCP server using the FastMCP framework. This server will expose callable tools for LLMs or other AI agents, and we'll upload it to the Clarifai platform as a custom model, making its functionalities accessible within the Clarifai ecosystem. ### Step 1: Perform Prerequisites #### Install Packages You need to install the following Python packages: - `clarifai` – The latest version of the Clarifai Python SDK required for integrating your MCP server with the Clarifai platform. This also installs the Clarifai [Command Line Interface (CLI)](https://docs.clarifai.com/resources/api-overview/cli), which we'll use for uploading the server. - `fastmcp` – This is the core framework used to define and manage the MCP server. - `pydantic` – [Pydantic](https://github.com/pydantic/pydantic) is essential for data validation and defining the structure (schema) of your tool arguments, particularly using `Field`. We'll pair it with `Any` and `Annotated` from Python's `typing` module to add vital metadata, like descriptions, to function arguments, which is key for `fastmcp`'s type hinting and automatic schema generation. You can run the following command to install them: pip install --upgrade fastmcp pydantic clarifai Or, you can define the packages in a [`requirements.txt`](#step-4-define-dependencies-in-requirementstxt) file and run the following command to install them: pip install -r requirements.txt #### Set a PAT Key You need to have a [Personal Access Token](https://docs.clarifai.com/control/authentication/pat) (PAT) to authenticate your connection with the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. Then, set the `CLARIFAI_PAT` as an environment variable. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE #### Create Files Create a project directory and organize your files as indicated below to fit the requirements of building custom models for the Clarifai platform. ```text your_model_directory/ ├── 1/ │ └── model.py ├── requirements.txt └── config.yaml └── client.py ``` :::tip You can automatically generate these files by running this CLI command: [`clarifai model init --toolkit mcp`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-init). You can then edit them as needed. ::: - **your_model_directory/** – The root directory containing all files related to your server. - **1/** – A subdirectory that holds the model file (_Note that the folder is named as **1**_). - **model.py** – Contains the main logic for the MCP server implementation. - **requirements.txt** – Lists the Python dependencies required to run your server. - **config.yaml** – Contains metadata and configuration settings needed for uploading the model to Clarifai. - **client.py** – Only contains an example client for demonstrating MCP usage after uploading the model to Clarifai. ### Step 2: Prepare `model.py` File `model.py` is the main file where you'll implement your MCP server's logic. Here's the `model.py` file for the custom model (or, in this case, a server) we want to build: {MCPModelPyFile} Let’s break down what each part of the file does. #### a. Initialize the FastMCP Server You'll start by creating an instance of the `FastMCP` class to set up your server. Here are the key arguments you can pass to the `FastMCP` constructor: * `name` – An optional identifier for your MCP server. This is useful for distinguishing it in logs or when managing multiple servers in client applications. * `instructions` – An optional, short description that helps clients understand the server's purpose or how to best interact with its functionalities. * `stateless_http` – A boolean flag (`True`/`False`) that configures the server to operate over stateless HTTP. Turning this on is ideal for lightweight deployments, as it simplifies communication by not requiring persistent sessions. #### b. Define Tools An empty MCP server isn't very useful; its power comes from the **tools** it exposes. We define these tools by applying the `@server.tool(...)` decorator directly to Python functions. This decorator registers the function as an MCP "tool," making it discoverable and invokable by any MCP client (like an LLM). Each tool is enriched with essential metadata: * Tool-level metadata – The `@server.tool()` decorator itself takes a `name` and `description` to explain the tool's overall purpose. * Input descriptions – The tool's input arguments are precisely described using `Annotated` in conjunction with `Field`. In our example, we define a tool and a resource: * `hello` – A tool that takes a `name` parameter and returns a greeting message. * `config://version` – A resource that returns the server version. :::tip For details on adding other useful MCP server components, like `Resources` and `Prompts`, you can refer to the [official MCP documentation](https://modelcontextprotocol.io/docs/concepts/resources). ::: #### c. Define Clarifai's Model Class The custom-named model class serves as the integration point between your MCP server and the Clarifai platform. You must define it by subclassing Clarifai's `MCPModelClass` and implementing the `get_server()` method. This method returns the `FastMCP` server instance (such as `server`) that Clarifai should use when running your model. When Clarifai runs the model, it calls `get_server()` to load your MCP server and expose its defined tools and capabilities to LLMs or other agents. :::tip Alternative MCP base classes - **`StdioMCPModelClass`** — Use this when wrapping an existing stdio-based MCP server (e.g., an open-source server launched as a subprocess). See [Deploy Open-Source MCP Servers](deploy-mcp.md) for a full walkthrough. - **`AgenticModelClass`** — Use this when building an LLM-based model that should autonomously discover and call MCP tools during inference. See the agentic model section in [Deploy Open-Source MCP Servers](deploy-mcp.md#get-an-agentic-model) for details. ::: ### Step 3: Prepare `config.yaml` File The `config.yaml` file is used to configure the build and deployment settings for a custom model on the Clarifai platform. It tells Clarifai how to build your model's environment and where to place it within your account. This is the `config.yaml` file for the custom model (or, in this case, a server) we want to build: {MCPConfigFile} Let’s break down what each part of the file does. * **`model.id`** – A unique identifier for your model. You can choose any name you want. * **`model.model_type_id`** – Set to `"mcp"` to indicate this is an MCP server model. * **`compute.instance`** – The compute instance type for deployment. Run `clarifai list-instances` to see all available options. * **`checkpoints`** – (Optional) Uncomment to auto-download model checkpoints from Hugging Face at runtime. > `user_id` and `app_id` are auto-filled from your [active context](https://docs.clarifai.com/resources/api-overview/cli#clarifai-config) at deploy time. You don’t need to add them manually. ### Step 4: Define Dependencies in `requirements.txt` The `requirements.txt` file lists all the Python packages your MCP server depends on. Clarifai uses this file during deployment to automatically install the necessary libraries, ensuring your server runs correctly. Here's the `requirements.txt` file for the custom model (or, in this case, a server) we want to build: {MCPRequirementsFile} ### Step 5: Test the Model Locally Before uploading your server to the Clarifai platform, you can test it locally to catch any typos or misconfigurations in the code. #### a. Local Runners You can use the Clarifai CLI to test and run your model as a local development runner. You can learn how to use the tool [here](https://docs.clarifai.com/compute/local-runners/). clarifai model serve #### b. MCP Inspector You can also use the [MCP Inspector](https://github.com/modelcontextprotocol/inspector) to easily test and debug your MCP server through a user-friendly, web-based interface that allows you to connect to your server and explore its capabilities. To launch the Inspector, run the following command. npx @modelcontextprotocol/inspector Once the interface opens, set the `Transport Type` to `Streamable HTTP` and provide the URL of your deployed MCP server on the Clarifai platform (_learn how to get the URL in the next step_). In the Authentication section, enter `Authorization` as the Header Name, and supply your Clarifai PAT in the `Bearer Token` field. After entering these details, click the **Connect** button to establish a connection and begin inspecting your server’s capabilities. MCP Inspector UI ![](/img/others/mcp-inspector.png) ### Step 6: Deploy to Clarifai You can deploy the MCP server to the Clarifai platform using a single command. Navigate to its root directory and run: clarifai model deploy This command handles everything — building, uploading, and deploying your model. The CLI automatically creates the required compute infrastructure (cluster, nodepool, deployment). :::note Once the deployment is complete, the output will display the URL of your deployed MCP server, which you'll use to enable your AI agents or clients to communicate with the server. This URL is constructed by combining the MCP API base URL (`https://api.clarifai.com/v2/ext/mcp/v1`) with your specific Clarifai identifiers: your user ID, app ID, and the model ID of your deployed MCP server. For example: `https://api.clarifai.com/v2/ext/mcp/v1/users/user-id/apps/app-id/models/model-id`. ::: Build Logs Example {BuildLogsExample} **Note:** If you make changes to your server code and re-deploy it, Clarifai automatically creates a new version of your model. :::tip Alternative: Upload Only If you prefer to upload the model first and deploy separately (e.g., to a specific nodepool), you can use `clarifai model upload` and then deploy via the UI or `clarifai deployment create`. See the [deployments guide](https://docs.clarifai.com/compute/deployments/deploy-model) for details. ::: ### Step 8: Interact With Server After uploading your server to Clarifai, you can create a separate client script to communicate with it and invoke its tools. Here’s an example `client.py` that demonstrates how to interact with your deployed MCP server: {MCPClientFile} Let’s break down what each part of the file does. - **Set up connection** – We establish the connection to your deployed MCP server. The `StreamableHttpTransport` instance handles this, serving as a specialized FastMCP transport for HTTP communication. Its `url` parameter points directly to your MCP model on Clarifai. Instead of hardcoding this URL, we use `ClarifaiUrlHelper().mcp_api_url()` as a convenient utility. Based on your current [context](https://docs.clarifai.com/resources/api-overview/cli#clarifai-context), this helper provides the base URL where your MCP server lives and exposes its tools, ensuring flexibility and correctness. - **Main logic** – We asynchronously open the MCP client session by initializing `fastmcp.Client` with the configured `transport`. Once connected, we list all available tools exposed by the server and call them. Note the use of `async` and `await`; FastMCP clients operate asynchronously, requiring an `async` function and an `async with Client` block to properly manage the client's lifecycle. - **Execution** – Finally, `asyncio.run(main())` executes the entire asynchronous interaction. To execute the client example, run the following command: python client.py :::note If you encounter a `Server error '503 Service Unavailable'` while calling the server, it typically indicates that the model is in a cold state and still warming up. You may wait a moment before trying the request again. ::: Example Output {MCPOutputExample} ## Additional Examples To learn more about building MCP servers using Clarifai, see the following examples: - [Build an Interactive AI Agent with Clarifai's LLMs and MCP Tools](https://github.com/Clarifai/examples/tree/main/agents/mcp/chat-mcp-agent) - [Empowering AI Agents: Clarifai LLMs with MCP Tools for Task Automation](https://github.com/Clarifai/examples/tree/main/agents/mcp/llm-mcp-agent) - [A FastMCP Server for Web Browse and Research Tools](https://github.com/Clarifai/runners-examples/tree/main/mcp/browser-tools) - [A FastMCP Server for Google Drive Integration](https://github.com/Clarifai/runners-examples/tree/main/mcp/google-drive) - [A FastMCP Server for PostgreSQL Database Operations](https://github.com/Clarifai/runners-examples/tree/main/mcp/postgres) --- ## Supported Cloud Instances # Supported Cloud Instances **Learn about the instance types we support** **Jump to a cloud provider:** - [Amazon Web Services (AWS)](#amazon-web-services-aws-instances) - [Google Cloud Platform (GCP)](#google-cloud-platform-gcp-instances) - [Vultr Cloud Servers](#vultr-cloud-servers-instances) - [Oracle Distributed Cloud](#oracle-distributed-cloud-instances) - [Microsoft Azure](#microsoft-azure-instances) - [DigitalOcean](#digitalocean-instances) - [Lambda](#lambda-instances) --- We offer a range of instance types designed to handle a variety of machine learning workloads. These cloud instances vary in their CPU, RAM (Random Access Memory), and GPU configurations, which allow you to orchestrate the right balance of performance and cost for your use case. The instances listed on this page are available **on demand** — they can be provisioned immediately when you create a nodepool. Clarifai also offers the latest NVIDIA, AMD, and Google accelerators (including B300, B200, H100, MI355X, and TPU v7) that require reservations and are available on request. :::info - See the [pricing page](https://www.clarifai.com/pricing) to learn more about pricing for each instance type. - [Contact us](https://www.clarifai.com/explore/contact-us) to access GPU types not listed here, including instances that require dedicated provisioning. ::: ## Amazon Web Services (AWS) Instances ### G7E Instances The AWS G7E series provides the latest generation of high-performance NVIDIA RTX PRO 6000 GPUs, scaling from single-GPU setups to large multi-GPU clusters. These instances are designed for the most demanding deep learning training and large-scale inference workloads. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |--------------------|-----------|-------------------------------|------------|---------------------------|----------| | `g7e.48xlarge` | us-east-1 | 8x NVIDIA-RTX-PRO-6000 (96Gi) | 768Gi | 191.1 cores (1,886.99Gi) | $33.156 | | `g7e.24xlarge` | us-east-1 | 4x NVIDIA-RTX-PRO-6000 (96Gi) | 384Gi | 95.3 cores (939.79Gi) | $16.524 | | `g7e.12xlarge` | us-east-1 | 2x NVIDIA-RTX-PRO-6000 (96Gi) | 192Gi | 47.4 cores (466.19Gi) | $10.368 | | `g7e.8xlarge` | us-east-1 | 1x NVIDIA-RTX-PRO-6000 (96Gi) | 96Gi | 31.5 cores (230.74Gi) | $5.256 | | `g7e.4xlarge` | us-east-1 | 1x NVIDIA-RTX-PRO-6000 (96Gi) | 96Gi | 15.5 cores (112.34Gi) | $5.004 | **Key Features** - NVIDIA RTX PRO 6000 GPU — 96 GiB of GPU memory per card, delivering high memory capacity for very large models and long-context inference. - Scalable multi-GPU configurations — From a single GPU up to 8 GPUs (768 GiB combined GPU memory), suitable for distributed training of frontier models. - High CPU & RAM — Up to 191 cores and ~1.9 TiB RAM in the 8-GPU configuration, ensuring data pipelines match GPU throughput. **Example Use Cases** - Fine-tuning or running inference on large language models that require more than 48 GiB of GPU memory per card (e.g., 70B+ parameter models on a single GPU). - Multi-GPU distributed training of foundation models where high per-GPU memory is critical. :::tip Looking for the latest GPUs or TPUs? The newest accelerators — including **NVIDIA B300, B200, H100**, **AMD MI355X, MI300X**, and **Google TPU v7** — are available with reservations and are not automatically provisioned on demand. [Contact us](https://www.clarifai.com/explore/contact-us) to discuss availability and reserve capacity. ::: ### G6 Instances The AWS G6 series introduces next-generation NVIDIA GPUs for the most demanding machine learning and simulation workloads. These instances scale from single-GPU mid-tier setups to multi-GPU, high-memory configurations capable of handling large-scale model training. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |------------------|-----------|--------------------------|------------|------------------------|----------| | `g6e.12xlarge` | us-east-1 | 4x NVIDIA-L40S (44.99Gi) | 179.95Gi | 47.4 cores (351.44Gi) | $13.104 | | `g6e.2xlarge` | us-east-1 | 1x NVIDIA-L40S (44.99Gi) | 44.99Gi | 7.5 cores (57.95Gi) | $2.808 | | `g6e.xlarge` | us-east-1 | 1x NVIDIA-L40S (44.99Gi) | 44.99Gi | 3.5 cores (28.35Gi) | $2.34 | | `g6.2xlarge` | us-east-1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 7.5 cores (28.35Gi) | $1.224 | | `g6.xlarge` | us-east-1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 3.5 cores (13.55Gi) | $1.008 | **Key Features** - Next-Gen GPUs — NVIDIA L4 GPUs target efficient inference and fine-tuning, while L40S GPUs deliver high throughput for large-scale training. - Scalable GPU memory — From 22.49 GiB (L4) to nearly 180 GiB (multi-L40S), supporting workloads from mid-sized tasks to multi-modal foundation models. - High vCPU & RAM options — Up to 47.4 cores and 351 GiB RAM in `g6e.12xlarge`, enabling massive parallelism and data-heavy preprocessing. - Flexible tiers — Ranges from cost-efficient single-GPU instances to powerful multi-GPU setups. **Example Use Cases** - G6 (L4 instances) support mid-tier workloads such as fine-tuning BERT-large, or computer vision tasks like text-to-image generation and object recognition. - G6e (L40S instances) support advanced training workloads, including large-scale language models (e.g., GPT-4, T5-XL) or multi-modal tasks requiring both vision and language. ### G5 Instances The AWS G5 series provides high-performance GPU capabilities for workloads that demand more memory and compute power. These instances are optimized for deep learning training, large-scale inference, and advanced computer vision tasks. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |----------------|-----------|--------------------------|------------|-----------------------|----------| | `g5.2xlarge` | us-east-1 | 1x NVIDIA-A10G (22.49Gi) | 22.49Gi | 7.5 cores (28.35Gi) | $1.512 | | `g5.xlarge` | us-east-1 | 1x NVIDIA-A10G (22.49Gi) | 22.49Gi | 3.5 cores (13.55Gi) | $1.26 | **Key Features** - NVIDIA A10G GPU — High compute throughput and memory bandwidth, enabling faster training for deep learning and support for more complex models compared to T4 GPUs. - Scalable CPU & memory — From ~3.5 to ~7.5 vCPUs and 13.55 to 28.35 GiB of RAM, supporting data-heavy preprocessing, augmentation, and orchestration alongside GPU tasks. - Balanced design — Efficient for both training and inference, bridging the gap between lightweight GPU instances (like G4dn) and specialized multi-GPU clusters. **Example Use Cases** - Training mid-sized NLP models like GPT-2 or T5 for text generation, or training image segmentation models like UNet or Mask R-CNN for medical imaging. - Running object tracking, pose estimation, or other GPU-accelerated pipelines for video analytics. :::tip How to Choose the Best GPU - [NVIDIA A10 or NVIDIA L40S?](https://www.clarifai.com/blog/nvidia-a10-vs-l40s-gpus-for-ai-workloads) - [NVIDIA A10 or NVIDIA A100?](https://www.clarifai.com/blog/nvidia-a10-vs-a100-choosing-the-right-gpu-for-ai-workloads) - [NVIDIA B200 or NVIDIA H100?](https://www.clarifai.com/blog/nvidia-b200-vs-h100) - [NVIDIA A100 or NVIDIA H100?](https://www.clarifai.com/blog/nvidia-a100-vs.-h100-choosing-the-right-gpu-for-your-ai-workloads) ::: ### G4DN Instances The AWS G4dn series is built for GPU-accelerated workloads at a moderate scale. These instances combine NVIDIA T4 GPUs with balanced CPU and memory resources, making them well-suited for small-to-medium machine learning and inference tasks. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |-----------------|-----------|----------------------|------------|-----------------------|----------| | `g4dn.xlarge` | us-east-1 | 1x NVIDIA-T4 (15Gi) | 15Gi | 3.5 cores (13.86Gi) | $0.648 | **Key Features** - NVIDIA T4 GPU — Designed for inference and light training, offering strong efficiency for workloads at a lower cost compared to heavier GPU families. - vCPUs and RAM — Provides ~3.5 vCPUs (baseline) and ~13.86 GiB memory, giving enough capacity to manage GPU-accelerated tasks, preprocessing, and orchestration. - Balanced performance — Ideal when you need GPU acceleration without the overhead of large, expensive GPU instances. **Example Use Cases** - Inference workloads, such as running NLP models such as BERT-base for summarization, classification, or question answering. - Light training smaller models or experimenting with prototypes before scaling to larger GPU families. ### T3A Instances The AWS T3A series is intended for cost‑effective, general‑purpose workloads that do not require GPU acceleration. It provides a balanced mix of CPU and memory, making it suitable for lightweight use cases. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |-----------------|-----------|-----|------------|----------------------|----------| | `t3a.2xlarge` | us-east-1 | – | – | 7.5 cores (28.35Gi) | $0.36 | | `t3a.xlarge` | us-east-1 | – | – | 3.5 cores (13.55Gi) | $0.18 | | `t3a.large` | us-east-1 | – | – | 1.5 cores (6.4Gi) | $0.108 | | `t3a.medium` | us-east-1 | – | – | 1.5 cores (2.89Gi) | $0.072 | **Key Features** - vCPU (virtual CPUs) performance — Burstable performance that adapts to workload spikes. For example, `t3a.medium` provides ~1.5 vCPUs, while `t3a.2xlarge` scales up to ~7.5 vCPUs. - Memory — Ranges from 2.89 GiB to 28.35 GiB, enabling efficient in-memory data handling for lightweight to moderately intensive workloads. - Efficiency — Optimized for cost savings compared to other instance families, making them budget-friendly for everyday use. **Example Use Case** - Running simple models such as for classification or regression tasks. > **Note**: The CPU values (e.g., 1.5 cores) are baseline vCPU allocations expressed as fractional units. The instance can burst up to its full vCPU count (e.g., 2 vCPUs for `t3a.medium`) by consuming CPU credits. ### T4G Instances The AWS T4G series provides ARM-based (AWS Graviton2) CPU-only instances for cost-effective general-purpose workloads. These instances offer a strong performance-per-dollar ratio for lightweight inference and preprocessing tasks that do not require GPU acceleration. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |----------------|-----------|-----|------------|----------------------|----------| | `t4g.xlarge` | us-east-1 | – | – | 3.5 cores (13.55Gi) | $2.34 | | `t4g.large` | us-east-1 | – | – | 1.5 cores (6.4Gi) | $2.34 | | `t4g.medium` | us-east-1 | – | – | 1.5 cores (2.89Gi) | $2.34 | **Key Features** - AWS Graviton2 ARM processors — Deliver up to 40% better price performance compared to comparable x86 instances, making them cost-efficient for CPU-bound workloads. - Memory — Ranges from 2.89 GiB to 13.55 GiB, suitable for lightweight model serving and data processing. - Cost efficiency — A budget-friendly alternative to T3A for workloads that are compatible with ARM64 architecture. **Example Use Case** - Running lightweight models or preprocessing pipelines where ARM compatibility is confirmed and cost savings are a priority. > **Note**: Ensure your model container is built for `linux/arm64` when using T4G instances. ## Google Cloud Platform (GCP) Instances ### G4 Instances The GCP G4 series features NVIDIA RTX PRO 6000 GPUs, offering 96 GiB of GPU memory per card — the highest per-GPU memory available in the GCP lineup. These instances scale from a single GPU to 8-GPU configurations and are well-suited for large model inference and training workloads that demand high GPU memory capacity. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |--------------------|-------------|-------------------------------|------------|---------------------------|-----------| | `g4-standard-48` | us-central1 | 1x NVIDIA-RTX-PRO-6000 (96Gi) | 96Gi | 47.2 cores (169.03Gi) | $5.652 | | `g4-standard-96` | us-central1 | 2x NVIDIA-RTX-PRO-6000 (96Gi) | 192Gi | 95.1 cores (345.43Gi) | $11.268 | | `g4-standard-192` | us-central1 | 4x NVIDIA-RTX-PRO-6000 (96Gi) | 384Gi | 190.9 cores (698.23Gi) | $22.5 | | `g4-standard-384` | us-central1 | 8x NVIDIA-RTX-PRO-6000 (96Gi) | 768Gi | 382.4 cores (1,403.83Gi) | $45 | **Key Features** - NVIDIA RTX PRO 6000 GPU — 96 GiB HBM per card, ideal for workloads requiring more GPU memory than H100/A100 instances provide. - Multi-GPU scaling — From 1 to 8 GPUs (up to 768 GiB combined GPU memory), supporting distributed inference and training at scale. - High CPU & RAM — Up to 382 cores and ~1.4 TiB RAM in the 8-GPU variant. **Example Use Cases** - Inference for very large language models (70B–400B+ parameters) where maximizing GPU memory per card reduces the need for model parallelism. - Multi-GPU distributed training where per-GPU memory is the primary bottleneck. ### A2 & A3 High-Performance Instances The A2 and A3 series are GCP's flagship high-performance GPU instances, designed for large-scale deep learning, high-performance inference, and real-time AI workloads. With NVIDIA A100 and H100 GPUs, they scale from single-GPU setups to multi-GPU powerhouse configurations capable of training foundation models. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |-------------------|-------------|---------------------------|------------|----------------------------|------------| | `a2-ultragpu-1g` | us-central1 | 1x NVIDIA-A100 (80Gi) | 80Gi | 11.3 cores (159.23Gi) | $2.34 | | `a3-highgpu-1g` | us-central1 | 1x NVIDIA-H100 (79.65Gi) | 79.65Gi | 25.3 cores (221.95Gi) | $2.34 | | `a2-ultragpu-1g` | us-east4 | 1x NVIDIA-A100 (80Gi) | 80Gi | 11.3 cores (159.23Gi) | $7.128 | | `a3-highgpu-1g` | us-east4 | 1x NVIDIA-H100 (79.65Gi) | 79.65Gi | 25.3 cores (221.95Gi) | $13.824 | | `a3-highgpu-8g` | us-east4 | 8x NVIDIA-H100 (79.65Gi) | 637.18Gi | 206.8 cores (1,827.19Gi) | $110.628 | **Key Features** - Next-generation GPUs — A100 (80 GiB) excels at large-scale training with strong throughput and memory bandwidth. H100 (80 GiB) delivers significant improvements for transformer-based models, enabling faster training and inference. Multi-GPU (`a3-highgpu-8g`) configurations scale this power dramatically. - Massive CPU & RAM scaling — From 11.3 cores / 159 GiB RAM in `a2-ultragpu-1g` to 206.8 cores / 1.8 TiB RAM in `a3-highgpu-8g`, ensuring parallel data pipelines can keep pace with GPU compute. - Flexible tiers — Options for single-GPU tasks or multi-GPU clusters, matching workloads of different scales and budgets. **Example Use Cases** - Single-GPU (A2 / A3-1g) can be used for training or fine-tuning mid-to-large language models (e.g., GPT-3, T5-XL) or advanced vision models. - Multi-GPU (A3-8g) can be used for training large-scale, next-generation foundation models (e.g., GPT-4, PaLM, multi-modal transformers) where scale and GPU memory aggregation are critical. - Deploying video analytics, autonomous systems, or robotics pipelines that demand real-time, ultra-low latency. ### G2-Standard Instances The GCP G2-Standard series provides GPU acceleration with NVIDIA L4 GPUs, designed for moderate machine learning and inference workloads. These instances scale from small setups to larger configurations, balancing cost with performance for small-to-medium tasks. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |-------------------|-------------|------------------------|------------|------------------------|----------| | `g2-standard-4` | us-central1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 3.4 cores (12.63Gi) | $2.34 | | `g2-standard-8` | us-central1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 7.3 cores (27.67Gi) | $2.34 | | `g2-standard-12` | us-central1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 11.3 cores (42.71Gi) | $2.34 | | `g2-standard-16` | us-central1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 15.3 cores (57.75Gi) | $2.34 | | `g2-standard-32` | us-central1 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 31.3 cores (118.07Gi) | $2.34 | | `g2-standard-4` | us-east4 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 3.4 cores (12.63Gi) | $0.9 | | `g2-standard-8` | us-east4 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 7.3 cores (27.67Gi) | $1.08 | | `g2-standard-12` | us-east4 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 11.3 cores (42.71Gi) | $1.26 | | `g2-standard-16` | us-east4 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 15.3 cores (57.75Gi) | $1.44 | | `g2-standard-32` | us-east4 | 1x NVIDIA-L4 (22.49Gi) | 22.49Gi | 31.3 cores (118.07Gi) | $2.16 | **Key Features** - NVIDIA L4 GPUs — Optimized for inference and light training, delivering strong efficiency for vision and NLP tasks at lower cost compared to heavier GPU families. - CPU & memory scaling — From ~3.4 to ~31.3 cores and 12.63 GiB to 118.07 GiB RAM, allowing smooth orchestration of preprocessing, data loading, and GPU-bound tasks. - Cost-performance balance — A versatile option for teams that need GPU acceleration without the expense of A100/H100-based instances. **Example Use Cases** - Running transformer-based models like BERT-base for summarization, classification, or Q&A. - Fine-tuning smaller computer vision models for object detection or image classification. ### TPU v6e & v7 High-Performance Instances Google's TPU v6e and v7 instances represent the latest generations of purpose-built AI accelerators, offering significant performance improvements over v5 for large-scale training and inference. | Instance Type | Region | Accelerators | Accelerator Memory | CPU | Price/hr | |----------------------|-------------|--------------------|--------------------|-------------------------|----------| | `ct6e-standard-1t` | us-central1 | 1x GOOGLE-TPU-v6e | – | 43.3 cores (165.11Gi) | $2.34 | | `tpu7x-standard-1t` | us-central1 | 1x GOOGLE-TPU-v7 | – | 55.2 cores (227.83Gi) | $2.34 | | `ct6e-standard-4t` | us-central1 | 4x GOOGLE-TPU-v6e | – | 178.9 cores (698.23Gi) | $2.34 | | `tpu7x-standard-4t` | us-central1 | 4x GOOGLE-TPU-v7 | – | 222.8 cores (933.43Gi) | $2.34 | **Key Features** - TPU v6e (Trillium) — Google's sixth-generation TPU, delivering ~4x the compute performance per chip compared to v5e. Optimized for both training and serving large transformer models. - TPU v7 — Google's latest-generation TPU, designed for frontier-scale training workloads with improved interconnect bandwidth and compute throughput over v6e. - No exposed accelerator memory — TPU memory is managed by the TPU runtime rather than exposed as a configurable VRAM value. - High CPU & RAM — From 43 cores / 165 GiB RAM (v6e 1-chip) to 222 cores / 933 GiB RAM (v7 4-chip), supporting large data pipelines alongside TPU compute. **Example Use Cases** - TPU v6e can be used for cost-efficient, high-throughput serving of large language models (e.g., Gemma, LLaMA) and training of mid-to-large scale transformer architectures. - TPU v7 can be used for training frontier models at scale where maximum TPU throughput and interconnect performance are required. ### TPU v5e & v5p High-Performance Instances Google's cloud TPU v5e and v5p instances are purpose-built accelerators optimized for deep learning training and inference. Unlike GPUs, TPUs (Tensor Processing Units) are specialized for matrix-heavy tensor operations, making them ideal for transformer-based models and large-scale distributed training. | Instance Type | Region | Accelerators | Accelerator Memory | CPU | Price/hr | |----------------------|-------------|--------------------|--------------------|-------------------------|----------| | `ct5lp-hightpu-1t` | us-central1 | 1x GOOGLE-TPU-v5e | – | 23.3 cores (42.71Gi) | $2.34 | | `ct5lp-hightpu-4t` | us-central1 | 4x GOOGLE-TPU-v5e | – | 111.1 cores (180.79Gi) | $2.34 | | `ct5p-hightpu-4t` | us-central1 | 4x GOOGLE-TPU-v5p | – | 206.8 cores (431.67Gi) | $2.34 | **Key Features** - Specialized Tensor Processing Units (TPUs) — TPU v5e provides a balanced design optimized for cost-efficiency in large-scale training and inference, which is great for productionizing ML workloads where throughput matters. TPU v5p provides higher-performance generation with faster interconnects and larger scaling potential, designed for frontier model training. - Scalable CPU & memory — From 23.3 cores / 42.71 GiB RAM in the 1-core v5e instance to 206.8 cores / 431.67 GiB RAM in the 4-core v5p, ensuring sufficient orchestration power for massive training workloads. - No exposed GPU memory — TPU memory is not presented like GPU VRAM but is instead managed by the TPU runtime for high-efficiency tensor operations. **Example Use Cases** - TPU v5e (1t, 4t) can be used for cost-efficient training of language models (e.g., BERT, T5-small/XL) and vision transformers (ViT). - TPU v5p (4t) can be used for training large foundation models such as PaLM, Gemini-like multi-modal architectures, or massive LLMs where performance and throughput at scale are critical. ### N2-Standard Instances The GCP N2-Standard series offers cost-effective, general-purpose compute for workloads that don't require GPU acceleration. These instances balance CPU and memory, making them well-suited for lightweight applications, preprocessing, and small-scale model deployment. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |-------------------|-------------|-----|------------|-----------------------|----------| | `n2-standard-2` | us-central1 | – | – | 1.4 cores (5.42Gi) | $2.34 | | `n2-standard-4` | us-central1 | – | – | 3.4 cores (12.63Gi) | $2.34 | | `n2-standard-8` | us-central1 | – | – | 7.3 cores (27.67Gi) | $2.34 | | `n2-standard-16` | us-central1 | – | – | 15.3 cores (57.75Gi) | $2.34 | | `n2-standard-2` | us-east4 | – | – | 1.4 cores (5.42Gi) | $0.144 | | `n2-standard-4` | us-east4 | – | – | 3.4 cores (12.63Gi) | $0.288 | | `n2-standard-8` | us-east4 | – | – | 7.3 cores (27.67Gi) | $0.54 | | `n2-standard-16` | us-east4 | – | – | 15.3 cores (57.75Gi) | $1.08 | **Key Features** - vCPUs — Baseline performance scales from ~1.4 to ~15.3 cores, with the ability to burst to the full allocation (2 to 16 vCPUs). Optimized for CPU-intensive tasks, such as running traditional models. - Memory (RAM) — From 5.42 GiB to 57.75 GiB, supporting in-memory data handling for lightweight to moderately intensive workloads. - Cost efficiency — Designed to deliver consistent performance at a lower cost, ideal for everyday compute tasks without GPU requirements. **Example Use Case** - Running small-scale machine learning models or serving simple inference workloads. ## Vultr Cloud Servers Instances ### GH200, MI300X & MI355X High-Performance GPU Instances Vultr offers high-performance GPU instances powered by NVIDIA and AMD accelerators. These instances are built for AI training, inference, and HPC (high-performance computing) workloads that demand extreme compute and memory bandwidth. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |---------------------------------|----------|----------------------------|-------------|----------------------------|------------| | `vbm-72c-480gb-gh200-gpu` | atlanta | 1x NVIDIA-GH200 (95.58Gi) | 95.58Gi | 71.4 cores (455.74Gi) | Contact Us | | `vbm-256c-2048gb-8-mi300x-gpu` | chicago | 8x AMD-MI300X (127.82Gi) | 1,022.53Gi | 255.4 cores (1,945.34Gi) | Contact Us | | `vbm-256c-3072gb-8-mi355x-gpu` | chicago | 8x AMD-MI355X (288Gi) | 2,304Gi | 255.4 cores (2,918.14Gi) | Contact Us | **Key Features** - NVIDIA GH200 superchip — Combines Hopper GPU architecture with Grace CPU integration, delivering ultra-fast memory bandwidth and low-latency compute, ideal for training massive AI models and real-time inference. - AMD MI300X GPUs — Designed for frontier AI workloads, offering huge HBM3 memory (128 GiB per GPU) and scaling efficiency with 8 GPUs per instance. Excellent for distributed training of very large models. - AMD MI355X GPUs — Next-generation AMD accelerator with 288 GiB HBM3 per card, pushing single-node GPU memory beyond MI300X for frontier-scale training and inference. - High CPU & RAM configurations — From 71 cores / 455 GiB RAM (GH200) up to 255 cores / ~2.85 TiB RAM (MI355X cluster), ensuring orchestration and preprocessing don't bottleneck GPU performance. **Example Use Cases** - NVIDIA GH200 (single GPU instance) can be used for training and inference for large language models (e.g., LLaMA 2–70B, GPT-3 scale). It can also be used for real-time multi-modal use cases requiring tight GPU-CPU integration, such as speech-to-speech AI assistants or interactive robotics. - AMD MI300X (8-GPU cluster instance) can be used for training frontier LLMs and multi-modal models (GPT-4, Gemini-class, or open LLMs at >100B parameters). - AMD MI355X (8-GPU cluster instance) can be used for frontier-scale workloads requiring maximum single-node GPU memory beyond what MI300X provides. ### VCG Instances The Vultr VCG series provides instances with dedicated NVIDIA GPUs, enabling acceleration for deep learning, inference, and GPU-intensive applications. These instances scale from entry-level GPU setups to multi-GPU clusters, making them versatile for workloads ranging from experimentation to frontier AI model training. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |--------------------------------|----------|----------------------------|-------------|----------------------------|------------| | `vcg-a16-6c-64g-16vram` | new-york | 1x NVIDIA-A16 (16Gi) | 16Gi | 5.4 cores (60.54Gi) | $0.576 | | `vcg-l40s-16c-180g-48vram` | new-york | 1x NVIDIA-L40S (44.99Gi) | 44.99Gi | 15.4 cores (170.74Gi) | $2.088 | | `vcg-a100-12c-120g-80vram` | new-york | 1x NVIDIA-A100 (80Gi) | 80Gi | 11.4 cores (113.74Gi) | $2.988 | | `vcg-b200-248c-2826g-1536vram` | atlanta | 8x NVIDIA-B200 (179.06Gi) | 1,432.49Gi | 247.4 cores (2,684.44Gi) | Contact Us | | `vcg-b200-248c-2826g-1536vram` | seattle | 8x NVIDIA-B200 (179.06Gi) | 1,432.49Gi | 247.4 cores (2,684.44Gi) | Contact Us | **Key Features** - Range of NVIDIA GPUs — From the A16 (lightweight inference) to the A100 (high-performance training), L40S (next-gen accelerated workloads), and B200 clusters (frontier-scale AI with 8 GPUs). - High vCPU and RAM configurations — Scales from 5.4 cores / 60 GiB RAM in entry-level instances to 255 cores / 1.9 TiB RAM in multi-GPU setups, ensuring GPU workloads are matched with sufficient CPU and memory. - Scalable GPU memory — Ranges from 16 GiB (A16) for smaller tasks up to 1.4 TiB (8 × B200) for extreme AI training. **Example Use Cases** - High-performance training and inference for large-scale deep learning models. - Running AI inference workloads with optimized GPU acceleration. ### VC2 Instances The Vultr VC2 series provides general-purpose compute instances optimized for workloads that do not require GPU acceleration. With a balance of CPU and memory, these instances are best suited for lightweight use cases. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | |-----------------|----------|-----|------------|-----------------------|----------| | `vc2-2c-4gb` | new-york | – | – | 1.4 cores (3.54Gi) | $0.036 | | `vc2-4c-8gb` | new-york | – | – | 3.4 cores (7.34Gi) | $0.072 | | `vc2-6c-16gb` | new-york | – | – | 5.4 cores (14.94Gi) | $0.144 | | `vc2-8c-32gb` | new-york | – | – | 7.4 cores (30.14Gi) | $0.288 | | `vc2-16c-64gb` | new-york | – | – | 15.4 cores (60.54Gi) | $0.54 | | `vc2-24c-96gb` | new-york | – | – | 23.4 cores (90.94Gi) | $1.08 | **Key Features** - Scalable CPU and RAM — Configurations range from 1.4 cores / 3.54 GiB RAM (`vc2-2c-4gb`) up to 23.4 cores / 90.94 GiB RAM (`vc2-24c-96gb`). - Cost-effective — Optimized for environments where GPU acceleration is unnecessary, making them a good fit for traditional compute workloads. - Flexibility — Suitable for a broad range of general-purpose tasks, with instance sizes that scale from small testing environments to larger services. **Example Use Cases** - Suitable for lightweight applications as well as development and testing environments. ## Oracle Distributed Cloud Instances ### AMD-MI300X GPU Instances The Oracle MI300X family provides cutting-edge AMD Instinct MI300X GPUs, purpose-built for large-scale AI training and HPC (high-performance computing) workloads. These instances feature massive GPU memory and CPU scaling, supporting frontier use cases. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | ----------------- | ------------ | -------------------------- | ----------- | ------------------------- | ---------- | | `BM.GPU.MI300X.8` | us-chicago-1 | 8 × AMD-MI300X (127.82Gi) | 1,022.53Gi | 111.5 cores (1,945.01Gi) | Contact Us | **Key Features** - 8 × AMD MI300X GPUs, each with 128 GiB HBM3 memory (over 1 TiB GPU memory total). - Extremely high CPU scaling, 111.5 cores, and nearly 2 TiB of system RAM. - Suited for frontier-scale AI training and supercomputing-class workloads. **Example Use Cases** - Training LLMs and multi-modal models (>100B parameters). - High-performance computing simulations, such as weather forecasting and scientific modeling. ### NVIDIA-A10G GPU Instances The Oracle A10G series provides NVIDIA GPUs optimized for inference, visualization, and moderate ML training tasks. These instances combine consistent vCPU allocations with scalable GPU configurations, from single-GPU VMs to multi-GPU bare metal nodes. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | -------------- | ------------ | -------------------------- | ---------- | ---------------------- | ---------- | | `VM.GPU.A10.1` | us-chicago-1 | 1 × NVIDIA-A10G (22.49Gi) | 22.49Gi | 14.8 cores (227.41Gi) | Contact Us | | `BM.GPU.A10.4` | us-chicago-1 | 4 × NVIDIA-A10G (22.49Gi) | 89.95Gi | 14.8 cores (227.41Gi) | Contact Us | **Key Features** - NVIDIA A10G GPUs with 22.49 GiB memory each. - Same CPU allocation (14.8 cores / 227 GiB RAM) across single- and four-GPU configurations. **Example Use Cases** - VM.A10.1 (single GPU) can be used for small-scale ML inference, 3D rendering, and graphics-heavy applications. - BM.A10.4 (four GPUs) can be used for larger inference workloads, distributed graphics rendering, or multi-GPU training of medium-sized models. ### Standard CPU-Only Instances The Oracle E6 Flex series provides CPU-only instances for workloads that do not require GPU acceleration. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | --------------------- | ------------ | --- | ---------- | -------------------- | ---------- | | `VM.Standard.E6.Flex` | us-chicago-1 | – | – | 0.8 cores (14.61Gi) | Contact Us | **Key Features** - No GPU, pure vCPU, and memory resources. - 0.8 cores and 14.61 GiB RAM, suitable for supporting tasks. - Cost-efficient for non-GPU workloads. **Example Use Cases** - Running control-plane services or lightweight applications alongside GPU clusters. - Workloads requiring basic compute and memory without acceleration. ## Microsoft Azure Instances ### High-Performance GPU Instances These Azure instances provide NVIDIA H100, A100, and AMD MI300X GPUs for large-scale deep learning training and inference workloads. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | ---------------------------- | ------ | ---------------------------- | ----------- | ---------------------------- | ---------- | | `Standard_NC40ads_H100_v5` | eastus | 1x NVIDIA-H100 (79.65Gi) | 79.65Gi | 39.7 cores (215.63Gi) | $2.34 | | `Standard_NC24ads_A100_v4` | eastus | 1x NVIDIA-A100 (80Gi) | 80Gi | 23.8 cores (148.13Gi) | $2.34 | | `Standard_ND96isr_MI300X_v5` | eastus | 1x AMD-MI300X (127.82Gi) | 127.82Gi | 95.6 cores (1,248.38Gi) | Contact Us | **Key Features** - NVIDIA H100 and A100 GPUs — Industry-leading GPU performance for transformer-based model training, large-scale inference, and fine-tuning of frontier LLMs. - AMD MI300X GPU — 128 GiB HBM3 memory per card with massive CPU and RAM backing (~1.2 TiB system RAM), suited for very large model inference and distributed training. - Scalable memory and compute — From 23.8 cores / 148 GiB RAM (A100) up to 95.6 cores / 1,248 GiB RAM (MI300X), supporting memory-intensive training pipelines. **Example Use Cases** - Fine-tuning or running inference on large language models (e.g., LLaMA-70B, GPT-3-scale) with A100 or H100 instances. - Training frontier LLMs (>100B parameters) or multi-modal models requiring extreme GPU memory with MI300X instances. ### Mid-Range GPU Instances These instances provide NVIDIA A10G, T4, and AMD V620 GPUs for moderate ML inference and training workloads. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | -------------------------- | ------ | ---------------------------- | ----------- | ---------------------------- | -------- | | `Standard_NV36ads_A10_v5` | eastus | 1x NVIDIA-A10G (22.49Gi) | 22.49Gi | 35.7 cores (296.63Gi) | $2.34 | | `Standard_NG32ads_V620_v1` | eastus | 1x AMD-V620 (32Gi) | 32Gi | 31.7 cores (42.82Gi) | $2.34 | | `Standard_NC4as_T4_v3` | eastus | 1x NVIDIA-T4 (15Gi) | 15Gi | 3.8 cores (18.52Gi) | $2.34 | **Key Features** - NVIDIA A10G GPU — High compute throughput for inference and moderate training, with a large CPU and RAM allocation (~296 GiB) suited for data-heavy pipelines. - AMD V620 GPU — 32 GiB GPU memory at a competitive price point, offering a balance between memory capacity and cost for inference tasks. - NVIDIA T4 GPU — Cost-efficient inference-focused GPU, ideal for deploying production NLP and vision models at smaller scale. **Example Use Cases** - Running transformer-based NLP models (e.g., BERT, T5) for classification, summarization, or Q&A with T4 or A10G instances. - Moderate fine-tuning of mid-sized models or computer vision tasks requiring more than 16 GiB GPU memory with the AMD V620. ### Standard CPU-Only Instances These Azure instances provide general-purpose compute for workloads that do not require GPU acceleration. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | ------------------- | ------ | --- | ---------- | --------------------- | -------- | | `Standard_A4` | eastus | – | – | 7.8 cores (9.07Gi) | $2.34 | | `Standard_A3` | eastus | – | – | 3.8 cores (4.35Gi) | $2.34 | | `Standard_A4m_v2` | eastus | – | – | 3.8 cores (21.22Gi) | $2.34 | | `Standard_L2aos_v4` | eastus | – | – | 1.8 cores (10.42Gi) | $2.34 | **Key Features** - Flexible CPU and memory — From 1.8 cores / 10.42 GiB RAM (`Standard_L2aos_v4`) up to 7.8 cores / 9.07 GiB RAM (`Standard_A4`), covering a range of lightweight workloads. - Memory-optimized option — `Standard_A4m_v2` provides 21.22 GiB RAM at 3.8 cores, offering higher memory per core for in-memory data processing. - Cost-effective — Suitable for preprocessing, control-plane services, lightweight inference, and development environments. **Example Use Cases** - Running simple classification or regression models that do not require GPU acceleration. - Preprocessing pipelines, orchestration services, or lightweight applications alongside GPU-accelerated nodepools. ## DigitalOcean Instances ### NVIDIA H100 GPU Instances DigitalOcean's H100 instances provide NVIDIA H100 GPUs for high-performance training and inference workloads, available in the `nyc1`, `nyc2`, and `sfo3` regions. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | -------------------- | ------ | ------------------------- | ---------- | ------------------------ | ---------- | | `gpu-h100x1-80gb` | nyc1 | 1x NVIDIA-H100 (80Gi) | 80Gi | 19.8 cores (238.24Gi) | Contact Us | | `gpu-h100x2-160gb` | nyc1 | 2x NVIDIA-H100 (160Gi) | 320Gi | 39.7 cores (483.24Gi) | Contact Us | | `gpu-h100x4-320gb` | nyc1 | 4x NVIDIA-H100 (320Gi) | 1,280Gi | 79.6 cores (973.24Gi) | Contact Us | | `gpu-h100x1-80gb` | nyc2 | 1x NVIDIA-H100 (80Gi) | 80Gi | 19.8 cores (238.24Gi) | Contact Us | | `gpu-h100x2-160gb` | nyc2 | 2x NVIDIA-H100 (160Gi) | 320Gi | 39.7 cores (483.24Gi) | Contact Us | | `gpu-h100x4-320gb` | nyc2 | 4x NVIDIA-H100 (320Gi) | 1,280Gi | 79.6 cores (973.24Gi) | Contact Us | | `gpu-h100x1-80gb` | sfo3 | 1x NVIDIA-H100 (80Gi) | 80Gi | 19.8 cores (238.24Gi) | Contact Us | | `gpu-h100x2-160gb` | sfo3 | 2x NVIDIA-H100 (160Gi) | 320Gi | 39.7 cores (483.24Gi) | Contact Us | | `gpu-h100x4-320gb` | sfo3 | 4x NVIDIA-H100 (320Gi) | 1,280Gi | 79.6 cores (973.24Gi) | Contact Us | **Key Features** - NVIDIA H100 GPUs — Leading GPU performance for transformer-based model training, large-scale inference, and fine-tuning of large language models. - Scalable multi-GPU configurations — From a single H100 (80 GiB) up to 4 GPUs (1,280 GiB combined GPU memory), with CPU and RAM scaling in step. - High memory backing — Up to 79.6 cores and ~973 GiB RAM in the 4-GPU configuration, ensuring data pipelines keep pace with GPU throughput. **Example Use Cases** - Fine-tuning or serving large language models (e.g., LLaMA-70B, Mistral) on a single or dual H100. - Multi-GPU distributed training of foundation models where high per-GPU memory is required. ### AMD MI300X & MI350X GPU Instances DigitalOcean's AMD Instinct instances provide extreme GPU memory capacity for frontier-scale AI training and inference, available in `atl1`, `nyc1`, `nyc2`, and `sfo3` regions. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | ----------------------- | ------ | -------------------------- | ---------- | ---------------------------- | ---------- | | `gpu-mi300x1-192gb` | atl1 | 1x AMD-MI300X (192Gi) | 192Gi | 19.8 cores (228.44Gi) | Contact Us | | `gpu-mi300x8-1536gb` | atl1 | 8x AMD-MI300X (192Gi) | 1,536Gi | 159.4 cores (1,874.84Gi) | Contact Us | | `gpu-mi350x1-288gb-contracted` | atl1 | 1x AMD-MI350X (288Gi) | 288Gi | 23.8 cores (244.12Gi) | Contact Us | | `gpu-mi350x8-2304gb-contracted` | atl1 | 8x AMD-MI350X (288Gi) | 2,304Gi | 191.3 cores (2,000.28Gi) | Contact Us | | `gpu-mi300x1-192gb` | nyc1 | 1x AMD-MI300X (192Gi) | 192Gi | 19.8 cores (228.44Gi) | Contact Us | | `gpu-mi300x8-1536gb` | nyc1 | 8x AMD-MI300X (192Gi) | 1,536Gi | 159.4 cores (1,874.84Gi) | Contact Us | | `gpu-mi350x1-288gb-contracted` | nyc1 | 1x AMD-MI350X (288Gi) | 288Gi | 23.8 cores (244.12Gi) | Contact Us | | `gpu-mi350x8-2304gb-contracted` | nyc1 | 8x AMD-MI350X (288Gi) | 2,304Gi | 191.3 cores (2,000.28Gi) | Contact Us | | `gpu-mi300x1-192gb` | nyc2 | 1x AMD-MI300X (192Gi) | 192Gi | 19.8 cores (228.44Gi) | Contact Us | | `gpu-mi300x8-1536gb` | nyc2 | 8x AMD-MI300X (192Gi) | 1,536Gi | 159.4 cores (1,874.84Gi) | Contact Us | | `gpu-mi350x1-288gb-contracted` | nyc2 | 1x AMD-MI350X (288Gi) | 288Gi | 23.8 cores (244.12Gi) | Contact Us | | `gpu-mi350x8-2304gb-contracted` | nyc2 | 8x AMD-MI350X (288Gi) | 2,304Gi | 191.3 cores (2,000.28Gi) | Contact Us | | `gpu-mi300x1-192gb` | sfo3 | 1x AMD-MI300X (192Gi) | 192Gi | 19.8 cores (228.44Gi) | Contact Us | | `gpu-mi300x8-1536gb` | sfo3 | 8x AMD-MI300X (192Gi) | 1,536Gi | 159.4 cores (1,874.84Gi) | Contact Us | | `gpu-mi350x1-288gb-contracted` | sfo3 | 1x AMD-MI350X (288Gi) | 288Gi | 23.8 cores (244.12Gi) | Contact Us | | `gpu-mi350x8-2304gb-contracted` | sfo3 | 8x AMD-MI350X (288Gi) | 2,304Gi | 191.3 cores (2,000.28Gi) | Contact Us | **Key Features** - AMD MI300X GPUs — 192 GiB HBM3 memory per card, delivering industry-leading GPU memory capacity for very large model inference and distributed training without model parallelism. - AMD MI350X GPUs — Next-generation AMD accelerator with 288 GiB per card, pushing single-GPU memory capacity beyond MI300X for frontier-scale workloads. - Multi-GPU scaling — From a single GPU up to 8-GPU clusters (up to 2,304 GiB combined GPU memory with MI350X), supported by up to 191 cores and ~2 TiB system RAM. - Multi-region availability — Both MI300X and MI350X instances are available in `atl1`, `nyc1`, `nyc2`, and `sfo3`, providing geographic redundancy. **Example Use Cases** - Running inference on very large language models (70B–400B+ parameters) where maximizing GPU memory per card reduces or eliminates model parallelism overhead. - Distributed training of frontier LLMs and multi-modal models requiring over 1 TiB of combined GPU memory. ### Standard CPU-Only Instances These DigitalOcean instances provide general-purpose compute for workloads that do not require GPU acceleration. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | ---------------- | ------ | --- | ---------- | -------------------- | ---------- | | `s-2vcpu-2gb` | nyc1 | – | – | 1.8 cores (1.3Gi) | Contact Us | | `s-4vcpu-8gb` | nyc1 | – | – | 3.8 cores (6.02Gi) | Contact Us | | `s-8vcpu-16gb` | nyc1 | – | – | 7.8 cores (13.24Gi) | Contact Us | | `s-2vcpu-2gb` | nyc2 | – | – | 1.8 cores (1.3Gi) | Contact Us | | `s-4vcpu-8gb` | nyc2 | – | – | 3.8 cores (6.02Gi) | Contact Us | | `s-8vcpu-16gb` | nyc2 | – | – | 7.8 cores (13.24Gi) | Contact Us | | `s-2vcpu-2gb` | sfo3 | – | – | 1.8 cores (1.3Gi) | Contact Us | | `s-4vcpu-8gb` | sfo3 | – | – | 3.8 cores (6.02Gi) | Contact Us | | `s-8vcpu-16gb` | sfo3 | – | – | 7.8 cores (13.24Gi) | Contact Us | **Key Features** - Scalable CPU and memory — From 1.8 cores / 1.3 GiB RAM (`s-2vcpu-2gb`) up to 7.8 cores / 13.24 GiB RAM (`s-8vcpu-16gb`). - Cost-effective — Suitable for lightweight workloads, preprocessing pipelines, and supporting services that do not require GPU acceleration. **Example Use Cases** - Running simple models, control-plane services, or development and testing environments alongside GPU-accelerated nodepools. ## Lambda Instances ### NVIDIA B200 GPU Instances Lambda's B200 instances provide NVIDIA B200 GPUs for frontier-scale AI training and inference workloads, available in the `us-south-3` region. | Instance Type | Region | GPU | GPU Memory | CPU | Price/hr | | ------------------- | ----------- | -------------------------- | ----------- | ---------------------------- | ---------- | | `cl-xl-gpu-8-b200` | us-south-3 | 8x NVIDIA-B200 (179.06Gi) | 1,432.49Gi | 207.3 cores (2,790.05Gi) | Contact Us | **Key Features** - NVIDIA B200 GPUs — Latest-generation Blackwell architecture, delivering significant performance improvements over H100 for large-scale transformer training and inference. - Massive GPU memory — 1,432 GiB of combined GPU memory across 8 GPUs, supporting frontier-scale models without requiring model parallelism across nodes. - Extreme CPU & RAM — 207 cores and ~2.8 TiB system RAM, ensuring data pipelines and preprocessing can fully saturate GPU throughput. **Example Use Cases** - Training or fine-tuning frontier LLMs (100B+ parameters) where maximum GPU memory and Blackwell compute throughput are required. - High-throughput inference for very large models, leveraging the combined 1.4 TiB GPU memory to serve multiple large models simultaneously. --- ## Deployments # Deployments > _**Note:** Although the video above showcases an older version of our UI, it still accurately illustrates how to deploy models on our platform._ Deployments bring your models to life by running them on dedicated compute, ready to serve predictions at scale with configurable replicas, autoscaling, and cost controls. A deployment defines how a model runs on a selected nodepool, acting as the bridge between your model and the underlying infrastructure. For advanced use cases, a single model can be deployed across [multiple nodepools](https://docs.clarifai.com/compute/deployments/deploy-model#step-4-set-dynamic-routing-optional) to optimize for different workloads, availability, or performance requirements. With deployments you can: - Serve models with low-latency, production-grade inference - Auto-scale replicas based on traffic demand - Monitor health, usage, and cost in real time - Run on dedicated infrastructure with full resource isolation import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Create Clusters and Nodepools # Create Clusters and Nodepools **Set up capabilities that match your computational needs** A compute cluster serves as the main environment where models are deployed, whether for training or inference. Each cluster can contain multiple nodepools, which are groups of virtual machine instances with similar configurations (such as CPU/GPU type, memory). After creating a custom cluster, you can configure nodepools within it to optimize resource usage. These nodepools will help tailor the infrastructure to meet the specific hardware, performance, cost, or regulatory compliance of your machine learning needs. For example, you may create a nodepool for GPU-intensive tasks and another for lighter workloads running on CPUs. With clusters and nodepools, you can organize and manage (_orchestrate_) the compute resources necessary for running your models and workflows. :::info Connect Your Own Cloud You can connect your existing AWS, Google Cloud (GCP), or Oracle infrastructure to leverage your current investments. This lets you maintain full control over your data while optimizing for cost and flexibility. [Contact](https://www.clarifai.com/explore/contact-us) our support team for assistance. ::: ## **Via the UI** ### Step 1: Start Creating a Cluster Log in to the Clarifai platform and select the **Compute** option in the collapsible left sidebar. Then, select the **Clusters** option from the dropdown list. You’ll be redirected to the clusters page, where you can view and manage all compute clusters across your environments. Click the **Create a Cluster** button to begin setting up a new cluster along with its associated nodepool. ![ ](/img/compute-orchestration/compute-2.png) You’ll be redirected to a page, where you can specify the configurations for your new cluster and nodepool. ### Step 2: Select an Instance Select an instance type that aligns with your specific requirements. The table displays essential details for each instance type, which helps you make an informed decision. :::tip See [Supported Cloud Instances](https://docs.clarifai.com/compute/cloud-instances) to learn more about the instance types we provide. ::: ![ ](/img/compute-orchestration/compute-3-1.png) You can narrow down the displayed options using the filters and tools provided in the top bar: - **Search bar** — Quickly find instance types by name. - **Provider filter** — Choose from available cloud providers to match your preferred infrastructure. - **Region filter** — Select the geographic location for the instance. Choosing a region closer to your users can reduce latency and improve performance. - **Instance type filter** — Directly filter for a specific instance type if you already know what you're looking for. - **Hardware filter** — Filter based on the instance's available hardware, such as CPU or GPU. - **Price/hour filter** — Filter instances by their hourly cost, helping you manage your budget. - **Sorting controls** — Click the sorting arrows next to each column header to sort values in ascending or descending order, making comparisons easier. :::note Before making your final choice, here are some key considerations to make: - **Workload requirements (CPU and memory)** — For applications demanding significant processing power, opt for instances with a higher number of CPU cores. For example, a `t3a.2xlarge` offers `7.6 cores`, considerably more than a `t3a.medium` with `1.6 cores`. Also, the `Gi` value displayed alongside the core count (such as `2.99Gi`) indicates the amount of RAM in Gigabytes. A higher `Gi` value signifies more available memory. - **Cost sensitivity** — The `PRICE/HR` column helps you evaluate cost. Choose an instance that balances price and performance according to your budget. - **GPU requirements** — If your workload involves intensive tasks like running models for video processing or other GPU-heavy tasks, you'll need to select instances equipped with GPUs. - **Regulatory compliance** – Ensure the chosen region and instance type comply with any relevant data residency or industry-specific regulations. ::: Once you've found the right instance, click the circular radio button to the left of the row to select it. :::tip > For this example, let's select the AWS `g4dn.xlarge` instance in the `us-east-1` region. For the remaining details to be filled, we'll use the already provided default options. After selecting an instance, you can click the [**Create Cluster**](#step-6-finalize-and-create-the-cluster) button to complete creating your cluster and nodepool. Otherwise, you can customize the rest of the settings to meet your needs. ::: ### Step 3: Set Node Autoscaling Range ![ ](/img/compute-orchestration/compute-7.png) Define the minimum and maximum number of nodes your nodepool can scale to based on workload demand. This ensures your system automatically adjusts its capacity by adding nodes during high traffic and scaling down during low usage, which balances performance and cost. For example, setting the autoscaling range to **1–5** nodes allows the nodepool to scale up to handle more requests and scale down when demand drops. - Setting a minimum of **1** ensures that at least one node is always running. This helps avoid cold start delays after periods of inactivity, which is crucial for maintaining low-latency response times. However, it also means incurring continuous compute costs. - Setting a minimum of **0** reduces costs during idle periods, as no nodes will be running. Keep in mind this can introduce cold starts when traffic resumes, potentially impacting response times. Choose your range based on the balance you need between cost-efficiency and responsiveness. :::info model replicas [Click here](deploy-model.md#model-replica) to learn how to configure model replicas to distribute your workload efficiently across multiple instances of a model. ::: ### Step 4: Enable Spot Instances You can enable this option (_default is off_) if you want to rent spare, unused compute capacity at significantly lower prices compared to regular on-demand instances. These spot instances are sourced from the underlying cloud provider (such as AWS or GCP) based on the region and instance type you've selected. If spot instances are unavailable, Clarifai will automatically fall back to on-demand instances to maintain service continuity. Keep in mind that spot instances can be terminated at any time if the capacity is reclaimed by the provider, which may cause temporary disruptions. For higher reliability and uninterrupted service, it's recommended to leave this option disabled and use on-demand instances only. ### Step 5: Provide Cluster and Nodepool Details Fill out the form to specify the details for your cluster and nodepool. ![ ](/img/compute-orchestration/compute-8.png) - **Personal Access Token (PAT)** — Use a [PAT](https://docs.clarifai.com/control/authentication/pat) to authenticate your identity when connecting to the cluster. Click the dropdown to choose from existing tokens, or create a new one by selecting **Create a new Personal Access Token** or by visiting the **Security** section in your personal settings. > **Note:** The token must have the required permissions to manage compute resources. Also, if the selected PAT is deleted, any associated compute functionality will stop working. - **Cluster ID** — Enter a unique identifier for your cluster. This ID is used when deploying models and should reflect the cluster’s purpose or workload. It is auto-filled based on your selected instance type, but you can modify it as needed. - **Cluster Description** — Optionally, provide a short description that summarizes the details related to the cluster. - **Nodepool ID** — Enter a unique identifier for your nodepool. This ID is used when deploying models and should reflect the nodepool’s purpose or workload. It is auto-filled based on your selected instance type, but you can modify it as needed. - **Nodepool Description** — Optionally, provide a short description that summarizes the details related to the nodepool. ### Step 6: Finalize and Create the Cluster Before you enter the required details for creating a cluster, the **Create Cluster** button in the upper-right corner will be disabled (greyed out). After providing the details, the button will become active. Click it to launch your cluster and nodepool. You'll then be redirected to your newly created cluster's page, where the details of its associated nodepool will also be displayed. ![ ](/img/compute-orchestration/compute-9.png) > **Note:** _You can also create a new nodepool from an existing cluster by clicking the **Create Nodepool** button in the upper-right corner of the cluster's page. You’ll be redirected to a page where you can specify the configurations for your new nodepool._ If you click on a nodepool displayed on the page, you'll be taken to its individual page, where you can view its detailed information, such as its activity status, extent of utilization, and any resource deployments associated with it. ![ ](/img/compute-orchestration/compute-10.png) ## **Via the API** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CO2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/create_compute_cluster.py"; import CL2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_create_cluster.sh"; import CO5 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/init_compute_cluster.py"; import CO7 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/create_nodepool.py"; import CL3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_create_nodepool.sh"; import CO10 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/init_nodepool.py"; import CO1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/compute_cluster_config.yaml"; import CO6 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/nodepool_config.yaml"; import CO11 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/deployment_config.yaml"; import CORouting from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/model-routing.yaml"; import Curl1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_1.sh"; import Curl2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_2.sh"; import Curl3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_3.sh"; ### Prerequisites #### Installation To begin, install the latest version of the `clarifai` Python package. This will also install the Clarifai [Command Line Interface](https://docs.clarifai.com/resources/api-overview/cli/#clarifai-login) (CLI), which we'll also use to demonstrate how to create clusters and nodepools. ```text pip install --upgrade clarifai ``` #### Get a PAT You need a [PAT (Personal Access Token)](https://docs.clarifai.com/control/authentication/pat) key to authenticate with the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. After generating it, set the token as an environment variable. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE > **Note**: When you set the PAT as an environment variable, you don’t need to hardcode it in your code. This also ensures that your CLI session is automatically authenticated with Clarifai. #### Set up Project Directory - Create an overarching directory to store your project files. - Inside this directory, create a Python file for your Compute Orchestration code. - Create a `configs` folder to store your YAML configuration files for clusters, nodepools, and deployments. Here’s the structure of the directory: ``` project-directory/ # Overarching project folder │ ├── compute_orchestration.py # Python file for your orchestration code │ └── configs/ # Folder for configuration files ├── compute_cluster_config.yaml ├── nodepool_config.yaml └── deployment_config.yaml ``` Then, add the following code snippets to their corresponding files in the `configs` folder. ##### 1. `compute_cluster_config.yaml` {CO1} - `compute_cluster` — Defines the top-level configuration for the compute cluster. - `compute_cluster.id` — A unique identifier for the cluster within the Clarifai workspace (here, named *test-compute-cluster*). - `compute_cluster.description` — A human-readable description of the cluster. - `compute_cluster.cloud_provider.id` — The cloud provider that will host the cluster. Clarifai supports a wide range of providers, such as `aws` and `gcp`. - `compute_cluster.region` — Geographic region for the resources. Must be a region supported by the selected provider. Choosing the right region reduces latency to your data and can affect cost. - `compute_cluster.managed_by` — Who is responsible for lifecycle management. `clarifai` means the platform will automatically handle patching, scaling, and health‑checks, rather than self-managed. - `compute_cluster.cluster_type` — Mode of operation. `dedicated` means the cluster’s compute resources are reserved exclusively for your workloads (not shared). - `compute_cluster.visibility.gettable` — A numeric permission level that defines who can see or access the cluster. `10` means it's visible privately. `50` means it's visible publicly. ##### 2. `nodepool_config.yaml` {CO6} - `nodepool` — Defines the top-level configuration for the nodepool. - `nodepool.id` — A unique identifier for the nodepool within the workspace (here, named *test-nodepool*). - `nodepool.compute_cluster.id` — The ID of the parent compute cluster that this nodepool belongs to. The cluster must already exist. - `nodepool.description` — A human-readable description of the nodepool. - `nodepool.instance_types` — A list that details the types of instances (virtual machines) that will make up this nodepool. - `nodepool.instance_types[0].id` — Specifies the type of instance to use; in this case, ["g5.2xlarge,"](https://docs.clarifai.com/compute/cloud-instances/#g5-instances) which is a type of GPU-optimized instance from AWS. - `nodepool.instance_types[0].compute_info` — A nested object that provides detailed specifications for the instance. - `nodepool.instance_types[0].compute_info.cpu_limit` — Number of virtual CPUs the instance provides. It's stored as a string (follows [Kubernetes notation](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/)), and the value must match the provider’s specification for the chosen type. - `nodepool.instance_types[0].compute_info.cpu_memory` — Amount of system memory (RAM) available to the node. The value can be expressed in gigabytes (Gi) or other supported memory units. - `nodepool.instance_types[0].compute_info.accelerator_type` — A list specifying the type of accelerator. In this case, it's an NVIDIA A10. - `nodepool.instance_types[0].compute_info.num_accelerators` — The number of accelerators (GPUs) per instance. - `nodepool.instance_types[0].compute_info.accelerator_memory` — The memory of the accelerator. - `nodepool.node_capacity_type.capacity_types` — A list of numerical values specifying allowed capacity modes for the nodepool. In this case, `1` correspond to "on-demand" instances (standard pricing) and `2` for "spot" instances (lower cost, but can be reclaimed). - `nodepool.min_instances` — The minimum number of instances that must run in the nodepool at any time. A value of `0` allows the pool to scale down to zero instances when idle. - `nodepool.max_instances` — The maximum number of instances the nodepool can scale to. A value of `0` disables the pool, while higher values (e.g., `10`) allow scaling out as demand increases. ##### 3. `deployment_config.yaml` _We'll use this later to [deploy the model](deploy-model.md#via-the-api)._ {CO11} - `deployment` — Defines the top-level deployment configuration, which specifies how a model runs in a compute environment. - `deployment.id` — A unique identifier for this deployment. (here, named *test-deployment*). - `deployment.description` — A human-readable description of the deployment. - `deployment.autoscale_config` — This section dictates how the deployment will automatically adjust its resources based on demand. - `deployment.autoscale_config.min_replicas` — The minimum number of running replicas the deployment is allowed to have. Setting it to 0 means the service can shut down completely when there is no traffic, saving costs. - `deployment.autoscale_config.max_replicas` — The maximum number of concurrent replicas. Setting it to 1 means the deployment will never have more than one copy of the model running. - `deployment.autoscale_config.traffic_history_seconds` — Length of the sliding window (in seconds) that the autoscaler looks at to decide whether to scale up or down. A longer window smooths out spikes; a shorter window reacts faster. - `deployment.autoscale_config.scale_down_delay_seconds` — The waiting period (in seconds) before scaling down after low traffic. This prevents “thrashing” when traffic briefly dips. - `deployment.autoscale_config.scale_to_zero_delay_seconds` — The waiting period (in seconds) before scaling all the way down to zero replicas after traffic stops. It must be greater than or equal to `scale_down_delay_seconds` to ensure that normal scale-down events happen before the system completely shuts down idle workers. - `deployment.autoscale_config.scale_up_delay_seconds` — The waiting period (in seconds) before scaling up after detecting higher demand. This gives the system a chance to see if the spike is sustained. - `deployment.autoscale_config.disable_packing` — Packing means placing multiple replicas on the same node when possible (to improve bin‑packing and reduce costs). `false` means packing is allowed (the default). Set to `true` if you want each replica on a dedicated node for isolation. - `deployment.worker` — This section specifies the core component of the deployment, which is the machine learning model itself. - `deployment.worker.model` — Specifies which model this deployment serves. - `deployment.worker.model.id` — The model ID. - `deployment.worker.model.model_version.id` — Refers to a specific version of the model. - `deployment.worker.model.user_id` — The model owner’s Clarifai user ID. - `deployment.worker.model.app_id` — The application (or project) under which the model lives. - `deployment.scheduling_choice` — An integer that maps to a pre‑defined scheduling policy. Typical mapping: 1 = Spread (maximise fault‑tolerance), **Note:** A deployment’s `min_replicas` applies only to the first nodepool in the list. A deployment’s `max_replicas` applies to **each** nodepool individually, not as a sum across them. Examples {CORouting} {Curl3} ::: ### Create a Cluster To create a new compute cluster, pass the `compute_cluster_id` and `config_filepath` as arguments to the `create_compute_cluster` method of the `User` class. {CO2} {CL2} {Curl1} Example Output ```text [INFO] 22:41:42.362861 Compute Cluster with ID 'test-compute-cluster' is created: code: SUCCESS description: "Ok" req_id: "sdk-python-11.7.5-1a5ceed7df3346dbb2451f2501f37f66" ``` After creating it, initialize the `ComputeCluster` class by providing the `user_id` and `compute_cluster_id` parameters. > **Note:** Initialization is essential because it establishes the specific user and compute cluster context, which allows the subsequent operations to accurately target and manage the intended resources. {CO5} ### Create a Nodepool To create a new nodepool within an already existing cluster, use the `create_nodepool` method with the `nodepool_id` and `config_filepath` parameters. {CO7} {CL3} {Curl2} Example Output ```text [INFO] 23:09:07.978155 Nodepool with ID 'test-nodepool' is created: code: SUCCESS description: "Ok" req_id: "sdk-python-11.7.5-99ad19030249400cabd9be8ade0df602" ``` After creating it, initialize the `Nodepool` class by providing the `user_id` and `nodepool_id` parameters. {CO10} --- ## Deploy a Model # Deploy a Model **Deploy a model into your created cluster and nodepool** Clarifai’s Compute Orchestration provides efficient capabilities for you to deploy any model on any compute infrastructure, at any scale. You can configure your compute environment and deploy your models into nodepools with your preferred settings, optimizing for both cost and scalability. With model deployment, you can quickly take a trained model and set it up for inference. :::tip Learn how Clarifai routes prediction requests for optimal performance [here](https://docs.clarifai.com/compute/inference/routing). ::: :::note Only one deployment per model version is allowed per nodepool. Attempting to deploy a model version that already has an existing deployment on the selected nodepool will result in an error. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CO12 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/create_deployment.py"; import CO15 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/init_deployment.py"; import CURLRestrictDeployment from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/restrict_deployment.sh"; import OutputCURLRestrictDeployment from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/output_restrict_deployment.txt"; import Curl1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_deployment.sh"; ## **Via the UI** ### Step 1: Start Creating a Deployment To create a deployment, go to the **Compute** option in the collapsible left sidebar. Then, select **Deployments** from the dropdown list. You'll be redirected to the **Deployments** page, where you can view and manage deployments across your Clarifai compute resources. To start creating a deployment, click the **Create Deployment** button. ![ ](/img/compute-orchestration/compute-12.png) :::note Alternatively You can create a deployment directly from other areas within the platform. For example, to deploy from a specific model, go to the individual page of the model you want to deploy and click the **Deploy Model** button. You can also create a deployment from a particular cluster or nodepool by navigating to its page and clicking **Deploy Model** there. > **Note:** On an individual model page, you can go to the **Deployments** section to check if it is already running on any compute environments. > ![ ](/img/compute-orchestration/compute-11.png) ::: ### Step 2: Select a Model You’ll be redirected to a page where you can choose a model to deploy. A selection modal will appear, allowing you to pick a model from the results before continuing with the deployment configuration. You can browse the listed models or use the search bar to quickly find a specific one. For this example, let's select the [gpt-oss-20b](https://clarifai.com/clarifai/agentic-model/models/gpt-oss-20b) model. ![ ](/img/compute-orchestration/compute-13.png) > **Note:** In the upper-right corner of the modal, there’s an account selector that lets you switch between different sources — such as featured models, your personal workspace, any organizations you belong to, or the public Clarifai model library. Selecting a different source updates the list to show the available models from that account or library. > **Note:** You can use the keyboard shortcuts shown at the bottom of the modal to navigate and select models: > - `↑ / ↓ (Navigate)` — Use the up and down arrow keys to move through the model list without using your mouse. > - `↵ (Select)` — Press Enter to select the currently highlighted model. > - `Esc (Close)` — Press Escape to close the dialog without making a selection. After selecting your desired model, it will be listed in the **Deployment Overview** section. If you want to change to another model, click the **Change Model** button. ![ ](/img/compute-orchestration/compute-13-3.png) You can also configure the following settings: * **Model Version** — Determines which version of the model will be used when the deployment is initialized. By default, the latest version is selected, but you can use the provided field to choose a different version if needed. * **Deployment ID** — A unique identifier for your deployment, used in API calls and URLs. You can keep the automatically generated ID or provide a custom one. * **Description** — Optionally add a description to provide additional context about the deployment. * **Deployment PAT** — A [Personal Access Token](https://docs.clarifai.com/control/authentication/pat) (PAT) authorizes access to the resources required for your deployment to run. You can use the automatically selected token or choose a different one. If you don’t already have a suitable PAT, you can generate one using the provided button. > **Caution:** Be cautious when changing the PAT after the deployment has been created, as this may cause the associated compute resources to stop functioning properly. :::note After selecting a model, a compatible nodepool will be automatically assigned. - If no compatible nodepool exists in your account, a new nodepool is automatically created and marked with a `Pending` status while the required infrastructure is being provisioned. - If a compatible nodepool already exists, it is selected automatically and assigned a `Provisioned` status, indicating that the infrastructure is already configured and ready to host the deployment. ::: ### Step 3: Select a Nodepool Next, choose a cloud instance type and hardware configuration that best fits your model’s compute requirements and performance goals. > **Note:** See [Supported Cloud Instances](https://docs.clarifai.com/compute/cloud-instances) for a full list of available GPUs, TPUs, and CPU-only options across supported cloud providers. To select or modify a nodepool, click the pencil icon in the upper-right corner. An **Edit Nodepool Configuration** modal will appear, allowing you to choose an already existing nodepool or select a recommended one that best fits your deployment requirements. The details of the nodepools will be displayed, such as GPU type, available memory, CPU cores, and cost per hour. ![ ](/img/compute-orchestration/compute-13-5.png) The modal allows you to accomplish various tasks, including: - **Search for an instance** — Use the search bar at the top to quickly find a specific nodepool by name. - **Filter the results** — Use the three dropdown filters to narrow down your options by Cloud Provider, GPU Hardware, and Relevance (sort order). - **Switch between tabs** — Choose between **Existing Nodepools** (nodepools already set up in your environment) and **Recommended** (pre-configured nodepools automatically suggested based on your model’s compatibility and resource needs). - **Select a nodepool** — Browse the listed nodepools and click one to select it. Each entry shows key details including the cloud provider, cluster, region, GPU specs, memory, cores, pricing per hour, and min/max scaling limits. - **Review incompatible instances** — Select the **Existing Nodepools** tab and expand the "See instances incompatible with this model" section at the bottom to understand which options are incompatible with your model. - **Create a custom nodepool** — If none of the listed options meet your needs, click **+ Create Custom** to configure a [new cluster and nodepool](clusters-nodepools.md) from scratch. - **Save or cancel** — Click **Save Changes** to apply your selected nodepool configuration, or **Cancel** to close the pop-up without making any changes. The selected nodepool will be displayed in the **Nodepool** section. :::note #### Edit Nodepool To update the settings for a nodepool instance, click the **gear icon** next to the nodepool entry. ![ ](/img/compute-orchestration/compute-13-6.png) A configuration modal will appear, allowing you to configure both the cluster and nodepool settings for the deployment. The modal includes the following configurable fields and options: - **Use Existing Cluster** — attach the nodepool to an existing cluster instead of creating a new one. - **Cluster ID** — specify or update the cluster identifier. - **Cluster Description** — add deployment notes or a description for the cluster. - **Nodepool ID** — define or update the nodepool identifier. - **Nodepool Description** — add notes or a description for the nodepool. - **Instance Autoscaling** — configure how the nodepool scales based on workload demands: - **Minimum Instances** — the minimum number of instances to keep running. Setting this value to `0` allows the nodepool to scale down completely when idle. - **Maximum Instances** — the maximum number of instances the nodepool can scale up to during periods of high demand. The modal also displays the selected nodepool type, machine configuration, cluster name, and deployment region. After updating the desired settings, click **Save Changes** to apply and finalize the configuration updates. ::: ### Step 4: Set Dynamic Routing (Optional) Dynamic routing allows you to attach additional nodepools to a deployment so that incoming requests can be distributed across multiple compute resources. This multi-nodepool deployment helps improve availability, scalability, and performance, especially during periods of high traffic. If you want to configure additional nodepools for spillover and load balancing, click the **Add Nodepools** button. A modal will appear that allows you to [select another nodepool](#step-3-select-a-nodepool) to include in the deployment. ![ ](/img/compute-orchestration/compute-13-7.png) Once you’ve selected the additional nodepool, click **Save Changes** to apply the configuration. The selected nodepools will then appear in the **Nodepool** section of your deployment setup. ![ ](/img/compute-orchestration/compute-13-2.png) With dynamic routing, you can control how requests are distributed across nodepools — either spilling over to another pool when the primary nodepool reaches capacity or following the scheduling strategy you’ve configured. > **Note:** To set the priority order, use the drag handle (**⠿**) to reorder the nodepools — **#1** will be scheduled first. To remove a nodepool from the scheduling configuration, click the remove (**x**) button. You can use the dropdown list to select one of the following scheduling strategies: - **Default** — Requests are routed by the priority order you set. The first nodepool is preferred; subsequent pools handle overflow. - **Performance** — Schedule to the fastest known option based on hardware performance characteristics. - **Price** — Choose the cheapest compute available across all nodepools. - **Utilization** — Send to the least-used nodepool based on its current capacity. - **Network** — Optimize based on network latency between client and compute. - **Random** — Randomly distribute across all available nodepools. - **Prefer Spot** — Prefer spot instances over on-demand for cost savings. Less reliable but cheaper. - **Prefer On-Demand** — Prefer on-demand instances over spot for reliability. More expensive but stable. ### Step 5: Customize Deployment Configuration Next, configure the replica scaling and timing settings for your deployment. You can keep the default configuration or customize it to better match your workload and performance requirements. To define the number of replicas, use the **Min** and **Max** fields. Each field includes arrow controls — click the down arrow (**∨**) to decrease the value or the up arrow (**∧**) to increase it. > **Note:** This specifies the minimum and maximum range of model replicas to deploy, adjusting based on your performance needs and anticipated workload. Adding replicas enables horizontal scaling, where the workload is distributed across several instances of the model rather than relying on a single one. However, increasing them consumes more resources and may lead to higher costs. ![ ](/img/compute-orchestration/compute-14.png) To make additional changes to the deployment configuration, click the pencil icon in the upper-right corner. A modal will appear with more advanced options for configuring the deployment. ![ ](/img/compute-orchestration/compute-14-6.png) These are the additional autoscaling settings and behavior you can configure for your deployment: - **Min Replicas** — As described earlier, this is the minimum number of replicas to maintain for the deployment. Setting this value to 1 or higher ensures the deployment remains persistent (always running), but may increase compute costs. - **Max Replicas** — As described earlier, this is the maximum number of replicas allowed during autoscaling. Ensure that your nodepool has sufficient capacity to support this upper limit. - **Scale Up** — This sets the waiting period (in seconds) before adding replicas in response to rising demand. Shorter delays respond faster, but may over-provision. - **Scale Down** — This sets the waiting period (in seconds) before removing replicas after a demand decrease. A longer delay helps avoid unnecessary scale-downs during brief drips in activity. Note that your nodepool will only scale down to the minimum number of replica(s) configured. - **To Minimum** — This sets the idle time (in seconds) the deployment waits after inactivity before scaling down to the configured minimum number of instances. - **Force Scale to Zero Replicas** — Toggle this on to enforce scaling down to zero replicas to save costs. Note that this overrides the minimum replicas setting. - **Disable Nodepool Packing** — Packing refers to placing multiple replicas on the same node to improve resource utilization and reduce costs. When set to `false` (default), replicas may be packed together for efficiency. When set to `true`, deployments are restricted to a single model replica per node, which can improve isolation or meet specific performance needs, but may result in underutilized nodes and higher costs. Once you’ve configured your preferred scaling parameters, click **Save Changes** to apply the updates. :::info node autoscaling range [Click here](clusters-nodepools.md#node-range) to find out how to set up node autoscaling ranges to automatically adjust your infrastructure based on traffic demand. ::: ### Step 6: Create a Deployment > **Note:** Review the price range displayed on the page to understand the estimated compute cost and credit runway for your deployment configuration. You can also click the **Add Credits** button to [add more credits](https://docs.clarifai.com/control/account-billing#billing) if needed. Learn more about pricing [here](https://www.clarifai.com/pricing). After completing all the steps, click the **Create Deployment** button to deploy the model on your configured infrastructure. ![ ](/img/compute-orchestration/compute-14-3.png) :::tip Deployment Runtime Reminder Email Clarifai may send an automated email when a deployment has been running for some time. The email includes the deployment name, its runtime, and a **View Deployment** link to open the deployment details page. If the deployment is no longer needed, you can scale it down or delete it to avoid unnecessary compute costs. ::: ### Step 7: Review Deployment Details Once the deployment is created, you’ll be redirected to the **Deployment** details page. ![ ](/img/compute-orchestration/compute-14-4.png) This page provides a comprehensive overview of your deployment, including the following sections: * **Deployment activity** — At the top of the page, you can view key activity and status information related to the deployment, including: * Deployment status, such as *Scaling Up*, which indicates that the platform is currently provisioning compute resources and starting replicas. * Created and last modified timestamps. * Replica indicator — for example, `0 / 5` shows the number of active replicas compared to the configured maximum; in this case, no replicas are running yet, but the system is preparing to scale up. * Usage statistics, which display deployment events from audit logs over the past 12 hours. * **Resource** — Displays the model or workflow that has been deployed. * **Nodepools** — Shows the compute nodepools provisioned to run the deployment, including their hardware configuration and cost. * **Logs** — Displays the deployment’s output stream and provides tools for monitoring and troubleshooting. In this section, you can: * Search logs and filter by `INFO`, `DEBUG`, `WARN`, and `ERROR` messages. * Use tabs to monitor different stages of the deployment lifecycle: * **Runner** — Shows real-time logs from the running deployment instance, useful for monitoring live behavior and troubleshooting runtime issues. * **Runner Events** — Displays system-level events related to the runner, such as instance startup, shutdown, and infrastructure or scheduling events. * **Builder** — Shows logs from the build process, such as when the model image is built or packaged for deployment. * **Builder Events** — Displays system-level events related to the build process, including build start, completion, or failure. * Expand or collapse log entries for easier navigation. * Download logs for offline review or debugging. * **Deployment management** — In the upper-right section of the page, you can manage the deployment using the available controls: * Edit the deployment configuration * [Try Model](https://docs.clarifai.com/getting-started/playground) to run inference requests * Open the three-dot menu to delete the deployment * **Configuration settings** — Shows the deployment configuration, including scaling behavior, scheduling strategy, visibility, and other operational settings that determine how workloads are handled. :::note legacy experience If you’d like to follow the guide for creating a deployment via the UI using the legacy experience, click [here](https://github.com/Clarifai/docs/blob/3c62d8b1f36487a27b159407c76890cfd9b3703b/docs/compute/deployments/deploy-model.md#via-the-ui). ::: ## **Via the CLI** The Clarifai CLI provides a one-command deployment workflow that handles all infrastructure creation automatically. No need to manually create clusters or nodepools first. ### One-Command Deploy Deploy a local model directory to the cloud in a single step: ```bash clarifai model deploy ./my-model --instance g5.xlarge ``` This uploads your model, creates a compute cluster and nodepool, deploys the model, and monitors until it's ready. ### Deploy an Already-Uploaded Model If your model is already uploaded to Clarifai, deploy it by URL: ```bash clarifai model deploy --model-url https://clarifai.com/user/app/models/my-model --instance g5.xlarge ``` ### Browse Available Instances View all available hardware configurations using the dedicated `list-instances` command: ```bash # List all available instances clarifai list-instances # Filter by cloud provider clarifai list-instances --cloud aws # Filter by GPU type clarifai list-instances --gpu L40S # Or use the deploy flag shortcut clarifai model deploy --instance-info clarifai model deploy --instance-info --cloud aws ``` ### Override Instance Type If your `config.yaml` already has a `compute.instance` value (auto-selected during `model init`), you can override it at deploy time. The `--instance` flag **always takes priority** over the config: ```bash # config.yaml has compute.instance: g5.xlarge, but deploy with a larger GPU clarifai model deploy ./my-model --instance g6e.2xlarge ``` ### Autoscaling Control how your deployment scales: ```bash clarifai model deploy ./my-model --instance g5.xlarge --min-replicas 2 --max-replicas 10 ``` ### Lifecycle Management After deployment, use these commands to manage it: ```bash # Check status clarifai model status --deployment # Stream logs clarifai model logs --deployment # Remove deployment clarifai model undeploy --deployment ``` For the full options reference, see the [CLI Reference](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy). ## **Via the API** ### Create a Deployment To deploy a model within a nodepool you've created, provide the `deployment_id` and `config_filepath` parameters to the `create_deployment` method of the `Nodepool` class. You can learn how to create the `deployment_config.yaml` file, which contains the deployment configuration details, [here](clusters-nodepools.md#3-deployment_configyaml). > **Note:** If you're creating a [multi-nodepool deployment](clusters-nodepools.md#multi-nodepool-deployment) using the Python SDK, initializing the `Nodepool` instance with the first nodepool only is sufficient. {CO12} {Curl1} Example Output ```text [INFO] 14:45:29.871319 Deployment with ID 'test-deployment' is created: code: SUCCESS description: "Ok" req_id: "sdk-python-11.7.5-1eb407b9e125478287d552fb76bc37dd" ``` After creating it, initialize the `Deployment` class by providing the `user_id` and `deployment_id` parameters. {CO15} ### Restrict Deployments You can specify the type of compute cluster an existing model you own is deployed to. By setting the `deploy_restriction` value, you can patch a model and define whether it runs on shared or dedicated resources. These are the values you can set: - `0` (`USAGE_RESTRICTION_NOT_SET`) — The default where no explicit restriction is set. - `1` (`NO_LIMITS`) — The model can be deployed on any kind of compute (shared or dedicated). There are no policy constraints. - `2` (`SHARED_COMPUTE_ONLY`) — The model can only run on shared compute resources. This is typically cheaper but may have lower isolation or performance guarantees. - `3` (`DEDICATED_COMPUTE_ONLY`) — The model can only run on dedicated compute resources. This is used when you need guaranteed performance, security isolation, or compliance. {CURLRestrictDeployment} Example Output {OutputCURLRestrictDeployment} --- ## Manage Your Compute # Manage Your Compute **Edit and delete deployments, nodepools, and clusters** You can efficiently manage your deployments, nodepools, and clusters within the Clarifai's platform to optimize performance and costs, as well as fine-tune your compute environment for tasks like model inference. You can easily edit configurations, adjust resource allocations, or remove unused resources to free up compute infrastructure as your workload requirements evolve. ## **Via the UI** ### Deployments The **Deployments** page provides a centralized workspace for viewing, monitoring, and managing all deployments across your compute infrastructure. To access the page, expand the **Compute** section in the collapsible left navigation sidebar and select **Deployments**. This opens a comprehensive dashboard where you can track deployment activity, monitor resource usage, and perform deployment-related actions from a single interface. ![](/img/compute-orchestration/compute-26-2.png) From the **Deployments** page, you can: - **Create a deployment** — Click the **Create Deployment** button in the upper-right corner to deploy a new model. - **Filter deployments** — Quickly narrow down deployments using the filter controls at the top of the page. You can filter by **Cluster**, **Nodepool**, **GPU Hardware**, **Model**, or deployment **Status**. - **View deployment details** — Each deployment row displays key information, including the deployment name, associated model, nodepool, region, current status, replica configuration, usage metrics, estimated cost, and last modified date. - **Monitor deployment status** — Easily identify the current state of a deployment, such as `Idle` when no replicas are running, along with configured minimum and maximum replica limits. - **Manage deployments** — Use the action controls on each deployment row to: - Pause traffic; that is, scale the deployment to zero replicas (stopping all traffic until restarted) - Open the deployed model in the [Playground](https://docs.clarifai.com/getting-started/quickstart-playground) for testing - View deployment logs and runtime details - **Access additional actions** — Click the three-dot menu at the end of a deployment row to open additional management options, including: - Copy the deployment ID - View the deployment - Edit the deployment - Delete the deployment - **Track activity trends** — The activity graph provides a quick visual overview of recent deployment activities and usage patterns over the past several days. > **Note:** The page also includes sorting and layout options, allowing you to switch between **List** and **Grid** views for easier deployment management. :::note Alternatively If you click a deployment listed on the page, you'll be redirected to its dedicated details page, where you can view deployment information, monitor runtime activity, manage scaling and traffic settings, access logs, and perform additional deployment management tasks. ![ ](/img/compute-orchestration/compute-5.png) ::: ### Nodepools The **Nodepools** page provides a centralized view for managing the compute resources available across your clusters. It allows you to monitor nodepool capacity, hardware configuration, deployment usage, and operational status from a single interface. To access the page, expand the **Compute** section in the collapsible left navigation sidebar and select **Nodepools**. This opens the **Nodepools** dashboard, where you can view, filter, and manage all nodepools across your infrastructure. ![ ](/img/compute-orchestration/compute-18.png) From the **Nodepools** page, you can: - **Create a nodepool** — Click the **Create Nodepool** button in the upper-right corner to provision a new nodepool with your preferred infrastructure and hardware configuration. - **Filter nodepools** — Use the filters at the top of the page to quickly locate nodepools by **Cluster**, **Region**, **Cloud Provider**, or **GPU Hardware**. - **View nodepool details** — Each nodepool row displays important information, including: - Associated cluster and region - Hardware specifications such as CPU, memory, and GPU type - Minimum and maximum instance limits - Hourly cost estimates - Number of active deployments - **Monitor compute capacity** — Easily track configured scaling limits and available hardware resources for each nodepool. - **View deployment usage** — The **Deploys** column shows how many deployments are currently using a nodepool, with quick access to view associated deployments. - **Manage nodepools** — Use the action controls on each row to: - Deploy a model to the nodepool - Edit the nodepool configuration - **Access additional actions** — Clicking the three-dot menu opens a pop-up menu with additional management options, including: - Copy the nodepool ID - Deploy a Model - Edit the nodepool - Delete the nodepool - **Track activity trends** — The activity graph provides a quick visual overview of recent nodepool activities and usage patterns over the past several days. > **Note:** The page also supports **List** and **Grid** layouts, allowing you to choose the view that best fits your workflow and monitoring preferences. :::note Alternatively If you click a nodepool listed on the page, you'll be redirected to its dedicated details page, where you can view additional information, monitor activity, and perform further management and configuration tasks for that nodepool. ![ ](/img/compute-orchestration/compute-17.png) ::: ### Clusters The **Clusters** page provides a centralized view for managing all compute clusters across your environments. It allows you to monitor cluster configuration, deployment usage, nodepool allocation, and overall infrastructure status from a single interface. To access the page, expand the **Compute** section in the collapsible left navigation sidebar and select **Clusters**. This opens the **Clusters** dashboard, where you can view, filter, and manage all available clusters. ![ ](/img/compute-orchestration/compute-6.png) From the **Clusters** page, you can: - **Create a cluster** — Click the **Create Cluster** button in the upper-right corner to provision a new cluster. - **Filter clusters** — Use the filter controls at the top of the page to quickly find clusters by **Cloud Provider** or **Region**. - **View cluster details** — Each cluster row displays important information, including: - Cloud provider and region - Visibility settings - Cluster type - Estimated cost - **Monitor infrastructure usage** — Quickly view how many nodepools and deployments are associated with each cluster, helping you track infrastructure utilization and workload distribution. - **Manage clusters** — Use the action controls on each row to: - Add a nodepool to the cluster - Edit cluster settings - **Access additional actions** — Clicking the three-dot menu opens a pop-up menu with additional management options, including: - Copy cluster ID - Deploy a model - Add a new nodepool - Edit the cluster - Delete the cluster > **Note:** The page supports both **List** and **Grid** layouts, allowing you to switch between viewing styles based on your workflow and monitoring preferences. :::note Alternatively If you click a cluster listed on the page, you'll be redirected to its dedicated details page, where you can view additional information, monitor cluster activity, manage associated nodepools and deployments, and perform further configuration and administrative tasks. ![ ](/img/compute-orchestration/compute-6-3.png) ::: ## **Via the API** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CO3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/get_compute_cluster.py"; import CO4 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/list_compute_cluster.py"; import CO8 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/get_nodepool.py"; import CO9 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/list_nodepool.py"; import CO13 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/get_deployment.py"; import CO14 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/list_deployment.py"; import CO16 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/delete_deployments.py"; import CO17 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/delete_nodepools.py"; import CO18 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/delete_clusters.py"; import CL5 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_list_cluster.sh"; import CL6 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_list_nodepool.sh"; import CL7 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_list_deployment.sh"; import CL8 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_delete_cluster.sh"; import CL9 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_delete_nodepool.sh"; import CL10 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/cli_delete_deployment.sh"; import GetDeploymentOutput from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/get_deployment_output.txt"; import CurlList1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_1.sh"; import CurlList2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_2.sh"; import CurlList3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_3.sh"; import CurlList4 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_4.sh"; import CurlOutput1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_output_1.txt"; import CurlOutput2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_output_2.txt"; import CurlOutput3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_output_3.txt"; import CurlOutput4 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_output_4.txt"; import CurlGetCluster from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_get_cluster.sh"; import CurlListClusters from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_clusters.sh"; import CurlEditCluster from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_edit_cluster.sh"; import CurlDeleteCluster from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_delete_cluster.sh"; import CurlDeleteNodepool from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_delete_nodepool.sh"; import CurlGetNodepool from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_get_nodepool.sh"; import CurlEditNodepool from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_edit_nodepool.sh"; import CurlListNodepools from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_nodepools.sh"; import CurlListNodepoolsReplicas from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_list_nodepools_replicas.sh"; import CurlGetDeployment from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_get_deployment.sh"; import CurlEditDeployment from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_edit_deployment.sh"; import CurlDeleteDeployment from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_delete_deployment.sh"; :::note tip A Personal Access Token (PAT) authenticates your connection to the Clarifai platform, including CLI sessions. Set the token as an environment variable [as explained previously](clusters-nodepools.md#get-a-pat). ::: ### Clusters #### Get a Cluster To get the details of your compute cluster, pass the `compute_cluster_id` to the `compute_cluster` method of the `User` class. {CO3} {CurlGetCluster} Example Output ```text Clarifai Compute Cluster Details: description=My AWS compute cluster, cloud_provider=id: "aws" name: "AWS" , region=us-east-1, created_at=seconds: 1757331634 nanos: 59523000 , modified_at=seconds: 1757331634 nanos: 59523000 , visibility=gettable: PRIVATE , cluster_type=dedicated, managed_by=clarifai, key=id: "****" , id=test-compute-cluster, user_id=alfrick ``` #### List All Clusters To list all your existing compute clusters, call the `list_compute_clusters` method of the `User` class. {CO4} {CL5} {CurlListClusters} Example Output ```text Available Compute Clusters: - ID: advanced-cluster-ebus, Description: , Region: us-east-1 - ID: test-compute-cluster, Description: My AWS compute cluster, Region: us-east-1 ``` #### Edit a Cluster You can update an existing cluster by setting `"action": "overwrite"` in the request body. {CurlEditCluster} #### Delete Compute Clusters To delete your compute clusters, provide a list of compute cluster IDs to the `delete_compute_clusters` method of the `User` class. {CO18} {CL8} {CurlDeleteCluster} --> Example Output ```text Compute Cluster Deleted code: SUCCESS description: "Ok" req_id: "sdk-python-11.7.5-dc2a5ef7b8824ed0999dad18b5594a12" ``` ### Nodepools #### Get a Nodepool To get the details of your nodepool, provide the `nodepool_id` to the `nodepool` method of the `ComputeCluster` class. {CO8} {CurlGetNodepool} Example Output ```text Nodepool Details: description=First nodepool in AWS in a proper compute cluster, created_at=seconds: 1757331678 nanos: 990816000 , modified_at=seconds: 1757331678 nanos: 990816000 , compute_cluster=id: "test-compute-cluster" description: "My AWS compute cluster" cloud_provider { id: "aws" name: "AWS" } region: "us-east-1" user_id: "alfrick" created_at { seconds: 1757331634 nanos: 59523000 } modified_at { seconds: 1757331634 nanos: 59523000 } visibility { gettable: PRIVATE } cluster_type: "dedicated" managed_by: "clarifai" key { id: "****" } , node_capacity_type=capacity_types: ON_DEMAND_TYPE , instance_types=[id: "g5.2xlarge" description: "g5.2xlarge" compute_info { cpu_memory: "29033Mi" num_accelerators: 1 accelerator_memory: "23028Mi" accelerator_type: "NVIDIA-A10G" cpu_limit: "7525m" } price: "42.000000" cloud_provider { id: "aws" name: "aws" } region: "us-east-1" ], max_instances=1, visibility=gettable: PRIVATE , enforced_max_instances=1, id=test-nodepool ``` #### List All Nodepools To list all the existing nodepools in your cluster, call the `list_nodepools` method of the `ComputeCluster` class. {CO9} {CL6} {CurlListNodepools} {CurlListNodepoolsReplicas} Example Output ```text Available Nodepools: - ID: test-nodepool, Description: First nodepool in AWS in a proper compute cluster, Min Instances: 0, Max Instances: 1 ``` #### Edit a Nodepool You can update an existing nodepool by setting `"action": "overwrite"` in the request body. {CurlEditNodepool} #### Delete Nodepools To delete your nodepools, provide a list of nodepool IDs to the `delete_nodepools` method of the `ComputeCluster` class. {CO17} {CL9} {CurlDeleteNodepool} --> Example Output ```text Nodepools Deleted code: SUCCESS description: "Ok" req_id: "sdk-python-11.7.5-d69f92a0263b41719b51083f44d6ed43" ``` ### Deployments #### Get a Deployment To get the details of your deployment, provide the `deployment_id` to the `deployment` method of the `Nodepool` class. {CO13} {CurlGetDeployment} Example Output {GetDeploymentOutput} #### List All Deployments To list all the existing deployments in your nodepool, call the `list_deployments` method of the `Nodepool` class. {CO14} {CL7} Example Output ```text Available Deployments: - ID: test-deployment, Description: some random deployment, Min Replicas: 0, Max Replicas: 5 ``` #### Edit a Deployment You can update an existing deployment by setting `"action": "overwrite"` in the request body. {CurlEditDeployment} #### Delete Deployments To delete your deployments, pass a list of deployment IDs to the `delete_deployments` method of the `Nodepool` class. {CO16} {CL10} {CurlDeleteDeployment} --> Example Output ```text Deployments Deleted code: SUCCESS description: "Ok" req_id: "sdk-python-11.7.5-a08b6c5f21674916ba5791df8eae5dd8" ``` ### List Cloud Providers You can retrieve the available cloud providers to use when creating clusters and nodepools. {CurlList1} Example Output {CurlOutput1} ### List Regions You can retrieve the geographic regions supported by a cloud provider. {CurlList2} Example Output {CurlOutput2} ### List Instance Types You can retrieve the instance types a cloud provider offers in a given region. {CurlList3} Example Output {CurlOutput3} ### List All Instance Types You can retrieve all instance types offered by all cloud providers across their supported regions. {CurlList4} Example Output {CurlOutput4} --- ## Model Inference # Model Inference **Perform predictions with models** Clarifai’s Compute Orchestration capabilities enable you to make prediction calls efficiently, tailored to a variety of use cases. You can use these features to run inferences seamlessly and get results from your model with ease. ## Predict With Compute Orchestration To make a prediction request using our Compute Orchestration capabilities, you'll need to [set up a cluster](https://docs.clarifai.com/compute/deployments/clusters-nodepools), create a nodepool, and [deploy your model](https://docs.clarifai.com/compute/deployments/deploy-model) to it. Once deployed, you’ll need to reference the model’s `deployment_id` — or alternatively, specify both the `compute_cluster_id` and `nodepool_id`. These parameters ensure that your prediction request is routed correctly to the intended compute resources. For example, you can route requests to a GCP cluster by selecting a corresponding deployment ID, use a different deployment ID for an AWS cluster, and yet another for an on-premises deployment. This gives you full control over performance, costs, and security, allowing you to focus on building cutting-edge AI solutions while we handle the infrastructure complexity. :::tip Learn how Clarifai routes prediction requests for optimal performance [here](https://docs.clarifai.com/compute/inference/routing/). ::: ## Structure of Prediction Methods :::note Supported Input and Output Data Types [Click here](https://docs.clarifai.com/compute/models/model-upload/data-types) to explore the wide range of input and output data types supported by Clarifai’s model framework. You'll also find client-side examples that show how to work with these rich data formats effectively. ::: Clarifai models are mostly built using three primary files: `model.py`, `requirements.txt`, and `config.yaml`. As described [here](https://docs.clarifai.com/compute/models/model-upload/#b-prediction-methods), the core prediction logic resides in `model.py`, which defines how your model processes inputs and generates outputs. When making predictions on the client side, the structure of the prediction methods directly reflects the method signatures defined in the `model.py` file. This one-to-one mapping allows you to make custom predictions with flexible naming and argument structures, giving you full control over how you invoke models. Here are some examples of this method mapping approach: | `model.py` Model Implementation | Client-Side Usage Pattern | |-------------------------------------------------|--------------------------| | `@ModelClass.method def predict(...)` | `model.predict(...)` | | `@ModelClass.method def generate(...)` | `model.generate(...)` | | `@ModelClass.method def stream(...)` | `model.stream(...)` | This design provides flexibility in how to make model predictions. For example, a method could be defined as `@ModelClass.method def analyze_video(...)` in `model.py`, and then you can call it on the client side using `model.analyze_video(...)`. Here are some key characteristics of this design: - Method names must match exactly between `model.py` and client usage. - Parameters retain the same names and types as defined in the method. - Return types follow the structure defined by the model's outputs. ## Predict With OpenAI-Compatible Format Clarifai supports various providers that you can use to interact with different models. We offer an OpenAI-compatible API endpoint, allowing any OpenAI-compatible library or client to seamlessly send requests directly to Clarifai. :::info Base URL for Clarifai's OpenAI endpoint: `https://api.clarifai.com/v2/ext/openai/v1`. ::: This integration capability offers several advantages, including: - **Access to diverse models** — Harness Clarifai's rich array of models directly within your OpenAI projects, expanding your AI capabilities. - **Standardized interaction** — Interact with Clarifai-hosted models using familiar OpenAI API patterns and interfaces, reducing the learning curve and streamlining development. - **Enhanced flexibility** — Leverage the power of Clarifai's platform while maintaining the flexibility of your chosen OpenAI development environment. :::note Usage-based billing is handled directly through Clarifai — not through OpenAI or any other provider. Also, while most OpenAI parameters are supported, certain advanced features may be unavailable depending on the specific model or endpoint. ::: import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Advanced Inference Options # Advanced Inference Options **Learn how to use our advanced inference operations** :::warning note To find out which advanced inference parameters a model supports, you can review its description and notes on the Clarifai Community platform, or run the snippet shown [here](https://docs.clarifai.com/compute/inference/clarifai/#get-method-signature) to inspect the model’s signature. ::: The advanced inference operations allow you to fine-tune how outputs are generated, giving you greater control to manipulate results according to their specific tasks and requirements. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [CLI](https://docs.clarifai.com/additional-resources/api-overview/cli), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeBP from "!!raw-loader!../../../code_snippets/python-sdk/inference/batch_predict.py"; import CodeBPTS from "!!raw-loader!../../../code_snippets/python-sdk/inference/batchPredict.ts"; import CodeDiffBase from "!!raw-loader!../../../code_snippets/python-sdk/inference/diff_baseurl.py"; import CodeRoot from "!!raw-loader!../../../code_snippets/python-sdk/inference/root_ca.py"; import PythonByModelVersion from "!!raw-loader!../../../code_snippets/api-guide/predict/python/prediction_parameters_by_model_version_id.py"; import PythonMaxConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/python/prediction_parameters_max_concepts.py"; import PythonMinPredictValue from "!!raw-loader!../../../code_snippets/api-guide/predict/python/prediction_parameters_min_predict_value.py"; import PythonSelectConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/python/prediction_parameters_select_concepts.py"; import JavaScriptByModelVersion from "!!raw-loader!../../../code_snippets/api-guide/predict/js/prediction_parameters_by_model_version_id.html"; import JavaScriptMaxConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/js/prediction_parameters_max_concepts.html"; import JavaScriptMinPredictValue from "!!raw-loader!../../../code_snippets/api-guide/predict/js/prediction_parameters_min_predict_value.html"; import JavaScriptSelectConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/js/prediction_parameters_select_concepts.html"; import NodeJSByModelVersion from "!!raw-loader!../../../code_snippets/api-guide/predict/node/prediction_parameters_by_model_version_id.js"; import NodeJSMaxConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/node/prediction_parameters_max_concepts.js"; import NodeJSMinPredictValue from "!!raw-loader!../../../code_snippets/api-guide/predict/node/prediction_parameters_min_predict_value.js"; import NodeJSSelectConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/node/prediction_parameters_select_concepts.js"; import JavaByModelVersion from "!!raw-loader!../../../code_snippets/api-guide/predict/java/prediction_parameters_by_model_version_id.java"; import JavaMaxConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/java/prediction_parameters_max_concepts.java"; import JavaMinPredictValue from "!!raw-loader!../../../code_snippets/api-guide/predict/java/prediction_parameters_min_predict_value.java"; import JavaSelectConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/java/prediction_parameters_select_concepts.java"; import PHPByModelVersion from "!!raw-loader!../../../code_snippets/api-guide/predict/php/prediction_parameters_by_model_version_id.php"; import PHPMaxConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/php/prediction_parameters_max_concepts.php"; import PHPMinPredictValue from "!!raw-loader!../../../code_snippets/api-guide/predict/php/prediction_parameters_min_predict_value.php"; import PHPSelectConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/php/prediction_parameters_select_concepts.php"; import CurlByModelVersion from "!!raw-loader!../../../code_snippets/api-guide/predict/curl/prediction_parameters_by_model_version_id.sh"; import CurlMaxConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/curl/prediction_parameters_max_concepts.sh"; import CurlMinPredictValue from "!!raw-loader!../../../code_snippets/api-guide/predict/curl/prediction_parameters_min_predict_value.sh"; import CurlSelectConcepts from "!!raw-loader!../../../code_snippets/api-guide/predict/curl/prediction_parameters_select_concepts.sh"; import CodeOutputExample4 from "!!raw-loader!../../../code_snippets/api-guide/predict/code_output_examples/prediction_parameters_model_version_id.txt"; ## Perform Batch Predictions You can process multiple inputs in a single request, streamlining the prediction workflow and saving both time and resources. :::info The batch size should not exceed 128. Learn more [here](https://docs.clarifai.com/create-manage/inputs/upload/#upload-limits). ::: {CodeBP} {CodeBPTS} ## Customize Base_URL You can obtain model predictions by customizing the `base_url`. This allows you to easily adapt your endpoint to different environments, providing a flexible and seamless way to access model predictions. :::info This feature is particularly useful for enterprises using on-premises deployments, allowing the `base_url` to be configured to point to their respective servers. ::: {CodeDiffBase} ## Add Root Certificate A root certificate provides an additional layer of security when communicating through APIs. As a self-signed certificate that verifies the legitimacy of other certificates, it establishes a chain of trust — ensuring that you are connecting to authentic APIs and that your data remains encrypted. You can add your own root certificates to further strengthen data security and protect user privacy. {CodeRoot} ## Prompt Types A prompt is a piece of text or set of instructions that you provide to generative AI models, such as Large Language Models (LLMs), to generate a specific response or action. Generative AI models are a type of artificial intelligence system that are designed to create new content, such as text, images, audio, or even videos, based on patterns learned from existing data. There are several prompting techniques you can use to communicate with generative AI models. For example, zero-shot prompting leverages a model’s inherent language understanding capabilities to generate responses without any specific preparation or examples. You can learn about other prompting techniques [here](https://docs.clarifai.com/portal-guide/agent-system-operators/prompter/). Here are some examples of prompts. ### Question Answering ```text Prompt: Who was president of the United States in 1955? ``` ### Grammar Correction ```text Prompt: Correct this to standard English: She no went to the market. Sample Response: She did not go to the market. ``` ### Summarize ```text Prompt: Summarize this: Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half times that of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history. It is named after the Roman god Jupiter.[19] When viewed from Earth, Jupiter can be bright enough for its reflected light to cast visible shadows,[20] and is on average the third-brightest natural object in the night sky after the Moon and Venus. Sample Response: Jupiter is the fifth planet from the Sun and is very big and bright. It can be seen with our eyes in the night sky and it has been known since ancient times. Its name comes from the Roman god Jupiter. It is usually the third brightest object in the night sky after the Moon and Venus. ``` ### Translation ```text Prompt: Translate this into 1. French, 2. Spanish, and 3. Japanese: What rooms do you have available?` Sample Response: Quels sont les chambres que vous avez disponibles? 2. ¿Qué habitaciones tienes disponibles? 3. どの部屋が利用可能ですか？ ``` :::tip [Click here](https://platform.openai.com/docs/examples) for more prompting examples. ::: ## Types of Inference Parameters When making predictions using the models on our platform, some of them offer the ability to specify various inference parameters to influence their output. These parameters control the behavior of the model during the prediction process, affecting aspects like creativity, coherence, and the diversity of the output. Let’s talk about them. ### Max Tokens (or Max Length) Max Tokens specifies the maximum number of tokens (words or characters) the model is allowed to generate in a single response. It limits the length of the output, preventing the model from generating overly long responses. As such, shorter token lengths will provide faster performance. This inference parameter helps in controlling the verbosity of the output, especially in applications where concise responses are required. Here is a usage example: inference_params = dict(max_tokens=100) Model(model_url).predict(inputs,inference_params=inference_params) ### Minimum Prediction Value The `min_value` specifies the minimum prediction confidence required to include a result in the output. For example if you want to see all concepts with a probability score of `.95` or higher, this parameter will allow you to accomplish that. Also note that if you don't specify the number of `max_concepts`, you will only see the top 20. If your result can contain more values you will have to increase the number of maximum concepts as well. Here is a usage example: output_config = dict(min_value=0.6) Model(model_url).predict(inputs,output_config=output_config) {PythonMinPredictValue} {JavaScriptMinPredictValue} {NodeJSMinPredictValue} {JavaMinPredictValue} {PHPMinPredictValue} {CurlMinPredictValue} ### Maximum Concepts The `max_concepts` parameter specifies how many concepts and their associated probability scores the Predict endpoint should return. If not set, the endpoint defaults to returning the top 20 concepts. You can currently set `max_concepts` to any value between 1 and 200. If your use case requires more than 200 concepts, please reach out to our [support team](mailto:support@clarifai.com) for assistance. output_config = dict(max_concepts=3) Model(model_url).predict(inputs,output_config=output_config) {PythonMaxConcepts} {JavaScriptMaxConcepts} {NodeJSMaxConcepts} {JavaMaxConcepts} {PHPMaxConcepts} {CurlMaxConcepts} ### Select Concepts The `select_concepts` specifies the concepts to include in the prediction results. By putting this additional parameter on your predict calls, you can receive predict value\(s\) for **only** the concepts that you want to. You can specify particular concepts by either their id and/or their name. The concept names and ids are case sensitive; and so, these must be exact matches. > **Note**: You can use the [`GetModelOutputInfo`](https://docs.clarifai.com/api-guide/model/create-get-update-and-delete#get-model-output-info-by-id) endpoint to retrieve an entire list of concepts from a given model, and get their ids and names. :::caution If you submit a request with not an exact match of the concept id or name, you will receive an invalid model argument error. However, if one or more matches while one or more do not, the API will respond with a Mixed Success. ::: output_config = dict(select_concepts=["concept_name"]) Model(model_url).predict(inputs,output_config=output_config) {PythonSelectConcepts} {JavaScriptSelectConcepts} {NodeJSSelectConcepts} {JavaSelectConcepts} {PHPSelectConcepts} {CurlSelectConcepts} ### Temperature Temperature is a decimal number (between 0 and 1) that controls the degree of randomness in the response. A low temperature (e.g., 0.2) makes the model more deterministic, leading to a more conservative and predictable output. A high temperature (e.g., 0.8) increases the randomness, allowing for more creative and varied responses. Adjusting temperature is useful when you want to balance between creative responses and focused, precise answers. Here is a usage example: inference_params = dict(temperature=0.2) Model(model_url).predict(inputs,inference_params=inference_params) ### Top-p (Nucleus) Top-p sampling is an alternative to temperature sampling that controls output diversity by considering the smallest set of tokens whose cumulative probability is greater than or equal to a specified threshold p (e.g., 0.9). Rather than restricting the model to a fixed number of top tokens, this method dynamically adjusts the selection based on token probabilities, ensuring that the most likely tokens are always included while maintaining flexibility in the number of tokens considered. It’s useful when you want to dynamically control the diversity of the generated output without setting a fixed limit on the number of tokens. ### Top-k Top-k sampling limits the model to only consider the top k most probable tokens when generating the next word, ignoring all others. A low k (e.g., 10) reduces diversity by restricting the choice of tokens, leading to more focused outputs. A high k increases diversity by allowing a broader range of possible tokens, leading to more varied outputs. It’s useful when you want to prevent the model from choosing rare, less likely words, but still allow for some diversity. ### Reasoning Effort The Reasoning Effort parameter controls how much internal reasoning the model performs before generating a response. You can set it to: - **Low** – Prioritizes faster responses and minimal token usage. - **Medium** – Strikes a balance between response time and depth of reasoning. - **High** – Emphasizes thorough reasoning, which may lead to slower but more detailed answers. You can adjust this setting based on your needs — whether you value speed, detail, or a balance of both. ### Number of Beams The Number of Beams inference parameter is integral to a method called beam search. Beam search is a search algorithm that keeps track of the top n (beam width) sequences at each step of generation, considering multiple possibilities before selecting the best one. It helps produce more coherent and optimized outputs by exploring multiple potential sequences. This parameter is particularly useful in tasks where the quality and diversity of the entire sequence is crucial, such as translation or summarization. ### Do Sample This parameter determines whether the model should sample from the probability distribution of the next token or select the token with the highest probability. If set to true, the model samples from the probability distribution, introducing randomness and allowing for more creative and diverse outputs. If set to false, the model selects the token with the highest probability, leading to more deterministic and predictable responses. Sampling is typically enabled (set to true) when you want the model to generate varied and creative text. When precision and consistency are more important, sampling may be disabled (set to false). ### Return Full Text This parameter determines whether the entire generated text should be returned or just a portion of it. If set to true, the model returns the full text, including both the prompt (if provided) and the generated continuation. If set to false, the model returns only the newly generated text, excluding the prompt. It’s useful when you need the complete context, including the prompt, in the output. This can be important for understanding the generated response in the context of the input. ### System Prompt A system prompt is a special input prompt provided to guide the model's behavior throughout the conversation or task. It sets the tone, style, or context for the model’s responses. It influences how the model generates responses by setting expectations or providing instructions that the model follows. It’s often used in conversational AI to define the role the model should play (e.g., a helpful assistant, a friendly chatbot) or in specialized tasks where specific behavior or output style is desired. It helps steer the model's responses in a consistent and contextually appropriate direction. ### Prompt Template A prompt template serves as a pre-configured piece of text used to instruct an LLM. It acts as a structured query or input that guides the model in generating the desired response. You can use a template to tailor your prompts for different use cases. Many LLMs require prompts to follow a specific template format. To streamline this process, we provide the `prompt_template` inference parameter, which automatically applies the correct template format for the LLM. This means that you do not need to manually format your prompts when using an LLM through our UI or [SDK](https://docs.clarifai.com/sdk/Inference-from-AI-Models/Text-as-Input#text-generation-using-llm#set-inference-parameters). By default, the `prompt_template` is set to the LLM's standard template, allowing you to simply enter your prompts without worrying about formatting. The prompts will be automatically adjusted to fit the required template. If you need more flexibility, you can customize the `prompt_template` parameter. When modifying this variable, make sure it includes the placeholder `{prompt}`, which will be replaced with the user's prompt input. For example, the [Openchat-3.6-8b](https://clarifai.com/openchat/openchat/models/openchat-3_6-8b-20240522) model supports the following chat template format: ```text GPT4 Correct User: {prompt}GPT4 Correct Assistant: ``` Let’s break down the meaning of the template: - `GPT4 Correct User`: — This delimiter indicates the start of a user's input. - `{prompt}`: — This substring will be replaced by the actual input or question from the user. It must be included in the prompt template. It works just like the [prompter node](https://docs.clarifai.com/portal-guide/agent-system-operators/prompter#zero-shot-prompting) in a workflow builder, which must contain the `{data.raw.text}` substring. When your text data is inputted at inference time, all occurrences of the `{prompt}` variable within the template will be replaced with the prompt text. - ``:— This delimiter indicates the end of a user's input. - `GPT4 Correct Assistant:` — This indicates the start of the assistant's (or the language model's) response, which should be a corrected or refined version of the user's input or an appropriate answer to the user's question. You can also add the `` delimiter, which specifically indicates the start of a turn; in this case, a user’s input. Here is an example: ```text GPT4 Correct User: {prompt}GPT4 Correct Assistant: ``` Another example is the [Llama 3.1-8b-Instruct](https://clarifai.com/meta/Llama-3/models/llama-3_1-8b-instruct) model, which supports the following chat template format: ```text system {system_prompt}user {prompt}assistant ``` The main purpose of this format is to clearly delineate the roles and contributions of different participants in the conversation: system, user, and assistant. Let’s break down its meaning: - `` — This delimiter marks the beginning of the text content. - `system` — This indicates the beginning of a system-level instruction or context. - `{system_prompt}` — This placeholder is for the actual system-level instruction or context. - `` — This indicates the end of a text unit; in this case, the system prompt. - `user` — This marks the beginning of a user's input. - `{prompt}` — As earlier described, this placeholder represents the actual prompt or query from the user. - `` — This marks the end of a text unit; in this case, the user's input. - `assistant` — This indicates the beginning of the assistant's response. ## Predict By Model Version ID Every time you train a custom model, it creates a new model version. By specifying `version_id` in your predict call, you can continue to predict on a previous version, for consistent prediction results. Clarifai also updates its pre-built models on a regular basis. If you are looking for consistent results from your predict calls, use `version_id`. If the model `version_id` is not specified, predict will default to the most current model. Below is an example of how you would set a model version ID and receive predictions from Clarifai's [`general-image-recognition`](https://clarifai.com/clarifai/main/models/general-image-recognition) model. {PythonByModelVersion} {JavaScriptByModelVersion} {NodeJSByModelVersion} {JavaByModelVersion} {PHPByModelVersion} {CurlByModelVersion} Output Example {CodeOutputExample4} --- ## Model Inference # Model Inference **Use Clarifai’s Reasoning Engine for optimized predictions** [Clarifai's Reasoning Engine](https://www.clarifai.com/press-release-clarifai-launches-reasoning-engine-optimized-for-agentic-ai-inference) delivers a fast, scalable, and streamlined way to run model inferences at production quality. Designed with a Python-first, developer-friendly architecture, it provides a powerful and flexible approach that minimizes operational overhead — allowing you to run predictions quickly, accurately, and cost-effectively. :::tip Check out [this repository](https://github.com/toswari-ai/demo-clarifai-api) for beginner-friendly examples on using Clarifai AI models for text, image, audio, and multimodal inference tasks. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CO1 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_1.py"; import NodeAvailableMethods from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/available_methods.js"; import NodeMethodSignatures from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/method_signatures.js"; import CO2 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_2.py"; import CO3 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_3.py"; import CO4 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_4.py"; ## Prerequisites ### Install Clarifai Packages - Install the latest version of the Clarifai [Python](https://github.com/Clarifai/clarifai-python/) SDK package: pip install --upgrade clarifai - Install the latest version of the Clarifai [Node.js](https://github.com/Clarifai/clarifai-nodejs) SDK package: npm install clarifai-nodejs ### Get a PAT Key You need a PAT (Personal Access Token) key to authenticate your connection to the Clarifai platform. Learn how to get it [here](https://docs.clarifai.com/control/authentication/pat/#how-to-create-a-pat-on-the-platform). ## Structure of Prediction Methods Before making a prediction with a model, it’s important to understand how its prediction methods are structured. You can learn more about the structure of model prediction methods [here](https://docs.clarifai.com/compute/inference/#structure-of-prediction-methods). ### Get Available Methods You can list all the methods implemented in the model's configuration that are available for performing model inference. {CO1} {NodeAvailableMethods} Example Output dict_keys(['predict', 'generate', 'chat']) ### Get Method Signature You can retrieve the method signature of a specified model's method to identify all its arguments and their type annotations, which are essential for performing model inference. A method signature defines the method's name, its input parameters (with types and default values), and the return type, helping you understand how to properly call the method. {CO2} {NodeMethodSignatures} Example Output def predict(prompt: str, image: data_types.Image, images: Any, chat_history: Any, max_tokens: float = 512.0, temperature: float = 1.0, top_p: float = 0.8, reasoning_effort: str = '"low"') -> str: ### Generate Example Code You can generate a sample code snippet to better understand how to perform inference using a model. {CO3} Example Output {CO4} import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Legacy Inference via API # Legacy Inference via API **Generate predictions using our older method** The legacy inference technique uses our previous API structure and is best suited for models built using the older techniques. While this method remains functional, we recommend transitioning to the [new inference method](https://docs.clarifai.com/compute/models/inference/api) for improved efficiency, scalability, and access to the latest features. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [CLI](https://docs.clarifai.com/additional-resources/api-overview/cli), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CO19 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/predict_with_model.py"; import CL22 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/cli_predict_with_model.sh"; import CO20 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/predict_with_model_2.py"; import CO21 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/predict_with_model_3.py"; import OutputExample1 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/output_example_1.txt"; import OutputExample2 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/output_example_2.txt"; import OutputExample3 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/output_example_3.txt"; import CodeVisualClassifier from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visual_classifier.py"; import CodeVisualClassifierTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visualClassifier.ts"; import CodeVisualDImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visual_d_image.py"; import CodeVisualDImageTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visualDetectorImage.ts"; import CodeVisualDVideo from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visual_d_video.py"; import CodeVisualDVideoTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visualDetectorVideo.ts"; import CodeVisualSegmenter from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visual_segmenter.py"; import CodeVisualSegmenterTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visualSegmenter.ts"; import CodeImageToText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/image_to_text.py"; import CodeImageToTextTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/imageToText.ts"; import CodeImageToImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/image_to_image.py"; import CodeImageToImageTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/imageToImage.ts"; import CodeVisualEmbed from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visual_embedder.py"; import CodeVisualEmbedTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/visualEmbedder.ts"; import CodeOutputVisualClassifier from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/visual_classifier.txt"; import CodeOutputVisualDImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/visual_d_image.txt"; import CodeOutputVisualDVideo from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/visual_d_video.txt"; import CodeOutputVisualSegmenter from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/visual_segmenter.txt"; import CodeOutputImageToText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/image_to_text.txt"; import CodeOutputVisualEmbed from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/visual_embedder.txt"; import CLIVisualClassifier from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/visual_classifier.sh"; import CLIVisualDetectorImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/visual_d_image.sh"; import CLIVisualDetectorVideo from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/visual_d_video.sh"; import CLIVisualSegmenter from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/visual_segmenter.sh"; import CLIImageToText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/image_to_text.sh"; import CLIImageToImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/image_to_image.sh"; import CLIVisualEmbedder from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/visual_embedder.sh"; import CodeTextClassifier from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_classifier.py"; import CodeTextClassifierTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/textClassifier.ts"; import CodeTextGenLLM from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_gen_llm.py"; import CodeTextGenLLMInference from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_gen_llm_inference.py"; import CodeTextGenLLMTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/textGenerationUsingLLM.ts"; import CodeTextClassLLM from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_class_llm.py"; import CodeTextClassLLMTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/textClassifierUsingLLM.ts"; import CodeTextImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_to_image.py"; import CodeTextImageTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/textToImage.ts"; import CodeTextAudio from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_to_audio.py"; import CodeTextAudioTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/textToAudio.ts"; import CodeTextEmbedder from "!!raw-loader!../../../../code_snippets/python-sdk/inference/text_embedder.py"; import CodeTextEmbedderTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/textEmbedder.ts"; import CLITextClassifier from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_classifier.sh"; import CLITextGenLLMInference from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_gen_llm_inference.sh"; import CLITextGenLLM from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_gen_llm.sh"; import CLITextClassLLM from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_class_llm.sh"; import CLITextImage from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_to_image.sh"; import CLITextAudio from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_to_audio.sh"; import CLITextEmbedder from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/text_embedder.sh"; import CodeOutputTextClassifier from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/text_classifier.txt"; import CodeOutputTextGenLLM from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/text_gen_llm.txt"; import CodeOutputTextClassLLM from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/text_class_llm.txt"; import CodeOutputTextEmbedder from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/text_embedder.txt"; import CodeAudioText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/audio_to_text.py"; import CodeAudioTextTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/audioToText.ts"; import CLIAudioText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/cli/audioToText.sh"; import CodeOutputAudioText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/audio_to_text.txt"; import CodeImageText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/imagetext_text.py"; import CodeImageTextTS from "!!raw-loader!../../../../code_snippets/python-sdk/inference/imageTextToText.ts"; import CodeImageText01 from "!!raw-loader!../../../../code_snippets/python-sdk/inference/imagetexttotext_localimage.py"; import CodeImageTextTS01 from "!!raw-loader!../../../../code_snippets/python-sdk/inference/imagetexttotext_localimage.ts"; import CodeOutputImageText from "!!raw-loader!../../../../code_snippets/python-sdk/inference/outputs/imagetext_text.txt"; import CodePythonViaURL from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/images_via_url.py"; import CodePythonViaURLMultiple from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/images_via_url_multiple.py"; import CodePythonViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/images_via_bytes.py"; import CodePythonImageDetection from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/image_detection.py"; import CodePythonImageSegmentation from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/image_segmentation.py"; import PythonImageToText from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/image_to_text.py"; import CodeJavaScriptViaURL from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/images_via_url.html" import CodeJavaScriptViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/images_via_bytes.html" import CodeJavaScriptImageDetection from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/image_detection.html" import CodeJavaScriptImageSegmentation from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/image_segmentation.html" import JavaScriptImageToText from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/image_to_text.html" import CodeNodeJSViaURL from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/images_via_url.js"; import CodeNodeJSViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/images_via_bytes.js"; import CodeNodeJSImageDetection from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/image_detection.js"; import CodeNodeJSImageSegmentation from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/image_segmentation.js"; import NodeJSImageToText from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/image_to_text.js"; import CodeJavaViaURL from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/images_via_url.java"; import CodeJavaViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/images_via_bytes.java"; import JavaImageDetection from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/image_detection.java"; import JavaImageSegmentation from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/image_segmentation.java"; import JavaImageToText from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/image_to_text.java"; import CodePHPViaURL from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/images_via_url.php"; import CodePHPViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/images_via_bytes.php"; import PHPImageDetection from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/image_detection.php"; import PHPImageSegmentation from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/image_segmentation.php"; import PHPImageToText from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/image_to_text.php"; import CurlViaURL from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/images_via_url.sh"; import CurlViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/images_via_bytes.sh"; import CurlImageDetection from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/image_detection.sh"; import CurlImageSegmentation from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/image_segmentation.sh"; import CurlViaURLMultiple from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/images_via_url_multiple.sh"; import CurlImageToText from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/image_to_text.sh"; import CodePythonViaURLVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/video_via_url.py"; import CodePythonViaBytesVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/video_via_bytes.py"; import CodeJavaScriptViaURLVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/video_via_url.html"; import CodeJavaScriptViaBytesVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/video_via_bytes.html"; import CodeNodeJSViaURLVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/video_via_url.js"; import CodeNodeJSViaBytesVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/video_via_bytes.js"; import CodeJavaViaURLVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/video_via_url.java"; import CodeJavaViaBytesVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/video_via_bytes.java"; import CodePHPViaURLVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/video_via_url.php"; import CodePHPViaBytesVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/video_via_bytes.php"; import CurlViaURLVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/video_via_url.sh"; import CurlViaBytesVideo from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/video_via_bytes.sh"; import CodePythonViaURLText from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/text_via_url.py"; import CodePythonViaBytesText from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/text_via_bytes.py"; import CodePythonViaRawText from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/text_via_raw.py"; import CodePythonImageGenerationText from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/image_generation.py"; import CodePythonTextToSpeechText from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/text_to_speech.py"; import CodeJavaScriptViaURLText from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/text_via_url.html"; import CodeJavaScriptViaBytesText from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/text_via_bytes.html"; import CodeJavaScriptViaRawText from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/text_via_raw.html"; import CodeJavaScriptImageGenerationText from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/image_generation.html"; import CodeJavaScriptTextToSpeechText from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/text_to_speech.html"; import CodeNodeJSViaURLText from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/text_via_url.js"; import CodeNodeJSViaBytesText from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/text_via_bytes.js"; import CodeNodeJSViaRawText from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/text_via_raw.js"; import CodeNodeJSImageGenerationText from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/image_generation.js"; import CodeNodeJSTextToSpeechText from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/text_to_speech.js"; import CodeJavaViaURLText from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/text_via_url.java"; import CodeJavaViaBytesText from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/text_via_bytes.java"; import CodeJavaViaRawText from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/text_via_raw.java"; import CodeJavaImageGenerationText from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/image_generation.java"; import CodeJavaTextToSpeechText from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/text_to_speech.java"; import CodePHPViaURLText from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/text_via_url.php"; import CodePHPViaBytesText from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/text_via_bytes.php"; import CodePHPViaRawText from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/text_via_raw.php"; import CodePHPImageGenerationText from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/image_generation.php"; import CodePHPTextToSpeechText from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/text_to_speech.php"; import CurlViaURLText from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/text_via_url.sh"; import CurlViaBytesText from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/text_via_bytes.sh"; import CurlImageGenerationText from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/image_generation.sh"; import CurlViaRawText from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/text_via_raw.sh"; import CurlTextToSpeechText from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/text_to_speech.sh"; import ThirdPartyKey from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/third_party_api_key.sh"; import CodePythonLLM from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/llms_3.py"; import CodeJavaScriptLLM from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/llms_3.html" import CodeNodeJSLLM from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/llms_3.js"; import CodeJavaLLM from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/llms_3.java"; import CodePHPLLM from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/llms_3.php"; import CodeCurlLLM from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/llms_3.sh"; import CodePythonParam from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/llms_param.py"; import CodeJavaScriptParam from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/llms_param.html" import CodeNodeJSParam from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/llms_param.js"; import CodeJavaParam from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/llms_param.java"; import CodePHPParam from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/llms_param.php"; import CodeCurlParam from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/llms_param.sh"; import PythonPredictURLAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/audio_via_url.py"; import PythonViaBytesAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/audio_via_bytes.py"; import JSPredictURLAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/audio_via_url.html"; import JSViaBytesAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/audio_via_bytes.html"; import NodePredictURLAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/audio_via_url.js"; import NodeViaBytesAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/audio_via_bytes.js"; import JavaPredictURLAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/audio_via_url.java"; import JavaViaBytesAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/audio_via_bytes.java"; import PHPPredictURLAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/audio_via_url.php"; import PHPViaBytesAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/audio_via_bytes.php"; import CurlPredictURLAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/audio_via_url.sh"; import CurlViaBytesAudio from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/audio_via_bytes.sh"; import CodePythonMultimodal from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/multimodal-to-text.py"; import CodeJavaScriptMultimodal from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/multimodal-to-text.html" import CodeNodeJSMultimodal from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/multimodal-to-text.js"; import CodeJavaMultimodal from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/multimodal-to-text.java"; import CodePHPMultimodal from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/multimodal-to-text.php"; import CodeCurlMultimodal from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/multimodal-to-text.sh"; import CodePythonEmbedding from "!!raw-loader!../../../../code_snippets/api-guide/predict/python/embeddings.py"; import CodeJavaScriptEmbedding from "!!raw-loader!../../../../code_snippets/api-guide/predict/js/embeddings.html" import CodeNodeJSEmbedding from "!!raw-loader!../../../../code_snippets/api-guide/predict/node/embeddings.js"; import CodeJavaEmbedding from "!!raw-loader!../../../../code_snippets/api-guide/predict/java/embeddings.java"; import CodePHPEmbedding from "!!raw-loader!../../../../code_snippets/api-guide/predict/php/embeddings.php"; import CodeCurlEmbedding from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/embeddings.sh"; import CurlUnaryStream from "!!raw-loader!../../../../code_snippets/api-guide/predict/curl/unary-stream.sh"; ## Legacy Inference via Compute Orchestration :::note Before making a prediction using our Compute Orchestration capabilities, ensure that your model has been [deployed](https://docs.clarifai.com/compute/deployments/deploy-model), [as explained here](https://docs.clarifai.com/compute/models/inference/). ::: ### Unary-Unary Predict Call This is the simplest type of prediction. In this method, a single input is sent to the model, and it returns a single response. This is ideal for tasks where a quick, non-streaming prediction is required, such as classifying an image. It supports the following prediction methods: - `predict_by_url` — Use a publicly accessible URL for the input. - `predict_by_bytes` — Pass raw input data directly. - `predict_by_filepath` — Provide the local file path for the input. {CO19} {CL22} Example Output {OutputExample1} ### Unary-Stream Predict Call The **Unary-Stream** predict call processes a single input, but returns a stream of responses. It is particularly useful for tasks where multiple outputs are generated from a single input, such as generating text completions from a prompt. It supports the following prediction methods: - `generate_by_url` — Provide a publicly accessible URL and handle the streamed responses iteratively. - `generate_by_bytes` — Use raw input data. - `generate_by_filepath` — Use a local file path for the input. {CO20} {CurlUnaryStream} Example Output {OutputExample2} ### Stream-Stream Predict Call The **stream-stream** predict call enables bidirectional streaming of both inputs and outputs, making it highly effective for processing large datasets or real-time applications. In this setup, multiple inputs can be continuously sent to the model, and the corresponding multiple predictions are streamed back in real-time. This is ideal for tasks like real-time video processing/predictions or live sensor data analysis. It supports the following prediction methods: - `stream_by_url` — Stream a list of publicly accessible URLs and receive a stream of predictions. It takes an iterator of inputs and returns a stream of predictions. - `stream_by_bytes` — Stream raw input data. - `stream_by_filepath` — Stream inputs from local file paths. {CO21} Example Output {OutputExample3} --> ## Legacy Inference via Traditional Methods ### Image as Input :::tip When you take an image with a digital device (such as a smartphone camera) the image's meta-information (such as the orientation value for how the camera is held) is stored in the image's [Exif's data](https://en.wikipedia.org/wiki/Exif). And when you use a photo viewer to check the image on your computer, the photo viewer will respect that orientation value and automatically rotate the image to present it the way it was viewed. This allows you to see a correctly-oriented image no matter how the camera was held. So, when you want to make predictions from an image taken with a digital device, you need to strip the Exif data from the image. Since the Clarifai platform does not account for the Exif data, removing it allows you to make accurate predictions using images in their desired rotation. ::: #### Visual Classifier You can use a [visual classifier](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22visual-classifier%22%5D%7D%5D) model to categorize images into predefined classes based on their visual content. You can provide image data either through URLs or by uploading files. ##### Predict via URL :::note You can send up to 128 images in a single API call, with each image file sized under 20MB. Learn more [here](https://docs.clarifai.com/create-manage/inputs/upload/#upload-limits). ::: {CodeVisualClassifier} {CodeVisualClassifierTS} {CLIVisualClassifier} {CodePythonViaURL} {CodeJavaScriptViaURL} {CodeNodeJSViaURL} {CodeJavaViaURL} {CodePHPViaURL} {CurlViaURL} Output {CodeOutputVisualClassifier} ##### Predict via Bytes Below is an example of how you would send the bytes of an image and receive model predictions. {CodePythonViaBytes} {CodeJavaScriptViaBytes} {CodeNodeJSViaBytes} {CodeJavaViaBytes} {CodePHPViaBytes} {CurlViaBytes} #### Predict Multiple Inputs To predict multiple inputs at once and avoid the need for numerous API calls, you can use the following approach. Note that these examples are provided for cURL and Python, but the same concept is applicable to any supported programming language. {CurlViaURLMultiple} {CodePythonViaURLMultiple} #### Visual Detector - Image Unlike image classification, which assigns a single label to an entire image, a [visual detector](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22visual-detector%22%5D%7D%5D) model identifies and outlines multiple objects or regions within an image, associating each with specific classes or labels. You can provide input images either through URLs or by uploading files. {CodeVisualDImage} {CodeVisualDImageTS} {CLIVisualDetectorImage} {CodePythonImageDetection} {CodeJavaScriptImageDetection} {CodeNodeJSImageDetection} {JavaImageDetection} {PHPImageDetection} {CurlImageDetection} Output {CodeOutputVisualDImage} #### Visual Segmenter You can use a [segmentation model](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22visual-segmenter%22%5D%7D%5D) to generate segmentation masks by providing an image as input. This enables detailed analysis by identifying distinct regions within the image and associating them with specific concepts. {CodeVisualSegmenter} Output {CodeOutputVisualSegmenter} {CodeVisualSegmenterTS} {CLIVisualSegmenter} {CodePythonImageSegmentation} {CodeJavaScriptImageSegmentation} {CodeNodeJSImageSegmentation} {JavaImageSegmentation} {PHPImageSegmentation} {CurlImageSegmentation} #### Image-to-Text You can use an [image-to-text](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22image-to-text%22%5D%7D%5D) model to generate meaningful textual descriptions from images. {CodeImageToText} Output {CodeOutputImageToText} {CodeImageToTextTS} {CLIImageToText} {PythonImageToText} {JavaScriptImageToText} {NodeJSImageToText} {JavaImageToText} {PHPImageToText} {CurlImageToText} #### Image-to-Image You can use an upscaling [image-to-image](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22image-to-image%22%5D%7D%5D) model to improve the quality of an image. {CodeImageToImage} {CodeImageToImageTS} {CLIImageToImage} #### Visual Embedder You can use an [embedding model](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22visual-embedder%22%5D%7D%5D) to generate embeddings from an image. Image embeddings are vector representations that capture the semantic content of an image, providing a powerful foundation for applications like similarity search, recommendation systems, and more. {CodeVisualEmbed} Output {CodeOutputVisualEmbed} {CodeVisualEmbedTS} {CLIVisualEmbedder} ### Video as Input :::note Configure FPS When processing a video input, the API returns a list of predicted concepts for each frame. By default, the video is analyzed at 1 frame per second (FPS), which corresponds to one prediction every 1000 milliseconds. This rate can be adjusted by setting the `sample_ms` parameter in your prediction request. The `sample_ms` defines the time interval, in milliseconds, between frames selected for inference. It must be a value between 100 and 60000. It is calculated as: FPS = 1000 / `sample_ms` For example, setting `sample_ms` to 1000 results in 1 FPS, which is the default rate. ::: #### Visual Detector - Video You can use a [visual detector](https://clarifai.com/explore/models?filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22visual-detector%22%5D%7D%5D&page=2&perPage=24) model to get predictions for every frame when processing a video input. You can also fine-tune your requests by adjusting parameters, such as the number of frames processed per second, giving you greater control over the speed and depth of the analysis. You can provide video inputs either through URLs or by uploading files. :::note When uploading via URL, videos must be no longer than 10 minutes in duration or 300MB in size. Learn more [here](https://docs.clarifai.com/create-manage/inputs/upload/#videos). ::: ##### Predict via URL Below is an example of how you would send video URLs and receive predictions. {CodeVisualDVideo} Output {CodeOutputVisualDVideo} {CodeVisualDVideoTS} {CLIVisualDetectorVideo} {CodePythonViaURLVideo} {CodeJavaScriptViaURLVideo} {CodeNodeJSViaURLVideo} {CodeJavaViaURLVideo} {CodePHPViaURLVideo} {CurlViaURLVideo} ##### Predict via Bytes Below is an example of how you would send the bytes of a video and receive predictions. {CodePythonViaBytesVideo} {CodeJavaScriptViaBytesVideo} {CodeNodeJSViaBytesVideo} {CodeJavaViaBytesVideo} {CodePHPViaBytesVideo} {CurlViaBytesVideo} ### Text as Input #### Text Classifier You can use a [text classifier](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22text-classifier%22%5D%7D%5D) model to automatically categorize text into predefined categories based on its content. You can provide the text data via URLs, file uploads, or by entering raw text directly. :::note The file size of each text input should be less than 20MB. Learn more [here](https://docs.clarifai.com/create-manage/inputs/upload/#text-files). ::: ##### Predict via URL Below is an example of how you would make predictions on passages of text hosted on the web. {CodeTextClassifier} {CodeTextClassifierTS} {CLITextClassifier} {CodePythonViaURLText} {CodeJavaScriptViaURLText} {CodeNodeJSViaURLText} {CodeJavaViaURLText} {CodePHPViaURLText} {CurlViaURLText} Output {CodeOutputTextClassifier} ##### Predict via Local Files Below is an example of how you would provide text inputs via local text files and receive predictions. {CodePythonViaBytesText} {CodeJavaScriptViaBytesText} {CodeNodeJSViaBytesText} {CodeJavaViaBytesText} {CodePHPViaBytesText} {CurlViaBytesText} ##### Predict via Raw Text Below is an example of how you would provide raw text inputs and receive predictions. {CodePythonViaRawText} {CodeJavaScriptViaRawText} {CodeNodeJSViaRawText} {CodeJavaViaRawText} {CodePHPViaRawText} {CurlViaRawText} #### Text Generation Using LLMs You can use [text generation](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22text-to-text%22%5D%7D%5D) models to dynamically create textual content based on user-defined prompts. {CodeTextGenLLM} {CodeTextGenLLMTS} {CLITextGenLLM} {CodePythonLLM} {CodeJavaScriptLLM} {CodeNodeJSLLM} {CodeJavaLLM} {CodePHPLLM} {CodeCurlLLM} Output {CodeOutputTextGenLLM} #### Set Inference Parameters When making predictions using LLMs on our platform, some models offer the ability to specify various inference parameters to influence their output. These parameters control the behavior of the model during the generation process, affecting aspects like creativity, coherence, and the diversity of the generated text. You can learn more about them [here](https://docs.clarifai.com/compute/inference/advanced). > **Note:** You can also find various examples of how to set inference parameters throughout this guide. {CodeTextGenLLMInference} {CLITextGenLLMInference} {CodePythonParam} {CodeJavaScriptParam} {CodeNodeJSParam} {CodeJavaParam} {CodePHPParam} {CodeCurlParam} #### Text Classification Using LLMs You can leverage [LLMs](https://clarifai.com/explore/models?filterData=%5B%7B%22field%22%3A%22use_cases%22%2C%22value%22%3A%5B%22llm%22%5D%7D%5D&page=1&perPage=24) to categorize text using carefully crafted prompts. {CodeTextClassLLM} Output {CodeOutputTextClassLLM} {CodeTextClassLLMTS} {CLITextClassLLM} #### Text-to-Image You can use a [text-to-image](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22text-to-image%22%5D%7D%5D) model to transform textual input into vibrant and expressive images. {CodeTextImage} {CodeTextImageTS} {CLITextImage} {CodePythonImageGenerationText} {CodeJavaScriptImageGenerationText} {CodeNodeJSImageGenerationText} {CodeJavaImageGenerationText} {CodePHPImageGenerationText} {CurlImageGenerationText} Output #### Text-to-Audio You can use a [text-to-audio](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22input_fields%22%2C%22value%22%3A%5B%22text%22%5D%7D%2C%7B%22field%22%3A%22use_cases%22%2C%22value%22%3A%5B%22speech-synthesis%22%2C%22text-to-speech%22%5D%7D%5D) model to convert written text into natural, expressive speech. {CodeTextAudio} {CodeTextAudioTS} {CLITextAudio} {CodePythonTextToSpeechText} {CodeJavaScriptTextToSpeechText} {CodeNodeJSTextToSpeechText} {CodeJavaTextToSpeechText} {CodePHPTextToSpeechText} {CurlTextToSpeechText} #### Text Embedder You can use an [embedding model](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22text-embedder%22%5D%7D%5D) to generate embeddings from text. These embeddings are vector representations that capture the semantic meaning of the text, making them ideal for applications such as similarity search, recommendation systems, document clustering, and more. :::note [Cohere Embed-v3](https://clarifai.com/cohere/embed/models/cohere-embed-english-v3_0) model requires an `input_type` parameter to be specified, which can be set using one of the following values: - `search_document` (default): For texts (documents) intended to be stored in a vector database. - `search_query`: For search queries to find the most relevant documents in a vector database. - `classification`: If the embeddings are used as input for a classification system. - `clustering`: If the embeddings are used for text clustering. ::: {CodeTextEmbedder} {CodeTextEmbedderTS} {CLITextEmbedder} {CodePythonEmbedding} {CodeJavaScriptEmbedding} {CodeNodeJSEmbedding} {CodeJavaEmbedding} {CodePHPEmbedding} {CodeCurlEmbedding} Output {CodeOutputTextEmbedder} ### Audio as Input #### Audio-to-Text You can use an [audio-to-text](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22audio-to-text%22%5D%7D%5D) model to convert audio files into text. This enables the transcription of spoken words for a variety of use cases, including transcription services, voice command processing, and more. ##### Predict via URL {CodeAudioText} {CodeAudioTextTS} {CLIAudioText} {PythonPredictURLAudio} {JSPredictURLAudio} {NodePredictURLAudio} {JavaPredictURLAudio} {PHPPredictURLAudio} {CurlPredictURLAudio} Output {CodeOutputAudioText} ##### Predict via Bytes {PythonViaBytesAudio} {JSViaBytesAudio} {NodeViaBytesAudio} {JavaViaBytesAudio} {PHPViaBytesAudio} {CurlViaBytesAudio} ### MultiModal as Input #### [Image,Text]-to-Text You can process multimodal inputs — combining multiple modalities, such as text, images, and/or other types of data — to generate accurate predictions. Below is an example of how you can send both image and text inputs to a model. ##### Predict via Image URL {CodeImageText} {CodeImageTextTS} {CodePythonMultimodal} {CodeJavaScriptMultimodal} {CodeNodeJSMultimodal} {CodeJavaMultimodal} {CodePHPMultimodal} {CodeCurlMultimodal} Output {CodeOutputImageText} ##### Predict via Local Image {CodeImageText01} Output {CodeOutputImageText} {CodeImageTextTS01} ### Use Third-Party API Keys :::info The ability to use third-party API keys is currently exclusively available to Enterprise users. Learn more [here](https://www.clarifai.com/pricing). ::: For the third-party models we've wrapped into our platform, like those provided by OpenAI, Anthropic, Cohere, and others, you can also choose to utilize their API keys as an option—in addition to using the default Clarifai keys. This Bring Your Own Key (BYOK) flexibility allows you to integrate your preferred services and APIs into your workflow, enhancing the versatility of our platform. Here is an example of how to add an OpenAI API key for [Dalle-3](https://clarifai.com/openai/dall-e/models/dall-e-3) for text-to-image tasks. {ThirdPartyKey} --- ## Inference via API # Inference via API **Generate predictions with models** :::tip [Click here](https://github.com/Clarifai/examples/tree/main/models/model_predict) for additional examples on how to perform model predictions using various SDKs — such as the Clarifai SDK, OpenAI client, and LiteLLM. The examples demonstrate various model types and include both streaming and non-streaming modes, as well as tool calling capabilities. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CO5 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_5.py"; import CO5Text from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_5-text.py"; import CO6 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_6.py"; import CO7 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_7.py"; import CO7Text from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_7_text.py"; import CO8 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_8.py"; import CO9 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_9.py"; import C10 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_10.py"; import CO9Output from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_9_output.txt"; import CO8Output from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_8_output.txt"; import RawProtobuf from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/raw_protobuf.py"; import RawProtobufOutput from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/raw_protobuf_output.txt"; import StreamProtobuf from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/stream_protobuf.py"; import StreamProtobufOutput from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/stream_protobuf_output.txt"; import NodePredictInterface from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/predict_interface.js"; import NodePredictInterfaceImageInputs from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/predict_interface_image_inputs.js"; import NodeStreamInterface from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/stream_interface.js"; import OutputUnaryTextInputs from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/output_1.txt"; import OutputUnaryImageInputs from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/output_2.txt"; import CO6Images from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/pythonic_6_images.py"; import NodeStreamInterfaceImages from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/stream_interface_images.js"; import PyToolCalling from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/tool_calling.py"; import Mask1 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/mask_1.py"; import Mask2 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/mask_2.py"; import Mask3 from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/mask_3.py"; import PyModelClient from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/py_model_client.py"; import NodeModelClient from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/node_model_client.js"; import AsyncPredict from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/async_predict.py"; import AsyncGenerate from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/async_generate.py"; import SetInferenceParams from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/set_inference_params.py"; import CLIUnaryUnary from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/cli_unary_unary.sh"; import CLIUnaryStream from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/cli_unary_stream.sh"; import CLIInferenceParams from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/cli_inference_params.sh"; import CLIUnaryUnaryImage from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/cli_unary_unary_image.sh"; import CLIUnaryStreamImage from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/cli_unary_stream_image.sh"; import SegmentPointsBoxes from "!!raw-loader!../../../../code_snippets/python-sdk/compute-orchestration/segment_points_boxes.txt"; ## Prerequisites ### Install Clarifai Packages - Install the latest version of the Clarifai [Python](https://github.com/Clarifai/clarifai-python/) SDK package. This also installs the [Command Line Interface (CLI)](https://docs.clarifai.com/resources/api-overview/cli). pip install --upgrade clarifai - Install the latest version of the Clarifai [Node.js](https://github.com/Clarifai/clarifai-nodejs) SDK package. npm install clarifai-nodejs ### Get a PAT Key You need a [PAT](https://docs.clarifai.com/control/authentication/pat/#how-to-create-a-pat-on-the-platform) (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. You can then set the PAT as an environment variable using `CLARIFAI_PAT`. This also authenticates your session when using the Clarifai’s CLI. > **Note:** Storing your PAT in an environment variable is more secure than hardcoding it directly in your code. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ## Prediction Tips #### Set Up Dedicated Deployment To use our dedicated Compute Orchestration capabilities, ensure your model is [deployed](https://docs.clarifai.com/compute/deployments/deploy-model). Then, specify the `deployment_id` parameter — alternatively, you can specify both `compute_cluster_id` and `nodepool_id`, as explained [here](https://docs.clarifai.com/compute/inference/#predict-with-compute-orchestration). :::info For dedicated deployments that belong to a different user or organization, alongside the `deployment_id`, also provide the user ID or the organization ID as the `Model`'s `deployment_user_id`. ::: ```text model = Model( url="MODEL_URL_HERE", deployment_id="DEPLOYMENT_ID_HERE", # For a deployment owned by a different user or organization: # deployment_user_id="USER_ID or ORGANIZATION_ID_HERE", # Or, set cluster and nodepool # compute_cluster_id = "COMPUTE_CLUSTER_ID_HERE", # nodepool_id = "NODEPOOL_ID_HERE" ) ``` :::caution note Only a curated set of featured models can be run using the default Clarifai [Shared SaaS (Serverless)](https://docs.clarifai.com/compute/overview#deployment-options) deployment option. To run inferences with any other models, you must deploy them to your own cluster and nodepool and specify the `deployment_id` parameter. ::: #### Initialize the Model Client You can initialize the model client using either explicit IDs or the full model URL. > **Note:** By default, the latest version of the model is used for inference. However, you can specify a different version when initializing the model. {PyModelClient} {NodeModelClient} #### Set Up Inference Parameters You can configure various [inference parameters](https://docs.clarifai.com/compute/inference/advanced/#types-of-inference-parameters) to customize your prediction requests to better suit your use case. Here is an example using the `max_tokens` parameter: {SetInferenceParams} {CLIInferenceParams} :::note Before making a prediction with a model, it’s important to understand how its prediction methods are structured. Learn more [here](https://docs.clarifai.com/compute/inference/clarifai/#structure-of-prediction-methods). ::: ## Unary-Unary Predict Call This is the simplest form of prediction: a single input is sent to the model, and a single response is returned. It’s ideal for quick, non-streaming tasks, such as classifying an image or analyzing a short piece of text. > **NOTE**: Streaming means that the response is streamed back token by token, rather than waiting for the entire completion to be generated before returning. This is useful for building interactive applications where you want to display the response as it's being generated. ### Text Inputs {CO5Text} {CLIUnaryUnary} {NodePredictInterface} Example Output {OutputUnaryTextInputs} ### Image Inputs #### Image-to-Text {CO5} {CLIUnaryUnaryImage} {NodePredictInterfaceImageInputs} Example Output {OutputUnaryImageInputs} #### Visual Segmentation ##### Example 1 Here’s an example that uses the [sam2_1-hiera-base-plus](https://clarifai.com/meta/segment-anything/models/sam2_1-hiera-base-plus) model to automatically generate masks for all objects in a given image. Arguments and Returns These are the model’s arguments and return values: ##### Arguments * **image (Image)**: The input image ##### Returns * **List[Region]**: List of generated masks, which can be accessed via `region.mask` or `region.proto.region_info.mask`. {Mask1} ##### Example 2 Here’s an example that uses the sam2_1-hiera-base-plus model to generate masks using points or boxes prompt. Arguments and Returns These are the model’s arguments and return values: ##### Arguments * **image (Image)**: The input image. * **regions (List[Region])**: A list of region prompts, where each region can specify either points or a bounding box. * To indicate a positive (object) point, set: `region.concepts = [Concepts(name="1", value=1)]` * To indicate a negative (background) point, set: `region.concepts = [Concepts(name="0", value=0)]` * **dict_inputs (Dict)**: Keyword arguments passed directly to the `SAM2ImagePredictor.predict(...)` method. Refer to [this example](https://github.com/facebookresearch/sam2/blob/main/notebooks/image_predictor_example.ipynb) for details. > **Note:** You may provide either `regions` or `dict_inputs`, but not both. * **multimask_output (bool)**: Whether to return multiple masks for each prompt. Defaults to `False`. * **return_type (str)**: Specifies the type of output to return. Must be one of: * `box` — return bounding boxes only * `mask` — return masks only * `all` — return both bounding boxes and masks ##### Returns * **List[Region]**: A list of generated regions. Masks can be accessed via `region.mask` or `region.proto.region_info.mask`. {Mask2} :::tip [Click here](https://docs.clarifai.com/compute/models/model-upload/data-types/) to explore how to make predictions with other data types. ::: ## Unary-Stream Predict Call This call sends a single input to the model but returns a stream of responses. This is especially useful for tasks that produce multiple outputs from one input, such as generating text completions or progressive predictions from a prompt. ### Text Inputs {CO6} {CLIUnaryStream} {NodeStreamInterface} ### Image Inputs {CO6Images} {CLIUnaryStreamImage} {NodeStreamInterfaceImages} ### Video Inputs Here’s an example that uses the [sam2_1-hiera-base-plus](https://clarifai.com/meta/segment-anything/models/sam2_1-hiera-base-plus) model to automatically track objects in a video. Arguments and Returns These are the model’s arguments and return values: ##### Arguments * **video (Video):** The input video. * **frames (List[Frame]):** A list of frames containing tracking prompts. For each frame: * `frame.data.regions` specifies object locations using points or bounding boxes, as described in the `predict` method. * The frame index is identified by `frame.frame_info.index`. * The object identity is specified using `frame.data.regions[...].track_id`. * **list_dict_inputs (List[Dict]):** A list of dictionaries following the `SAM2VideoPredictor.add_new_points_or_box()` method signature. Each dictionary may include `points`, `box`, `obj_id`, `labels`, and `frame_idx`. Refer to [this example](https://github.com/facebookresearch/sam2/blob/main/notebooks/video_predictor_example.ipynb) for details. * **return_type (str):** Specifies the type of output to return. Must be one of: * `box` — return bounding boxes only * `mask` — return masks only * `all` — return both bounding boxes and masks > **Note:** You may provide either `frames` or `list_dict_inputs`, but not both at the same time. ##### Returns * **Iterator[Frame]:** An iterator over frames containing the generated masks and tracked object results for each frame. {Mask3} ## Stream-Stream Predict Call This call enables bidirectional streaming of both inputs and outputs, making it ideal for real-time applications and processing large datasets. In this setup, multiple inputs can be continuously streamed to the model, while predictions are returned in real time. It’s especially useful for use cases like live video analysis or streaming sensor data. ### Text Inputs {CO7Text} ### Audio Inputs {CO7} ## Multimodal Predictions You can make predictions using models that support multimodal inputs, such as a combination of images and text. Here is an example: {C10} ## Batch Prediction Handling Clarifai’s model framework seamlessly supports both single and batch predictions through a unified interface. It can adapt to the input format, so no code changes are needed. The system automatically detects the type of input provided: - If you pass a single input, it’s treated as a singleton batch; - If you pass multiple inputs as a list, they are handled as a parallel batch. This means you can pass either a single input or a list of inputs, and the system will automatically process them appropriately — making your code cleaner and more flexible. ### Text Inputs {CO9} Example Output {CO9Output} ### Image Inputs {CO8} Example Output {CO8Output} ## Tool Calling Tool calling in LLMs is a capability that allows models to autonomously decide when and how to call external tools, functions, or APIs during a conversation — based on the user’s input and the context. You can learn more about it [here](https://docs.clarifai.com/compute/models/inference/open-ai#tool-calling). {PyToolCalling} ## Asynchronous Inference Asynchronous inference enables non-blocking execution of model prediction tasks. Instead of waiting for each prediction to complete before proceeding, you can use the `async_predict` and `async_generate` methods to submit multiple requests concurrently and retrieve the results once they're ready. ### Async Prediction You can use this for standard prediction tasks that return a complete result in a single response. The output is typically a structured object, like a dictionary or JSON. {AsyncPredict} ### Async Generation You can use this for generative models that produce output incrementally — such as large language models that stream tokens one by one. The response is an asynchronous stream, which you iterate over. {AsyncGenerate} ## Raw Protobuf Response Information By default, prediction methods in the [`Model`](https://docs.clarifai.com/resources/api-references/python#model) class — such as `predict()`, `generate()`, and others — return only the processed, user-friendly response. If you need deeper insight, you can pass `with_proto=True`, which makes the method return a tuple containing both the processed result and the underlying raw Protobuf response. Note that `with_proto=False` is the default behavior, meaning only the processed result is returned without the raw protobuf data. > **Note:** This parameter is supported in both [synchronous and asynchronous](#asynchronous-inference) model operations. ### Predict With Protobuf {RawProtobuf} Example Output {RawProtobufOutput} ### Stream With Protobuf {StreamProtobuf} Example Output {StreamProtobufOutput} :::tip Retrieving raw Protobuf response information works with any custom method defined in the `ModelClient` class. For example: ```python result, proto = model.my_custom_method( input_data="some data", temperature=0.7, with_proto=True ) ``` ::: --- ## Inference via UI # Inference via UI **Generate model or workflow predictions on the UI** You can perform predictions using a model directly through the Clarifai's User Interface (UI) — no code required. This method is ideal for quick testing, demos, and visual validation of your model's performance. You can simply upload an input (such as text, image, or video) and view the output predictions in real-time within an intuitive user experience. The Clarifai [Community](https://clarifai.com/explore) platform offers a wide selection of the latest AI models that you can use to run inferences. You can also [build and upload](https://docs.clarifai.com/compute/upload/) your own custom models and use them for predictions. :::info Predict With Compute Orchestration If you want to make predictions using Clarifai’s Compute Orchestration capabilities, you’ll need to set up a cluster, create a nodepool, and [deploy](https://docs.clarifai.com/compute/deployments/deploy-model) your model to it. Once deployed, you can select this deployment for running inferences. If you don’t configure a custom deployment, the platform will use the default Clarifai Shared deployment where applicable, [as explained here](https://docs.clarifai.com/compute/inference/#predict-with-compute-orchestration). ::: ## Generate Chat Predictions The Clarifai platform lets you generate human-like text in a conversational format using large language models (LLMs). You simply provide text as input, known as a prompt, and these models generate coherent, context-aware text-based responses. These models are ideal for tasks such as answering questions, summarizing content, generating code, or engaging in dialogue. ### Step 1: Get a Model After finding a language model you want to use, go to its individual page and click the **Open in Playground** button located in the upper-right corner. For this example, we'll use the [Llama-3.2-3B-Instruct](https://clarifai.com/meta/Llama-3/models/Llama-3_2-3B-Instruct) model. ![](/img/community_2/ui_inference_1.png) ### Step 2: Run Your Inference You’ll be taken to Clarifai’s AI Playground — an intuitive interface that lets you quickly run inferences with models and explore the platform’s full capabilities without any additional setup. ![](/img/community_2/ui_inference_2.png) > _Alternatively, you can access the Playground directly from the top navigation bar. Once inside, use the model selector in the upper-left corner to search and choose the model you want to use for inference._ > **Note:** If you click the **View Model Page** icon (the file symbol) next to the model selector, you’ll be taken to the model’s individual page, where you can view detailed information about it. Next to it, you’ll find the **Copy Model URL** icon, which allows you to quickly copy the model’s URL for easy sharing or reference. In the message box at the bottom of the Playground, enter your desired prompt to generate text using the selected model. You can also choose from the predefined prompt examples provided. Once your input is ready, click the **arrow icon** in the message box to submit your request. The model’s response will stream in real time, allowing you to see the output as it’s being generated — just like a live chat experience. Clarifai’s streaming capabilities allow language models to return responses token by token, enabling natural and interactive conversations. :::note For these examples, we’re using the default deployment settings (**Clarifai Shared**) and default [inference parameters](https://docs.clarifai.com/compute/inference/advanced). You can customize these settings as needed to suit more advanced use cases. ::: > _You can also toggle the button in the upper-left section of the Playground to view ready-to-use API code snippets in multiple programming languages — just copy and paste them into your project._ > ![](/img/community_2/ui_inference_3.png) ## Generate Multimodal Predictions Clarifai supports multimodal models — models that can process and understand more than one type of input at a time, such as images combined with text. These models are designed to handle complex scenarios where context from multiple input types improves prediction quality. Their common use cases include tasks like image captioning, visual question answering (VQA), and multimodal classification. ### Step 1: Get a Model After finding a multimodal model you want to use, go to its individual page and click the **Open in Playground** button located in the upper-right corner, as illustrated earlier. For this example, we’ll use the [GPT-4o](https://clarifai.com/openai/chat-completion/models/gpt-4o) model, which can process both an image and a text prompt simultaneously. ### Step 2: Run Your Inference While on the AI Playground, navigate to the message box at the bottom and click the plus (**+**) button. A pop-up window will appear, allowing you to select and upload an image. Once the image is uploaded, enter your accompanying text prompt in the message box. Then, click the arrow icon to submit your request. The model’s response will be streamed in real time, allowing you to see the output as it’s being generated. ![](/img/community_2/ui_inference_4.png) ## Generate Visual Predictions The Clarifai platform allows you to leverage powerful AI models to analyze and understand visual data such as images and videos. By providing visual input, you can prompt a model to analyze the content and generate an output based on its learned visual understanding. Depending on the model type, the output may include [image classifications](https://docs.clarifai.com/create/models/deep-fine-tuning/visual-classifier), [object detections](https://docs.clarifai.com/create/models/deep-fine-tuning/visual-detector), [segmentation masks](https://docs.clarifai.com/create/models/deep-fine-tuning/visual-segmenter), or even AI-generated visuals. :::note The Playground automatically detects the appropriate mode based on the selected model — intelligently switching between **Chat** and **Vision** modes to match the model's capabilities. ::: ### Step 1: Get a Model After finding a visual model you want to use, go to its individual page and click the **Open in Playground** button located in the upper-right corner, as illustrated earlier. For this example, we'll use the [general-image-recognition](https://clarifai.com/clarifai/main/models/general-image-recognition) model to identify and classify a variety of concepts in images. ### Step 2: Run Your Inference While on the AI Playground, click the plus (**+**) blue button on the left side. A pop-up will appear, offering the following options for making predictions: - Try your own image or video - Batch predict on app inputs ![](/img/community_2/ui_inference_5.png) #### Try Your Own Images or Videos This option lets you add an input and see its predictions without leaving the Playground screen. If you click the button, a small window will pop up, allowing you to upload your input. ![](/img/community_2/model_predictions-2.png) After uploading the image, the model will analyze it and return a list of concepts identified on the right side of the page. #### Batch Predict on App Inputs This option lets you select an app and a [dataset](https://docs.clarifai.com/create/datasets/). If you click the button, a small window will pop up, allowing you to choose an app and a dataset with the inputs you want to view their predictions. ![](/img/community_2/model_predictions-3.png) After selecting an app and a dataset, click the **Try Inputs** button. You’ll be redirected to the [single Input-Viewer](https://docs.clarifai.com/create/labeling/ui/#single-input-viewer) screen with the default mode set to **Predict**, allowing you to see the predictions on an input based on your selections. ![](/img/community_2/model_predictions-4.png) ## Generate Predictions Within Input-Viewer > The single Input-Viewer is the main page that showcases the details of a single input available in your app. If you click an input listed on the [Inputs-Manager](https://docs.clarifai.com/portal-guide/inputs-manager/) page, you'll be redirected to the viewer page for that input, where you can view and interact with it. ### Step 1: Get a Model To make predictions on an input within the single Input-Viewer, start by switching to predict mode by toggling the **Predict** button located in the upper-left corner of the page. Next, click the **Choose a model or workflow** button in the right sidebar to select the model you want to use. ![ ](/img/compute-orchestration/compute-27.png) In the window that appears, choose your desired model. You can choose your own customized model or workflow, or look for a public one from the Community platform. You can also create your own model or workflow from there. > To select a public model or workflow from the Community, click the **Explore Community Models / Workflows** button. In the pop-up window, use the search bar to find the desired model or workflow. ![ ](/img/compute-orchestration/compute-27-1.png) For this example, let’s choose the Community’s [Qwen2_5-VL-7B-Instruct](https://clarifai.com/qwen/qwen-VL/models/Qwen2_5-VL-7B-Instruct) model, which is a vision-language model that excels in visual recognition tasks. ### Step 2: Run Your Inference Next, select a deployment from the **Deployment** dropdown. For this example, we’re using the default deployment settings (**Clarifai Shared**). If needed, you can also [create a new deployment](https://docs.clarifai.com/compute/deployments/deploy-model#via-the-ui) from this window. ![ ](/img/compute-orchestration/compute-28.png) Lastly, click the **Predict** button at the bottom of the sidebar to start making the predictions. The model will process the input and return predictions in real time, allowing you to immediately view the results within the Input-Viewer screen. ![ ](/img/compute-orchestration/compute-29.png) :::note For models that output concepts, the **Prediction Threshold** slider allows you to control which predictions are displayed by setting a minimum confidence level. Only predictions with probabilities that meet or exceed this threshold will be shown. You can also use the **Filter by concept** search field to quickly locate specific concepts and view their associated predictions on the page. ::: --- ## LiteLLM # LiteLLM **Run inferences on Clarifai models using LiteLLM** LiteLLM provides a universal interface that simplifies working with LLMs across multiple providers. It offers a single, consistent API for making inferences, allowing you to interact with a wide range of models using the same method, regardless of the underlying provider. LiteLLM natively supports OpenAI-compatible APIs, making it easy to run inferences on [Clarifai-hosted](https://docs.litellm.ai/docs/providers/clarifai) models with minimal setup. :::tip [Click here](https://github.com/Clarifai/examples/tree/main/models/model_predict) for additional examples on how to perform model predictions using various SDKs — such as the Clarifai SDK, OpenAI client, and LiteLLM. The examples demonstrate various model types and include both streaming and non-streaming modes, as well as tool calling capabilities. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Example5 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/litellm_1.py"; import Example6 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/litellm_2.py"; import Example7 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/litellm_3.py"; import Example8 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/litellm_4.py"; import TestOpenai from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/litellm_5.py"; import TestCurl from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/litellm_6.sh"; ## Prerequisites ### Install LiteLLM Install the `litellm` package. pip install litellm ### Get a PAT Key You need a [PAT](https://docs.clarifai.com/control/authentication/pat) key to authenticate your connection to the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. You can then set the PAT as an environment variable using `CLARIFAI_PAT`: export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Get a Clarifai Model Go to the Clarifai [Community](https://clarifai.com/explore) platform and select the model you want to use for predictions. LiteLLM supports all models in the Clarifai community. :::note When using a Clarifai model with LiteLLM, reference it using the `clarifai/`-prefixed model ID in the following format: `clarifai/..`. For example: `clarifai/openai.chat-completion.gpt-oss-20b`. ::: ## Chat Completions In LiteLLM, the [`completion()`](https://docs.litellm.ai/docs/completion) function is the primary method for interacting with language models that follow the OpenAI Chat API format. It supports both traditional completions and chat-based interactions by accepting a list of messages — similar to OpenAI’s [`chat.completions.create()`](https://docs.clarifai.com/compute/inference/open-ai/#chat-completions-api). {Example5} Example Output Hey there! I'm doing great—thanks for asking. How about you? Anything fun or interesting on your mind today? ## Streaming When [streaming](https://docs.litellm.ai/docs/completion/stream) is enabled by setting `stream=True`, the `completion` method returns an iterator that yields partial responses in real time as the model generates them, instead of a single complete dictionary. {Example7} ## Tool Calling Clarifai models accessed via LiteLLM fully support [tool calling](https://docs.clarifai.com/compute/inference/open-ai/#tool-calling), enabling advanced interactions such as function execution during a conversation. {Example6} Tool Calling Implementation Example {Example8} ## Usage with LiteLLM Proxy Here’s how to call Clarifai models through the LiteLLM Proxy Server. ### Install LiteLLM with Proxy Support pip install 'litellm[proxy]' ### Set Key Set your Clarifai PAT as an environment variable, as illustrated [above](#get-a-pat-key). ### Start the Proxy Create a `config.yaml`. ```yaml model_list: - model_name: clarifai-model litellm_params: model: clarifai/openai.chat-completion.gpt-oss-20b api_key: ${CLARIFAI_PAT} ``` Then, start the LiteLLM proxy: ```bash litellm --config /path/to/config.yaml ``` The server will run at: ``` http://0.0.0.0:4000 ``` ### Test the Proxy {TestCurl} {TestOpenai} --- ## MCP Servers # MCP Servers **Extend agentic models inference capabilities with MCP Servers** [Model Context Protocol](https://docs.clarifai.com/compute/agents/mcp) (MCP) servers enable LLMs with agentic capabilities on Clarifai to dynamically discover and invoke external tools during inference. By integrating MCP servers with agentic-enabled models, you can extend model behavior beyond normal uses to include tool usage, data retrieval, and action execution — all within a single inference flow. This allows AI agents on Clarifai to reason, call tools exposed by MCP servers, and iteratively use their outputs to produce more accurate and capable responses. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import OpenAIStreaming from "!!raw-loader!../../../code_snippets/python-sdk/inference/open_ai_streaming.py"; import OpenAINonStreaming from "!!raw-loader!../../../code_snippets/python-sdk/inference/open_ai_non_streaming.py"; import MCPServerFastMCP from "!!raw-loader!../../../code_snippets/python-sdk/inference/mcp_server_fast_mcp.py"; import OutputOpenAIStreaming from "!!raw-loader!../../../code_snippets/python-sdk/inference/output_open_ai_streaming.txt"; ## Prerequisites ### Get an Agentic Model For a model to support agentic behavior through MCP servers on the Clarifai platform, it must extend the standard `OpenAIModelClass` with the `AgenticModelClass`. This extension enables the following capabilities: * Tool discovery and execution managed by the agentic model class * Iterative tool calling within a single predict or generate request * Compatibility with Clarifai SDKs and the OpenAI-compatible API * Support for both streaming and non-streaming inference modes You can see an example implementation of `AgenticModelClass` in [this `1/model.py`](https://github.com/Clarifai/runners-examples/blob/main/llm/agentic-gpt-oss-20b/1/model.py) file. > **Note:** To upload a model with agentic capabilities, simply use `AgenticModelClass`. All other steps and functionality remain the same as when [uploading a standard model](https://docs.clarifai.com/compute/upload/) on Clarifai. You can follow [this example](https://github.com/Clarifai/runners-examples/tree/main/llm/agentic-gpt-oss-20b) to get started. These are some example models with agentic capabilities enabled: * [Qwen3-30B-A3B-Instruct-2507](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-Instruct-2507) * [Qwen3-30B-A3B-Thinking-2507](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-Thinking-2507) * [Qwen3-Coder-30B-A3B-Instruct](https://clarifai.com/qwen/qwenCoder/models/Qwen3-Coder-30B-A3B-Instruct) ### Get an MCP Server The Clarifai platform provides MCP servers that you can use out of the box. Here are some examples: - [Weather Server](https://clarifai.com/clarifai/mcp/models/weather-mcp-server) — Provides weather information - [Browser Server](https://clarifai.com/clarifai/mcp/models/browser-mcp-server) — Enables web browsing capabilities You can also [build your own MCP server](https://docs.clarifai.com/compute/agents/mcp#build-an-mcp-server) or use an [open-source MCP server](https://docs.clarifai.com/compute/agents/deploy-mcp) and deploy it on the Clarifai platform. > **Note:** You can specify multiple MCP servers in the [`mcp_servers`](#chat-completions-streaming) list to give the model access to multiple tool sets. ### Install Packages You need to install the following Python packages: - `clarifai` – Install the latest version of the Clarifai Python SDK package. This also installs the [Command Line Interface (CLI)](https://docs.clarifai.com/additional-resources/api-overview/cli). - `openai` — This leverage Clarifai’s [OpenAI-compatible endpoint](https://docs.clarifai.com/compute/inference/#predict-with-openai-compatible-format) endpoint to run inferences using the OpenAI client library - `fastmcp` — The core framework for interacting with MCP servers. You can run the following command to install them: pip install --upgrade clarifai openai fastmcp ### Get a PAT Key You need a [PAT](https://docs.clarifai.com/control/authentication/pat/#how-to-create-a-pat-on-the-platform) (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. You can then set the PAT as an environment variable using `CLARIFAI_PAT`. This also authenticates your session when using the Clarifai’s CLI. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ## Integrate with LLMs You can pass an MCP server as a tool source to an agentic LLM on Clarifai. The model will automatically discover available tools and call them as needed during completion. ### Chat Completions (Non-Streaming) The [OpenAI Chat Completions](https://platform.openai.com/docs/api-reference/chat) API endpoint lets you produce a model response by providing a list of messages that constitute a conversation. {OpenAINonStreaming} Example Output {OutputOpenAIStreaming} The above snippet demonstrates how to: - Initialize the OpenAI client and point it to Clarifai’s OpenAI-compatible endpoint, instead of OpenAI’s servers. - Specify which MCP servers the model can use. During inference, the model can discover and call these tools autonomously if it decides they’re useful. - Create a chat completion conversation with MCP support: - An agentic model is used to discover tools from MCP servers, execute them, and iterate on tool calls if needed — for example, weather lookup or browsing. - `extra_body={"mcp_servers": mcp_servers}` tells Clarifai which MCP servers to make available to the model for this request. ### Chat Completions (Streaming) You can enable token-by-token streaming by setting `stream=True`. This allows the model’s response to arrive incrementally, providing partial results in real time instead of waiting for the full completion. {OpenAIStreaming} ## Use with FastMCP Client You can call MCP tools directly using the FastMCP client without involving an LLM, giving you full programmatic control over tool execution. {MCPServerFastMCP} --- ## OpenAI # OpenAI **Run inferences on Clarifai models using OpenAI** You can run inferences on Clarifai-hosted models using the OpenAI client library by leveraging the Clarifai’s [OpenAI-compatible API endpoint](README.mdx#predict-with-openai-compatible-format). This allows you to use the same code and tools you would with OpenAI, in either Python or JavaScript, by simply configuring the client to point to Clarifai and providing your PAT (Personal Access Token) key. :::tip [Click here](https://github.com/Clarifai/examples/tree/main/models/model_predict) for additional examples on how to perform model predictions using various SDKs — such as the Clarifai SDK, OpenAI client, and LiteLLM. The examples demonstrate various model types and include both streaming and non-streaming modes, as well as tool calling capabilities. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PyOpenAI from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/openai_1.py"; import NodeOpenAI from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/openai_1.ts"; import PyOpenAIStreaming from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_streaming.py"; import Example2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/openai_2.py"; import TSExample2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/openai_2.ts"; import Example3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/openai_3.py"; import Example4 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/openai_4.txt"; import PyImageGeneration from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_image_generation.py"; import PyEmbeddings from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_embeddings.py"; import PyResponses from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_responses.py"; import PyMultimodal from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_multimodal.py"; import PyResponsesMultimodal from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_responses_multimodal.py"; import PyResponsesStreaming from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_responses_streaming.py"; import PyResponsesToolCalling from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_ai_responses_tool_calling.py"; import ResponsesToolCallingExample from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/responses_tool_calling_1.py"; import ToolCallingOutput1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/tool_calling_output_1.txt"; import CurlOpenAIText from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/curl_openai_text.sh"; import PyStandardJSON from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/py_standard_json.py"; import PyPydanticIntegrated from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/py_pydantic_integrated.py"; ## Prerequisites ### Install OpenAI Package Install the `openai` package. pip install openai npm install openai ### Get a PAT Key You need a [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat) key to authenticate your connection to the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. You can then set the PAT as an environment variable using `CLARIFAI_PAT`: export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Get a Clarifai Model Go to the Clarifai [Community](https://clarifai.com/explore) platform and select the model you want to use for making predictions. Some Clarifai models that support OpenAI https://clarifai.com/openai/chat-completion/models/gpt-oss-120b https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-0528-Qwen3-8B https://clarifai.com/meta/Llama-3/models/Llama-3_2-3B-Instruct https://clarifai.com/anthropic/completion/models/claude-sonnet-4 https://clarifai.com/qwen/qwenLM/models/Qwen3-14B https://clarifai.com/mistralai/completion/models/Devstral-Small-2505_gguf-4bit https://clarifai.com/clarifai/main/models/general-image-recognition https://clarifai.com/xai/chat-completion/models/grok-3 https://clarifai.com/openai/chat-completion/models/gpt-4o https://clarifai.com/openai/chat-completion/models/gpt-4_1 https://clarifai.com/gcp/generate/models/gemini-2_5-flash https://clarifai.com/anthropic/completion/models/claude-3_5-haiku https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-GGUF https://clarifai.com/gcp/generate/models/gemini-2_0-flash https://clarifai.com/gcp/generate/models/gemma-3-12b-it https://clarifai.com/microsoft/text-generation/models/Phi-4-reasoning-plus https://clarifai.com/openbmb/miniCPM/models/MiniCPM3-4B https://clarifai.com/microsoft/text-generation/models/phi-4-mini-instruct https://clarifai.com/qwen/qwen-VL/models/Qwen2_5-VL-7B-Instruct https://clarifai.com/microsoft/text-generation/models/phi-4 https://clarifai.com/xai/chat-completion/models/grok-2-vision-1212 https://clarifai.com/xai/image-generation/models/grok-2-image-1212 https://clarifai.com/xai/chat-completion/models/grok-2-1212 https://clarifai.com/qwen/qwenLM/models/QwQ-32B-AWQ https://clarifai.com/gcp/generate/models/gemini-2_0-flash-lite https://clarifai.com/anthropic/completion/models/claude-opus-4 https://clarifai.com/openai/chat-completion/models/o4-mini https://clarifai.com/openai/chat-completion/models/o3 https://clarifai.com/openbmb/miniCPM/models/MiniCPM-o-2_6-language https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-7B https://clarifai.com/qwen/qwenCoder/models/Qwen2_5-Coder-7B-Instruct ## Responses API The [OpenAI Responses](https://platform.openai.com/docs/api-reference/responses) API endpoint provides a powerful interface for generating model responses, allowing you to leverage OpenAI's most advanced capabilities. It’s highly versatile, supporting both text and image inputs and producing text outputs, as well as advanced features like streaming and tool calling. ### Text Inputs {PyResponses} Example Output Under a silver‑shimmering moon, a gentle unicorn named Lira tiptoed into the sleepy meadow, her horn casting soft, glittering lullabies over the swaying wildflowers. As the night wind whispered sweet dreams, she gathered a handful of moon‑kissed dew and sprinkled it over the slumbering forest creatures, coax {PyResponsesMultimodal} Example Output ### Streaming The OpenAI Responses API supports supports streaming, which means you get the response one token at a time instead of waiting for the full completion. {PyResponsesStreaming} Example Output ### Tool Calling The OpenAI Responses API supports supports tool calling (also known as function calling), allowing you to enable large language models (LLMs) to interact with external tools. Based on user input, the LLM can autonomously decide when and how to invoke functions — such as fetching real-time data from an API or running custom code. The following example demonstrates a simple tool-calling interaction. It simulates a weather API and shows how the model automatically “calls” that tool when asked about the weather. {PyResponsesToolCalling} Example Output Tool Calling Implementation Example {ResponsesToolCallingExample} --> ## Chat Completions API The [OpenAI Chat Completions](https://platform.openai.com/docs/api-reference/chat) API endpoint enables you to generate a model response by providing a list of messages that constitute a conversation. :::note Supported Message Roles Clarifai's LLM endpoint supports the following message roles: `system`, `user`, `assistant`, `developer`, and `tool`. The `developer` role is used for developer-level instructions (similar to `system`), while `tool` is used for passing back the results of tool calls during agentic interactions. ::: ### Text Inputs {PyOpenAI} {NodeOpenAI} {CurlOpenAIText} Example Output I’m ChatGPT, an AI language model created by OpenAI. I’ve been trained on a wide range of text so I can help answer questions, brainstorm ideas, explain concepts, and assist with many other tasks. Think of me as a virtual assistant you can chat with— ### Multimodal Inputs The OpenAI Chat Completions API supports multimodal inputs. {PyMultimodal} Example Output Response: The image shows a ginger tabby cat lying down on a stone surface. The cat has a striped pattern on its fur and is looking directly at the camera with alert, bright eyes. The background features a textured wall, and the lighting highlights the cat's features, giving a warm and cozy atmosphere. ### Streaming The OpenAI Chat Completions API supports streaming. {PyOpenAIStreaming} Example Output Assistant's Response: I’m ChatGPT, an AI language model created by OpenAI. I’ve been trained on a wide range of text data so I can help answer questions, explain concepts, brainstorm ideas, draft writing, solve problems, and more. Think of me as a virtual assistant that can converse with you on many topics— ### Tool Calling The OpenAI Chat Completions API supports tool calling (also known as function calling). Here is an example code that sets up a basic tool-calling interaction. It simulates a weather API and shows how the LLM would "call" that tool when asked about the weather. {Example2} {TSExample2} Example Output {ToolCallingOutput1} Tool Calling Implementation Example {Example3} {Example4} ### Structured Outputs The Chat Completions API supports generating structured JSON outputs from any [supported](#get-a-clarifai-model) OpenAI-compatible large language model. This feature lets you go beyond plain text generation by enforcing that the model’s output follows a strict, predictable data structure. Structured responses make it easier to integrate model outputs into downstream applications with greater safety, consistency, and reliability. To define, validate, and serialize these structured responses, [Pydantic](https://docs.pydantic.dev/latest/) is used — a powerful data modeling library for Python. You can install Pydantic with: ```bash pip install pydantic ``` We support two main ways of generating the structured JSON responses: - Standard JSON schema extraction - Pydantic-integrated extraction. #### Standard JSON Schema Extraction This technique calls the chat completion endpoint and explicitly requests the model to return its response in a [JSON](https://json-schema.org/) format that conforms to a specified Pydantic schema. {PyStandardJSON} Example Output ```text {"age": 29, "email": "john.miller@example.com", "name": "John Miller"} ``` #### Pydantic-Integrated Extraction This technique uses a convenience method (`client.chat.completions.parse`) that handles the structured output request and attempts to automatically parse the JSON response into a Pydantic object, simplifying the extraction process. {PyPydanticIntegrated} Example Output ```text name='John Miller' age=29 email='john.miller@example.com' ``` ## Images Generate API The [OpenAI Images Generate](https://platform.openai.com/docs/api-reference/images) API endpoint enables you to generate an image by providing a prompt. Here is an example of how to generate an image using a model that supports Clarifai's OpenAI-compatible API endpoint. {PyImageGeneration} Example Output ```text ImagesResponse(created=None, data=[Image(b64_json=None, revised_prompt='A high-resolution photograph of a cat perched on a branch in a lush, green tree during the daytime. The cat, possibly a tabby, is the central focus of the image, looking slightly to the side with its fur naturally positioned. The background features a soft, slightly blurred forest setting with sunlight filtering through the leaves, creating a serene and natural environment. The composition avoids any distracting elements, ensuring the cat remains the primary subject in a peaceful outdoor scene.', url='https://imgen.x.ai/xai-imgen/xai-tmp-imgen-41202340-c0e1-4669-bed5-e70f7b491176.jpeg')], usage=None) ``` ## Embeddings API The [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings) API allows you to convert text into numerical vector representations (embeddings) that capture semantic meaning. These embeddings can then be used to measure the similarity or relatedness between different pieces of text. Here’s an example of how to generate embeddings using a model that supports Clarifai’s OpenAI-compatible API endpoint. {PyEmbeddings} Example Output ```text [0.007715723942965269, -0.005610561463981867, 0.0038386366795748472, -0.023672660812735558, -0.0028357207775115967, 0.0058668977580964565, 0.005136339459568262, -0.029632480815052986, 0.009997117333114147, -0.027863759547472, 0.016123555600643158, 0.010894294828176498, -0.0065814354456961155, -0.03468230739235878, -0.019545646384358406, 0.003260277910158038, 0.032759785652160645, -0.006728828884661198, 0.00939472671598196, -0.020865777507424355, -0.02409561537206173, 0.01959691382944584, 0.0123361861333251, -0.003186581190675497, -0.007440162356942892, 0.016213273629546165, 0.0012400270206853747, -0.017956361174583435, 0.009266559034585953, 0.017943544313311577, 0.0027379924431443214, -0.005963024217635393, -0.03734820336103439, 0.001370598329231143, -0.014213849790394306, -0.017328336834907532, -0.019648181274533272, 0.005863693542778492, 0.003191387513652444, 0.004690954927355051, 0.003135313745588064, -0.014265117235481739, 0.012977027334272861, 0.008625717833638191, -0.04129578545689583, 0.012047807686030865, 0.007613189052790403, 0.0061424593441188335, -0.01573905162513256, -0.019276492297649384, 0.011778654530644417, -0.01596975326538086, -0.011560768820345402, -0.013201321475207806, 0.0214809849858284, 0.017174534499645233, -0.007286360487341881, -0.009708738885819912, 0.02837643213570118, -0.015290462411940098, -0.009221700020134449, 0.02190393954515457, -0.021827038377523422, -0.018379315733909607, -0.019789164885878563, -0.01197731588035822, -0.012836041860282421, 0.00048784009413793683, 0.0025665676221251488, 0.0067160120233893394, 0.0037296938244253397, 0.011150631122291088, 0.012803999707102776, -0.01639270968735218, 0.018968889489769936, -0.012880900874733925, -0.008503957651555538, 0.00356307509355247, -0.004127014894038439, . . .] ``` --- ## OpenHands # OpenHands **Run inference with Clarifai models locally using OpenHands** [OpenHands](https://docs.all-hands.dev/) is an AI-powered code assistance framework designed to streamline development by harnessing the flexibility of large language models (LLMs). Let's walk through how to run inference with Clarifai models locally using OpenHands, while leveraging its intelligent assistance to boost your productivity. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import OpenHands1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_hands_1.py"; import OpenHands2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/open_hands_2.txt"; ## Step 1: Perform Prerequisites - **Get a PAT (Personal Access Token) key** — To use OpenHands with Clarifai models, you need a [PAT](https://docs.clarifai.com/control/authentication/pat) for authentication. [Log in](https://clarifai.com/login) or [sign up](https://clarifai.com/signup) for a Clarifai account, then go to **Settings** in the collapsible left sidebar and select **Secrets**. From there, you can create a new token or copy an existing one to use with your requests. - **Get a model** — Clarifai's [Community platform](https://clarifai.com/explore) offers a wide selection of cutting-edge language models that you can run using OpenHands. Browse the platform to find a model that best fits your use case. For this example, we'll use the [GPT-OSS-120B](https://clarifai.com/openai/chat-completion/models/gpt-oss-120b) model. - **Install Docker Desktop** — OpenHands runs inside a Docker container, so you’ll need Docker installed and running on your system. You can download and install Docker Desktop for your operating system from the [official Docker website](https://www.docker.com/products/docker-desktop). Be sure to follow the installation steps specific to your OS (Windows, macOS, or Linux). - **Get a GitHub token** — If you want to integrate OpenHands with GitHub, you’ll need a personal access token for authentication. You can follow [GitHub’s documentation](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to generate one. ## Step 2: Run OpenHands Locally ### Option A: CLI Launcher (Recommended) The simplest way to run OpenHands locally is by using the CLI launcher with the [`uv`](https://docs.astral.sh/uv/) package manager. This approach provides better isolation from your existing project’s virtual environment and is needed for OpenHands’ default MCP servers. If you don’t already have `uv` installed, follow the official [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/) for the latest instructions for your platform. Then, use either of the following commands to launch OpenHands. Launch by starting the GUI server: ```bash uvx --python 3.12 openhands serve ``` Or, start in CLI mode: ```bash uvx --python 3.12 openhands ``` Example Output {OpenHands2} ### Option B: Docker You can also run OpenHands using Docker. Note that OpenHands uses a dedicated Docker image to provide a sandboxed execution environment. Pulling this image ahead of time from the All-Hands AI Docker registry can speed up the initial startup process. > **Note:** Always check the [official OpenHands documentation](https://github.com/OpenHands/OpenHands?tab=readme-ov-file#option-2-docker) for the latest recommended image tag, as it may change over time. docker pull docker.openhands.dev/openhands/runtime:0.62-nikolaik Next, start OpenHands using the following comprehensive `docker run` command. > **Note:** This command works best in Linux or macOS environments, where Docker supports native Linux-style paths. If you're using Windows, you'll need to modify the volume mount paths and run the command within a WSL2 (Linux) shell for proper compatibility. {OpenHands1} The above command: * Launches a new Docker container to run OpenHands. * Sets environment variables for the runtime image and logging. * Grants the container access to the Docker engine for sandboxing. * Maps port 3000 on your computer host to port 3000 in the container. _This is how you access the OpenHands web interface or API locally._ * Mounts the `~/.openhands` folder for persistent user data. * Enables the container to communicate with the host machine. * Assigns the container the name `openhands-app`. * Uses the OpenHands `0.62` image from the official registry. * Automatically deletes the container after it exits. ### Run OpenHands After launching OpenHands, monitor the terminal for log output. Once the application finishes its startup process, open your preferred web browser and go to: ``` http://localhost:3000 ``` At this point, OpenHands is successfully installed and running on your local machine. ![](/img/new-docs/openhands_1.png) ## Step 3: Connect to an LLM To configure OpenHands to use a Clarifai model as its cognitive engine, click the settings icon in the bottom-left corner of the sidebar. In the menu that appears, select **LLM** to open the LLM configuration panel. Here, you can set up the large language model that OpenHands will use for reasoning and task execution. ![](/img/new-docs/openhands_2.png) Fill in the following fields: * **LLM Provider** — From the list of providers, select `Clarifai`. * **LLM Model** — Choose the Clarifai model you want OpenHands to use. * **API Key** — Enter your Clarifai PAT as described [earlier](#step-1-perform-prerequisites). After filling in the fields, click the **Save Changes** button in the bottom-right corner of the interface. :::info Advanced LLM Settings To access [advanced](https://docs.openhands.dev/openhands/usage/settings/llm-settings#advanced-llm-settings) configuration options, toggle the **Advanced** switch in the LLM configuration panel. ![](/img/new-docs/openhands_6.png) This allows you to specify a custom Clarifai model and adjust additional LLM options. This is helpful when the model you want to use is not available in the basic dropdown selection. Then, fill in the following fields: * **Custom Model** — Enter the Clarifai model URL you want to use. To ensure OpenAI compatibility, prefix the model path with `openai/`, followed by the full Clarifai model URL. Here is an example: `openai/https://clarifai.com/openai/chat-completion/models/gpt-oss-120b`. * **Base URL** — Enter Clarifai's [OpenAI-compatible](https://docs.clarifai.com/compute/inference/#predict-with-openai-compatible-format) API endpoint: `https://api.clarifai.com/v2/ext/openai/v1`. * **API Key** — Enter your PAT, as explained [earlier](#step-1-perform-prerequisites). > **Note:** You can leave all other fields at their default values, including the selected agent, unless you require custom behavior. After filling in the fields, click the **Save Changes** button at the bottom-right corner of the interface. ::: ## Step 4: Integrate with GitHub Integrating OpenHands with GitHub allows you to take advantage of version control, repository access, and collaboration features. To set this up, open the **Settings** panel and navigate to the **Integrations** tab. Then, enter your GitHub personal access token in the designated field. > **Optionally:** If you are using a self-hosted or enterprise instance, you may also provide your GitHub host URL. ![](/img/new-docs/openhands_3.png) > **Note:** You can also integrate with GitLab or Bitbucket by entering the corresponding token and host details. Once everything is filled in, click **Save Changes** in the bottom-right corner to apply your integration settings. ## Step 5: Start a Conversation Next, click the plus (**+**) **Start new conversation** button at the top of the sidebar. From there, connect to a repository by selecting the desired repo and its branch. Once selected, click the **Launch** button to begin your session. ![](/img/new-docs/openhands_4.png) > **Tip:** You can also start a new conversation from scratch that is not connected to an existing repository. ## Step 6: Start Building In the main interface, use the input field to prompt the agent and begin working on your task. You can continue interacting with the agent to refine the results, extend the functionality, or troubleshoot issues as they arise. ![](/img/new-docs/openhands_5.png) OpenHands sends each request to the large language model you configured — powered by Clarifai — and the model generates the responses. Once you're satisfied with the output, you can seamlessly push your changes to GitHub directly from the interface, making it easy to commit, publish, and share your work. That's it! --- ## Request Routing # Request Routing **How Clarifai routes prediction requests for optimal performance** When you send a prediction request to a deployed model, Clarifai's routing system determines where that request is processed. Routing happens at two levels: - First selecting the right nodepool (which hardware); - Then, selecting the right replica (which pod within that hardware). This page covers both levels, what Clarifai optimizes automatically, and what you can configure. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PredictionCaching from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/prediction_caching.txt"; import KVCaching from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/kv_caching.txt"; ## Nodepool Selection When a request arrives, Clarifai picks a nodepool in this order: 1. **Explicit routing** — If you specify a `deployment_id` (or `compute_cluster_id` + `nodepool_id`) in the request, the request goes directly to that nodepool. 2. **Deployment lookup** — If you don't specify routing, Clarifai finds active deployments for the model version and picks the best nodepool based on queue depth and latency. 3. **Shared compute fallback** — For Clarifai-owned models with no user deployment, the request falls back to Clarifai's shared infrastructure. ### Multi-Nodepool Deployments A single deployment can span multiple nodepools with priority ordering. When you add nodepools to a deployment, you assign each a priority rank (1 being highest). Requests are routed to the highest-priority nodepool that has available capacity, with overflow spilling to the next. > **Note:** You can configure multi-nodepool deployments in the [UI](https://docs.clarifai.com/compute/deployments/deploy-model#step-4-set-dynamic-routing-optional) by dragging nodepools into your preferred priority order, or via the [API](https://docs.clarifai.com/compute/deployments/clusters-nodepools#multi-nodepool-deployment) using `deployment_nodepools` with `sequence` values. This enables several powerful patterns: - **Cross-cloud failover** — Place nodepools in different cloud providers (AWS, GCP) so traffic fails over automatically if one region has issues. - **Cross-hardware routing** — Since Clarifai's Compute Orchestration supports any hardware, you can route between NVIDIA and AMD GPUs, or even between GPU and CPU nodepools, within a single deployment. This means you can mix x86 and ARM architectures, or combine spot and on-demand instances for cost optimization. - **Tiered performance** — Use a high-performance GPU nodepool as primary and a cheaper instance as overflow. ### Explicit Request-Level Routing You can override the default routing behavior for individual requests by specifying a particular [deployment ID or nodepool](https://docs.clarifai.com/compute/inference/clarifai/api#set-up-dedicated-deployment). ```python # Route to a specific deployment response = model.predict(inputs, deployment_id="my-deployment") # Route to a specific nodepool response = model.predict(inputs, compute_cluster_id="my-cluster", nodepool_id="my-nodepool") ``` ## Automatic Replica Optimization Once a nodepool is selected, Clarifai intelligently routes the request to the replica most likely to deliver the fastest response. This is fully automatic — you don't configure it, but understanding what it does helps you design better deployments. ### KV Cache Routing (Prefix Cache Routing) For models backed by inference engines with KV caching (such as vLLM and SGLang), Clarifai detects shared prompt prefixes and routes requests to replicas that already hold those prefixes in their KV cache. This avoids redundant recomputation of attention keys and values, resulting in higher throughput and lower time-to-first-token (TTFT). > **Note:** This is fully automatic — no configuration or code changes required. This is especially effective for: - **Shared system prompts** — Multiple users hitting the same model with the same system instruction share the same prefix. Rather than recomputing it on every replica, requests are routed to replicas that already have that prefix cached. - **RAG pipelines** — Users querying the same knowledge base receive similar retrieved-document prefixes, which are automatically reused across requests. - **Multi-turn conversations** — Follow-up messages share the prior conversation as a prefix, so subsequent turns reuse cached state from earlier turns. When a matching prefix is found, the request routes to the best-matched replica. When no prefix match exists, the system falls back gracefully — there is no performance penalty compared to routing without this feature. Example Output {KVCaching} ### Session Awareness Clarifai tracks which replicas have recently served each user and favors routing subsequent requests from the same user to the same replica. This improves cache reuse for conversational workloads without any client-side session management. KV cache routing (prefix cache routing) and session awareness work together in a priority chain: **prefix cache match → session affinity → random selection**. Prefix cache routing handles shared prefixes *across* users (e.g., shared system prompts), while session awareness handles reuse *within* a single user's conversation. If neither signal applies, the system falls back to random selection with no degradation. ## Autoscaling Autoscaling controls how many replicas are running within a nodepool. You configure this per-nodepool within a deployment: | Setting | Default | Description | |---|---|---| | **Min Replicas** | `0` | Minimum instances always running. Set `≥ 1` to avoid cold starts | | **Max Replicas** | `5` | Ceiling for autoscaling | | **Scale Up Delay** | `300s` | Wait before adding replicas after traffic increases | | **Scale Down Delay** | `300s` | Wait before removing replicas after traffic drops | | **Scale to Zero Delay** | `1800s` | Idle time before scaling to zero (only when min replicas = 0) | | **Traffic History** | `300s` | How far back to look at traffic data for scaling decisions | | **Disable Packing** | `false` | When `true`, restricts to one replica per node for isolation | > **Note:** You can configure autoscaling in the [UI](https://docs.clarifai.com/compute/deployments/clusters-nodepools/#step-3-set-node-autoscaling-range) during deployment, via the [Python SDK](https://docs.clarifai.com/compute/deployments/clusters-nodepools/#3-deployment_configyaml), or via the [CLI](https://docs.clarifai.com/compute/deployments/deploy-model#autoscaling). For advanced autoscaling settings (scale-to-zero delays, traffic history, packing), use the UI deployment flow or the API. ```bash clarifai model deploy ./my-model --min-replicas 1 --max-replicas 10 ``` :::tip Cold Start Reduction Clarifai pre-warms popular instance types automatically so new replicas are ready faster when traffic increases or spills across nodepools. For specific GPU types not covered by default, set `min_replicas ≥ 1` on your deployment to keep your preferred hardware warm and eliminate cold starts entirely. ::: More replicas means more capacity and better cache distribution across your deployment. :::info Cache and Scaling When new replicas scale up, they start with an empty [KV cache](#kv-cache-routing-prefix-cache-routing) — prefix cache routing effectiveness improves as replicas warm up over time. When replicas scale down, cached state on those replicas is lost. For latency-sensitive workloads, setting `min_replicas ≥ 1` ensures you always have warm replicas with populated caches. ::: ## Prediction Caching Prediction caching stores and reuses previously computed model outputs for identical requests, eliminating redundant inference and significantly reducing latency and compute costs. For repeated requests with the same input, model, and model version, you can bypass inference entirely by enabling prediction caching with the `use_predict_cache` parameter. ```curl curl -X POST "https://api.clarifai.com/v2/users/moonshotai/apps/chat-completion/models/Kimi-K2_6/outputs" \ -H "Authorization: Key YOUR_PAT_HERE" \ -H "Content-Type: application/json" \ -d '{ "inputs": [ { "data": { "text": { "raw": "How do I check if a Python object is an instance of a class?" } } } ], "use_predict_cache": true }' ``` Example Output {PredictionCaching} When enabled, Clarifai caches the full prediction response for identical `input + model + version` combinations. Subsequent matching requests are served directly from cache without invoking model compute. > **Note:** Prediction caching is different from KV cache routing (prefix cache routing). > > * Prediction caching skips inference entirely by returning a previously computed response. > * Prefix cache routing still performs inference, but accelerates it by reusing cached prompt state. ## Inference Cost Savings and Cache Hits For requests using KV cache routing, if a request is routed to a replica that already contains cached prompt state, the cached portion of the prompt is billed at a **90% discount** (10% of standard prompt token pricing). The Clarifai Python SDK automatically extracts `cached_tokens` from the model response metadata and forwards it to the output proto. If `cached_tokens` is absent or zero, the field is omitted from the output. You can therefore treat the presence of `cached_tokens` as a reliable indicator that a cache hit occurred. If you're unsure whether a specific model reports cached tokens, [contact us](https://www.clarifai.com/company/contact) for confirmation. --- ## Vercel AI SDK # Vercel AI SDK **Run inferences on Clarifai models using Vercel** Vercel offers a TypeScript toolkit called the [AI SDK](https://vercel.com/docs/ai-sdk), which streamlines integration with language models in modern web applications. The SDK supports the [OpenAI Compatible Provider](https://ai-sdk.dev/providers/openai-compatible-providers) package that enables seamless interaction with any OpenAI-compatible API — including [Clarifai’s](https://ai-sdk.dev/providers/openai-compatible-providers/clarifai) OpenAI-compatible endpoint. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import VercelExample1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/vercel_1.ts"; import Output1 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/vercel_output_1.txt"; import VercelExample2 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/vercel_2.ts"; import VercelExample3 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/vercel_3.ts"; import VercelExample4 from "!!raw-loader!../../../code_snippets/python-sdk/compute-orchestration/vercel_4.ts"; ## Prerequisites ### Install Package Vercel offers a Clarifai provider, which is available through the `@ai-sdk/openai-compatible` module and is compatible with the OpenAI API. You can install it using: pnpm add @ai-sdk/openai-compatible npm install @ai-sdk/openai-compatible yarn add @ai-sdk/openai-compatible ### Get a PAT Key You need a [PAT](https://docs.clarifai.com/control/authentication/pat) (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. You can then set the PAT as an environment variable using `CLARIFAI_PAT`: export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Get a Clarifai Model Go to the Clarifai [Community](https://clarifai.com/explore) platform and select the model you want to use for making predictions. Some Clarifai models that support Vercel AI SDK, and their capabilities | Model | Image Input | Tool Usage | Tool Streaming | | --- | --- | --- | --- | | [gpt-oss-120b](https://clarifai.com/openai/chat-completion/models/gpt-oss-120b) | ✅ | ✅ | ✅ | | [DeepSeek R1 0528 Qwen3 8B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-0528-Qwen3-8B) | ✅ | ✅ | ✅ | | [Llama 3.2 3B Instruct](https://clarifai.com/meta/Llama-3/models/Llama-3_2-3B-Instruct) | ✅ | ✅ | ✅ | | [claude Sonnet 4](https://clarifai.com/anthropic/completion/models/claude-sonnet-4) | ✅ | ✅ | ✅ | | [Qwen3 14B](https://clarifai.com/qwen/qwenLM/models/Qwen3-14B) | ✅ | ✅ | ✅ | | [Devstral Small 2505.gguf 4bit](https://clarifai.com/mistralai/completion/models/Devstral-Small-2505_gguf-4bit) | ✅ | ✅ | ✅ | | [grok 3](https://clarifai.com/xai/chat-completion/models/grok-3) | ❌ | ✅ | ✅ | | [gpt 4o](https://clarifai.com/openai/chat-completion/models/gpt-4o) | ✅ | ✅ | ✅ | | [gpt 4.1](https://clarifai.com/openai/chat-completion/models/gpt-4_1) | ✅ | ✅ | ✅ | | [gemini 2.5 Flash](https://clarifai.com/gcp/generate/models/gemini-2_5-flash) | ✅ | ✅ | ✅ | | [claude 3.5 Haiku](https://clarifai.com/anthropic/completion/models/claude-3_5-haiku) | ✅ | ✅ | ✅ | | [Qwen3 30B A3B GGUF](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-GGUF) | ✅ | ✅ | ✅ | | [gemini 2.0 Flash](https://clarifai.com/gcp/generate/models/gemini-2_0-flash) | ✅ | ✅ | ✅ | | [gemma 3 12b It](https://clarifai.com/gcp/generate/models/gemma-3-12b-it) | ✅ | ✅ | ✅ | | [Phi 4 Reasoning Plus](https://clarifai.com/microsoft/text-generation/models/Phi-4-reasoning-plus) | ✅ | ✅ | ✅ | | [phi 4 Mini Instruct](https://clarifai.com/microsoft/text-generation/models/phi-4-mini-instruct) | ✅ | ✅ | ✅ | | [Qwen2.5 VL 7B Instruct](https://clarifai.com/qwen/qwen-VL/models/Qwen2_5-VL-7B-Instruct) | ✅ | ❌ | ❌ | | [phi 4](https://clarifai.com/microsoft/text-generation/models/phi-4) | ✅ | ✅ | ✅ | | [grok 2 Vision 1212](https://clarifai.com/xai/chat-completion/models/grok-2-vision-1212) | ✅ | ✅ | ✅ | | [grok 2 1212](https://clarifai.com/xai/chat-completion/models/grok-2-1212) | ❌ | ✅ | ✅ | | [QwQ 32B AWQ](https://clarifai.com/qwen/qwenLM/models/QwQ-32B-AWQ) | ✅ | ✅ | ✅ | | [gemini 2.0 Flash Lite](https://clarifai.com/gcp/generate/models/gemini-2_0-flash-lite) | ✅ | ✅ | ✅ | | [claude Opus 4](https://clarifai.com/anthropic/completion/models/claude-opus-4) | ✅ | ✅ | ✅ | | [o4 Mini](https://clarifai.com/openai/chat-completion/models/o4-mini) | ✅ | ✅ | ✅ | | [o3](https://clarifai.com/openai/chat-completion/models/o3) | ✅ | ✅ | ✅ | | [MiniCPM-o 2.6 Language](https://clarifai.com/openbmb/miniCPM/models/MiniCPM-o-2_6-language) | ✅ | ❌ | ❌ | | [DeepSeek R1 Distill Qwen 7B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-7B) | ✅ | ❌ | ❌ | | [Qwen2.5 Coder 7B Instruct](https://clarifai.com/qwen/qwenCoder/models/Qwen2_5-Coder-7B-Instruct) | ✅ | ✅ | ✅ | ### Create Provider Instance To use Clarifai with Vercel for inference, create a custom provider instance using the `createOpenAICompatible` function from `@ai-sdk/openai-compatible`. {VercelExample4} ## Generating Text Here's an example of how you can generate text using the Vercel AI SDK with a Clarifai model. {VercelExample1} Example Output {Output1} ## Streaming You can use the Vercel AI SDK to stream responses with a Clarifai model. {VercelExample2} ## Tool Calling The Vercel AI SDK supports [tool calling](https://docs.clarifai.com/compute/models/inference/api#tool-calling). > **Note:** To run this example, install the Zod validation library: `npm install zod`. {VercelExample3} --- ## Local Runners # Local Runners **Run and test models locally, while exposing them publicly through Clarifai’s API** [Local Runners](https://www.clarifai.com/products/local-runners) are a powerful feature that let you securely expose your locally running models or servers via a public URL, allowing you to quickly develop, test, and share any models running on your own hardware. Instead of running solely in the cloud, you can run your models anywhere that supports Python and has an internet connection — whether it's your workstation or on-premise server. With Local Runners, you can connect your own models to Clarifai's compute plane. This seamless integration enables you to leverage the Clarifai cloud API, workflows, and other platform capabilities with your custom models. Your model can securely receive and process requests from anywhere, just as it would in a production cloud deployment. > **Note:** A runner is the actual running instance of your model import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import LocalDev from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/local-dev.txt"; import LocalDevExampleCode from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/local-dev-example-code.py"; ![ ](/img/others-2/local-runners.png) ## Prerequisites ### Sign Up or Log In [Log in to](https://clarifai.com/login) your existing Clarifai account or [sign up](https://clarifai.com/signup) for a new one. Next, retrieve the following credentials required for setup: - **User ID** – In the collapsible left sidebar, select **Settings** and choose **Account** from the dropdown list. Then, locate your user ID. - **Personal Access Token (PAT)** – From the same **Settings** option, choose **Secrets** to generate or copy your [PAT](https://docs.clarifai.com/control/authentication/pat). This token is used to authenticate your connection with the Clarifai platform. ### Install Clarifai CLI Install the latest version of the [Clarifai CLI](https://docs.clarifai.com/sdk/cli) tool. It includes built-in support for Local Runners. pip install --upgrade clarifai > **Note:** You'll need **[Python 3.11 or 3.12](https://docs.clarifai.com/resources/api-overview/python-sdk#python-requirements)** installed to successfully run the Local Runners. --- :::note ## Quick Start Once you've completed the prerequisites above, run the following commands and follow the prompts in your terminal to quickly get started with Local Runners. ### Log in to Clarifai Connect your environment to the Clarifai platform and [create a context](#step-2-create-a-context-optional) profile. clarifai login ### Set up a Model Scaffold a sample [model](#step-1-build-a-model) with the necessary files. You can start from a blank Python template, or pick a built-in toolkit pre-configured for your inference engine: {`# Local LLM servers (most common for local runners): clarifai model init --toolkit ollama --model-name llama3.1 clarifai model init --toolkit lmstudio --model-name # GPU LLM serving (if running on a local GPU): clarifai model init --toolkit vllm --model-name Qwen/Qwen3-0.6B clarifai model init --toolkit sglang --model-name meta-llama/Llama-3-8B clarifai model init --toolkit huggingface --model-name bert-base-uncased # Other: clarifai model init --toolkit mcp my-mcp-server # MCP tool server (FastMCP) clarifai model init --toolkit openai my-openai-wrapper # OpenAI-compatible wrapper # Blank Python template: clarifai model init`} Each toolkit pre-fills `model.py` with a working `ModelClass` implementation, `config.yaml` with reasonable defaults, and `requirements.txt`. After scaffolding, the next step is to serve the model locally. ### Serve Your Model Locally Start the model using `clarifai model serve`. The CLI will guide you through creating any required platform resources (compute clusters, nodepools, and deployments) — described [below](#table). clarifai model serve > **Note:** The `local-runner` command is still supported as an alias for `serve`. Once your runner launches successfully, your model will be running and accessible via a public URL. You can then open a new terminal, copy the sample code provided in the output, and [test your model](#step-4-test-with-snippet)! ::: --- ## Use Cases for Local Runners - **Streamlined model development** — Local Runners make it easy to build and test new models directly within your local environment. You can spin up a runner on your machine, route API calls through our public cloud endpoint, and watch requests hit your model in real time. This allows you to debug, set breakpoints, return results, and validate outputs. - **Leverage your own compute resources** — If you have powerful hardware, you can take advantage of that local compute without relying on Clarifai's [autoscaling](https://docs.clarifai.com/compute/deployments/clusters-nodepools/#step-3-set-node-autoscaling-range) infrastructure. Your model remains accessible through our API with full authentication, even though it runs locally. - **Locally connect agents** — Because Local Runners execute on your chosen hardware, they can interact with local file systems, make OS-level calls, or access private data stores. With our [MCP (Model Context Protocol)](https://docs.clarifai.com/compute/agents/mcp) model type, you can give your cloud-hosted agents or any MCP-enabled clients authenticated access to your locally controlled information, regardless of their deployment location. - **Run models anywhere** — Whether on a [local development machine](https://www.clarifai.com/blog/how-to-run-ai-models-locally-2025-tools-setup-tips), an on-premises server, or a private cloud cluster, Local Runners seamlessly connect your models to our platform. This enables you to keep sensitive data and custom-built models securely within your own environment. ## Step 1: Build a Model Start by building the model you want to run using Local Runners. You can either create a custom model from scratch or leverage pre-trained models from external sources like Hugging Face. If you're building your own model, follow our comprehensive [step-by-step](https://docs.clarifai.com/compute/models/model-upload/) guide to get started. You can also explore our [examples repository](https://github.com/Clarifai/runners-examples) to see models built for compatibility with the Clarifai platform. :::note tip You can automatically generate a default model by running the [`clarifai model init`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-init) command in the terminal from your current directory. After the model's files are created, you can modify them as needed or go with the default options. ::: ## Step 2: Create a Context (Optional) Running the local development runner relies on certain variables defined in your current context. The _context_ refers to the active environment settings that determine how your commands interact with the Clarifai platform. > **Note:** You can create this context using the provided default values when you run `clarifai login` and `local-runner` commands. Any configurations you create locally — such as the computer cluster and app — will also be created on the Clarifai platform, making them reusable whenever you test your model with the local development runner. :::tip [Click here](https://docs.clarifai.com/resources/api-overview/cli) to learn how to create and manage various aspects of your Clarifai context, including switching contexts and editing your configuration information. ::: These are the variables required to create a runner: | Variable | Description | | ----------------------------- | ----------------------------------------------------------------------- | | CLARIFAI_PAT | Personal Access Token (PAT) for authentication | | CLARIFAI_USER_ID (`user_id`) | User ID of the account owning the model | | CLARIFAI_APP_ID (`app_id`) | App ID containing the model | | CLARIFAI_MODEL_ID (`model_id`) | The model ID for the model to be run locally | | CLARIFAI_COMPUTE_CLUSTER_ID (`compute_cluster_id`) | [Compute cluster](https://docs.clarifai.com/compute/deployments/clusters-nodepools) where the Local Runner will reside. _Note that the `user_id` of the compute cluster must match the `user_id` of the model._ | | CLARIFAI_NODEPOOL_ID (`nodepool_id`) | Nodepool within the compute cluster | | CLARIFAI_DEPLOYMENT (`deployment_id`) | [Deployment](https://docs.clarifai.com/compute/deployments/deploy-model) for a model into the cluster and nodepool | | CLARIFAI_RUNNER_ID (`runner_id`) | Auto-generated unique runner ID, created by the API and stored in the context | :::note important The Clarifai CLI follows a defined order of precedence when launching Local Runners. If a value is set as an environment variable in your terminal, it takes priority and is used first. If no corresponding environment variable is found, the CLI automatically falls back to the value defined in the active context. For example, you can override the model ID by setting it as an environment variable before starting the Local Runner: ```bash export CLARIFAI_MODEL_ID=new-model-id ``` In this case, the `CLARIFAI_MODEL_ID` from your terminal session will take precedence over the one stored in your active context. ::: ## Step 3: Run Your Model To run your model with the local development runner, navigate to the directory where your custom model is located. Then, follow these steps. ### Log in to Clarifai Run the following command to log in to the Clarifai platform and establish a connection. clarifai login After running the `login` command, you'll be prompted to enter the following details to authenticate your connection: context name (default: "default"): user id: personal access token value (default: "ENVVAR" to get our env var rather than config): - **Context name** — You can provide a custom name for your Clarifai configuration context, or simply press Enter to use the default name, "default". This helps you manage different configurations if needed. - **User ID** — Enter your Clarifai user ID. - **PAT** — Enter your Clarifai [PAT](#get-a-pat-key-and-user-id). Note that if you press Enter, and you have set the `CLARIFAI_PAT` environment variable, it will use that token automatically. ### Serve Your Model Start the model using `clarifai model serve` (or the alias `local-runner`): clarifai model serve Or with a specific model path: clarifai model serve [OPTIONS] [MODEL_PATH] > **Note:** `MODEL_PATH` is an optional path to the model directory. If omitted, the current directory is used by default. :::note Options | Option | Default | Description | |--------|---------|-------------| | `--mode` | `none` | `none` (current env), `env` (auto-creates a virtualenv), `container` (Docker) | | `--grpc` | off | Standalone gRPC server — no Clarifai login required | | `--port` | 8000 | Server port (with `--grpc`) | | `--concurrency` | 32 | Max concurrent requests (equivalent to `--pool_size`) | | `--keep-image` | off | Keep Docker image after exit (container mode only) | | `--health-check-port` | auto | Port for the health check HTTP server | | `--disable-health-check` | off | Disable the health check server entirely | | `--auto-find-health-check-port` | off | Automatically find an available port if the default is in use | | `-v, --verbose` | off | Show detailed logs (useful for debugging toolkit servers like [Ollama](https://docs.clarifai.com/compute/toolkits/ollama/)) | ::: #### Standalone gRPC Mode For offline development without a Clarifai account, use `--grpc` to start a standalone gRPC server: ```bash # Start standalone gRPC server (no login required) clarifai model serve ./my-model --grpc # Custom port clarifai model serve ./my-model --grpc --port 9000 ``` This is useful for quick local testing and debugging without needing an internet connection or Clarifai credentials. #### Virtual Environment Mode Use `--mode env` to automatically create a virtualenv with the model's dependencies: clarifai model serve ./my-model --mode env This isolates the model's packages from your system Python, preventing dependency conflicts. If the runner doesn't detect the necessary [context configurations](#step-2-create-a-context-optional) in your environment, it will prompt you to create them using default values. This ensures that all essential components required for Local Runners are properly set up or included in your configuration context, including: - A compute cluster and nodepool configured for Local Runners. > **Note:** This cluster is created exclusively for Local Runners. It is not designed to support other tasks and lacks features like autoscaling to handle variable traffic demands, among other cloud-specific capabilities. You also cannot use other types of clusters for Local Runners — only the special cluster created for this purpose is supported. - An app with a model and model version representing the local runner. - A deployment that places the model version into the designated nodepool. > **Note:** All resources created by `clarifai model serve` — the app, model, model version, and deployment — are set to **public visibility** by default. This means anyone with the model URL can send predictions to it. You can adjust visibility after creation in the Clarifai platform if needed. Example Output {LocalDev} > **Note:** If the [`config.yaml`](https://docs.clarifai.com/compute/models/upload/#step-2-prepare-the-configyaml-file) file does not contain model information that matches the `user_id`, `app_id`, and `model_id` defined in your current context, it will be automatically updated to include the new model details. This ensures that the model started by the local development runner is the same one you intend to call via the API. If needed, you can back up the existing `config.yaml` file as `config.yaml.bk`. > **Note:** The `model_type_id` is loaded from the `config.yaml` file, rather than from the context, ensuring consistency between the model configuration and the actual model. ## Step 4: Test Your Runner When the local development runner starts in your terminal, it automatically generates an example client code snippet based on the [model's signature](https://docs.clarifai.com/compute/inference/#structure-of-prediction-methods). You can run this snippet in a separate terminal (within the same directory) to receive the model’s response output. Example Code Snippet {LocalDevExampleCode} The terminal also displays a link to the [AI Playground](https://docs.clarifai.com/getting-started/quickstart-playground), which you can copy and use to interact with the model directly. Alternatively, while the runner is active in your terminal, you can open the [**Runners**](https://clarifai.com/compute/runners) dashboard on the Clarifai platform, find your runner in the table, click the three-dot menu, and select **Open in Playground** to chat with the model. ![](/img/others/runners-dashboard.png) When you’re done testing, simply close the terminal running the local development runner to shut it down. ## Additional Examples - [Simple example for running Hello World model locally with Clarifai’s Local Runners](https://github.com/Clarifai/runners-examples/tree/main/hello-world) - [YouTube video on how to build your own local AI agent using Clarifai’s Local Runners](https://www.youtube.com/watch?v=JOdtZDmCFfk) --- ## Compute Orchestration # Compute Orchestration **Orchestrate your AI workloads better, avoid vendor lock-in, and use compute spend efficiently** Clarifai’s Compute Orchestration provides efficient capabilities for you to deploy any model on any compute infrastructure, at any scale. These new platform capabilities bring the convenience of serverless autoscaling to any environment, regardless of deployment location or hardware, and dynamically scale resources to meet workload demands. With Compute Orchestration, we are providing users with the ability to organize and manage (_orchestrate_) the compute resources necessary for running their models and workflows. These capabilities enable our enterprise customers to deploy production models with enhanced control, performance, and scalability — while addressing specific problems around compute costs, latency, and control over hosted models. Clarifai handles the containerization, model packing, time slicing, and other performance optimizations on your behalf. ## Deployment Options Compute Orchestration allows us to provide multiple deployment options — all of which can be customized with your preferred settings for autoscaling, cold start, and more, ensuring maximum cost efficiency and performance. These are the deployment options we provide: - **Shared SaaS (Serverless)** — This deployment option provides a fully managed, serverless environment that eliminates the need to configure or maintain any infrastructure, allowing you to run predictions effortlessly. It is available only for a curated set of featured models that are uploaded and owned by Clarifai. - **Dedicated SaaS** — Provides access to Clarifai‑managed, isolated nodes with customizable configurations. For example, you can launch a dedicated deployment in AWS US‑East, with plans to expand to other cloud providers and hardware options in the future. - **Self-Managed VPC (Virtual Private Cloud)** — Connect your own cloud provider’s VPC to Clarifai, allowing Clarifai to run and orchestrate deployments within your environment. This approach lets you maintain full control of your infrastructure while leveraging existing cloud resources and spend commitments. - **Self-Managed On-Premises** — Connect your own on‑premises or bare‑metal infrastructure to Clarifai, allowing you to utilize existing compute resources. Clarifai then orchestrates model deployments within your environment, making the most of your infrastructure investments. - **Multi-Site Deployment** — Enables deployments across multiple self‑managed compute environments, with support for a mix of on‑premises, cloud, or edge locations — and a roadmap for future multi‑cloud or multi‑region SaaS options. - **Full Platform Deployment** — Designed for organizations with stringent security and compliance needs, this option allows you to run both the Clarifai control and compute planes within your chosen cloud, on‑premises, or air‑gapped infrastructure, ensuring complete isolation and control. ![ ](/img/compute-orchestration/intro-1.png) ## Compute Clusters and Nodepools We use [clusters and nodepools](https://docs.clarifai.com/compute/deployments/clusters-nodepools) to organize and manage the compute resources required for the Compute Orchestration capabilities. :::info Cluster A compute cluster in Clarifai acts as the overarching computational environment where models are executed, whether for training or inference. ::: :::info nodepool A nodepool refers to a set of dedicated nodes (virtual machine instances) within a cluster that share similar configurations and resources, such as CPU or GPU type, memory size, and other performance parameters for running your models. ::: Cluster configuration lets you specify where and how your models are run, ensuring better performance, lower latency, and adherence to regional regulations. You can specify a cloud provider, such as AWS, that will provide the underlying compute infrastructure for hosting and running your models. You can also specify the geographic location of the data center where the compute resources will be hosted. Nodepools are an important part of how compute resources are operated within a cluster. They provide flexibility in choosing the type of instances used to run your machine learning models and workflows and help determine how resources are scaled to meet demand. Nodepools specify the accelerator and instance that will run your models and other workloads. Accelerators are specialized hardware resources, such as GPUs or dedicated ML chips used for computation. :::tip Supported Instance Types See the [Supported Cloud Instances](https://docs.clarifai.com/compute/cloud-instances) page for a full list of available GPU, TPU, and CPU-only instance types across AWS, GCP, Vultr, and Oracle — including pricing and hardware specs. ::: Each nodepool can run containers or workloads, and you can have multiple nodepools within a single cluster to support different types of workloads or performance requirements. These nodes execute tasks like model training, inference, and workflow orchestration within a compute cluster. With compute orchestration, you can ensure these nodepools are properly scaled up or down depending on the workload's size, complexities, and costs. ## Benefits of Compute Orchestration ![ ](/img/compute-orchestration/intro-2.png) ### 1. Performance and Deployment Flexibility - It provides access to a wide range of [GPU, TPU, and CPU instance types](https://docs.clarifai.com/compute/cloud-instances) across AWS, GCP, Vultr, and Oracle — including NVIDIA, AMD, and Google accelerators — tailored to your use case. You can configure multiple compute clusters each tailored to your AI development stage, performance requirements, and budget. You can also run affordable proof of concepts or compute-heavy LLMs or LVMs in production all from a single product. - It offers flexibility to [make deployments](https://docs.clarifai.com/portal-guide/compute-orchestration/deploy-model) in any cloud service provider, on-premises, air-gapped, or Kubernetes-supported environment. Or, you can make deployments in Clarifai’s compute to avoid having to worry about managing infrastructure. This allows users to leverage their hardware of choice without being locked into a single vendor. - You can customize auto-scaling settings to prevent cold-start issues and handle traffic swings; and scale down to zero for cost efficiency. The ability to scale from zero to infinity ensures both flexibility and cost management. - We ensure efficient resource usage and cost savings through [GPU fractioning](https://www.clarifai.com/blog/gpu-fractioning-explained-how-to-run-multiple-ai-workloads-on-a-single-gpu) (running multiple models per GPU), time slicing, and other optimizations. ### 2. Enhanced Security - Users can run compute planes within their own cloud service provider or on-premise environments and securely connect to Clarifai’s control plane, while only having to open outbound ports for traffic. This reduces networking complexities and security risks compared to opening inbound access or configuring cloud Identity and Access Management (IAM) access roles within your VPC. - Nodepool-based compute allows users to keep their resources isolated and provides precise control over scaling models and nodes. This allows users to specify where models are executed, addressing compliance and security needs for regulated industries. - Clarifai offers fine-grained access control across apps, teams, users, and compute resources. - Users can group CPU and GPU types into dedicated scaling nodepools, enabling them to handle diverse workloads or team-specific requirements while enhancing security and resource management. ### 3. Use Compute Cost-Efficiently and Abstract Away Complexity - An intuitive control plane enables users to efficiently govern access to AI resources, monitor performance, and manage costs. Clarifai’s expertly designed platform takes care of dependencies and optimizations, offering features like model packing, streamlined dependency management, and customizable autoscaling options — including scale-to-zero for both model replicas and compute nodes. - The advanced optimizations deliver exceptional efficiency, with model packing reducing compute usage by up to 3.7x and enabling support for over 1.6 million inputs per second with an impressive 99.9997% reliability. Depending on the chosen configuration, customers can achieve cost savings of at least 60%, and in some cases, up to 90%. - Organizations with pre-committed cloud spend or compute contracts with major cloud service providers, like AWS, Azure, or GCP, or existing GPU and hardware investments, can efficiently leverage their compute using Clarifai Compute Orchestration. ### 4. New Inference Capabilities and Developer Experience Improvements - Resourceful features such as inference streaming improve time-to-first-token for LLM generations. - Faster cold starts and optimized frameworks improve performance for critical workloads. - Continuous batching is available to reduce costs by processing multiple inference requests in batches. - Clarifai containerizes your desired models into Docker images, ensuring model package requirements are encapsulated in a portable environment and dependencies are handled automatically. - Low-latency deployment minimizes gRPC hops, speeding up communication. - New model types are easily supported with a unified protobuf format, and local inference runners allow users to [test models](https://docs.clarifai.com/compute/local-runners) before deploying to the cloud. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Pipelines # Pipelines **Orchestrate asynchronous, long-running, multi-step AI processes** Clarifai Pipelines let you design and run asynchronous, long-running, multi-step processes for complex AI and MLOps tasks. They act as the orchestration backbone for building advanced AI agents and managing end-to-end machine learning operations. Pipelines enable you to define, execute, and monitor container-based jobs directly on the Clarifai platform, giving you fine-grained control over each step. With Clarifai Pipelines, you get a scalable, end-to-end automation engine that simplifies and unifies your entire MLOps lifecycle. :::info Explore a collection of ready-to-use ML pipeline templates for training, evaluating, and running models on the Clarifai platform in our [Examples repository](https://github.com/Clarifai/pipeline-examples). See the [Training Models](https://docs.clarifai.com/create/models/) documentation to learn how to use the pipeline templates to train different types of models. ::: ## Why Clarifai Pipelines * **Schedule training on your inference clusters.** Training, evaluation, and inference workloads share the same Compute Orchestration clusters — no separate cluster to provision and pay for. Run a training job overnight on the same nodepool that serves your daytime inference traffic. * **Outputs land in the model registry.** Trained and quantized models from pipeline runs are registered directly into Clarifai's model registry, ready to serve or evaluate further without an export step. * **Python or YAML — both first-class.** Author pipelines as decorated Python functions with the [DSL](dsl-reference.md), or as YAML config with [scaffold templates](create-api.md). Same engine, same templates, same upload path — pick whichever fits your team. ## Use Cases for Pipelines Clarifai Pipelines support a wide range of AI and MLOps workflows. Here are some common use cases: * **Model training and fine-tuning** — Fine-tune LLMs and train custom image classifiers, object detectors, and other models on your own data, with the full training loop running on GPU compute and the resulting model registered in the Clarifai model registry. * **Model evaluation and benchmarking** — Measure model performance with standardized benchmarks like IFBench, SWE-bench, or MMLU, or with task-specific evaluations that score trained models against ground-truth labels, with per-task metrics stored alongside the model version under test. * **Model quantization** — Shrink a model's memory and compute footprint with post-training quantization, orchestrating calibration, conversion, and accuracy evaluation end-to-end and registering the quantized model alongside the original. * **Draft model training for speculative decoding** — Train smaller "draft" models that accelerate inference on larger target models via speculative decoding, including dataset preparation, draft training, and alignment evaluation against the target. * **AI agent orchestration** — Design, test, and deploy AI agents that execute multi-step logic autonomously — orchestrating [tool usage](https://docs.clarifai.com/compute/inference/open-ai/#tool-calling), LLM calls, and decision logic across long-running tasks that can take hours or days. * **Complex workflow automation** — Orchestrate sophisticated, multi-stage AI workflows across components and services — large-scale batch processing, data transformations, and other distributed jobs — with built-in state management, fault tolerance, and scalable execution. ## Quick Start import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; The fastest way to get a working pipeline is to scaffold one from a template, upload it, and run it. The [`lora-pipeline-unsloth-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/lora-pipeline-unsloth-quick-start) template fine-tunes an LLM with LoRA — all parameters are pre-set to sensible defaults so the first run works without edits. ### Install and Authenticate Install the latest `clarifai` SDK (which also installs the CLI) and set your [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat): {`pip install --upgrade clarifai export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE`} {`pip install --upgrade clarifai set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE`} ### Scaffold a Pipeline From a Template {`clarifai pipeline init --template lora-pipeline-unsloth-quick-start cd lora-pipeline-unsloth-quick-start`} This creates a working LoRA fine-tuning project that trains [`unsloth/Qwen3-0.6B`](https://huggingface.co/unsloth/Qwen3-0.6B) on the [`mlabonne/FineTome-100k`](https://huggingface.co/datasets/mlabonne/FineTome-100k) dataset using [Unsloth](https://github.com/unslothai/unsloth) — with `num_epochs`, `learning_rate`, LoRA rank, sequence length, and other hyperparameters all pre-configured. No edits required to run the first time. > **Tip:** Browse all available starter templates with `clarifai pipelinetemplate list`. See the [Templates](templates.md) page for the full catalog. ### Upload and Run {`clarifai pipeline upload clarifai pipeline run --instance=g5.xlarge`} The pipeline fine-tunes the model on a single-GPU instance (the template's compute step requests `num_accelerators: 1` with a `NVIDIA-*` GPU). `--instance=g5.xlarge` auto-provisions a compute cluster and nodepool — no separate cluster setup required, and the same nodepool can later serve inference for the fine-tuned model. That's it — you have a fine-tuned model registered in your Clarifai model registry, ready to serve, evaluate, or refine further. :::tip Two upload paths The LoRA template ships with both a YAML scaffold (`config.yaml`) and a Python DSL form (`dsl.py`). The default `clarifai pipeline upload` uses the YAML scaffold. To upload from the DSL Python file instead: ```bash clarifai pipeline upload dsl.py ``` Both paths produce the same pipeline. The [YAML config](create-api.md) is convenient for explicit, version-controlled configuration. The [Pipeline DSL](dsl-reference.md) is convenient for code-first authoring with type hints, `@step` decorators, and `>>` DAG composition. Use whichever fits your team. ::: ### Customize Before Running To override defaults at init time, pass `--set key=value` flags. For example, to fine-tune a different base model and change the number of epochs: {`clarifai pipeline init --template lora-pipeline-unsloth-quick-start \\ --set base_model_name="unsloth/Llama-3.2-1B-Instruct" \\ --set num_epochs=3`} Or `cd` into the scaffolded directory and edit `config.yaml` directly. See [Templates](templates.md) for the full reference on `--template`, `--set`, and template discovery. ### Author a Pipeline From Scratch in Python For custom workflows that don't fit a template, use the [Pipeline DSL](dsl-reference.md) — write `@step`-decorated Python functions, compose them into a DAG, and upload directly from a `.py` file. See the [DSL reference](dsl-reference.md) for the full API. ### Where to Go Next * **[Templates](templates.md)** — full catalog of starter pipelines for training, fine-tuning, evaluation, and benchmarking. * **[Pipeline DSL reference](dsl-reference.md)** — `@step`, `step_ref`, `>>` composition, `ComputeInfo`, secrets, codegen. * **[Manage Pipelines](manage.md)** — list, validate, and inspect pipelines on the platform. * **[Manage Pipeline Runs](manage-run.md)** — monitor, pause, resume, and cancel runs. * **[YAML / config-based pipelines](create-api.md)** — the YAML config authoring path, useful for explicit version-controlled configuration, YAML-first workflows, or as a reference for the scaffold-directory file structure. ### Pipeline outputs → Artifacts Pipeline runs produce [Artifacts](https://docs.clarifai.com/create/artifacts) — versioned, addressable binary outputs such as model checkpoints, training logs, embeddings, or datasets. Once a pipeline run completes, its outputs can be managed, downloaded, and reused via the artifacts CLI and SDK without re-running the pipeline. See the [Artifacts documentation](https://docs.clarifai.com/create/artifacts) for managing pipeline-produced outputs. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## YAML / Config-Based Pipelines # YAML / Config-Based Pipelines **YAML / config-based authoring flow for pipelines, with the full file structure reference** :::tip Prefer Python? Clarifai Pipelines support two authoring paths: this YAML / config-based flow, and the [Pipeline DSL](dsl-reference.md) for code-first authoring with `@step` decorators and `>>` DAG composition. Both paths reach the same engine and produce the same pipeline — pick whichever fits your team. This page documents the YAML path: scaffold a directory of `config.yaml` + `pipeline_step.py` files with `clarifai pipeline init`, edit them, and upload the directory. ::: [Clarifai Pipelines](README.mdx) let you design and launch asynchronous, multi-step AI workflows on our platform. You can effortlessly automate complex processes, orchestrate AI agents, and run long-running jobs at scale. This page walks through the YAML / config-based authoring flow. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PipelineInit from "!!raw-loader!../../../code_snippets/new-docs/pipelines/pipeline-init.txt"; import CompilePipeline from "!!raw-loader!../../../code_snippets/new-docs/pipelines/compile-pipeline.txt"; import PipelineDefinition from "!!raw-loader!../../../code_snippets/new-docs/pipelines/pipeline-definition.py"; import ConfigYamlRoot from "!!raw-loader!../../../code_snippets/new-docs/pipelines/config-yaml-root.yaml"; import ConfigYamlStep from "!!raw-loader!../../../code_snippets/new-docs/pipelines/config-yaml-step.yaml"; import RequirementsTXT from "!!raw-loader!../../../code_snippets/new-docs/pipelines/requirements-txt.txt"; import PipelineStep from "!!raw-loader!../../../code_snippets/new-docs/pipelines/pipeline-step.py"; import UploadPipeline from "!!raw-loader!../../../code_snippets/new-docs/pipelines/upload-pipeline.txt"; import RunPipeline from "!!raw-loader!../../../code_snippets/new-docs/pipelines/run-pipeline.txt"; import RunPipelineDev from "!!raw-loader!../../../code_snippets/new-docs/pipelines/run-pipeline-dev.txt"; import DockerfileExample from "!!raw-loader!../../../code_snippets/new-docs/pipelines/dockerfile-example.py"; import DockerfileExample2 from "!!raw-loader!../../../code_snippets/new-docs/pipelines/dockerfile-example_2.py"; import LockfileExample from "!!raw-loader!../../../code_snippets/new-docs/pipelines/lockfile-example.yaml"; import RootLockfileExample from "!!raw-loader!../../../code_snippets/new-docs/pipelines/root-lockfile-example.yaml"; ## Step 1: Perform Prerequisites ### Sign Up or Log In To get started, [log in](https://clarifai.com/login) to your Clarifai account or [sign up](https://clarifai.com/signup) for a new one. If you’re creating a new account, a default application will be automatically created for you. Next, gather the following details — they’re needed to create and manage pipelines programmatically: * **App ID** – Go to your application’s page and choose the [**Overview**](https://docs.clarifai.com/create/applications/manage#app-overview) option in the collapsible left sidebar. Get the app ID from there. * **User ID** – Go to **Settings** in the collapsible left sidebar and choose the **Account** option. Then, copy your user ID from that page. * **PAT (Personal Access Token)** – From the same **Settings** menu, navigate to the **Secrets** page to generate or copy your [PAT](https://docs.clarifai.com/control/authentication/pat). This token authenticates your requests to the Clarifai platform. Set your PAT as an environment variable before continuing: export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Install the Clarifai Python SDK Install the latest version of the `clarifai` Python SDK. It includes the [Clarifai CLI](https://docs.clarifai.com/additional-resources/api-overview/cli), which you can use to interact with pipelines from the command line. ```bash pip install --upgrade clarifai ``` ### Create a Cluster and Nodepool A compute cluster and nodepool define where your pipeline runs within Clarifai’s compute environment. They are only required if you plan to run your pipeline on managed infrastructure (see [Option B](#option-b-run-on-cluster-and-nodepool) in Step 5). If you need a cluster and nodepool, [create](https://docs.clarifai.com/compute/deployments/clusters-nodepools) them and get their IDs. If you prefer on-demand instance compute (see [Option A](#option-a-run-on-on-demand-instance-compute)), you can skip this step. ## Step 2: Initialize a Pipeline Project You can set up your pipeline project in two ways: scaffold a boilerplate with `init`, or compile one from a Python definition using `compile`. Both approaches produce the same project structure. ### Option A: Use `pipeline init` Run the following command to create a new pipeline project in your current directory: clarifai pipeline init > **Note:** > - You can initialize a project in a specific location by providing a `PIPELINE_PATH`. For example, running `clarifai pipeline init ` creates a new directory named `` and populates it with all the required pipeline project files. > - You can shorten `pipeline` to `pl` when running pipeline commands. For example: `clarifai pl init`. > - By default, `clarifai pipeline init` fetches templates from the official Clarifai pipeline templates repository. To point to a custom or private template repository, set the `CLARIFAI_PIPELINE_TEMPLATES_GIT_REPO_URL` environment variable to your repository URL before running the command. After running the command, you’ll be prompted to provide the following details: * `User ID` — Your Clarifai user ID. * `App ID` — Your Clarifai application ID where the pipeline lives. * `Pipeline ID` — The unique ID for your pipeline (or press **Enter** to use the default). * `Number of steps` — How many steps your pipeline should have (or press **Enter** to use the default). You need to specify at least one step. * `Step names` — A name for each pipeline step (or press **Enter** to use the defaults). Example Output {PipelineInit} Once scaffolded, proceed to [Step 3](#step-3-modify-the-files) to customize the generated files. ### Option B: Use `pipeline compile` If you prefer to define your pipeline in Python, you can use the `compile` command to generate the YAML/config-based pipeline assets from a Python pipeline definition — so you can skip manual YAML editing in Step 3. Start by creating a `pipeline_definition.py` file that declares your pipeline and its steps using the Clarifai SDK: Example: pipeline_definition.py {PipelineDefinition} > **Note:** You can find another example pipeline definition file in the [Clarifai examples repository](https://github.com/Clarifai/clarifai-python/blob/master/examples/pipeline_dsl_text_pipeline.py). Then run the `compile` command, pointing to your definition file and specifying an output directory: ```bash clarifai pipeline compile pipeline_definition.py --output-dir ./my-pipeline ``` The `--output-dir` flag (required) specifies the directory where the compiled `config.yaml` and step folders will be written. This gives you the same project structure as `pipeline init`, but derived directly from your Python definition. Example Output {CompilePipeline} :::note ### Pipeline Steps A step is a self-contained unit of execution in a Clarifai Pipeline. It represents one stage of your workflow and performs a specific task, then optionally passes its output to the next step. For example, a step might preprocess input data, call an AI model, transform results, or interact with an external API. In your pipeline project structure, each step is represented by its own folder. Steps are executed in sequence, parallel, or according to the definitions in the pipeline configuration. Each step runs in an isolated, containerized environment with: * Its own dependencies (`requirements.txt`) * Its own configuration (`config.yaml`) * Its own implementation logic (`/1/pipeline_step.py`) This design makes steps modular, reusable, and independently versioned — allowing you to update or improve one step without impacting the rest of the pipeline. > **Note:** You can also manage individual pipeline steps using the `pipelinestep` (or `ps`) command. This allows you to create, update, and reuse pipeline steps independently of the full pipeline. For example: > * `clarifai pipelinestep init` – Initialize a new pipeline step project structure. > * `clarifai pipelinestep local-run` – Run a pipeline step locally in a Docker container. > * `clarifai pipelinestep upload` – Upload a pipeline step to the Clarifai platform. ::: Regardless of which option you chose above, the resulting project structure looks like this: ```text ├── config.yaml # Pipeline configuration ├── stepA/ # First pipeline step │ ├── config.yaml # Step A configuration │ ├── requirements.txt # Step A dependencies │ └── 1/ │ └── pipeline_step.py # Step A implementation logic ├── stepB/ # Second pipeline step │ ├── config.yaml # Step B configuration │ ├── requirements.txt # Step B dependencies │ └── 1/ │ └── pipeline_step.py # Step B implementation logic └── README.md # Generated project documentation ``` ## Step 3: Modify the Files The Clarifai CLI generates a structured, multi-step pipeline project, where each step is independently configurable, versioned, and deployable. Next, you can customize the generated files to define your pipeline’s behavior. Let’s walk through what each file does in the default project setup. ### `config.yaml` (root) Example: config.yaml (root) {ConfigYamlRoot} The root `config.yaml` file defines your pipeline’s identity, structure, and execution logic. It tells Clarifai what the pipeline is, which steps it contains, and how those steps should be orchestrated at runtime. :::tip Argo Workflows Pipeline execution is powered by [Argo Workflows](https://argoproj.github.io/workflows/), an open-source, Kubernetes-native orchestration engine used to schedule and manage containerized jobs. Argo models multi-step workflows as a sequence of tasks and captures the dependencies between them using a directed acyclic graph (DAG). Clarifai leverages this under the hood to coordinate step execution, manage dependencies, and handle long-running workloads. ::: Let’s break down the `config.yaml` file section by section. #### Pipeline Metadata ```yaml pipeline: id: "hello-world-pipeline" user_id: "user-id" app_id: "app-id" ``` These fields define your pipeline’s identity and location when uploaded to the Clarifai platform. They are automatically filled based on the values you provide during initialization. #### Step Directories ```yaml step_directories: - stepA - stepB ``` This section tells Clarifai which folders in your project represent pipeline steps and how they should be ordered during execution. The execution order is stipulated by the Argo Workflow Definition, as explained below. Each listed directory corresponds to one pipeline step created during initialization. As mentioned [earlier](#pipeline-steps), each step contains its own `config.yaml`, `requirements.txt`, and executable logic, allowing steps to be configured and maintained independently. > **Note:** `step_directories` is optional when all `templateRef` entries in the Argo Workflow Definition already include a pinned version ID. In that case, Clarifai resolves the step implementations directly from those versioned references and does not need to upload local step directories. This is useful when the pipeline steps are already deployed to the platform and you only need to update or re-upload the pipeline definition itself. #### Argo Workflow Definition ```yaml orchestration_spec: argo_orchestration_spec: | apiVersion: argoproj.io/v1alpha1 kind: Workflow spec: entrypoint: sequence arguments: parameters: - name: input_text value: "Input Text Here" ``` This section is the control center of your pipeline. It defines how Clarifai uses Argo Workflows to orchestrate and execute your multi-step workflow as a DAG. Here’s what each part does: * `apiVersion` — Specifies the version of the Argo Workflow API being used. * `kind: Workflow` — Tells the system this is a multi-step Argo workflow. * `entrypoint` — Defines the starting point of execution. In this case, the workflow begins at a [template](https://argo-workflows.readthedocs.io/en/release-3.5/workflow-concepts/) named `sequence`. * Input parameters — These are runtime variables passed into your pipeline when you run it. Here, a parameter called `input_text` is defined. If no value is provided at runtime, it defaults to `"Input Text Here"`. This value can then be consumed by any pipeline step. #### Step Execution Order ```yaml templates: - name: sequence steps: - - name: step-0 templateRef: name: users/user-id/apps/app-id/pipeline_steps/stepA template: users/user-id/apps/app-id/pipeline_steps/stepA arguments: parameters: - name: input_text value: "{{workflow.parameters.input_text}}" - - name: step-1 templateRef: name: users/user-id/apps/app-id/pipeline_steps/stepB template: users/user-id/apps/app-id/pipeline_steps/stepB arguments: parameters: - name: input_text value: "{{workflow.parameters.input_text}}" ``` The section defines how individual pipeline steps are connected and the exact order in which they run. * `step-0` — Executes first. It references the deployed pipeline step `stepA` and receives the `input_text` parameter. * `step-1` — Executes after `step-0` completes. It references `stepB` and also receives the same `input_text` parameter. > **Note:** This structure defines a sequential workflow. Under the hood, Argo models this sequence as a DAG, ensuring that `stepB` only runs once its dependency (`stepA`) has successfully completed. The `templateRef` fields point to the deployed step implementations in your project directories (`stepA/` and `stepB/`). When deployed, Clarifai transforms these into executable Argo templates that power the pipeline. > **Note:** You can also structure your pipeline as a [parallel workflow](https://argo-workflows.readthedocs.io/en/release-3.4/walk-through/steps/), where multiple steps run simultaneously instead of sequentially. #### Optional: Secrets Configuration ```yaml config: step_version_secrets: step-0: API_KEY: users/user-id/apps/secrets/my-api-key DB_PASSWORD: users/user-id/apps/secrets/db-secret step-1: EMAIL_TOKEN: users/user-id/apps/secrets/email-token ``` This section lets you securely inject [environment secrets](https://docs.clarifai.com/control/authentication/environment-secrets) into specific pipeline steps. It’s especially useful for handling sensitive data, like third-party API keys, without hardcoding them into your source code. Each key (for example, `API_KEY` or `DB_PASSWORD`) becomes an environment variable inside the corresponding step’s runtime environment. The values reference secrets stored in the Clarifai secrets manager, which ensures they are encrypted, access-controlled, and never exposed directly in your project files. > **Note:** Once the secrets are mounted, you can access them in your [`pipeline_step.py`](#1pipeline_steppy) file using standard environment variable calls, for example: ```python import os api_key = os.environ['API_KEY'] ``` > The secret value is then available to your code at runtime. ### `step/config.yaml` Example: step/config.yaml {ConfigYamlStep} Each pipeline step includes its own [`config.yaml`](https://docs.clarifai.com/compute/upload/#prepare-configyaml) file, which defines it as a self-contained, containerized unit of execution. These files usually share a common structure and similar resource settings, helping keep your pipeline consistent, predictable, and easy to manage. Let’s break down the `step/config.yaml` file section by section. #### Step Metadata ```yaml pipeline_step: id: "stepA" user_id: "user-id" app_id: "app-id" ``` This section defines how Clarifai identifies, scopes, and stores this pipeline step. These values are automatically filled with the information you provide when you [initialize](#step-2-initialize-a-pipeline-project) the pipeline project. > **Note:** Clarifai Pipelines support cross-application orchestration for reusing logic. To integrate a shared step into your pipeline, simply configure it with the corresponding `user_id`, `app_id`, or step `id` where the resource is hosted. #### Input Interface ```yaml pipeline_step_input_params: - name: input_text description: "Text input for processing" ``` This defines the data this step expects when it runs. The description helps document what this input is used for and how it should be provided by upstream steps or users. #### Runtime Environment ```yaml build_info: python_version: "3.12" ``` This specifies the environment used to run the step, ensuring [compatibility](https://docs.clarifai.com/resources/api-overview/python-sdk#python-requirements) with your code and dependencies. #### Compute Resources Allocation ```yaml pipeline_step_compute_info: cpu_limit: "500m" cpu_memory: "500Mi" num_accelerators: 0 ``` This section controls the resources allocated to the step during execution. It is the most critical part for performance and cost. * `cpu_limit: "500m"` allocates half of one CPU core's processing power (500 millicores = 0.5 CPU cores). This indicates the step is a lightweight task (like text parsing), not a heavy calculation. * `cpu_memory: "500Mi"` limits memory usage to 500 Mebibytes of RAM. * `num_accelerators: 0` means no GPU or other [accelerators](https://docs.clarifai.com/compute/cloud-instances) are used. This keeps costs low, as you are running on standard CPU infrastructure. ### `step/requirements.txt` Example: step/requirements.txt {RequirementsTXT} Each pipeline step has its own [`requirements.txt`](https://docs.clarifai.com/compute/upload/#prepare-requirementstxt) file, which specifies the Python packages required for that specific step to run. These dependencies are installed in an isolated, containerized environment, meaning one step can use a different set of libraries or versions without affecting other steps in the pipeline. ### `1/pipeline_step.py` Example: 1/pipeline_step.py {PipelineStep} Each pipeline step has its own `pipeline_step.py` file, which contains the core implementation for that specific step. This is where you define all the logic the step executes — including data processing, API interactions, model inference, and output transformations. When the pipeline runs, Clarifai spins up this step’s container and executes this script. > **Note:** The `pipeline_step.py` file is nested inside the `step/1/` directory. The folder is named as **1** to fit Clarifai’s naming convention. Let’s break down the example of a `pipeline_step.py` file section by section. #### Command-line Input Handling ```python parser = argparse.ArgumentParser(description='stepA processing step.') parser.add_argument('--input_text', type=str, required=True, help='Text input for processing') args = parser.parse_args() ``` This section receives its inputs as command-line arguments when Clarifai runs it. The `--input_text` corresponds to the `pipeline_step_input_params` you defined in the step’s [`config.yaml`](#stepconfigyaml). Clarifai automatically maps pipeline parameters to these arguments when the step is executed. #### Logging the Clarifai SDK Version ```python logger.info(clarifai.__version__) ``` This section logs the version of the Clarifai Python SDK installed inside the step’s container. It’s useful for debugging and verifying that the correct environment is being used. #### Step Logic Placeholder ```python # TODO: Implement your pipeline step logic here logger.info(f"stepA processed: {args.input_text}") ``` This section is where your step’s core logic lives. In this example, it simply logs the input it receives, but in a real pipeline, this is where you would implement the actual work of the step, such as preprocessing input data or calling LLM models. ## Step 4: Upload the Pipeline Run the following command in your current directory to upload the pipeline with associated pipeline steps to Clarifai. clarifai pipeline upload > **Note:** You can specify a path to the pipeline configuration file or directory containing the root `config.yaml` file. If not specified, the current directory is used by default. Example Output {UploadPipeline} When you run the `upload` command, the CLI reads your pipeline configuration, uploads each step, and automatically builds a container image for every step using its code, configs, and dependencies. - **Dockerfile** — For each step, it auto-generates a `Dockerfile` behind the scenes, which defines how the container image for the step is built (base image, Python version, dependency installation, and entrypoint setup). Example: Dockerfile (auto-generated) {DockerfileExample} You can also define a custom `Dockerfile` to install additional binaries or dependencies during the image build process, giving you full control over the runtime environment. Example: Dockerfile (custom) {DockerfileExample2} >**Note:** The key requirement here is where the pipeline step files live inside the container image. Specifically, `requirements.txt` and the step directory (`1/` containing `pipeline_step.py`) should be placed under `/home/nonroot/main/`. This directory layout is what Clarifai expects at runtime to correctly discover, validate, and execute the pipeline step. In the above custom Dockerfile, that requirement is satisfied by copying the step implementation to `/home/nonroot/main/1` and the supporting files (`requirements.txt`, `config.yaml`) to `/home/nonroot/main/`, ensuring the pipeline step is executable once the image is built. - **Lock file** — If the CLI detects that no lock file is existing, it auto-creates a new `config-lock.yaml` to “freeze” the exact step configuration and build details for that version. This lock file captures the resolved configuration, including the Python runtime (`3.12`), compute resources, input parameters, and a unique version ID. The `step/config.yaml` file also stores an MD5 hash of the step contents, which is used to track changes and ensure reproducible builds. > **Tip:** Use the `--no-lockfile` flag to disable generation of `config-lock.yaml`. Example: `clarifai pipeline upload --no-lockfile`. Example: config-lock.yaml (root) {RootLockfileExample} Example: step/config-lock.yaml {LockfileExample} - **Container image** — Using the generated Dockerfile and the locked configuration, Clarifai then builds a multi-architecture container image on its infrastructure. This image becomes a versioned, immutable artifact that’s tightly coupled with the `config-lock.yaml`, ensuring every future pipeline execution runs with the exact same code, environment, and resource settings. > **Note:** If you modify your pipeline and upload it again to the Clarifai platform, a new version is automatically created. Only the updated sections are uploaded, ensuring efficient version management. :::note ### Upload Pipeline Steps To upload a specific pipeline step, navigate to the step’s directory and run `clarifai pipelinestep upload`. Alternatively, you can provide the path to the pipeline step directory directly in the command. By default, a `Dockerfile` is automatically generated for the pipeline step during upload. If you need more control, you can provide your own custom Dockerfile, as described [earlier](#docker1). If you want to skip generating a Dockerfile — so you can use an already existing one — use the `--skip_dockerfile` flag: `clarifai pipelinestep upload --skip_dockerfile`. ::: :::note ### Test a Pipeline Step Locally Before uploading, you can run an individual pipeline step locally in a Docker container using `clarifai pipelinestep local-run`. This lets you validate your step’s logic and dependencies without deploying to Clarifai, making it faster to iterate during development. clarifai pipelinestep local-run Run this command from inside the step’s directory (for example, `stepA/`). The CLI builds a Docker image for the step using its `Dockerfile`, `requirements.txt`, and `pipeline_step.py`, then executes it locally. ::: ## Step 5: Run the Pipeline After successfully uploading your pipeline, you can execute it using the Clarifai CLI. You can run your pipeline using either on-demand instance compute or a preconfigured cluster and nodepool. ### Option A: Run on On-Demand Instance Compute Instead of relying on an existing nodepool and compute cluster, you can automatically provision or reuse compute at runtime by specifying an instance type: clarifai pipeline run --instance=g6e.xlarge This approach removes the need to manage infrastructure, making it ideal for quick experiments or simplified workflows. See the [available instance types](https://docs.clarifai.com/compute/cloud-instances) to choose one that best matches your workload and performance requirements. ### Option B: Run on Cluster and Nodepool If you've already set up a compute cluster and nodepool, you can run the pipeline by explicitly targeting those resources. Run the following command from your current project directory to start the pipeline and monitor its progress until completion or timeout. > **Note:** You must specify both a compute [cluster ID and a nodepool ID](#create-a-cluster-and-nodepool). ```text clarifai pipeline run --compute_cluster_id cluster_id_here --nodepool_id nodepool_id_here ``` > **Note:** The `clarifai pipeline run` command requires the user ID, app ID, pipeline version ID, and pipeline version run ID, which it reads directly from `config-lock.yaml`. Example Output {RunPipeline} ### Option C: Iterate with `--dev` During development, you can use the `--dev` flag to run your pipeline against a throwaway dev copy without touching your production pipeline. This lets you iterate quickly on step code without creating new versions of your main pipeline. When `--dev` is set, the CLI: 1. Creates (or updates) an ephemeral pipeline named `{pipeline_id}-dev`. 2. Re-uploads only the steps whose local files have changed, using hash-based detection to skip unchanged steps. 3. Writes a separate `config-lock-dev.yaml` — the production `config-lock.yaml` is never modified. 4. Runs the dev pipeline on the compute you specify. Run from your pipeline directory, or pass a path to the directory (or `config.yaml`) as the optional `PATH` argument: ```text clarifai pipeline run --dev --compute_cluster_id cluster_id_here --nodepool_id nodepool_id_here ``` Example Output {RunPipelineDev} ### Run Command Options The table below summarizes the available options you can use with the `clarifai pipeline run` command. | Option | Type | Description | Defaults / Notes | | :--- | :--- | :--- | :--- | | **Targeting & Identity** | | | | | `PATH` | positional `PATH` | Optional path to a pipeline directory or `config.yaml`. Config resolution order: `config-lock.yaml` > `config.yaml`. | Default: current directory | | `--pipeline_id` | `TEXT` | ID of the pipeline to execute | | | `--user_id` | `TEXT` | User ID owning the pipeline | | | `--app_id` | `TEXT` | App ID containing the pipeline | | | `--pipeline_version_id` | `TEXT` | Specific version of the pipeline to run | | | `--pipeline_url` | `TEXT` | Full URL to the pipeline resource | Alternative to providing IDs separately | | **Execution Control** | | | | | `--dev` | `FLAG` | Upload local code to an ephemeral `{pipeline_id}-dev` pipeline before running. Only re-uploads changed steps (hash-based). Writes `config-lock-dev.yaml` — production lockfile is unaffected. | | | `--config` | `PATH` | Path to a local configuration file for the run | Deprecated — use the positional `PATH` argument instead | | `--pipeline_version_run_id`| `TEXT` | Custom ID for this specific execution run | A UUID is generated automatically if omitted | | `--nodepool_id` | `TEXT` | Specific Nodepool to execute on | | | `--compute_cluster_id` | `TEXT` | Specific Compute Cluster to execute on | | | **Inputs & Parameters** | | | | | `--set` | `TEXT` | Override parameter values inline | Format: `key=value`. Can be used multiple times. Example `--set prompt="hello"`, `--set temperature="0.7"`| | `--overrides-file` | `PATH` | Path to JSON/YAML file for bulk parameter overrides| | | **Monitoring & Logging** | | | | | `--monitor` | `FLAG` | Watch an *existing* run instead of starting a new one| **Requires** `--pipeline_version_run_id` | | `--monitor_interval` | `INTEGER`| Frequency of status checks | Default: **10 seconds**. | | `--timeout` | `INTEGER`| Max time to wait for completion | Default: **3600 seconds** (1 hour) | | `--log_file` | `PATH` | File path to save execution logs | If omitted, logs output to console | That's it! --- ## Pipeline DSL Reference # Pipeline DSL Reference **Define pipelines in Python — steps, composition, compute, and upload — with no separate YAML to maintain** The Clarifai pipeline DSL lets you define entire pipelines in Python. Steps are decorated functions; pipelines are composed inside a `Pipeline` context; parameters come from function signatures. The DSL compiles to the same underlying pipeline bundle used by the YAML / scaffold-directory flow, but you never have to write or maintain the YAML yourself. This page is the complete reference for the DSL. For a runnable end-to-end walkthrough, see the [Quick Start](README.mdx#quick-start). For a more elaborate example with mixed local and remote steps, see [`examples/pipeline_dsl_text_pipeline.py`](https://github.com/Clarifai/clarifai-python/blob/master/examples/pipeline_dsl_text_pipeline.py) in the `clarifai-python` repo. ## Public API The DSL is exported from `clarifai.runners.pipelines`: ```python from clarifai.runners.pipelines import ( ComputeInfo, Pipeline, step, step_ref, load_pipeline_from_file, # advanced: load a Pipeline instance from a .py path ) ``` ## Defining Steps with `@step` Use the `@step` decorator to turn a Python function into a pipeline step. The function signature becomes the step's input parameters; the return value becomes the step's output. ```python from clarifai.runners.pipelines import ComputeInfo, step @step( id="prepare-text", # optional; defaults to function name with _→- requirements=["transformers>=4.0"], # optional; pip requirements for the step assets=["./text_utils.py"], # optional; files copied alongside step code compute=ComputeInfo(cpu_limit="500m", cpu_memory="500Mi"), # optional; compute requirements python_version="3.12", # optional; defaults to "3.12" secrets={"OPENAI_API_KEY": "users/me/secrets/openai-key"}, # optional; step-level secrets ) def prepare_text(input_text: str) -> str: """Normalize text before downstream processing.""" return input_text.strip().lower() ``` ### `@step` Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `id` | `str` | The step's identifier. Defaults to the function name with underscores converted to hyphens (e.g. `prepare_text` → `prepare-text`). Must be a valid Clarifai resource ID. | | `requirements` | `list[str]` or `str` | Pip requirements for the step. Accepts a list of requirement strings, a single requirement string, or a path to a `requirements.txt` file. | | `assets` | `list[str]` | Files or directories to bundle alongside the step's code. Paths are resolved relative to the source file containing the step. | | `compute` | `ComputeInfo` | Compute resources for the step (CPU, memory, accelerators). See [Compute Requirements](#compute-requirements). | | `python_version` | `str` | Python version to use inside the step's container. Defaults to `"3.12"`. | | `base_image` | `Optional[str]` | Custom base Docker image for the step's container — e.g., a CUDA-enabled image for GPU steps. Defaults to Clarifai's standard base image. | | `platform` | `Optional[str]` | Build platform for the step's container image (e.g., `"linux/amd64"`). Defaults to multi-architecture build. | | `secrets` | `dict[str, str]` | Environment variable name → secret resource path. Secrets are mounted as environment variables at step runtime. | ### Step Input Parameters A step's input parameters are inferred from its function signature. Type annotations become parameter type hints; default values become parameter defaults. ```python @step def train( dataset_path: str, # required parameter base_model: str = "google/vit-base-patch16-224", # optional with default epochs: int = 10, learning_rate: float = 0.001, ) -> str: ... ``` No separate `pipeline_step_input_params` YAML block to maintain — the function signature is the source of truth. ### Calling a Step Function Directly Step functions can be called directly outside of a `Pipeline` context. This is useful for local testing: ```python result = prepare_text("Some Input") # runs the underlying function locally # or, equivalently: result = prepare_text.test(input_text="Some Input") ``` Inside a `Pipeline` context, the same call adds a node to the DAG instead of executing the function. ## Compute Requirements `ComputeInfo` declares the compute resources for a step: ```python from clarifai.runners.pipelines import ComputeInfo ComputeInfo( cpu_limit="500m", # 500 millicores = 0.5 CPU cores cpu_memory="500Mi", # 500 MiB of memory num_accelerators=0, # number of GPUs/accelerators accelerator_type=["NVIDIA-A10G"], # optional accelerator type constraints ) ``` `ComputeInfo` is the same proto type used in `config.yaml`'s `pipeline_step_compute_info` block — see the [Compute Resources](create-api.md#compute-resources-allocation) section of the advanced YAML reference for the full set of fields. ## Composing Pipelines with `Pipeline(...)` A `Pipeline` instance is the container for your DAG. Use it as a context manager (`with Pipeline(...) as pipeline:`) so step calls register nodes against it: ```python from clarifai.runners.pipelines import Pipeline with Pipeline(id="my-pipeline") as pipeline: raw = pipeline.input("input_text", default="hello world") prepared = prepare_text(input_text=raw) summary = summarize(input_text=prepared.output()) prepared >> summary ``` After running `clarifai login`, `user_id` and `app_id` are read automatically from your CLI context — you don't need to pass them explicitly. To target a different user or app, pass them as kwargs: ```python with Pipeline(id="my-pipeline", user_id="other-user", app_id="other-app") as pipeline: ... ``` ### `Pipeline` Constructor | Parameter | Type | Description | | :--- | :--- | :--- | | `id` | `str` | The pipeline's identifier. Required. | | `user_id` | `Optional[str]` | User ID owning the pipeline. Defaults to the user from your `clarifai login` context. | | `app_id` | `Optional[str]` | App ID containing the pipeline. Defaults to the app from your `clarifai login` context. | | `visibility` | `str` | Visibility setting; defaults to `"PRIVATE"`. | If neither explicit args nor `clarifai login` context provide `user_id` and `app_id`, the constructor raises a `ValueError` pointing you to run `clarifai login` first. When the `with` block exits, the pipeline is validated automatically — missing dependencies and DAG cycles raise a `ValueError` before upload. ### Workflow Inputs `pipeline.input(name, default=None)` declares a workflow-level input parameter — a value provided at runtime when you `clarifai pipeline run`. The return value is a reference you can pass into step calls: ```python with Pipeline(id="my-pipeline") as pipeline: raw = pipeline.input("input_text", default="hello") cleaned = prepare_text(input_text=raw) ``` ### Connecting Steps There are two ways to express dependencies between steps: **1. Pass one step's output to another step's input.** This creates an implicit dependency: ```python prepared = prepare_text(input_text=raw) summary = summarize(input_text=prepared.output()) # summary depends on prepared ``` `step_node.output(param_name="result")` returns a reference to a named output of an upstream step. The default name is `"result"`. **2. Use the `>>` operator for explicit ordering.** This is useful when you need a dependency that isn't expressed by data flow, or for fan-in / fan-out: ```python # Linear: a → b → c a >> b >> c # Fan-out: prepared feeds two parallel branches prepared >> [summary, sentiment] # Fan-in (diamond DAG): both branches feed the final report step prepared >> [summary, sentiment] >> report ``` The `>>` operator returns its right-hand operand so chained expressions read naturally. ## Referencing Existing Remote Steps with `step_ref` To compose pipelines that include already-uploaded pipeline steps (your own or shared from another app), use `step_ref`. The referenced step is pinned to a specific version and is not re-uploaded with the pipeline: ```python from clarifai.runners.pipelines import step_ref # By IDs: classify_sentiment = step_ref( id="classify-sentiment", user_id="demo-user", app_id="shared-app", version_id="sentiment-v3", ) # By URL: summarize = step_ref.from_url( "https://api.clarifai.com/v2/users/demo-user/apps/shared-app/pipeline_steps/summarize/versions/summary-v1", secrets={"OPENAI_API_KEY": "users/demo-user/secrets/openai-key"}, ) ``` You can mix locally-defined steps and `step_ref` steps freely in the same pipeline. ### `step_ref` Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `id` | `str` | The remote step's ID. | | `version_id` | `str` | A specific version of the step. Required — referencing the "latest" version is intentionally not supported, to preserve reproducibility. | | `user_id` | `str` | The step's owning user. Defaults to the pipeline's `user_id`. | | `app_id` | `str` | The step's owning app. Defaults to the pipeline's `app_id`. | | `secrets` | `dict[str, str]` | Step-level secrets, same shape as `@step`. | ### `step_ref.from_url(url, *, secrets=None)` Convenience constructor that parses a versioned pipeline step URL or resource path of the form: ``` users/{user_id}/apps/{app_id}/pipeline_steps/{step_id}/versions/{version_id} ``` ## Step-Level Secrets Both `@step` and `step_ref` accept a `secrets` dictionary mapping environment variable names to [Clarifai secret](https://docs.clarifai.com/control/authentication/environment-secrets) resource paths: ```python @step( id="call-openai", secrets={"OPENAI_API_KEY": "users/me/secrets/openai-key"}, ) def call_openai(prompt: str) -> str: import os api_key = os.environ["OPENAI_API_KEY"] ... ``` Secrets are injected as environment variables at step runtime. They are not visible in your pipeline source or in the generated bundle. ## Upload Upload a pipeline directly from its `.py` file: ```bash clarifai pipeline upload my_pipeline.py ``` The CLI loads the `Pipeline` instance from the file, generates the underlying step bundles into a temporary `generated-/` directory next to the file, and uploads. You can also upload programmatically: ```python pipeline_version_id = pipeline.upload(no_lockfile=False) ``` The pipeline's source file must contain exactly one `Pipeline` instance. If you want multiple pipelines in the same project, put them in separate files. ## Generating the YAML Bundle (Codegen) To produce the underlying scaffold-directory bundle without uploading — useful for inspection, version control, or handing off to the YAML/CLI flow — there are two options. **From the CLI:** ```bash clarifai pipeline compile my_pipeline.py --output-dir ./generated-bundle ``` `clarifai pipeline compile` takes any `.py` file containing a `Pipeline` instance and writes the corresponding scaffold-directory bundle to the directory you specify. **From Python:** ```python pipeline.generate("./generated-bundle") ``` The `pipeline.generate(output_dir)` method is what `clarifai pipeline compile` calls under the hood — use it directly if you want to expose codegen from your own script: ```python # my_pipeline.py import argparse from clarifai.runners.pipelines import Pipeline, step # ... step and pipeline definitions ... def main(): parser = argparse.ArgumentParser() parser.add_argument("--out", default="./generated-pipeline") args = parser.parse_args() pipeline.generate(args.out) if __name__ == "__main__": main() ``` The generated directory has the same shape as a scaffold-directory pipeline: a root `config.yaml`, per-step subdirectories with their own `config.yaml`, `requirements.txt`, and a `1/pipeline_step.py` containing an `argparse`-wrapped version of your `@step` function. You can upload it the same way as a scaffold-directory pipeline: ```bash clarifai pipeline upload ./generated-bundle ``` ## Programmatic Run You can run a pipeline from Python without going through the CLI: ```python result = pipeline.run( inputs={"input_text": "hello world"}, timeout=3600, monitor_interval=10, nodepool_id="...", compute_cluster_id="...", ) ``` If the pipeline hasn't been uploaded yet, `run()` uploads first. See the [Manage Pipeline Runs](manage-run.md) page for monitoring, pausing, and cancelling runs after they're started. ## Validation Pipelines are validated automatically when the `with Pipeline(...) as pipeline:` block exits. Validation checks for: - **Missing dependencies** — any `>>` reference to a step not defined in the pipeline raises a `ValueError`. - **DAG cycles** — circular dependencies between steps raise a `ValueError` with the offending node. For explicit validation outside a `with` block, call `pipeline.validate()` directly. ## See Also * **[Quick Start](README.mdx#quick-start)** — minimal end-to-end DSL example. * **[Templates](templates.md)** — starter `.py` files for common workflows. * **[Advanced: YAML / config-based pipelines](create-api.md)** — the scaffold-directory authoring flow the DSL compiles down to. * **[Working example](https://github.com/Clarifai/clarifai-python/blob/master/examples/pipeline_dsl_text_pipeline.py)** — a complete DSL pipeline with helpers, remote `step_ref` steps, and a diamond DAG. --- ## Manage Pipeline Runs # Manage Pipeline Runs **Control pipeline runs in real time — monitor, pause, resume, and cancel** The `clarifai pipelinerun` command group lets you manage pipeline version runs after they have started. You can monitor execution, pause or resume long-running workflows, and cancel runs that are no longer needed. ```bash clarifai pipelinerun [OPTIONS] COMMAND [ARGS]... ``` > **Note:** You can also use the shorthand alias `pr` instead of `pipelinerun` in all commands. ## Monitor a Pipeline Run Use `monitor` to watch the current status and logs of an existing, already running [pipeline run](create-api.md#step-5-run-the-pipeline). ```bash clarifai pr monitor ``` You can also customize monitoring behavior. For example, you can wait up to 2 hours and check the status every 5 seconds: ```bash clarifai pr monitor \ --timeout=7200 \ --monitor_interval=5 ``` ## Pause a Pipeline Run Use `pause` to temporarily halt a pipeline run. Pausing is only allowed when the run is in the `Queued` or `Running` state. This is useful if you need to free up resources or temporarily stop execution. ```bash clarifai pr pause ``` ## Resume a Pipeline Run Use `resume` to continue a pipeline run that was previously paused. Resuming is only allowed when the run is in the `Paused` state. ```bash clarifai pr resume ``` ## Cancel a Pipeline Run Use `cancel` to permanently stop a pipeline run. Once canceled, a run cannot be resumed. Cancellation is only allowed if the pipeline is not already in a terminal state (such as `Completed` or `Failed`). ```bash clarifai pr cancel ``` ## Additional Tips ### Use Flags Instead of providing the `pipeline_version_run_id` as a positional argument, you can supply it explicitly as a flag. This approach improves readability and is especially useful in scripts or commands with multiple parameters. For example: ```bash clarifai pipelinerun monitor --pipeline_version_run_id= ``` ### Provide Explicit Parameters The `clarifai pipelinerun` command automatically reads all required parameters — except `pipeline_version_run_id` — from the `config-lock.yaml` file, which is generated when you [upload a pipeline](create-api.md#step-4-upload-the-pipeline). This allows you to pass the `pipeline_version_run_id` as a positional argument after the action (for example, `monitor`). However, if the `config-lock.yaml` file has been modified, is unavailable, or you're working in an environment with multiple users or pipelines, you can explicitly provide the full set of identifiers instead of relying on the lock file. For example: ```bash clarifai pr monitor \ --user_id USER_ID \ --app_id APP_ID \ --pipeline_id PIPELINE_ID \ --pipeline_version_id VERSION_ID ``` --- ## Manage Pipelines # Manage Pipelines **Manage and scale pipelines across their entire lifecycle** Managing your Clarifai Pipelines across their lifecycle keeps them organized and production-ready, so you can scale confidently and maintain consistent performance. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## List Pipelines The `list` (or `ls`) command displays all pipelines available in your specified application. This is useful for verifying that your pipeline was successfully uploaded to the Clarifai platform and discovering pipelines across different apps. It has the following options: - `--app_id` — Only list pipelines from a specific app - `--page_no` — (Optional) Page number to display (useful when there are many pipelines) - `--per_page` — (Optional) Number of pipelines to return per page ### List All Pipelines Run the following command to list all the pipelines for the user. ```bash # List pipelines from a specific app clarifai pipeline list --app_id my-app # List pipelines with pagination clarifai pipeline list --app_id my-app --page_no 2 --per_page 10 ``` Example Output ```text clarifai pipeline list --app_id pipelines-1 ID USER_ID APP_ID VERSION_ID DESCRIPTION CREATED_AT MODIFIED_AT hello-world-pipeline alfrick pipelines-1 48862434906f482e94a2ec638a4233a1 2025-11-26T14:15:51Z 2025-11-26T14:15:51Z clarifai pipeline list --app_id pipelines-1 --page_no 1 --per_page 10 ID USER_ID APP_ID VERSION_ID DESCRIPTION CREATED_AT MODIFIED_AT hello-world-pipeline alfrick pipelines-1 48862434906f482e94a2ec638a4233a1 2025-11-26T14:15:51Z 2025-11-26T14:15:51Z ``` ### List All Pipeline Steps Run the following command to list only the [pipeline steps](create-api.md#pipeline-steps) for the user. ```bash # clarifai pipelinestep list --app_id my-app # Or: clarifai ps ls --app_id my-app ``` Example Output ```text PIPELINE_STEP_ID USER_ID APP_ID VERSION_ID DESCRIPTION CREATED_AT MODIFIED_AT stepA alfrick pipelines-1 e7b2839d02b0463e8a45e7a37afc767b Pipeline step version 2025-11-28T05:23:46Z 2025-11-28T05:23:53Z stepB alfrick pipelines-1 5e93c74ef8ae456ab353aa5e60e46f97 Pipeline step version 2025-11-26T14:15:36Z 2025-11-26T14:15:43Z stepA alfrick pipelines-1 36e752e546334fa28d73bcbdc86d37a7 Pipeline step version 2025-11-26T14:14:02Z 2025-11-26T14:15:29Z ``` ## Validate Lock File The `validate-lock` command verifies that your root [`config-lock.yaml`](create-api.md#step-4-upload-the-pipeline) file matches the expected schema. It checks that all referenced steps, versions, and configurations are valid. You can use this command before re-uploading or running your pipeline to catch configuration errors early and ensure everything is consistent. ```bash # Validate lock file in the current directory clarifai pipeline validate-lock # Validate a specific lock file clarifai pipeline validate-lock ./my-pipeline/config-lock.yaml ``` > **Note:** If no path is provided, the CLI automatically looks for a `config-lock.yaml` file in your current working directory. Example Output ```text clarifai pipeline validate-lock [INFO] 15:45:54.415983 ✅ Lockfile config-lock.yaml is valid | thread=8329666752 [INFO] 15:45:54.416070 Pipeline: hello-world-pipeline | thread=8329666752 [INFO] 15:45:54.416102 User: alfrick | thread=8329666752 [INFO] 15:45:54.416129 App: pipelines-1 | thread=8329666752 [INFO] 15:45:54.416149 Version: 48862434906f482e94a2ec638a4233a1 | thread=8329666752 ``` --- ## Pipeline Templates # Pipeline Templates **Start from a working pipeline template for common workflows** Pipeline templates are starter projects for common AI/ML workflows — LoRA fine-tuning, image classification, object detection, model quantization, evaluation. They give you a working pipeline with sensible defaults that you can edit and run, instead of starting from a blank file. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## Available CLI-First Templates These five templates are designed to be initialized and run from the Clarifai CLI. All have working defaults so the first run requires no edits. | Template | What it does | | :--- | :--- | | [`lora-pipeline-unsloth-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/lora-pipeline-unsloth-quick-start) | Fine-tune an LLM with LoRA using [Unsloth](https://github.com/unslothai/unsloth). Defaults to `unsloth/Qwen3-0.6B` on `mlabonne/FineTome-100k`. | | [`llmcompressor-pipeline-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/llmcompressor-pipeline-quick-start) | Quantize a model with [`llm-compressor`](https://github.com/vllm-project/llm-compressor) for reduced memory and compute footprint. | | [`classifier-pipeline-resnet-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/classifier-pipeline-resnet-quick-start) | Train an image classifier (ResNet) on your dataset. | | [`detector-pipeline-yolof-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/detector-pipeline-yolof-quick-start) | Train an object detector (YOLOF) on your dataset. | | [`detector-pipeline-eval-yolof-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/detector-pipeline-eval-yolof-quick-start) | Evaluate a YOLOF object detector against ground-truth labels. | ## Discover Templates From the CLI Use `clarifai pipelinetemplate list` to see all registered templates: clarifai pipelinetemplate list Example Output ```text NAME TYPE classifier-pipeline-resnet classifier classifier-pipeline-resnet-quick-start classifier detector-pipeline-dfine detector detector-pipeline-eval-yolof-quick-start detector detector-pipeline-yolof detector detector-pipeline-yolof-quick-start detector llmcompressor-pipeline-quick-start llmcompressor lora-pipeline-unsloth-quick-start lora Found 8 template(s) total Available types: classifier, detector, llmcompressor, lora ``` > **Note:** The `ls` alias works the same as `list` — `clarifai pipelinetemplate ls` is equivalent to `clarifai pipelinetemplate list`. You'll notice the CLI returns three additional templates that aren't in the table above (`classifier-pipeline-resnet`, `detector-pipeline-dfine`, `detector-pipeline-yolof`). Those are **UI-triggerable templates** — they're the back-end implementations of the model-training workflows in the Clarifai web app, not standalone CLI experiences. See [Model-type guides](#model-type-guides) below for those. Templates are pulled from the [official Clarifai pipeline-examples repository](https://github.com/Clarifai/pipeline-examples). To point at a custom or private template repository, set `CLARIFAI_PIPELINE_TEMPLATES_GIT_REPO_URL` before running the command. ## Initialize a Project From a Template Use `clarifai pipeline init --template ` to scaffold a working starter project: clarifai pipeline init --template lora-pipeline-unsloth-quick-start The command creates a project in the current directory with all configuration and step files in place. You can then edit the generated files to point at your data and customize behavior, and upload with `clarifai pipeline upload`. ### Override Defaults at Initialization Use `--set key=value` to override template parameters at init time. Each `--set` flag sets one parameter; you can pass it multiple times: ```bash clarifai pipeline init --template lora-pipeline-unsloth-quick-start \ --set base_model_name="unsloth/Llama-3.2-1B-Instruct" \ --set num_epochs=3 ``` This is the scriptable / agent-friendly entry point for scaffolding — no interactive prompts required. ### Specify User and App Context By default, the scaffold uses your authenticated user (from `clarifai login`). You can override with `--user_id` and `--app_id` to scaffold for a different user or app: ```bash clarifai pipeline init --template lora-pipeline-unsloth-quick-start \ --user_id your_user_id \ --app_id your_app_id ``` ## What Gets Scaffolded Templates today scaffold a YAML / config-based pipeline project — a directory containing a root `config.yaml`, per-step subdirectories, and a `pipeline_step.py` for each step. The full file structure is documented under [Advanced: YAML / config-based pipelines](create-api.md#step-3-modify-the-files). > **Forward-looking note:** Clarifai is moving toward Python-first pipeline authoring via the [Pipeline DSL](dsl-reference.md). Templates are being migrated to scaffold DSL-based `.py` starters in addition to YAML scaffold directories. The CLI commands documented on this page will remain stable; what changes is the shape of the generated files. ## Use Templates Without `init` You don't have to use the `init` command to use a template. Each template is just a project layout in the [pipeline-examples repository](https://github.com/Clarifai/pipeline-examples) — you can also clone or download templates directly, edit them in place, and `clarifai pipeline upload` them like any other pipeline. For workflows where you'd rather author from scratch in Python without a starter, see the [Pipeline DSL reference](dsl-reference.md). ## Model-Type Guides The CLI-first templates above have UI-triggerable counterparts (`classifier-pipeline-resnet`, `detector-pipeline-dfine`, `detector-pipeline-yolof`) that power the model-training workflows in the Clarifai web app. For an end-to-end walkthrough that uses those, see: * [Train a Visual Classifier](https://docs.clarifai.com/create/models/visual-classifier) * [Train a Visual Detector](https://docs.clarifai.com/create/models/visual-detector) * [Training Models overview](https://docs.clarifai.com/create/models/) — index of model-type guides The model-type guides take a "I want to train a specific kind of model from the UI" angle; this page is the canonical reference for the CLI-first template surface. --- ## Toolkits # Toolkits **Download and run AI models locally with Clarifai toolkits** Toolkits let you run large language models (LLMs) and other generative AI models locally on your machine. They provide a standardized way to initialize a new Clarifai model directory structure that’s fully compatible with the Clarifai platform. Using toolkits, you can easily download and set up models from popular repositories such as Hugging Face or Ollama, making it simple to prepare them for deployment or integration with Clarifai. To download and initialize a model using a toolkit we support, run: ```bash clarifai model init --toolkit ``` import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Hugging Face # Hugging Face **Download and run Hugging Face models locally and make them available via a public API** [Hugging Face](https://huggingface.co/) is an open-source platform for sharing, exploring, and collaborating on a wide range of pre-trained models and related assets. With Clarifai’s [Local Runners](README.mdx), you can run these models directly on your machine, expose them securely via a public URL, and tap into Clarifai’s powerful platform — all while preserving the speed, privacy, and control of local deployment. > **Note:** After initializing a model using the Hugging Face toolkit, you can [upload](https://docs.clarifai.com/compute/upload/#step-4-upload-the-model-to-clarifai) it to Clarifai to leverage the platform’s capabilities. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import HFModelPy from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_model_py.py"; import HFConfig from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_config.yaml"; import HFSupportedModels from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_supported_models.txt"; import ClarifaiLogin from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_clarifai_login.txt"; import RequirementsTXT from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_requirements_txt.txt"; import HFModelInit from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_model_init.txt"; import StartRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_start_runner.txt"; import TestRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_test_runner.py"; ## Step 1: Perform Prerequisites ### Get User ID and PAT Start by [logging in](https://clarifai.com/login) to your existing Clarifai account or [signing up](https://clarifai.com/signup) for a new one. Once logged in, you’ll need your **Personal Access Token (PAT)** for authentication: - In the collapsible left sidebar, select **Settings** and choose **Secrets** to generate or copy your [PAT](https://docs.clarifai.com/control/authentication/pat). You can then set the PAT as an environment variable using `CLARIFAI_PAT`. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Install Clarifai CLI Install the latest version of the [Clarifai CLI](https://docs.clarifai.com/sdk/cli) tool. It includes built-in support for Local Runners. pip install --upgrade clarifai > **Note:** You'll need **[Python 3.11 or 3.12](https://docs.clarifai.com/resources/api-overview/python-sdk#python-requirements)** installed to successfully run the Local Runners. ### Get Hugging Face Token A Hugging Face access token is required to authenticate with Hugging Face services, especially when downloading models from private or restricted repositories. You can create one by following [these instructions](https://huggingface.co/docs/hub/en/security-tokens). Once you have it, provide the token either in your model’s `config.yaml` file (as described [below](#configyaml)) or as an environment variable. > **Note**: If `hf_token` is not specified in the `config.yaml` file, the CLI automatically falls back to the `HF_TOKEN` environment variable to authenticate with Hugging Face. export HF_TOKEN="YOUR_HF_ACCESS_TOKEN_HERE" set HF_TOKEN="YOUR_HF_ACCESS_TOKEN_HERE" ### Install Hugging Face Hub The [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) library is used under the hood to fetch files from the Hugging Face Hub. While you won’t interact with it directly, it’s required for downloading the models and resources automatically. pip install huggingface_hub ## Step 2: Initialize a Model With the Clarifai CLI, you can download and set up any supported Hugging Face model directly in your local environment. Use the `--model-name` flag to specify a HuggingFace model: clarifai model init --toolkit huggingface --model-name Qwen/Qwen2-0.5B This creates a `Qwen2-0.5B/` directory with all required files pre-configured. The CLI auto-selects an appropriate GPU instance based on the model's VRAM requirements. > **Note:** You can initialize a model in a specific location by passing a [`MODEL_PATH`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-init). Example Output {HFModelInit} You can customize or optimize the model by editing the generated files as needed. :::tip To initialize with a default model ([`unsloth/Llama-3.2-1B-Instruct`](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct)), omit `--model-name`: clarifai model init --toolkit huggingface Supported Models {HFSupportedModels} ::: > **Note:** Some models are quite large and require substantial memory or GPU resources. Ensure your machine has sufficient compute capacity to load and run the model locally before initializing it. The generated structure includes: ``` ├── 1/ │ └── model.py ├── requirements.txt └── config.yaml ``` ### `model.py` Example: model.py {HFModelPy} The [`model.py`](https://docs.clarifai.com/compute/upload/#prepare-modelpy) file, which is located inside the `1` folder, defines the logic for the Hugging Face model, including how predictions are made. ### `config.yaml` Example: config.yaml {HFConfig} The `config.yaml` file defines your Hugging Face model’s configuration in a simplified format: - **`model.id`** — A unique identifier for your model. Auto-generated from the Hugging Face model name when you use `--model-name`. - **`build_info.python_version`** — The Python version to use (default: `3.11`). - **`compute.instance`** — The GPU instance type, auto-selected based on the model’s VRAM requirements. Run `clarifai list-instances` to see all available options. - **[`checkpoints`](https://docs.clarifai.com/compute/upload/#hugging-face-model-checkpoints)** — Defines how to retrieve model weights. If you’re using a gated repository, add your Hugging Face access token via `hf_token` or set the `HF_TOKEN` environment variable. > `user_id` and `app_id` are auto-filled from your [active context](https://docs.clarifai.com/resources/api-overview/cli#clarifai-config) at deploy time. You don’t need to add them manually. > > **Tip:** Use `when: runtime` (the default) for large models to reduce image size and improve load times. ### `requirements.txt` Example: requirements.txt {RequirementsTXT} The `requirements.txt` file lists Python dependencies needed by your model. If you haven’t installed them yet, run the following command to install the dependencies: pip install -r requirements.txt ## Step 3: Log In to Clarifai Run the following command to log in to the Clarifai platform, create a configuration [context](https://docs.clarifai.com/compute/local-runners/#step-2-create-a-context-optional), and establish a connection: ```bash clarifai login ``` You’ll be prompted to provide a few details for authentication: * **User ID** – Enter your Clarifai user ID. * **PAT** – Enter your Clarifai PAT. If you’ve already set the `CLARIFAI_PAT` environment variable, type `ENVVAR` to use it automatically. * **Context name** – Assign a custom name to this configuration context, or press Enter to accept the default name, `"default"`. This is helpful if you manage multiple environments or configurations. Example Output {ClarifaiLogin} ## Step 4: Serve the Model Locally Start the model using `clarifai model serve`: ```bash clarifai model serve ``` > **Note:** The older `clarifai model local-runner` command still works as an alias. If the necessary context configurations aren’t detected, the CLI will guide you through creating them using default values. This setup ensures all required components — such as compute clusters, nodepools, and deployments — are properly included in your configuration context, which are described [here](https://docs.clarifai.com/compute/local-runners/#step-2-create-a-context-optional). Example Output {StartRunner} ## Step 5: Test Your Runner Once the local runner starts, it provides a sample client code snippet you can use for quick testing. You can run the snippet in a separate terminal within the same directory to see the model’s response. Here’s an example snippet: {TestRunner} ## Deploy to Cloud After testing locally, you can deploy your Hugging Face model to Clarifai's cloud compute with a single command. All infrastructure (compute cluster, nodepool, deployment) is created automatically. ### Step 1: Initialize If you haven't already, scaffold a Hugging Face model project: clarifai model init --toolkit huggingface --model-name Qwen/Qwen2-0.5B The CLI auto-selects an appropriate GPU instance based on the model's VRAM requirements. ### Step 2: Deploy clarifai model deploy ./Qwen2-0.5B --instance g5.xlarge If your `config.yaml` already has a `compute.instance` value (set during `init`), you can omit the `--instance` flag: clarifai model deploy ./Qwen2-0.5B To override the instance with a larger GPU, use the `--instance` flag — it always takes priority over the config: clarifai model deploy ./Qwen2-0.5B --instance g5.xlarge # override to a larger GPU Browse available GPU instances with `clarifai list-instances` or `clarifai model deploy --instance-info`. ### Step 3: Monitor and Manage ```bash # Check deployment status clarifai model status --deployment # Stream live logs clarifai model logs --deployment # Run predictions clarifai model predict user/app/models/Qwen2-0.5B "Explain AI in one sentence" # Clean up when done clarifai model undeploy --deployment ``` For the full deploy options reference, see the [CLI Reference](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy). --- ## LM Studio # LM Studio **Download and run LM Studio models locally and expose them via a public API** [LM Studio](https://lmstudio.ai/) is a desktop application that lets you run open-source LLMs locally on your machine. Combined with Clarifai's Local Runners, you can serve LM Studio models from your machine, expose them via a public API, and access them through the Clarifai platform — all while keeping the speed, privacy, and control of local inference. > **Important:** Clarifai's LM Studio integration currently supports **macOS only** (Apple devices). For other platforms, consider using [Ollama](https://docs.clarifai.com/compute/toolkits/ollama) or [vLLM](https://docs.clarifai.com/compute/toolkits/vllm) instead. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import LMStudioInit from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_init.txt"; import LMStudioLogin from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_login.txt"; import LMStudioLocalRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_local_runner.txt"; import LMStudioTestRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_test_runner.py"; import LMStudioModel from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_model.py"; import LMStudioConfig from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_config.yaml"; import LMStudioRequirements from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/lm_studio_requirements.txt"; ## Step 1: Install Prerequisites ### Install LM Studio Go to the [LM Studio website](https://lmstudio.ai/download) and install the desktop application for macOS. After installing, enable the `lms` CLI tool so Clarifai can detect your models: ```sh ~/.lmstudio/bin/lms bootstrap ``` Restart your terminal, then verify with `lms --version`. Keep LM Studio open and running before starting the local runner — it provides the model runtime that Clarifai connects to. ### Install Clarifai pip install --upgrade clarifai > **Note:** Python 3.11 or 3.12 is required. The `openai` package is included with `clarifai`. ## Step 2: Log In clarifai login You'll be prompted for your user ID and [PAT](https://docs.clarifai.com/control/authentication/pat). This saves your credentials locally so you don't need to set environment variables manually. Example Output {LMStudioLogin} ## Step 3: Initialize a Model Scaffold a model project using any model from the [LM Studio Model Catalog](https://lmstudio.ai/models): clarifai model init --toolkit lmstudio --model-name google/gemma-3-4b The CLI auto-detects LM Studio models already downloaded on your machine. Change `--model-name` to any other model from the catalog. Example Output {LMStudioInit} This creates a `./gemma-3-4b/` directory: ``` gemma-3-4b/ ├── 1/ │ └── model.py # LM Studio inference logic ├── requirements.txt # Python dependencies └── config.yaml # Model config (user_id/app_id auto-filled from login) ``` > **Note:** Some models are very large and may require significant memory. Check your machine's capacity before initializing. model.py {LMStudioModel} config.yaml {LMStudioConfig} requirements.txt {LMStudioRequirements} ## Step 4: Serve Locally Start the model as a local runner: clarifai model serve ./gemma-3-4b > **Note:** Make sure LM Studio is open and running before starting the runner. Add `-v` for verbose logs. Example Output {LMStudioLocalRunner} When ready, the CLI prints: - A model URL for API calls - A Playground link for browser-based testing - A sample code snippet Press `Ctrl+C` to stop the runner. ## Step 5: Run Inference While the local runner is active, test it using the OpenAI-compatible client: {LMStudioTestRunner} Or use the Clarifai CLI: ```sh clarifai model predict https://clarifai.com//local-runner-app/models/local-runner-model "Explain AI in one sentence" ``` You can also open the **[Runners](https://clarifai.com/compute/runners)** dashboard, find your runner, and select **Open in Playground** from the three-dot menu. When you're done, close the terminal running the local runner to shut it down. --- ## Ollama # Ollama **Download and run Ollama models locally and expose them via a public API** Ollama is an open-source tool that lets you run LLMs directly on your local machine. Combined with Clarifai's Local Runners, you can serve Ollama models from your machine, expose them via a public API, and access them through the Clarifai platform — all while keeping the speed, privacy, and control of local inference. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import OllamaInit from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/ollama-init.txt"; import OllamaRun from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/ollama-run.txt"; import OllamaModelPy from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/ollama-modelpy.py"; import OllamaRequirements from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/ollama-requirements.txt"; import OllamaConfig from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/ollama-config.txt"; ## Step 1: Install Prerequisites ### Install Ollama Go to the [Ollama website](https://ollama.com/download) and install the appropriate version for your system (macOS, Windows, or Linux). > **Note:** On Windows, restart your machine after installing to ensure environment variables are applied. ### Install Clarifai pip install --upgrade clarifai > **Note:** Python 3.11 or 3.12 is required. The `openai` package is included with `clarifai`. ## Step 2: Log In clarifai login You'll be prompted for your user ID and [PAT](https://docs.clarifai.com/control/authentication/pat). This saves your credentials locally so you don't need to set environment variables manually. ## Step 3: Initialize a Model Scaffold a model project using any model from the [Ollama library](https://ollama.com/library): clarifai model init --toolkit ollama --model-name gpt-oss:20b Example Output {OllamaInit} This creates a `./gpt-oss:20b/` directory with three files: ``` gpt-oss:20b/ ├── 1/ │ └── model.py # Ollama inference logic ├── requirements.txt # Python dependencies └── config.yaml # Model config (user_id/app_id auto-filled from login) ``` :::tip Omitting `--model-name` will install the default model (`llama3.2`): ```sh clarifai model init --toolkit ollama ``` Use `ollama list` to see downloaded models and `ollama rm` to remove one. ::: > **Note:** Some models are very large and may require significant memory or GPU resources. Check your machine's capacity before initializing. model.py {OllamaModelPy} requirements.txt {OllamaRequirements} config.yaml {OllamaConfig} ## Step 4: Serve Locally Start the model as a local runner: clarifai model serve "./gpt-oss:20b" > **Note:** The first run pulls the model weights from Ollama, which may take a few minutes. Add `-v` for verbose logs. Example Output {OllamaRun} When ready, the CLI prints: - A model URL for API calls - A Playground link for browser-based testing - A sample code snippet Press `Ctrl+C` to stop the runner. ## Step 5: Run Inference While the local runner is active, test it using the OpenAI-compatible client: ```python import os from openai import OpenAI client = OpenAI( base_url="https://api.clarifai.com/v2/ext/openai/v1", api_key=os.environ['CLARIFAI_PAT'], ) response = client.chat.completions.create( model="https://clarifai.com//local-runner-app/models/local-runner-model", messages=[ {"role": "system", "content": "Talk like a pirate."}, {"role": "user", "content": "How do I check if a Python object is an instance of a class?"}, ], stream=False, # stream=True also works ) print(response) ``` Or use the Clarifai CLI: ```sh clarifai model predict https://clarifai.com//local-runner-app/models/local-runner-model "Explain AI in one sentence" ``` You can also open the **[Runners](https://clarifai.com/compute/runners)** dashboard, find your runner, and select **Open in Playground** from the three-dot menu. ![](/img/others/runners-dashboard-ollama.png) When you're done, close the terminal running the local runner to shut it down. ## Additional Resources - [More Ollama Python examples](https://github.com/ollama/ollama-python/tree/main/examples) - [Clarifai inference examples with Ollama](https://docs.clarifai.com/compute/inference/clarifai/api) - [Ollama runner example on GitHub](https://github.com/Clarifai/runners-examples/tree/main/local-runners/ollama-model-upload) - [YouTube: Running OpenAI's GPT-OSS-20B with Ollama](https://www.youtube.com/watch?v=TfS2p8LZYBE) --- ## OpenAI # OpenAI **Run OpenAI-compatible models locally and expose them via a public API** OpenAI's API specification has become the industry standard for interacting with large language models. With Clarifai's [Local Runners](https://docs.clarifai.com/compute/local-runners/), you can deploy any OpenAI-compatible model locally, whether it's from OpenAI's official services, open-source alternatives, or your own fine-tuned models, and make them available via secure public endpoints. This approach gives you the flexibility of OpenAI's familiar API interface while maintaining data privacy, reducing latency, and having full control over your deployment environment. > **Note:** After setting up your OpenAI-compatible model locally, you can [upload](https://docs.clarifai.com/compute/upload/#step-4-upload-the-model-to-clarifai) it to Clarifai to leverage the platform's capabilities, such as versioning, monitoring, and auto‑scaling. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import OpenAIModelPy from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_model_py.py"; import OpenAIConfig from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_config.yaml"; import OpenAIRequirements from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_requirements.txt"; import OpenAIModelInit from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_model_init.txt"; import StartRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_start_runner.txt"; import TestRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_test_runner.py"; import OpenAILogin from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/openai_login.txt"; import OpenAIResponse from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/open_ai_response.txt"; ## Step 1: Perform Prerequisites ### Get User ID and PAT Start by [logging in](https://clarifai.com/login) to your existing Clarifai account or [signing up](https://clarifai.com/signup) for a new one. Once logged in, you'll need your **Personal Access Token (PAT)** for authentication: - In the collapsible left sidebar, select **Settings** and choose **Secrets** to generate or copy your [PAT](https://docs.clarifai.com/control/authentication/pat). You can then set the PAT as an environment variable using `CLARIFAI_PAT`. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Install Clarifai CLI Install the latest version of the [Clarifai CLI](https://docs.clarifai.com/sdk/cli) tool. It includes built-in support for Local Runners. pip install --upgrade clarifai > **Note:** You'll need **[Python 3.11 or 3.12](https://docs.clarifai.com/resources/api-overview/python-sdk#python-requirements)** installed to successfully run the Local Runners. ### Install the OpenAI Package Install the `openai` package — it’s required to perform inference with models that support the [OpenAI-compatible](https://docs.clarifai.com/compute/inference/#predict-with-openai-compatible-format) format. pip install openai ### Set Up Your OpenAI-Compatible Server Before initializing the model, ensure you have an OpenAI-compatible server running locally. Popular options include: - [vLLM](https://docs.clarifai.com/compute/toolkits/vllm#install-vllm) — Usually runs at `http://localhost:8000/v1` (default port: 8000, unless you specify `--port`). - [LM Studio](https://docs.clarifai.com/compute/toolkits/lmstudio#install-lm-studio) — Usually at `http://localhost:1234/v1` (default port: 1234, shown in the LM Studio UI) - [Ollama](https://docs.clarifai.com/compute/toolkits/ollama#install-ollama) (OpenAI mode) — Usually at `http://localhost:11434/v1` (default port: 11434, unless changed via the config or startup flags) You’ll use this address in the [`model.py`](#modelpy) file. ## Step 2: Initialize a Model With the Clarifai CLI, you can set up any OpenAI-compatible model to work with your local server. The command below scaffolds a default OpenAI-compatible model template. You can pass a `MODEL_PATH` to control where the directory is created: clarifai model init --toolkit openai my-wrapper > **Note:** You can initialize a model in a specific location by passing a [`MODEL_PATH`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-init). Example Output {OpenAIModelInit} This command generates a model directory structure that's compatible with the Clarifai platform and configured to work with OpenAI-compatible APIs. The generated structure includes: ``` ├── 1/ │ └── model.py ├── requirements.txt └── config.yaml ``` ### `model.py` Example: model.py {OpenAIModelPy} The [`model.py`](https://docs.clarifai.com/compute/upload/#prepare-modelpy) file, located inside the `1` folder, acts as the bridge between Clarifai’s model execution environment and your local (or remote) OpenAI-compatible server. It includes an extension of the Clarifai’s `OpenAIModelClass`. This base class is designed specifically for wrapping OpenAI-compatible model servers and exposing them through Clarifai’s inference infrastructure. This class implements the following: - The `OpenAI` client, which connects to your model server (e.g. vLLM, LM Studio, Ollama) via its `/v1` API endpoint. - The `predict()` method, which handles standard (non-streaming) chat completions. - The `generate()` method, which supports streaming token generation. - `build_openai_messages()`, which automatically converts Clarifai inputs into OpenAI-compatible message format. These are the key components you can configure: - `base_url`: Your local model server endpoint address (default: `http://localhost:8000/v1`). Update this to match your server’s address and port. - `api_key`: Required only when calling the model through OpenAI’s hosted API. If you’re using a local OpenAI-compatible server, an API key isn’t needed — you can simply provide any dummy value. - `model`: The ID of the model to use. You can leave this as is to let it be automatically detected from your OpenAI-compatible server, or explicitly set it to a specific model ID (for example, `"gpt-4"`). ### `config.yaml` Example: config.yaml {OpenAIConfig} The `config.yaml` file defines your OpenAI-compatible model’s configuration in a simplified format: - **`model.id`** — A unique identifier for your model. - **`model.model_type_id`** — Set to `"openai"` for OpenAI-compatible models. - **`compute.instance`** — The GPU instance type for deployment. Run `clarifai list-instances` to see all available options. - **`checkpoints`** — (Optional) Uncomment to auto-download model checkpoints from Hugging Face at runtime. > `user_id` and `app_id` are auto-filled from your [active context](https://docs.clarifai.com/resources/api-overview/cli#clarifai-config) at deploy time. You don’t need to add them manually. > **When to use checkpoints:** Most OpenAI models are accessed via the API, so you won’t need a [checkpoints](https://docs.clarifai.com/compute/toolkits/hf#configyaml) block. If you are serving a self‑hosted Hugging Face model, you can uncomment the checkpoints section and set the required values. ### `requirements.txt` Example: requirements.txt {OpenAIRequirements} The `requirements.txt` file specifies all the Python dependencies your model needs to run. If these packages are not already installed in your environment, install them by running the following command: pip install -r requirements.txt ## Step 3: Log In to Clarifai Run the following command to log in to the Clarifai platform, create a configuration [context](https://docs.clarifai.com/compute/local-runners/#step-2-create-a-context-optional), and establish a connection: ```bash clarifai login ``` You'll be prompted to provide: * **User ID** – Enter your Clarifai user ID. * **PAT** – Enter your Clarifai PAT. If you've already set the `CLARIFAI_PAT` environment variable, type `ENVVAR` to use it automatically. * **Context name** – Assign a custom name to this configuration context, or press Enter to accept the default name, `"default"`. Example Output {OpenAILogin} ## Step 4: Serve the Model Locally Start the model using `clarifai model serve`: ```bash clarifai model serve ``` > **Note:** The older `clarifai model local-runner` command still works as an alias. The CLI will guide you through creating any necessary context configurations with default values, ensuring all components (compute clusters, nodepools, deployments) are properly set up. Example Output {StartRunner} > **Tip:** If your underlying model is running on a specific port (like 8000), ensure your `model.py` points to that port, and that the Local Runner does not try to bind to the same port. ## Step 5: Test Your Runner Once the local runner starts, it provides a sample client code snippet for testing. You can run this in a separate terminal within the same directory. Here's an example test snippet: {TestRunner} Example Output {OpenAIResponse} That’s it! When you’re done testing, simply stop the terminal running the local development runner and the process hosting your OpenAI-compatible server. --- ## SGLang # SGLang **Serve any HuggingFace LLM locally or deploy to cloud compute** [SGLang](https://github.com/sgl-project/sglang) is a high-performance open-source inference engine for LLMs with support for structured generation and multimodal reasoning. With Clarifai, you can deploy SGLang models to cloud GPUs with a single command. The workflow is: **init → deploy** > **Important:** SGLang requires **Linux with an NVIDIA GPU** (Ampere or newer, compute capability >= 8.0). It does not run on macOS or Windows. For local testing on non-Linux machines, use [vLLM](https://docs.clarifai.com/compute/toolkits/vllm) or [Ollama](https://docs.clarifai.com/compute/toolkits/ollama) instead. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import SGLModelInit from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/sgl_model_init.txt"; import ClarifaiLogin from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/sgl_clarifai_login.txt"; import SGLModelPy from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/sgl_model_py.py"; import SGLConfig from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/sgl_config.yaml"; import RequirementsTXT from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/sgl_requirements_txt.txt"; import TestRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/sgl_test_runner.py"; ## Step 1: Install Clarifai pip install --upgrade clarifai > **Note:** Python 3.11 or 3.12 is required. The `openai` package is included with `clarifai`. ## Step 2: Log In clarifai login You'll be prompted for your user ID and [PAT](https://docs.clarifai.com/control/authentication/pat). This saves your credentials locally so you don't need to set environment variables manually. Example Output {ClarifaiLogin} ## Step 3: Initialize a Model Scaffold a model project using a HuggingFace model name: clarifai model init --toolkit sglang --model-name Qwen/Qwen3-4B The CLI auto-selects an Ampere+ GPU instance based on the model's VRAM requirements. You can initialize any model [supported by SGLang](https://docs.sglang.ai/references/supported_models.html) — just change `--model-name` to a different HuggingFace repo ID. Example Output {SGLModelInit} This creates a `./Qwen3-4B/` directory: ``` Qwen3-4B/ ├── 1/ │ └── model.py # SGLang inference logic ├── requirements.txt # Lightweight deps (SGLang is pre-installed in the base image) └── config.yaml # Model config (user_id/app_id auto-filled from login) ``` > **Note:** For private or gated models (Llama, Gemma, etc.), set `HF_TOKEN` in your environment before initializing: > ```sh > export HF_TOKEN=your_token_here > ``` model.py {SGLModelPy} config.yaml {SGLConfig} The `config.yaml` has 4 sections: - **`model.id`** — Auto-generated from the HuggingFace model name - **`build_info.image`** — `lmsysorg/sglang:latest` (SGLang, PyTorch, and CUDA pre-installed) - **`compute.instance`** — Auto-selected based on estimated VRAM (Ampere+ only). Run `clarifai list-instances` to see all options - **`checkpoints`** — HuggingFace model weights config. Add `hf_token` here or set `HF_TOKEN` for gated models requirements.txt {RequirementsTXT} ## Step 4: Deploy to Cloud Deploy to dedicated cloud compute. Clarifai automatically provisions all required infrastructure. Since `config.yaml` already has a `compute.instance` value (auto-selected during `init`), you can deploy directly: clarifai model deploy ./Qwen3-4B To override the instance type: clarifai model deploy ./Qwen3-4B --instance g5.xlarge To see all available instance types and pricing: clarifai list-instances > **Tip:** If you have a local Linux GPU and want to test before deploying, run `clarifai model serve ./Qwen3-4B --mode env` first. > **Note:** Clarifai automatically optimizes replica routing using KV cache affinity, improving throughput for shared system prompts, RAG pipelines, and multi-turn conversations. See [Request Routing](https://docs.clarifai.com/compute/inference/routing) for details. ## Step 5: Run Inference {TestRunner} Or use the Clarifai CLI: ```sh clarifai model predict https://clarifai.com//main/models/Qwen3-4B "Explain AI in one sentence" ``` ### Manage Your Deployment ```bash # Stream live logs clarifai model logs --deployment # Check status clarifai model status --deployment # Remove deployment when done (stops billing) clarifai model undeploy --deployment ``` For the full CLI reference, see [CLI Reference](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy). --- ## vLLM # vLLM **Serve any HuggingFace LLM locally or deploy to cloud compute** [vLLM](https://docs.vllm.ai/en/latest/) is a high-performance open-source inference engine for LLMs. With Clarifai, you can deploy vLLM models to cloud GPUs with a single command. The workflow is: **init → deploy** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import VLLMInit from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_init.txt"; import VLLMLogin from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_login.txt"; import VLLMLocalRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_local_runner.txt"; import VLLMTestRunner from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_test_runner.py"; import VLLMModel from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_model.py"; import VLLMConfig from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_config.yaml"; import VLLMRequirements from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/vllm_requirements.txt"; ## Step 1: Install Clarifai pip install --upgrade clarifai > **Note:** Python 3.11 or 3.12 is required. The `openai` package is included with `clarifai`. ## Step 2: Log In clarifai login You'll be prompted for your user ID and [PAT](https://docs.clarifai.com/control/authentication/pat). This saves your credentials locally so you don't need to set environment variables manually. Example Output {VLLMLogin} ## Step 3: Initialize a Model Scaffold a model project using a HuggingFace model name: clarifai model init --toolkit vllm --model-name Qwen/Qwen3-0.6B The CLI auto-selects the optimal GPU instance based on the model's VRAM requirements. You can initialize any model [supported by vLLM](https://docs.vllm.ai/en/v0.9.2/models/supported_models.html) — just change `--model-name` to a different HuggingFace repo ID. Example Output {VLLMInit} This creates a `./Qwen3-0.6B/` directory: ``` Qwen3-0.6B/ ├── 1/ │ └── model.py # vLLM inference logic ├── requirements.txt # Lightweight deps (vLLM is pre-installed in the base image) └── config.yaml # Model config (user_id/app_id auto-filled from login) ``` > **Note:** For private or gated models (Llama, Gemma, etc.), set `HF_TOKEN` in your environment before initializing: > ```sh > export HF_TOKEN=your_token_here > ``` model.py {VLLMModel} config.yaml {VLLMConfig} The `config.yaml` has 4 sections: - **`model.id`** — Auto-generated from the HuggingFace model name - **`build_info.image`** — `vllm/vllm-openai:latest` (vLLM, PyTorch, and CUDA pre-installed) - **`compute.instance`** — Auto-selected based on estimated VRAM. Run `clarifai list-instances` to see all options - **`checkpoints`** — HuggingFace model weights config. Add `hf_token` here or set `HF_TOKEN` for gated models requirements.txt {VLLMRequirements} ## Step 4: Deploy to Cloud Deploy to dedicated cloud compute. Clarifai automatically provisions all required infrastructure. Since `config.yaml` already has a `compute.instance` value (auto-selected during `init`), you can deploy directly: clarifai model deploy ./Qwen3-0.6B To override the instance type: clarifai model deploy ./Qwen3-0.6B --instance a10g To see all available instance types and pricing: clarifai list-instances > **Tip:** If you have a local GPU and want to test before deploying, run `clarifai model serve ./Qwen3-0.6B --mode env` first. > **Note:** Clarifai automatically optimizes replica routing using KV cache affinity, improving throughput for shared system prompts, RAG pipelines, and multi-turn conversations. See [Request Routing](https://docs.clarifai.com/compute/inference/routing) for details. ## Step 5: Run Inference {VLLMTestRunner} Or use the Clarifai CLI: ```sh clarifai model predict https://clarifai.com//main/models/Qwen3-0.6B "Explain AI in one sentence" ``` ### Manage Your Deployment ```bash # Stream live logs clarifai model logs --deployment # Check status clarifai model status --deployment # Remove deployment when done (stops billing) clarifai model undeploy --deployment ``` For the full CLI reference, see [CLI Reference](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy). --- ## Build and Upload Models # Build and Upload Models **Build and import models, including from sources like Hugging Face** The Clarifai Python SDK allows you to upload custom models easily. Whether you're working with a pre-trained model from an external source like Hugging Face, or one you've built from scratch, Clarifai allows seamless integration of your models, enabling you to take advantage of the platform’s powerful capabilities. Once imported to our platform, your model can be utilized alongside Clarifai's vast suite of AI tools. It will be automatically deployed and ready to be evaluated, combined with other models and agent operators in a workflow, or used to serve inference requests as it is. Let’s walk through how to build and upload a custom model to the Clarifai platform. This example model appends the phrase `Hello World` to any input text and also supports streaming responses. :::note Explore [this repository](https://github.com/Clarifai/runners-examples) for examples on uploading different model types. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PrepareModelPyFile from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/prepare_model_py_file.py"; import ConfigModelInfo from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/model_info.yaml"; import HFCheckpoints from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/hf_checkpoints.yaml"; import ModelConcepts from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/model_concepts.yaml"; import BuildLogs from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/build_logs.txt"; import UnaryUnaryPredict from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/unary-unary-predict.py"; import UnaryStreamPredict from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/unary-stream-predict.py"; import StreamStreamPredict from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/stream-stream-predict.py"; import PredictOutput1 from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/predict_output_1.txt"; import PredictOutput2 from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/predict_output_2.txt"; import PredictOutput3 from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/predict_output_3.txt"; import DockerFile from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/dockerfile.txt"; ## Step 1: Perform Prerequisites ### Sign Up and Log In [Log in to](https://clarifai.com/login) your existing Clarifai account, or [sign up](https://clarifai.com/signup) for a new one. If you’re creating a new account, a default application will be provided for you. Then authenticate using the CLI: ```bash clarifai login ``` The CLI will prompt for your **Personal Access Token (PAT)** — find it in **Settings** > **Secrets** in the left sidebar. It will auto-detect your user ID and save everything locally. You can verify with `clarifai whoami`. > **Note:** Alternatively, you can set `export CLARIFAI_PAT=YOUR_PAT_HERE` as an environment variable, but `clarifai login` is the recommended approach since it also configures your user ID and app context. ### Install Clarifai Package Install the latest version of the `clarifai` Python package. This will also install the Clarifai [Command Line Interface](https://docs.clarifai.com/resources/api-overview/cli) (CLI), which we'll use for testing and uploading the model. pip install --upgrade clarifai ### Set Up Compute (For Manual Deploy Only) :::tip One-Command Deploy If you plan to use `clarifai model deploy` (recommended), you can **skip this step entirely** — the CLI automatically creates the compute cluster, nodepool, and deployment for you. See the [Getting Started guide](https://docs.clarifai.com/getting-started/upload-model) for the simplified flow. ::: If you prefer to manage compute infrastructure manually, your model requires a dedicated compute environment consisting of a [cluster and a nodepool](https://docs.clarifai.com/compute/deployments/clusters-nodepools/). > **Note:** A cluster is the foundation of your compute environment, while a nodepool is a single compute node or a group of nodes within a cluster that provides the resources your model requires. ### Set Up Docker or a Virtual Environment To test, run, and upload your model, you need to set up either a Docker container or a Python virtual environment. This ensures proper dependency management and prevents conflicts in your project. Both options allow you to work with different Python versions. For example, you can use Python 3.11 for uploading one model and Python 3.12 for another — configured via the [`config.yaml`](#build-info) file. > If Docker is installed on your system, it is highly recommended to use it for running the model. Docker provides better isolation and a fully portable environment, including for Python and system libraries. You should ensure your local environment has sufficient memory and compute resources to handle model loading and execution, especially during [testing](https://docs.clarifai.com/sdk/compute-orchestration/test-models-locally). ### Create Project Directory :::tip You can automatically generate the required files by running the [`clarifai model init`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-init) command in the terminal from your current directory. After the files are created, you can modify them as needed. ::: Create a project directory and organize your files as indicated below to fit the requirements of uploading models to the Clarifai platform. ```text your_model_directory/ ├── 1/ │ └── model.py ├── requirements.txt └── config.yaml ``` - **your_model_directory/** – The root directory containing all files related to your custom model. - **1/** – A subdirectory that holds the model file (_Note that the folder is named as **1**_). - **model.py** – Contains the code that defines your model, including loading the model and running inference. - **requirements.txt** – Lists the Python dependencies required to run your model. - **config.yaml** – Contains model metadata and configuration details necessary for building the model, defining compute resources, and more. ## Step 2: Build a Model Let's talk about the general steps you'd follow to upload any type of model to the Clarifai platform. ### Scaffold the Project With `clarifai model init` (Recommended) The fastest way to start is to scaffold a complete model project — `model.py`, `config.yaml`, and `requirements.txt` — with one command. Pick a built-in toolkit for a common inference engine, or scaffold a blank Python template you can fill in: {`# GPU LLM serving: clarifai model init --toolkit vllm --model-name Qwen/Qwen3-0.6B clarifai model init --toolkit sglang --model-name Qwen/Qwen3-0.6B clarifai model init --toolkit huggingface --model-name bert-base-uncased # Local LLM servers: clarifai model init --toolkit ollama --model-name llama3.1 clarifai model init --toolkit lmstudio --model-name llama-3.2-3b-instruct # Other use cases: clarifai model init --toolkit mcp my-mcp-server # MCP tool server clarifai model init --toolkit openai my-openai-wrapper # OpenAI-compatible wrapper # Blank Python template: clarifai model init my-model`} Each toolkit pre-fills `model.py` with a working `ModelClass` implementation, `config.yaml` with reasonable compute defaults, and `requirements.txt` with the right dependencies. After scaffolding, you can run the model locally (Step 3), upload it (Step 4), and deploy it (Step 5) — often with no further code changes. You can also clone an existing example from a GitHub URL with `clarifai model init --github-url ` — useful for starting from a known-good reference (see the [`runners-examples`](https://github.com/Clarifai/runners-examples) repo). The rest of this step walks through `model.py`, `config.yaml`, and `requirements.txt` in detail — either as reference for what gets generated, or as a guide if you'd rather build everything from scratch. ### Prepare `model.py` The `model.py` file contains the core logic for your model, including how the model is loaded and how predictions are made. This file must define a custom class that inherits from `ModelClass` and implements the required methods. This is the `model.py` file for the custom model we want to upload: {PrepareModelPyFile} Let’s break down what each part of the file does. #### a. `load_model` Method The `load_model` method is optional but recommended, as it prepares the model for inference by handling resource-heavy initializations. It is particularly useful for: - One-time setup of heavy resources, such as loading trained models or initializing data transformations. - Executing tasks during model container startup to reduce runtime latency. - Loading essential components like tokenizers, pipelines, and other model-related assets. Here is an example: ```python def load_model(self): self.tokenizer = AutoTokenizer.from_pretrained("model/") self.pipeline = transformers.pipeline(...) ``` #### b. Prediction Methods The `model.py` file must include at least one method decorated with `@ModelClass.method` to define the prediction endpoints. In the example model we want to upload, we defined a method that appends the phrase `Hello World` to any input text and added support for different types of [streaming responses](#step-6-predict-with-model). > **Note:** The structure of prediction methods on the client side directly mirrors the method signatures defined in your `model.py` file. This one-to-one mapping provides flexibility in defining prediction methods with varying names and arguments. > Here are some examples of method mapping: > | `model.py` Model Implementation | Client-Side Usage Pattern | > |-------------------------------------------------|--------------------------| > | `@ModelClass.method def predict(...)` | `model.predict(...)` | > | `@ModelClass.method def generate(...)` | `model.generate(...)` | > | `@ModelClass.method def stream(...)` | `model.stream(...)` | > You can learn more about the structure of prediction methods [here](https://docs.clarifai.com/compute/models/inference/#structure-of-prediction-methods). :::warning [Supported Input and Output Data Types](https://docs.clarifai.com/compute/models/model-upload/data-types/) Each parameter in the class methods must be annotated with a type, and the return type must also be specified. Clarifai's model framework supports rich data typing for both inputs and outputs. Supported types include `Text`, `Image`, `Audio`, `Video`, and more. ::: ### Prepare `config.yaml` The `config.yaml` file is essential for specifying the model’s metadata, compute resource requirements, and model checkpoints. :::tip Simplified Config Format When using `clarifai model init --toolkit`, the CLI generates a minimal config that works out of the box. Fields like `user_id`, `app_id`, and `model_type_id` are auto-filled from your CLI context at deploy time: ```yaml build_info: image: vllm/vllm-openai:latest checkpoints: repo_id: Qwen/Qwen3-0.6B type: huggingface when: runtime compute: instance: g4dn.xlarge model: id: qwen3-06b ``` - **`build_info.image`** specifies a Docker base image with the toolkit pre-installed (e.g., vLLM, SGLang). This means `requirements.txt` only needs lightweight dependencies. - **`compute.instance`** is auto-selected based on the model's estimated VRAM needs (weights + KV cache + framework overhead). It's expanded into the full `inference_compute_info` section when deploying. Use `clarifai list-instances` to browse available instance types, or `clarifai model deploy --instance-info`. For a simpler end-to-end flow, use [`clarifai model deploy`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy) which handles both upload and deployment in one step. ::: This is the `config.yaml` file for the custom model we want to upload: {ConfigModelInfo} Let’s break down what each part of the file does. #### Model Info This section defines your unique model ID (any arbitrary name you choose), along with the Clarifai user ID and app ID you retrieved [earlier](#sign-up-or-log-in). These values will determine where the model is uploaded on the Clarifai platform. > **Note:** If you reference an app that doesn’t exist, the CLI will prompt you to create it during the [model upload](#step-4-upload-the-model-to-clarifai) process. #### Build Info This section specifies details about the environment used to build or run the model. You can include the `python_version`, which is useful for ensuring compatibility between the model and its runtime environment, as different Python versions may have varying dependencies, library support, and performance characteristics. > **Note:** We currently support Python 3.11 and Python 3.12 (default). :::info multi-architecture support By default, Docker model images are built for the **linux/amd64** architecture. If needed, you can provide a comma-separated list of additional target platforms to build a multi-architecture image. Enabling multi-architecture builds allows the system to route requests to model-serving [nodepools](https://docs.clarifai.com/compute/deployments/clusters-nodepools) running on different hardware architectures (such as x86 or Arm). During deployment configuration and request routing, the system automatically validates which node instance types are compatible with the model’s supported platforms. This unified build process enhances cost efficiency and hardware flexibility. For example, you can build a single model version that works across multiple architectures, eliminating the need to maintain and manage separate model versions for each hardware type. > **Note**: You can also use the `--platform` flag when uploading a model to specify the target platform(s). This overrides any platform settings defined in your `config.yaml`. ```bash # Specify multiple platforms clarifai model upload --platform "linux/amd64,linux/arm64" # Combine with other options clarifai model upload --platform "linux/arm64" --stage upload ``` ::: #### Compute Resources To deploy your model on Clarifai’s dedicated compute, you must define the minimum compute resources required for running it, including CPU, memory, and optional GPU specifications. > **Note:** For [local execution](https://docs.clarifai.com/compute/local-runners/), the `inference_compute_info` section is optional — the model runs entirely on your machine using local CPU/GPU resources. These are some parameters you can define: - **`cpu_limit`** – Number of CPUs allocated for the model (follows [Kubernetes notation](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/), e.g., "1", "2"). - **`cpu_requests`** (default: `500m` (500 millicores)) – Specifies the minimum amount of CPU resources to request. Follows Kubernetes notation (e.g., "100m", "1", "4.5"), where 1 equals one full CPU core. - **`cpu_memory`** – Minimum memory required for the CPU (uses Kubernetes notation, e.g., "1Gi", "1500Mi", "3Gi"). - **`cpu_memory_requests`** (default: `500Mi` (500 mebibytes)) – Specifies the minimum amount of memory to request for the CPU. Also follows Kubernetes notation, such as 1Ki (1 kibibyte), 1500Mi (1500 mebibytes), 3Gi(3 gibibytes), and 4Ti (4 tebibytes). - **`num_accelerators`** – Number of GPUs or TPUs to use for inference. - **`accelerator_type`** – Specifies the type of hardware [accelerators](https://docs.clarifai.com/compute/cloud-instances/) (e.g., GPU or TPU) supported by the model (e.g., "NVIDIA-A10G"). _Note that instead of specifying an exact accelerator type, you can use a wildcard `(*)` to automatically match all available accelerators that fit your use case. For example, using `["NVIDIA-*"]` will enable the system to choose from all NVIDIA options compatible with your model._ - **`accelerator_memory`** – Minimum memory required for the GPU or TPU. - **`num_threads`** – Number of threads allocated to the model runner container. Useful for CPU-bound models where controlling thread concurrency improves performance or prevents resource contention. #### Hugging Face Model Checkpoints If you're using a model from Hugging Face, you can automatically download its checkpoints by specifying the appropriate configuration in this section. For private or restricted Hugging Face repositories, make sure to include an access token. Learn how to generate one [here](https://huggingface.co/docs/hub/en/security-tokens). See the [additional examples](#additional-examples) below for how to define Hugging Face checkpoints. {HFCheckpoints} :::note The `when` parameter in the `checkpoints` section determines when model checkpoints should be downloaded and stored. It must be set to one of the following options: - `runtime` (_default_) – Downloads checkpoints when loading the model in the `load_model` method. - `build` – Downloads checkpoints during the image build process. - `upload` – Downloads checkpoints before uploading the model. For larger models, we highly recommend downloading checkpoints at `runtime`. Doing so prevents unnecessary increases in Docker image size, which has some advantages: - Smaller image sizes - Faster build times - Quicker uploads and inference on the Clarifai platform Downloading checkpoints at `build` or `upload` time can significantly increase image size, resulting in longer upload times and increased cold start latency. ::: #### Model Concepts or Labels This section is required if your model outputs concepts or labels and is not being directly loaded from Hugging Face. So, you must define a `concepts` section in the `config.yaml` file. The following model types output concepts or labels: - `visual-classifier` - `visual-detector` - `visual-segmenter` - `text-classifier` - `visual-keypointer` > **Note:** For `visual-classifier` and `visual-detector` models, you can now specify explicit concept IDs in `config.yaml` in addition to concept names. When concept IDs are defined, the platform uses them directly instead of auto-generating IDs from names. This ensures the concept IDs in your model's `output_info` match those returned in prediction outputs, preventing ID mismatches. {ModelConcepts} :::note If you're using a model from Hugging Face and the `checkpoints` section is defined, the Clarifai platform will automatically infer concepts. In this case, you don’t need to manually specify them. ::: ### Prepare `requirements.txt` The `requirements.txt` file lists all the Python dependencies your model needs. This is the `requirements.txt` file for the custom model we want to upload: ```text clarifai>=11.8.2 ``` If your model requires Torch, we provide optimized pre-built Torch images as the base for machine learning and inference tasks. These images include all necessary dependencies, ensuring efficient execution. The available pre-built Torch images are: - `2.4.1-py3.11-cuda124` — Based on PyTorch 2.4.1, Python 3.11, and CUDA 12.4. - `2.5.1-py3.11-cuda124` — Based on PyTorch 2.5.1, Python 3.11, and CUDA 12.4. - `2.4.1-py3.12-cuda124` — Based on PyTorch 2.4.1, Python 3.12, and CUDA 12.4. - `2.5.1-py3.12-cuda124` — Based on PyTorch 2.5.1, Python 3.12, and CUDA 12.4. To use a specific Torch version, define it in your requirements.txt file like this: ```text torch==2.5.1 ``` This ensures the correct pre-built image is pulled from Clarifai's container registry, ensuring the correct environment is used. This minimizes cold start times and speeds up model uploads and runtime execution — avoiding the overhead of building images from scratch or pulling and configuring them from external sources. We recommend using either `torch==2.5.1` or `torch==2.4.1`. If your model requires a different Torch version, you can specify it in requirements.txt, but this may slightly increase the model upload time. ## Step 3: Test the Model Locally Before uploading your model to the Clarifai platform, it's important to test it locally to catch any typos or misconfigurations in the code. Learn how to run and test your models locally [here](https://docs.clarifai.com/compute/upload/test-locally). ## Step 4: Upload the Model to Clarifai Once your model is ready, you can upload it to the platform using Clarifai CLI. To upload your model, run the following command in your terminal: clarifai model upload ./your/model/path/here Alternatively, navigate to the directory containing your custom model and run the command without specifying the directory path: clarifai model upload This command builds the model’s Docker image using the defined compute resources and uploads it to Clarifai, where it can be served in production. The build logs will be displayed in your terminal, which helps you troubleshoot any upload issues. > **Note**: If you make any changes to your model and upload it again to the Clarifai platform, a new version of the model will be created automatically. :::note ### Skip Dockerfile By default, when you upload a model, the CLI automatically generates a `Dockerfile` in the root of your model directory. This ensures your model can be built and deployed with the correct environment. In some cases, though, you may prefer to provide your own custom [Dockerfile](https://docs.docker.com/reference/dockerfile/), such as when a specific base image is required for model inference. To do this, use the `--skip_dockerfile` flag. This tells the CLI to skip automatic Dockerfile generation and instead rely on the one you’ve created. clarifai model upload --skip_dockerfile Automatically Generated Dockerfile Example {DockerFile} ::: ## Step 5: Deploy the Model After your model is uploaded, you need to deploy it to make it available for inference. :::tip One-Command Deploy You can skip the separate upload and deploy steps entirely by using `clarifai model deploy`, which handles everything in one command — including auto-creating the compute infrastructure: ```bash clarifai model deploy ./your-model --instance g5.xlarge ``` See the [CLI deploy reference](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy) for all options. ::: Alternatively, after uploading with `clarifai model upload`, follow the on-screen prompts to: - Choose an existing cluster and nodepool where your model will run. - Specify the deployment configuration, including defining the minimum and maximum number of [replicas](https://docs.clarifai.com/compute/deployments/deploy-model#model-replica) to control scalability. Take note of the generated `deployment_id`. Build Logs Example {BuildLogs} ## Step 6: Predict With Model Once the model is successfully deployed, you can generate predictions either programmatically or through the [UI Playground](https://docs.clarifai.com/getting-started/quickstart-playground). ### Unary-Unary Predict Call You can make a [unary-unary](https://docs.clarifai.com/compute/models/inference/api/#unary-unary-predict-call) predict call using the model. {UnaryUnaryPredict} Example Output {PredictOutput1} ### Unary-Stream Predict Call You can make a [unary-stream](https://docs.clarifai.com/compute/models/inference/api/#unary-stream-predict-call) predict call using the model. {UnaryStreamPredict} Example Output {PredictOutput2} {StreamStreamPredict} Example Output {PredictOutput3} --> ## Additional Examples :::tip You can find various up-to-date model upload examples [here](https://github.com/Clarifai/runners-examples), which demonstrate different use cases and optimizations. Here is an example of how to [download](https://docs.clarifai.com/resources/api-overview/cli#initialize-with-github-template) a model: `clarifai model init --github-url https://github.com/Clarifai/runners-examples/tree/main/local-runners/ollama-model-upload`. ::: - [An OpenAI-compatible model](https://github.com/Clarifai/runners-examples/tree/main/llm/vllm-gemma-3-1b-it) built with Clarifai’s `OpenAIModelClass` - [A Llama-3.2-1B-Instruct model](https://github.com/Clarifai/runners-examples/tree/main/llm/hf-llama-3_2-1b-instruct) built with Clarifai’s `ModelClass` - [An NSFW Image Classifier model](https://github.com/Clarifai/runners-examples/tree/main/image-classifier/nsfw-image-classifier) built with Clarifai’s `VisualClassifierClass` - [A DETR ResNet Image Detector model](https://github.com/Clarifai/runners-examples/tree/main/image-detector/detr-resnet-image-detection) built with Clarifai’s `VisualDetectorClass` - Sandboxed execution with [Daytona](https://www.daytona.io/) and Clarifai - [A YouTube video](https://www.youtube.com/watch?v=S9pMcLzqbgo) on how to upload and run a custom model on Clarifai, start a local runner, and execute the model within a Daytona Sandbox. The model’s repository is [here](https://github.com/Clarifai/runners-examples/tree/daytona-code-generator-model/code-generation/daytona-code-generator). --- ## Input and Output Data Types # Input and Output Data Types **Learn about supported input and output data types along with usage examples** Clarifai's model framework supports rich data typing for both inputs and outputs, enabling flexible and type-safe model development. Note that when [preparing the `model.py` file](https://docs.clarifai.com/compute/models/model-upload#step-3-prepare-the-modelpy-file/) for uploading a model to the Clarifai platform, each parameter in the class methods must be annotated with a type, and the return type must also be specified. The [supported types](https://github.com/Clarifai/runners-examples/blob/main/SUPPORTED_DATATYPE.md) are categorized into Core Primitive, Python Primitive & Generic, and Custom Structured types. ## Core Primitive Types These are fundamental data types supported by Clarifai's framework for handling common data formats. | Type | Python Class | Description | Initialization Examples | |------------|-------------|-----------------------------------------------|-------------------------| | **Text** | `Text` | UTF-8 encoded text | `Text("Hello World")` `Text(url="https://example.com/text.txt")` | | **Image** | `Image` | RGB images (PNG/JPG format) | `Image(bytes=b"")` `Image(url="https://example.com/image.jpg")` `Image.from_pil(pil_image)` | | **Audio** | `Audio` | Audio data (WAV/MP3 format) | `Audio(bytes=b"")` `Audio(url="https://example.com/audio.mp3")` | | **Video** | `Video` | Video data (MP4/AVI format) | `Video(bytes=b"")` `Video(url="https://example.com/video.mp4")` | | **Frame** | `Frame` | Video frame with metadata | `Frame(time=1.5, image=Image(...))` | | **Concept** | `Concept` | Label with confidence score | `Concept("cat", 0.97)` `Concept(name="dog", value=0.92)` | | **Region** | `Region` | Bounding box and list of Concepts | `Region(box=[0.7, 0.3, 0.9, 0.7], [Concept("cat", 0.7), Concept(name="dog", value=0.2)])` | | **NameFields** | `dict` | Structured data | `{"scores": [0.9, 0.1]}` | ## Python Primitive & Generic Types Clarifai's framework also supports standard Python primitive and generic types for flexible data handling. These types enable type-safe processing of complex structures while maintaining compatibility with Python's native type system. ### Standard Python Primitive Types These fundamental data types are supported as both inputs and outputs. | Type | Example Inputs | Example Outputs | |--------|--------------------------------------|----------------------| | **int** | `42`, `user_age: int = 30` | `return 100` | | **float** | `0.95`, `temperature: float = 36.6` | `return 3.14159` | | **str** | `"Hello"`, `prompt: str = "Generate..."` | `return "success"` | | **bool** | `True`, `flag: bool = False` | `return is_valid` | | **bytes** | `b'raw_data'`, `file_bytes: bytes` | `return processed_bytes` | | **None** | `None` | `return None` | Here is an example of a primitive type usage: ```python class MyModel(ModelClass): @ModelClass.method def calculate_bmi( self, height_cm: float, weight_kg: float ) -> float: """Calculate Body Mass Index""" return weight_kg / (height_cm/100) ** 2 ``` ## Generic Container Types Clarifai supports generic types for handling complex structures while maintaining compatibility with Python’s type system. ### List[T] This handles homogeneous collections of any supported type, such as a list of images. Here is an example of using `List[T]` for batch processing: ```python class MyModel(ModelClass): def load_model(self): self.model = ... @ModelClass.method def predict_images(self, images: List[Image]) -> List[str]: """Process multiple images simultaneously""" return [self.model(img) for img in images] ``` Here is a client usage example: ```python images = [ Image(file_path="img1.jpg"), Image(url="https://example.com/img2.png") ] predictions = model.predict_images(images=images) ``` #### Dynamic Batch Prediction Handling Clarifai's model framework automatically handles both single and batch predictions through a unified interface. It dynamically adapts to the input format, eliminating the need for code changes to support different input types. Input detection is carried out automatically by: - **Single input** — Automatically processed as a singleton batch. - **Multiple inputs** — When inputs are provided as a list, the system processes them as a parallel batch. This flexibility allows you to pass either a single input or a list of inputs, and the system will handle them appropriately without requiring additional configuration. Here is an example of a model configuration that supports both single and batch predictions: ```python class TextClassifier(ModelClass): @ModelClass.method def predict(self, text: Text) -> float: """Single text classification (automatically batched)""" return self.model(text.text) ``` Here is a client usage example: ```python # Single prediction single_result = model.predict(Text("Positive review")) # Batch prediction batch_results = model.predict([ Text("Great product"), Text("Terrible service"), Text("Average experience") ]) ``` ### Dict[K, V] This supports JSON-like structures with string keys. Here is an example of using `Dict[K, V]` for handling model configuration: ```python class MyModel(ModelClass): @ModelClass.method def configure_model( self, params: Dict[str, float] ) -> Dict[str, str]: """Update model parameters""" self.threshold = params.get('threshold', 0.5) return {"status": "success", "new_threshold": str(self.threshold)} ``` ### Tuple[T1, T2, ...] This handles fixed-size heterogeneous data. Here is an example of using `Tuple[T1, T2, ...]` for multi-output models: ```python class MyModel(ModelClass): @ModelClass.method def analyze_document( self, doc: List[Text] ) -> Tuple[List[Text], Dict[str, float]]: """Return keywords and sentiment scores""" return (doc, {"docs": len(doc)}) ``` ## Custom Structured Types with NamedFields The `NamedFields` class enables creation of custom structured data types for handling complex inputs and outputs. This is particularly useful for models requiring multi-field data or producing compound results. Here is an example of using `NamedFields` to define a custom document metadata type: ```python DocumentMetadata = NamedFields( author=str, title=str, page_count=int, keywords=List[str] ) class MyModel(ModelClass): @ModelClass.method def process_document( self, content: Text, metadata: DocumentMetadata ) -> NamedFields( summary=Text, sentiment=float, topics=List[str]): ... ``` Here is an example of streaming complex structured data using `Stream[NamedFields]`: ```python class RealTimeAnalytics(ModelClass): @ModelClass.method def monitor_sensors( self, sensor_stream: Stream[NamedFields( temperature=float, pressure=float, timestamp=float )]) -> Stream[NamedFields( status=str, anomaly_score=float )]: for reading in sensor_stream: yield self._analyze_reading(reading) ``` Here is a client usage example: ```python sensor_data = [ {"temperature": 25.6, "pressure": 1013, "timestamp": 1625097600}, {"temperature": 26.1, "pressure": 1012, "timestamp": 1625097610}, {"temperature": 27.5, "pressure": 1011, "timestamp": 1625097620} ] for status in model.monitor_sensors(iter(sensor_data)): if status.anomaly_score > 0.9: return True ``` --- ## Model Upload Examples # Model Upload Examples **Learn how to upload and customize your own models on the Clarifai platform** This section provides examples that guide you through uploading custom models to Clarifai. You’ll learn how to adapt pre-built examples, configure model settings, and prepare your project for deployment on the platform. :::tip All the examples here are available in the Clarifai Runners Examples [GitHub repository](https://github.com/Clarifai/runners-examples). You can use the Clarifai CLI to [download a model template](https://docs.clarifai.com/resources/api-overview/cli#initialize-with-github-template). For example: ```bash clarifai model init --github-url https://github.com/Clarifai/runners-examples/tree/main/local-runners/ollama-model-upload ``` ::: To use these examples, create a project directory and organize your files as shown below. This structure is required for successfully uploading models to the Clarifai platform: ``` your_model_directory/ ├── 1/ │ └── model.py ├── requirements.txt └── config.yaml ``` * **`your_model_directory/`** – Root directory containing all files related to your custom model. * **`1/`** – A subdirectory (named `1`) that contains the main model file. * **`model.py`** – Defines your model logic, including loading the model and handling inference. * **`requirements.txt`** – Lists all Python dependencies required to run your model. * **`config.yaml`** – Includes metadata and configuration details for building the model, such as compute resources and environment settings. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import HelloWorldPy from "!!raw-loader!../../../code_snippets/runners-examples/hello_world.py"; import HelloWorldTxt from "!!raw-loader!../../../code_snippets/runners-examples/hello_world_requirements.txt"; import HelloWorldConfig from "!!raw-loader!../../../code_snippets/runners-examples/hello_world_config.yaml"; import NSFWImageClassifierPy from "!!raw-loader!../../../code_snippets/runners-examples/nsfw-image-classifier.py"; import NSFWImageClassifierTxt from "!!raw-loader!../../../code_snippets/runners-examples/nsfw-image-classifier.txt"; import NSFWImageClassifierConfig from "!!raw-loader!../../../code_snippets/runners-examples/nsfw-image-classifier.yaml"; import DetrresnetimagedetectionPy from "!!raw-loader!../../../code_snippets/runners-examples/detr-resnet-image-detection.py"; import DetrresnetimagedetectionTxt from "!!raw-loader!../../../code_snippets/runners-examples/detr-resnet-image-detection_requirements.txt"; import DetrresnetimagedetectionConfig from "!!raw-loader!../../../code_snippets/runners-examples/detr-resnet-image-detection_config.yaml"; import mask2formeradePy from "!!raw-loader!../../../code_snippets/runners-examples/mask2former-ade.py"; import mask2formeradeTxt from "!!raw-loader!../../../code_snippets/runners-examples/mask2former-ade.txt"; import mask2formeradeConfig from "!!raw-loader!../../../code_snippets/runners-examples/mask2former-ade.yaml"; import stablediffusion2depthPy from "!!raw-loader!../../../code_snippets/runners-examples/stable-diffusion-2-depth.py"; import stablediffusion2depthTxt from "!!raw-loader!../../../code_snippets/runners-examples/stable-diffusion-2-depth.txt"; import stablediffusion2depthConfig from "!!raw-loader!../../../code_snippets/runners-examples/stable-diffusion-2-depth.yaml"; import SglangSmolLM217BInstructPy from "!!raw-loader!../../../code_snippets/runners-examples/Sglang-SmolLM2-1_7B-Instruct.py"; import SglangOpenaiServer from "!!raw-loader!../../../code_snippets/runners-examples/Sglang-openai_server_starter.py"; import SglangSmolLM217BInstructTxt from "!!raw-loader!../../../code_snippets/runners-examples/Sglang-SmolLM2-1_7B-Instruct.txt"; import SglangSmolLM217BInstructConfig from "!!raw-loader!../../../code_snippets/runners-examples/Sglang-SmolLM2-1_7B-Instruct.yaml"; import hfllama321binstructPy from "!!raw-loader!../../../code_snippets/runners-examples/hf-llama-3_2-1b-instruct.py"; import hfllama321binstructTxt from "!!raw-loader!../../../code_snippets/runners-examples/hf-llama-3_2-1b-instruct_requirements.txt"; import hfllama321binstructConfig from "!!raw-loader!../../../code_snippets/runners-examples/hf-llama-3_2-1b-instruct_config.yaml"; import lmdeployLlama323BInstructPy from "!!raw-loader!../../../code_snippets/runners-examples/lmdeploy-Llama-3_2-3B-Instruct.py"; import lmdeployLlama323BInstructTxt from "!!raw-loader!../../../code_snippets/runners-examples/lmdeploy-Llama-3_2-3B-Instruct_requirements.txt"; import lmdeployLlama323BInstructConfig from "!!raw-loader!../../../code_snippets/runners-examples/lmdeploy-Llama-3_2-3B-Instruct_config.yaml"; import vllmgemma31bitPy from "!!raw-loader!../../../code_snippets/runners-examples/vllm-gemma-3-1b-it.py"; import vllmgemma31bitTxt from "!!raw-loader!../../../code_snippets/runners-examples/vllm-gemma-3-1b-it_requirements.txt"; import vllmgemma31bitConfig from "!!raw-loader!../../../code_snippets/runners-examples/vllm-gemma-3-1b-it_config.yaml"; import vllmtoolcallingllama318bPy from "!!raw-loader!../../../code_snippets/runners-examples/vllm-tool-calling-llama-3.1-8b.py"; import vllmtoolcallingllama318bTxt from "!!raw-loader!../../../code_snippets/runners-examples/vllm-tool-calling-llama-3.1-8b_requirements.txt"; import vllmtoolcallingllama318bConfig from "!!raw-loader!../../../code_snippets/runners-examples/vllm-tool-calling-llama-3.1-8b_config.yaml"; import ollamamodeluploadPy from "!!raw-loader!../../../code_snippets/runners-examples/ollama-model-upload.py"; import ollamamodeluploadTxt from "!!raw-loader!../../../code_snippets/runners-examples/ollama-model-upload_requirements.txt"; import ollamamodeluploadConfig from "!!raw-loader!../../../code_snippets/runners-examples/ollama-model-upload_config.yaml"; import browsertoolsPy from "!!raw-loader!../../../code_snippets/runners-examples/browser-tools.py"; import browsertoolsClient from "!!raw-loader!../../../code_snippets/runners-examples/browser-tools-client.py"; import browsertoolsTxt from "!!raw-loader!../../../code_snippets/runners-examples/browser-tools_requirements.txt"; import browsertoolsConfig from "!!raw-loader!../../../code_snippets/runners-examples/browser-tools_config.yaml"; import codeexecutiondockerversionPy from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-docker-version.py"; import codeexecutiondockerversionClient from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-docker-version-client.py"; import codeexecutiondockerversionTxt from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-docker-version_requirements.txt"; import codeexecutiondockerversionConfig from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-docker-version_config.yaml"; import codeexecutionwithoutdockerversionPy from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-without-docker-version.py"; import codeexecutionwithoutdockerversionClient from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-without-docker-version-client.py"; import codeexecutionwithoutdockerversionTxt from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-without-docker-version_requirements.txt"; import codeexecutionwithoutdockerversionConfig from "!!raw-loader!../../../code_snippets/runners-examples/code-execution-without-docker-version_config.yaml"; import googledrivePy from "!!raw-loader!../../../code_snippets/runners-examples/google-drive.py"; import googledriveClient from "!!raw-loader!../../../code_snippets/runners-examples/google-drive-client.py"; import googledriveTxt from "!!raw-loader!../../../code_snippets/runners-examples/google-drive.txt"; import googledriveConfig from "!!raw-loader!../../../code_snippets/runners-examples/google-drive.yaml"; import mathPy from "!!raw-loader!../../../code_snippets/runners-examples/math.py"; import mathClient from "!!raw-loader!../../../code_snippets/runners-examples/math-client.py"; import mathTxt from "!!raw-loader!../../../code_snippets/runners-examples/math_requirements.txt"; import mathConfig from "!!raw-loader!../../../code_snippets/runners-examples/math_config.yaml"; import postgresPy from "!!raw-loader!../../../code_snippets/runners-examples/postgres.py"; import postgresClient from "!!raw-loader!../../../code_snippets/runners-examples/postgres-client.py"; import postgresTxt from "!!raw-loader!../../../code_snippets/runners-examples/postgres_requirements.txt"; import postgresConfig from "!!raw-loader!../../../code_snippets/runners-examples/postgres_config.yaml"; import websearchPy from "!!raw-loader!../../../code_snippets/runners-examples/web-search.py"; import websearchClient from "!!raw-loader!../../../code_snippets/runners-examples/web-search-client.py"; import websearchTxt from "!!raw-loader!../../../code_snippets/runners-examples/web-search_requirements.txt"; import websearchConfig from "!!raw-loader!../../../code_snippets/runners-examples/web-search_config.yaml"; import Qwen25VL3BInstructpythonicvllmPy from "!!raw-loader!../../../code_snippets/runners-examples/Qwen2_5-VL-3B-Instruct-pythonic-vllm.py"; import Qwen25VL3BIopenaiserverstarterPy from "!!raw-loader!../../../code_snippets/runners-examples/Qwen-openai_server_starter.py"; import Qwen25VL3BInstructpythonicvllmTxt from "!!raw-loader!../../../code_snippets/runners-examples/Qwen2_5-VL-3B-Instruct-pythonic-vllm_requirements.txt"; import Qwen25VL3BInstructpythonicvllmConfig from "!!raw-loader!../../../code_snippets/runners-examples/Qwen2_5-VL-3B-Instruct-pythonic-vllm_config.yaml"; import nanonetsocrsPy from "!!raw-loader!../../../code_snippets/runners-examples/nanonets-ocr-s.py"; import nanonetsocrsTxt from "!!raw-loader!../../../code_snippets/runners-examples/nanonets-ocr-s_requirements.txt"; import nanonetsocrsConfig from "!!raw-loader!../../../code_snippets/runners-examples/nanonets-ocr-s_config.yaml"; import jinaembeddingsv3Py from "!!raw-loader!../../../code_snippets/runners-examples/jina-embeddings-v3.py"; import jinaembeddingsv3xt from "!!raw-loader!../../../code_snippets/runners-examples/jina-embeddings-v3_requirements.txt"; import jinaembeddingsv3Config from "!!raw-loader!../../../code_snippets/runners-examples/jina-embeddings-v3_config.yaml"; import fluxschnellPy from "!!raw-loader!../../../code_snippets/runners-examples/flux-schnell.py"; import fluxschnellTxt from "!!raw-loader!../../../code_snippets/runners-examples/flux-schnell_requirements.txt"; import fluxschnellConfig from "!!raw-loader!../../../code_snippets/runners-examples/flux-schnell_config.yaml"; ## Hello World `1/model.py` {HelloWorldPy} `requirements.txt` {HelloWorldTxt} `config.yaml` {HelloWorldConfig} ## NSFW Image Classifier `1/model.py` {NSFWImageClassifierPy} `requirements.txt` {NSFWImageClassifierTxt} `config.yaml` {NSFWImageClassifierConfig} ## DETR ResNet Image Detector `1/model.py` {DetrresnetimagedetectionPy} `requirements.txt` {DetrresnetimagedetectionTxt} `config.yaml` {DetrresnetimagedetectionConfig} ## Image Segmenter ### Mask2Former ADE `1/model.py` {mask2formeradePy} `requirements.txt` {mask2formeradeTxt} `config.yaml` {mask2formeradeConfig} ## Image-Text-to-Image ### Stable Diffusion 2 Depth `1/model.py` {stablediffusion2depthPy} `requirements.txt` {stablediffusion2depthTxt} `config.yaml` {stablediffusion2depthConfig} ## LLM ### SmolLM2 1.7B Instruct (SGLang) `1/model.py` {SglangSmolLM217BInstructPy} `1/openai_server_starter.py` {SglangOpenaiServer} `requirements.txt` {SglangSmolLM217BInstructTxt} `config.yaml` {SglangSmolLM217BInstructConfig} ### LLaMA 3.2 1B Instruct (Hugging Face) `1/model.py` {hfllama321binstructPy} `requirements.txt` {hfllama321binstructTxt} `config.yaml` {hfllama321binstructConfig} ### LLaMA 3.2 3B Instruct (LMDeploy) `1/model.py` {lmdeployLlama323BInstructPy} `requirements.txt` {lmdeployLlama323BInstructTxt} `config.yaml` {lmdeployLlama323BInstructConfig} ### Gemma 3 1B Instruct (vLLM) `1/model.py` {vllmgemma31bitPy} `requirements.txt` {vllmgemma31bitTxt} `config.yaml` {vllmgemma31bitConfig} ### LLaMA 3.1 8B Tool Calling (vLLM) `1/model.py` {vllmtoolcallingllama318bPy} `requirements.txt` {vllmtoolcallingllama318bTxt} `config.yaml` {vllmtoolcallingllama318bConfig} ## Local Runners ### Ollama `1/model.py` {ollamamodeluploadPy} `requirements.txt` {ollamamodeluploadTxt} `config.yaml` {ollamamodeluploadConfig} ## MCP ### Browser Tools `1/model.py` {browsertoolsPy} `client.py` {browsertoolsClient } `requirements.txt` {browsertoolsTxt} `config.yaml` {browsertoolsConfig} ### Code Execution `1/model.py` {codeexecutiondockerversionPy} `client.py` {codeexecutiondockerversionClient} `requirements.txt` {codeexecutiondockerversionTxt} `config.yaml` {codeexecutiondockerversionConfig} ### Code Execution Without Docker Version `1/model.py` {codeexecutionwithoutdockerversionPy} `client.py` {codeexecutionwithoutdockerversionClient} `requirements.txt` {codeexecutionwithoutdockerversionTxt} `config.yaml` {codeexecutionwithoutdockerversionConfig} ### Google Drive `1/model.py` {googledrivePy} `client.py` {googledriveClient} `requirements.txt` {googledriveTxt} `config.yaml` {googledriveConfig} ### Math `1/model.py` {mathPy} `client.py` {mathClient} `requirements.txt` {mathTxt} `config.yaml` {mathConfig} ### Postgres `1/model.py` {postgresPy} `client.py` {postgresClient} `requirements.txt` {postgresTxt} `config.yaml` {postgresConfig} ### Web Search `1/model.py` {websearchPy} `client.py` {websearchClient} `requirements.txt` {websearchTxt} `config.yaml` {websearchConfig} ## Multimodal Models ### Qwen2.5 VL 3B Instruct `1/model.py` {Qwen25VL3BInstructpythonicvllmPy} `1/openai_server_starter.py` {Qwen25VL3BIopenaiserverstarterPy} `requirements.txt` {Qwen25VL3BInstructpythonicvllmTxt} `config.yaml` {Qwen25VL3BInstructpythonicvllmConfig} ## OCR ### NanoNets OCR Small `1/model.py` {nanonetsocrsPy} `requirements.txt` {nanonetsocrsTxt} `config.yaml` {nanonetsocrsConfig} ## Text Embedder ### Jina Embeddings v3 `1/model.py` {jinaembeddingsv3Py} `requirements.txt` {jinaembeddingsv3xt} `config.yaml` {jinaembeddingsv3Config} ## Text-to-Image ### FLUX Schnell `1/model.py` {fluxschnellPy} `requirements.txt` {fluxschnellTxt} `config.yaml` {fluxschnellConfig} --- ## Test Models Locally # Test Models Locally **Learn how to test your custom models locally** Before uploading a custom model to the Clarifai platform, always test and debug it locally. It ensures smooth performance, verifies dependency compatibility, and streamlines the deployment process. This step helps you detect problems like setup file errors, typos, code misconfigurations, or incorrect model implementations. This saves you time and avoids upload failures by validating the model's behavior on the target hardware you plan to deploy to. :::note You should ensure your local environment has sufficient memory and compute resources to handle model loading and execution during the testing process. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import TestMethod from "!!raw-loader!../../../code_snippets/python-sdk/model-upload/test-models-locally.py"; ## Prerequisites ### Build a Model You can either build a custom model from scratch or leverage pre-trained models from external repositories like Hugging Face. If you're developing your own model, our [step-by-step guide](https://docs.clarifai.com/compute/upload/) provides detailed instructions to get started. You can also explore [this examples repository](https://github.com/Clarifai/runners-examples) to learn how to build models compatible with the Clarifai platform. ### Install Clarifai CLI Install the latest version of the [Clarifai CLI](https://docs.clarifai.com/resources/api-overview/cli) tool. We'll use this tool to test models in the local development environment. pip install --upgrade clarifai ### System Requirements Before running the test commands below, ensure your local environment meets the following requirements: * **Python version** — Python 3.11 or 3.12 is required. * **GPU support** — For models that require GPU acceleration, your environment must have NVIDIA GPU support installed and properly configured. CPU-only models can still be tested without a GPU. * **Docker (optional)** — Docker is recommended for container-based testing (`--mode container`), but it is not mandatory. Without Docker, you can use `--mode env` to test in a virtual environment. ## Test With the `serve` Command The `clarifai model serve` command is the primary way to test your model locally. It has two modes: - **API-connected mode** (default) — Connects to the Clarifai platform and exposes your model via a public URL, just like a cloud deployment. - **Standalone gRPC mode** (`--grpc`) — Runs your model as a local gRPC server with no Clarifai connection needed. Ideal for offline development. ### Option A: API-Connected Mode This mode connects your local model to the Clarifai platform, giving you a public URL you can use to test predictions through the API or the [AI Playground](https://docs.clarifai.com/getting-started/quickstart-playground). clarifai model serve > **Note:** You must be logged in (`clarifai login`) to use API-connected mode. You can specify additional options: | Flag | Description | Default | |------|-------------|---------| | `--mode` | How to run: `none` (current env), `env` (virtual env), or `container` (Docker) | `none` | | `--port` | Port for the local server | `8000` | | `--concurrency` | Number of concurrent requests to handle | `1` | | `--keep-image` | Keep the Docker image after stopping (for `container` mode) | `false` | | `-v, --verbose` | Show detailed SDK and server logs | `false` | You can also specify the path to the model directory. If omitted, the current directory is used: clarifai model serve ./path/to/my-model --mode env ### Option B: Standalone gRPC Mode (Offline) Use the `--grpc` flag to run the model as a standalone gRPC server without any Clarifai connection. This is ideal for offline development — no PAT or login required. clarifai model serve --grpc --port 8000 Once the server is running, set the `CLARIFAI_API_BASE` environment variable to point to it: export CLARIFAI_API_BASE="localhost:8000" set CLARIFAI_API_BASE="localhost:8000" You can then make [inference requests](https://docs.clarifai.com/compute/inference/clarifai/api) using the Clarifai Python SDK. ## Implement a `test` Method To enable quick validation, implement a `test` method in your [`model.py`](https://docs.clarifai.com/compute/upload/#prepare-modelpy) file. This method should internally call your model's other methods and verify the output. Below is a sample `model.py` file with an example implementation of the `test` method: {TestMethod} --- ## Account and Billing # Account and Billing **View and manage your personal account and billing settings** The **Account** and **Billing** pages serve as your central hub for managing every aspect of your profile and subscription. These pages give you full control over your account by allowing you to customize your personal information and handle billing-related activities. ## Account The **Account** page allows you to view and manage your personal identity and security preferences on the Clarifai platform. To access it, [log in](https://clarifai.com/login) to the Clarifai platform. Then, in the collapsible left sidebar, select **Settings** and choose **Account** from the dropdown list. ![ ](/img/new-docs/account_1.png) ### Profile Information This section displays the personal and professional details you provided when creating your account. You can customize how you appear across the Community platform — such as on shared model or app pages — by uploading a profile avatar. If no image is uploaded, your initials will be shown instead. To upload or change your avatar, click **Change Avatar**. A pop-up will appear, allowing you to select an image from your device. > **Note:** The image must be in JPEG or PNG format and no larger than **20MB**. You can also update any of your profile details in this section. After making changes, click **Update User Info** to save them. ### Profile ![ ](/img/new-docs/account_2.png) This section displays your unique User ID, which serves as your username handle across the platform — appearing in both API references and platform URLs. Your User ID is not case-sensitive and may include one or more alphanumeric strings separated by hyphens or underscores. To update it, click the **Edit** button next to the field. A pop-up will appear, allowing you to enter a new User ID. > **Caution:** Changing your User ID may lead to unintended side effects, as we do not create redirects for old app URLs and we do not re-route requests that reference your previous User ID. If you make this change, you are responsible for notifying any collaborators who may be affected. ### Email This section lists and manages all email addresses associated with your account and indicates which one is set as your **Primary** contact. To link an additional email address, enter it into the **Add new email** field and click **Add Email**. ### Danger Zone ![ ](/img/new-docs/account_3.png) This section contains actions that have significant and potentially irreversible effects on your account. To manage your user profile visibility, click **Update Visibility** and choose whether your profile should be public (visible to everyone) or private. To permanently remove your account, click **Delete Account**. This action is irreversible and will permanently erase all data associated with your profile. Proceed with extreme caution when using this option. ## Billing The **Subscription & Billing** page provides a centralized place to view and manage your credits, plan, payment methods, and invoices. To access it, [log in](https://clarifai.com/login) to the Clarifai platform. Then, in the collapsible left sidebar, select **Settings** and choose **Billing** from the dropdown list. ![ ](/img/new-docs/billing_1.png) ### Transition to New Plan We’ve simplified Clarifai’s pricing by introducing a single Pay-As-You-Go plan with no minimum monthly commitments and far fewer feature gates — most of Clarifai is now accessible out of the box, including Compute Orchestration with auto-provisioned GPUs. Just add credits, build across the platform, and go. All new users are automatically onboarded to this plan. Existing users can switch by clicking the **Review the new plan** button in the banner displayed on the page. In the window that appears, click **Change Plan** to complete the transition. ![ ](/img/new-docs/billing_2.png) After switching, click the notification bar and select **Claim credits** to verify your phone number and receive a one-time, free $5 credits, which can be used to try anything from serverless model access to dedicated deployments. > **Note:** The $5 welcome bonus has a 30-day expiration period. ![ ](/img/new-docs/billing_3.png) > **Note:** Each user is eligible for up to two welcome bonuses — one for a personal account and another for an organization account that the user can create with their peers. ![ ](/img/new-docs/billing_7.png) ### Recharge Your Account After transitioning to the new plan, you can view your current plan and the details of your remaining credits at the top of the page. To recharge your account, click the **Recharge** button. If your phone number has not yet been verified, a dialog will prompt you to enter your number and receive a verification code via SMS. ![ ](/img/new-docs/billing_4.png) > **Note:** The same phone number can be verified for up to three different Clarifai accounts, regardless of whether the accounts are personal or organizational. However, only 1 personal and 1 Org account is eligible for the $5 bonus. After successful verification, you can proceed to recharge your account credits using your card. ![ ](/img/new-docs/billing_5.png) :::info Auto recharge > To enable automatic recharging, turn on the **Auto recharge** button. A dialog will appear where you can set a balance threshold — for example, when credits drop to $5, automatically recharge up to $20. > ![ ](/img/new-docs/billing_6.png) ::: ### Change Your Plan As mentioned previously, we offer a Pay-As-You-Go plan that allows you to pay only for what you use. It offers the following features: - Run models directly or via dedicated compute instance - Clarifai Reasoning Engine — Accelerate Agentic AI workloads and large reasoning models - Full platform access - Promotional free access to [Local Runners](https://docs.clarifai.com/compute/local-runners/) - Up to 100 requests per second > **Note:** The Pay-As-You-Go plan has a maximum monthly spend limit of $100. Contact us to request a higher limit. For organizations operating at scale or with specialized requirements, we also offer an Enterprise plan. It offers the following features: - Access to top GPUs - Clarifai’s SaaS or private control plane - Optional air-gapped deployments and private data planes - Multi-organizations, teams management and [audit logging](https://docs.clarifai.com/control/control-center/teams-logs) - 24/7 dedicated support To upgrade to the Enterprise plan, click the **Talk to our AI Experts** button. You’ll be redirected to a page where you can schedule a meeting with our AI team to discuss your specific needs. ![ ](/img/new-docs/billing_8.png) ### Add a Payment Method In the **Payment Methods** section, you can add a new payment option by clicking **Add a Credit Card**. A pop-up window will appear, allowing you to securely enter your card details. > **Note:** If you haven’t verified your phone number yet, you’ll be prompted to do so. ![ ](/img/new-docs/billing_9.png) ### Invoices The **Invoices** section provides a complete history of your billing activity, displaying past transactions and downloadable statements for your records. --- ## Authentication # Authentication **Secure, compliant AI** Authentication ensures secure access to the Clarifai platform. You can log in with your username and password through the web interface or use Google or GitHub single-sign-on (SSO). > **Note:** The length of a login session depends on how you sign in: SSO sessions remain active for seven days. When using a regular username and password, selecting “remember me” extends the session to 30 days, while logging in without it limits the session to just one day. For API and SDK interactions, authentication is handled using a Personal Access Token (PAT) or API Key, which provide secure access without sharing your password. This section walks you through the available authentication methods, how to set them up, and best practices for keeping your credentials safe. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Authorize # Authorize **Authorize before securely making API requests** After creating your access credentials, you are ready to make API calls. Most of our API clients require setting up authentication during initialization; though, it can be changed for particular requests, if needed. If you are using a REST API, you will need to add the `Authorization` header as illustrated in the cURL example below. ## Authorization Keys The key used for authorization can either be: * [A Personal Access Token \(PAT\)](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens), which is tied to a user; or, * [An API Key](https://docs.clarifai.com/clarifai-basics/authentication/app-specific-api-keys), which is tied to a specific application. :::important IMPORTANT NOTE - PAT is the primary authentication mechanism we use. ::: Using a PAT is more powerful than an API Key. A PAT represents you when accessing the Clarifai API. It allows you to access multiple applications with a single key. With a PAT, you can access your own apps as well as any other apps you have permissions to use, such as public apps, apps you're added as a collaborator, or apps belonging to your organization's team. Also, certain endpoints support only PATs, such as creating a new application or a new API Key. When using a PAT to call the API, you need to specify your user ID alongside the application ID to which the request should be applied. On the other hand, an API Key restricts your access only to a single app. So, it could be suitable for accessing resources that are specifically locked down to a single app. When using an app-specific API Key, you do not need to specify either the user ID or the application ID as they are already part of the API Key. :::tip Clarifai provides various clients you can use to access the API in your favorite programming language. Learn how to install your preferred client [here](https://docs.clarifai.com/api-guide/api-overview/api-clients/). ::: ## Authorization Examples import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ```python ############################################################################################### # Initialize the gRPC-based client to communicate with the Clarifai platform. ############################################################################################### # Import the Clarifai gRPC-based objects needed from clarifai_grpc.channel.clarifai_channel import ClarifaiChannel from clarifai_grpc.grpc.api import resources_pb2, service_pb2, service_pb2_grpc from clarifai_grpc.grpc.api.status import status_pb2, status_code_pb2 # Construct the communications channel channel = ClarifaiChannel.get_grpc_channel() # Construct the V2Stub object for accessing all the Clarifai API functionality stub = service_pb2_grpc.V2Stub(channel) ############################################################################################## # This is where you set up the metadata object that's used to authenticate. # This authorization will be used by every Clarifai API call. # Change the following authorization key to your own credentials # Example: metadata = (('authorization', 'Key ' + 'a123457612345678'),) ############################################################################################## metadata = (('authorization', 'Key ' + 'YOUR_CLARIFAI_PAT_HERE'),) # Or, if you were to use an API Key: # metadata = (('authorization', 'Key ' + 'YOUR_CLARIFAI_API_KEY_HERE'),) # Yes, the word 'Key' appears in addition to the alphanumeric PAT or API Key ############################################################################################## # A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: # user_id (your user id) and app_id (app id that contains the model of interest). # Both of them are specified as string values. ############################################################################################## userDataObject = resources_pb2.UserAppIDSet(user_id='YOUR_USER_ID_HERE', app_id='YOUR_APPLICATION_ID_HERE') ``` ```javascript ///////////////////////////////////////////////////////////////////////////////////////////// // Initialize the gRPC-based client to communicate with the Clarifai platform. //////////////////////////////////////////////////////////////////////////////////////////// // Import the Clarifai gRPC-based client const {ClarifaiStub, grpc} = require("clarifai-nodejs-grpc"); // Construct the stub object for accessing all the Clarifai API functionality const stub = ClarifaiStub.grpc(); //////////////////////////////////////////////////////////////////////////////////////////// // This is where you set up the Metadata object that's used to authenticate. // This authorization will be used by every Clarifai API call. // Change the following authorization key to your own credentials // Example: metadata.set("authorization", "Key " + "a123457612345678"); //////////////////////////////////////////////////////////////////////////////////////////// const metadata = new grpc.Metadata(); metadata.set("authorization", "Key " + "YOUR_CLARIFAI_PAT_HERE"); // Or, if you were to use an API Key: // metadata.set("authorization", "Key " + "YOUR_CLARIFAI_API_KEY_HERE"); // Yes, the word 'Key' appears in addition to the alphanumeric PAT or API Key ///////////////////////////////////////////////////////////////////////////////////////////// // A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: // user_id (your user id) and app_id (app id that contains the model of interest). // Both of them are specified as string values. ///////////////////////////////////////////////////////////////////////////////////////////// user_app_id: { "user_id": "YOUR_USER_ID_HERE", "app_id": "YOUR_APPLICATION_ID_HERE" } ``` ```java ///////////////////////////////////////////////////////////////////////////////////////////// // Initialize the gRPC-based client to communicate with the Clarifai platform. //////////////////////////////////////////////////////////////////////////////////////////// // Import the Clarifai gRPC-based objects needed import com.clarifai.grpc.api.*; import com.clarifai.channel.ClarifaiChannel; import com.clarifai.credentials.ClarifaiCallCredentials; ///////////////////////////////////////////////////////////////////////////////////////////// // Construct the communications channel. // Construct the stub object for accessing all the Clarifai API functionality. // Set up the authorization that will be used by every Clarifai API call. ///////////////////////////////////////////////////////////////////////////////////////////// Channel channel = ClarifaiChannel.INSTANCE.getGrpcChannel(); V2Grpc.V2BlockingStub stub = V2Grpc.newBlockingStub(channel) .withCallCredentials(new ClarifaiCallCredentials("YOUR_CLARIFAI_PAT_HERE")); // Or, if you were to use an API Key: //.withCallCredentials(new ClarifaiCallCredentials("YOUR_CLARIFAI_API_KEY_HERE")); ///////////////////////////////////////////////////////////////////////////////////////////// // A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: // user_id (your user id) and app_id (app id that contains the model of interest). // Both of them are specified as string values. ///////////////////////////////////////////////////////////////////////////////////////////// .setUserAppId(UserAppIDSet.newBuilder().setUserId("YOUR_USER_ID_HERE").setAppId("YOUR_APPLICATION_ID_HERE")) ``` ```php ///////////////////////////////////////////////////////////////////////////////////////////// // Initialize the gRPC-based client to communicate with the Clarifai platform. //////////////////////////////////////////////////////////////////////////////////////////// // Enable use of the ClarifaiClient object from the Clarifai namespace use Clarifai\ClarifaiClient; // Construct the initialization object for accessing all the Clarifai API functionality $client = ClarifaiClient::grpc(); //////////////////////////////////////////////////////////////////////////////////////////// // This is where you set up the Metadata object that's used to authenticate. // This authorization will be used by every Clarifai API call. // Change the following authorization key to your own credentials // Example: $metadata = ['Authorization' => ['Key a123457612345678']]; //////////////////////////////////////////////////////////////////////////////////////////// $metadata = ['Authorization' => ['Key YOUR_CLARIFAI_PAT_HERE']]; // Or, if you were to use an API Key: // $metadata = ['Authorization' => ['Key YOUR_CLARIFAI_API_KEY_HERE']]; // Yes, the word 'Key' appears in addition to the alphanumeric PAT or API Key ///////////////////////////////////////////////////////////////////////////////////////////// // A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: // user_id (your user id) and app_id (app id that contains the model of interest). // Both of them are specified as string values. ///////////////////////////////////////////////////////////////////////////////////////////// use Clarifai\Api\UserAppIDSet; $userDataObject = new UserAppIDSet([ 'user_id' => 'YOUR_USER_ID_HERE', 'app_id' => 'YOUR_APPLICATION_ID_HERE' ]); ``` ```bash curl -X POST \ -H "Authorization: Key YOUR_CLARIFAI_PAT_HERE" \ -H "Content-Type: application/json" \ ``` ```csharp using System.Threading.Tasks; using Clarifai.API; namespace YourNamespace { public class YourClassName { public static async Task Main() { var client = new ClarifaiClient("YOUR_API_KEY"); } } } ``` ```objectivec ClarifaiApp *app = [[ClarifaiApp alloc] initWithApiKey:@"YOUR_API_KEY"]; ``` :::caution - If your PAT or API Key does not have the required scope\(s\) to execute a given request, you will get an error message reporting the missing scopes and/or endpoints that are needed to execute the request. - An invalid PAT or API Key may be reported as 'API key not found'. - Failure to include the required PAT or API Key may result in 'Invalid request'. ::: ## CORS Policy Our APIs adhere to industry standards for Cross-Origin Resource Sharing (CORS) to ensure security and proper access control. The Access-Control-Allow-Origin header is limited to a predefined list of allowed origins. This means that only requests originating from allowed origins will receive a successful CORS response. If your Javascript application(s) encounter a CORS error while calling our APIs, it is likely that the origin is not on the allowed list. To avoid such issues, we recommend following established patterns such as using a backend server to proxy API requests or ensuring your application is hosted on an approved origin. For more information about CORS and best practices, you can refer to [this resource](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). If you need additional support or would like to request for your custom domain, please reach out to support@clarifai.com for assistance. --- ## Environment Secrets # Environment Secrets **Create, store, and manage environment variables and secrets** Environment secrets are securely encrypted values that function as environment variables within your workflows. They enable you to store and manage sensitive configuration data, such as third-party API keys, without hardcoding them into your scripts. ## Via the UI The Clarifai platform offers an intuitive interface for creating, managing, and securing your environment secrets. To create a secret, [log in](https://clarifai.com/login) to the Clarifai platform. Then, in the collapsible left sidebar, select **Settings** and choose **Secrets** from the dropdown list. ![](/img/others/third_party_key_1.png) On the ensuing **Secrets** page, click the **Create Environment Secrets** button. Next, on the form that pops up, enter a required name for your secret, an optional short description, and the required value. Lastly, click the **Create Environment Secret** button. ![](/img/others/third_party_key_2.png) The new secret will be listed in the **Environment Secrets** section, where you can copy, view, edit, or delete it. ![listed pat](/img/others/third_party_key_3.png) :::tip note - Environment secrets do not expire. In case your secret gets compromised, you should delete it, and create a new one. - We recommend that you do **not** share your secrets with other users. ::: ## Via the API ### Prerequisites #### Install Python SDK Install the latest version of the Clarifai [Python](https://github.com/Clarifai/clarifai-python/) SDK package. pip install --upgrade clarifai #### Get Credentials Go to your Clarifai account and retrieve the following credentials: - **User ID** – In the collapsible left sidebar, select **Settings** and choose **Account** from the dropdown list. Then, locate your user ID. - **PAT** – From the same **Settings** option, choose **Secrets** to generate or copy your [Personal Access Token](pat.md) (PAT). This token is used to authenticate your connection with the Clarifai platform. You can then set the PAT as an environment variable using `CLARIFAI_PAT`. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CurlSecretsCreate from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_create.sh"; import OutputCurlSecretsCreate from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_create_output.txt"; import CurlSecretsGet from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_get.sh"; import OutputSecretsGet from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_get_output.txt"; import PySecretsCreate from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_create.py"; import PySecretsUpdate from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_update.py"; import PySecretsGet from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_get.py"; import PySecretsDelete from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_delete.py"; import PySecretsList from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_list.py"; import CurlSecretsList from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_list.sh"; import OutputSecretsList from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_list_output.txt"; import OutputSecretsPatch from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_patch_output.txt"; import OutputSecretsDelete from "!!raw-loader!../../../code_snippets/api-guide/authentication/secrets_delete_output.txt"; ### Create You can create a new environment secret by providing its ID and value. {PySecretsCreate} {CurlSecretsCreate} Example Output {OutputCurlSecretsCreate} ### Get You can get an existing environment secret by providing its ID. {PySecretsGet} {CurlSecretsGet} Example Output {OutputSecretsGet} ### List You can list all the environment secrets you have. {PySecretsList} {CurlSecretsList} > **Note:** If you specify `page_no` but not `per_page`, [pagination](https://docs.clarifai.com/resources/api-overview/pagination/) defaults to 16 items per page. If both `page_no` and `per_page` are left as `None`, all resources are listed without pagination. Example Output {OutputSecretsList} ### Update You can update an existing environment secret by providing its ID. {PySecretsUpdate} Example Output {OutputSecretsPatch} ### Delete You can delete an existing environment secret by providing its ID. {PySecretsDelete} Example Output {OutputSecretsDelete} ## Use Case Examples ### Custom Models The `config.yaml` file defines the build, deployment, and runtime configuration for a custom model on the Clarifai platform. You can also use this file to reference sensitive configuration values without hardcoding them into your application code. The following example shows how to define a secret when deploying the [GitHub MCP server](https://github.com/Clarifai/runners-examples/blob/main/mcp/github-mcp-server/README.md#2-configure-secrets-optional) to the Clarifai platform: ```yaml model: id: github-mcp-server model_type_id: mcp build_info: python_version: "3.11" node_version: "20.11.1" compute: instance: t3a.2xlarge secrets: - id: "github_personal_access_token" type: "env" env_var: "GITHUB_PERSONAL_ACCESS_TOKEN" description: "GitHub personal access token" mcp_server: command: "npx" args: ["-y", "@modelcontextprotocol/server-github"] ``` In this configuration: * `id` specifies the secret identifier stored in Clarifai’s secret manager. * `env_var` defines the environment variable name that will be available inside the runtime environment. * The secret value is injected securely during deployment and never stored directly in the project files. Your application can then read the secret using a standard environment variable: ```python import os github_token = os.environ["GITHUB_PERSONAL_ACCESS_TOKEN"] ``` ### Pipelines Clarifai Pipelines allow you to design and run asynchronous, multi-step AI workflows on the platform. The root `config.yaml` file defines the pipeline’s identity, structure, and orchestration logic. It specifies which steps the pipeline contains and how they are executed. You can also use this configuration to securely inject secrets into specific pipeline steps. Below is an example showing how to reference secrets in the [root `config.yaml` file](https://docs.clarifai.com/compute/pipelines/create-api#optional-secrets-configuration) of a pipeline project: ```yaml pipeline: id: "hello-world-pipeline" user_id: "user-id" app_id: "app-id" step_directories: - stepA - stepB orchestration_spec: argo_orchestration_spec: | apiVersion: argoproj.io/v1alpha1 kind: Workflow spec: entrypoint: sequence arguments: parameters: - name: input_text value: "Input Text Here" templates: - name: sequence steps: - - name: step-0 templateRef: name: users/user-id/apps/app-id/pipeline_steps/stepA template: users/user-id/apps/app-id/pipeline_steps/stepA arguments: parameters: - name: input_text value: "{{workflow.parameters.input_text}}" - - name: step-1 templateRef: name: users/user-id/apps/app-id/pipeline_steps/stepB template: users/user-id/apps/app-id/pipeline_steps/stepB arguments: parameters: - name: input_text value: "{{workflow.parameters.input_text}}" config: step_version_secrets: step-0: API_KEY: users/user-id/apps/secrets/my-api-key DB_PASSWORD: users/user-id/apps/secrets/db-secret step-1: EMAIL_TOKEN: users/user-id/apps/secrets/email-token ``` In this configuration: * Each key (for example, `API_KEY` or `DB_PASSWORD`) becomes an environment variable within the corresponding pipeline step. * The values reference secrets stored in the Clarifai secrets manager, ensuring they remain encrypted and access-controlled. Once the pipeline runs, the secrets are available in your step code through standard environment variables. Example (`pipeline_step.py`): ```python import os api_key = os.environ["API_KEY"] ``` At runtime, the platform securely injects the secret value into the environment, allowing your code to use it without exposing sensitive information in your project files. --- ## App-Specific API Keys # App-Specific API Keys **Authenticate access to resources within the scope defined by the key** App-specific API Keys are used to authorize access to your Clarifai applications. You can use an API Key to access the resources within the scope of the app defined by that key. A key is automatically generated when you create a new application. You can also create a new one, as described [below](#how-to-create-api-keys-on-the-platform). :::info Each API Key is associated with a specific user and a specific app. It ties in `user_id` and `app_id`, causing only resources in that app to be usable. ::: When using an app-specific API Key to make a request, you do not need to specify either the user ID or the application ID, as they are already part of the key. An API Key allows you to have fine-grained control over the data exposed through your app. You can control the scope of your key through a simple checkbox interface displayed when you create a new key or edit a key. :::caution API Key Versus PAT You cannot use an API Key to access models, model versions, workflows, or other resources that are not part of the app that the key is associated with. You need a [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) to do so. For example, to access any of Clarifai's resources, you need to use a PAT while specifying Clarifai's `user_id` and the `app_id` to which the resource belongs. ::: ## How to Create API Keys on the Platform In the collapsible left sidebar, select the **Projects** option. Then, choose the application you want to generate an API key for. ![App settings create key](/img/others/create_api_key_select_app.png) You'll be redirected to the application's individual page. Select the **Settings** option in the app's collapsible left sidebar. Next, on the **App Settings** page, within the **API Keys** section, click the **Create API Key** button. ![App settings create key](/img/others/create_api_key_community.png) Then, use the form that pops up to generate a new key for your application — provide a short description, select the [scopes](https://docs.clarifai.com/control/authentication/scopes), and click the **Create API Key** button. ![App key create form](/img/others/create_api_key_community_form.png) The new key will be listed in the **API Keys** section, where you can copy it to the clipboard, edit it, or delete it. ![copy, edit, delete api key](/img/others/create_api_key_community_2.png) ## How to Create API Keys Programmatically For enterprise users, it is also possible to generate keys programmatically. If you are managing the work of multiple users, who's data, models, and concepts that need to be segregated, we recommend you create keys this way. This ensures that each individual user only has access to their own private resources. :::note You need to use a [Personal Access Token (PAT)](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) to create an API Key. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonKey from "!!raw-loader!../../../code_snippets/api-guide/authentication/key.py"; import CurlKey from "!!raw-loader!../../../code_snippets/api-guide/authentication/key.sh"; ```bash curl --location --request POST "https://api.clarifai.com/v2/users/YOUR_USER_ID_HERE/keys" \ --header "Content-Type: application/json" \ --header "Authorization: Key YOUR_PAT_HERE" \ --data-raw '{ "keys": [ { "description": "All permissions", "scopes": [ "All" ], "apps": [ { "id": "YOUR_APP_ID_HERE", "user_id": "YOUR_USER_ID_HERE" } ] } ] }' ``` :::note - API Keys do not expire. In case your key gets compromised, you should delete it, and create a new one with the same scopes. - We recommend that you do **not** share your API Key with other users. ::: ## How to Use an API Key Example Here is an example of how to use an API Key to make a prediction request from your own model. Note that your `user_id` and `app_id` are already tied to the key, so no need to specify them. :::tip Set API Key as an Environment Variable It's a good practice to load your API Key from an environment variable. Keeping your key in a secrets manager, and not in the source code, improves its security and management. ::: {PythonKey} {CurlKey} --- ## Personal Access Tokens # Personal Access Tokens **Authenticate access to your own resources and those outside the scope of your apps** A Personal Access Token \(usually shortened as PAT\) is a kind of key that authenticates your connection to the Clarifai platform. It's not linked to a specific application. A PAT represents you when accessing the Clarifai API. It's a powerful way of accessing your resources within the Clarifai platform. You can use PATs to accomplish various tasks, including: - Access multiple apps with a single key. This lets you access your own apps as well as any other apps you have permission to use, such as public apps, apps where you're added as a collaborator, or apps belonging to your organization's team. - Create apps and app-specific keys programmatically through the API. This is crucial for programs that segregate the data of each of their end-users into different apps. > _**Note:** PAT is the primary authentication mechanism we use. For example, when using a PAT to access a resource, you need to specify the **user ID** alongside the **application ID** of the owner of the resource you want to access — be it your own or for another user. If accessing your own resources, you specify your own `user_id` and `app_id`; if you don't own them, specify the owner's `user_id` and `app_id`._ :::caution PAT Versus API Key A PAT allows you to make inferences on resources that are outside the scope of your apps. [An API Key](https://docs.clarifai.com/clarifai-basics/authentication/app-specific-api-keys) only allows you to access resources scoped to the app defined by that key. So, you can use an API Key to access your own resources, but not Clarifai's or other public resources. ::: ## How to Create a PAT on the Platform A default PAT is automatically generated for you when you create an account on the Clarifai platform. Nonetheless, you can also create a new PAT explicitly on the platform. To create it, [log in](https://clarifai.com/login) to the Clarifai platform. Then, in the collapsible left sidebar, select **Settings** and choose **Secrets** from the dropdown list. ![Create new PAT on Community](/img/others/create_pat_community.png) On the ensuing **Secrets** page, click the **Create Personal Access Token** button. Next, on the form that pops up, provide a short token description, set the [scopes](https://docs.clarifai.com/control/authentication/scopes/) you want to apply, and click the **Create Personal Access Token** button. ![create pat](/img/others/pat_dialog_box.png) The new PAT will be listed in the **Personal Access Token** section, where you can copy, view, edit, or delete it. ![listed pat](/img/others/pat_section_1.png) :::tip note - PATs do not expire. In case your PAT gets compromised, you should delete it, and create a new one with the same scopes. - We recommend that you do **not** share your PAT with other users. ::: ## Set PAT as an Environment Variable It’s recommended to load your PAT from an environment variable. Storing it this way, instead of hardcoding it in your code, enhances both security and manageability. Here is how you can set it as an environment variable. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ## Example Here is an example of how to use a PAT to make a prediction request. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonPat from "!!raw-loader!../../../code_snippets/api-guide/authentication/pat.py"; import PythonSDKPat from "!!raw-loader!../../../code_snippets/api-guide/authentication/pat-sdk.py"; import CurlPat from "!!raw-loader!../../../code_snippets/api-guide/authentication/pat.sh"; {PythonSDKPat} {PythonPat} {CurlPat} --- ## Scopes # Scopes **Improve your apps' security** Scopes provide control over the set of functionality and features available when using our API. Modifying scopes allows for fine-grained control over the data that users have access to, and can help keep your app secure from unauthorized intrusions. You can control scopes for your apps at three different levels: - **[App-Specific API Keys](https://docs.clarifai.com/clarifai-basics/authentication/app-specific-api-keys)** — Control access to resources used by a specific app and a specific user. - **[Personal Access Tokens (PATs)](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens)** — Control access to resources available to a specific user. - **[Collaboration](https://docs.clarifai.com/clarifai-basics/applications/collaboration)** — Control access to resources available to collaborators. ## Combining Scopes A variety of use cases can be addressed by selecting different combinations of scopes. For example, you might want to create an app that only has access to the Search endpoint, but for search to work properly, it needs to access Predict at the operation level. This lets you perform advanced visual searches like searching by an image crop, which first needs to be understood with the prediction before search is performed. By giving the combination of predict op-level, but only search endpoint, you can create an app that can perform searches, but not model predictions \(like [PostModelOutputs](https://docs.clarifai.com/api-guide/predict/images)\). Since collaborators need to create an API key to access the app they are invited to \(or use a PAT\), the scopes that are attached to the collaborators invited to an app will be intersected with the scopes attached to the API key or PAT. Therefore, the permissions allowed when making requests will be the minimum set of scopes from that intersection. This ensures that the app owner remains in full control of the permissions they want their collaborators to have. ## Operations and Endpoints Scopes You have control over both the operation and endpoint level scopes. :::tip You can check an up-to-date list of the scopes available in your plan when creating or editing an API key or a PAT. ::: - **Operation level scopes** — These provide control over the ability to read, write, or delete a given resource type. Examples include `Annotations:Add (Write Annotations)`, `Concepts:Get (Read Concepts)`, and `Models:Delete (Delete Models)`. - **Endpoint level scopes** — These give you control over access to specific endpoints. Examples include `/clarifai.api.V2/GetConcept`, `/clarifai.api.V2/DeleteInputs`, and `/clarifai.api.V2/PostModelOutputs`. ### Annotation: Add and Remove Annotations on Inputs - Annotations:Add - Annotations:Get - Annotations:Delete - /clarifai.api.V2/DeleteAnnotation - /clarifai.api.V2/DeleteAnnotations - /clarifai.api.V2/GetAnnotation - /clarifai.api.V2/ListAnnotations - /clarifai.api.V2/PatchAnnotations - /clarifai.api.V2/PatchAnnotationsStatus - /clarifai.api.V2/PostAnnotations ### Collaborator: Add, Remove and Patch Collaborators - Collaborators:Add - Collaborators:Get - Collaborators:Delete - /clarifai.api.V2/DeleteCollaborators - /clarifai.api.V2/ListCollaborators - /clarifai.api.V2/PatchCollaborators - /clarifai.api.V2/PostCollaborators ### Collector: Add, Remove and Patch Input Collectors - Collectors:Add - Collectors:Get - Collectors:Delete - /clarifai.api.V2/DeleteCollectors - /clarifai.api.V2/GetCollector - /clarifai.api.V2/ListCollectors - /clarifai.api.V2/PatchCollectors - /clarifai.api.V2/PostCollectors ### Concept: Add, Remove and Patch Concept - Concepts:Add - Concepts:Get - Concepts:Delete - /clarifai.api.V2/GetConcept - /clarifai.api.V2/GetConceptCounts - /clarifai.api.V2/GetConceptLanguage - /clarifai.api.V2/ListConceptLanguages - /clarifai.api.V2/ListConceptReferences - /clarifai.api.V2/ListConcepts - /clarifai.api.V2/PatchConceptLanguages - /clarifai.api.V2/PatchConcepts - /clarifai.api.V2/PostConceptLanguages - /clarifai.api.V2/PostConcepts - /clarifai.api.V2/PostConceptsSearches ### Input: Add and Remove Inputs to an App - Inputs:Add - Inputs:Get - Inputs:Delete - /clarifai.api.V2/DeleteInput - /clarifai.api.V2/DeleteInputs - /clarifai.api.V2/GetInput - /clarifai.api.V2/GetInputCount - /clarifai.api.V2/ListInputs - /clarifai.api.V2/ListModelInputs - /clarifai.api.V2/PatchInputs - /clarifai.api.V2/PostInputs - /clarifai.api.V2/PostInputsFile ### Model: Add, Remove and Train Custom Models - Models:Add - Models:Get - Models:Delete - Models:Train - /clarifai.api.V2/DeleteModel - /clarifai.api.V2/DeleteModelVersion - /clarifai.api.V2/DeleteModels - /clarifai.api.V2/GetModel - /clarifai.api.V2/GetModelOutputInfo - /clarifai.api.V2/GetModelVersion - /clarifai.api.V2/GetModelVersionMetrics - /clarifai.api.V2/ListModelVersions - /clarifai.api.V2/ListModels - /clarifai.api.V2/PatchModelVersions - /clarifai.api.V2/PatchModels - /clarifai.api.V2/PostModelVersionMetrics - /clarifai.api.V2/PostModelVersions - /clarifai.api.V2/PostModels - /clarifai.api.V2/PostModelsSearches ### Predict: Predict on Public and Custom Models - Predict - /clarifai.api.V2/PostModelOutputs - /clarifai.api.V2/PostWorkflowResults ### Search: Search over Inputs an App - Search - /clarifai.api.V2/DeleteSearch - /clarifai.api.V2/GetSearch - /clarifai.api.V2/ListSearches - /clarifai.api.V2/PatchSearches - /clarifai.api.V2/PostSearches - /clarifai.api.V2/PostSearchesByID ### Task: Add, Remove and Patch Scribe labeling Tasks - Tasks:Add - Tasks:Get - Tasks:Delete - /clarifai.api.V2/DeleteTasks - /clarifai.api.V2/GetTask - /clarifai.api.V2/ListTasks - /clarifai.api.V2/PatchTasks - /clarifai.api.V2/PostTasks ### Vocab: Add, Remove and Delete lists of Concepts - /clarifai.api.V2/DeleteVocab - /clarifai.api.V2/DeleteVocabConcept - /clarifai.api.V2/DeleteVocabConcepts - /clarifai.api.V2/DeleteVocabs - /clarifai.api.V2/GetVocab - /clarifai.api.V2/ListVocabConcepts - /clarifai.api.V2/ListVocabs - /clarifai.api.V2/PatchVocabs - /clarifai.api.V2/PostVocabConcepts - /clarifai.api.V2/PostVocabs ### Workflow: Add, Remove and Delete Workflows of Models - Workflows:Add - Workflows:Get - Workflows:Delete - /clarifai.api.V2/DeleteWorkflow - /clarifai.api.V2/DeleteWorkflows - /clarifai.api.V2/GetWorkflow - /clarifai.api.V2/ListWorkflows - /clarifai.api.V2/PatchWorkflows - /clarifai.api.V2/PostWorkflows --- ## Two-Factor Authentication # Two-Factor Authentication **Use 2FA to add an extra layer of security to your account** Two-factor authentication (2FA) is an optional sign-in security feature that provides an additional layer of security to your account. ## Why 2FA? Passwords have been the main form of authentication since the start of the digital revolution. However, this security measure is far from infallible. Here are some concerns about the traditional password-based authentication: * 90% of passwords can be cracked in less than six hours; * 2/3 of people use the same password everywhere; * Sophisticated cyber attackers have the power to test billions of passwords every second. The vulnerability of passwords is the main reason for requiring and using 2FA. ### Who Needs 2FA? As a security best practice, you should enable 2FA to minimize the risk of cybersecurity attacks on your account. Despite the interruption in the sign-in process, the additional security benefit is significant. For enterprise users, 2FA might fall under security-related controls for compliance reasons. Organizations might want to enforce 2FA for their users to ensure account and data safety. ### How 2FA Works on the Clarifai Platform Clarifai offers two-factor authentication to add an extra layer of security to your account. When enabled, 2FA prompts you for a unique code from your authenticator application in addition to your username and password during login. Here's what to expect: - **Simple activation**: You can enable or disable 2FA directly from your user security settings; - **Recovery option**: A 2FA recovery method is available in case you lose your authentication device; - **Seamless workflow**: Enabling 2FA only adds a single step to the standard sign-in process, ensuring a smooth user experience. ## 2FA Set Up To enable 2FA, sign in to the platform first. Then, in the collapsible left sidebar, select **Settings** and choose **Account Security** from the dropdown list. You'll be directed to the **Security** settings page. ![](/img/others-2/2fa-1.png) > **Note:** On the **Security** page, you can update the password used to access your Clarifai account by entering your current password and a new one, or by generating a new password automatically. To set up 2FA, toggle the **Two Factor Authentication** "Enable" button. ![](/img/others-2/2fa-2.png) Once the button is enabled: **1.** Scan the given image with a two-factor authentication app, such as Duo, Google Authenticator, or Authy. **2.** After scanning the image, the app will display a six-digit code that you can enter in the provided field. Then, click the **Verify** button. After enabling 2FA for your account, you'll receive a recovery code. This code serves as a backup to reset two-factor authentication in case you lose access to your authenticator app. > **Note:** It's important to copy, print, or download your recovery code and store it in a secure location. ![](/img/others-2/2fa-3.png) Lastly, click the **Close** button to finish the two-factor authentication setup. ### 2FA Sign-In When signing in to the Clarifai platform with the correct username and password, 2FA is activated. A pop-up prompt will appear on the screen requesting the authentication code. Enter the six-digit code from the authenticator app and click the **Verify** button. You will be successfully signed in to the platform. ![](/img/others-2/2fa-4.png) ### 2FA Disabling To disable 2FA, follow these steps: **1.** Go to the **Security** settings page, as earlier described; **2.** Toggle the **Two Factor Authentication** "Disable" button; **3.** You will be prompted to enter a six-digit code from your authenticator application; ![](/img/others-2/2fa-8.png) **4.** Once you enter the correct code and click the **Disable** button, 2FA will be successfully removed from your account. ## 2FA Recovery ### Self-Serve If you lose access to your authentication app, you won't be able to sign in with 2FA. To regain access to the Clarifai platform, follow these steps: **1.** Go to the sign-in page; **2.** Enter your correct username and password; **3.** When prompted for the 2FA code, click on the **Enter recovery code** link; ![](/img/others-2/2fa-5.png) **4.** A window will appear, asking for the recovery code that was created during the initial setup process; ![](/img/others-2/2fa-6.png) **5.** Enter the recovery code and click the **Recover** button; **6.** A message will be sent to your account's email address to confirm the TOTP reset. Click the link provided in the email; ![](/img/others-2/2fa-7.png) **7.** 2FA will be disabled for your account, allowing you to [set up](#2fa-set-up) a new two-factor authentication process. ### Last-Resort If you lose your recovery code, no self-serve method exists for regaining access to the Clarifai platform. In this case, you'll need to contact support@clarifai.com for assistance in recovering access. --- ## Clarifai Organizations # Clarifai Organizations **Consolidate multiple accounts and enhance collaboration** Clarifai Organization, commonly referred to as the Org feature, is a functionality within our platform that lets you consolidate multiple Clarifai accounts into an organization, allowing you to enhance collaboration and drive better results with your AI-powered software solutions. With the Org feature, you can create and centrally manage your company’s machine learning projects. It allows you to consolidate your team’s capabilities so that you can realize the compliance, security, and budgetary goals of your enterprise. ## Capabilities of Organizations The Org feature offers the following capabilities: - **Centralize management of your Clarifai operations** — An Organization enables you to bring your collaborators, teams, applications, and other resources within Clarifai under a single umbrella. For example, if you’re an administrator of an organization, you can invite existing Clarifai users to be members of your organization, which allows you to manage their accounts from a central point. This centralized control makes it easy to track your usage of the Clarifai platform and get the most out of it. - **Consolidate billing for all members of an organization** — A Clarifai Organization lets you consolidate the billing for all member accounts so that you can conveniently track and pay for each account. Having one bill across your organization allows you to monitor the charges across multiple accounts and optimize the cost performance of your organization. - **Enact access policies** — As an administrator, you can set up policies that dictate how users can access the resources within your organization. You can also enact role-based security to restrict access to resources based on a member’s privileges within the organization. You can allow members to access only the applications necessary to perform their assigned duties effectively. - **Set up siloed access permissions** — As an administrator, you can set up various teams and attach different access permissions to them. You can specify which applications and individual actions the users in each team can access. For example, if you have members within your organization who must access only the resources that meet certain data regulatory requirements, you can aggregate those accounts in their own team. You then define a policy that prevents members of that team from accessing resources that do not comply with those regulatory standards. ## Org Feature Availability The Clarifai Organization feature is available to all users of our platform. However, the extent of its functionality depends on your [pricing plan](https://www.clarifai.com/pricing). Below is a table outlining the feature limitations for each plan. :::tip Learn more about the scopes and access levels of organization members [here](https://docs.clarifai.com/portal-guide/clarifai-organizations/security#scopes-and-access-levels-of-organization-members). ::: | Type of Plan | Org Functionalities Available| |-----------------------|------------------------| | Pay-As-You-Go | Limited to a single organization and does not support teams or custom roles | | Enterprise | Supports up to 20 organizations, along with multiple teams and configurable roles | ## How to Create an Organization To create an Org account, start by logging in to the [Clarifai platform](https://clarifai.com/explore). Next, click the plus (**+**) button in the upper-right corner of the Clarifai platform dashboard. A dropdown menu will appear with several options. From the list, select **New Organization**. ![Add an organization](/img/clarifai_orgs/add_an_organization.png) In the dialog box that appears, provide the details of your organization. ![Create an organization](/img/clarifai_orgs/create_organization.png) - **Organization name:** Enter a unique name for your organization, such as _test-org_. - **Organization ID:** Enter a unique identifier for your organization, such as _123455_. This is what will appear in the URL when browsing the Clarifai platform as an organization. - **Billing email:** Enter your billing email, such as test-org@example.com. Click the **Create organization** button to finalize creating the organization. And your new Clarifai Org will be created and available under your account! :::info You can create an Org account when registering for an account on the Clarifai platform. When activating your account, you'll be presented with a form that allows you to create an organization and invite members to join it. ::: ## How to Use an Org After creating a Clarifai organization, you can begin interacting with our platform as an organization — just like a normal user would, albeit with some other added capabilities. To start accessing the platform with your organization account, click the workspace switcher icon in the collapsible left sidebar and select your organization in the dropdown list. You’ll notice that your user’s profile will change to your organization’s profile. ![Select your organization’s profile](/img/clarifai_orgs/select_organization_profile-2.png) ## How to Manage an Org To manage your organization, click your user profile icon in the upper-right corner of the Clarifai dashboard and select **Manage Organizations** from the dropdown menu. ![Select your organization’s profile](/img/clarifai_orgs/select_organization_profile.png) You’ll be redirected to the **Organizations** management page, which provides a centralized location for accessing and managing all your Clarifai organizations. ![Create an organization](/img/clarifai_orgs/how_to_use_org.png) From this page, you can: - View the list of organizations you belong to - Switch between different organizations - Create a new organization - Leave an organization you belong to - Delete an organization you own To open an organization, locate it in the **Organization ID** column and click on its corresponding row. ### Organization Settings When [logged in](#how-to-use-an-org) as an organization, select the **Settings** option in the collapsible left sidebar to access a dropdown with settings for managing different aspects of your Clarifai organization. ![Select your organization’s profile ](/img/clarifai_orgs/select_organization_profile_2.png) - **Account** — Allows you to update your organization’s profile, delete the account information related to your organization, or leave the organization. - **Security** — Allows you to enforce the [security](security#enforce-your-clarifai-organizations-security) of your organization by setting password policies and creating identity providers. - **Billing** — Allows you to access and manage your subscription and billing information, including adding a credit card. - **Members** — Allows you to access and manage the [members](members-teams/#organization-members) associated with your organization. - **Teams** — Allows you to access and manage the [teams](members-teams#organization-teams) associated with your organization. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Apps, Models, Workflows, & Compute # Apps, Models, Workflows, & Compute **Create and manage apps, models, workflows, and dedicated compute** In a Clarifai organization, you have the ability to create and manage various resources, such as [apps](https://docs.clarifai.com/create/applications/), [models](https://docs.clarifai.com/create/models/), [workflows](https://docs.clarifai.com/create/workflows/), and even set up [dedicated compute](https://docs.clarifai.com/compute/deployments/clusters-nodepools) for deploying models. The process for handling these resources is consistent with how they are managed across the rest of our platform. This ensures a seamless experience whether you are working within an organization or individual account. You can visit the respective sections within this documentation to understand how to effectively create and manage these resources. This will help you make the most of the Org feature and enhance collaboration within your teams. ## Transfer an App You can transfer an app you own to an organization you belong to, provided you have the necessary [permissions](https://docs.clarifai.com/portal-guide/clarifai-organizations/security#scopes-and-access-levels-of-organization-members) to create apps. This transfer enhances collaboration and boosts productivity within your team. To begin, go to your personal workspace and, from the collapsible left sidebar, select **Projects**, then choose **All Resources** from the dropdown list. This opens a page displaying all the resources you own, with the **Apps / Templates** tab selected by default. From there, select the app you want to transfer to an organization. ![List apps you own](/img/clarifai_orgs/transfer_app_org_1.png) > **Alternatively:** You can use the **Select Application** dropdown in the collapsible left sidebar to quickly locate and choose the app you wish to transfer. You’ll then be redirected to the app’s individual page. Select the **Settings** option in the collapsible left sidebar. ![App setting](/img/clarifai_orgs/transfer_app_org_2.png) On the **App Settings** page, scroll down to the **Transfer ownership** section and click the **Transfer** button. ![Transfer app ownership](/img/clarifai_orgs/transfer_app_org_3.png) On the dialog box that appears, select the app’s destination organization and give it a new name, if you want to. Click the **Confirm** button to finalize the transfer. ![Give app a new name](/img/clarifai_orgs/transfer_app_org_4.png) The app will be successfully moved to the organization you selected, and it will no longer appear on your listed apps page. ![Org apps](/img/clarifai_orgs/transfer_app_org_5.png) --- ## Members and Teams # Members and Teams **Learn how to add and manage members and teams** Having members and teams within your Clarifai organization enhances productivity, ensures efficient task management, and supports the collaborative development of your machine learning projects. Let's illustrate how to add and manage members and teams within your Clarifai organization. ## Organization Members Organization members are individuals who have been invited to join an organization. They are assigned specific roles that define their level of access, responsibilities, and permissions within the organization. ### How to Add Members [Log into](README.mdx#how-to-use-an-org) your organization’s account. Then, in the collapsible left sidebar, select **Settings** and choose **Members** from the dropdown list. You’ll be redirected to the **Members** page, where you can add new members and carry out various member management tasks. ![Account option](/img/clarifai_orgs/account_option.png) To add a new team member to your organization, you need to send them an invitation first. Click the **Invite Members** button to do so. In the dialog box that appears, enter the invitee’s email address and select a role you wish to assign to them, [as described below](#roles-for-members). > **Note:** Role-based access control is available only on the Enterprise [plan](https://www.clarifai.com/pricing). Pay-As-You-Go users can invite members with Admin roles only. ![Invite member dialog box](/img/clarifai_orgs/invite_member_dialog_box.png) After providing the details, click the **Confirm** button. > When you send an invitation, the invitee will receive a notification in their email address as well as in the platform's notification bar for them to decide whether to accept or decline the request. If they do not already have an account with Clarifai, they will need to register first before joining your organization. > You can track the invitations in the **Pending invitations** tab. If you want to rescind any pending invitations, you can also delete them from there. > ![Pending Invitations](/img/clarifai_orgs/pending_invitation.png) :::note #### Roles for Members | **Role** | **Permissions Description** | |----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Admin** | The invitee will have administrative privileges in the organization | | **Financial Manager** | The invitee will have access to key [financial data](https://docs.clarifai.com/control/control-center/costs), including current spending based on the organization's pricing plan and budget settings, enabling better financial oversight and management | | **Organization Contributor (all apps)** | The invitee will be able to contribute to all the apps within the organization, but without some admin privileges | | **Organization User (all apps)** | The invitee will have access privileges similar to those of an Organization Contributor for all apps, but with view-only permissions without ability to perform certain CRUD operations — create, update, or delete resources | | **Team Contributor (select apps)** | The invitee will be able to contribute only to some selected applications | | **Infrastructure Manager** | The invitee will be able to create, modify, and delete [clusters and nodepools](https://docs.clarifai.com/compute/deployments/clusters-nodepools) | ::: :::tip [Click here](security.md#scopes-and-access-levels-of-organization-members) to learn more about the scopes and access levels allowed for each user type within a Clarifai organization. ::: ### How to Manage Members Once invitees have accepted your invitation, their details will appear in the **Organization members** tab. From there, you can view team members, reassign roles, or delete members. ![Current members section](/img/clarifai_orgs/current_members_section.png) :::warning multi-org membership Multi-organization membership is available exclusively on the Enterprise plan, allowing users to create, join, and participate in multiple organizations simultaneously. ::: ## Organization Teams Teams allow you to bring different members of your organization together to accomplish specific tasks. For example, you can aggregate users with the same access privileges to certain apps together in a single team. > **Note:** The teams functionality is available only to users on the Enterprise plan. ### How to Add Teams [Log into](README.mdx#how-to-use-an-org) your organization’s account. Then, in the collapsible left sidebar, select **Settings** and choose **Teams** from the dropdown list. You’ll be redirected to the **Teams** page, where you can create new teams and carry out various team management tasks. ![ ](/img/clarifai_orgs/organization_teams_1.png) Click the **Create Team** button to add a new team to your organization. ![Create team dialog box](/img/clarifai_orgs/create_team_dialog_box-2.png) In the dialog box that appears, enter the team ID, team name, and set the permissions for the team (the permissions are [described below](#permission-types-for-teams)). Click the **Confirm** button to finalize creating the team. ![Create team dialog box](/img/clarifai_orgs/create_team_dialog_box.png) Your newly created team will appear on the **Teams** listing page. You can view other teams you've added and delete them from that table. ![Teams listing page](/img/clarifai_orgs/teams_listing_page.png) :::note #### Permission Types for Teams | **Role** | **Description** | **Capabilities** | **Notes / Best Practices** | |-------------------|---------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------| | **Full Access** | Highest level of access with complete administrative control over the app | Access to all app features and functionalitiesFull CRUD (Create, Read, Update, & Delete) operations on all resources | Should only be granted to team members requiring full administrative control | | **Model Trainer** | For team members focused on model development and training | View all modelsCreate, update, delete modelsTrain modelsAccess model metrics & performance dataView inputs & conceptsAccess datasetsDeploy and view models in clustersView clusters and nodepools | Assign to members responsible for developing and maintaining AI models | | **Labeler Lead** | Supervisory role for labeling operations and quality control | All Labeler Worker permissions, plus:Create labeling tasksAssign tasksMonitor task progressView team member details (first/last names)Access task analyticsReview annotationsProvide feedbackCreate, modify, delete concepts | Serves as supervisors for labeling operations and ensures quality control | | **Labeler Worker**| Basic role focused on performing labeling tasks | Access assigned labeling tasksRead access to: concepts within tasks, input data for assigned tasks, task instructions, and guidelinesWrite access to: annotations, task status updates, progress reportsEvaluate & assess completed work for quality and consistency | Entry-level labeling role with limited permissions | ::: ### How to Manage a Team When you click on a team from the **Teams** listing page, you’ll be taken to its individual page, where you can manage and perform various team-related tasks. For example, click the three-dot menu in the upper-right corner to open a dropdown list, where you can select either **Edit Team Name** to rename the team or **Delete Team** to remove it. ![manage a team](/img/clarifai_orgs/how_to_manage_team.png) #### Add and Manage Team Members To add members to a team, first select the **Members** tab. Then, click the **Add Members** button in the upper-right section of the page. ![Individual team page](/img/clarifai_orgs/individual_team_page.png) In the dialog box that appears, search for a member(s) of your organization and click the **Add users** button to add them to the team. Note that you can only add existing members of your organization to a team. ![Add members to your team](/img/clarifai_orgs/add_team_members_dialog_box.png) The newly added member will appear in the **Members** tab, where you can also view all existing members and remove them if needed. ![ ](/img/clarifai_orgs/add_team_members_2.png) #### Add and Manage Team Apps To assign an app to a team, first select the **Apps** tab. Then, click the **Add App** button in the upper-right section of the page. ![add app to team](/img/clarifai_orgs/add_app_to_team.png) > **Note**: You can set the default access level for your team members when adding apps — Full Access, Labeler Worker, Labeler Lead, or Model Trainer, [as explained previously](#permission-types-for-teams). Use the dropdown menu on the right side of the apps listing page to set these permissions, which you can always customize later if needed. In the dialog box that appears, search for the app(s) you want to add to your team. Note that you can only add apps that already belong to your organization — either those you created or those [transferred](apps-models-workflows.md#transfer-an-app) into it. From the same dialog, you can also set the permissions for the added app(s). Next, click the **Add apps** button in the dialog box to add the app(s) you've selected to the team. ![Add apps to your team](/img/clarifai_orgs/apps_dialog_box.png) The newly added app will appear on the apps listing page, where you can also view other apps, edit their permissions, or delete them. ![newly added app](/img/clarifai_orgs/gs_3.png) :::note tip On the **App Settings** page of an organization app, you can view the teams that have access to the app, edit their permissions, and add or delete teams. ![newly added app](/img/clarifai_orgs/gs_4.png) ::: --- ## Security # Security **Learn how to enhance the security of your Clarifai organization** At Clarifai, we understand that the security of your organization is of utmost importance. We are committed to prioritizing security and have designed our platform to cater to the needs of even the most security-conscious companies. Our robust security measures ensure the protection and confidentiality of your valuable data. ## Scopes and Access Levels of Organization Members Scopes and access levels play a crucial role in defining the boundaries of the components, resources, or data that members of an organization can access or manipulate. Every member of a Clarifai organization has their own pre-set scopes that determine what actions they can perform and what resources they can access. The permissions allowed for each user type help to protect your organization’s resources from unauthorized infiltration, maintain the privacy of your assets, and ensure trackability of resources by knowing who has access to what. [Click here](members-teams.md#roles-for-members) to see a summary of the permissions assigned to each role. Here is a table detailing the scopes and access levels allowed for each user type within a Clarifai organization. | Permission | Administrator | Financial Manager | Organization Contributor (all apps) | Team Contributor (select apps) | Organization User (all apps) | Infrastructure Manager | | --------------------------------------------- | ------------- | ----------------- | ----------------------------------- | ------------------------------ | ---------------------------- | ---------------------- | | View Sidebar Items | Yes | | Yes | Yes | Yes | | | Create Models | Yes | | Yes | Yes | No | | | Create Workflows | Yes | | Yes | Yes | No | | | Edit App ID | Yes | | Yes | No | No | | | Edit App Description | Yes | | Yes | Yes | No | | | Edit App Notes | Yes | | Yes | Yes | No | | | View Collaborators | Yes | | Yes | Yes | Yes | | | Edit Collaborators | Yes | | Yes | No | No | | | Base Workflow View | Yes | | Yes | Yes | Yes | | | Base Workflow Edit | Yes | | Yes | No | No | | | Delete All Models | Yes | | Yes | Yes | No | | | Delete App | Yes | | Yes | No | No | | | Invite / Modify / Delete Members | Yes | | No | No | No | | | Create / Edit / Delete Teams | Yes | | Yes | No | No | | | Add Members to Teams | Yes | | Yes | No | No | | | Add Apps to Teams | Yes | | Yes | No | No | | | Create Apps | Yes | | Yes | No | No | | | Change Visibility of Apps | Yes | | Yes | No | No | | | Change Visibility of Apps Resources | Yes | | Yes | Yes | No | | | Use Modules | Yes | | Yes | Yes | Yes | | | Create / Delete Modules | Yes | | Yes | Yes | No | | | Install / Uninstall Modules | Yes | | Yes | Yes | No | | | Create / Edit / Delete Clusters and Nodepools | Yes | No | No | No | No | Yes | | Deploy and View Models in Clusters | Yes | No | Yes | No | No | Yes | | View Clusters and Nodepools | Yes | No | Yes | No | Yes | Yes | | View Financial Data | Yes | Yes | No | No | No | No | ## Enforce Your Clarifai Organization's Security You can take some additional actions to augment the security of your organization at Clarifai. To access the security settings of your organization, [log into](README.mdx#how-to-use-an-org) your organization’s account. Then, in the collapsible left sidebar, select **Settings** and choose **Security** from the dropdown list. ![Security organization settings](/img/clarifai_orgs/security_option.png) On the **Security** page, you can view the password policies associated with your organization’s account. You can also create and view identity providers to reinforce the security of your organization. For example, if you want to add a security identity provider, such as SAML (Security Assertion Markup Language), click the **Create Identity Provider** button. > **Note:** SAML is a reliable mechanism that can add to the security of your organization’s account by ensuring authentication and authorization data are securely transferred between parties. On the next page, provide the required details, toggle the enable button, and click the **Create** button to finalize the process. ![Create identity provider](/img/clarifai_orgs/create_identity_provider.png) --- ## Control Center # Control Center **A unified interface for centralized visibility into your platform usage** Clarifai Control Center is a unified dashboard, a single pane of glass to monitor everything happening within your account in our platform. It serves as the sole source of truth for various information dimensions, enabling you to make informed decisions based on data from different sources. The Control Center is organized into multiple tabs and offers a versatile and intuitive way to explore specific details. The key functionalities include charts of different types, tables, date ranges, as well as filtering and sorting options. > _**Note:** Although the video above showcases an older version of our UI, it still accurately illustrates how the Control Center dashboard functions._ ## Your Central Hub for AI Operations The Control Center is your go-to dashboard for managing and monitoring everything you do on the Clarifai platform. It is designed to support you in a variety of scenarios: * **Unified operations view** — Access all your Clarifai operations from one screen, reducing the need to switch between tools or browser tabs. * **Utilization tracking** — Visualize resource usage through interactive charts, graphs, and tables to make data-driven decisions and maximize AI adoption across your organization. * **Cost and resource management** — Centralize financial and usage data to monitor spending in real time, explore detailed cost breakdowns, and identify opportunities for optimization. * **Insight generation** — Convert raw data, such as model predictions, search activities, input storage, and training hours, into actionable insights. * **System health monitoring** — Track the technical performance of your models and deployments, including latency, error rates, and overall usage. * **Team activity insights** — Analyze team behavior to understand its impact on operational efficiency and platform spending. * **Audit and traceability** — Review user actions with full audit trails to ensure accountability and trace the origin of any event. ## How to Access the Control Center To access the Control Center, log in to Clarifai’s [platform](https://clarifai.com/explore). Then, in the collapsible left sidebar, select **Control Center** and choose **Overview** from the dropdown list. You’ll be taken to the Control Center's **[Overview](overview)** tab. ![](/img/community/control-center/control_center_1.png) import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Audit Logging via API # Audit Logging via API **Track what was performed, who did it, and the results** Clarifai's Audit Logging feature helps you monitor platform activities for better visibility, security, and governance. It captures detailed logs of operations so you can know what was done, who did it, and the results. With Audit Logging, you can assess the actions performed on users and their resources, track resource changes to identify potential security issues, and maintain comprehensive activity logs to meet regulatory requirements. :::tip To learn how to perform audit tracking via the UI, see [Teams & Logs](teams-logs.md) within the Control Center. ::: :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [CLI](https://docs.clarifai.com/additional-resources/api-overview/cli), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonListAuditLogs from "!!raw-loader!../../../code_snippets/api-guide/audit-log/list_audit_logs.py"; import JSListAuditLogs from "!!raw-loader!../../../code_snippets/api-guide/audit-log/list_audit_logs.html"; import NodeListAuditLogs from "!!raw-loader!../../../code_snippets/api-guide/audit-log/list_audit_logs.js"; import CURLListAuditLogs from "!!raw-loader!../../../code_snippets/api-guide/audit-log/list_audit_logs.sh"; import PythonOperation from "!!raw-loader!../../../code_snippets/api-guide/audit-log/operation.py"; import JSOperation from "!!raw-loader!../../../code_snippets/api-guide/audit-log/operation.html"; import NodeOperation from "!!raw-loader!../../../code_snippets/api-guide/audit-log/operation.js"; import CURLOperation from "!!raw-loader!../../../code_snippets/api-guide/audit-log/operation.sh"; import PythonTime from "!!raw-loader!../../../code_snippets/api-guide/audit-log/time.py"; import JSTime from "!!raw-loader!../../../code_snippets/api-guide/audit-log/time.html"; import NodeTime from "!!raw-loader!../../../code_snippets/api-guide/audit-log/time.js"; import CURLTime from "!!raw-loader!../../../code_snippets/api-guide/audit-log/time.sh"; import PythonUser from "!!raw-loader!../../../code_snippets/api-guide/audit-log/user.py"; import JSUser from "!!raw-loader!../../../code_snippets/api-guide/audit-log/user.html"; import NodeUser from "!!raw-loader!../../../code_snippets/api-guide/audit-log/user.js"; import CURLUser from "!!raw-loader!../../../code_snippets/api-guide/audit-log/user.sh"; import PythonTarget from "!!raw-loader!../../../code_snippets/api-guide/audit-log/target.py"; import JSTarget from "!!raw-loader!../../../code_snippets/api-guide/audit-log/target.html"; import NodeTarget from "!!raw-loader!../../../code_snippets/api-guide/audit-log/target.js"; import CURLTarget from "!!raw-loader!../../../code_snippets/api-guide/audit-log/target.sh"; import PythonSuccess from "!!raw-loader!../../../code_snippets/api-guide/audit-log/success.py"; import JSSuccess from "!!raw-loader!../../../code_snippets/api-guide/audit-log/success.html"; import NodeSuccess from "!!raw-loader!../../../code_snippets/api-guide/audit-log/success.js"; import CURLSuccess from "!!raw-loader!../../../code_snippets/api-guide/audit-log/success.sh"; import PythonSource from "!!raw-loader!../../../code_snippets/api-guide/audit-log/source.py"; import JSSource from "!!raw-loader!../../../code_snippets/api-guide/audit-log/source.html"; import NodeSource from "!!raw-loader!../../../code_snippets/api-guide/audit-log/source.js"; import CURLSource from "!!raw-loader!../../../code_snippets/api-guide/audit-log/source.sh"; import PythonSort from "!!raw-loader!../../../code_snippets/api-guide/audit-log/sort.py"; import JSSort from "!!raw-loader!../../../code_snippets/api-guide/audit-log/sort.html"; import NodeSort from "!!raw-loader!../../../code_snippets/api-guide/audit-log/sort.js"; import CURLSort from "!!raw-loader!../../../code_snippets/api-guide/audit-log/sort.sh"; import PythonPage from "!!raw-loader!../../../code_snippets/api-guide/audit-log/page.py"; import JSPage from "!!raw-loader!../../../code_snippets/api-guide/audit-log/page.html"; import NodePage from "!!raw-loader!../../../code_snippets/api-guide/audit-log/page.js"; import CURLPage from "!!raw-loader!../../../code_snippets/api-guide/audit-log/page.sh"; import OutputExample1 from "!!raw-loader!../../../code_snippets/api-guide/audit-log/output-1.json"; ## Supported Operations The Audit Logging feature currently supports tracking the following critical resource operations (_we're planning to support more resources in the future_): - **[Organization and team membership activities](https://docs.clarifai.com/control/clarifai-organizations/)** — Includes creating, updating, or deleting organizations and teams, sending invitations, and managing team users and applications. - **[Compute Orchestration](https://docs.clarifai.com/compute/overview)** – Cover creating, modifying, deleting clusters, nodepools, deployments - **[Module activities](https://docs.clarifai.com/create/modules/)** — Tracks the creation, updating, and deletion of modules. - **[Model activities](https://docs.clarifai.com/create/models/)** — Tracks actions such as creating, training, publishing, and deleting models. - **[Workflow activities](https://docs.clarifai.com/create/workflows/)** — Covers the creation, publishing, updating, and deletion of workflows. - **[Application activities](https://docs.clarifai.com/create/applications/create)** — Includes creating, updating, duplicating, and deleting applications. - **[Collaborator activities](https://docs.clarifai.com/create/applications/manage#collaborators)** — Includes adding collaborators, editing their scopes, and removing them. Here is a table that showcases the supported operations. |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `EVENT_TYPE_NOT_SET` | 0 | Event type is not specified, lists all the supported audit log events | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `ORGANIZATION_MEMBER_ADD` | 100 | Organization members added | |`ORGANIZATION_MEMBER_CHANGE` | 101 | Organization members' roles changed | | `ORGANIZATION_MEMBER_REMOVE` | 102 | Organization members removed | | `ORGANIZATION_MEMBER_INVITATION_CREATE` | 103 | Organization member invitations created | | `ORGANIZATION_MEMBER_INVITATION_CANCEL` | 104 | Organization member invitations canceled | | `ORGANIZATION_MEMBER_INVITATION_ACCEPT` | 105 | Organization member invitations accepted | | `ORGANIZATION_MEMBER_INVITATION_DECLINE` | 106 | Organization member invitations declined | | `ORGANIZATION_TEAM_CREATE` | 107 | Organization teams created | | `ORGANIZATION_TEAM_UPDATE` | 108 | Organization teams updated | | `ORGANIZATION_TEAM_DELETE` | 109 |Organization teams deleted | | `ORGANIZATION_TEAM_MEMBER_ADD` | 110 | Organization team members added | | `ORGANIZATION_TEAM_MEMBER_REMOVE` | 111 | Organization team members removed | | `ORGANIZATION_TEAM_APP_ADD` | 112 | Organization team applications added | | `ORGANIZATION_TEAM_APP_REMOVE` | 113 | Organization team applications removed | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `COMPUTE_CLUSTER_CREATE` | 900 | Compute clusters created | | `COMPUTE_CLUSTER_DELETE ` | 901 | Compute clusters deleted | | `NODEPOOL_CREATE ` | 1000 | Nodepools created | | `NODEPOOL_UPDATE` | 1001 | Nodepools updated | | `NODEPOOL_DELETE` | 1002 | Nodepool deleted | | `DEPLOYMENT_CREATE ` | 1100 | Deployments created | | `DEPLOYMENT_UPDATE` | 1101 | Deployments updated | | `DEPLOYMENT_DELETE` | 1102 | Deployments deleted | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `MODULE_CREATE` | 200 | Modules created | | `MODULE_UPDATE` | 201 | Modules updated | | `MODULE_DELETE` | 202 | Modules deleted | | `MODULE_VERSION_CREATE` | 203 | Module versions created | | `MODULE_VERSION_UPDATE` | 204 | Module versions updated | | `MODULE_VERSION_DELETE` | 205 | Module versions deleted | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `MODEL_CREATE` | 300 |Models created | | `MODEL_UPDATE` | 301 | Models updated | | `MODEL_DELETE` | 302 | Models deleted | | `MODEL_VERSION_CREATE` | 303 | Model versions created | | `MODEL_VERSION_UPDATE` | 304 | Model versions updated | | `MODEL_VERSION_DELETE` | 305 | Model versions deleted | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `WORKFLOW_CREATE` | 400 | Workflows created | | `WORKFLOW_UPDATE` | 401 | Workflows updated | | `WORKFLOW_DELETE` | 402 | Workflows deleted | | `WORKFLOW_VERSION_CREATE` | 403 | Workflow versions created | | `WORKFLOW_VERSION_UPDATE` | 404 | Workflow versions updated | | `WORKFLOW_VERSION_DELETE` | 405 | Workflow versions deleted | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `APPLICATION_CREATE` | 600 | Applications created | | `APPLICATION_UPDATE` | 601 | Applications updated | | `APPLICATION_DELETE` | 602 | Applications deleted | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `COLLABORATOR_ADD` | 700 | Collaborators added | | `COLLABORATOR_UPDATE` | 701 | Collaborators updated | | `COLLABORATOR_REMOVE` | 702 | Collaborators removed | |Event Type |Code |Description | |--------------------------------------|--------------------|---------------------------| | `USER_UPDATE` | 800 |Users updated | ## List Audit Log Events Here is how you can list all the supported audit log events performed by a user on the Clarifai platform. {PythonListAuditLogs} {JSListAuditLogs} {NodeListAuditLogs} {CURLListAuditLogs} Output Example {OutputExample1} ## Filter Searches You can optionally refine your searches to retrieve only the operations of interest. This enables targeted audit trails for the activities performed on the Clarifai platform. ### Operation-Based Filtering You can apply filters to target specific operation types. {PythonOperation} {JSOperation} {NodeOperation} {CURLOperation} ### Time-Based Filtering You can specify precise time ranges for your queries to track exactly when operations were performed. Note that for HTTP+JSON requests, timestamps are formatted as `2024-08-25T00:00:00Z`. And for gRPC requests, timestamps use separate `seconds` and `nanos` fields, as detailed here: [Protobuf Timestamp Reference](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp). {PythonTime} {JSTime} {NodeTime} {CURLTime} ### User-Based Filtering You can define specific users or groups in your queries to track who performed each operation. {PythonUser} {JSUser} {NodeUser} {CURLUser} ### Target-Based Filtering You can define the target of your query; that is, specify the resource on which an operation recorded in the audit log was performed. **Target Types Supported** | Target | |-----------------| | `User user` | | `Role role` | | `Team team` | | `App app` | | `Module module` | | `ModuleVersion module_version` | | `Workflow workflow` | | `WorkflowVersion workflow_version ` | | `Model model` | | `ModelVersion model_version` | {PythonTarget} {JSTarget} {NodeTarget} {CURLTarget} ### Success-Based Filtering You can filter logs based on the operation outcome, such as whether it was successful. This also enables you to identify failed attempts and take appropriate action. {PythonSuccess} {JSSuccess} {NodeSuccess} {CURLSuccess} ### Source-Based Filtering You can filter logs based on the IP address where the request originated from. {PythonSource} {JSSource} {NodeSource} {CURLSource} ## Sorting Logs You can specify the sorting options for the audit logs. - If `true`, logs are sorted by timestamp in ascending order (oldest to newest). - If `false`, logs are sorted in descending order (newest to oldest). {PythonSort} {JSSort} {NodeSort} {CURLSort} ## Pagination You can split the results into [pages](https://docs.clarifai.com/api-guide/advanced-topics/pagination), which makes it easier to navigate and review the data. {PythonPage} {JSPage} {NodePage} {CURLPage} ## Additional Actions That Generate Logs In addition to the explicitly defined audit log event types, other API actions — such as `Delete`, `Patch`, and `Post` operations on various Clarifai resources — will automatically generate audit logs when they match supported event types. These actions are part of the broader set of activities tracked by the Audit Logging feature. * `PostAppDuplications` * `PostApps` * `PostCollaborators` * `PostInstalledModuleVersions` * `PostModels` * `PostModelVersions` * `PostModelVersionsPublish` * `PostModelVersionsUnPublish` * `PostModelVersionsUpload` * `PostModules` * `PostModuleVersions` * `PostOrganizationInvitations` * `PostOrganizationMember` * `PostTeamApps` * `PostTeams` * `PostTeamUsers` * `PostWorkflows` * `PostWorkflowVersionsPublish` * `PostWorkflowVersionsUnPublish` * `PatchApp` * `PatchAppOwner` * `PatchApps` * `PatchCollaborators` * `PatchModelCheckConsents` * `PatchModelIds` * `PatchModelLanguages` * `PatchModels` * `PatchModelToolkits` * `PatchModelUseCases` * `PatchModelVersions` * `PatchModules` * `PatchOrganizationInvitations` * `PatchOrganizationMember` * `PatchTeams` * `PatchWorkflows` * `DeleteApp` * `DeleteCollaborators` * `DeleteInstalledModuleVersions` * `DeleteModel` * `DeleteModels` * `DeleteModelVersion` * `DeleteModules` * `DeleteModuleVersions` * `DeleteOrganizationMember` * `DeleteRequestingUserFromOrganization` * `DeleteTeamApps` * `DeleteTeams` * `DeleteTeamUsers` * `DeleteWorkflow` * `DeleteWorkflows` * `DeleteWorkflowVersions` --- ## Costs & Budget Tab # Costs & Budget Tab **Get detailed financial data about your operations** The Costs & Budget tab offers detailed insights into your financial data, allowing self-service users to easily view and monitor the costs associated with billable operations on our platform. It simplifies expense tracking and management, offering greater control over your spending. To access it, go to the [Control Center](README.mdx#how-to-access-the-control-center) and select the Costs & Budget tab in the collapsible left sidebar. ![](/img/community/control-center/costs_budget_1_1.png) The overview section of the Costs & Budget page provides a quick analysis of your total expenses on the Clarifai platform for the selected time period. It tells you: - The total cost of all billable operations - The total cost of [Compute Orchestration](https://docs.clarifai.com/compute/overview) tasks - The total cost of model predictions - The total cost of training models - The total cost of search operations - The total cost of storing inputs :::note Pricing for historical data is based on current rates, without accounting for any past price changes. ::: ## Common Chart Features The Costs & Budget page has interactive charts that help you visualize your expenses on the Clarifai platform for the selected period. Some of the key functionalities include the ability to drag charts, pin your favorites, switch between different chart types, and use tooltips for detailed insights. ![](/img/community/control-center/costs_budget_2-1.png) > **Note:** The charts in the Costs & Budget tab are linked to related charts in the [Usage & Operations](usage.md) tab, and vice versa. These cross-navigation links allow you to seamlessly explore related insights. For example, the Cost of Operations chart includes a link in the lower-right corner that directs you to the corresponding Total Number Operations chart. If you click the **View Report Details** button in the lower-right corner of a chart, you’ll be redirected to a detailed report page. Here, you'll find a comprehensive breakdown of each cost type for the selected period, with options to filter charts by specific costs. The page also includes a table that displays the date each expense was incurred, along with other detailed insights. ![](/img/community/control-center/costs_budget_3.png) :::note Learn more Learn more about using these chart features [here](./usage/#common-chart-features). ::: ## Total Costs This section displays the total cost of all your billable operations, along with the average daily cost, during the selected time period. It helps you track spending across each operation. ![](/img/community/control-center/costs_budget_4.png) ## Cost of Operations This section displays the total cost of all your operations, along with the average daily cost, during the selected time period. It helps you track spending across your model predictions and search operations. ![](/img/community/control-center/costs_budget_8.png) ## Models Cost by Type and Date This section displays the total cost of all your model predictions, along with the average daily cost, during the selected time period. It helps you track spending trends by model type across specific dates. ![](/img/community/control-center/costs_budget_6.png) ## Models Cost by ID and Date This section displays the total cost of all your model predictions, along with the average daily cost, during the selected time period. It helps you track spending trends by model ID across specific dates. ![](/img/community/control-center/costs_budget_7.png) ## Models Cost by Type This section displays the total cost of all your model predictions, along with the average daily cost, during the selected time period. It helps you track spending across each model type. ![](/img/community/control-center/costs_budget_7_2.png) ## Models Cost by ID This section displays the total cost of all your model predictions, along with the average daily cost, during the selected time period. It helps you track spending across each model ID. ![](/img/community/control-center/costs_budget_7_3.png) ## Training Cost This section displays the total cost of training your [deep training](https://docs.clarifai.com/portal-guide/model/deep-training/) and [transfer learning](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning) models, along with the average daily cost, during the selected time period. ![](/img/community/control-center/costs_budget_5.png) ## Stored Inputs Cost This section displays the total cost of storing inputs on our platform during the selected time period. ![](/img/community/control-center/costs_budget_9.png) ## Compute Cost by Instance and Date This section displays the total compute cost for instance types, as well as the average daily cost, during the selected time period. It helps you track spending across each instance type. ![](/img/community/control-center/costs_budget_10.png) ## Token Type Cost By Date This section displays the total cost of all your LLM tokens, along with the average daily cost, during the selected time period. It helps you track spending trends by token type across specific dates. ![](/img/community/control-center/costs_budget_11.png) --- ## Overview Tab # Overview Tab **Get a summary of your key metrics at a glance** The Overview tab allows you to quickly access the charts and data you choose from other function-specific tabs within the Control Center. To access it, select **Control Center** in the collapsible left sidebar and choose **Overview** from the dropdown list. You can look at the section to get a quick summary of the activities you undertook. ![](/img/community/control-center/control_center_3-1.png) In the Overview highlights section, you can get quick analysis of how you used our platform during the selected period. It tells you: - The total number of operations you’ve performed; that is, model predictions and search operations - The total compute time for [Compute Orchestration](https://docs.clarifai.com/compute/overview) (CO) tasks (in hours) - The average number of inputs you’ve added and stored in the platform - The total training hours of your models - The total number of models you’ve used - The total number of model predictions you’ve made - The total costs of all your Compute Orchestration tasks - The total costs of all your billable operations - The total costs of training your models - The total number of model types you’ve used ## Smart Insights Smart Insights at the top of the page allows you to uncover usage insights through a conversational interface. It enables you to interact with your data naturally — by simply asking questions in plain language — rather than navigating multiple reports or charts. ![](/img/community/control-center/control_center_4-4.png) In the search field, you can type a query such as _“How much am I spending each month”_ or any other question related to your usage, costs, or operations. Once you submit your question, the AI system processes your query and retrieves relevant insights based on your usage data. These insights are displayed directly in the interface, which records your past interactions for easy reference. ## Chart Features Charts in the Overview and across the Control Center share several common features that make data visualization and interaction more intuitive. Let’s take a closer look at these features. ### Configure Page You can customize the display of charts on the page to suit your preferences. To do so, click the **Configure page** button located at the top of the page. This opens the chart configuration panel, where you can resize, reorder, and adjust the layout of the charts. ![](/img/community/control-center/control_center_4-2.png) To align a chart to the left side of the page, click the left arrow (**←**) icon next to the chart name. To align it to the right, click the right arrow (**→**) icon. If you want the chart to span the full width of the page, click the double-sided arrow (**↔**) icon. The alignment arrows also serve to rearrange charts, allowing you to place them side by side or change their relative order. Charts can also be reordered easily using the drag handle (**⋮⋮**) next to each chart name. You can click and hold this handle to move charts up or down within the page. > **Note:** This flexibility is especially useful for comparison — when two charts are aligned next to each other, they automatically resize to half their original width, making it easier to view related insights together. For example, you might place the [**Total Number of Operations**](usage.md#total-number-of-operations) chart beside the [**Cost of Operations**](costs.md#cost-of-operations) chart to compare usage and expenditure trends directly. Once you’ve finished adjusting your layout, click the close (**X**) button in the upper-right corner of the configuration panel to exit. > **Alternatively:** In addition to the main configuration panel, each chart includes its own configuration options. By clicking the **configure icon** in the upper-right corner of a chart, you can use the drop-down to quickly align it to the left, right, or full width, or open the main configuration panel for more advanced layout adjustments. > You can also reorder a chart easily using the drag handle (**⋮⋮**) next to its name. > ![](/img/community/control-center/control_center_4-3.png) ### Date Range Selection You can use the date range control to filter and view data for specific time periods. The feature allows you to select a range as a criterion to view the output of the Overview section. To select a date range, click the button in the upper-right corner of the page. A small window will drop down, which allows you to select your preferred date range. By default, the display of data is filtered based on your current month-to-month billing cycle. You can customize the output by selecting a predefined date range — from today to the last 6 months. You can also use the date picker button at the bottom of the drop-down window to select a custom date range. ![](/img/community/control-center/control_center_3.png) After choosing your preferred date range, click the **Apply** button. ### Pin Charts Once you pin a chart from other tabs within the Control Center, it will automatically appear in the Overview. > **Tip:** You can also click the **Pin to overview page** button directly within the Overview to quickly add the suggested chart to the page. > ![](/img/community/control-center/control_center_4.png) In the [next section](./usage/#common-chart-features), we’ll explore charts in more detail and walk you through how to add them to the Overview tab. You can remove a chart from the Overview by simply clicking the delete button (**X**) in the upper-right corner. This action will only remove the chart from the Overview, without deleting it from the respective data section of the Control Center. ![](/img/community/control-center/control_center_5.png) ### Download as CSV Click the **Download as CSV** icon to export the chart data in CSV format. ![](/img/community/control-center/control_center_5-2.png) --- ## Teams & Logs Tab # Teams & Logs Tab **Gain insights into platform activities with enhanced visibility, security, and governance** The Teams & Logs tab allows you to monitor platform operations by capturing detailed audit logs, helping you track what activities were performed, who performed them, and their outcomes. It provides the ability to assess user actions and resource changes, identify potential security issues, maintain comprehensive logs to meet regulatory requirements, and monitor request origins to troubleshoot failed operations effectively. :::tip [Click here](audit-log.md) to learn how to perform audit tracking via the API. ::: :::info This audit logging feature is currently only available to users on our Enterprise [subscription plan](https://www.clarifai.com/pricing). ::: ## Supported Operations You can use the Teams & Logs tab to track the following critical resource operations (_we're planning to support more resources in the future_): - **[Organization and team membership activities](https://docs.clarifai.com/control/clarifai-organizations/)** — Includes creating, updating, or deleting organizations and teams, sending invitations, and managing team users and applications. - **[Compute Orchestration](https://docs.clarifai.com/compute/overview)** – Cover creating, modifying, deleting clusters, nodepools, deployments - **[Collaborator activities](https://docs.clarifai.com/create/applications/manage#collaborators)** — Includes adding collaborators, editing their scopes, and removing them. - **[Application activities](https://docs.clarifai.com/create/applications/create)** — Includes creating, updating, duplicating, and deleting applications. - **[Model activities](https://docs.clarifai.com/create/models/)** — Tracks actions such as creating, training, publishing, and deleting models. - **[Workflow activities](https://docs.clarifai.com/create/workflows/)** — Covers the creation, publishing, updating, and deletion of workflows. - **[Module activities](https://docs.clarifai.com/create/modules/)** — Tracks the creation, updating, and deletion of modules. ## Audit Log Details To access the tab, go to the [Control Center](README.mdx#how-to-access-the-control-center) and select the **Teams & Logs** option in the collapsible left sidebar. ![](/img/community/control-center/teams_logs_1.png) You can get details of the following activity logs: - **When** — Timestamp of the operation. - **Who** — Identity of the user performing the action. - **What** — Description of the operation. - **Where** — IP address where the request originated from. - **Status** — Success or failure results of the operation. ## Filter by Apps By default, the page displays activities across all your apps. You can customize the view by selecting specific app(s) using the drop-down menu in the upper-right corner. ![](/img/community/control-center/teams_logs_2.png) ## Filter by Dates You can filter and view data for specific time periods using the date range selector in the upper-right corner. If you click the tool, a calendar will drop down, which allows you to specify a date range to narrow down the activity logs. > **Note:** The current date is not selected by default. You can learn more about the date range tool [here](overview.md#date-range-selection). ![](/img/community/control-center/teams_logs_3.png) ## Collapsible Operation Details The **Operation** column includes collapsible fields that allow you to expand or collapse details for each logged event. When expanded, the field reveals additional information about the specific action taken and the IDs of the resources impacted. Conversely, when collapsed, only a summarized description of the operation is visible, helping to maintain a cleaner and more organized view of the logs. You can toggle this view by clicking the small arrow icon next to each operation entry. ![](/img/community/control-center/teams_logs_4.png) --- ## Usage & Operations Tab # Usage & Operations Tab **Get detailed insights into your consumption of the Clarifai platform** The Usage & Operations tab provides detailed insights into your consumption of the Clarifai platform. It offers metrics, charts, and reports that allow you to monitor usage patterns, track performance, and optimize resource allocation. To access it, go to the Control Center and select the **Usage & Operations** tab in the collapsible left sidebar. ![](/img/community/control-center/control_center_6.png) The overview section of the Usage & Operations page gives you a quick analysis of how you used the Clarifai platform during the selected period. It tells you: - The total compute time for your [Compute Orchestration](https://docs.clarifai.com/compute/overview) tasks (in hours) - The total number of model predictions you’ve made - The total number of search operations you’ve made - The average number of inputs you’ve added and stored in the platform - The total training hours of your [transfer learning](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning) models - The total training hours of your [deep training](https://docs.clarifai.com/portal-guide/model/deep-training/) models ## Common Chart Features The Usage & Operations page has several charts that help you visualize your utilization of our platform during a selected period. Let’s talk about some functionalities that are common across the charts. ![](/img/community/control-center/control_center_7_1.png) :::note Learn more about other common chart features, such as page configuration and date range selection, [here](overview.md#chart-features). ::: - **Drag charts** — You can easily drag a chart up and down within the page by using the drag handle (**⋮⋮**) in its upper-left corner. This allows you to customize the layout and manage the charts that appear on the page. - **Pin icon** — The pin icon allows you to "pin" or "save" charts for quick access. If you click it, the chart will be displayed in the [Overview](overview.md) section within the Control Center. This ensures that it remains easily accessible in the Control Center. If the pin icon is highlighted in the Usage & Operations page, it indicates that the respective chart has already been saved to the Overview section. - **Change chart display** — You can choose to display the type of chart you want by clicking its icon in the upper-right corner — a bar chart, a line graph, a donut chart, or any other chart available. - **Tooltip** — If you hover over a chart, a tooltip is displayed, which shows the exact number of different actions performed for that particular date. In the tooltip, the type of action that is currently hovered over is highlighted, allowing you to easily identify which item is in focus. - **Color-coding** — Each type of task in the charts within the page is represented by its own distinct color. This color-coding provides a clear visual distinction between the types of activities, making it easier for users to track and compare their trends across the displayed dates. - **Timestamp** — A **Last updated** label appears at the bottom-left corner of each chart, indicating when the displayed metrics were last refreshed. - **Cross-navigation links** — The charts in the Usage & Operations tab are linked to related charts in the [Costs & Budget tab](costs.md), and vice versa. These cross-navigation links allow you to seamlessly explore related insights. For example, the Total Number of Operations chart includes a link in the lower-right corner that directs you to the corresponding Cost of Operations chart. ### View Report Details If you click the **View Report Details** button in the lower-right corner of a chart, you’ll be redirected to a detailed report page. This page provides a comprehensive breakdown of each type of action. The chart in the detailed report page displays all the activities performed, plotted against specific dates. If you want to focus on a particular activity, simply click the block at the top showing its corresponding number. This action will filter the chart to display only the selected activity. The table on the page includes the date each activity was performed. Additionally, you can sort the items in the table for easier navigation. ![](/img/community/control-center/control_center_9.png) After viewing the detailed report, you can go back to the Usage & Operations page by clicking the **Back** button in the upper-left corner. ## Total Number of Operations This section shows the total number of operations you've performed, along with the daily average, during the selected time period. It includes both model predictions and search operations. A stacked bar chart (displayed by default) or a line graph shows the number of each type of operation plotted against specific dates. Here is a bar chart: ![](/img/community/control-center/control_center_7-1.png) Here is a line graph: ![](/img/community/control-center/control_center_7.png) ## Model Training Hours This section displays the total hours spent training your models, along with the daily average, for the selected time period. It breaks the hours down by [transfer learning](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning) models and [deep training](https://docs.clarifai.com/portal-guide/model/deep-training/) models. A stacked bar chart (displayed by default) or a line graph shows the number of training hours for each type of model plotted against specific dates. Here is a bar chart: ![](/img/community/control-center/control_center_10.png) Here is a line graph: ![](/img/community/control-center/control_center_10_1.png) ## Total Inputs Stored This section displays the number of inputs you’ve added and stored in the Clarifai platform for the selected time period. The average number of stored inputs across the time period is displayed in the top section. A bar chart (displayed by default) or a line graph shows the number of inputs plotted against specific dates. Here is a bar chart: ![](/img/community/control-center/control_center_13.png) Here is a line graph: ![](/img/community/control-center/control_center_14.png) ## Compute Time by Instance and Date This section shows both the total and average compute time consumed by each [instance type](https://docs.clarifai.com/compute/cloud-instances) over the selected time period. A stacked bar chart (displayed by default) or a line graph shows the number of compute hours per instance type across specific dates. Here is a bar chart: ![](/img/community/control-center/control_center_27.png) Here is a line chart: ![](/img/community/control-center/control_center_28.png) ## Models Operations by Type This section displays the total number of model operations, and the daily average, for the selected time period. It shows the operations by [model type](https://docs.clarifai.com/create/models/#model-types). A bar chart (displayed by default) shows the type of models plotted against the number of operations. In a donut chart, the number of models operations is visualized and grouped by type. Here is a bar chart: ![](/img/community/control-center/control_center_16.png) Here is a donut chart: ![](/img/community/control-center/control_center_17.png) ## Models Operations by ID This section displays the total number of model operations, and the daily average, for the selected time period. It shows the operations by unique model ID. A bar chart (displayed by default) shows the ID of models plotted against the number of operations. In a donut chart, the number of models operations is visualized and grouped by type. Here is a bar chart: ![](/img/community/control-center/control_center_19.png) Here is a donut chart: ![](/img/community/control-center/control_center_20.png) ## Models Operations by Type and Date This section displays the total number of model operations, and the daily average, for the selected time period. It shows the operations by model type and date. A stacked bar chart (displayed by default) or a line graph shows the number of operations by model type plotted against specific dates. Here is a bar chart: ![](/img/community/control-center/control_center_22.png) Here is a line chart: ![](/img/community/control-center/control_center_23.png) ## Models Operations by ID and Date This section displays the total number of model operations, and the daily average, for the selected time period. It shows the operations by model ID and date. A stacked bar chart (displayed by default) or a line graph shows the number of operations by model ID plotted against specific dates. Here is a bar chart: ![](/img/community/control-center/control_center_25.png) Here is a line chart: ![](/img/community/control-center/control_center_26.png) ## Token Type By Date This section displays the total number of tokens used, and the daily average, for the selected time period, broken down by token type. A stacked bar chart (displayed by default) or a line graph shows the number of tokens plotted against specific dates. Each bar represents the total tokens used per day, segmented by token type, such as `prompt_token` and `completion_token`. Here is a bar chart: ![](/img/community/control-center/control_center_29.png) Here is a line chart: ![](/img/community/control-center/control_center_30.png) ## Tokens by Model Type and Date This section shows the total number of tokens used, and the daily average, for the selected time period, broken down by model type. A stacked bar chart (displayed by default) or a line graph shows the number of tokens plotted against specific dates. Each bar represents the total tokens used per day, segmented by model type, such as `text-to-text` or `multimodal-to-text`. Here is a bar chart: ![](/img/community/control-center/control_center_31.png) Here is a line chart: ![](/img/community/control-center/control_center_32.png) ## Tokens by Model Type This section displays the total number of tokens used, and the daily average, for the selected time period. It shows the tokens by model type. A horizontal bar chart (displayed by default) compares token usage across different model types. In a donut chart, the number of tokens consumed is visualized and grouped by model type. Here is a bar chart: ![](/img/community/control-center/control_center_33.png) Here is a donut chart: ![](/img/community/control-center/control_center_34.png) ## Tokens by Model ID and Date This section displays the total number of tokens used, and the daily average, for the selected time period. It shows the tokens by model ID. A stacked bar chart (displayed by default) or a line graph shows the number of tokens consumed across specific dates. Each bar represents the total tokens used per day, segmented by model ID. Here is a bar chart: ![](/img/community/control-center/control_center_35.png) Here is a line chart: ![](/img/community/control-center/control_center_36.png) ## Tokens by Model ID This section displays the total number of tokens used, and the daily average, for the selected time period. It shows the tokens by model ID. A horizontal bar chart (displayed by default) compares token usage across different model IDs. In a donut chart, the number of tokens consumed is visualized and grouped by model ID. Here is a bar chart: ![](/img/community/control-center/control_center_37.png) Here is a donut chart: ![](/img/community/control-center/control_center_38.png) --- ## Agent System Operators # Agent System Operators **Learn about our agent system operators** Agent system operators are fixed-function operators that act as "non-trainable models". They help you connect, direct, and network your models in a workflow. These operators that can be “chained” together with models to automate tasks. Below, we outline the different agent system operators we offer. ![model types](/img/others/agent-system-operators.png) import DocCardList from '@theme/DocCardList'; --- ## Aggregate # Aggregate **Learn about our aggregate operators** Aggregate operators are designed to process and synthesize data across multiple regions of an image or multiple frames in a video sequence. These operators are crucial for applications requiring consolidated information from dispersed data points, such as object counting or text aggregation within images. ### **Features** * **Enhanced Data Synthesis**: Aggregate operators combine data from multiple inputs to produce a coherent output, simplifying further analysis. * **Customizable Workflows**: Easily integrated into custom workflows, these operators allow flexible configurations to suit varied use cases. ## Text Aggregation Operator **Input**: `regions[…].data.text` **Output**: `text` The Text Aggregation Operator consolidates detected text elements within an image, organizing them into a coherent body of text. This operator is especially beneficial in workflows that process documents or extract readable information from various visual media formats. This model arranges text detections spatially from left to right and then from top to bottom, mimicking the natural reading order. This functionality ensures the output text follows a logical sequence, increasing its usability for subsequent processing or analysis. The operator adjusts to the positional dynamics of text within the image, making it versatile across different types of textual content. ### **Example Scenario:** **Text** **Blocks**: Multiple addresses on a package **Sorting** **Order**: Left to right, top to bottom **Operation**: The operator organizes the detected addresses into a sequence replicating the text's layout as it appears on the package. ### **Workflow Integration and Setup** Let's demonstrate how you can use the Text Aggregator alongside an OCR operator. This is because the OCR operator identifies regions with text and provides the text read with a score. The text aggregator then uses that output as input. 1. Go to the workflow builder page. Search for the OCR operator in the left-hand sidebar and drag it onto the empty workspace. Then, select a model from the pop-up on the right-hand sidebar. 2. Search for the Text Aggregation operator in the left-hand sidebar, drag it onto the workspace, and set up its output configuration based on the requirements. 1. Connect the OCR with the Text Aggregator and save the workflow. ![alt text]() To see it in action, upload the inputs from your local device or use the inputs in the app. When you upload inputs, the workflow will give the output based on the configurations done. ![alt text]() You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Text-Aggregation?version=d5a9822e504e4bd68c57c51177d031f1). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Object Counter **Input**: `regions[…].data.concepts` **Output**: `metadata` The Object Counter operator is tailored to quantify objects within images or video frames based on specific concepts. This operator is instrumental in environments where accurate object tracking and counting are required across successive frames, such as in surveillance, traffic monitoring, and automated inventory systems. This model excels in recognizing and counting occurrences of predefined concepts within the designated regions of interest, frame by frame. By leveraging advanced detection algorithms, the Object Counter ensures precise counts, enhancing data analysis and decision-making processes in real-time applications. ### **Example Scenario:** **Object of Interest**: "Vehicle" **Operation**: The operator counts the number of vehicles across video frames, providing real-time data essential for traffic flow analysis. ### **Object Counter Setup** Let's demonstrate how you can use the Object Counter alongside a [Visual Detector](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector). The reason behind this is that the Visual Detector identifies regions and provides the confidence scores that the Object Counter uses to count the regions effectively. Without the Visual Detector, the Object Counter would lack the necessary data to perform its counting function. 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection). 2. Search for the Object Counter option in the left-hand sidebar and drag it onto the workspace. 1. Connect the Visual Detector with the Object Counter operator and save your workflow. ![alt text]() 1. To see it in action, upload the inputs from your local device or use the inputs in the app. When you upload inputs, the workflow will give the output based on the configurations done. For this example, [this](https://samples.clarifai.com/05/26/e0/d96cab4e0cb85c430f2ef763b3.jpg) image is used. ![alt text]() To ensure accurate counting, the Visual Detector identifies the regions of interest and assigns confidence scores to them, which the Object Counter then uses to perform the counting function. However, due to the complex nature of this operation, the workflow's output is not supported directly in the web viewer. Users must access the detailed results in a JSON format to review the count and other metadata effectively. This approach guarantees precise data interpretation and enhances decision-making processes. \ \ JSON for the above output is given below, and at the end, the "**object_count": 8** is given. ```json { "json": { "id": "6ef88a0a76c94da6a02c8ea58feb13d5", "status": { "code": 10000, "description": "Ok" }, "created_at": "2024-09-04T00:28:29.404495670Z", "model": { "id": "workflow-model-61fd-d06227acd7d1", "name": "workflow-model-61fd-d06227acd7d1", "created_at": "2024-09-04T00:28:26.852951Z", "modified_at": "2024-09-04T00:28:26.852951Z", "app_id": "Text-Generator", "model_version": { "id": "fd2bdda515164775bbbd1b24a2d620a6", "created_at": "2024-09-04T00:28:27.141447Z", "status": { "code": 21100, "description": "Model is trained and ready for deployment" }, "visibility": { "gettable": 10 }, "app_id": "Text-Generator", "user_id": "mohit01", "metadata": {}, "inference_compute_info": {} }, "user_id": "mohit01", "model_type_id": "object-counter", "visibility": { "gettable": 10 }, "toolkits": [], "use_cases": [], "languages": [], "languages_full": [], "check_consents": [], "workflow_recommended": false }, "data": { "metadata": { "object_count": 8 } } } } ``` You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Object-Counter?version=97dbdbae2b78452cafbefc6ddc63d69b). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: --- ## Algorithmic Predict # Algorithmic Predict **Learn about our algorithmic predict operators** Algorithmic Predict refers to a category of operators that leverage predefined algorithms to make predictions or generate outputs based on input data. You can use the prediction results to understand, classify, or organize your data. You can also use them to drive behaviors in other nodes in your workflow. These operators take specific input types and then return predictions about things like concepts, regions, characters, words, or the abstract visual characteristics of your inputs. :::tip Since the algorithmic predict operators can be "chained" together with models to automate tasks in a workflow, you can learn how to create workflows [here](https://docs.clarifai.com/portal-guide/workflows/input-nodes#create-your-workflow). ::: ## Regex-Based Classifier **Input**: Text **Output**: Concepts This operator allows you to classify text using regular expressions. When the specified regex pattern matches the text, the text is assigned to one of the predefined concepts. Let's demonstrate how you can use the Regex-Based Classifier, alongside [a Prompter template](https://docs.clarifai.com/portal-guide/agent-system-operators/prompter), to efficiently classify text. **1**. Go to the workflow builder page. Then, search for the **prompter** template node in the left sidebar and drag and drop it onto the empty workspace. Use the pop-up that appears on the right sidebar to set up the template text. For this example, let's use this template text: ```text [INST]>Classify the following description into one of the following classes: ["cat," "dog," "cheetah," "lion"]. Respond only with one of the provided classes.>[/INST]\n{data.text.raw} ``` :::note Since we'll use the [llama2-13b-chat](https://clarifai.com/meta/Llama-2/models/llama2-13b-chat) model to help with the classification, we format the prompt text using the special tokens it requires for the specific structure of its prompts. We also include the `{data.text.raw}` placeholder to meet the requirements of the **Prompter** template format. ::: **2**. Search for the **text-to-text** node in the left sidebar and drag and drop it onto the workspace. Then, search for the **llama2-13b-chat** model on the right sidebar and connect it to the prompter model. **3**. Search for the **regex-based classifier** node in the left sidebar and drag and drop it onto the workspace. On the right sidebar, click the **SELECT CONCEPTS** button and use the pop-up that appears to select the relevant [concepts](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete) already existing in your application. For this example, we select the following concepts: `cat, dog, cheetah, lion`. After selecting the concepts, click the **OK** button. ![](/img/others/regex_1_1.png) In the **regex** field, provide the regex pattern that will be used to classify the text. If the pattern matches, the text will be classified as the selected concept. For this example, we provide `\bcat\b`, which would match the word "cat" in instances where it appears as a whole word, surrounded by word boundaries. **4**. Connect the **text-to-text** model with the **regex-based classifier**. Lastly, click the **Save Workflow** button to save your workflow. ![](/img/others/regex_1.png) To observe it in action, navigate to the workflow's individual page and click the **+** button to input your text. For this example, let's provide the following input: ```text A small, four-legged mammal with soft fur, typically characterized by its whiskers, sharp retractable claws, and acute senses. Known for its independent and curious nature, it often displays a variety of behaviors such as grooming itself, purring, and occasionally hunting. What is this animal? ``` This is the prompt text we get for the model: ```text [INST]>Classify the following description into one of the following classes: [''cat'', ''dog'', ''cheetah'', ''lion'']. Respond only with one of the provided classes.>[/INST]\nA small, four-legged mammal with soft fur, typically characterized by its whiskers, sharp retractable claws, and acute senses. Known for its independent and curious nature, it often displays a variety of behaviors such as grooming itself, purring, and occasionally hunting. What is this animal?[/INST] ``` The model will process the input and classify the description into one of the provided classes. ![](/img/others/regex_2.png) Then, the Regex-Based Classifier will categorize the response into one of the provided concepts, which you can feed into other downstream tasks, such as an [Annotation Writer](https://docs.clarifai.com/portal-guide/agent-system-operators/push#annotation-writer) to create annotations for inputs. You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Regex-Based-Classifier?version=ee5e074735ab46e6b3a50c3fb6627cd0). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Language Identification Operator **Input**: Text **Output**: Concepts The Language Identification Operator is designed to automatically detect the language of a given text. It takes in text input, which can be in any form — paragraphs, sentences, or even shorter phrases. It then analyzes the text to automatically determine which language it is written in. The output of the operator is typically a language code (e.g., `en` for English, `fr` for French, `es` for Spanish) that corresponds to the detected language. The operator leverages either of the following libraries: - `langdetect` — It's an open-source tool for language detection. This library is known for its ability to recognize a broad range of languages based on text samples. It uses algorithms that compare the text against language profiles created from a large corpus of multilingual data. It assigns a probability score to each language, which helps to identify the most likely language for the given text. - `fastText` — Developed by Facebook's AI Research (FAIR) lab, this open-source library provides efficient language identification capabilities. It can recognize a large number of languages and is particularly fast, making it suitable for processing large volumes of text. It is based on word embeddings and character-level n-grams, which allows it to handle short or informal texts well. To use the operator, go to the workflow builder page and search for the `language-id-operator` node in the left sidebar. Drag and drop it onto the empty workspace and connect it to the `IN` element. You can use the right sidebar to set up the following output configurations: - **library** — Select the library you want to use for the language identification — either `langdetect` or `fastText`. - **topk** — Set the maximum number of predicted languages. - **threshold** — Set a confidence score that determines the likelihood that the detected language is correct. Languages with a confidence level above the set threshold will be returned. - **lowercase** — If set to true, the provided text will be converted to lowercase letters. Lastly, click the **Save Workflow** button to save your workflow. ![](/img/agent-system-operators/language-identification-operator.png) To observe it in action, navigate to the operator's individual page and click the **+** button to input your text. For this example, let's provide [this text](https://samples.clarifai.com/model-gallery/Text/text-romance-french.txt). The operator will process the text and identify its language. ![](/img/agent-system-operators/language-identification-operator-1.png) ## Barcode Operator **Input**: Image **Output**: regions[…].data.text The Barcode Operator is used to detect and recognize a wide range of barcode types within images. It processes image inputs and detects barcodes contained in them. It works by recognizing and decoding the information encoded within each barcode. This includes reading the patterns of bars and spaces or, in the case of QR codes, interpreting the matrix of squares. For each detected barcode, the operator assigns a specific region within the image that contains the barcode text. This helps in isolating and extracting the exact area of the image where the barcode is located. It then outputs the recognized text or data from the barcode alongside the region information, which allows for easy extraction and use of the barcode data. It supports the following types of barcodes: - **EAN/UPC** — Commonly used in retail for product identification. - **Code 128** — A versatile barcode often used in logistics and packaging. - **Code 39** — Frequently used in inventory management and non-retail settings. - **Interleaved 2 of 5** — Typically used for encoding numeric information in industries like warehousing. - **QR Code** — A matrix barcode that can encode a variety of data types, including URLs, text, and more. To use the operator, go to the workflow builder page and search for the `barcode-operator` node in the left sidebar. Drag and drop it onto the empty workspace and connect it to the `IN` element. Lastly, click the **Save Workflow** button to save your workflow. ![](/img/agent-system-operators/barcode-operator.png) To observe it in action, navigate to the operator's individual page and click the **+** button to input your image. For this example, let's provide [this image](https://samples.clarifai.com/model-gallery/images/QRcode-001.jpeg). The operator will process the image and detect the QR code. ![](/img/agent-system-operators/barcode-operator-1.png) You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Barcode-Operator?version=6b78ce81239a4381a08ac1502775a90d). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: --- ## Edit # Edit **Learn about our edit operators** Edit operators are specific type of agent system operators that are specialized for data transformation tasks. These operators are responsible for modifying and/or augmenting your data as it passes through the workflow. They can be used to crop out regions of an image, align an image based on the pose of a face, or even map predictions from one model to another. ### Common Characteristics - **Purpose**: Designed to manipulate and prepare images for enhanced analysis or visual presentation. - **Configurability**: Users can configure each operator with specific parameters to adjust the image processing according to the needs of their application, making these tools versatile across different scenarios. - **Enhanced Workflow Efficiency**: By modifying images as per specified requirements early in the process, these operators streamline the workflow, reducing the workload and computational requirements of downstream processes. :::tip Since the transform operators can be "chained" together with models to automate tasks in a workflow, you can learn how to create workflows [here](https://docs.clarifai.com/portal-guide/workflows/input-nodes#create-your-workflow). ::: ## Image Cropper **Input:** `image`, `regions` **Output:** `regions[...].data.image` The Image Cropper model crops input images based on regions specified either in the workflow or detected by a preceding model, making it particularly useful in workflows where precise image manipulation is required. This model works seamlessly across different types of visual inputs. It is designed to ensure that only the specified regions of an image are retained for further processing, enhancing both the accuracy and relevance of the results. The cropper can look back along the workflow to find the input image if the preceding model does not output an image itself, allowing for a streamlined process in image-based workflows. **Example Scenario:** - **Region of Interest:** "Face" - **Margin Setting:** 10 pixels - **Operation:** If a face is detected in an image, the cropper will extract this region with a 10-pixel margin around it. If no face is detected, the image may either be passed as is or not included in the output, depending on configuration. ### Workflow Integration and Setup 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. This model will detect the regions to be cropped. 2. Search for the Image Cropper option in the left-hand sidebar and drag it onto the workspace. Use the pop-up that appears on the right-hand sidebar to configure the margin and other settings. 3. Set up its output configuration on the below-given parameter: - **Margin:** Adjust the slider to set the desired margin around the cropped areas. 1. Connect the region detection model (such as a face detector or general object detector) with the Image Cropper operator and configure the workflow. ![Image Cropper Setup]() To see it in action, upload the inputs from your local device or use the inputs in the app. As soon as you upload inputs where regions are detected, the workflow will execute and output the cropped images based on the configurations done. ![alt text]() You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Image-Cropper?version=1f3c662d2fd540d190a354711fa1ad48). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Image Tiler **Input:** `image` **Output:** `regions[...].data.image`, `regions|[...].region_info.bounding_box` The Image Tiler model divides input images into multiple equal-sized tiles. This operator is essential in workflows where large images need to be broken down into smaller, more manageable segments for detailed analysis or parallel processing. This model effectively handles visual inputs by splitting them into uniformly sized tiles. It ensures that detailed analyses can be performed on each segment without the computational burden of processing large images in their entirety. The tiler is configurable to adapt to various sizes and formats of images, making it versatile for different application needs. **Example Scenario:** - **Tile Size:** 512x512 pixels - **Max Object Size:** 120 pixels - **Operation:** An aerial photograph will be tiled into 512x512 pixel segments, ensuring that any object smaller than 120 pixels remains within a single tile to facilitate accurate analysis. ### Workflow Integration and Setup Workflow Integration and Setup 1. Go to the workflow builder page. From the tool palette on the left-hand sidebar, select the Image Tiler option. 2. Drag the Image Tiler onto the workspace and use the configuration options in the right-hand sidebar to set the tile size and max object size. 3. Set up its output configuration on the below-given parameters: - **Tile Size:** Specify the size for each tile. - **Max Object Size:** Set the maximum size for objects that should not be split across tiles. 1. Save your workflow. No need to connect it to other models unless specific regions within tiles need further analysis or processing. ![alt text]() To see it in action, upload the inputs from your local device or use the inputs in the app. As soon as you upload inputs, the workflow will execute, and you will see the images split into the specified tile sizes. This tiling is crucial for handling large-scale images efficiently. ![alt text]() --- ## Embed # Embed **Learn about our embed operators** The embed operators include: - Visual embedder - Text embedder - Audio embedder - Multimodal embedder - Visual detector embedder --- ## Filter # Filter **Learn about our filter operators** Filtering helps you to remove unwanted data from your workflow. This data might take the form of inputs (like images, video, and text) or it might be an output from another model, like a predicted concept. One very common use of filters in workflows is to eliminate predictions that fall below a certain confidence threshold. That's exactly what filter operators do. Filter operators process and analyze data by selectively passing information based on defined criteria. This guide covers three primary filter operators used in various data processing workflows: Region Thresholder, Concept Thresholder, and Random Sample. Each operator is designed to enhance the efficiency and accuracy of data handling, ensuring that only relevant data is analyzed in subsequent stages. ### ​​**Common Features of Filter Operators** All three operators share several key features and functionalities: - **Input Filtering**: Each operator takes input data and applies a set of predefined rules or conditions to filter the data, ensuring that only relevant data points proceed to the next stage of processing. - **Enhanced Workflow Efficiency**: By eliminating irrelevant or less significant data early in the process, these operators reduce the workload and computational requirements of downstream processes. - **Configurability**: Users can configure each operator with specific parameters and thresholds to meet the unique needs of their application, making these tools versatile across different scenarios and datasets. ## Region Thresholder **Input:** `regions[…].data.concepts` **Output:** `regions[…].data.concepts` The Region Thresholder model filters image regions based on the confidence scores assigned to detected concepts and works with the visual detection node. It ensures that only regions meeting specific confidence criteria are passed on for further processing, enhancing the workflow results' accuracy and relevance. It uses a thresholding mechanism to filter out regions according to threshold criteria set by the user, ensuring that only regions with required scores are considered for further analysis. It’s threshold mechanism is discussed below: - **Threshold Type:** The threshold can be set using various operators such as >, >=, =, \, then only regions with a concept's confidence score greater than the threshold will be passed on. - **Overall Region Filtering:** If all concepts within a region are filtered out (i.e., none meet the threshold criteria), the entire region is removed from the output. #### Example Scenario - **Concept:** "Laptop" - **Threshold Type:** > - **Threshold Value:** 0.8 - **Operation:** If the confidence score of "Laptop" in a detected region is greater than 0.8, that region is passed through. Otherwise, it is filtered out. Let's demonstrate how you can use the Region Thresholder, alongside a [detection model](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector). The reason behind this is that the Visual Detector identifies regions and provides the confidence scores that the Thresholder uses to filter those regions effectively. Without the Visual Detector, the Region Thresholder would lack the necessary data to perform its filtering function. 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. 2. Search for the Region-Thresholder option in the left-hand sidebar and drag it onto the workspace. Now set up its output configuration on below given parameters: - **concepts -** Select the concepts and their confidence threshold value. - **concept_threshold_type -** Select the concept threshold type from the dropdown. 3. Connect the visual-detector model with the Region Thresholder operator and save your workflow. ![Region Thresholder Setup]() To see it in action, upload the inputs from your local device or use the inputs in the app. As soon as you upload inputs, the workflow will give the output based on the configurations done. ![Region Thresholder Output]() You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Region-Thresholder?version=c6e19afea2a740738cbaad0bb53913f5). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Concept Thresholder **Input:** `concepts` **Output:** `concepts` The Concept Thresholder model filters entire datasets based on the confidence scores assigned to specific concepts. This model works effectively across different types of data inputs, not limited to visual data. It ensures that only data points meeting specific confidence criteria are passed on for further processing, enhancing the accuracy and relevance of the workflow results. It uses a thresholding mechanism to filter out data according to criteria set by the user, ensuring that only data with required scores are considered for further analysis. Its threshold mechanism is discussed below: - **Threshold Type:** The threshold can be set using various operators such as >, >=, =, \, then only data points with a concept's confidence score greater than the threshold will be passed on. - **Overall Data Filtering:** If all concepts within a data point are filtered out (i.e., none meet the threshold criteria), the entire data point is removed from the output. #### Example Scenario - **Concept:** "Bridge" - **Threshold Type:** > - **Threshold Value:** 0.75 - **Operation:** If the confidence score of "Bridge" in a data point is greater than 0.75, that data point is passed through. Otherwise, it is filtered out. Let's demonstrate how you can use the Concept Thresholder to efficiently manage large datasets. This model is pivotal for tasks where specific concept relevancy is critical, such as filtering customer feedback or social media posts. 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. 2. Search for the concept-thresholder option in the left-hand sidebar and drag it onto the empty workspace. Now set up its output configuration on the below-given parameters: - **concepts** - Select the concepts and their confidence threshold value. - **concept_threshold_type** - Select the concept threshold type from the dropdown. 3. Connect the visual-detector model with the Concept Thresholder operator and save your workflow. ![Concept Thresholder Setup]() To see it in action, upload the inputs from your local device or use the inputs in the app. As soon as you upload inputs, the workflow will give the output based on the configurations done. ![Concept Thresholder Output]() You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Concept-Thresholder?version=ff04af9f5fff471aa7a4691a23474607). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Random Sampler **Input:** `any` **Output:** `any` The Random Sample model randomly selects a subset of data from the input based on a specified sample size or percentage, making it an essential tool for statistical analysis and model training. This operator ensures that the sample is representative of the whole dataset, thus maintaining the integrity and variability of the data. It employs a sampling mechanism to randomly pick data points, ensuring that every item in the dataset has an equal chance of being included in the sample. This approach is crucial for reducing bias in the analysis results. Its sampling mechanism is discussed below: - **Sampling Type:** The sampling can be configured to select either a fixed number of items or a percentage of the total dataset. - **Random Selection:** Each item in the dataset is given an equal probability of being selected, ensuring a fair and unbiased sample. #### Example Scenario - **Sampling Type:** Percentage - **Sampling Value:** 0.1 - **Operation:** If the dataset consists of 1,000 items, the Random Sample operator will randomly select 100 items to be passed on for further processing. Let's demonstrate how you can use the Random Sample operator to handle large datasets efficiently or prepare data for machine learning model training. 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. 2. Search for the random-sample option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to specify the sampling criteria ie; **keep fraction**. This is the fraction of input to randomly keep. This is implemented as simply if keep_fraction > rand() then output this input from the model . This is applied independently for each input sent in a batch to the model. 3. Save your workflow without the need to connect it directly to a data processing model, as it functions independently to reduce the dataset size. ![Random Sampler Setup]() To see it in action, upload the inputs from your local device or use the inputs in the app. As soon as you upload inputs, the workflow will give the output based on the configurations done, displaying a randomly selected subset of the data. ![Random Sampler Output]() You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Random-Sampler?version=cc26641fca6b4a5d8a8075e6883deb54). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: --- ## Map # Map **Learn about our map operators** ## Concept Synonym Mapper **Output**: Concepts Allows you to map the input concepts to output concepts by following synonym concept relations in the knowledge graph of your app. --- ## Prompter # Prompter **Learn about our prompter operator** **Output**: Text A prompt template serves as a pre-configured piece of text used to instruct a large language model (LLM). It acts as a structured query or input that guides the model in generating the desired response. :::note Prompt Templates Many tasks you’ll need LLMs for are repeatable. With [prompt templates](https://docs.clarifai.com/portal-guide/ppredict/generative-ai#prompt-template), you can simplify repeat LLMs use cases. By making these part of your Clarifai application, you won’t need to re-enter prompts the same way every time. ::: LLMs are trained on massive datasets of text and code, and they can be used to perform a variety of tasks, including text generation, question answering, translation, summarization, text completion, and more. LLMs are designed to understand and generate text based on the instructions or prompts they receive. Prompting an LLM allows you to leverage the model’s pre-trained language capabilities and control its outputs so that it can deliver what is relevant to your needs. The quality and relevance of the generated texts depend on the specific wording and context of the prompt. Prompts can be structured in various ways, and their effectiveness often depends on how well they convey the desired task or instruction to the model. :::note Prompt Engineering The process of designing and refining prompts to effectively interact with and elicit desired responses is called prompt engineering. It involves crafting specific questions, statements, or input formats that guide the model to generate useful, relevant, and accurate outputs. ::: There are several prompting techniques you can use. ## Zero-Shot Prompting Zero-shot prompting leverages the model's inherent language understanding to perform tasks without any specific labeled training data. It simply requires providing a clear prompt for the LLM to generate relevant responses. The LLM is then able to use its pre-trained knowledge to complete the task. Let’s demonstrate how you can create a zero-shot prompter on the [workflow builder](https://docs.clarifai.com/portal-guide/workflows/workflow-builder/) for a text classification task. :::tip Your prompt template should include at least one instance of the placeholder `{data.text.raw}`. When you input your text data at inference time, all occurrences of `{data.text.raw}` within the template will be replaced with the provided text. ::: **1.** Search for the **Prompter** template option in the left-hand sidebar of the workflow builder and drag it onto the empty workspace. **2.** Use the pop-up that appears on the right-hand sidebar to define the template text. For this example, let’s use the following text: ``` Please label the sentiment towards the movie of the given movie review. The sentiment label should be "positive" or "negative". ### Text: {data.text.raw} ### Sentiment: ``` ![zero-shot prompting](/img/agent-system-operators/prompter_1.png) Note that: - The template text is a single-line statement. - We included the `{data.text.raw}` placeholder to meet the requirements of the **Prompter** template. - We placed the instructions at the beginning of the template text and used the ["###" delimiter](https://docs.clarifai.com/portal-guide/model/model-types/text-to-text/#training-data) to separate the instruction and context. The delimiter is important when giving instructions to LLMs because it signifies the beginning and end of different sections within the text. This ensures clarity and facilitates easy parsing and processing during the prediction phase. **3.** You can then connect the prompter model to a text-to-text model like [GPT-4](https://clarifai.com/openai/chat-completion/models/GPT-4). **4.** Save your workflow. To observe it in action, navigate to the workflow's individual page and click the **+** button to input your text. For example, you could input the following as your input text: ``` I’ll bet the video game is a lot more fun than the film ``` Click the **Submit** button. Once the model has completed processing your input, you'll see the results, starting with the earlier template text, now adapted to your input. In this case, the prompt text becomes: ``` Please label the sentiment towards the movie of the given movie review. The sentiment label should be "positive" or "negative". ### Text: I’ll bet the video game is a lot more fun than the film. ### Sentiment: ``` And the output becomes: ``` Negative ``` ![zero-shot prompting output](/img/agent-system-operators/prompter_2.png) Note that in the above zero-shot prompt, we did not give the model any examples of text alongside their classifications. The model already knows what "sentiment" means without needing any extra information — that's how its zero-shot abilities work. You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Zero-Shot-Prompter?version=aeb30786c61f4b2cbaea7415cdae59e4). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Few-Shot Prompting In few-shot prompting, the large language model is given a limited number of examples or "shots" to adapt to a particular task. With just a few examples, it can perform more specialized tasks that require specific context or knowledge. This is in contrast to zero-shot prompting, which does not require any examples. Few-shot prompting is often used for more complex tasks where zero-shot prompting is not sufficient. For example, few-shot prompting can be used to train an LLM to classify new types of data, translate languages that it has not been trained on, or generate different creative text formats. Let’s demonstrate how you can create a few-shot prompter on the [workflow builder](https://docs.clarifai.com/portal-guide/workflows/workflow-builder/) for a text classification task. **1.** Search for the **Prompter** template option in the left-hand sidebar of the workflow builder and drag it onto the empty workspace. **2.** Use the pop-up that appears on the right-hand sidebar to define the template text. For this example, let’s use the following text: ``` Please label the sentiment towards the movie of the given movie review. The sentiment label should be "positive" or "negative". ### Text: (lawrence bounces) all over the stage, dancing, running, sweating, mopping his face and generally displaying the wacky talent that brought him fame in the first place. ### Sentiment: positive. ### Text: Despite all evidence to the contrary, this clunker has somehow managed to pose as an actual feature movie, the kind that charges full admission and gets hyped on tv and purports to amuse small children and ostensible adults. ### Sentiment: negative. ### Text: For the first time in years, de niro digs deep emotionally, perhaps because he's been stirred by the powerful work of his co-stars. ### Sentiment: positive. ### Text: {data.text.raw} ### Sentiment: ``` ![few-shot prompting](/img/agent-system-operators/prompter_4.png) Note that just like in the zero-shot prompting example above, we included the `{data.text.raw}` placeholder and the "###" delimiter in the template text. **3.** You can then connect the prompter model to a text-to-text model like [GPT-4](https://clarifai.com/openai/chat-completion/models/GPT-4). **4.** Save your workflow. To observe it in action, navigate to the workflow's individual page and click the **+** button to input your text. For example, you could input the following as your input text: ``` I'll bet the video game is a lot more fun than the film ``` Click the **Submit** button. Once the model has completed processing your input, you'll see the results, starting with the earlier template text, now adapted to your input. In this case, the prompter text becomes: ``` Please label the sentiment towards the movie of the given movie review. The sentiment label should be "positive" or "negative". ### Text: (lawrence bounces) all over the stage, dancing, running, sweating, mopping his face and generally displaying the wacky talent that brought him fame in the first place. ### Sentiment: positive. ### Text: Despite all evidence to the contrary, this clunker has somehow managed to pose as an actual feature movie, the kind that charges full admission and gets hyped on tv and purports to amuse small children and ostensible adults. ### Sentiment: negative. ### Text: For the first time in years, de niro digs deep emotionally, perhaps because he''s been stirred by the powerful work of his co-stars. ### Sentiment: positive. ### Text: I'll bet the video game is a lot more fun than the film ### Sentiment: ``` And the output becomes: ``` negative ``` ![few-shot prompting output](/img/agent-system-operators/prompter_3.png) Note that in the above few-shot prompt, we provided a few examples to guide the model to perform better. The demonstrations enabled in-context learning and acted as conditioning for successive examples in which we wanted the model to produce a response we desired. :::info The prompt template can be used for a variety of prompting methods. Let's delve into the descriptions of other different prompting techniques below. ::: You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Few-Shot-Prompter?version=9bf7a4dca6834e71906e72b106d6a94e). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Task-Specific Prompt Here is an example of a text classification task with a task-specific prompt. ``` ### Instruction: You are a customer service agent that is classifying emails by type. ### Input: Hi -- My Mixmaster4000 is producing a strange noise when I operate it. It also smells a bit smoky and plasticky, like burning electronics. I need a replacement. Categories are: (A) Pre-sale question (B) Broken or defective item (C) Billing question (D) Other (please explain) ### Assistant: My answer is ( ``` Response: ``` B) Broken or defective item ``` ## Translation Here is an example of a prompt text for a text translation task. ``` Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Translate the following phrase into French. ### Input: I miss you ### Response: ``` Response: ``` Je te manque ``` ## Pre-Encoded Knowledge QA This involves utilizing the model's built-in pre-encoded knowledge base to respond to questions. The model is provided with a large collection of facts and relationships, which it uses to generate answers when given prompts or questions. The pre-existing knowledge base equips the model with the ability to answer questions that demand a good understanding of the world. Here is an example. ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: How did Julius Caesar die? ### Response: ``` Response: ``` Julius Caesar was assassinated by a group of up to 60 conspirators, led by Gaius Cassius Longinus and Marcus Junius Brutus, in the Senate House on the Ides of March (15 March) of 44 BC. ``` ## Close-Book QA Close-book QA, also known as zero-shot QA, refers to the ability of an LLM to answer questions without access to any additional information or context beyond its internal knowledge base. This stands in contrast to open-book QA, where the LLM can access and process external sources of information, such as documents, web pages, or knowledge bases. Here is an example of a close-book QA with a model specific prompt. ``` Below is a document, followed by a question. Answer the question using information in the document. Respond using only information in the document below. Do not provide any other information that is not in the document. ### Input: {{DOCUMENT}} ### Instruction: {{QUESTION}} ### Response: ``` ## Text Extraction Here is an example of a prompt for a text extraction task. ``` Please precisely copy any email addresses from the following text and then write them, one per line. Only write an email address if it's precisely spelled out in the input text. If there are no email addresses in the text, write "N/A". Do not say anything else. ### Input: {{TEXT}}. ### Response: ``` --- ## Push # Push **Learn about our push operators** Push operators help you to automate processes. You can trigger a wide variety of actions based on predictions made by models in your workflow. For example, you can automatically add a label to an image based on predicted concepts. :::tip Since the push operators can be "chained" together with models to automate tasks in a workflow, you can learn how to create workflows [here](https://docs.clarifai.com/portal-guide/workflows/input-nodes#create-your-workflow). ::: ## Annotation Writer **Input**: Any **Output**: Any Annotation Writer allows you to write the input data to the database in the form of an annotation with a specified status as if a specific user created the annotation. We'll demonstrate how to use the Annotation Writer to automatically annotate text and image inputs in your app. ### How to Annotate Text Inputs Let's start by demonstrating how you can use an Annotation Writer, alongside a [Regex-Based Classifier](https://docs.clarifai.com/portal-guide/agent-system-operators/algorithmic-predict#regex-based-classifier), to automatically label text inputs. :::caution Regex-Based Classifier **Input**: Text **Output**: Concepts This operator allows you to classify text using regular expressions. When the specified regex pattern matches the text, the text is assigned to one of the predefined concepts. ::: **1.** Create a concept that you want to be automatically assigned to your input. You can learn how to create concepts [here](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete). For this demonstration, let's create the `bucket` concept. **2.** Create a [labeling task](https://docs.clarifai.com/portal-guide/annotate/create-a-task). Remember to choose `Classification` as the modeling objective. Then, go to the **Tasks** listing page and copy the ID of the task. ![](/img/others/annotation_writer_1.png) **3.** Go to the [workflow builder](https://docs.clarifai.com/portal-guide/workflows/workflow-builder/) page. Search for the `regex-based-classifier` node in the left-hand sidebar and drag and drop it onto the empty workspace. ![](/img/others/annotation_writer_1_1.png) In the right-hand sidebar, click the **SELECT CONCEPTS** button and use the pop-up that appears to select the relevant concept(s) already existing in your application. For this example, we select the `bucket` concept. In the **regex** field, provide the regex pattern that will be used to classify the text. If the pattern matches, the text will be classified as the selected concept. For this example, we provide `\bbucket\b`, which would match the word "bucket" in instances where it appears as a whole word, surrounded by word boundaries. **4.** Search for the **annotation-writer** node in the left-hand sidebar and drag and drop it onto the workspace. Within the right-hand sidebar, specify the output configuration values: ![](/img/others/annotation_writer_2.png) - Set the `annotation_status` as `ANNOTATION_SUCCESS`. This will write the annotations with the annotation success status. - Set the `annotation_user_id`. This is the `user_id` for which to write the annotations on their behalf as if they manually did the work themselves. You could also choose your own `user_id`. - For this example, let's leave the `annotation_info` field empty. - Set the `task_id`. This is the ID the task annotation work belongs to. You can retrieve it from the **Tasks** listing page — as mentioned earlier. :::warning Annotation Status The **annotation_status** field lets you specify the [status of the annotation](https://docs.clarifai.com/api-guide/advanced-topics/status-codes#annotation-related-codes-24xxx) that will be created. These are the possible values: - `ANNOTATION_SUCCESS` (code 24150) — The annotation is created successfully. - `ANNOTATION_PENDING` (code 24151) — The annotation status is pending. - `ANNOTATION_AWAITING_REVIEW` (code 24157) — Annotation is still waiting for review before it's finalized. - `ANNOTATION_AWAITING_CONSENSUS_REVIEW` (code 24159) — Annotation is still waiting for a consensus review before it's finalized. ::: **5.** Connect the operators and save your workflow. **6.** Go to your app's settings page and change the [Base Workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/#how-to-change-a-base-workflow) to the workflow you just created. **7.** [Upload a text input](https://docs.clarifai.com/portal-guide/data/#upload-inputs) into your app, such as `I'm looking for a bucket list`. If you go to the Input-Viewer page, you'll notice that the text input has been automatically labeled with the `bucket` concept. ![](/img/others/annotation_writer_3.png) The Regex-Based Classifier successfully classified the input and forwarded it to the Annotation Writer for labeling. You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Annotation-Writer?version=2c31251b6b524a308d723310137a9670). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ### How to Annotate Image Inputs Next, let's demonstrate how you can use an Annotation Writer to automatically label image inputs. For this demonstration, we'll create a workflow that automatically classifies images of dogs and cats, assigning them the appropriate labels. **1.** Create the concepts that you want to be automatically assigned to your input. You can learn how to create concepts [here](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete). For this demonstration, let's create `cat` and `dog` concepts. **2.** Create a [visual classification](https://docs.clarifai.com/portal-guide/model/model-types/visual-classifier) model and train it with the `cat` and `dog` concepts. You can learn how to create the model [here](https://docs.clarifai.com/portal-guide/model/deep-training/#how-to-fine-tune-a-model). **3.** Create a [labeling task](https://docs.clarifai.com/portal-guide/annotate/create-a-task). Remember to choose `Classification` as the modeling objective. Then, go to the **Tasks** listing page and copy the ID of the task — as mentioned earlier. **4.** Go to the [workflow builder](https://docs.clarifai.com/portal-guide/workflows/workflow-builder/) page. Search for the `visual-classifier` node in the left-hand sidebar and drag and drop it onto the empty workspace. Connect it to the `IN` element. :::caution visual classifier **Input**: Image **Output**: Concepts It allows you to classify images into a set of concepts. ::: ![](/img/others/annotation_writer_4.png) Use the pop-up that appears in the right sidebar to search for the classification model you created and add it to the node. After selecting the model, we'll use the default settings of the other output configuration options for the purpose of this illustration. **5.** Search for the `concept-thresholder` node in the left sidebar and drag it onto the workspace. Connect it to the `visual-classifier` node. :::caution concept-thresholder **Input**: Concepts **Output**: Concepts This operator allows you to threshold input concepts according to both a threshold and an overall operator (>, >=, =, \ " (GREATER_THAN) threshold type. Thus, only the `cat` and `dog` concepts outputted by the visual classifier model above a certain threshold will be processed further downstream. ::: ![](/img/others/annotation_writer_5.png) Click the **SELECT CONCEPTS** button in the right sidebar. In the window that appears, select the relevant concepts already existing in your application. For this example, let's select the `cat` and `dog` concepts, and use the slider to set their threshold values to 0.80 each. This threshold will determine which concepts are accepted to be used for the auto-annotation. Click the **OK** button to save the changes. You'll see the selected concepts highlighted in the right sidebar, along with their threshold values. Lastly, select the `concept_threshold_type` as `GREATER_THAN`. Keep the other configuration options set to their default values. **6.** Search for the **annotation-writer** node in the left-hand sidebar and drag and drop it onto the workspace. Within the right-hand sidebar, specify the output configuration values: ![](/img/others/annotation_writer_6.png) - Set the `annotation_status` as `ANNOTATION_SUCCESS`. This will write the annotations with the annotation success status. - Set the `annotation_user_id`. This is the `user_id` for which to write the annotations on their behalf as if they manually did the work themselves. You could also choose your own `user_id`. - For this example, let's leave the `annotation_info` field empty. - Set the `task_id`. This is the ID the task annotation work belongs to. You can retrieve it from the **Tasks** listing page — as mentioned earlier. **7.** Click the **Save Workflow** button to save your workflow. **8.** Go to your app's settings page and change the [Base Workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/#how-to-change-a-base-workflow) to the workflow you just created. **9.** [Upload an image input](https://docs.clarifai.com/portal-guide/data/#upload-inputs) into your app, such as [this image](https://samples.clarifai.com/cat1.jpeg ) of a cat. If you go to the Input-Viewer page, you'll notice that the image input has been automatically labeled with the `cat` concept. ![](/img/others/annotation_writer_7.png) The model successfully classified the input, the thresholder filtered the concepts, and the filtered concept was forwarded to the Annotation Writer for automatic labeling. --- ## RAG Prompter # RAG Prompter **Learn how to enhance the quality and relevance of LLM-generated text** The RAG Prompter is a prompt template operator that leverages the [Retrieval-Augmented Generation (RAG)](https://www.clarifai.com/blog/what-is-rag-retrieval-augmented-generation) technique to improve the performance of Large Language Models (LLMs). LLMs have revolutionized natural language understanding across various applications and industries. These models have introduced a new era of quick problem-solving and AI-powered interactions. However, LLMs are often plagued by these two main limitations: - Most traditional LLMs have a static knowledge base that is limited to a specific period; for example, as of this writing, ChatGPT's knowledge cut-off is in October 2023. This makes them outdated and unable to respond accurately to recent events. - Most traditional LLMs are trained on an extensive, generalized collection of data, which may not be applicable to your specific use case. Whenever LLMs confront voids in their training data, they might generate seemingly plausible yet erroneous information, a phenomenon referred to as "hallucination." RAG helps solve these issues by incorporating your data into the existing dataset that is accessible to LLMs. With the RAG AI framework, you can retrieve facts from an external knowledge base, thereby anchoring LLMs with the most precise and current information available. Additionally, RAG reduces the necessity for users to continuously train a model on new data and update its parameters to accommodate evolving circumstances. RAG allows users to do much cheaper in-context learning and reduces the need for expensive LLM fine-tuning. :::note - The RAG Prompter model type should not be set as an app’s [base workflow]( https://docs.clarifai.com/portal-guide/workflows/base-workflows/); otherwise, it would cause a dependency cycle. - This model generates search billable events. ::: ## How RAG Works As its name implies, RAG works through two phases: retrieval and content generation. When a user uploads external data to be used for RAG purposes into a Clarifai application, the data is first chunked into bite-sized pieces. The chunks are then passed through an embedding model, which transforms the data into indexed vectors and stores them in a vector database. :::info [Embedding models](https://docs.clarifai.com/api-guide/predict/embeddings) are the type of models usually used to convert data into numerical vectors, while preserving meaningful relationships between them. These vectors are like condensed summaries of the data, capturing its important aspects in a way that machine learning models can understand. By using these vectors, the models can now reason about the data, compare different pieces of information, and perform tasks like similarity search. ::: When an LLM is asked a question, the prompt will first pass through an embedding model and also be converted into a vector. Algorithms will search the Clarifai vector store to *retrieve* the chunks most relevant to that provided user’s prompt. The most relevant chunks are then appended to the user’s query and served as context – extending the prompt to the LLM with a lot of background information. Next, in the generative phase, the LLM leverages the *augmented* prompt along with its internal representation of training data to craft a compelling response tailored to the user's query at that moment. ![](https://www.clarifai.com/hs-fs/hubfs/rag-query-drawio%20(1)-png-2.png?width=2056&height=1334&name=rag-query-drawio%20(1)-png-2.png) Image source: [Clarifai portal](https://www.clarifai.com/hs-fs/hubfs/rag-query-drawio%20(1)-png-2.png?width=2056&height=1334&name=rag-query-drawio%20(1)-png-2.png) ## How to Create a RAG Prompter Let’s demonstrate how you can create a RAG Prompter model on the Clarifai platform. :::note objective Our intention is to search an app containing textual data for relevant information related to a provided text query. We extract the top k most relevant results from the app, and augment these results into the prompt we provide to the model. By leveraging the additional context, the model should be able to generate a more informative response to the question. ::: ### Step 1: Create an Application [Click here](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/#create-an-application-on-the-portal) to learn how to create an application on the Clarifai portal. :::info Base Workflow When creating the application, select the **Text/Document** option as the primary input type. And in the collapsible **Advanced Settings** field, select an embeddings workflow, such as the [baai-general-embedding-base-en](https://clarifai.com/clarifai/main/workflows/baai-general-embedding-base-en) as the [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/). The base workflow will convert the uploaded data into indexed vectors, which makes them searchable – as explained earlier. ::: ### Step 2: Upload Data Next, upload the external data you want to use to optimize the output of a large language model. [Click here](https://docs.clarifai.com/portal-guide/inputs-manager/upload-inputs) to learn how you can upload data to your application. You can also [create a dataset](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete/#create-datasets) and add your inputs to it, then specify the dataset ID when configuring the RAG Prompter. This ensures the RAG Prompter performs context-based searches within the specified dataset, which leads to more accurate and relevant outputs. If a dataset ID is not specified, the RAG Prompter will search across all inputs in your app, spanning all datasets. This can result in mixed context searches and less precise outputs due to jumbled context hits from unrelated datasets. Similarly, you can attach JSON metadata to your inputs when uploading them to the Clarifai platform. Metadata serve as additional information associated with your inputs and, like dataset IDs, can be used to narrow down search results. By specifying metadata, you can filter search results to match specific conditions, which further improves the relevance and accuracy of the context provided by the RAG Prompter. :::warning RAG in four lines of code [Click here]( https://www.clarifai.com/blog/retrieval-augmented-generation-rag-in-4-lines-of-code) to learn how to build a RAG system in four lines of code. You’ll also learn how to upload documents seamlessly to your Clarifai application. ::: For this example, let's add the following inputs to a dataset in our app: ``` The audit findings indicate that $460,000 was expended on machinery repairs during the previous twelve months. ``` ``` The total expenditure allocated to machinery repairs in the 2022 fiscal year amounted to $500,000. ``` ``` The expenses for machinery repairs in the year 2021 totaled $550,000, reflecting increased maintenance requirements. ``` ``` The financial records indicate that $480,000 was disbursed for machinery repairs during 2020 calendar year. ``` ``` During the 2019 fiscal year, the organization disbursed $475,000 on machinery repairs to ensure operational efficiency. ``` ### Step 3: Create a Workflow Go to the [workflow builder](https://docs.clarifai.com/portal-guide/workflows/workflow-builder/). Then, search for the **rag-prompter** node in the left-hand sidebar and drag it onto the empty workspace. ![](/img/others/rag-prompter-1.png) Use the pop-up that appears on the right-hand sidebar to set up the template text as a single-line statement. For this example, let's use this prompt template text: ``` Context information is below: {data.hits} Given the context information and not prior knowledge, answer the query.Query: {data.text.raw} Answer: ``` :::tip Your prompt template must include at least one instance of each of these placeholders: `{data.text.raw}` and `{data.hits}`. During inference, all instances of `{data.text.raw}` within the template will be substituted with the user query provided. Similarly, `{data.hits}` will represent a newline-separated list of search results retrieved from your app’s data and ordered by similarity. ::: To customize your search experience, you can: - Adjust the `min_score` parameter if you desire a minimum threshold for search result scores. - Modify `max_results` to specify the maximum number of relevant search results included in the prompt. - Provide a comma-separated list of `dataset_ids` or a single dataset ID for the RAG Prompter to search within. This ensures the search results are confined to specific datasets, which improves relevance and precision, as earlier explained. - Define the `metadata` in JSON format to filter the search results further. Just like specifying a dataset ID, using [metadata](https://docs.clarifai.com/portal-guide/psearch/pfilter/#filter-by-metadata) enhances context generation and boosts the accuracy of the results. To finalize creating your workflow, connect the **rag-prompter** to a text-to-text node, and choose a text-to-text LLM from the Clarifai Community, such as [GPT-4 Turbo](https://clarifai.com/openai/chat-completion/models/gpt-4-turbo). Then, click the **Save Workflow** button to save your workflow. ### Step 4: Use the Workflow After saving the workflow, you’ll be directed to its individual page, where you can start using it. Click the **+** button to provide your query text. For example, you could provide the following as your input text: ``` How much was spent on machinery repairs in 2020? ``` Click the **Submit** button. Once the workflow has completed processing your input, you'll see the results, starting with the earlier template text, now adapted to your input. As you can see below, the LLM model leveraged the additional context to generate an accurate response to the question. ![](/img/others/rag-prompter-2.png) ## How to Edit a RAG Prompter After creating your RAG Prompter, you can edit it by navigating to its individual page and clicking the **Edit workflow** button in the upper-right section. ![](/img/others/rag-prompter-3.png) You'll be redirected to the workflow editor page, where you can make any changes needed, such as updating the prompt template text or other parameters. ![](/img/others/rag-prompter-4.png) Once you've made your changes, click the **Save as new version** button to save the updated RAG Prompter under a new version — without exiting the workflow editor. :::note You can easily switch between different versions of the RAG Prompter by selecting the respective version ID from the left sidebar in the workflow editor. ::: Note that clicking the **Update Workflow** button creates a new version of your RAG Prompter and exits the workflow editor, redirecting you to its main page. You can then select the version to use for inferencing. ![](/img/others/rag-prompter-5.png) --- ## Tracker # Tracker **Learn about our tracker operators** Tracker operators are a specific type of agent system operators designed for object tracking in computer vision. Object tracking involves following the movement of objects in a sequence of images or frames in a video. Tracker models use detection-based tracking algorithms that don't require training and help them identify and track objects over time. The goal of object tracking is to maintain the identity of the object(s) over time, despite changes in position, scale, orientation, and lighting conditions. :::tip Since the tracker operators can be "chained" together with models to automate tasks in a workflow, you can learn how to create workflows [here](https://docs.clarifai.com/portal-guide/workflows/input-nodes#create-your-workflow). ::: ## BYTE Tracker **Input**: `frames[…].data.regions[…].data.concepts`, `frames[…].data.regions[…].region_info.bounding_box` **Output**: `frames[…].data.regions[…].track_id` [BYTE Tracker](https://arxiv.org/abs/2110.06864) is a multi-object tracking by-detection model built upon the [Simple Online and Real-time Tracking](https://arxiv.org/abs/1602.00763) (SORT) principles. Multi-object tracking aims to predict the bounding boxes and identities of objects within video sequences. BYTE tracker can also be seen as an enhanced version of the Kalman Filter Hungarian Tracker. Most tracking techniques retrieve identities by associating detection boxes whose scores are higher than a threshold. Unlike simpler trackers that ditch detections with low confidence scores, BYTE Tracker considers them, too, making it better at handling situations like temporary occlusions or lighting changes. Typically, it works in two stages: 1. **High Confidence Matches**: First, BYTE Tracker focuses on high-scoring detections (bounding boxes around objects). It uses a combination of motion similarity (how much the object moved between frames) and appearance similarity (features extracted from the object) to match these detections with existing tracks (tracklets). A motion prediction technique is then used to predict the position of these tracks in the next frame. 2. **Low Confidence Recovery**: Here's where BYTE Tracker differs. It revisits the low confidence detections (discarded by simpler trackers) and unmatched tracklets from the previous stage. Using the same motion similarity metric, BYTE Tracker tries to re-associate these with each other, potentially recovering tracks that were lost due to occlusions or low initial confidence. With this powerful operator, you can seamlessly integrate object tracking into your detect-track workflows and unlock advanced capabilities. Let's demonstrate how you can use the BYTE Tracker, alongside [a detection model](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector), to efficiently track objects in videos. **1.** Go to the workflow builder page. Search for the **visual-detector** option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. **2.** Search for the **byte-tracker** option in the left-hand sidebar and drag it onto the workspace. You can set up its output configuration parameters, which are outlined below. **3.** Connect the **visual-detector** model with the **byte-tracker** operator and save your workflow. ![](/img/agent-system-operators/byte_tracker_1.png) To observe it in action, navigate to the workflow's individual page and click the **+** button to input your video. For this example, let's provide [this video](https://samples.clarifai.com/kep_CLIP_20180509-114253.mp4). The workflow will analyze the video and identify objects consistently throughout its duration. ![alt text]() You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/BYTE-Tracker?version=29cfec1d5dbd456c82a54e9302fd37ee). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Centroid Tracker **Input**: `frames[…].data.regions[…].data.concepts`, `frames[…].data.regions[…].region_info.bounding_box` **Output**: `frames[…].data.regions[…].track_id` Centroid trackers rely on the Euclidean distance between centroids of regions in different video frames to assign the same track ID to detections of the same object. Here's a breakdown of how they operate: 1. **Object Detection**: In the first step, an object detector or a segmentation model (not part of the centroid tracker itself) identifies objects in each frame of a video. The detector outputs bounding boxes around the identified objects. 2. **Centroid Calculation**: For each bounding box, the centroid tracker calculates its centroid. The centroid is simply the center point of the box, typically represented by its X and Y coordinates. 3. **Distance Comparison**: The tracker then compares the centroids of objects detected in the current frame with the centroids of objects from the previous frame. It calculates the Euclidean distance, which is a straight-line distance between two points in space. 4. **Track Assignment**: Based on a predefined threshold value, the tracker assigns track IDs. Objects in the current frame whose centroids are within a certain distance of a centroid in the previous frame are considered to be the same object and are assigned the same track ID. Objects with centroids exceeding the threshold distance are assumed to be new objects and assigned new track IDs. Let's demonstrate how you can use the centroid tracker, alongside [a detection model](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector), to efficiently track objects in videos. **1.** Go to the workflow builder page. Search for the **visual-detector** option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. **2.** Search for the **centroid-tracker** option in the left-hand sidebar and drag it onto the workspace. You can set up its output configuration parameters, which are outlined below. **3.** Connect the **visual-detector** model with the **centroid-tracker** operator and save your workflow. ![](/img/agent-system-operators/tracker-1.png) To observe it in action, navigate to the workflow's individual page and click the **+** button to input your video. For this example, let's provide [this video](https://samples.clarifai.com/kep_CLIP_20180509-114253.mp4). The workflow will analyze the video and identify objects consistently throughout its duration. ![](/img/agent-system-operators/tracker-2.png) You can try this workflow [here](https://clarifai.com/clarifai/Sample-Workflows-for-Docs/workflows/Centroid-Tracker?version=ae5e933187e84428bf316caf79b8b3eb). :::note Before trying to access the workflow, please make sure that you have a Clarifai account and are logged in to the Clarifai platform to access the example workflow. If you do not have clarifai account you can signup [here](https://clarifai.com/explore). ::: ## Kalman Filter Hungarian Tracker **Output:** `frames|...].data.regions...].data.concepts, frames[...].data.regions[...].region_info.bounding_box` **Input:** `frames|...].data.regions...] track_id` Kalman filter tracker rely on the Kalman filter algorithm to estimate an object's next position based on its position and velocity in previous frames. Then, detections are matched to predictions using the Hungarian algorithm. This sophisticated model excels in environments where objects move predictably, such as controlled traffic scenes or automated industrial processes. **Key Features:** - **Kalman Filter**: Utilizes a state prediction model that accounts for the linear dynamics of moving objects, calculating their future states based on velocities and directional movements. - **Hungarian Algorithm**: Assigns new detections to predicted states by solving an optimization problem that minimizes the total cost of assigning predictions to observations, thus ensuring the best possible tracking accuracy. **Operational Details:** - **State Prediction**: The Kalman filter predicts the future location of each tracked object based on its current state and motion prediction. This is crucial for maintaining track continuity in environments with predictable movement patterns. - **Detection Matching**: After prediction, detections from the current frame are matched to these predictions using the Hungarian algorithm. This method effectively handles assignments even under conditions where objects may occlude each other or momentarily disappear from view. Let's demonstrate how you can use the Kalman filter tracker, alongside [a detection model](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector), to efficiently track objects in videos. 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. 2. Search for the Kalman filter tracker option in the left-hand sidebar and drag it onto the workspace. You can set up its output configuration parameters, which are outlined below. 3. Connect the visual-detector model with the centroid-tracker operator and save your workflow. ![alt text]() To observe it in action, navigate to the workflow's individual page and click the + button to input your video. For this example, let's provide [this video](https://samples.clarifai.com/JUFNERPILCQEHCZXSAZVCKZBYVUAQUXF.mp4). The workflow will analyze the video and identify objects consistently throughout its duration. ![alt text]() ## Kalman Reid Tracker **Output:** `frames|...].data.regions|...].data.concepts` **Input:** `frames|...].data.regions...] track_id` The Kalman Reid Tracker is an advanced version of the Kalman filter tracking system, enhanced with Re-Identification (ReID) capabilities using appearance embeddings. This tracker is particularly effective in environments where objects frequently occlude each other or experience significant appearance changes, such as in crowded urban areas or complex indoor environments. **Key Features:** - **Embedding Utilization**: Leverages appearance embeddings to provide a secondary layer of identity verification, which helps in accurately re-identifying objects even after they have been occluded or altered in appearance. - **Adaptive Tracking**: Integrates dynamic adjustments to tracking strategies based on the embedding distances, allowing for more flexible and robust tracking capabilities. **Operational Details** - **Kalman Prediction**: Continuously predicts the next positions of objects based on their measured positions and velocities, using a state estimation technique that considers the physics of motion. - **ReID Application**: When there are discrepancies between predicted and observed positions, the system uses appearance embeddings to reassess and realign track IDs. This is particularly useful for maintaining tracking accuracy in scenarios where objects undergo significant appearance changes. Let's demonstrate how you can use the Kalman Reid tracker, alongside [a detection model](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector), to efficiently track objects in videos. 1. Go to the workflow builder page. Search for the visual-detector option in the left-hand sidebar and drag it onto the empty workspace. Then, use the pop-up that appears on the right-hand sidebar to search for a detection model, such as [general-image-detection](https://clarifai.com/clarifai/main/models/general-image-detection), and select its version. You can also set the other configuration options — including selecting the concepts you want to filter. 2. Search for the Kalman Reid tracker option in the left-hand sidebar and drag it onto the workspace. Then, you can set its output configuration parameters, which are outlined below. 3. Connect the visual-detector model with the centroid-tracker operator and save your workflow. ![alt text]() To observe it in action, navigate to the workflow's individual page and click the + button to input your video. For this example, let's provide [this video](https://samples.clarifai.com/JUFNERPILCQEHCZXSAZVCKZBYVUAQUXF.mp4). The workflow will analyze the video and identify objects consistently throughout its duration. ![alt text]() ## Tracker Operators Parameters Here is a table outlining the various output configuration parameters you can configure for each operator (the ✓ symbol represents the operator that supports the parameter). | Parameter | Description | BYTE Tracker | Centroid Tracker | Kalman Filter Hungarian Tracker | Kalman Reid Tracker | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ---------------- | ------------------------------- | ------------------- | | `min_confidence` | This is the minimum confidence score for detections to be considered for tracking | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | `min_visible_frames` | Only return tracks with minimum visible frames > min_visible_frames | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | `track_id_prefix` | Prefix to add on to track and eliminate conflicts | ✓ | ✓ | | ✓ | ✓ | ✓ | | `max_disappeared` | This is the number of maximum consecutive frames a given object is allowed to be marked as “disappeared” until we need to deregister the object from tracking | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | `new_track_confidence_thresh` | Initialize a new track if the confidence score of the new detection is greater than the setting | ✓ | | | | | | | `confidence_thresh` | This is used to categorize high score detections for the first association if their scores are greater, and the second association if not | ✓ | | | | | | | `high_confidence_match_thresh` | The distance threshold for high-score detection | ✓ | | | | | | | `low_confidence_match_thresh` | The distance threshold for low-score detection | ✓ | | | | | | | `unconfirmed_match_thresh` | The distance threshold for unconfirmed tracks, usually tracks with only one beginning frame. `{“min”: 0, “max”: 1}` | ✓ | | | | | | | `max_distance` | Associate tracks with detections only when their distance is below max_distance | ✓ | ✓ | | ✓ | ✓ | ✓ | | `filtered_probability` | If false, return original detection probability; if true, return processed probability from the tracker | | | ✓ | | | | | `max_detection` | Maximum detection per frame | | | ✓ | | | | | `has_probability` | | | | ✓ | | | | | `has_embedding` | | | | ✓ | | | | | `association_confidence` | The list of association confidences to perform for each round | | | | ✓ | ✓ | | | `covariance_error` | Magnitude of the uncertainty on the initial state | | | | ✓ | ✓ | | | `observation_error` | Magnitude of the uncertainty on detection coordinates | | | | ✓ | ✓ | | | `distance_metric` | Distance metric for Hungarian matching | | | | ✓ | ✓ | | | `initialization_confidence` | Confidence for starting a new track. Must be > min_confidence to have an effect | | | | ✓ | ✓ | | | `project_track` | How many frames in total to the project box when detection isn’t recorded for track | | | | ✓ | ✓ | | | `use_detect_box` | How many frames to project the last detection box, should be less than project_track_frames (1 is the current frame) | | | | ✓ | ✓ | | | `project_without_detect` | Whether to keep projecting the box forward if no detect is matched | | | | ✓ | ✓ | | | `project_fix_box_size` | Whether to fix the box size when the track is in a project state | | | | ✓ | ✓ | | | `detect_box_fall_back` | Rely on the detect box if the association error is above this value | | | | ✓ | ✓ | | | `keep_track_in_image` | If this is 1, then push the tracker predict to stay inside image boundaries | | | | ✓ | ✓ | | | `match_limit_ratio` | Multiplier to constrain association ( max_distance before assignment | | | | ✓ | ✓ | | | `max_emb_distance` | Maximum embedding distance to be considered a re-identification | | | | ✓ | ✓ | | | `max_dead` | Maximum number of frames for track to be dead before we re-assign the ID | | | | ✓ | ✓ | | | `var_tracker` | String that determines how embeddings from multiple timestamps are aggregated, defaults to “na” (most recent embedding overwrites past embeddings) | | | | ✓ | ✓ | | | `reid_model_path` | The path to the linker | | | | ✓ | ✓ | | --- ## Applications # Applications **Create applications to work with AI resources on Clarifai** Applications are the basic building blocks for creating projects on the Clarifai platform. An application is literally what it sounds like: an application of AI to an existing challenge. It’s a self-contained project for storing and handling, data, annotations, models, concepts, datasets, workflows (chaining of models together), searches, modules, and more. You can create as many applications as you want and edit or delete them as you see fit. You can can divide your use among them to segment data into collections and manage access accordingly. Usually, you would create a new application for each new set of related tasks you want to accomplish. :::info An operation performed in one application will return results from data within that application, but will be blind to data in other applications. ::: Applications include a variety of resources and provide control over access to those resources. Access to each application can be controlled using Personal Access Tokens (PATs) or app-specific API Keys. ![](/img/applications.jpg) import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Apps Creation # Apps Creation **Learn how to create a Clarifai application** [As mentioned previously](README.mdx), an app on the Clarifai platform acts as a central repository for models, datasets, inputs, and other resources. ## **Create via the UI** To create an app, start by [logging in to](https://clarifai.com/login) to your account. Next, click the plus (**+**) button in the upper-right corner of the Clarifai platform dashboard. A dropdown menu will appear with several options. From the list, select **New Application**. ![create app portal](/img/others/create-new-app-new-1.png) > **Alternatively:** _Open the collapsible left sidebar and select **Projects**. From the dropdown that appears, choose **Select Application** to open a panel displaying a searchable list of your existing applications. Scroll to the bottom of this list and click the **Create new Application** option, marked with a **+** icon._ > _![create app portal](/img/others/alternate-create-new-app-new.png)_ Next, on the window that pops up, select the **Start with a Blank App** option to create a new application from scratch. ![application creation window](/img/create-new-app-new.png) > **Tip:** You can also select the **Use an App Template** option to [use a template](https://docs.clarifai.com/clarifai-basics/app-templates) to create a new application. Next, on the window that appears, provide the information required to create a new application. ![](/img/others/create-new-app-new-2.png) - **Owner** — Choose the owner to associate with the app. By default, this is set to your personal account, but you can also select an organization you belong to, provided you have the necessary [permissions](https://docs.clarifai.com/control/clarifai-organizations/security#scopes-and-access-levels-of-organization-members) to create apps within it. - **App ID** — This serves as a unique identifier for your application. It’s important to choose a unique and memorable ID as it will be used for URLs and redirections. - **Short Description** — Optionally, provide a brief description that outlines the purpose or features of your app. - **Primary Input Type** — Select the primary type of inputs that will be available in your app: either images/videos or texts/documents. - **Advanced Settings** — This collapsible field allows you to choose the [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/) for your app. Base workflows index your data and furnish your app with a default knowledge base. While a base workflow is automatically selected based on your primary input type, you can choose another one that best suits your specific use case. :::note ID Validation Application names and other names in our platform must follow a few rules: - Names must be 1 to 32 letters or numbers in length, with hyphens or underscores as separators. - Spaces, periods, etc., are not allowed as separators. ::: Lastly, click the **Create App** button in the upper-right corner to finalize and create your application. ### Copy (Duplicate) Apps You can also create a new app by cloning an existing one. Cloning is a convenient way to start a new project using an established setup or to branch off from an existing application. To do so, open the collapsible left sidebar, select **Projects**, and then choose **All Resources** from the dropdown list. This will take you to a page that displays all the resources you own, with the **Apps / Templates** tab selected by default. From there, locate the app you want to clone. Click the three dots in the bottom-right corner of the app card, and from the dropdown menu, select **Duplicate** to copy its contents into a new application. ![](/img/app_duplication.png) > **Note:** The dropdown menu also allows you to copy the app ID or delete the app. Next, a pop-up window will appear where you can choose the destination user or organization, as well as the target application. You can either create a new app as the destination or copy the contents into an existing app. The window also allows you to specify which resources you want to duplicate. ![](/img/others/app_duplication-1.png) Lastly, click the **Confirm** button, and the copied app will be automatically created for you. ## **Create via the API** For enterprise customers, it is possible to generate applications programmatically. Note that you need to use a [Personal Access Token (PAT)](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) to create an application. ### Create App With Default Base Workflow [The base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/) acts as the default knowledge base for your app and provides the basic structure for indexing your data. If you create an application without specifying the base workflow, a default one will be automatically created for you. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeCreateApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/create_app.py"; import CodeCreateAppTS from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/createApp.ts"; import CodeCreateAppBase from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/create_app_base_workflow.py"; import CodeOutputCreateApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/outputs/create_app.txt"; {CodeCreateApp} {CodeCreateAppTS} ```javascript curl --location --request POST "https://api.clarifai.com/v2/users/YOUR_USER_ID_HERE/apps/" \ --header "Content-Type: application/json" \ --header "Authorization: Key YOUR_PAT_HERE" \ --data-raw '{ "apps": [ { "id": "test-app" } ] }' ``` Example Output {CodeOutputCreateApp} ### Create App With Different Base Workflow You can create an app with a different base workflow. You can choose from a range of available base workflows, including **Empty**, **Universal**, **Language Understanding**, and **General**. This enables you to seamlessly integrate and customize the fundamental structure of your app, ensuring it aligns perfectly with your project requirements. {CodeCreateAppBase} --- ## Apps Management # Apps Management **Learn how to manage your Clarifai applications** ## **Manage via the UI** ### App Overview The App Overview page gives a glimpse of your app and its purpose. To access it, go to your application's individual page and select the **Overview** option from the collapsible left sidebar. You'll be redirected to a page that gives an overview of the contents of your app. ![app overview](/img/app_overview_page.png) You can accomplish various tasks on that page. - Click the pencil icon next to the app ID to change it. - Click the **Edit Visibility** button in the upper-right corner to edit the visibility of your app to either public or private. - Click the star icon in the upper-right corner to highlight your app. This marks it as a favorite and makes it easier to access among other listed apps. - Click the three dots in the upper-right corner to reveal a dropdown list that lets you duplicate the app, copy its ID, or delete it. - View the number of each type of resource contained in your app — inputs, datasets, models, workflows, and modules. If the number is large and has been truncated in the display, click on the tooltip (?) to reveal the precise count of the resource. - Click the plus icon (**+**) within any resource type section to add the resource to your app. If you click the button, you’ll be redirected to a subsequent page that allows you to add the resource you want. - Click the "view" link within any resource type section to view all the respective resources available in your app. - Add notes that describe what your app is about. - Upload a cover image and use it as a reference for the app. - Edit the app’s description. - View the app’s base workflow, last updated date, and default language. - Copy the app's URL and share it across various platforms and social media channels for wider visibility and reach. - View the types of models contained in the app. - View and edit the [collaborators](#add-collaborators) associated with the app. ### App Settings You can change the settings of an application at any time. To do so, go to the application's individual page and select the **Settings** option in the collapsible left sidebar. You'll be redirected to the **App Settings** page, where you can change the application's settings. ![app settings](/img/edit_application.png) Let's talk about the changes you can make on the page. :::note This page has intuitive elements that let you search for specific items and sort displayed items based on your preferences. ::: #### API Keys ##### Create API Keys Click the **Create API Key** button to add a new [API key](https://docs.clarifai.com/control/authentication/key/) to your app. Then, use the form that pops up to generate a new API key — provide a short description, select the [scopes](https://docs.clarifai.com/control/authentication/scopes), and click the **Create API Key** button. ![api keys](/img/application_settings_1.png) The new app-specific key will be listed in the **API Keys** section, where you can carry out various management tasks on it. ##### Manage API Keys The **Actions** column allows you to complete the following actions: ![copy an api key](/img/application_settings_2.png) - Copy an API key to the clipboard by clicking the copy button. - Reveal an API key by clicking the show button. You can also hide it by clicking the same button. - Edit an API key by clicking the edit button. A form will pop up that allows you to update the description and scopes of your API key. - Delete an API key by clicking the delete button. #### Collaborators Collaboration is a functionality that allows you to share your apps so that you can work with your team members to label data, create models, and more. With this feature, you can control the permissions available in your apps and manage the capabilities available to each user. For example, you can give a trusted collaborator full access privileges. You can also invite a worker and grant them limited permissions, such as only to annotate data but not to delete them. ##### Add Collaborators To add a collaborator to your app, click the **Add Collaborators** button in the **Collaborators** section. A form will appear that allows you to invite a collaborator. ![add a collaborator](/img/application_settings_6.png) Enter the email address associated with the collaborator’s Clarifai account and specify the scope of resources you want them to access. Lastly, click the **Confirm** button. The invited collaborator will receive an email notification informing them that they have been added to the app. :::note - The collaborator you want to invite must have a registered account on the Clarifai platform. - [Personal Access Tokens (PATs)](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) are automatically generated for collaborators, providing them access to the Clarifai API. - Collaborators can also invite other collaborators. ::: > _Alternatively, you can add a collaborator directly from the **[App Overview](#app-overview)** page._ > _![](/img/others-2/app-collaboration-1.png)_ > _In the upper-right corner of the **App Overview** page, click the add collaborator button. Next, use the form that pops up to invite a collaborator — provide an email address associated with their Clarifai account, select their scopes, and click the **Confirm** button._ :::info Choose Scopes You can select specific scopes when adding a collaborator. Modifying these scopes provides fine-grained control over the data that users can access, enhancing your app's security and preventing unauthorized intrusions. In the **Add Collaborator** form, you can click the **Select All Scopes** button to choose all scopes at once. Alternatively, you can select individual scopes from the listed **BASIC SCOPES**, or click on the **ADVANCED SCOPES** option to access more advanced choices. You can edit a collaborator's scopes at a later time. These are the basic scopes you can choose: ```text Annotation: Add and Remove Annotations on Inputs Collaborator: Add, Remove and Patch Collaborators Collector: Add, Remove and Patch Input Collectors Concept: Add, Remove and Patch Concept Input: Add and Remove Inputs to an App Model: Add, Remove and Train Custom Models Predict: Predict on Public and Custom Models Search: Search over Inputs an App Task: Add, Remove and Patch Scribe labeling Tasks Vocab: Add, Remove and Delete lists of Concepts Workflow: Add, Remove and Delete Workflows of Models ``` [Click here](https://docs.clarifai.com/clarifai-basics/authentication/scopes/) to learn more about scopes. ::: ##### Manage Collaborators You can edit a collaborator’s scopes or remove them entirely by using the corresponding buttons in the **Actions** column. ![manage collaborator](/img/application_settings_7.png) ##### View Collaborations You can easily access the apps where you've been added as a collaborator. To do so, click your user profile icon in the upper-right corner of the Clarifai dashboard and select **Collaborations** from the dropdown menu. This will take you to a page that lists all the apps you've been invited to collaborate on. ![](/img/others-2/view-collaborations.png) #### Base Workflow [A base workflow](https://docs.clarifai.com/create/workflows/base-workflows/) is the workflow you choose as the default knowledge base when creating an application. It optimizes your custom model performance. ![change base workflow](/img/application_settings_8.png) You can change your app's base workflow by selecting another one from the list that drops down when you click the workflows search box. You can choose public workflows or workflows you've created in your app. > **Note:** If you select the **Reindex existing inputs** checkbox, updating the base workflow will trigger a reindex of your app, reprocessing all inputs through the new base workflow. This process may take some time and could incur additional costs. To avoid these costs, you can delete all your inputs before updating the base workflow. After selecting your preferred base workflow, click the **Change Base Workflow** button. :::warning Performing any of the actions below is dangerous. So, you need to proceed with caution. ::: #### Update Visibility To change your app’s visibility, click the **Edit Visibility** button. In the pop-up window, select either **Public** to make the app accessible to anyone or **Private** to restrict access. Then, click **Confirm** to apply the change. > **Note:** You can also enable the **Mark App as a template** option to make the app reusable as a starting point. Marking an app as a [template](https://docs.clarifai.com/create/applications/templates) allows its resources — such as inputs, datasets, models, workflows, and modules — to be used as the foundation for creating new applications. ![update visibility](/img/application_settings_update_visibility.png) #### Delete Application You can delete the application by clicking the **Delete App** button. A small window will pop up, prompting you to confirm the action. ![delete app](/img/application_settings_11.png) :::warning Remember that once you delete an application, we cannot recover it. You will also lose all images, concepts, models, and other resources associated with that application. ::: #### Delete all Models You can delete all models by clicking the **Delete all Models** button. A small window will pop up, prompting you to confirm the action. ![delete all models](/img/application_settings_10.png) #### Transfer Ownership You can [transfer your application](https://docs.clarifai.com/control/clarifai-organizations/apps-models-workflows#transfer-an-app) to an organization you belong to and have permission to create apps. This would greatly enhance collaboration and boost your productivity. To do so, click the **Transfer** button. On the small window that pops up, select the app’s destination organization and give it a new name, if you want to. Click the **Confirm** button to finalize the transfer. ![transfer application](/img/application_settings_9.png) The app will be successfully moved to the organization you selected, and it will no longer appear under your listed apps. ## **Manage via the API** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodePatchApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/patch_app.py"; import CodeCreateAppBase from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/create_app_base_workflow.py"; import CodeListApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/list_apps.py"; import CodeListAppTS from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/listingApps.ts"; import CodeDeleteApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/delete_app.py"; import CodeDeleteAppTS from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/deleteApp.ts"; import CodeCreateConcept from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/create_concept.py"; import CodeCreateConceptTS from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/createConcept.ts"; import CodeListConcept from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/list_concept.py"; import CodeListConceptTS from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/listConcept.ts"; import CodeOutputListApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/outputs/list_apps.txt"; import CodeOutputDeleteApp from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/outputs/delete_app.txt"; import CodeOutputCreateConcept from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/outputs/create_concept.txt"; import CodeOutputListConcept from "!!raw-loader!../../../code_snippets/python-sdk/create-apps/outputs/list_concept.txt"; ### Patch App You can perform patch operations on an app by merging, removing, or overwriting data. By default, all actions support overwriting, with specific behaviors for lists of objects. - The `merge` action updates an existing `key:value` pair with `key:new_value` or appends to an existing list. For dictionaries, it merges objects that share a matching `id` field. - The `remove` action is only used to delete the app's cover image on the platform UI. - The `overwrite` action fully replaces an existing object with a new one. Below is an example of performing patch operations on an app, where the base workflow is updated, the app is switched to an [app template](https://docs.clarifai.com/create-manage/applications/templates/), and changes are made to the app's description, notes, default language, and image URL. {CodePatchApp} ### List Apps You can use the `list_apps` method to retrieve all the apps associated with your account. This method supports [pagination](https://docs.clarifai.com/additional-resources/api-overview/pagination), allowing you to tailor the results using parameters like `page_no` and `per_page` to fit your specific needs. {CodeListApp} Example Output {CodeOutputListApp} {CodeListAppTS} ### Delete App You can remove an app from your account by specifying its app ID. Note that you need to be certain that you want to delete a particular app as the operation cannot be undone. {CodeDeleteApp} Example Output {CodeOutputDeleteApp} {CodeDeleteAppTS} ### Create Concepts You can create [concepts](https://docs.clarifai.com/portal-guide/inputs-manager/concepts) by uploading input data to your app along with a specified label. This automatically generates new concepts, making it easier to integrate and manage new ideas or entities within your application. {CodeCreateConcept} Example Output {CodeOutputCreateConcept} {CodeCreateConceptTS} ### List Concepts You can use the `list_concepts` method to retrieve all the concepts within your application. This method supports [pagination](https://docs.clarifai.com/additional-resources/api-overview/pagination), allowing you to tailor the results using parameters like `page_no` and `per_page` to fit your specific needs. {CodeListConcept} Example Output {CodeOutputListConcept} {CodeListConceptTS} --- ## App Templates # App Templates **Create applications faster with pre-built blueprints** Clarifai app templates are pre-built blueprints that provide a starting point for creating your own applications. They are apps with their contents grouped by some use case — enabling you to easily get started building your applications. When you choose a template to create an app, the configurations and resources available in the template will be preemptively applied to your new application. You can use the pre-built components to quickly apply AI to your specific use case. :::info Apps on the Clarifai platform are like self-contained projects. They store and handle your data, annotations, [models](https://docs.clarifai.com/portal-guide/model/), [workflows](https://docs.clarifai.com/portal-guide/workflows/working-with-workflows), [modules](https://docs.clarifai.com/portal-guide/modules/), and other resources. ::: ## Why Use an App Template? You might choose an app template over building a custom application from scratch for several reasons. - **Time efficiency** — App templates provide a head start in your AI development journey. They offer a ready-made foundation to launch your application much faster than starting from scratch. - **Leverage pre-existing resources** — App templates empower you to leverage pre-existing resources, such as models, workflows, or modules. Instead of starting from scratch, you can begin with a fully equipped application that addresses a particular use case. - **Ease of customization** — While app templates provide a foundation for AI development, they are often highly customizable. You can easily modify the template to suit your preferences. ## App Templates Examples We provide a variety of application templates in various categories to help you hit the ground running. :::note on rapid expansion - We're constantly expanding our library of templates, with new additions released on a regular basis. You can check [here](https://clarifai.com/explore/apps?activeToggle=Templates&page=1&perPage=24) for them. - We'll also introduce the ability to build your own app templates soon. ::: Here are some examples: - **[Chatbot Template](https://clarifai.com/clarifai/chatbot-template)** — It serves as an extensive guide for building AI chatbot assistants swiftly and effectively, utilizing the capabilities of Clarifai's Large Language Models (LLMs). - **[Content Generation](https://clarifai.com/clarifai/content-generation-template)** — It empowers you to effortlessly craft a variety of high-quality textual content, from compelling emails and blog posts to engaging social media content and captivating stories. This helps boost your communication skills and spark your creativity. - **[Document Summarization](https://clarifai.com/clarifai/document-summarization)** — It lets you condense a longer text document into a shorter version that captures the most important points or information. This template provides different methods for summarization that cater to different lengths and complexities of text. - **[Image Moderation](https://clarifai.com/clarifai/image-moderation)** — It provides diverse AI-powered workflows for automatically filtering and categorizing inappropriate or harmful images based on various criteria. - **[RAG Template](https://clarifai.com/clarifai/rag-template)** — It streamlines the creation of Retrieval-Augmented Generation (RAG) applications with Clarifai. This lets you enhance LLMs with external knowledge for accurate, up-to-date information generation. - **[Text Moderation](https://clarifai.com/clarifai/text-moderation)** — It explores various text moderation use cases and provides several ready-to-use workflows and models. It leverages different NLP models and LLMs to address diverse scenarios. - **[Coding Template](https://clarifai.com/clarifai/coding-template)** — It explores a variety of coding scenarios and includes pre-built workflows tailored to address specific use cases, utilizing specialized models for each unique situation. ## How to Create an App Using a Template There are two ways to create an application using a template. - Via the create-an-app modal - Via an app’s template page ### Via the create-an-app modal To create an app using a template, start by [logging in to](https://clarifai.com/login) to your account. Next, click the plus (**+**) button in the upper-right corner of the Clarifai platform dashboard. A dropdown menu will appear with several options. From the list, select **New Application**. ![](/img/others/app-template-9.png) > **Alternatively:** _Open the collapsible left sidebar and select **Projects**. From the dropdown that appears, choose **Select Application** to open a panel displaying a searchable list of your existing applications. Scroll to the bottom of this list and click the **Create new Application** option, marked with a **+** icon._ > _![](/img/others/app-template-10.png)_ Next, on the window that pops up, select the **Use an App Template** option to create a new application using a template. ![](/img/others/app-template-2.png) > **Tip:** You can also select the **Start with a Blank App** option to start [creating a new application](https://docs.clarifai.com/create/applications/create#create-via-the-ui) from scratch. The ensuing page lets you select a template for creating your application. You can opt for a template provided by the community or select one of your own creations. For this illustration, let’s select the **text-moderation** template, which is provided by the community. ![](/img/others/app-template-3.png) On the template’s individual page, you'll find an overview of its available resources as well as a description of its functionality. ![](/img/others/app-template-3-1.png) When you click the view link of a resource, a modal will pop up, allowing you to easily view the available resources in the template, including inputs, datasets, models, workflows, and modules. For example, if you click the **View models** link, a modal will appear displaying the models included in the template. You can also browse through the other tabs to explore additional resources present in the template. ![](/img/others/app-template-3-2.png) Next, click the **Select Template** button in the upper-right corner of the page. ![](/img/others/app-template-4.png) Next, specify the setup of your new application. - Provide an app ID, which serves as a unique identifier for your application. It’s important to choose a unique and memorable ID as it will be used for URLs and redirections. - Optionally, provide a brief description that outlines the purpose or features of your app. Lastly, click the **Create App** button in the upper-right corner of the page. ![](/img/others/app-template-5.png) Once the app has been created, you can begin using it immediately. For example, you can use one of its workflows for text moderation tasks. ![](/img/others/app-template-5-1.png) ### Via an app’s template page You can also create an app from the page of an existing template. To do so, open the collapsible left sidebar, select **Projects**, and choose **All Resources** from the dropdown list. This will open a page showing all the resources you own, with the **Apps / Templates** tab selected by default. From there, select the **Templates** tab in the right side of the page. A list of the available templates will be populated on that page. Select the template you want to use. ![](/img/others/app-template-6.png) You’ll be directed to the template’s individual page. ![](/img/others/app-template-7.png) Next, click the **Use Template** button in the upper-right corner of the page. And in the window that pops up, provide the details for creating your new application. The pop-up window also displays the types of resources included within the template. Lastly, click the **Create App** button to finalize the creation of your app. --- ## Artifacts # Artifacts **Programmatically manage binary, file-based assets** Clarifai artifacts are binary, file-based assets associated with your apps. An artifact acts as a logical container for any binary or structured output produced by a user’s workload. At a high level, artifacts are: * **User-defined** — Created explicitly by users or automatically by [Pipelines](https://docs.clarifai.com/compute/pipelines/create-api). Pipelines support long-running workloads such as model training, fine-tuning, and data preprocessing, which produce artifact [files](#common-artifact-examples) used in downstream tasks. * **Versioned** — Each artifact contains one or more immutable `ArtifactVersions`, representing distinct snapshots or checkpoints over time. * **Taggable and searchable** — Artifacts support metadata and tagging, making them easy to organize, filter, and discover. * **Linked** — Artifacts can be associated with Clarifai resources such as `PipelineVersionRuns`, Models, or Datasets, establishing clear data lineage and traceability. * **Timestamped** — Artifacts track lifecycle timestamps including `created_at`, `modified_at`, and `deleted_at`. * **Resumable** — Runs can resume from stored artifact checkpoints (exact URI and checksum) instead of recomputing from scratch, significantly reducing training time and cost. * **Reproducible and auditable** — Pipeline runs can persist the exact files used to produce results (weights, checkpoints, configs, logs), ensuring experiments are reproducible and verifiable. * **Access-controlled and workspace-scoped** — Artifacts inherit app and project permissions, enabling secure sharing and [collaboration](https://docs.clarifai.com/create/applications/manage#collaborators). Shared artifacts can be reused across multiple pipeline version runs without duplication. * **Modality-agnostic** — Unlike entities such as inputs, models, or datasets — which are semantically tied to specific modalities or roles — an artifact provides a generic abstraction for storing arbitrary data, regardless of format or purpose. ## Common Artifact Examples Artifacts can store a wide range of assets, including: * Model checkpoints and weights (`.ckpt`, `.pt`, `.bin`) * Training logs or metrics (`.json`, `.csv`) * Preprocessed embeddings (`.npy`, `.pkl`) * Packaged datasets (`.tar.gz`, `.parquet`) * Serialized configuration or tokenizer files (`.yaml`, `.json`) ## Artifact Resource Paths Artifacts are addressed using fully qualified Clarifai resource paths: ``` users/{user-id}/apps/{app-id}/artifacts/{artifact-id} ``` Artifact versions are addressed as: ``` users/{user-id}/apps/{app-id}/artifacts/{artifact-id}/versions/{version-id} ``` These paths are used consistently across the CLI and SDK when managing artifacts and their versions. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Artifacts Management # Artifacts Management **Create, upload, download, list, get, and delete** [Clarifai artifacts](README.mdx) are binary, file-based assets associated with your apps — such as model weights, checkpoints, configuration files, or other supporting resources. You can create, upload, download, list, inspect, and delete artifacts and their versions directly from the command line or using our Python SDK. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PyCreate1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-1.py"; import PyUpload1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-2.py"; import PyUpload2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-3.py"; import PyUpload3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-4.py"; import PyUpload4 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-5.py"; import PyDownload1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-6.py"; import PyDownload2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-7.py"; import PyDownload3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-8.py"; import PyDownload4 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-9.py"; import PyList1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-10.py"; import PyList2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-11.py"; import PyGet1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-12.py"; import PyGet2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-13.py"; import PyDelete1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-14.py"; import PyDelete2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-15.py"; import CLI1Upload1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-1.sh"; import CLIUpload2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-2.sh"; import CLIUpload3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-3.sh"; import CLIUpload4 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-4.sh"; import CLIUpload5 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-5.sh"; import Download1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-6.sh"; import Download2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-7.sh"; import Download3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-8.sh"; import Download4 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-9.sh"; import Download5 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-10.sh"; import Download6 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-11.sh"; import List1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-12.sh"; import List2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-13.sh"; import Get1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-14.sh"; import Get2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-15.sh"; import Delete1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-16.sh"; import Delete2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-17.sh"; import Delete3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/cli-18.sh"; import ModelPt from "!!raw-loader!../../../code_snippets/new-docs/artifacts/model-pt.py"; import Output1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-1.txt"; import Output2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-2.txt"; import Output3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-3.txt"; import Output4 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-4.txt"; import Output5 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-5.txt"; import Output6 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-6.txt"; import Output7 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-7.txt"; import Output8 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/output-8.txt"; ## Prerequisites ### Install Clarifai Package Install the latest version of the `clarifai` Python package. This also installs the Clarifai Command Line Interface (CLI), which we'll use for running artifact commands. pip install --upgrade clarifai ### Get User ID and PAT You’ll need the following credentials from the [Clarifai platform](https://clarifai.com/explore) for managing artifacts: - **User ID** – In the collapsible left sidebar, select **Settings** and choose **Account** from the dropdown list. Then, locate your user ID. - **Personal Access Token (PAT)** – From the same **Settings** option, choose **Secrets** to generate or copy your [PAT](https://docs.clarifai.com/control/authentication/pat). This token is used to authenticate your connection with the Clarifai platform. You can then set the PAT as an environment variable using `CLARIFAI_PAT`. export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ### Create an App In the Clarifai platform, [create an app](https://docs.clarifai.com/create/applications/create) where your artifact will be stored. After creating the app, make note of the app ID, as it is required when referencing [artifact paths](README.mdx#artifact-resource-paths) in the CLI or SDK. :::note CLI commands When using the Clarifai CLI, run `clarifai artifact --help` to view the complete list of available commands and their descriptions. > **Note:** You can use **`af`** as a shorthand alias for the `artifact` command. Example: `clarifai af --help`. > These are the available commands: > - `cp` – Upload or download artifact files > - `list` (`ls`) – List artifacts or artifact versions > - `get` – Retrieve artifact or version details > - `delete` (`rm`) – Delete artifacts or artifact versions ::: ## Create an Artifact With the Python SDK, you can create an artifact object under your user and app by specifying an artifact ID. Once created, it enables you to store files as versioned assets and easily upload, track, and manage changes over time. {PyCreate1} ## Upload Artifacts You can upload an artifact file by specifying an artifact ID; this automatically creates a new artifact version. > **Note:** If the artifact does not already exist, it will be created automatically before the upload. Example Artifact: model.pt {ModelPt} {PyUpload1} {CLI1Upload1} Example Output {Output1} You can upload with a description and visibility. > **Note:** These are the visibility options you can set: `private` (default) restricts access to you, `org` allows access within your organization, and `public` makes the artifact accessible to anyone. {PyUpload2} {CLIUpload2} You can upload directly to a specific artifact version by providing a version ID. {PyUpload3} {CLIUpload3} You can specify an expiration time using [RFC3339](https://www.rfc-editor.org/rfc/rfc3339) format (e.g., `2026-12-31T23:59:59.999Z`) > **Note:** Artifacts do not expire by default. An artifact version is only deleted if an `expires_at` value is explicitly set at upload time. {PyUpload4} {CLIUpload4} With the CLI, you can use `--force` or `-f` to overwrite existing files. {CLIUpload5} ## Download Artifacts You can download the latest artifact version to the current directory. > **Note:** If no output path is provided, the file is automatically saved to the current working directory using an appropriate filename. When using the SDK, you should specify the `version_id`. {PyDownload1} {Download3} Example Output {Output2} You can download the artifact directly to the current directory using a specific filename. {PyDownload2} {Download2} > **Note:** This is how you can use the CLI to download a specific artifact version and save it as a named file in the current directory. {Download4} You can use `force` (or `-f` for the CLI) to overwrite existing local files. {PyDownload3} {Download6} You can download the latest artifact version by specifying an already existing local destination directory. {PyDownload4} {Download1} > **Note:** This is how you can use the CLI to download a specific artifact version and save it into an existing local directory. {Download5} ## List Artifacts You can list all artifacts within your app. > **Note:** When using the CLI, you can use `ls` as a shorthand alias for the `list` command. {PyList1} {List1} Example Output {Output3} You can list all versions of a specific artifact. {PyList2} {List2} Example Output {Output4} ## Get Artifact Details You can get the details of a specific artifact. {PyGet1} {Get1} Example Output {Output5} You can get the details of a specific artifact version. {PyGet2} {Get2} Example Output {Output6} ## Delete Artifacts You can delete an artifact (including all versions) by specifying its `artifact_id`. > **Note:** When using the CLI, you can use `rm` as a shorthand alias for the `delete` command. {PyDelete1} {Delete1} Example Output {Output7} You can delete a specific artifact version by including its version ID. {PyDelete2} {Delete2} Example Output {Output8} With the CLI, you can use `--force` or `-f` to delete without a confirmation prompt. {Delete3} --- ## Artifacts Use Cases # Artifacts Use Cases **Examples of how artifacts are used** [Artifacts](README.mdx) provide a flexible, versioned way to store and manage files produced throughout the ML and data lifecycle. They are especially useful for preserving intermediate outputs, enabling reproducibility, and sharing results across teams and environments. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import UseCase1 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-use-case-1.py"; import UseCase2 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-use-case-2.py"; import UseCase3 from "!!raw-loader!../../../code_snippets/new-docs/artifacts/py-use-case-3.py"; ## Example Use Cases ### Model Training You can upload model checkpoints to the Clarifai platform after each training run or epoch. If training is interrupted, you can resume from the most recent checkpoint instead of starting from scratch. {UseCase1} ### Model Deployment You can retrieve the latest approved model artifact and download it locally for deployment in production environments. {UseCase2} ### Pipeline Outputs You can store evaluation metrics, preprocessed embeddings, serialized configurations, or logs produced by pipeline steps. These artifacts can be reused in downstream pipelines, audited later, or shared across teams. ### Experiment Tracking You can use artifacts as a versioned record of experiment outputs. Track how model performance evolves over time, compare results across hyperparameter configurations, and ensure experiments remain reproducible. ## Error Handling Examples You can handle common failure cases gracefully when uploading or downloading artifacts. {UseCase3} --- ## Create and Manage Concepts # Create and Manage Concepts **Put a word to it** A concept in machine learning is a label or tag used to describe an entity, similar to a keyword. It helps annotate data, allowing models to learn from patterns associated with that concept. Over time, through training on annotated data, models can recognize or classify new data based on the concepts they've learned. Concepts are often called "classes" in machine learning tasks. On the Clarifai platform, you can create and manage these concepts within your app to enhance your model's ability to recognize and categorize data. Here are some best practices when creating concepts: * Avoid special characters and spaces * Avoid long concept names. Typically, 1-3 word concepts are best import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Concepts Relations # Concepts Relations **Connect and organize concepts for smarter AI** The Knowledge Graph establishes semantic and hierarchical relationships between your concepts — such as synonyms, hyponyms, and hypernyms — laying the foundation for more intelligent, context-aware systems. These relationships are crucial for building robust knowledge representations that enhance both computer vision and natural language processing capabilities. Whether you're managing a product catalog, powering a search engine, or organizing complex datasets, using concept relationships allow you to structure, search, and retrieve information more effectively — making your AI applications significantly smarter and more precise. ## Types of Concepts Relations - **Hyponym** — This represents an _“is a kind of”_ relationship. For instance, if you define the relationship as _'honey' (subject), 'hyponym' (predicate), 'food' (object),_ it can be interpreted as _"honey is a kind of food."_ This helps systems understand that "honey" belongs to a broader category. - **Hypernym** — This is the inverse of a hyponym, signifying a _“parent of”_ relationship. Using the same example, if you define _'food' (subject), 'hypernym' (predicate), 'honey' (object),_ it is read as _"food is a parent of honey."_ When a hyponym is defined, the corresponding hypernym is automatically inferred in queries, and vice versa, ensuring consistency in how concept hierarchies are interpreted. - **Synonym** — This relationship connects two concepts that essentially mean the same thing, similar to an _“is”_ statement. For example, defining _'puppy' as a synonym of 'pup'_ allows the model to treat them interchangeably. This relationship is bidirectional, so adding one synonym automatically implies the reverse, making searches and classifications more flexible and inclusive. ## Use Case Examples - **Enhanced search and retrieval** — Concept relationships enable more intelligent searches. For example, a search for “Animal” can automatically include related hyponyms like “Dog” and “Cat,” returning broader and more relevant results. - **Improved data organization** — Hierarchical relationships help structure complex datasets more effectively. For example, hypernyms allow models to group specific entities under broader categories, improving organizational structures. - **Contextual understanding** — When models grasp the semantic links between concepts, they can better interpret context. For instance, recognizing that “Puppy” is a synonym of “Dog” ensures all relevant information is considered during classification or prediction. - **Dynamic content delivery** — In use cases like personalized content, search recommendations, or targeted advertising, concept relationships allow systems to infer user intent and deliver more relevant, meaningful results. :::tip [Click here](https://docs.clarifai.com/create-manage/search/api/rank#search-by-concepts) to learn how to leverage concept relations for more powerful and precise search queries. ::: ## How to Create Relations To create a relation between two concepts, you first have to create them in your app. You can see the [previous section](../concepts/create.md) on how to create concepts. Each relation should have a specified predicate, which can be _hyponym_, _hypernym_, or _synonym_. Below is an example of how to create a relation between two concepts. :::tip Learn how to create concept relations via the UI [here](https://docs.clarifai.com/create-manage/labeling/ui/create#update-annotations). ::: :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonSDKCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/sdk_knowledge_graph_create.py"; import PythonCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/knowledge_graph_create.py"; import PythonListRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/knowledge_graph_list.py"; import PythonDeleteRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/knowledge_graph_delete.py"; import PythonSearchRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/search_relations.py"; import PythonSearchSpecificRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/search_specific_relations.py"; import PythonSearchRelationsShowTree from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/search_relations-show-tree.py"; import JavaScriptCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/knowledge_graph_create.html"; import JavaScriptListRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/knowledge_graph_list.html"; import JavaScriptDeleteRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/knowledge_graph_delete.html"; import NodeJSCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/knowledge_graph_create.js"; import NodeJSListRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/knowledge_graph_list.js"; import NodeJSDeleteRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/knowledge_graph_delete.js"; import JavaCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/knowledge_graph_create.java"; import JavaListRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/knowledge_graph_list.java"; import JavaDeleteRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/knowledge_graph_delete.java"; import PHPCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/knowledge_graph_create.php"; import PHPListRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/knowledge_graph_list.php"; import PHPDeleteRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/knowledge_graph_delete.php"; import CurlCreateRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/knowledge_graph_create.sh"; import CurlListRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/knowledge_graph_list.sh"; import CurlDeleteRelations from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/knowledge_graph_delete.sh"; import JSONOutputExample1 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/knowledge_graph_create.js"; import JSONOutputExample2 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/knowledge_graph_list.js"; import JSONOutputExample3 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/knowledge_graph_delete.js"; {PythonSDKCreateRelations} {PythonCreateRelations} {JavaScriptCreateRelations} {NodeJSCreateRelations} {JavaCreateRelations} {PHPCreateRelations} {CurlCreateRelations} Raw Output Example {JSONOutputExample1} ## List Existing Relations ### List All Relations Below is an example of how to list all the existing relations between concepts. {PythonSearchRelations} {PythonListRelations} {JavaScriptListRelations} {NodeJSListRelations} {JavaListRelations} {PHPListRelations} {CurlListRelations} Raw Output Example {JSONOutputExample2} ### List Specific Concept Relations Below is an example of how to list the specific relations between concepts. {PythonSearchSpecificRelations} ### List Relations in Tree Structure You can set the `show_tree` argument to `True` in `search_concept_relations()` to display concept relationships in a rich, hierarchical tree structure. {PythonSearchRelationsShowTree} ## Delete Relations Below is an example of how to delete relations between concepts. :::tip You can use either of the following ways to retrieve the `CONCEPT_RELATION_IDS`: - Use the above [List Existing Relations](https://docs.clarifai.com/api-guide/concepts/knowledge_graph#list-existing-relations) method to list ALL existing relations between concepts. Remember to omit the `predicate` parameter. - Log in to the Portal and access the relations details of your concept. Then, inspect the network activity under your browser's Network Tab. The IDs are under the `relations` category. ::: {PythonDeleteRelations} {JavaScriptDeleteRelations} {NodeJSDeleteRelations} {JavaDeleteRelations} {PHPDeleteRelations} {CurlDeleteRelations} --- ## Create Concepts # Create Concepts **Learn how to create concepts within your app** ## **Create via the UI** To create a new concept, head to your application's individual page and select the **Inputs** option in the collapsible left sidebar. You'll be redirected to the Inputs-Manager page, where you can create new concepts and use them to complete various operations. ![](/img/community_2/inputs_viewer_page.png) There are several ways to create concepts on the platform. Let's illustrate two of them: - Via the **Labels** section - Via the inputs uploader ### Via the **Labels** Section To create a new concept, go to the **Labels** section on the Inputs-Manager page and click the plus sign (**+**) next to the **Select or add concepts** search box. Then, type the new concept name in the search box. The new name you've typed will appear underneath the search box. Click the **Add new label** button to create the concept. ![](/img/community_2/add_concept.png) The new concept will be successfully added to your app. You can follow the same process to create other concepts for your app. ![](/img/community_2/new_concept_added.png) ### Via the Inputs Uploader You can also create new concepts when uploading inputs to your app. To do so, click the **Upload Inputs** button at the upper-right corner of the Inputs-Manager page. > The window that pops up allows you to upload your inputs — either by uploading them directly from your local device or by providing a publicly accessible URL. In the **Concepts** section of the pop-up window, click the plus sign (**+**) next to the **Select or add concepts** search box. Then, type the new concept name in the search box. The new name you've typed will appear underneath the search box. Click the **Add new concept** button to create the concept. ![](/img/community_2/concepts_upload_inputs_new_concepts.png) The new concept will be successfully added to your app. > You can also click the **Upload inputs** button at the bottom of the pop-up window to finalize uploading your input. ![](/img/community_2/concepts_newly_added_concept.png) ## **Create via the API** :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonSDKCreateConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/sdk_create_concepts.py"; import PythonAddConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/add_concepts.py"; import JavaScriptAddConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/add_concepts.html"; import NodeJSAddConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/add_concepts.js"; import JavaAddConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/add_concepts.java"; import PHPAddConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/add_concepts.php"; import CurlAddConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/add_concepts.sh"; import JSONOutputExample1 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/add_concepts.js"; ### Create Concepts To create a new concept in your app, you POST the concept with an id and name. You can also post more than one concept in the same request by sending a list of concepts. Below is an example of how to add concepts. {PythonSDKCreateConcepts} {PythonAddConcepts} {JavaScriptAddConcepts} {NodeJSAddConcepts} {JavaAddConcepts} {PHPAddConcepts} {CurlAddConcepts} Raw Output Example {JSONOutputExample1} --- ## Manage Concepts # Manage Concepts **Learn how to get, update, and delete concepts** :::warning Delete Concepts We currently do not support deleting concepts solitarily since they have such an integral tie across almost all other data structures in the platform, like inputs, models, searches, etc. ::: ## **Via the API** :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: Within your app, you can retrieve and modify concepts after they've been created. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonGetConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/get_concept.py"; import PythonListConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/list_concepts.py"; import PythonListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/list_model_concepts.py"; import PythonUpdateConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/python/update_concept.py"; import JavaScriptGetConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/get_concept.html"; import JavaScriptListConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/list_concepts.html"; import JavaScriptListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/list_model_concepts.html"; import JavaScriptUpdateConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/js/update_concept.html"; import NodeJSGetConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/get_concept.js"; import NodeJSListConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/list_concepts.js"; import NodeJSUpdateConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/node/update_concept.js"; import JavaGetConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/get_concept.java"; import JavaListConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/list_concepts.java"; import JavaUpdateConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/java/update_concept.java"; import PHPGetConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/get_concept.php"; import PHPListConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/list_concepts.php"; import PHPListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/list_model_concepts.php"; import PHPUpdateConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/php/update_concept.php"; import CurlGetConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/get_concept.sh"; import CurlListConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/list_concepts.sh"; import CurlListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/list_model_concepts.sh"; import CurlUpdateConcept from "!!raw-loader!../../../code_snippets/api-guide/concepts/curl/update_concept.sh"; import JSONOutputExample2 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/get_concept.js"; import JSONOutputExample3 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/list_concepts.js"; import JSONOutputExample4 from "!!raw-loader!../../../code_snippets/api-guide/concepts/code_output_examples/update_concept.js"; ### Get #### Get Concept by ID Below is an example of how to get a single concept by its ID. {PythonGetConcept} {JavaScriptGetConcept} {NodeJSGetConcept} {JavaGetConcept} {PHPGetConcept} {CurlGetConcept} Raw Output Example {JSONOutputExample2} #### List Concepts You can get a list of concepts within your app with a GET call. This call supports [pagination](https://docs.clarifai.com/api-guide/advanced-topics/pagination/). Below is an example of how to list concepts. {PythonListConcepts} {JavaScriptListConcepts} {NodeJSListConcepts} {JavaListConcepts} {PHPListConcepts} {CurlListConcepts} Raw Output Example {JSONOutputExample3} #### List Model Concepts You can get a list of concepts within your model with a GET call. This call supports [pagination](https://docs.clarifai.com/api-guide/advanced-topics/pagination/). Below is an example of how to list the concepts in your model. :::note If you are using any of the Clarifai gRPC clients, the `ListModelConcepts` endpoint, which lists concepts in a model, is only available from release 8.10.0. ::: {PythonListModelConcepts} {JavaScriptListModelConcepts} {PHPListModelConcepts} {CurlListModelConcepts} ### Update #### Update Concept Name Below is an example of how to update a concept's name given its id by using the "overwrite" action. You can also patch multiple concepts by sending a list of concepts. {PythonUpdateConcept} {JavaScriptUpdateConcept} {NodeJSUpdateConcept} {JavaUpdateConcept} {PHPUpdateConcept} {CurlUpdateConcept} Raw Output Example {JSONOutputExample4} --- ## Datasets # Datasets **Manage the datasets for training, testing, and evaluating your models** A dataset is a structured collection of data that serves as the foundation for working with machine learning models. With Clarifai datasets, you can manage the inputs you want to use for visual search, model training, evaluation, and more. Effective dataset management is essential to ensure data quality, consistency, and accessibility throughout the entire development lifecycle, from initial data collection to model deployment. Clarifai’s robust platform empowers you to simplify and optimize interactions with datasets, enabling seamless organization, annotation, and utilization of your data for AI applications. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Datasets Creation # Datasets Creation **Learn how to create datasets and their versions** ## **Create via the UI** ### Create a New Dataset To create a new dataset, head to the individual page of your application. Then, select the **Datasets** option in the collapsible left sidebar. You'll be redirected to the **Datasets** manager page, where you can create new datasets and view already created ones. Click the **Create Dataset** button in the upper-right corner of the page. ![](/img/community/datasets/dataset_1.png) On the **New Dataset** page, provide an ID and, optionally, a short description of the dataset. Then, click the **Create** button. ![](/img/community/datasets/dataset_2.png) You'll be redirected to the created dataset's page, where you can accomplish various tasks. Let's talk about them. ![](/img/community/datasets/dataset_3.png) #### View in Leaderboard To view the dataset in the Leaderboard, click the **View in Leaderboard** button in the upper section of the page. You'll be redirected to the **Leaderboard** page, where you can assess how the dataset performs when used to evaluate a model. [Click here](https://docs.clarifai.com/portal-guide/evaluate/leaderboard/#evaluation-dataset--version) to learn how the leaderboard feature works. #### Edit Visibility To change the visibility of the dataset, whether from private to public or vice versa, click the **Edit Visibility** button located in the upper section of the page. A pop-up window will appear, allowing you to modify the dataset's visibility settings. ![](/img/community/datasets/dataset_3-1.png) :::note If you make a specific version of a dataset public, your user ID and app ID will also be set to public. ::: After editing the settings, click the **Confirm** button. #### Upload Inputs To add inputs to the dataset, click the **Upload Inputs** button in the upper section of the page. You'll be redirected to the Inputs-Manager page, where you can upload inputs to the dataset. [Click here](https://docs.clarifai.com/portal-guide/data/#upload-inputs) to learn how to upload inputs to our platform. #### Create Labeling Task To create a labeling task, click the **Create Labeling Task** button in the upper section of the page. You'll be redirected to the **New Labeling Task** page, where you can create a new labeling task to label the inputs in your dataset either manually or automatically. [Click here](https://docs.clarifai.com/portal-guide/annotate/create-a-task) to learn how to create a labeling task. #### Train Model To train a model, click the **Train Model** button in the upper section of the page. You'll be redirected to a page where you can create a new custom model for your use case. [Click here](https://docs.clarifai.com/portal-guide/model/pcustom-model-walkthrough) to learn how to create and train a model. ### Create Dataset Version A dataset can change over time for various reasons, such as by adding or removing inputs. With dataset versioning, you can assign a unique identifier to a specific version of a dataset. A dataset version can help you achieve many things, such as: - Refer to a specific dataset version and recreate the same results. This can help you to have a clear reference to what data was used at a particular point in time. - Ensure everyone in your team is working on the same dataset. This reduces confusion and errors, and leads to accurate results. - Track the changes you've made to a dataset over time. This can help you to determine whether you're improving the quality and quantity of your dataset. After adding inputs to a dataset, you can create a version that bookmarks the state of your data so that you can apply a specific version of the dataset for future iterations. To create a new dataset version, go to the individual page of your created dataset and click the **New version** button. ![](/img/community/datasets/dataset_10.png) Finally, click the **Update status** button. ![](/img/community/datasets/dataset_11.png) The total number of inputs and their respective annotations are displayed in the **Overview** tab. :::note If you click the **Explore Inputs** button, you'll be redirected to the Inputs-Manager page, where all the inputs in your dataset are displayed. ::: ![](/img/community/datasets/dataset_12.png) The versions of the datasets you've created are displayed in the **Versions** tab. ![](/img/community/datasets/dataset_13.png) As you can see in the screenshot above, the **Versions** tab has a chart that displays the total number of inputs in your dataset over time, with data plotted against specific dates. By default, it shows the annotation metrics based on the dataset version, with each annotation type represented by a distinct color. This makes it easy to track and compare trends across different dates. You can switch the chart to display metrics by input type by clicking the **by inputs** button in the upper-right corner. Each type of input is marked with its own color. ![](/img/community/datasets/dataset_13_1.png) If you hover over the chart, a tooltip is activated that provides detailed information for that specific date, including the exact count of the type of annotation or type of input. The hovered item is highlighted in the tooltip, which allows for quick identification. Additionally, a table is provided below the chart. It lists each dataset version alongside the creation date, description, input count, and annotation count. The table includes a search function for easy lookup and allows the sorting of columns for streamlined navigation. #### Auto-Generated Versions As you navigate through the **Versions** tab, you might come across auto-generated dataset versions. These are some cases where **"auto-generated-*"** dataset versions could be created: - If you train a model and only select a dataset, but not a corresponding dataset version. - During the model evaluation process, the **auto-generated** dataset versions are used to store the different ground truths and predictions, which are then used to further calculate the actual evaluation metrics. ![](/img/community/datasets/autogen_version.png) ## **Create via the API** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeCreateDataset from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/create_dataset.py"; import CodeCreateDatasetTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/creatingDatasets.ts"; import CodeCreateDatasetV from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/create_data_version.py"; import CodeCreateDatasetVTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/createDatasetVersion.ts"; import curlCreateDataset from "!!raw-loader!!../../../code_snippets/api-guide/data/datasets/create_dataset.sh"; import curlAddDatasetVersion from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/add_dataset_version.sh"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ### Create a Dataset You can create a dataset by specifying a unique dataset ID. {CodeCreateDataset} {CodeCreateDatasetTS} {curlCreateDataset} ### Create a Dataset Version After making changes to a dataset, such as adding new inputs, you can create a new version to reflect those updates, [as previously explained](#create-dataset-version). {CodeCreateDatasetV} {CodeCreateDatasetVTS} {curlAddDatasetVersion} --- ## Data Utils # Data Utils **Learn how to use the Clarifai Data Utils library** [Data Utils](https://github.com/Clarifai/clarifai-python-datautils) is an open-source Clarifai library that provides a powerful suite of multimedia data utilities to simplify and accelerate your data management and processing activities. It addresses several key challenges associated with data preparation, fostering a more efficient and professional workflow. Additionally, this library seamlessly integrates with the Python SDK, empowering you to unlock AI-driven solutions for diverse use cases. With Data Utils, you can effortlessly extract, transform, and load unstructured data — such as images, videos, and text — into the Clarifai platform. Once uploaded, you can leverage the data for various purposes within the Clarifai platform, such as training a custom image classification model to identify objects in images. The library offers two key features: - **[Image Annotation Loader](https://docs.clarifai.com/sdk/data-utils/image-annotation-loader)** — This framework enables you to load various annotated image datasets and upload them directly to the Clarifai platform. It also supports converting between different annotation formats, ensuring compatibility and flexibility for your projects. - **[Data Ingestion Pipelines](https://docs.clarifai.com/sdk/data-utils/ingestion-pipelines)** — This framework provides robust pipelines to pre-process images and text documents (such as PDFs, HTML, and Word docs), transform and chunk the content, and seamlessly ingest them into the Clarifai platform for further processing and analysis. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Image Annotation Loader # Image Annotation Loader **Load existing annotated image datasets and convert between supported formats** The [Image Annotation Loader](https://github.com/Clarifai/examples/blob/main/Data_Utils/Image%20Annotation/image_annotation_loader.ipynb) framework, part of the Data Utils library, enables you to load already annotated image datasets and upload them to the Clarifai platform. This framework eliminates the hurdle of format incompatibility by supporting a wide range of industry-standard annotation formats; you can also use it to convert between different annotation formats. This allows seamless integration of existing labeled datasets, regardless of the initial annotation tool used. It also facilitates a smooth upload process, enabling you to leverage the Clarifai platform's infrastructure for various use cases. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CreateDataset from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/create_dataset.py"; import GetDatasetInfo from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/get_dataset_info.py"; import UploadToClarifai from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/upload_to_clarifai.py"; import DownloadFromKaggle from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/download_from_kaggle.py"; import UploadFromKaggle from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/upload_from_kaggle.py"; import ConvertDataset from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/convert_dataset.py"; import ExportClarifai1 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/export_clarifai_1.py"; import ExportClarifai2 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/export_clarifai_2.py"; import SupportedFormats from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/supported_formats.py"; import FormatDetection from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/format_detection.py"; import Example1 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/example_1.txt"; import Example2 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/example_2.txt"; import Example3 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/example_3.txt"; :::tip Run the following command to clone the repository containing various examples for using the Data Utils library: `git clone https://github.com/Clarifai/examples.git`. After cloning, navigate to the **Data_Utils** folder to follow along with this tutorial. :::: ## Prerequisites ### Install Python SDK and Data Utils Install the latest version of the `clarifai` Python SDK package. Also, install the Data Utils library. ```bash pip install --upgrade clarifai pip install clarifai-datautils ``` ### Install Extra Dependencies The Image Annotation Loader requires additional libraries to function properly. To keep the core library lightweight, we moved these optional dependencies under `annotations`. _(Python extras allow projects to specify additional dependencies for optional functionality.)_ To install them, run: pip install 'clarifai-datautils[annotations]' The above command also installs [Datumaro](https://github.com/openvinotoolkit/datumaro), a dataset management framework essential for the Image Annotation Loader. Note that Datumaro requires a [Rust compiler](https://www.rust-lang.org/) to be installed on your machine for a smooth installation. ### Get a PAT You need a PAT (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can generate it in your Personal Settings page by navigating to the [Security section](https://clarifai.com/settings/security). Then, set it as an environment variable in your script. ```bash import os os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # replace with your own PAT key ``` ### Create a Dataset [Create a dataset](https://docs.clarifai.com/sdk/managing-datasets#creating-datasets) on the Clarifai platform to use for uploading your annotated image datasets. {CreateDataset} ## Utility Features​ ### Supported Formats You can retrieve and display all the annotation formats that the Image Annotation Loader framework supports. {SupportedFormats} Note that: - The `ImageAnnotations` class is imported from the `clarifai_datautils.image` package. This class provides utilities for working with annotated image datasets. Output Example {Example2} Here is a table that illustrates the annotation formats that the framework supports. | Annotation Type | Format | TASK | |------------------------|--------------------|--------------| | [ImageNet](http://image-net.org/) | imagenet | classification | | [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) | cifar | classification | | [MNIST](http://yann.lecun.com/exdb/mnist/) | mnist | classification | | [VGGFace2](https://github.com/ox-vgg/vgg_face2) | vgg_face2 | classification | | [LFW](http://vis-www.cs.umass.edu/lfw/) | lfw | classification | | [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/index.html) | voc_detection | detection | | [YOLO](https://github.com/AlexeyAB/darknet#how-to-train-pascal-voc-data) | yolo | detection | | [COCO](http://cocodataset.org/#format-data) | coco_detection | detection | | [CVAT](https://opencv.github.io/cvat/docs/manual/advanced/xml_format/) | cvat | detection | | [Kitti](http://www.cvlibs.net/datasets/kitti/index.php) | kitti | detection | | [LabelMe](http://labelme.csail.mit.edu/Release3.0) | label_me | detection | | [Open Images](https://storage.googleapis.com/openimages/web/download.html) | open_images | detection | | [Clarifai](https://github.com/Clarifai/examples/tree/main/Data_Utils) | clarifai | detection | | [COCO(segmentation)](http://cocodataset.org/#format-data) | coco_segmentation | segmentation | | [Cityscapes](https://www.cityscapes-dataset.com/) | cityscapes | segmentation | | [ADE](https://ade20k.csail.mit.edu/) | ade20k2017 | segmentation | ### Format Detection​ You can identify the annotation format that a dataset uses. {FormatDetection} Note that: - The `LOCAL_FOLDER_PATH` parameter specifies the local directory path where the annotated dataset is stored. Output Example {Example3} ### Dataset Information​ You can get the details of a dataset you want to upload to the Clarifai platform. {GetDatasetInfo} Note that: - The `import_from` method of the `ImageAnnotations` class is used to load the dataset from the specified local folder. - The `format` parameter specifies the format of the annotations. You can specify any supported annotation type. Output Example {Example1} ## Uploading to Clarifai To upload a pre-labeled dataset from your local environment to the Clarifai platform, you need to initialize the `Dataset` object and specify where the dataset will be uploaded — using the Python SDK library. Then, call the `upload_dataset()` method on the `Dataset` object. This method takes a `dataloader` as an argument, which iterates over the dataset and yield data in a format compatible with the Clarifai platform. {UploadToClarifai} ## Uploading From Kaggle to Clarifai You can download a dataset from [Kaggle](https://www.kaggle.com/datasets) and upload it to the Clarifai platform. To begin, install the `opendatasets` Python package, which enables direct dataset downloads from Kaggle. pip install -q opendatasets Next, download the dataset from Kaggle. For example, here is how you could download this [dogs-vs-wolves](https://www.kaggle.com/datasets/harishvutukuri/dogs-vs-wolves) dataset. {DownloadFromKaggle} Then, you can upload it to Clarifai. {UploadFromKaggle} ## Convert Between Supported Formats You can convert datasets between various annotation formats in your local development environment. For example, you can convert a dataset from COCO format to VOC format. {ConvertDataset} ## Export a Clarifai Dataset to Another Format You can export a dataset version from the Clarifai platform and convert it into various formats. This process involves two simple steps. First, use the Clarifai SDK to export the dataset from the platform. The dataset will be downloaded as a ZIP file to your specified local directory. If the directory does not already exist, it will be automatically created for you. {ExportClarifai1} Next, extract the contents of the ZIP file to a folder. Then, pass the folder path to `ImageAnnotations` and convert the dataset into your desired format. {ExportClarifai2} --- ## Data Ingestion Pipelines # Data Ingestion Pipelines **Pre-process and ingest diverse data formats, including images and text-based documents** The [Data Ingestion Pipelines](https://github.com/Clarifai/examples/tree/main/Data_Utils/Ingestion%20pipelines) framework, part of the Data Utils library, offers a comprehensive suite of robust functions — commonly referred to as pipelines — designed to pre-process, transform, and prepare images and text documents for seamless ingestion into the Clarifai platform. These ready-to-use pipelines enable efficient processing of unstructured data, including partitioning, chunking, cleaning, and extracting valuable information, ensuring the data is optimized for downstream use cases such as [Retrieval Augmented Generation](https://docs.clarifai.com/portal-guide/agent-system-operators/rag-prompter) (RAG). Leveraging the capabilities of the open-source [Unstructured](https://docs.unstructured.io/welcome) library, this framework is designed to streamline data processing workflows, making it an essential tool for working with Large Language Models (LLMs) and other AI-driven applications. It supports these file formats: - PDF - Text (.txt) - Docx - Markdown import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CreateDataset from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/create_dataset.py"; import PDFPartition from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/pdf_partition.py"; import PDFPartitionMultimodal from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/pdf_partition_multimodal.py"; import BuildingPipelines from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/building_pipelines.py"; import TextPartition from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/text_partition.py"; import DocxPartition from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/docx_partition.py"; import MarkdownPartition from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/markdown_partition.py"; import CleanExtraWhitespace from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_extra_whitespace.py"; import ReplaceUnicodeQuotes from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/replace_unicode_quotes.py"; import CleanDashes from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_dashes.py"; import CleanBullets from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_bullets.py"; import GroupBrokenParagraphs from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/group_broken_paragraphs.py"; import RemovePunctuation from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/remove_punctuation.py"; import ConvertByteStrings from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/convert_byte_strings.py"; import CleanNonASCII from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_non_ascii.py"; import CleanOrderedBullets from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_ordered_bullets.py"; import CleanPrefix from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_prefix.py"; import CleanPostfix from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/clean_postfix.py"; import Datetime from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/datetime.py"; import ExtractEmail from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/extract_email.py"; import ExtractIP from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/extract_ip.py"; import ExtractIPName from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/extract_ip_name.py"; import ExtractTextAfter from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/extract_text_after.py"; import ExtractTextBefore from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/extract_text_before.py"; import ImageSummarization from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/image_summarization.py"; import Example1 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/ingestion_example_1.txt"; import Example2 from "!!raw-loader!../../../../code_snippets/python-sdk/datautils/building_pipelines_example.txt"; ## Prerequisites ### Install Python SDK and Data Utils Install the latest version of the `clarifai` Python SDK package. Also, install the Data Utils library. pip install --upgrade clarifai pip install clarifai-datautils ### Install Extra Dependencies The Data Ingestion Pipelines framework requires additional libraries to function properly. First, create a `requirements-dev.txt` file and add the following dependencies: unstructured[pdf] @ git+https://github.com/clarifai/unstructured.git@support_clarifai_model llama-index-core==0.10.33 llama-index-llms-clarifai==0.1.2 pi_heif==0.18.0 markdown==3.7 python-docx==1.1.2 schema==0.7.5 > _Note that this command `pip install unstructured[pdf] @ git+https://github.com/clarifai/unstructured.git@support_clarifai_model` installs the `support_clarifai_model` branch from the [Clarifai fork of `unstructured`library](https://github.com/Clarifai/unstructured/tree/support_clarifai_model)._ Then, run the following command to install the required dependencies: pip install -r requirements-dev.txt You can also install the following system dependencies if they are not already available on your system. Based on the document types you're handling, you may not need all of them. - [`opencv-python-headless`](https://pypi.org/project/opencv-python-headless/) — A lightweight version of OpenCV (Open Source Computer Vision Library) designed for environments where GUI functionalities (such as image or video display) are not needed. You can install it by running: `pip install opencv-python-headless`. - [`poppler-utils`](https://pypi.org/project/poppler-utils/) — Essential for processing and extracting data from PDF files. You can install it by running: `sudo apt update && sudo apt install poppler-utils`. - [`tesseract-ocr`](https://github.com/tesseract-ocr/tessdoc) — Required for performing OCR on images or scanned documents to extract text. You can install it by running: `sudo apt update && sudo apt install tesseract-ocr`. - [`libgl1-mesa-glx`](https://mesa3d.org/) — Ensures compatibility with graphical operations, which may be required by certain libraries (e.g., OpenCV) even in headless environments. You can install it by running: `sudo apt update && sudo apt install libgl1-mesa-glx`. - [`punkt_tab`](https://www.nltk.org/api/nltk.tokenize.html) — Enables tokenization of text data with tab-separated values; it's part of the NLTK library. You can install it by running: `nltk.download('punkt_tab')`. - [`averaged_perceptron_tagger_eng`](https://www.nltk.org/api/nltk.tag.html) — Provides a pre-trained model for accurate part-of-speech tagging in English; it's part of the NLTK library. You can install it by running: `nltk.download('averaged_perceptron_tagger_eng')`. ### Get a PAT You need a PAT (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can generate it in your Personal Settings page by navigating to the [Security section](https://clarifai.com/settings/security). Then, set it as an environment variable in your script. import os os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # replace with your own PAT key ### Create a Dataset [Create a dataset](https://docs.clarifai.com/sdk/managing-datasets#creating-datasets) on the Clarifai platform to use for uploading your pre-processed data. {CreateDataset} ## Building Pipelines When working with unstructured documents like PDFs, building pipelines is a crucial step to automate the processing and transformation of data. Here is an example of a basic pipeline for PDF partitioning. {BuildingPipelines} Output Example {Example2} Note that: - `Pipeline` and `PDFPartition` classes are imported from `clarifai_datautils.multimodal`. These are used to define and execute processing pipelines for PDF documents. - A `Pipeline` object is created with the name `"basic_pdf"`. You can provide any arbitrary name for the pipeline. The name can be used to identify or call the pipeline. - `PDFPartition()` uses [default parameters](https://github.com/Clarifai/clarifai-python-datautils/blob/main/clarifai_datautils/constants/pipeline.py) (such as `max_characters=500`) for ingesting PDFs. - After loading a predefined pipeline, you can view its details. ## Partitioning & Chunking [Partitioning](https://docs.unstructured.io/open-source/core-functionality/partitioning) is the first step in document processing. It breaks down a raw, unstructured document into smaller, meaningful units called document elements, while preserving the document’s semantic structure. These elements — such as paragraphs, titles, tables, and images — help maintain the original context. The process involves reading the document, segmenting it into sections, categorizing those sections, and extracting the relevant text. Chunking follows partitioning and involves grouping or rearranging document elements generated by partitioning into "chunks" based on specific size constraints or criteria. This step ensures that the resulting segments are optimized for use cases like search, summarization, or content retrieval. :::info Once a chunk of text or image data is uploaded to the Clarifai platform, [metadata fields](https://docs.clarifai.com/portal-guide/input-viewer/) — such as `filename`, `page_number`, `orig_elements`, and `type` — are automatically added to provide detailed information about the uploaded input. Example ![](/img/others/metadata-datautils.png) ::: ### PDF Partitioning PDF partitioning helps transform PDFs into a structured format that can be used for further processing. {PDFPartition} Output Example {Example1} Note that: - The transformation step uses the `PDFPartition` object to partition the PDF into smaller chunks. - `chunking_strategy` is set to `"by_title"`, meaning the document is split based on its title sections. - `max_characters` limits each chunk to 1024 characters for better processing and retrieval efficiency. The default behavior is 500 characters. - The `loader=False`, which is the default, argument ensures the transformed chunks are returned as Python objects (`elements`), allowing for local inspection or further processing. Conversely, setting `loader=True` directly ingests the transformed chunks into a Clarifai dataset instead of just returning them locally. - The partitioned and chunked PDF elements are uploaded to a Clarifai dataset. The uploaded data is automatically [annotated](https://docs.clarifai.com/portal-guide/input-viewer/) with the pipeline name on the Clarifai platform. This makes it easy to identify and distinguish between data processed through different pipelines. :::note tips You can also configure the following arguments for the `PDFPartition` object: - Set `chunking_strategy="basic"` for the document to be chunked purely based on character length and sequential order rather than structural elements like section titles or page boundaries. It's useful when you simply want to group text into evenly sized chunks without preserving the document’s logical structure. - Set `ocr=True` to enable OCR for extracting text from scanned or image-based PDFs. Set it to `False`, which is the default, to disable OCR. - By default, `overlap=None` or `overlap=0` ensures no overlap between chunks; that is, chunks are created without any shared text between them. To enable overlap, provide an integer value (e.g., `overlap=100`) to specify the number of overlapping characters between consecutive chunks. - Set `overlap_all=True` to enable overlapping across all chunks. Set it to `False`, which is the default, to disable this behavior. - Set `strategy="ocr_only"` to force the document to be processed using the Tesseract OCR strategy. If Tesseract is unavailable and the document contains extractable text, it falls back to the `"fast"` strategy. Set `strategy="fast"` to extract text using `pdfminer`, which is faster and suitable for text-based PDFs. Otherwise, `strategy="auto"` is the default that selects the partitioning strategy based on document characteristics and the function kwargs. - Use `clarifai_ocr_model` to set the URL of a Clarifai OCR model for processing the document. The default is `None`. ::: ### PDF Partitioning Multimodal The `PDFPartitionMultimodal` ingestion pipeline supports multimodal scenarios, where files containing a mix of text, images, and other elements are to be processed and ingested into the Clarifai platform. We use the Clarifai-hosted YOLOX object detection model to process the PDFs containing embedded images. {PDFPartitionMultimodal} Note that: - The`PDFPartitionMultimodal` supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. :::note tips You can also configure the following arguments for the `PDFPartitionMultimodal` object: - By default, `extract_images_in_pdf=True` extracts images from a PDF file. In that case, the partitioning strategy is set as `strategy="hi_res"`, which is intended to identify the layout of the document and gain additional information about the document elements. Otherwise, set `extract_images_in_pdf=False` to disable this behavior. - Set `extract_image_block_types=["Image"]` to specify that you want to extract a list of image block types. - Set `extract_image_block_to_payload=True` to allow for the conversion of extracted images from a PDF into base64 format (return images as bytes). Note that to use this feature, you must set the `strategy` parameter to `hi_res` and `extract_images_in_pdf` to `True`. Otherwise, set `extract_image_block_to_payload=False` to disable this behavior. ::: ### Text Partitioning Text partitioning transforms unstructured `.txt` documents into text elements, making them easier to process, analyze, and utilize in downstream applications. {TextPartition} Note that: - The `TextPartition` object supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. ### Docx Partitioning Docx partitioning processes `.docx` files, extracting and partitioning their contents into structured text elements. {DocxPartition} Note that: - The `DocxPartition` object supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. ### Markdown Partitioning Markdown partitioning processes `.md` files, breaking them down into structured text elements for improved usability in downstream applications. {MarkdownPartition} Note that: - The `MarkdownPartition` object supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. ## Image Summarization The Image Summarizer pipeline enables you to utilize a Clarifai's multimodal-to-text model to generate text summaries for the uploaded image data. Each summary is stored as an individual input on the Clarifai platform, and you can view its metadata field to see the source image it’s associated with. The generated summaries are concise, optimized for retrieval, and enriched with relevant keywords, making them highly effective for search and indexing. {ImageSummarization} Example ![](/img/others/summarizer_datautils.png) ## Text Cleaning The Data Ingestion Pipelines framework allows you to prepare and refine raw text data by removing or correcting unwanted elements to improve readability, consistency, and usability for downstream applications. :::note The following examples use the `PDFPartition` object, but they can also be applied to any other supported partitioning objects. ::: ### Clean Extra Whitespaces You can remove unnecessary spaces, tabs, or newlines from documents. {CleanExtraWhitespace} ### Replace Unicode Quotes You can replace Unicode quotes with ASCII quotes for standardization. {ReplaceUnicodeQuotes} ### Clean Dashes You can remove unnecessary dashes from texts. {CleanDashes} ### Clean Bullets You can remove unnecessary bullets from texts. {CleanBullets} ### Group Broken Paragraphs You can merge fragmented paragraphs that were unintentionally split, restoring proper text flow and improving readability. {GroupBrokenParagraphs} ### Remove Punctuations You can remove unnecessary punctuations from texts. {RemovePunctuation} ### Convert Byte Strings You can convert a byte string (such as `b'hello'`) into a regular string (`'hello'`), ensuring proper text formatting and usability. {ConvertByteStrings} ### Clean Non-ASCII Characters You can remove non-ASCII characters from text, ensuring compatibility with systems that only support standard ASCII encoding. {CleanNonASCII} ### Clean Ordered Bullets You can remove ordered bullet points(such as `1.`, `2)`, or `III.`) from text. {CleanOrderedBullets} ### Clean Prefix You can remove a specified prefix from a document. The `Clean_prefix` object supports the following arguments: - `pattern` — Defines the prefix to remove. The pattern must be provided, and it can be a simple string or a regex pattern. - `ignore_case` (optional, default is `False`) — Determines whether to ignore case. If `True`, ensures case-insensitive matching. - `strip` (optional, default is `True`) — If `True`, removes any leading whitespace after the prefix is removed. {CleanPrefix} ### Clean Postfix You can remove a specified postfix from a document using the `Clean_postfix` object, which supports the same arguments as `Clean_prefix`. {CleanPostfix} ## Text Extraction The Data Ingestion Pipelines framework allows you to identify and retrieve meaningful texts from documents. ### Extract Email Addresses You can extract email addresses from texts. Note that if a chunk contains the addresses, they will be extracted and stored in the `email_address` metadata field of the uploaded input on the Clarifai platform, [as previously mentioned](#partitioning--chunking). {ExtractEmail} ### Datetime With Time Zones You can extract datetime values with time zones from texts, ensuring accurate timestamp retrieval. Note that if a chunk contains the values, they will be extracted and stored in the `date_time` metadata field of the uploaded input on the Clarifai platform. {Datetime} ### Extract IP Addresses You can extract IP addresses from texts. Note that if a chunk contains the addresses, they will be extracted and stored in the `ip_address` metadata field of the uploaded input on the Clarifai platform. {ExtractIP} ### Extract IP Addresses Names You can extract IP addresses along with associated names from texts. Note that if a chunk contains the names, they will be extracted and stored in the `ip_address_name` metadata field of the uploaded input on the Clarifai platform. {ExtractIPName} ### Extract Text After You can extract text appearing after a specified string in a given text input. The `ExtractTextAfter` object supports the following string arguments: - `key` — Key to store the extracted text in the metadata field of the uploaded input on the Clarifai platform. - `string` — The reference string after which the text will be extracted. {ExtractTextAfter} ### Extract Text Before You can extract text appearing before a specified string in a given text input. The `ExtractTextBefore` object supports the following string arguments: - `key` — Key to store the extracted text in the metadata field of the uploaded input on the Clarifai platform. - `string` — The reference string before which the text will be extracted. {ExtractTextBefore} --- ## Dataset Annotation Filters # Dataset Annotation Filters **Use filter and search functionality over your datasets** The dataset annotation filters do not affect the inputs that belong to a dataset. Inputs are explicitly added or removed from datasets. Rather, they affect the annotations (for the inputs in the dataset) that are used whenever the dataset is converted into a dataset version and used for training or evaluation. We calculate the metrics for the dataset version whenever it is created (each time a dataset is used for training or evaluation). Therefore, if a user wants to train a model on the exact same set of inputs, but by using annotations created by person A vs. the ones created by person B, they would update the dataset annotation filter for the same dataset to create two different dataset versions (both containing the same inputs, but one with person A's annotations and the other with person B's). Comparing the performance of those two models is a common technique for assessing the quality of the two labelers' annotations. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import curlAddDatasetFilters from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/add_dataset_filters.sh"; import curlListDatasetFilters from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/list_dataset_filters.sh"; import curlGetDatasetFilter from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/get_dataset_filter.sh"; import curlChangeDatasetFilter from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/change_dataset_filter.sh"; import curlDeleteDatasetFilter from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/delete_dataset_filter.sh"; ## Add Dataset Annotation Filters {curlAddDatasetFilters} ## List Dataset Annotation Filters {curlListDatasetFilters} ## Get a Dataset Annotation Filter {curlGetDatasetFilter} ## Change a Dataset Annotation Filter {curlChangeDatasetFilter} ## Delete a Dataset Annotation Filter {curlDeleteDatasetFilter} --- ## Datasets Management # Datasets Management **Use vector search to sort, rank, and retrieve images** ## **Via the UI** ### Get Datasets #### List of Datasets To get a list of datasets, go to the individual page of your application. Then, select the **Datasets** option in the collapsible left sidebar. You'll be redirected to the **Datasets** manager page, where you can get the already created datasets in your application. ![](/img/community/datasets/dataset_7.png) #### Export a Dataset You can export the inputs in your dataset, alongside their annotations, to an external storage system. To do so, start by clicking the icon at the extreme end of a dataset field to select the format you want to use for exporting your dataset. From the list that drops down, you can select any of the following data formats: - **Clarifai-Data-Protobuf**, which is the default — This is a [protocol buffer](https://protobuf.dev/) (protobuf) format used by Clarifai to structure data for machine learning tasks. Protocol buffers are a language-agnostic, platform-neutral mechanism for serializing structured data. - **Clarifai-Data-JSON** — This is a JSON (JavaScript Object Notation) format used by Clarifai to structure data. JSON is a text-based, lightweight data-interchange format that's easy to read and write. - **COCO** — This is the [COCO](https://cocodataset.org/#format-data) (Common Objects in Context) format used by Clarifai to structure data. COCO is a large-scale, popular dataset used in machine learning and computer vision tasks. After selecting your preferred export format, click the **Generate** button. Once the export file has been processed, the **Generate** button will become a **Download** button, which you can click to download your dataset. ![](/img/community/datasets/dataset_8.png) :::note The export feature only works after adding inputs to a dataset and creating and selecting a dataset version. Learn how to create dataset version [here](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete#create-dataset-version). ::: #### Dataset ID or Version ID To copy a dataset ID to the clipboard, go to its individual page and click the copy button next to the dataset's ID. To copy a dataset version ID to the clipboard, click the copy button next to the **Selected Version** search field. ![](/img/community/datasets/dataset_9.png) ### Update Datasets #### Update a Dataset Version After making some changes to your dataset — such as adding or removing inputs, or adding or removing annotations — you may want to update your dataset version to reflect the changes. To update a dataset version, go to the individual page of the dataset and select the **Refresh Metrics** option that drops down after clicking the ellipsis in the upper-right corner of the page. ![](/img/community/datasets/dataset_14.png) Finally, click the **Update status** button. ![](/img/community/datasets/dataset_15.png) The updated inputs and annotations in your dataset will be displayed in the **Overview** tab. ![](/img/community/datasets/dataset_16.png) You can also choose the dataset version you'd like to use from the **Selected Version** drop-down list. #### Update Cover Image To update a dataset's cover image, click the **Change cover image** button. A window will appear that allows you to upload an image for the dataset. ![](/img/community/datasets/dataset_16_1.png) ### Merge Datasets You can merge datasets by transferring inputs and their annotations from a source dataset to a destination dataset. Note that this process does not remove the inputs from the source dataset; they remain intact while being duplicated to the destination. Start by selecting the **Inputs** option in your app's collapsible left sidebar. You'll be redirected to the Inputs-Manager page, where the inputs in your app are displayed. Next, navigate to the **Datasets** section and select the dataset from which you want to transfer inputs. Once selected, all the available inputs in the dataset will be displayed on the page. To choose the inputs for transfer, hover over each of them and click the small empty box in the upper-left corner to select them. :::tip multi-select feature - **Mouse click**: Selects a single item or input. - **Shift + mouse click**: Selects a range of inputs between the first and last clicked item. ::: Next, click the **Dataset...** button that appears at the bottom section of the page. ![](/img/community/datasets/dataset_18.png) The small window that pops up allows you to add or remove inputs from the selected datasets. ![](/img/community/datasets/dataset_19.png) Select the **Add** option, which lets you add inputs to the destination dataset (the option is selected by default). Then, select the destination dataset from the **Select Datasets** search field. :::tip If you select the **Apply to all search results** button, all the inputs that are visually similar to the one(s) you've initially selected will also be added. This allows you to merge datasets easily and fast. ::: If you want to create a new destination dataset: ![](/img/community/datasets/dataset_20.png) - Click the plus sign (**+**) next to the **Select Datasets** search field. - Type the new dataset name in the search field. The new name you've typed will appear underneath the search field. - Click the **Add new dataset** button to create the dataset. The new dataset will be successfully added to your app and selected as a destination. Finally, click the **Add Inputs** button at the bottom of the pop-up window to complete adding the selected inputs to the destination dataset. Alternatively, you can remove inputs from a dataset by selecting the **Remove** option, selecting the desired dataset, and clicking the **Remove Inputs** button. ![](/img/community/datasets/dataset_21.png) After merging the datasets, remember to [update the dataset version](#update-a-dataset-version) of your destination dataset to ensure the latest version reflects the newly added inputs and annotations. ### Delete Datasets To delete a dataset, go to the individual page of the dataset and select the **Delete Dataset** option that drops down after clicking the ellipsis in the upper-right corner of the page. :::caution Please proceed with extreme caution, as deleted datasets cannot be recovered. ::: ![](/img/community/datasets/dataset_17.png) ## **Via the API** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodePatchDataset from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/patch_dataset.py"; import CodeExport from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/export.py"; import CodeSDH from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/sdh.py"; import CodeDV from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/delete_version.py"; import CodeDVTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/deleteDatasetVersion.ts"; import CodeDelete from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/delete.py"; import CodeDeleteTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/deleteDataset.ts"; import MergeDataset from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/merge_dataset.py"; import ListDatasetInputs from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/list_dataset_inputs.py"; import curlListDatasets from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/list_datasets.sh"; import curlGetDatasets from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/get_datasets.sh"; import curlUpdateDatasets from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/update_datasets.sh"; import curlUpdateDatasetsDefaultFilter from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/update_datasets_default_filter.sh"; import curlDeleteDatasets from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/delete_datasets.sh"; import curlListInputsDatasets from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/list_inputs_datasets.sh"; import curlGetDatasetInputs from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/get_dataset_input.sh"; import curlDeleteInputs from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/delete_inputs.sh"; import curlListDatasetVersions from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/list_dataset_versions.sh"; import curlGetDatasetVersions from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/get_dataset_versions.sh"; import curlChangeDatasetVersion from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/change_dataset_version.sh"; import curlDeleteDatasetVersion from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/delete_dataset_version.sh"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ### List Datasets You can list the datasets in your app. {curlListDatasets} ### List Dataset Versions You can list all the versions associated with your dataset to view its update history and changes over time. {curlListDatasetVersions} ### Get a Dataset You can retrieve the details of a specific dataset by providing its ID. {curlGetDatasets} ### Get a Dataset Version You can retrieve the details of a specific dataset version by providing its version ID. {curlGetDatasetVersions} ### Get a Dataset Input You can retrieve an input in a dataset by specifying its ID. {curlGetDatasetInputs} ### List Dataset Inputs You can list the inputs in a dataset by providing the dataset ID. {ListDatasetInputs} {curlListInputsDatasets} ### Export Dataset You can download your datasets in a compressed ZIP file format for easy storage, sharing, or offline access. :::info The `clarifai-data-protobuf.zip` file can be downloaded from the [export dataset](#export-a-dataset) section on the platform UI. ::: {CodeExport} ### SDH Enabled Inputs Download You can download inputs that have been enhanced or optimized using [Secure Data Hosting](https://docs.clarifai.com/product-updates/upcoming-api-changes/secure-data-hosting/#what-is-secure-data-hosting) (SDH) technology. This feature leverages the power of SDH to deliver a faster, more efficient download experience — offering performance and flexibility tailored to modern computing needs. {CodeSDH} ### Patch a Dataset You can apply patch operations to a dataset — merging, removing, or overwriting data. While all these actions support overwriting by default, they have specific behaviors when handling lists of objects. - The `merge` action replaces a `key:value`pair with `key:new_value`, or appends to an existing list. For dictionaries, it merges entries that share the same `id` field. - The `remove` action is only used to delete the dataset's cover image on the platform UI. - The `overwrite` action completely replaces an existing object with a new one. Below is an example of patching a dataset to update its description, notes, and image URL. {CodePatchDataset} Below is an example of updating dataset's description and metadata. {curlUpdateDatasets} #### Patch With a Default Filter Below is an example of updating a dataset with a default filter. {curlUpdateDatasetsDefaultFilter} #### Patch Dataset Version Below is an example of updating a dataset version's name. {curlChangeDatasetVersion} ### Merge Datasets Here’s an example of merging a dataset with the ID `merge_dataset_id` into another dataset with the ID `dataset_id` using the `merge_dataset` feature from the `Dataset` class. Note that all inputs from the source dataset (`merge_dataset_id`) will be added to the target dataset (`dataset_id`). {MergeDataset} ### Delete Dataset Inputs You can delete the inputs in a dataset by specifying their IDs. {curlDeleteInputs} ### Delete Dataset Version You can easily remove specific versions of your dataset. :::caution Be certain that you want to delete a particular dataset version as the operation cannot be undone. ::: {CodeDV} {CodeDVTS} {curlDeleteDatasetVersion} ### Delete Dataset You can easily remove a dataset by specifying its ID. :::caution Be certain that you want to delete a particular dataset as the operation cannot be undone. ::: {CodeDelete} {CodeDeleteTS} {curlDeleteDatasets} --- ## Upload Data to Dataset via API # Upload Data to Dataset via API **Learn how to upload data to a dataset via the API** Uploading data to a dataset in Clarifai is essential for training and evaluating your machine learning models. Whether you're working with images, videos, text, audio, or other data types, we provide flexible and efficient methods to upload data from various sources. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: :::tip [Click here](https://docs.clarifai.com/additional-resources/api-references/api-reference/#dataset) to learn more about working with the `Dataset` class. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeUpImageAN from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/up_image_an.py"; import CodeUpImageANTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/uploadImageWithAnnotation.ts"; import CodeUpVideoAN from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/up_video_an.py"; import CodeUpVideoANTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/uploadVideoWithAnnotation.ts"; import CodeUpTextAN from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/up_text_an.py"; import CodeUpTextANTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/uploadTextWithAnnotation.ts"; import CodeUpBatch from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/batch.py"; import CodeUpImageMask from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/up_image_annot_mask.py"; import CodeUpImageMaskTS from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/up_image_annot_mask.ts"; import CodeRetry from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/retry_log.py"; import curlAddInputsDatasets from "!!raw-loader!../../../code_snippets/api-guide/data/datasets/add_inputs_datasets.sh"; import UploadFromFolder from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/upload_from_folder.py"; import UploadFromCSV from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/upload_from_csv.py"; import NodeUploadFromFolder from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/node_upload_from_folder.js"; import NodeUploadFromCSV from "!!raw-loader!../../../code_snippets/python-sdk/managing-datasets/node_upload_from_csv.js"; :::note ## Customize Batch Size When uploading inputs to the Clarifai platform, there are limits on the size and number of inputs per upload, as detailed [here](https://docs.clarifai.com/create-manage/inputs/upload/#upload-limits). However, by using methods from the `Dataset` class — such as `Dataset.upload_from_folder()` or `Dataset.upload_from_csv()` — you can bypass these restrictions and efficiently upload larger volumes of inputs. For example, when uploading images in bulk, such methods incrementally process and upload them in multiple batches, ensuring that each batch contains a maximum of 128 images and does not exceed 128MB in size – which ensures adherence to the upload restrictions. You can also customize the `batch_size` variable, which allows for concurrent upload of inputs and annotations. For example, if your images folder exceeds 128MB, you can set the variable to ensure that each batch contains an appropriate number of images while staying within the 128MB per batch limit. The default `batch_size` is set to 32, but you can customize it to any value between 1 (minimum) and 128 (maximum). Here is an example: ```text dataset.upload_from_folder(folder_path='/path/to/your/folder', input_type='image', labels=True, batch_size=50) ``` ::: ## Add Inputs to a Dataset After uploading inputs to the Clarifai platform, you can add them to a dataset by specifying their input IDs. {curlAddInputsDatasets} ## Upload From Folder The `upload_from_folder` method lets you bulk-upload images or text files from a local folder directly into a Clarifai dataset. {UploadFromFolder} {NodeUploadFromFolder} Note that: - The `upload_from_folder` method only supports `"image"` and `"text"` input types. - Ensure your dataset (`dataset_id`) already exists before calling this method. - Large datasets should be uploaded with an appropriate [`batch_size`](#customize-batch-size) (default 128). - If `labels=True`, the folder name is assigned as the input’s concept label. - The filename (without extension) is used as the `input_id` in Clarifai. - When uploading text data, the target app should be configured to accept text inputs. Set the primary input type to **Text/Document** when [creating the app](https://docs.clarifai.com/create/applications/create#create-via-the-ui). ## Upload From CSV The `upload_from_csv` method lets you bulk-upload data into a Clarifai dataset using a CSV file. This method is useful when your data is already structured in tabular form with URLs, local file paths, or raw text. {UploadFromCSV} {NodeUploadFromCSV} Example CSV Files File-path based (for local files) ```text inputid,input,concepts,metadata img1,"data/metro-north.jpg","train","{'source': 'local'}" img2,"data/puppy.jpeg","dog","{'source': 'local'}" ``` Raw text dataset (only valid with input_type="text") ```text inputid,input,concepts,metadata txt1,"The sky is clear and blue","weather","{'lang': 'en'}" txt2,"The puppy is playing in the garden","dog","{'lang': 'en'}" ``` With geopoints ```text inputid,input,concepts,metadata,geopoints img1,"data/metro-north.jpg","train","{'source': 'clarifai-samples'}","-73.935242,40.730610" img2,"data/puppy.jpeg","dog","{'source': 'clarifai-samples'}","-118.243683,34.052235" ``` Note that: - The `upload_from_csv` method supports `"image"`, `"text"`, `"video"`, and `"audio"` file types. - The `csv_type` parameter defines how the CSV file will be interpreted. It can be: - `"url"` — Inputs are hosted online, and the CSV provides URLs. - `"file_path"` — Inputs are stored locally, and the CSV provides file paths. - `"raw"` — Only valid for text datasets; the CSV provides raw text strings. - If `labels=True` (default), the CSV must include a `concepts` column with labels. If `False`, inputs are uploaded without labels. - The [`batch_size`](#customize-batch-size) (default = 128) parameter defines the maximum number of inputs to upload concurrently in one batch. - The CSV file must include column headers. These are the supported headers: - `inputid` — Unique identifier for the input. - `input` — URL, file path, or raw text depending on `csv_type`. - `concepts` — Concept labels (if `labels=True`). - `metadata` — JSON metadata, formatted with single quotes inside. Example: `"{'source': 'web'}"`. - `geopoints` — Geolocation in `"longitude,latitude"` format. - All the data in the CSV file should be enclosed in double quotes. - When uploading text data, ensure the target app is configured to accept text inputs. Set the primary input type to **Text/Document** when creating the app. ## Upload Image Data With Annotations You can [upload image data](https://docs.clarifai.com/create/inputs/upload/api#upload-image-data) together with bounding box annotations into a Clarifai dataset, adding richer context and detail to your visual data. {CodeUpImageAN} {CodeUpImageANTS} ## Upload Image Data With Mask Annotations You can add masks to image data in a Clarifai dataset by providing polygon coordinates with the image, enabling precise region-based annotations. {CodeUpImageMask} {CodeUpImageMaskTS} ## Upload Video Data With Annotations You can [upload videos](https://docs.clarifai.com/create/inputs/upload/api#upload-video-data) in a Clarifai dataset with enriched annotations by including bounding box coordinates that define regions of interest within individual frames, adding valuable context to your video content. {CodeUpVideoAN} {CodeUpVideoANTS} ## Upload Text Data With Annotations You can [upload text data](https://docs.clarifai.com/create/inputs/upload/api#upload-text-data) in a Clarifai dataset and enrich it by attaching metadata, categorizing the content, or adding detailed annotations to enhance structure and context. {CodeUpTextAN} {CodeUpTextANTS} ## Batch Upload Image Data While Tracking Status You can actively monitor the status of your dataset upload, giving you clear visibility into the progress and making it easy to track and analyze the data transfer process. {CodeUpBatch} ## Retry Upload From Log File You can retry uploads for failed inputs directly from the logs. When using the `upload_dataset` function, any failed inputs are automatically logged to a file, which can later be used to resume and retry the upload process seamlessly. :::info Set `retry_duplicates` to `True` if you want to retry duplicate with new `input_id` in current dataset. ::: {CodeRetry} --- ## Create and Manage Your Inputs # Create and Manage Your Inputs **Manage your inputs and training data on the Clarifai platform** Clarifai provides an easy-to-use UI that lets you explore your data and maximize the performance of your models. The platform offers several features to help you effectively manage and optimize your machine learning projects, including: - **Data exploration:** You can upload, visualize, and organize your datasets, including images, videos, and text. The UI provides tools for searching and filtering data to better understand and manage the information. - **Annotation:** The interface includes robust labeling tools, enabling you to prepare your data for training. This includes adding labels, drawing bounding boxes, and applying other relevant annotations. ## Inputs-Manager Screen If you click the **Inputs** option in your app's collapsible left sidebar, you'll be redirected to the Inputs-Manager screen, which is the main page showcasing all the inputs in your app. :::note An input is any data added to the Clarifai platform. ::: On this screen, you can: ![](/img/community_2/data_explorer_1.png) - **Perform a smart search**: Use the search field to rank, sort, and retrieve inputs based on a predicted match to your query caption text. Learn more about it [here](https://docs.clarifai.com/portal-guide/psearch/visual-search/#smart-image-search-by-caption). - **Upload inputs**: Add new inputs to your app by clicking the **Upload Inputs** button in the upper-right section of the page. Learn more about it [here](https://docs.clarifai.com/portal-guide/inputs-manager/upload-inputs). - **Filter inputs**: Use the **Input Filters** section to customize how inputs are displayed. By default, inputs are sorted by creation date, with the newest at the top. You can filter inputs by dataset, labels, input type (audio, image, text, or video), or metadata. The number of inputs matching your criteria will be displayed at the top. - **Zoom in/out**: Adjust the zoom level of the displayed inputs to suit your preference. - **Toggle layout**: Toggle the layout of the displayed items. ### Input Mode The **Input Mode** is the default mode of the Inputs-Manager screen. It displays a gallery showcasing all inputs in your app. Image inputs are shown as thumbnails, while text inputs are truncated. ![](/img/community_2/data_explorer_2.png) When you hover over an input, icons will appear in the left corner of the highlighted input, allowing you to: - Click the small empty box to select the input. - Click the magnifying glass to perform a smart search by similarity. This lets you rank, sort, and retrieve inputs based on their similarity to a provided query input. Learn more about it [here](https://docs.clarifai.com/portal-guide/psearch/visual-search#smart-image-search-by-visual-similarity). - Click the info button to see details of the input, such as input ID, creation date, and resolution. - Click the delete button to remove the input from your app. When you select an input by clicking the small empty box in its upper-left corner, a section will appear at the bottom of the page, allowing you to: - Click the **Label as…** button to add or remove annotations from the selected input. Learn more about it [here](https://docs.clarifai.com/portal-guide/annotate/create-get-update-delete#via-the-inputs-manager-page). - Click the **Edit Metadata** button to add or remove metadata information associated with the selected input. - Click the **Dataset…** button to add or remove the selected input from a dataset. - Click the **Delete** button to remove the selected input from your app. ### Object Mode The **Object Mode** is a gallery that shows the objects that have been labeled on your inputs. You can select it by clicking the **Object Mode** button in the upper-right section of the Inputs-Manager screen. ![](/img/community_2/data_explorer_3.png) If you hover over an input and select it by clicking the small empty box in its upper-left corner, several options will appear, allowing you to perform various management tasks. - The magnifying glass icon appearing in the left side of the input allows you to use it to perform [Smart Object Search](https://docs.clarifai.com/portal-guide/psearch/prank) and retrieve annotated objects within other inputs based on their content and similarity. - The delete icon allows you to delete the input from your app. - The **Add Concepts** button that appears at the bottom of the page allows you to create new object annotations with the selected concepts. - The **Remove Annotations** button that appears at the bottom of the page allows you to remove the selected annotation from the input. Additionally, while in **Object Mode**, you can click the ellipsis next to the **Labels** section to reveal various options for filtering the display of your inputs. You can filter the display by: Operator — `or` or `and`, [Labeler](https://docs.clarifai.com/portal-guide/annotate/create-a-task#step-6-add-collaborators), or [Status](https://docs.clarifai.com/api-guide/advanced-topics/status-codes#annotation-related-codes-24xxx) — `Success`, `Awaiting Review`, `Awaiting Consensus Review`, or `Pending`. ![](/img/community_2/data_explorer_4.png) ## Inputs and Outputs Guide ### Example When choosing one of Clarifai's pre-built models, you might see something like this from some of our models: Input Type | Output Type ------------- | ------------- image | regions[...].data.concepts, regions[...].region\_info.bounding\_box These inputs and outputs can be clarified with the following table explaining these data types: ### Table of Uploadable Data Types Data Type | Meaning ---------- | ------------- text | This is freeform plain text which can be uploaded via raw text or specified with a URI. image | This is an image in an accepted format, which currently includes JPG, PNG, TIFF, BMP, WEBP, CSV, and TSV. It can be uploaded via base64 bytes or specified with a URI. video | This is video in an accepted format, which currently includes AVI, MP4, WMV, MOV, GIF, and 3GPP. It can be uploaded via base64 bytes or specified with a URI. All these data formats are read in as raw bytes in base64 format. ### Table of Single Data Types Passed Between Models Data Type | Meaning ---------- | ------------- embeddings | Vector representions of data passed from model to model. These are not uploadable by users. clusters | These are IDs that identify clusters. These are primarily used for image search. concepts | The list of concepts used in a model. For the general model, these would be the top 20 concepts with classified with the highest confidence. ### Table of `regions[...]` Data Types The notation of `[...]` means that the variable is a list of things, so `regions[...]` represents a list of regions of data. This could be parts of an image, text, video, or audio: Data Type | Meaning ------------- | ------------- regions[...].region_info.point | This is a list of points which specify regions of an image. regions[...].region_info.bounding\_box | This is a list of regions each containing the four corners of a bounding box in a specific region of an image. Each corner coordinate is normalized to [0,1]. regions[...].region_info.mask | The mask is an overlay of the entire image, with the specific concepts pixels set to a certain color. regions[...].data.text | This is a list of regions and their associated text. This could be OCR data for an image, or subtext within a larger text for NLP. regions[...].data.embeddings | This is a list of regions and their associated vector representions. regions[...].data.concepts | This is a list of regions and their associated or high confidence concepts. ### Table of `frames[...]` Data Types The notation of `[...]` means that the variable is a list of things, so `frames[...]` represents a list of frames of video or audio, and therefore `frames[...].data.regions[...]` represents a 2D matrix of the number of frames by the number of regions in each frame. Data Type | Meaning ------------- | ------------- frames[...].data.regions[...].region\_info.bounding\_box | These are the four corners of a bounding box in a specific region of a specific frame of video. Each corner coordinate is normalized to [0,1]. frames[...].data.regions[...].data.concepts | This is the matrix of frames and regions containing the concepts used in a model. For the general model, these would be the top 20 concepts classified with the highest confidence in a specific region of a specific frame of video. frames[...].data.regions[...].track_id | This is the matrix of frames and regions containing the tracking ids used to track objects across frames of a video. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Manage Inputs via API # Manage Inputs via API **Learn how to use our API to manage, organize, and process your inputs** You can manage inputs on the Clarifai platform by organizing, updating, deleting, and performing various data processing tasks. Whether you're working with images, text, or videos, the platform provides powerful capabilities to help you maintain full control over your inputs throughout their lifecycle. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import BulkDeleteAnnotations from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/bulk_delete_annotations.py"; import DownloadInputs from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/download_inputs.py"; import CodeListInput from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/list_input.py"; import CodeDeleteInput from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/delete_input.py"; import CodePatchInputs1 from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/patch_inputs_1.py"; import CodePatchInputs2 from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/patch_inputs_2.py"; import CodePatchInputs3 from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/patch_inputs_3.py"; import CodePatchInputs4 from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/patch_inputs_4.py"; import CodeOutputListInput from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/outputs/list_input.txt"; import CodeOutputDeleteInput from "!!raw-loader!../../../code_snippets/python-sdk/managing-inputs/outputs/delete_input.txt"; import PythonListAllInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/list_all_inputs.py"; import PythonListInputsStreaming from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/list_inputs_streaming.py"; import PythonGetInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/get_input_by_id.py"; import PythonGetInputsStatus from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/get_inputs_status.py"; import PythonUpdateInputConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/update_input_concepts.py"; import PythonBulkUpdateInputsConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/bulk_update_inputs_concepts.py"; import PythonDeleteConceptsInput from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/delete_concepts_input.py"; import PythonBulkDeleteConceptsInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/bulk_delete_concepts_inputs.py"; import PythonDeleteInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/delete_input_by_id.py"; import PythonDeleteListInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/py/delete_list_inputs.py"; import JSListAllInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/list_all_inputs.html"; import JSListInputsStreaming from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/list_inputs_streaming.html"; import JSGetInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/get_input_by_id.html"; import JSGetInputsStatus from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/get_inputs_status.html"; import JSUpdateInputConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/update_input_concepts.html" import JSBulkUpdateInputsConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/bulk_update_inputs_concepts.html"; import JSDeleteConceptsInput from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/delete_concepts_input.html"; import JSBulkDeleteConceptsInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/bulk_delete_concepts_inputs.html"; import JSDeleteInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/delete_input_by_id.html"; import JSDeleteListInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/js/delete_list_inputs.html" import NodeListAllInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/list_all_inputs.js"; import NodeListInputsStreaming from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/list_inputs_streaming.js"; import NodeGetInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/get_input_by_id.js"; import NodeGetInputsStatus from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/get_inputs_status.js"; import NodeUpdateInputConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/update_input_concepts.js" import NodeBulkUpdateInputsConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/bulk_update_inputs_concepts.js"; import NodeDeleteConceptsInput from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/delete_concepts_input.js"; import NodeBulkDeleteConceptsInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/bulk_delete_concepts_inputs.js"; import NodeDeleteInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/delete_input_by_id.js"; import NodeDeleteListInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/node/delete_list_inputs.js"; import JavaListAllInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/list_all_inputs.java"; import JavaListInputsStreaming from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/list_inputs_streaming.java"; import JavaGetInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/get_input_by_id.java"; import JavaGetInputsStatus from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/get_inputs_status.java"; import JavaUpdateInputConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/update_input_concepts.java" import JavaBulkUpdateInputsConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/bulk_update_inputs_concepts.java"; import JavaDeleteConceptsInput from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/delete_concepts_input.java"; import JavaBulkDeleteConceptsInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/bulk_delete_concepts_inputs.java"; import JavaDeleteInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/delete_input_by_id.java"; import JavaDeleteListInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/java/delete_list_inputs.java"; import CurlListAllInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/list_all_inputs.sh"; import CurlListInputsStreaming from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/list_inputs_streaming.sh"; import CurlGetInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/get_input_by_id.sh"; import CurlGetInputsStatus from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/get_inputs_status.sh"; import CurlUpdateInputConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/update_input_concepts.sh" import CurlBulkUpdateInputsConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/bulk_update_inputs_concepts.sh"; import CurlDeleteConceptsInput from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/delete_concepts_input.sh"; import CurlBulkDeleteConceptsInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/bulk_delete_concepts_inputs.sh"; import CurlDeleteInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/delete_input_by_id.sh"; import CurlDeleteListInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/curl/delete_list_inputs.sh" import PHPListAllInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/list_all_inputs.php"; import PHPListInputsStreaming from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/list_inputs_streaming.php"; import PHPGetInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/get_input_by_id.php"; import PHPGetInputsStatus from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/get_inputs_status.php"; import PHPUpdateInputConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/update_input_concepts.php" import PHPBulkUpdateInputsConcepts from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/bulk_update_inputs_concepts.php"; import PHPDeleteConceptsInput from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/delete_concepts_input.php"; import PHPBulkDeleteConceptsInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/bulk_delete_concepts_inputs.php"; import PHPDeleteInputId from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/delete_input_by_id.php"; import PHPDeleteListInputs from "!!raw-loader!../../../code_snippets/api-guide/data/create_get_update_delete/php/delete_list_inputs.php"; ## List Inputs ### List All Inputs You can retrieve all inputs available in your app. If you added inputs with concepts, they will be returned in the response as well. Note that this request supports [pagination](https://docs.clarifai.com/additional-resources/api-overview/pagination), allowing you to navigate through large sets of inputs efficiently. {CodeListInput} {PythonListAllInputs} {JSListAllInputs} {NodeListAllInputs} {JavaListAllInputs} {PHPListAllInputs} {CurlListAllInputs} ```csharp var response = client.ListInputs( new ListInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, Page = 1, PerPage = 10 }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> Output {CodeOutputListInput} ### List Inputs \(Streaming\) This is another method for listing inputs, which was built to scalably list an app's inputs in an iterative / streaming fashion. `StreamInputs` will return `per_page` number of inputs from a certain input onward, controlled by the optional `last_id` parameter \(defaults to the first input\). By default, the stream will return inputs from oldest to newest. Set the `descending` field to true to reverse that order. {PythonListInputsStreaming} {JSListInputsStreaming} {NodeListInputsStreaming} {JavaListInputsStreaming} {PHPListInputsStreaming} ## Get Inputs ### Get Input by ID If you'd like to get the details of a specific input by its `id`, you can do that as well. {PythonGetInputId} {JSGetInputId} {NodeGetInputId} {JavaGetInputId} {PHPGetInputId} {CurlGetInputId} ```csharp var response = client.ListInputs( new ListInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, InputId = "{YOUR_INPUT_ID}" }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Get Inputs' Status If you add inputs in bulk, they will be procesed in the background. You can get the status of all your inputs \(processed, to\_process, and errors\) like this: {PythonGetInputsStatus} {JSGetInputsStatus} {NodeGetInputsStatus} {JavaGetInputsStatus} {PHPGetInputsStatus} {CurlGetInputsStatus} ```csharp var response = client.GetInputCount( new GetInputCountRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); Console.WriteLine(response.Counts); ``` --> ## Download Inputs Below is an example of how to download inputs from your app. {DownloadInputs} ## Patch Inputs You can apply patch operations to an input, allowing for the merging or removal of items. By default, these actions overwrite existing data, but they behave differently when handling lists of objects. - The `merge` action replaces a `key:value` pair with a `key:new_value`, or appends new values to an existing list. When dealing with dictionaries, it merges entries that share the same `id` field. - The `remove` action replaces a `key:value` pair with a `key:new_value`, or removes any items from a list that match the IDs of the provided values. - The `overwrite` action fully replaces an existing object with a new one. ### Patch Metadata Here is an example of how to patch the metadata of an input. {CodePatchInputs3} ### Patch Bounding Box Annotation Here is an example of how to patch a bounding box annotation on an input. {CodePatchInputs1} ### Patch Polygon Annotation Here is an example of how to patch a polygon annotation on an input. {CodePatchInputs2} ### Update Input With Concepts To update an input with a new concept, or to change a concept value from true/false, you can do the following: {CodePatchInputs4} {PythonUpdateInputConcepts} {JSUpdateInputConcepts} {NodeUpdateInputConcepts} {JavaUpdateInputConcepts} {PHPUpdateInputConcepts} {CurlUpdateInputConcepts} ```csharp var response = client.PatchInputs( new PatchInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" } Action = "merge", Inputs = { new List() { new Input() { InputId = "{YOUR_INPUT_ID}", Data = new Data() { Concepts = { new List { new Concept { Id = "tree", Value = 1 } }, new Concept { Id = "water", Value = O } } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Bulk Update Inputs With Concepts You can update existing inputs using their `ids`. This is useful if you'd like to add concepts to inputs after they have already been added. Below is an example of how to update multiple inputs with concepts at once. {PythonBulkUpdateInputsConcepts} {JSBulkUpdateInputsConcepts} {NodeBulkUpdateInputsConcepts} {JavaBulkUpdateInputsConcepts} {PHPBulkUpdateInputsConcepts} {CurlBulkUpdateInputsConcepts} ```csharp var response = client.PatchInputs( new PatchInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" } Action = "merge", Inputs = { new List() { new Input() { InputId = "{YOUR_INPUT_ID_1}", Data = new Data() { Concepts = { new List { new Concept { Id = "tree", Value = 1 } }, new Concept { Id = "water", Value = O } } } } }, { new Input() { InputId = "{YOUR_INPUT_ID_2}", Data = new Data() { Concepts = { new List { new Concept { Id = "animal", Value = 1 } }, new Concept { Id = "fruit", Value = O } } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ## Bulk Delete Input Annotations Below is an example of how to delete all the annotations associated with a given input by setting the input ID(s). The `annotation_ids` parameter is optional. However, if provided, the number and order of `annotation_ids` must match the corresponding `input_ids`. {BulkDeleteAnnotations} ## Delete Inputs :::caution Be certain that you want to delete a particular input as the operation cannot be undone. ::: ### Delete Concepts From an Input To remove concepts that were already added to an input, you can do this: {PythonDeleteConceptsInput} {JSDeleteConceptsInput} {NodeDeleteConceptsInput} {JavaDeleteConceptsInput} {PHPDeleteConceptsInput} {CurlDeleteConceptsInput} ```csharp var response = client.PatchInputs( new PatchInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" } Action = "remove", Inputs = { new List() { new Input() { InputId = "{YOUR_INPUT_ID}", Data = new Data() { Concepts = { new List { new Concept { Id = "tree" } } } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Bulk Delete Concepts From a List of Inputs Below is an example of how to bulk delete multiple concepts from a list of inputs. {PythonBulkDeleteConceptsInputs} {JSBulkDeleteConceptsInputs} {NodeBulkDeleteConceptsInputs} {JavaBulkDeleteConceptsInputs} {PHPBulkDeleteConceptsInputs} {CurlBulkDeleteConceptsInputs} ```csharp var response = client.DeleteInput( new DeleteInputRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" } Action = "remove", Inputs = { new List() { new Input() { InputId = "{YOUR_INPUT_ID_1}", Data = new Data() { Concepts = { new List { new Concept { Id = "tree" } }, { new Concept { Id = "water" } } } } } }, { new Input() { InputId = "{YOUR_INPUT_ID_2}", Data = new Data() { Concepts = { new List { new Concept { Id = "animal" } }, { new Concept { Id = "fruit" } } } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Delete Input by ID Below is an example of how to delete a single input by its `id`. {PythonDeleteInputId} {JSDeleteInputId} {NodeDeleteInputId} {JavaDeleteInputId} {PHPDeleteInputId} {CurlDeleteInputId} ```csharp var response = client.DeleteInput( new DeleteInputRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, inputId = "{YOUR_INPUT_ID}" }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Delete a List of Inputs You can also delete multiple inputs in one API call. This will happen asynchronously. :::info We currently support a batch size of 128 inputs per request. So, you can provide a list of 128 input IDs and delete them in one API call. ::: {PythonDeleteListInputs} {JSDeleteListInputs} {NodeDeleteListInputs} {JavaDeleteListInputs} {PHPDeleteListInputs} {CurlDeleteListInputs} ```csharp var response = client.DeleteInput( new DeleteInputRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, Ids = { new List() { "{YOUR_INPUT_ID_1}", "{YOUR_INPUT_ID_2}" } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Delete All Inputs Below is an example of how to delete all inputs from your app. {CodeDeleteInput} Output {CodeOutputDeleteInput} --- ## Upload Inputs # Upload Inputs **Add structured or unstructured data to the Clarifai platform** An input is any piece of structured or unstructured data added to the Clarifai platform. This includes images, text, videos, and more — you can add as many inputs as you want. Whether your data is hosted online via URLs, stored locally as file paths, or represented as bytes, our platform supports a wide range of formats, ensuring flexibility and ease of use. You can also upload zipped archive files (ZIP format) containing mixed data types, such as text and images. Once uploaded, you can organize your inputs into [datasets](https://docs.clarifai.com/portal-guide/datasets/) to support a wide range of tasks, including: - [Data labeling](https://docs.clarifai.com/portal-guide/labeling-tasks/) - [Performing searches](https://docs.clarifai.com/portal-guide/psearch/) - [Training new models](https://docs.clarifai.com/portal-guide/model/) - [Evaluating models](https://docs.clarifai.com/portal-guide/evaluate/) - [Making predictions](https://docs.clarifai.com/portal-guide/ppredict/) :::info - As each input is uploaded, it is automatically indexed using the specified [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows) for your app. This indexing enables you to perform searches over the uploaded inputs, leveraging Clarifai’s custom-built [vector database](https://docs.clarifai.com/portal-guide/psearch/#powered-by-a-vector-database) for fast and efficient search capabilities. - Stored inputs are subject to a recurring storage fee, meaning your account balance may decrease over time even without active platform usage. ::: ## Upload Limits When uploading data to the Clarifai platform, your inputs should meet the conditions outlined below. Note that if the size of your input, such as a video or an audio file, exceeds these limits, you will need to split ithem into smaller chunks. Otherwise, the processing will time out, and you will receive an error response. ### Images - The supported image formats include `JPEG`, `PNG`, `TIFF`, `BMP`, `WEBP`, and `GIF`. - Each request can include up to 128 image inputs per batch. - Each image file must be a maximum of 85 megapixels and less than 20MB in size. - The total batch size (in bytes) for each request must be less than 128MB. ### Videos - The supported video formats include `AVI`, `MP4`, `WMV`, `MOV`, and `3GPP`. - Each request can include only 1 video input. - If uploading via URL, the video can be up to 300MB or 10 minutes long. - If uploading via direct file upload (byte data), the video must be less than 128MB. ### Text Files - The supported text formats include plain texts, CSV files, and TSV files. - Each request can include up to 128 text files per batch. - Each text file must be less than 20MB. - The total batch size (in bytes) must be less than 128MB. ### Audio Files - The supported audio format is `WAV`. - Each request can include up to 128 audio files per batch. - Each audio file must be less than 20MB in size (suitable for a 48kHz, 60-second, 16-bit recording). - The total batch size (in bytes) must be less than 128MB. :::note bypass upload limits When uploading data to the Clarifai platform using the Python SDK — such as with `upload_from_bytes()` or `upload_from_url()` methods (demonstrated [here](api.md)) — the standard upload limits apply. However, you can bypass these limits by using the [`upload_from_folder()`](https://docs.clarifai.com/sdk/managing-datasets/upload-data) method from the `Dataset` class, which efficiently handles larger volumes of inputs by automatically batching them while adhering to upload restrictions. For example, when uploading images in bulk, the method incrementally processes and uploads them in multiple batches, ensuring that each batch contains a maximum of 128 images and does not exceed 128MB in size. You can also customize the `batch_size` variable, which allows for concurrent upload of inputs and annotations. For example, if your folder exceeds 128MB, you can set the variable to ensure that each batch contains an appropriate number of images while staying within the 128MB per batch limit. ::: import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Upload via API # Upload via API **Add data to the Clarifai platform via the API** :::tip Data Utils The Clarifai's [Data Utils library](https://docs.clarifai.com/sdk/data-utils/) allows you to effortlessly extract, transform, and load unstructured data — such as images, videos, and text — into the Clarifai platform. ::: :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: Note that input uploads are processed asynchronously. Your files will be indexed in the background via your app's default base workflow, which may take some time depending on volume and file types. To verify successful indexing, you can check the [input status](https://docs.clarifai.com/create-manage/inputs/manage#get-inputs-status) for code `30000` (`INPUT_IMAGE_DOWNLOAD_SUCCESS`). This confirms the input is fully processed and ready for use. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeImageData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/image_data.py"; import CodeImageDataTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/imageDataInputs.ts"; import CodeTextData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/text_data.py"; import CodeTextDataTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/textDataInputs.ts"; import CodeAudioData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/audio_data.py"; import CodeAudioDataTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/audioDataInputs.ts"; import CodeVideoData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/video_data.py"; import CodeVideoDataTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/videoDataInputs.ts"; import CodeMMData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/multimodal_data.py"; import CodeMMDataTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/multimodalDataInputs.ts"; import CodeImageM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/image_metadata.py"; import CodeImageMTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/imageWithMetadata.ts"; import CodeVideoM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/video_metadata.py"; import CodeVideoMTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/videoWithMetadata.ts"; import CodeTextM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/text_metadata.py"; import CodeTextMTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/textWithMetadata.ts"; import CodeAudioM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/audio_data_metadata.py"; import CodeAudioMTS from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/audioWithMetadata.ts"; import CodeGeoInfo from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/geo_info.py"; import BoundingBoxAnnotation from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/bounding_box_annotation.py"; import PolygonAnnotation from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/polygon_annotation.py"; import ConceptsAnnotation from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/concepts_annotation.py"; import CodeOutputImageData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/image_data.txt"; import CodeOutputTextData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/text_data.txt"; import RemoveUnicode from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/remove_unicode.py"; import CodeOutputAudioData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/audio_data.txt"; import CodeOutputVideoData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/video_data.txt"; import CodeOutputMMData from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/multimodal_data.txt"; import CodeOutputImageM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/image_metadata.txt"; import CodeOutputVideoM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/video_metadata.txt"; import CodeOutputTextM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/text_metadata.txt"; import CodeOutputAudioM from "!!raw-loader!../../../../code_snippets/python-sdk/managing-inputs/outputs/audio_data_metadata.txt"; import PythonAddInputsViaURL from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/py/add_inputs_via_url.py"; import PythonAddInputsViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/py/add_inputs_via_bytes.py"; import PythonAddMultipleInputsIds from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/py/add_multiple_inputs_with_ids.py"; import PythonAddInputsConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/py/add_inputs_with_concepts.py"; import PythonAddInputsMultipleConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/py/add_inputs_with_multiple_concepts.py"; import PythonAddInputsCustomMetadata from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/py/add_inputs_custom_metadata.py"; import JSAddInputsViaURL from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/js/add_inputs_via_url.html"; import JSAddInputsViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/js/add_inputs_via_bytes.html"; import JSAddMultipleInputsIds from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/js/add_multiple_inputs_with_ids.html"; import JSAddInputsConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/js/add_inputs_with_concepts.html"; import JSAddInputsMultipleConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/js/add_inputs_with_multiple_concepts.html"; import JSAddInputsCustomMetadata from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/js/add_inputs_custom_metadata.html"; import NodeAddInputsViaURL from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/node/add_inputs_via_url.js"; import NodeAddInputsViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/node/add_inputs_via_bytes.js"; import NodeAddMultipleInputsIds from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/node/add_multiple_inputs_with_ids.js"; import NodeAddInputsConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/node/add_inputs_with_concepts.js"; import NodeAddInputsMultipleConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/node/add_inputs_with_multiple_concepts.js"; import NodeAddInputsCustomMetadata from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/node/add_inputs_custom_metadata.js"; import JavaAddInputsViaURL from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/java/add_inputs_via_url.java"; import JavaAddInputsViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/java/add_inputs_via_bytes.java"; import JavaAddMultipleInputsIds from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/java/add_multiple_inputs_with_ids.java"; import JavaAddInputsConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/java/add_inputs_with_concepts.java"; import JavaAddInputsCustomMetadata from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/java/add_inputs_custom_metadata.java"; import CurlAddInputsViaURL from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/curl/add_inputs_via_url.sh"; import CurlAddInputsViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/curl/add_inputs_via_bytes.sh"; import CurlAddMultipleInputsIds from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/curl/add_multiple_inputs_with_ids.sh"; import CurlAddInputsConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/curl/add_inputs_with_concepts.sh"; import CurlAddInputsCustomMetadata from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/curl/add_inputs_custom_metadata.sh"; import PHPAddInputsViaURL from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/php/add_inputs_via_url.php"; import PHPAddInputsViaBytes from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/php/add_inputs_via_bytes.php"; import PHPAddMultipleInputsIds from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/php/add_multiple_inputs_with_ids.php"; import PHPAddInputsConcepts from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/php/add_inputs_with_concepts.php"; import PHPAddInputsCustomMetadata from "!!raw-loader!../../../../code_snippets/api-guide/data/create_get_update_delete/php/add_inputs_custom_metadata.php"; ## Upload Image Data Below is an example of how to upload image data. {CodeImageData} Output {CodeOutputImageData} {CodeImageDataTS} ## Upload Text Data Below is an example of how to upload text data. {CodeTextData} Output {CodeOutputTextData} {CodeTextDataTS} ## Write Custom Functions for Data Processing You can add your own custom functions for data processing with ease. Below is an example of how to clean text data by removing Unicode characters before uploading it to the Clarifai platform. {RemoveUnicode} ## Upload Audio Data Below is an example of how to upload audio data. {CodeAudioData} Output {CodeOutputAudioData} {CodeAudioDataTS} ## Upload Video Data Below is an example of how to upload video data. {CodeVideoData} Output {CodeOutputVideoData} {CodeVideoDataTS} ## Upload Multimodal Data Below is an example of how to upload a combination of different input types, such as images and text, to the Clarifai platform. Currently, Clarifai supports specific multimodal input combinations, such as `[Image, Text] -> Text`. This allows you to process and analyze interconnected data types for advanced use cases. {CodeMMData} Output {CodeOutputMMData} {CodeMMDataTS} ## Upload Custom Metadata When using the Clarifai SDKs, you can enhance your inputs by attaching custom metadata alongside concepts. This feature enables you to include additional contextual information, such as categorization, filtering criteria, or reference data, making it easier to organize and retrieve your inputs later. Below are examples of how to upload inputs with custom metadata. In these examples, the metadata includes details about the filename and the dataset split (e.g., train, validate, or test) to which the input belongs. ### Image With Metadata {CodeImageM} Output {CodeOutputImageM} {CodeImageMTS} ### Video With Metadata {CodeVideoM} Output {CodeOutputVideoM} {CodeVideoMTS} ### Text With Metadata {CodeTextM} Output {CodeOutputTextM} {CodeTextMTS} ### Audio With Metadata {CodeAudioM} Output {CodeOutputAudioM} {CodeAudioMTS} ## Upload Inputs with Geospatial Information When uploading inputs to Clarifai, you can enrich them by including geospatial data, such as longitude and latitude coordinates from the GPS system. This allows you to associate each input with a specific geographic location. Note that each input can have at most one geospatial point associated with it. {CodeGeoInfo} ## Upload Inputs With Annotations You can upload inputs along with their corresponding annotations, such as bounding boxes or polygons. ### Bounding Box Annotations Below is an example of how to label a new rectangular bounding box for a specific region within an image. The bounding box coordinates should be normalized to the image dimensions, with values scaled to the range of [0, 1.0]. This ensures that the coordinates are independent of the image resolution, making the annotations consistent across different image sizes. {BoundingBoxAnnotation} ### Polygon Annotations Below is an example of how to annotate any polygon-shaped region within an image. A polygon is defined by a list of points, each specified by: - **row** — The row position of the point, represented as a value between 0.0 and 1.0, where 0.0 corresponds to the top row and 1.0 corresponds to the bottom. - **col** — The column position of the point, represented as a value between 0.0 and 1.0, where 0.0 corresponds to the left column of the image and 1.0 corresponds to the right column. {PolygonAnnotation} ### Concepts Annotations Below is an example of how to annotate different types of inputs with [concepts](https://docs.clarifai.com/portal-guide/inputs-manager/concepts). {ConceptsAnnotation} ## Upload Inputs Options (via URLs, bytes, concepts, metadata, etc) You can add inputs one by one or in bulk. If you send them in bulk, you are limited to sending 128 inputs at a time, as mentioned [above](#upload-limits). ### Upload Inputs via URL Below is an example of how to add inputs via a publicly accessible URL. {PythonAddInputsViaURL} {JSAddInputsViaURL} {NodeAddInputsViaURL} {JavaAddInputsViaURL} {PHPAddInputsViaURL} {CurlAddInputsViaURL} ```csharp var response = client.PostInputs( new PostInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, Inputs = { new List() { new Input() { Data = new Data() { Image = new Image() { Url = "https://samples.clarifai.com/metro-north.jpg", AllowDuplicateUrl = true // optional } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Upload Inputs via Bytes Below is an example of how to add inputs via bytes. :::important Note The data must be base64 encoded. When you add a base64 image to our servers, a copy will be stored and hosted on our servers. If you already have an image hosting service, we recommend using it and adding images via the `url` parameter. ::: {PythonAddInputsViaBytes} {JSAddInputsViaBytes} {NodeAddInputsViaBytes} {JavaAddInputsViaBytes} {PHPAddInputsViaBytes} {CurlAddInputsViaBytes} ```csharp var response = client.PostInputs( new PostInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, Inputs = { new List() { new Input() { Data = new Data() { Image = new Image() { Base64 = "{YOUR_IMAGE_BYTES_STRING}", AllowDuplicateUrl = true // optional } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Upload Multiple Inputs With IDs In cases where you have your own `id` and you only have one item per image, you are encouraged to send inputs with your own `id`. This will help you later match the input to your own database. If you do not send an `id`, one will be created for you. If you have more than one item per image, it is recommended that you put the product `id` in the metadata. {PythonAddMultipleInputsIds} {JSAddMultipleInputsIds} {NodeAddMultipleInputsIds} {JavaAddMultipleInputsIds} {PHPAddMultipleInputsIds} {CurlAddMultipleInputsIds} ```csharp var response = client.PostInputs( new PostInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, Inputs = { new List() { new Input() { Id = "train1" Data = new Data() { Image = new Image() { Url = "https://samples.clarifai.com/metro-north.jpg", AllowDuplicateUrl = true // optional } } } }, { new Input() { Id = "puppy1" Data = new Data() { Image = new Image() { Url = "https://samples.clarifai.com/puppy.jpeg", AllowDuplicateUrl = true // optional } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Upload Inputs With Concepts You can add inputs with concepts via URLs or bytes. [Concepts](https://docs.clarifai.com/api-guide/concepts/) play an important role in creating your own models. Concepts also help you [search](https://docs.clarifai.com/api-guide/search/) for inputs. When you add a concept to an input, you need to indicate whether the concept is present in it or not. {PythonAddInputsConcepts} {JSAddInputsConcepts} {NodeAddInputsConcepts} {JavaAddInputsConcepts} {PHPAddInputsConcepts} {CurlAddInputsConcepts} ```csharp var response = client.PostInputs( new PostInputsRequest() { UserAppId = new UserAppIDSet() { UserId = "{YOUR_USER_ID}", AppId = "{YOUR_APP_ID}" }, Inputs = { new List() { new Input() { Data = new Data() { Image = new Image() { Url = "https://samples.clarifai.com/puppy.jpeg", AllowDuplicateUrl = true // optional }, Concepts = { new List { new Concept { Id = "charlie", Value = 1 } } } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); ``` --> ### Upload Inputs With Multiple Concepts You can also add an input with multiple concepts in a single API call. You can provide the concepts in a list and iterate through it. You can add the inputs via URLs or bytes. {PythonAddInputsMultipleConcepts} {JSAddInputsMultipleConcepts} {NodeAddInputsMultipleConcepts} ### Upload Inputs With Custom Metadata In addition to adding an input with concepts, you can also add an input with custom metadata. This metadata will then be searchable. Metadata can be any arbitrary JSON. If you have more than one item per image, it is recommended to put the `id` in the metadata like: ```text { "product_id": "xyz" } ``` {PythonAddInputsCustomMetadata} {JSAddInputsCustomMetadata} {NodeAddInputsCustomMetadata} {JavaAddInputsCustomMetadata} {PHPAddInputsCustomMetadata} {CurlAddInputsCustomMetadata} ## Upload Inputs From Cloud Storage You can add inputs from various cloud storage platforms, such as S3 (Amazon Simple Storage Service) and GCP (Google Cloud Platform), by simply providing their corresponding URLs. In cases where access credentials are necessary, you can include them as part of the request. This simplifies the process of adding inputs to our platform, offering a more efficient alternative to the conventional method of using the **PostInputs** endpoint for users who already have data stored in the cloud platforms. :::note This functionality has been introduced starting from the [10.1 release](https://docs.clarifai.com/product-updates/changelog/release101#api). ::: :::info - Image files stored in the cloud platforms will be treated as image inputs, video files as video inputs, etc. Archives will be extracted, and their contents will also be processed like this. - We do not support extraction of archives located inside other archives. - The cloud URL will serve as a filter prefix. For instance, in the case of an S3 URL like `s3:/bucket/images_folder/abc`, files within the `images_folder` will be processed starting with `abc`, or within a subfolder beginning with `abc`. For example, files such as `bucket/images_folder/abcImage.png` or `bucket/images_folder/abc-1/Data.zip` will be processed accordingly. ::: ### Upload Inputs via Cloud Storage URLs Below is an example of pulling inputs from a subfolder of an S3 bucket. import PythonCloudStorageURLs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.py"; import JSCloudStorageURLs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.html"; import NodeCloudStorageURLs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.js"; import JavaCloudStorageURLs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.java"; import PHPCloudStorageURLs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.php"; import CurlCloudStorageURLs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.sh"; import PythonTrackUploadProcess from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.py"; import JSTrackUploadProcess from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.html"; import NodeTrackUploadProcess from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.js"; import JavaTrackUploadProcess from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.java"; import PHPTrackUploadProcess from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.php"; import CurlTrackUploadProcess from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.sh"; import PythonListInputsJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.py"; import JSListInputsJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.html"; import NodeListInputsJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.js"; import JavaListInputsJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.java"; import PHPListInputsJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.php"; import CurlListInputsJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.sh"; import PythonCancelJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cancel_jobs.py"; import JSCancelJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cancel_jobs.html"; import NodeCancelJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cancel_jobs.js"; import JavaCancelJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cancel_jobs.java"; import PHPCancelJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cancel_jobs.php"; import CurlCancelJobs from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cancel_jobs.sh"; import PythonConceptsDatasets from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.py"; import JSConceptsDatasets from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.html"; import NodeConceptsDatasets from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.js"; import JavaConceptsDatasets from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.java"; import PHPConceptsDatasets from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.php"; import CurlConceptsDatasets from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.sh"; import OutputExample1 from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/cloud-storage-urls.txt"; import OutputExample2 from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/track-upload-process.txt"; import OutputExample3 from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/list_inputs_jobs.txt"; import OutputExample4 from "!!raw-loader!../../../../code_snippets/api-guide/data/cloud-storage/concepts_datasets.txt"; {PythonCloudStorageURLs} {JSCloudStorageURLs} {NodeCloudStorageURLs} {JavaCloudStorageURLs} {PHPCloudStorageURLs} {CurlCloudStorageURLs} Output Example {OutputExample1} ### Track Upload Process After starting to pull the inputs from a cloud storage service, you can track the progress of the exercise. Note that we’ll use the `inputs_extraction_job_id` returned after running the extraction job. {PythonTrackUploadProcess} {JSTrackUploadProcess} {NodeTrackUploadProcess} {JavaTrackUploadProcess} {PHPTrackUploadProcess} {CurlTrackUploadProcess} Output Example {OutputExample2} ### List Inputs Extraction Jobs You can list all your inputs extraction jobs and get their details. {PythonListInputsJobs} {JSListInputsJobs} {NodeListInputsJobs} {JavaListInputsJobs} {PHPListInputsJobs} {CurlListInputsJobs} Output Example {OutputExample3} ### Cancel Extraction Jobs You can cancel the process of extraction of inputs from a cloud storage service. Note that we’ll use the `inputs_extraction_job_id` returned after starting the extraction process. {PythonCancelJobs} {JSCancelJobs} {NodeCancelJobs} {JavaCancelJobs} {PHPCancelJobs} {CurlCancelJobs} ### Upload Inputs With Concepts and Datasets You can also add inputs from cloud storage platforms while attaching relevant concepts, assigning them to an already existing [dataset](https://docs.clarifai.com/api-guide/data/datasets/dataset-basics), or adding other metadata information to them. The `input_template` parameter allows you to do that. {PythonConceptsDatasets} {JSConceptsDatasets} {NodeConceptsDatasets} {JavaConceptsDatasets} {PHPConceptsDatasets} {CurlConceptsDatasets} Output Example {OutputExample4} --- ## Upload via UI # Upload via UI **Add data to the Clarifai platform via the UI** To upload inputs, navigate to your individual app's page and select the **Inputs** option in the collapsible left sidebar. Then, click the **Upload Inputs** button located in the upper-right corner of the page. The inputs uploader window that pops up allows you to upload any type of input data — files from your local directory, texts, or publicly accessible URLs. Note that you can also use the inputs uploader modal to: - Organize your inputs into [datasets](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete). - [Add concepts to inputs](https://docs.clarifai.com/portal-guide/inputs-manager/concepts#via-the-inputs-uploader). - Attach JSON metadata for additional context. Metadata are additional pieces of information you attach to your inputs when uploading them to the Clarifai platform. This can include product IDs, user IDs, or any other relevant details necessary for achieving specific outcomes This modal simplifies the process of managing and enriching your inputs on the platform. ## Upload Files ![](/img/community_2/data_upload_inputs.png) :::tip [Click here](https://docs.clarifai.com/portal-guide/advanced-topics/csv-and-tsv/) to learn how to upload data using files in **.csv** or **.tsv** formats. ::: To upload files containing any supported data type from your local directory, select the **Files** tab in the inputs uploader window. Then, click the upload button to select your files or drag and drop them directly into the designated area. > If you want to make multiple uploads without closing the uploader window, select the **Keep window open for multiple uploads** checkbox. After uploading your inputs, click the **Upload Inputs** button located in the lower section of the uploader window. ## Upload Texts ![](/img/community_2/data_upload_inputs-4.png) To upload text data directly through the UI, select the **Text** tab in the inputs uploader window and enter your text. Each input can contain a maximum of 500 words. ## Upload URLs ![](/img/community_2/data_upload_inputs-5.png) To upload a URL containing any supported data type, select the **URL** tab in the inputs uploader window and enter each URL on a separate line. > If you want to allow uploading identical URLs, toggle the **Allow duplicate URLs** button on. ## Uploads Status The **Uploads** status window appears in the lower-right section of the page, enabling you to monitor the percentage progress of your upload. You can check the progress of any of your active uploads in the **Active** tab. ![](/img/community_2/data_upload_inputs-2.png) Once the upload process is complete, the **Inactive** tab will display a **Complete** status. :::note If you select the **Refresh when jobs finish** checkbox, the window will automatically refresh to display the status as soon as the upload process is finished. You'll also be notified if there is an issue with any of your inputs during uploading. ::: ![](/img/community_2/data_upload_inputs-3.png) --- ## Labeling # Labeling **Assign relevant tags or keywords to inputs** Labeling, also known as annotation, is the process of assigning one or more relevant tags or keywords — commonly referred to as [concepts](https://docs.clarifai.com/create-manage/concepts/) — to describe the attributes or content of your inputs accurately. For example, annotations might indicate whether an image contains a jogger or a cyclist, which words were spoken in a recorded audio file, or if a concrete block contains cracks. Successfully labeling data is a key ingredient to any custom AI solution. You can use a concept to annotate an input if that input has that entity. That’s how you prepare training data to teach your models to recognize new concepts. After training your model using the annotated concepts, the model can learn to recognize them when it encounters data without those tags. Clarifai offers custom tools for labeling image, text, video, and audio inputs, as well as delegate and manage labeling projects of any size. ![](/img/scribe.svg) import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Labeling via the API # Labeling via the API **Learn how to use our API to easily label your data** import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Delete Annotations # Delete Annotations :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ### Delete Annotation by Input ID and Annotation ID Below is an example of how to delete a single annotation by input ID and annotation ID. {PythonDeleteAnnotationInputAnnotationIds} {JSDeleteAnnotationInputAnnotationIds} {NodeDeleteAnnotationInputAnnotationIds} {JavaDeleteAnnotationInputAnnotationIds} {PHPDeleteAnnotationInputAnnotationIds} {CurlDeleteAnnotationInputAnnotationIds} ### Bulk Delete Annotations by Input IDs and Annotation IDs You can delete multiple annotations in one API call. You need to provide a list of input IDs and a list of annotation IDs. The number of input IDs has to match the number of annotation IDs. Below is an example of how to do that. {PythonbulkDeleteAnnotationsInputAnnotationIds} {JSbulkDeleteAnnotationsInputAnnotationIds} {NodebulkDeleteAnnotationsInputAnnotationIds} {JavabulkDeleteAnnotationsInputAnnotationIds} {PHPbulkDeleteAnnotationsInputAnnotationIds} {CurlbulkDeleteAnnotationsInputAnnotationIds} ### Bulk Delete All Annotations by Input IDs To delete all annotations of a given input, you just need to set their input ID\(s\). This will delete all annotations for these input\(s\), EXCEPT the input level annotations, which only get deleted if you delete the inputs themselves. Below is an example of how to do that. {PythonBulkDeleteAnnotationsInputIds} {JSBulkDeleteAnnotationsInputIds} {NodeBulkDeleteAnnotationsInputIds} {JavaBulkDeleteAnnotationsInputIds} {PHPBulkDeleteAnnotationsInputIds} {CurlBulkDeleteAnnotationsInputIds} import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonbulkDeleteAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/bulk_delete_annotations_input_annotation_ids.py"; import PythonBulkDeleteAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/bulk_delete_annotations_input_ids.py"; import PythonDeleteAnnotationInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/delete_annotation_input_annotation_ids.py"; import JSbulkDeleteAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/bulk_delete_annotations_input_annotation_ids.html"; import JSBulkDeleteAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/bulk_delete_annotations_input_ids.html"; import JSDeleteAnnotationInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/delete_annotation_input_annotation_ids.html"; import NodebulkDeleteAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/bulk_delete_annotations_input_annotation_ids.js"; import NodeBulkDeleteAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/bulk_delete_annotations_input_ids.js"; import NodeDeleteAnnotationInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/delete_annotation_input_annotation_ids.js"; import JavabulkDeleteAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/bulk_delete_annotations_input_annotation_ids.java"; import JavaBulkDeleteAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/bulk_delete_annotations_input_ids.java"; import JavaDeleteAnnotationInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/delete_annotation_input_annotation_ids.java"; import PHPbulkDeleteAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/bulk_delete_annotations_input_annotation_ids.php"; import PHPBulkDeleteAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/bulk_delete_annotations_input_ids.php"; import PHPDeleteAnnotationInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/delete_annotation_input_annotation_ids.php"; import CurlbulkDeleteAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/bulk_delete_annotations_input_annotation_ids.sh"; import CurlBulkDeleteAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/bulk_delete_annotations_input_ids.sh"; import CurlDeleteAnnotationInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/delete_annotation_input_annotation_ids.sh"; import PythonAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_mask.py"; import JavascriptAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_mask.html"; import NodeAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_mask.js"; import JavaAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_mask.java"; import PHPAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_mask.php"; import CURLAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_mask.sh"; --- ## List Annotations # List Annotations You can get a list of annotations within your app with a GET call. Annotations will be returned from oldest to newest. These requests are [paginated](https://docs.clarifai.com/resources/api-overview/pagination/). By default each page will return 20 annotations. ### List All User Created Annotations in Your App Below is an example of how to list all your user labelled annotations. :::important note This will not show annotations by models in your workflow. To include model created annotations, you need to set `list_all_annotations` to `True`. ::: {PythonlistUserCreatedAnnotationsApp} {JSlistUserCreatedAnnotationsApp} {NodelistUserCreatedAnnotationsApp} {JavalistUserCreatedAnnotationsApp} {PHPlistUserCreatedAnnotationsApp} {CurllistUserCreatedAnnotationsApp} ### List All Annotations in Your App Below is an example of how to list all annotations, including those created by models. {PythonListAllAnnotationsApp} {JSListAllAnnotationsApp} {NodeListAllAnnotationsApp} {JavaListAllAnnotationsApp} {PHPListAllAnnotationsApp} {CurlListAllAnnotationsApp} ### List User Created Annotations by Input IDs Below is an example of how to list all user created annotations for certain input \(one or several\) by providing a list of input IDs. :::important note This will not show annotations by models in your workflow. To include model created annotations, you need to set `list_all_annotations` to `True`. ::: {PythonListUserCreatedAnnotationsInputIds} {JSListUserCreatedAnnotationsInputIds} {NodeListUserCreatedAnnotationsInputIds} {JavaListUserCreatedAnnotationsInputIds} {PHPListUserCreatedAnnotationsInputIds} {CurlListUserCreatedAnnotationsInputIds} ### List Annotations by Input IDs and Annotation IDs You can list annotations by input IDs and their corresponding annotation IDs. Number of input IDs and annotation IDs should be the same. Since we are finding annotation by IDs, this will match any user or model created annotations. Below is an example of how to do that. :::tip - When listing annotations, both input IDs and annotation IDs are optional. If you do not provide any input ID or annotation ID, we will return all annotations based on the creation time of each input. - You can also list annotations by providing input IDs only. - However, if you want to list annotations by providing annotation IDs, then input IDs are also required so that we know the inputs that correspond to the annotation IDs provided in the request. In this case, the number of input IDs should be equal to the number of annotation IDs. ::: {PythonListAnnotationsInputAnnotationIds} {JSListAnnotationsInputAnnotationIds} {NodeListAnnotationsInputAnnotationIds} {JavaListAnnotationsInputAnnotationIds} {PHPListAnnotationsInputAnnotationIds} {CurlListAnnotationsInputAnnotationIds} ### List Annotations by User IDs An annotation is created by either a user or a model. You can list annotations created by specific user\(s\) by providing their user IDs. Below is an example of how to do that. {PythonListAnnotationsUserIds} {JSListAnnotationsUserIds} {NodeListAnnotationsUserIds} {JavaListAnnotationsUserIds} {PHPListAnnotationsUserIds} {CurlListAnnotationsUserIds} ### List Annotations by Model Version IDs An annotation is created by either a user or a model. For example, if your workflow has a detection model, when you add an input, the model will detect objects in your input. You can see these detected objects by listing the annotations created in the detection model. You can also label these regions by using `Post annotation` with the region ID returned from this call. Below is an example of how to list annotations by model version IDs. {PythonListAnnotationsModelVersionIds} {JSListAnnotationsModelVersionIds} {NodeListAnnotationsModelVersionIds} {JavaListAnnotationsModelVersionIds} {PHPListAnnotationsModelVersionIds} {CurlListAnnotationsModelVersionIds} import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonListAllAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/list_all_annotations_app.py"; import PythonListAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/list_annotations_input_annotation_ids.py"; import PythonListAnnotationsModelVersionIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/list_annotations_model_version_ids.py"; import PythonListAnnotationsUserIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/list_annotations_user_ids.py"; import PythonlistUserCreatedAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/list_user_created_annotations_app.py"; import PythonListUserCreatedAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/list_user_created_annotations_input_ids.py"; import JSListAllAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/list_all_annotations_app.html"; import JSListAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/list_annotations_input_annotation_ids.html"; import JSListAnnotationsModelVersionIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/list_annotations_model_version_ids.html"; import JSListAnnotationsUserIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/list_annotations_user_ids.html"; import JSlistUserCreatedAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/list_user_created_annotations_app.html"; import JSListUserCreatedAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/list_user_created_annotations_input_ids.html"; import NodeListAllAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/list_all_annotations_app.js"; import NodeListAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/list_annotations_input_annotation_ids.js"; import NodeListAnnotationsModelVersionIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/list_annotations_model_version_ids.js"; import NodeListAnnotationsUserIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/list_annotations_user_ids.js"; import NodelistUserCreatedAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/list_user_created_annotations_app.js"; import NodeListUserCreatedAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/list_user_created_annotations_input_ids.js"; import JavaListAllAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/list_all_annotations_app.java"; import JavaListAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/list_annotations_input_annotation_ids.java"; import JavaListAnnotationsModelVersionIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/list_annotations_model_version_ids.java"; import JavaListAnnotationsUserIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/list_annotations_user_ids.java"; import JavalistUserCreatedAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/list_user_created_annotations_app.java"; import JavaListUserCreatedAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/list_user_created_annotations_input_ids.java"; import PHPListAllAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/list_all_annotations_app.php"; import PHPListAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/list_annotations_input_annotation_ids.php"; import PHPListAnnotationsModelVersionIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/list_annotations_model_version_ids.php"; import PHPListAnnotationsUserIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/list_annotations_user_ids.php"; import PHPlistUserCreatedAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/list_user_created_annotations_app.php"; import PHPListUserCreatedAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/list_user_created_annotations_input_ids.php"; import CurlListAllAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/list_all_annotations_app.sh"; import CurlListAnnotationsInputAnnotationIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/list_annotations_input_annotation_ids.sh"; import CurlListAnnotationsModelVersionIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/list_annotations_model_version_ids.sh"; import CurlListAnnotationsUserIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/list_annotations_user_ids.sh"; import CurllistUserCreatedAnnotationsApp from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/list_user_created_annotations_app.sh"; import CurlListUserCreatedAnnotationsInputIds from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/list_user_created_annotations_input_ids.sh"; import PythonAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_mask.py"; import JavascriptAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_mask.html"; import NodeAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_mask.js"; import JavaAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_mask.java"; import PHPAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_mask.php"; import CURLAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_mask.sh"; --- ## Update Annotations # Update Annotations Changing annotation data is possible by PATCHing existing annotations. The application owner can change any user-created annotations. Collaborators are not allowed to change annotations made by other collaborators. Generally speaking, you should send PATCH when you want to change the data you have posted; for example, changing the concept from positive to negative or adjusting the bounding box coordinates. If you want to add more tags, you can always POST new annotations. There is no limit on how many annotations an input can have. Update supports `overwrite`, `merge`, and `remove` actions. You can update from 1 up to 128 annotations in a single API call. ### Update Annotation With Concepts Below is an example of how to update an annotation of an image with a new concept, or change a concept value from true to false \(or vice versa\). {PythonUpdateAnnotationConcepts} {JSUpdateAnnotationConcepts} {NodeUpdateAnnotationConcepts} {JavaUpdateAnnotationConcepts} {PHPUpdateAnnotationConcepts} {CurlUpdateAnnotationConcepts} ### Update Annotation With Concepts in a Region When you update region data, you must nest this new data within `region.data`. Set the `region_id` to the current `region_id` if you do not want to change or remove this region. Below is an example of how to update annotation with concepts in a region. {PythonUpdateAnnotationConceptsRegion} {JSUpdateAnnotationConceptsRegion} {NodeUpdateAnnotationConceptsRegion} {JavaUpdateAnnotationConceptsRegion} {PHPUpdateAnnotationConceptsRegion} {CurlUpdateAnnotationConceptsRegion} ### Update Annotation Region Coordinates You can update region bounding boxes coordinates. When changing the region, you should use `overwrite` action. With `overwrite` action, you need to provide the data you want to keep in this annotation. Below is an example of how to do that. {PythonUpdateAnnotationRegionCoordinates} {JSUpdateAnnotationRegionCoordinates} {NodeUpdateAnnotationRegionCoordinates} {JavaUpdateAnnotationRegionCoordinates} {PHPUpdateAnnotationRegionCoordinates} {CurlUpdateAnnotationRegionCoordinates} ### Update Annotation Status Below is an example of how to update an annotation status. {PythonUpdateAnnotationStatus} {JSUpdateAnnotationStatus} {NodeUpdateAnnotationStatus} {JavaUpdateAnnotationStatus} {PHPUpdateAnnotationStatus} {CurlUpdateAnnotationStatus} import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonUpdateAnnotationConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/update_annotation_concepts.py"; import PythonUpdateAnnotationConceptsRegion from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/update_annotation_concepts_region.py"; import PythonUpdateAnnotationRegionCoordinates from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/update_annotation_region_coordinates.py"; import PythonUpdateAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/update_annotation_status.py"; import JSUpdateAnnotationConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/update_annotation_concepts.html"; import JSUpdateAnnotationConceptsRegion from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/update_annotation_concepts_region.html"; import JSUpdateAnnotationRegionCoordinates from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/update_annotation_region_coordinates.html"; import JSUpdateAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/update_annotation_status.html"; import NodeUpdateAnnotationConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/update_annotation_concepts.js"; import NodeUpdateAnnotationConceptsRegion from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/update_annotation_concepts_region.js"; import NodeUpdateAnnotationRegionCoordinates from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/update_annotation_region_coordinates.js"; import NodeUpdateAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/update_annotation_status.js"; import JavaUpdateAnnotationConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/update_annotation_concepts.java"; import JavaUpdateAnnotationConceptsRegion from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/update_annotation_concepts_region.java"; import JavaUpdateAnnotationRegionCoordinates from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/update_annotation_region_coordinates.java"; import JavaUpdateAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/update_annotation_status.java"; import PHPUpdateAnnotationConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/update_annotation_concepts.php"; import PHPUpdateAnnotationConceptsRegion from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/update_annotation_concepts_region.php"; import PHPUpdateAnnotationRegionCoordinates from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/update_annotation_region_coordinates.php"; import PHPUpdateAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/update_annotation_status.php"; import CurlUpdateAnnotationConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/update_annotation_concepts.sh"; import CurlUpdateAnnotationConceptsRegion from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/update_annotation_concepts_region.sh"; import CurlUpdateAnnotationRegionCoordinates from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/update_annotation_region_coordinates.sh"; import CurlUpdateAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/update_annotation_status.sh"; import PythonAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_mask.py"; import JavascriptAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_mask.html"; import NodeAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_mask.js"; import JavaAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_mask.java"; import PHPAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_mask.php"; import CURLAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_mask.sh"; --- ## Create Annotations # Create Annotations **Label your data** Annotations \(also known as labels\) describe your inputs. When you add inputs to your app, we will create an input level annotation for each input. This input level annotation contains any data you provided in `POST /inputs` call. Models in your default workflow can also write annotations. Once your input is successfully indexed, you can add additional annotations, such as concepts and bounding boxes. You can label your inputs by calling the `POST /annotations` endpoint. For example, you can add concept\(s\) to an image, draw a bounding box, or label concept\(s\) in a video frame. When you add an annotation, the app's default workflow will be run by default. This means that any newly added annotations will be immediately available for AI based search and training. :::tip You can add from 1 up to 128 annotations in a single API call. ::: Each annotation should contain at most one region. If it is a video, each annotation should contain 1 frame. If there are multiple regions in a frame you want to label, you can add multiple annotations for each region and each annotation will be contained within the same frame but in a different region. ### Annotate Images With Concepts Below is an example of how to annotate a concept present anywhere in an image. {PythonAnnotateImagesConcepts} {JSAnnotateImagesConcepts} {NodeAnnotateImagesConcepts} {JavaAnnotateImagesConcepts} {PHPAnnotateImagesConcepts} {CurlAnnotateImagesConcepts} ### Annotate Images With Multiple Concepts Below is an example of how to annotate an image with multiple concepts in a single API call. You can provide the concepts in a list and iterate through it. {PythonAnnotateImagesMultipleConcepts} {JSAnnotateImagesMultipleConcepts} {NodeAnnotateImagesMultipleConcepts} ### Annotate New Bounding Boxes in an Image Below is an example of how to label a new rectangular bounding box for a region. These are the bounding box coordinates you need to provide: - **top_row**—The top left of the bounding box normalized to the data dimension to be within [0-1.0]; - **left_col**—The left column of the bounding box normalized to the data dimension to be within [0-1.0]; - **bottom_row**—The bottom row of the bounding box normalized to the data dimension to be within [0-1.0]; - **right_col**—The right col of the bounding box normalized to the data dimension to be within [0-1.0]. {PythonAnnotateNewBoundingBoxesImage} {JSAnnotateNewBoundingBoxesImage} {NodeAnnotateNewBoundingBoxesImage} {JavaAnnotateNewBoundingBoxesImage} {PHPAnnotateNewBoundingBoxesImage} {CurlAnnotateNewBoundingBoxesImage} ### Annotate Polygons in an Image Below is an example of how to provide annotations within any polygon-shaped region of an image. These are the list of points that connect together to form a polygon: - **row**—The row location of the point. This has a [0.0-1.0] range with 0.0 being top row and 1.0 being the bottom row; - **col**—The column location of the point. This has a [0.0-1.0] range with 0.0 being left col and 1.0 being the right col; - **z**—Depth, if applicable, of the point. {PythonAnnotatePolygonsImage} {JSAnnotatePolygonsImage} {NodeAnnotatePolygonsImage} {JavaAnnotatePolygonsImage} {PHPAnnotatePolygonsImage} {CurlAnnotatePolygonsImage} ### Annotate Existing Regions in an Image When you add an input, detection models \(such as `Face Detection` or `General Detection`\) will detect regions in your image where there appear to be relevant objects. You can get the IDs of these detected regions by [listing model's annotations](https://docs.clarifai.com/api-guide/annotate/annotations#list-annotations). Your labels should be contained within `Region.data`. Each annotation can only have 1 region. If you want to label multiple regions, it is possible to label multiple annotations in a single API call. Below is an example of how to annotate existing regions in an image. {PythonAnnotateExistingRegionsImage} {JSAnnotateExistingRegionsImage} {NodeAnnotateExistingRegionsImage} {JavaAnnotateExistingRegionsImage} {PHPAnnotateExistingRegionsImage} {CurlAnnotateExistingRegionsImage} ### Annotate Images With Mask Below is an example of how to add a mask to an Image using a single API call. In this example, we provide an image mask as a `base64` string. Click [here](https://docs.clarifai.com/portal-guide/annotate/label-types/#how-to-choose-a-label-type) to learn more about image mask annotations. {PythonAnnotateMask} {JavascriptAnnotateMask} {NodeAnnotateMask} {JavaAnnotateMask} {PHPAnnotateMask} {CURLAnnotateMask} ### Annotate Images With Different `user_id` and `status` Each annotation is tied to a user or a model in your workflow. By default, when a user posts an annotation, this user is the owner of the annotation. Sometimes, however, you might want to post an annotation as another user; for example, when assigning an image to another user. In such a case, you can create an annotation with another `user_id` \(and status `PENDING`\). :::important note Only the app owner can post an annotation with other user's `user_id`; collaborators cannot. ::: Below is an example of how to annotate images with different `user_id` and `status`. {PythonAnnotateImagesUserIdStatus} {JSAnnotateImagesUserIdStatus} {NodeAnnotateImagesUserIdStatus} {JavaAnnotateImagesUserIdStatus} {PHPAnnotateImagesUserIdStatus} {CurlAnnotateImagesUserIdStatus} import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonAnnotateExistingRegionsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_existing_regions_image.py"; import PythonAnnotateImagesConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_images_concepts.py"; import PythonAnnotateImagesMultipleConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_images_multiple_concepts.py"; import PythonAnnotateImagesUserIdStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_images_user_id_status.py"; import PythonAnnotateNewBoundingBoxesImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_new_bounding_boxes_image.py"; import PythonAnnotatePolygonsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_polygons_image.py"; import JSAnnotateExistingRegionsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_existing_regions_image.html"; import JSAnnotateImagesConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_images_concepts.html"; import JSAnnotateImagesMultipleConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_images_multiple_concepts.html"; import JSAnnotateImagesUserIdStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_images_user_id_status.html"; import JSAnnotateNewBoundingBoxesImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_new_bounding_boxes_image.html"; import JSAnnotatePolygonsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_polygons_image.html"; import NodeAnnotateExistingRegionsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_existing_regions_image.js"; import NodeAnnotateImagesConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_images_concepts.js"; import NodeAnnotateImagesMultipleConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_images_multiple_concepts.js"; import NodeAnnotateImagesUserIdStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_images_user_id_status.js"; import NodeAnnotateNewBoundingBoxesImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_new_bounding_boxes_image.js"; import NodeAnnotatePolygonsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_polygons_image.js"; import JavaAnnotateExistingRegionsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_existing_regions_image.java"; import JavaAnnotateImagesConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_images_concepts.java"; import JavaAnnotateImagesUserIdStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_images_user_id_status.java"; import JavaAnnotateNewBoundingBoxesImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_new_bounding_boxes_image.java"; import JavaAnnotatePolygonsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_polygons_image.java"; import PHPAnnotateExistingRegionsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_existing_regions_image.php"; import PHPAnnotateImagesConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_images_concepts.php"; import PHPAnnotateImagesUserIdStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_images_user_id_status.php"; import PHPAnnotateNewBoundingBoxesImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_new_bounding_boxes_image.php"; import PHPAnnotatePolygonsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_polygons_image.php"; import CurlAnnotateExistingRegionsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_existing_regions_image.sh"; import CurlAnnotateImagesConcepts from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_images_concepts.sh"; import CurlAnnotateImagesUserIdStatus from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_images_user_id_status.sh"; import CurlAnnotateNewBoundingBoxesImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_new_bounding_boxes_image.sh"; import CurlAnnotatePolygonsImage from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_polygons_image.sh"; import PythonAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/annotate_mask.py"; import JavascriptAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/annotate_mask.html"; import NodeAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/annotate_mask.js"; import JavaAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/annotate_mask.java"; import PHPAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/annotate_mask.php"; import CURLAnnotateMask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/annotate_mask.sh"; --- ## Positive and Negative Annotations # Positive and Negative Annotations **Learn how to make positive and negative annotations** Positive and negative annotations play a critical role in training machine learning models. Positive annotations help the model identify and learn the features associated with a particular input. Negative annotations, on the other hand, help the model distinguish between similar inputs by highlighting what the input is not. Some inputs may also have overlapping features, especially in complex datasets. Providing both positive and negative annotations reduces the risk of the model misclassifying similar but distinct inputs, leading to more precise predictions. By annotating your data with both positive and negative labels, you provide your model with the comprehensive information it needs to learn effectively, which results in better performance and more reliable outcomes. :::warning value - The `value` parameter is used to specify the presence of a concept in a request. A `value` of 1.0 indicates that the concept is present (true), while a `value` of 0.0 indicates that the concept is not present (false). - When making HTTP requests, if the `value` parameter is omitted, it defaults to a positive annotation. In contrast, for gRPC requests, a missing `value` is interpreted as a negative annotation. To avoid unintended behavior, it's recommended to always explicitly set the `value` parameter in your requests. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonPositive from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/positive_negative_1.py"; import JSPositive from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/positive_negative_1.html"; import NodePositive from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/positive_negative_1.js"; import JavaPositive from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/positive_negative_1.java"; import PHPPositive from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/positive_negative_1.php"; import CurlPositive from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/positive_negative_1.sh"; import PythonNegative from "!!raw-loader!../../../../code_snippets/api-guide/annotate/py/positive_negative_2.py"; import JSNegative from "!!raw-loader!../../../../code_snippets/api-guide/annotate/js/positive_negative_2.html"; import NodeNegative from "!!raw-loader!../../../../code_snippets/api-guide/annotate/node/positive_negative_2.js"; import JavaNegative from "!!raw-loader!../../../../code_snippets/api-guide/annotate/java/positive_negative_2.java"; import PHPNegative from "!!raw-loader!../../../../code_snippets/api-guide/annotate/php/positive_negative_2.php"; import CurlNegative from "!!raw-loader!../../../../code_snippets/api-guide/annotate/curl/positive_negative_2.sh"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ## Positive Annotation If an input is tagged with a positive annotation, it is considered a positive label for it. {PythonPositive} {JSPositive} {NodePositive} {JavaPositive} {PHPPositive} {CurlPositive} ## Negative Annotation If an input is tagged with a negative annotation, it is treated as a negative label for that concept. It emphasizes that the input does not belong to that concept. {PythonNegative} {JSNegative} {NodeNegative} {JavaNegative} {PHPNegative} {CurlNegative} --- ## Tasks : Create, Get, Update, Delete # Tasks : Create, Get, Update, Delete **Group your labeling work into tasks that can be delegated** Task is the work that needs to be done for labeling the inputs in an app. It's a powerful way to help your team annotate inputs fast and efficiently. These are some parameters you can specify when working with tasks: - **Task type** — It can be: - `CONCEPTS_CLASSIFICATION` — Concepts classification tasks annotate [concepts](https://docs.clarifai.com/api-guide/concepts/) for the overall image, frame of video, or section of text. - `BOUNDING_BOX_DETECTION` — Bounding box detection tasks annotate rectangular bounding box regions around each concept in an image, frame of video, or section of text. - `POLYGON_DETECTION` — Polygon detection tasks annotate free-form regions around concepts in an image, frame of video, or section of text. - `TYPE_NOT_SET` — This is the default task type. It should be used when creating an auto-annotation task. - **Worker** — Task worker includes information about the workers who will work on the task. For manual labeling tasks, the workers can only be users; no limitation on number of workers. For auto-annotation tasks, the worker can be either a model or a workflow; currently only supports 1 worker. - **Concepts** — List of concept IDs used in the work on the task. The concepts should already be existing in your app. - **Task worker strategy** — It can be: - `DYNAMIC` — Each worker will dynamically get 10 inputs assigned at a time. No inputs are assigned at task creation. It's the recommended way to set a task worker strategy. - `PARTITIONED` — The inputs will be partitioned in several partitions. Each worker will label one or more input partitions. All inputs are assigned at task creation. - `FULL` — Each worker will label all inputs from the input source. All inputs are assigned at task creation. - **Input source** — It can be: - `ALL_INPUTS` — Use all inputs in the app. - `DATASET` — Use inputs from a [dataset](https://docs.clarifai.com/api-guide/data/datasets/). - **[`sample_ms`](https://docs.clarifai.com/api-guide/predict/video#configuring-fps)** — Used in video model predictions. It specifies the sample delay for video predictions (1 frame per N milliseconds). - **Review strategy** — It can be: - `NONE` — No review is needed. - `MANUAL` — Manual review strategy. - `CONSENSUS` — Consensus review strategy. - **Partitioned strategy info** — It can be: - `EVENLY` — Each worker will label (approximately) the same number of inputs. - `WEIGHTED` — Each worker will have an assigned weight. - **Workers per input** — The number of workers who will label each input. - **Auto-annotation config** - The concepts configurations for setting up an auto-annotation labeling task using a model or a workflow. You can set: - `annotation_data_types` — An integer for filtering annotations by their annotation data type. It's a bit-mask field that holds multiple annotation data type values that are combined in an OR fashion. For example, if `annotation_data_types = 34`, then we filter annotations that appear as a mask or a bounding box, because `MASK = 32` and `BOUNDING_BOX = 2`. You can look for the various annotation data types values [here](https://github.com/Clarifai/clarifai-go-grpc/blob/master/proto/clarifai/api/resources.pb.go). For example, `annotation_data_types=1` corresponds to `AnnotationDataType_TAG`. - `threshold_range` — It specifies a range of predictions values based on the lower and upper bounds, and it defines whether these bounds are inclusive or exclusive. For example, if you set `is_lower_inclusive = true`, `is_upper_inclusive = true`, `lower = 0.7` and `upper = 1.0`, it is interpreted as the prediction range includes all values from 0.7 to 1.0, including both 0.7 and 1.0. - `status_code` — It specifies the [code](https://docs.clarifai.com/api-guide/advanced-topics/status-codes#annotation-related-codes-24xxx) related to the status of the annotation. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PyAssignedTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/assigned_task.py"; import PyTaskPartitionedWorkerStrategy from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_partitioned_worker_strategy.py"; import PyConsensusReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_consensus_review.py"; import PyGetTaskByID from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/get_task_by_id.py"; import PyListAllTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_all_tasks.py"; import PyListTasksAssignedUser from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user.py"; import PyListTasksAssignedUserReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user_review.py"; import PyUpdateTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/update_task.py"; import PyDeleteMultipleTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/delete_multiple_tasks.py"; import PyAutoAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/auto_annotation.py"; import JSAssignedTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/assigned_task.html"; import JSTaskPartitionedWorkerStrategy from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_partitioned_worker_strategy.html"; import JSConsensusReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_consensus_review.html"; import JSGetTaskByID from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/get_task_by_id.html"; import JSListAllTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_all_tasks.html"; import JSListTasksAssignedUser from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user.html"; import JSListTasksAssignedUserReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user_review.html"; import JSUpdateTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/update_task.html"; import JSDeleteMultipleTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/delete_multiple_tasks.html"; import JSAutoAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/auto_annotation.html"; import NodeAssignedTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/assigned_task.js"; import NodeTaskPartitionedWorkerStrategy from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_partitioned_worker_strategy.js"; import NodeConsensusReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_consensus_review.js"; import NodeDeleteMultipleTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/delete_multiple_tasks.js"; import NodeGetTaskByID from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/get_task_by_id.js"; import NodeListAllTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_all_tasks.js"; import NodeListTasksAssignedUser from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user.js"; import NodeListTasksAssignedUserReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user_review.js"; import NodeAutoAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/auto_annotation.js"; import JavaAssignedTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/assigned_task.java"; import JavaTaskPartitionedWorkerStrategy from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_partitioned_worker_strategy.java"; import JavaConsensusReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_consensus_review.java"; import JavaUpdateTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/update_task.java"; import JavaDeleteMultipleTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/delete_multiple_tasks.java"; import JavaGetTaskByID from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/get_task_by_id.java"; import JavaListAllTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_all_tasks.java"; import JavaListTasksAssignedUser from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user.java"; import JavaListTasksAssignedUserReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user_review.java"; import JavaAutoAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/auto_annotation.java"; import PHPTaskPartitionedWorkerStrategy from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_partitioned_worker_strategy.php"; import PHPConsensusReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_consensus_review.php"; import PHPGetTaskByID from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/get_task_by_id.php"; import PHPListAllTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_all_tasks.php"; import PHPListTasksAssignedUser from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user.php"; import PHPListTasksAssignedUserReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user_review.php"; import PHPAssignedTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/assigned_task.php"; import PHPUpdateTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/update_task.php"; import PHPDeleteMultipleTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/delete_multiple_tasks.php"; import PHPAutoAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/auto_annotation.php"; import CurlAssignedTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/assigned_task.sh"; import CurlTaskPartitionedWorkerStrategy from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_partitioned_worker_strategy.sh"; import CurlConsensusReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_consensus_review.sh"; import CurlGetTaskByID from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/get_task_by_id.sh"; import CurlListAllTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_all_tasks.sh"; import CurlListTasksAssignedUser from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user.sh"; import CurlListTasksAssignedUserReview from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/list_tasks_assigned_user_review.sh"; import CurlUpdateTask from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/update_task.sh"; import CurlDeleteMultipleTasks from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/delete_multiple_tasks.sh"; import CurlAutoAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/auto_annotation.sh"; import JSTaskAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_annotations.html"; import CurlTaskAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_annotations.sh"; import PyTaskAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_annotations.py"; import JavaTaskAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_annotations.java"; import PHPTaskAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_annotations.php"; import NodeTaskAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/annotate/tasks/task_annotations.js"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ## Create To create a new task in your app, you `POST` the task information to the `v2/task` endpoint. ### Assigned Task A task should be assigned to a list of users. These users will do the labeling work, so they're also called workers. A task may also be assigned to a list of users for review purposes. {PyAssignedTask} {JSAssignedTask} {NodeAssignedTask} {JavaAssignedTask} {PHPAssignedTask} {CurlAssignedTask} {PyTaskPartitionedWorkerStrategy} {JSTaskPartitionedWorkerStrategy} --> {NodeTaskPartitionedWorkerStrategy} --> {JavaTaskPartitionedWorkerStrategy} {PHPTaskPartitionedWorkerStrategy} {CurlTaskPartitionedWorkerStrategy} :::info * It is not required for the weights to add up to 100. For example, the weights \[9, 1\] are equivalent with weights \[90, 10\]. * The partitioning is approximate. This means that the number of assigned inputs to each worker may have a small error margin, but it will be close to the assigned weight percentage. ::: --> ### Task With Consensus Review You can also create tasks with `CONSENSUS` review strategy. When enough workers label an input in the same way, it will automatically be approved, with no need for the reviewer to spend time to check. In this way, the reviewer will be able to focus on the inputs where the workers don't agree. Note that an approval threshold must be set. It is the number of labelers that need to agree in order to automatically approve an annotation. For example, in case of 3 workers and `approval_threshold` set to 2, if an input is labeled in the same way by 2 workers, they form a majority and the group reaches a consensus. {PyConsensusReview} {JSConsensusReview} {NodeConsensusReview} {JavaConsensusReview} {PHPConsensusReview} {CurlConsensusReview} ### Auto-Annotation Task You can create an auto-annotation task and automatically label the inputs in your dataset. You need to specify a model or a workflow you want to use its predictions to automatically generate annotations or labels for your data. :::tip You can learn how to perform auto-annotation via the UI [here](https://docs.clarifai.com/portal-guide/annotate/auto-annotation). ::: {PyAutoAnnotation} {JSAutoAnnotation} {NodeAutoAnnotation} {JavaAutoAnnotation} {PHPAutoAnnotation} {CurlAutoAnnotation} ## Get ### Get Task by ID You can get the details of a single task by its ID, which is automatically generated upon task creation. You can output the API response to obtain this task ID. Alternatively, you can also locate the task ID within the [Clarifai platform's user interface](https://docs.clarifai.com/portal-guide/annotate/create-a-task#tasks-listing). {PyGetTaskByID} {JSGetTaskByID} {NodeGetTaskByID} {JavaGetTaskByID} {PHPGetTaskByID} {CurlGetTaskByID} ### List All Tasks You can get a list of tasks within your app with a `GET` call. This call supports [pagination](https://docs.clarifai.com/api-guide/advanced-topics/pagination/). {PyListAllTasks} {JSListAllTasks} {NodeListAllTasks} {JavaListAllTasks} {PHPListAllTasks} {CurlListAllTasks} ### List Tasks Assigned to User Get only the tasks assigned to a specific user for work. {PyListTasksAssignedUser} {JSListTasksAssignedUser} {NodeListTasksAssignedUser} {JavaListTasksAssignedUser} {PHPListTasksAssignedUser} {CurlListTasksAssignedUser} ### List Tasks Assigned to User for Review Get only the tasks assigned to a specific user for review. {PyListTasksAssignedUserReview} {JSListTasksAssignedUserReview} {NodeListTasksAssignedUserReview} {JavaListTasksAssignedUserReview} {PHPListTasksAssignedUserReview} {CurlListTasksAssignedUserReview} ## Update Currently, we only support updating a task by providing all information at once. ### Update Task {PyUpdateTask} {JSUpdateTask} {JavaUpdateTask} {PHPUpdateTask} {CurlUpdateTask} ## Task Annotations In order to keep track of each user's work assigned to a task, all the annotations of this user related to this task should be linked to the task ID. Therefore, when a user creates an annotation, the task ID should be provided as below: {PyTaskAnnotations} {JSTaskAnnotations} {NodeTaskAnnotations} {JavaTaskAnnotations} {PHPTaskAnnotations} {CurlTaskAnnotations} ## Delete You can delete a task by specifying its ID. You can also delete multiple tasks by specifying a list of their IDs. {PyDeleteMultipleTasks} {JSDeleteMultipleTasks} {NodeDeleteMultipleTasks} {JavaDeleteMultipleTasks} {PHPDeleteMultipleTasks} {CurlDeleteMultipleTasks} --- ## Labeling via the UI # Labeling via the UI **Label inputs on the Inputs-Manager or Input-Viewer page** ## Inputs-Manager Page If you click the **Inputs** option in your app's collapsible left sidebar, you'll be redirected to the Inputs-Manager screen, which is the main page showcasing all the inputs in your app. [Click here](https://docs.clarifai.com/create-manage/inputs/#inputs-manager-screen) to learn more about the Inputs-Manager page. ![](/img/community/annotate/create_annotations_1.png) ## Single Input-Viewer The single Input-Viewer is the main page that showcases the details of a single input available in your app. If you click an input listed on the Inputs-Manager page, you'll be redirected to the viewer page for that input, where you can view and interact with it. ![](/img/community_2/explorer_1.png) ### Key Features and Tasks You can complete several tasks on the Input-Viewer screen, including: - Use the assistive tools at the top of the page to manually label your inputs. Learn more about them [below](#manual-labeling-tools). - Generate annotations for your inputs automatically using AI-assisted labeling. Learn more about it [here](https://docs.clarifai.com/portal-guide/annotate/ai-assist). - Use the **Search concepts** field in the right sidebar to find specific concepts and display their annotations on the page. - Use the film strip at the left sidebar to swiftly move from one input to another. You can scroll through to get the specific image you want to annotate. If you click any image on the sidebar, it'll appear on the main workspace. - Set the page's mode to **Predict**, and get predictions from your models and workflows. Learn more about it [here](https://docs.clarifai.com/portal-guide/input-viewer/predictions). - Click the **View Metadata** button in the lower-left section to see details of the input, such as input ID, creation date, and resolution. You can download the original asset directly from the page. If you right-click on the canvas area, a button will appear, which allows you to download the original input. ![](/img/community_2/explorer_1-2.png) The Input-Viewer page offers a variety of keyboard shortcuts, or hotkeys, to streamline and speed up the labeling process. To view the available shortcuts for canvas-related activities, click the keyboard icon in the upper-right corner of the page. Additionally, there are other hotkeys you can use in the labeling process, which will be demonstrated later. ![](/img/others-2/annotate-10.png) ### Manual Labeling Tools The single Input-Viewer page has various assistive tools to help you make the most of the labeling exercise. #### Select / Edit Tool The **Select / Edit** tool allows you to navigate and interact with the interface. You can use it to select already defined bounding boxes or polygons. The keyboard shortcut for using this tool is **V**. ![](/img/others-2/annotate-1.png) #### Bounding Box Tool The **Bounding Box** tool allows you to draw a rectangle on an image to define a region of interest. It's useful for identifying objects in images. The keyboard shortcut for using this tool is **B**. ![](/img/others-2/annotate-2.png) #### Polygon Tool The **Polygon** tool allows you to draw custom, multi-sided shapes on an image. It's ideal for annotating irregularly shaped areas or objects. The keyboard shortcut for using this tool is **P**. ![](/img/others-2/annotate-4.png) #### Hand Tool The **Hand** allows you to click and drag an image in order to pan or move it around the workspace. It's useful for examining different areas of a large or zoomed-in image without changing the zoom level. The keyboard shortcut for using this tool is **H**. ![](/img/others-2/annotate-3.png) #### Display Controls The **Display Controls** tool allows you to use an Electronic Light Table to adjust the visual settings of your input. It supports adjusting for brightness, saturation, and contrast. You can also invert the colors by selecting the **Invert colors** checkbox. If you select the **Save settings for future inputs** checkbox, your adjusted settings will be applied persistently across all inputs on the Input-Viewer page. You can click the **Reset** button at the upper-right corner to restore the settings to their default values. ![](/img/others-2/annotate-5.png) #### Rotate Tool The **Rotate** tool allows you to adjust the orientation of an image to suit your needs. Whether you want to rotate the image clockwise, counterclockwise, or flip it to a specific angle, this tool provides precise control for achieving your desired alignment. ![](/img/others-2/annotate-6.png) #### Zoom Tool The **Zoom** tool allows you to zoom in for a closer look or zoom out to view the image at a broader scale. ![](/img/others-2/annotate-7.png) #### Canvas Options Tool The **Canvas Options** tool allows you to manage the success notifications for actions performed in the canvas area. You can toggle this feature on or off based on your preference. ![](/img/others-2/annotate-8.png) import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## AI-Assisted Labeling # AI-Assisted Labeling **Automatically generate annotations on Input-Viewer page** > _Note that our UI has been updated since the release of the above video. However, the AI-assist functionality described in the video remains fully compatible with the new interface._ AI-assisted labeling is an innovative Clarifai feature that leverages artificial intelligence technology to assist and optimize the process of annotating data. You can request predictions from any model or workflow available to you on a particular input, and then review, correct, or validate them before converting them into annotations. AI-Assist provides you with several benefits, including: ![](/img/annotate/ai_assist_image-1.png) - Significantly accelerate the labeling process, reducing the time required to create labeled datasets; - Automation can reduce the labor costs associated with manual labeling; - AI models can provide consistent labeling, minimizing human errors; - It allows for the efficient and scalable handling of large datasets or rapidly changing data. :::caution feature availability This AI-assist feature is only available to users on our [paid plans](https://www.clarifai.com/pricing). ::: Let’s demonstrate how you can perform AI-assisted labeling using the **Annotate** mode of the Input-Viewer screen. ## 1. Activate AI-Assist Settings On the Input-Viewer, click the **Edit** button located in the upper-right corner of the page. ![](/img/annotate/ai_assist_1-2.png) You'll be redirected to the AI-assist sidebar that enables you to choose a model or workflow for using in labeling your inputs. Ensure the **AI Assist** toggle is switched on. ![navigate to input-viewer screen](/img/annotate/ai_assist_1.png) ## 2. Choose a Model or Workflow Use the **Select Model or Workflow** search box to choose a model or workflow to generate predictions and assist with annotations. You can choose your own customized model or workflow, or look for a public one from the Community platform. ![choose a model or workflow](/img/annotate/ai_assist_2.png) To select a public model or workflow from the Community, click the **Explore Community Models / Workflows** button. In the pop-up window, use the search bar to find the desired model or workflow. ![](/img/annotate/ai_assist_2-1.png) :::tip When working with image inputs, you need to choose a model or workflow that outputs concepts or objects (bounding box regions). This ensures the generation and display of annotation suggestions. ::: :::note Objective In this example, we'll illustrate how to generate annotations using a [visual classification]( https://docs.clarifai.com/portal-guide/model/model-types/visual-classifier) model and a [visual detection]( https://docs.clarifai.com/portal-guide/model/model-types/visual-detector) workflow. ::: ### Classifications Labeling First, let’s choose the Community’s [general-image-recognition]( https://clarifai.com/clarifai/main/models/general-image-recognition) model, which is a visual classification model that identifies a variety of concepts in images. ![ general-image-recognition model](/img/annotate/ai_assist_3.png) ### Objects Labeling Similarly, let’s select another input on the Input-Viewer screen. And on the **AI Assist** sidebar, let’s choose the Community’s [General-Detection](https://clarifai.com/clarifai/main/workflows/General-Detection) workflow, which identifies a variety of common objects in images. ![General-Detection workflow](/img/annotate/ai_assist_4.png) ## 3. Generate Annotations After choosing a model or workflow, it could take a few moments to automatically generate the annotations. The generated labels are sorted in descending order based on their concept probability values. ### Classifications Labeling The **Classifications** pane lists the concepts generated by the classification model, alongside their probability values. ![generate classification annotations](/img/annotate/ai_assist_5.png) ### Objects Labeling The **Objects** pane displays the bounding boxes identified by the detection workflow, alongside their probability values. ![generate bounding box annotations](/img/annotate/ai_assist_6.png) :::note Manually Generate annotations The **Select or add concepts** field in either the **Classifications** or **Objects** pane lets you choose existing concepts in your app for annotating your inputs. You can also add new concepts for the annotation. ::: ## 4. Review and Accept Predictions Finally, you can review the model or workflow prediction suggestions and accept them as needed. The **AI assist probability threshold** section in the right sidebar allows you to filter predictions by probability values. Use the slider control to display only the predictions that fall within your selected probability range. To accept all AI-assisted suggestions for annotating your input(s) with the selected threshold, simply click the **Accept all AI assist predictions** button. :::info When you accept an AI-assisted prediction suggestion, it will be added to your application as a [concept](https://docs.clarifai.com/portal-guide/inputs-manager/concepts) and automatically applied as a label to your input. ::: The **Search concepts** field allows you to find specific concepts and display their annotations on the page. ### Classifications Labeling The **Classifications** pane allows you to review and accept classification predictions. ![review and accept classification predictions](/img/annotate/ai_assist_7.png) - **Concepts count display** — The pane displays the number of concepts used to annotate an image. For example, "Classifications (9)" means the model suggested nine concepts based on the defined probability threshold. - **Accept or remove annotations** — To manually accept a prediction, simply click on the checkmark next to it. You will notice a color change, and a confirmation message will indicate that the annotation has been successfully added to your app and annotated with your input. You can also remove an annotation by deselecting the checkmark. _The removed annotation will revert to being a suggestion rather than being entirely removed from the list._ - **View individual annotators** — If you hover over the person icon on an annotation field, you can view the annotator(s) who added that annotation. The displayed number indicates how many annotators labeled that input. For example, it can be the annotator who manually accepted the suggestion and the model. - **Update annotations** — To update an already accepted annotation, hover over it and click the pencil icon that appears. Enter a new name for the annotation in the provided text field, then click the "Update" button to save your changes. The updated concept will be added to your app and annotated with your input. This updated concept is referred to as a concept relation, with the original concept shown as a superscript next to it. If you create an annotation using the original concept, its relation will be used instead. Note that if you edit the concept relation back to its original value, the concept relation annotation on the input will be removed. However, it will only be removed from the input and not from your app. - **Delete annotation** — If you hover over an already accepted annotation field, a delete icon will appear that you can use to remove the annotation. A small pop-up will appear, prompting you to confirm the deletion. - **Hotkey assignment** — Each annotation is assigned a hotkey number. Clicking this number will delete the corresponding annotation from the input. For example, clicking "1" will remove the `animal` annotation from the image. Up to 20 hotkeys can be assigned to an input. ### Objects Labeling The **Objects** pane allows you to review and accept objects predictions. It displays categories of concepts used for annotations, along with individual annotation instances. Reviewing and accepting object predictions is largely similar to classification labeling, with a few minor differences. ![review and accept objects predictions](/img/annotate/ai_assist_8.png) - **Annotation count display** — The pane shows the number of annotation suggestions for each instance. For example, "Objects (6)" indicates that the workflow suggested six categories of concepts. Similarly, "sheep (8)" means the workflow suggested eight instances of the `sheep` concept. Individual annotations are numbered sequentially. For example, "sheep.1" represents the first annotation labeled with the `sheep` concept, "sheep.2" represents the second annotation, and so on. - **Update and hide suggested annotations** — Hovering over a concept category field reveals a pencil icon for updating the concept name (as described earlier) and an eye icon for hiding all the annotations in that category. - **Drawing mode** — Clicking the bounding box icon or the polygon icon in a category field activates the drawing mode. This allows you to manually create bounding box or [polygon annotations](https://docs.clarifai.com/portal-guide/input-viewer/annotate#segmentation-labeling), respectively, using the selected concept. The drawing mode box, located in the upper-right corner of the canvas, displays the concept currently in use for detection labeling. To exit drawing mode, simply click the "Exit" button. - **Instance-specific actions** — Hovering over an individual annotation instance reveals these icons: pencil icon for reassigning the annotation to a different concept, eye icon for hiding the specific annotation instance, and delete icon for removing the specific annotation instance. - **Edit suggestions** — Clicking an individual annotation instance highlights its corresponding bounding box in the canvas, enabling easy editing by resizing, repositioning, or adjusting to better suit your needs. You can also edit a region prediction by clicking on the labeled bounding box and dragging it to cover the specific areas you want it to encompass. After editing the bounding box, the label will be automatically added to your input. - **Hotkey assignment** — Each concept is assigned a hotkey, allowing quick labeling for that concept. For example, pressing "1" enables you to draw bounding boxes labeled with the *sheep* concept. Up to 20 hotkeys can be assigned to your concepts. --- ## Bulk Labeling # Bulk Labeling **Label multiple inputs at once** Clarifai offers several convenient methods to label multiple inputs simultaneously, helping you save time and streamline your annotation process. Whether you're managing a large dataset or need to apply consistent labels across multiple inputs, the platform provides tools designed for efficiency and accuracy. :::note The examples provided here demonstrate how to perform bulk labeling for visual classification tasks. However, these methods can also be applied to bulk label other types of machine learning tasks, such as text classification tasks. ::: ## Bulk Label at Upload Time You can label all your inputs as you upload them to the Clarifai platform. To do so, go to the Inputs-Manager and click the **Upload Inputs** button located in the upper-right corner of the page. The [inputs uploader window](https://docs.clarifai.com/portal-guide/inputs-manager/upload-inputs) that pops up allows you to upload any type of data supported by the Clarifai platform. Next, use the **Concepts** section to select or add the [**concept(s)**](https://docs.clarifai.com/portal-guide/inputs-manager/concepts) you want to label your inputs with. If the concept already exists in your app, you can select it from the dropdown menu. ![](/img/others/bulk_labeling_1.png) If you want to add a new concept, click the plus sign (**+**) next to the **Select or add concepts** field. Then, type the new concept name in the field. Once the new name appears below the search field, click the **Add new concept** button to create the concept. The new concept will be added to your app and attached as a label to your inputs. ![](/img/others/bulk_labeling_2.png) Finally, click the **Upload inputs** button at the bottom of the uploader window to upload your inputs in bulk, complete with the assigned labels. ## Bulk Label Uploaded Inputs After uploading your inputs, you can quickly add labels directly from the Inputs-Manager page. To bulk label inputs, simply hover over the desired items and click the checkmark in the upper-left corner of each. :::tip multi-select feature - **Mouse click:** Selects a single item or input. - **Shift + mouse click:** Selects a range of inputs between the first and last clicked item. ::: Next, click the **Label as…** button that appears at the bottom of the page. ![](/img/others/bulk_labeling_3.png) The small window that pops up allows you to label the selected inputs with the chosen concept(s). Ensure the **Add** option is selected (this is the default). This option lets you add labels to your inputs. :::tip Optionally, you can select the **Apply to all search results** checkbox to label all inputs visually similar to the ones you initially selected. This is a quick way to apply bulk labeling. ::: Next, select or add the concepts you want to label with your inputs in the **Concepts** section. If your app already has existing concepts, you can select them from the dropdown menu; otherwise, add them as described earlier. ![](/img/others/bulk_labeling_4.png) Finally, click the **Add to inputs** button at the bottom of the uploader window to complete bulk labeling your inputs. ## Bulk Label Using Smart Search You can leverage Clarifai’s Smart Search capabilities to efficiently label large batches of inputs. For example, you can use the visual similarity search feature to identify and label similar images in bulk. To start a visual similarity search, hover over an image input and click the magnifying glass icon that appears on its left. ![](/img/others/bulk_labeling_7.png) A thumbnail of the image will be added to the search bar, and your results will be displayed in ranked order, from most to least visually similar. ![](/img/others/bulk_labeling_8.png) Once the results are displayed, you can quickly annotate the visually similar inputs with the same concept(s), as illustrated previously. [Click here](https://docs.clarifai.com/portal-guide/psearch/) to learn more about Clarifai’s Smart Search capabilities. ## Bulk Add Metadata Metadata provides additional information about each input. To add the same metadata to multiple inputs, start by hovering over the desired items and clicking the checkmark in the upper-left corner of each. Then, click the **Edit Metadata** button that appears at the bottom of the page. ![](/img/others/bulk_labeling_5.png) A small window will pop up, allowing you to add metadata to all the selected inputs. ![](/img/others/bulk_labeling_6.png) Ensure the **Add** option is selected (this is the default). Next, add the metadata you want to label with your inputs in the **Metadata** field. You need to add the metadata as a valid JSON object. Finally, click the **Add to inputs** button at the bottom of the pop-up window to complete bulk labeling your inputs with the metadata information. :::info As demonstrated earlier, you can also add metadata information in bulk via the inputs uploader pop-up window when uploading inputs to your application. ::: --- ## Create Annotations # Create Annotations **Learn how to create annotations** We support different types of labeling methods, each suited for different tasks and data characteristics. This lets you create high-quality training data depending on the objective you want your AI model to achieve. These are the different types of labels we support for your image, video, and text inputs: - **Classification** — Categorizes images, videos, and texts into categories; - **Detection** — Detects where an object of interest is and draws a bounding box around it; - **Segmentation** (polygons for segmentation) — Outlines the exact shape or contour of the object; - **Masks** — A type of image segmentation that defines the exact boundaries of an object at a pixel level. :::info AI-assisted labeling [Click here](https://docs.clarifai.com/portal-guide/annotate/ai-assist) to learn how to use the AI-assisted labeling feature to carry out different label types. ::: :::info Labeling Tasks Tool [Click here](https://docs.clarifai.com/portal-guide/annotate/labeling-tools) to learn how to use the Labeling Tasks tool to carry out different label types. ::: ## Label on Inputs-Manager Page To label your inputs, head to your application's individual page. Then, select the **Inputs** option in the collapsible left sidebar. You'll be redirected to the Input Mode of the Inputs-Manager page, where you can create and manage the annotations of your inputs. It is the default mode of the Inputs-Manager page. ![](/img/community/annotate/create_annotations_1.png) You can use different ways to label your inputs. ### Classification Labeling To label an image input for classification tasks on the Inputs-Manager page, hover over it and click the small empty box in the upper-left corner to select it. :::tip multi-select feature - **Mouse click**: Selects a single item or input. - **Shift + mouse click**: Selects a range of inputs between the first and last clicked item. ::: Next, click the **Label as…** button that appears at the bottom section of the page. ![](/img/community/annotate/create_annotations_2.png) The small window that pops up allows you to annotate the selected input(s) with concepts. ![](/img/community/annotate/create_annotations_4.png) Select the **Add** option, which lets you add annotations to your inputs (the option is selected by default). If you want to create a new concept and use it for labeling your inputs: - Click the plus sign (**+**) next to the **Select or add concepts** search field; - Type the new concept name in the search field. The new name you've typed will appear underneath the search field; - Click the **Add new concept** button to create the concept. The new concept will be successfully added to your app; - Finally, click the **Add to inputs** button at the bottom of the pop-up window to complete labeling your inputs with the newly created concept. :::tip If you select the **Apply to all search results** button, all the inputs that are visually similar to the one(s) you've initially selected will also be annotated. This allows you to label your inputs easily and fast. ::: If you've already created concepts and want to use them for annotating your input(s), simply select them from the **Concepts** field. ![](/img/community/annotate/create_annotations_3.png) After selecting the already existing concepts, click the **Add to inputs** button at the bottom of the pop-up window to complete labeling your inputs with them. ## Label on Input-Viewer Page If you click an input listed on the Inputs-Manager page, you'll be redirected to the viewer page for that input, where you can view and interact with it. To carry out manual annotation on the Input-Viewer page, ensure the page's mode is set to **Annotate**, which is the default status. You can find the mode settings in the upper-left corner of the page. ![](/img/others-2/label-types-1.png) ### Classification Labeling The classification label type lets you assign annotations to an entire image, a single frame of video, or a piece of text. #### Image Classification To manually classify an image on the Input-Viewer page, start by clicking the **Select / Edit** tool in the navigation bar (this tool is selected by default). Next, use the **Select label** menu that drops down to select the concept you want to use to annotate the image — that is, if the concept already exists in your app. ![](/img/others-2/label-types-2.png) If you want to annotate an image with a new concept that does not already exist in your app, click the **+** (plus) button in the dropdown menu and type the concept's name in the field. Then, click the dropdown box that appears with the concept name beneath that field. The new concept will be added to your app and labeled with your input. ![](/img/others-2/label-types-3.png) You can add as many annotations as you want. The added annotations will appear in the **Select label** dropdown menu as well as in the **Classifications** pane in the right sidebar of the page. ![](/img/others-2/label-types-4.png) :::note Alternatively, you can manually classify an image on the Input-Viewer page by navigating to the **Classifications** pane. Then, use the **Select or add concepts** search box to choose or add concepts for annotating your inputs, as described earlier. ::: #### Text Classification You can classify your text inputs into predefined categories in the same way as described earlier for image classification. :::note Ensure you select the appropriate [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows) when creating an app for text inputs. ::: ![](/img/others-2/label-types-5.png) #### Video Classification Support for video labeling within this tool is coming soon. If you need to label videos, you can create a [Labeling Task](https://docs.clarifai.com/portal-guide/labeling-tasks/create-a-task) and label a dataset with videos. ### Detection Labeling The detection label type lets you identify the objects in your inputs and also draw bounding boxes around them. #### Detection for Still Images To manually add detection labels on the Input-Viewer page, start by clicking the **Bounding Box** tool in the navigation bar. Next, use the **Select label** menu that drops down to select the concept you want to use to annotate the image — that is, if the concept already exists in your app. If it's not already existing, you'll need to add it as described earlier. ![](/img/others-2/annotate-9.png) Next, draw a rectangle as accurately as possible around the region of interest in the image. You can also create cascading bounding boxes; that is, a hierarchy of annotations where one bounding box is nested or dependent on another. After creating the bounding box, you can edit it by clicking on it until dotted lines appear around the edges. This will enable you to resize, reposition, or adjust the bounding box to better suit your needs. ![](/img/others-2/label-types-6.png) :::note drawing mode After selecting a concept for labeling, the drawing mode will be activated, allowing you to draw bounding boxes to annotate your image(s) easily. The drawing mode box, located in the upper-right corner of the canvas, displays the concept currently in use for detection labeling. To exit drawing mode, simply click the **Exit** button. ::: You can add as many detection annotations as you want for each concept. The added annotations will appear in the **Select label** dropdown menu as well as in the **Objects** pane in the right sidebar of the page. The **Objects** pane displays categories of concepts used for annotations, along with individual annotation instances. It supports several detection labeling activities, including: - **Annotation count display** — The pane shows the number of annotations for each instance. For example, "Objects(2)" indicates that two concept categories were used for the detection annotations in the image, while "sheep(4)" means that four instances are labeled with the `sheep` concept. Individual annotations are numbered sequentially. For example, "sheep.1" represents the first annotation labeled with the `sheep` concept, "sheep.2" the second annotation, and so on. - **Hotkey assignment** — Each concept is assigned a hotkey. Pressing a hotkey initiates labeling for that concept. For example, pressing "1" enables you to draw bounding boxes labeled with the `sheep` concept. Up to 20 hotkeys can be assigned to your concepts. - **Category-specific actions** — These are the actions you can complete on an annotation category: - Clicking the bounding box icon in a category field activates the drawing mode, which allows you to create annotations with the selected concept. - Hovering over a category of annotations reveals these icons: pencil icon for updating the concept name (as described earlier), eye icon for hiding all the annotations in that category, and delete icon for removing all the annotations in that category. - **Instance-specific actions** — These are the actions you can complete on an individual annotation instance: - To delete an individual annotation, deselect its checkmark. - Hovering over the person icon on an annotation instance shows the annotator(s) responsible for that annotation. The number displayed represents how many annotators labeled the input. - Hovering over an individual annotation instance reveals these icons: pencil icon for reassigning the annotation to a different concept, eye icon for hiding the specific annotation instance, and delete icon for removing the specific annotation instance. - Clicking an individual annotation instance highlights its corresponding bounding box in the canvas, enabling easy editing or deletion (by clicking the delete button on the keyboard). #### Detection for Video Support for video labeling is coming soon. :::info object mode [Click here](https://docs.clarifai.com/portal-guide/inputs-manager/#object-mode) to learn how to get the objects that have been labeled on your inputs using the Object Mode of the Inputs-Manager page. ::: ### Segmentation Labeling The segmentation label type lets you outline a boundary of an object using a series of vertices that define a closed polygonal shape. It's ideal for annotating irregularly shaped areas or objects. To manually add segmentation labels to an image on the Input-Viewer page, first click the **Polygon** tool on the navigation bar. Next, use the **Select label** menu that drops down to select the concept you want to use to annotate the image — that is, if the concept already exists in your app. If it's not already existing, you'll need to add it as described earlier. Next, use the dots to draw a contour as closely as possible around the image's region of interest. After creating the initial shape by placing your dots, you'll need to close the loop. Simply click on the first dot you made again, and this action will close the loop, completing the polygon. ![](/img/others-2/label-types-8.png) Note that segmentation labeling works just as the previously described detection labeling. For example, if you click the polygon icon in an annotation category field, the drawing mode will be activated, enabling you to create annotations with the selected concept. ### Masks Labeling The mask label type lets you label each pixel within the region of interest. It provides pixel-level labeling that allows you to precisely identify and delineate objects within an image. Currently, you can only minimally review existing image mask annotations on the Input-Viewer page. After creating mask images via the API and uploading them to our platform, you can view them on that page. [Click here](https://docs.clarifai.com/api-guide/annotate/annotations#annotate-images-with-mask) to learn how to add image mask annotations using our API. :::note You can delete an entire image mask annotation for an input directly from the Input-Viewer page. However, the Input-Viewer page does not currently support creating or editing mask annotations. ::: ![](/img/others-2/label-types-7.png) --- ## Manage Annotations # Manage Annotations **Learn how to manage annotations** ## Manage on Inputs-Manager Page ### Get Annotations Let's illustrate how you can get an image's labels. If you check the **Labels** section in the Inputs-Manager page, you can see the concepts you've created listed there. ![](/img/community/annotate/create_annotations_5.png) ### Update and Delete Annotations You can remove an annotation from an input and relabel it with a different annotation. Note that this action only removes the annotation from the input — it does not delete the [concept](https://docs.clarifai.com/create-manage/concepts/) from your app. Let's illustrate how you can edit the annotations of an image via the Inputs-Manager page. ![](/img/community/annotate/update_annotations_1.png) 1. Hover over the image and click the small empty box that appears in the upper-left corner to select it — just like we demonstrated previously; 2. Click the **Label as…** button that appears in the bottom section of the page. The small window that pops up allows you to edit the labels associated with the input; 3. Select the **Remove** option, which lets you remove the annotations you've added to the input; 4. In the **Concepts** field, select the concepts you want to remove from the input. After selecting a concept, it'll appear beneath the search field. Alternatively, you can check the **Remove all Concepts** button to delete all the annotations present in the input; 5. Finally, click the **Remove from inputs** button to delete the selected concepts from the input. ## Manage on Input-Viewer Page Within the Input-Viewer page, you can get the annotations already added to inputs, remove them from inputs, or carry out other management tasks with them. ### View Individual Annotators If you hover over the person icon on an annotation field in the **Classifications** pane, you can view the annotator(s) who added that annotation. The displayed number indicates how many annotators labeled that input. ### Update Annotations ![](/img/others-2/label-types-4-1.png) To update an annotation, hover over its field in the right sidebar and click the pencil icon that appears. Next, enter a new name for the annotation in the text field that appears, then click the **Update** button to save your changes. The new concept will be added to your app and annotated with your input. This newly added concept is referred to as a [**concept relation**](https://docs.clarifai.com/create-manage/concepts/concepts-relations), with the original concept displayed as a superscript next to it. As a result, if you create an annotation using the original concept, the updated concept will be used instead. Note that if you edit the concept relation back to its original value, the existing concept relation annotation on the input will be removed. It will only be removed from the input, and not from your app. ### Delete Annotations There are several ways of deleting an annotation via the Input-Viewer page: - If you hover over an annotation field, a delete icon will appear that you can use to remove the annotation. Note that deleting an annotation only removes it from the input; it does not delete the concept from your app. - You can remove an annotation by deselecting the checkmark next to it in the annotation field. - Additionally, each annotation is assigned a hotkey number. Clicking this number will delete the corresponding annotation from the input. For example, clicking "2" will remove the **sheep** annotation from the image. Up to 20 hotkeys can be assigned to an input. --- ## Object Tracking # Object Tracking **(Beta) Track people and objects in video** ## Track people and objects across multiple frames ![](/img/object-tracker.jpg) Videos are actually made up of a sequence of still images. Detection models can identify individual people and objects in still images, but they are not able to maintain these identities across frames. That is, they are not able to say that the person in one frame video is the same as the person in the next frame of video. This is where object tracking comes in. Clarifai offers convenient, pre-built object tracking workflows that help you identify and track objects and people across multiple frames of video. ## Build an object tracking workflow ![](/img/person-tracker-wkflw.jpg) To build an object tracking workflow, just visit model mode and click the create workflow button in the upper righthand corner of the screen. From here you will need to add two models to your workflow: | Model | ID | | :--- | :--- | | efficient person detector | 5750faf62ed9d514b9ee9d2d163f172e | | Person Tracker | c9d65bb4f86c487b848e9400725168d4 | Just add these two models to your workflow and connect the efficient person detector to the input node on the Person Tracker. ## Limitations Please keep in mind that video support is in Beta. Any video of 100MBs or less that is less than 1 minute can be processed at 10 FPS for any workflow. Download is 20 second limit per job assuming 5MB/sec. --- ## Labeling Tasks # Labeling Tasks **Label your data with tasks** Tasks enable you to label the inputs in your dataset easily, fast, and conveniently. You can create new labeling tasks, distribute these tasks to your workforce, and review their work in one convenient tool. ![](/img/community/tasks/task_1.png) import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Auto-Annotation # Auto-Annotation **Learn how to automatically label inputs with ease** Auto-annotation, also known as automatic annotation or automated labeling, refers to the use of machine learning techniques to automatically generate annotations or labels for data. Clarifai’s innovative auto-annotation feature comes with several ground-breaking capabilities, including: - **Annotate with a single click**: Annotate all the inputs in your dataset with just one click, regardless of its size. - **Fully automated annotation**: Utilize a completely automated approach without relying on the semi-automated, human-in-the-loop method of [AI-assist](https://docs.clarifai.com/portal-guide/annotate/create-a-task#step-5-turn-off-or-enable-ai-assist). - **Set confidence thresholds**: Determine which concepts to annotate automatically and which to review manually based on user-defined confidence thresholds. - **Targeted review**: Review only those inputs that fall below your specified accuracy thresholds. - **Customizable model selection**: Choose the model or workflow that best suits your dataset during the labeling task creation process. - **Partitioned review**: Assign inputs to multiple reviewers, allowing for faster review processes and minimizing single points of dependence. - **Efficient review process**: Easily review, accept, or reject inputs annotated by a specific labeler. - **Enhanced user experience**: Access key information such as "% Reviewed" and "Labeled by" to streamline the user experience. :::warning Supported Label Types Auto-annotation is a versatile feature designed to streamline data labeling processes by supporting various [data types](https://docs.clarifai.com/portal-guide/annotate/label-types), including classification, bounding boxes, and polygons for images, as well as text. This capability is compatible with all models and workflows, including foundation models and large language models (LLMs). ::: Here is a video that showcases how to perform auto-annotation: Let’s demonstrate how you can set up auto-annotation when creating labeling tasks. ## Prerequisites - [An application](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/) - [A dataset](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete/) containing the inputs you want to label. For this example, we’ll create a dataset containing a few images of pants, shirts, shoes, shorts, and skirts sourced from this [clothing dataset](https://www.kaggle.com/datasets/agrigorev/clothing-dataset-full/data) - [A model](https://docs.clarifai.com/portal-guide/model/pcustom-model-walkthrough) or [workflow](https://docs.clarifai.com/portal-guide/workflows/). You can create your own models or workflows, or choose from those publicly available in the Community ## Step 1: Start the Labeling Task Select **Labeling Tasks** from the collapsible left sidebar, then click the **New Labeling Task** button located in the upper-right corner of the **Labeling Tasks** page. ![](/img/images1/auto-annotation-1.png) ## Step 2: Select Dataset You'll be redirected to the **New Labeling Task** page, where you can provide the details for creating a new task. Start by selecting the dataset containing the inputs you want to label automatically. To do so, click the **Select a dataset** search box and choose the dataset you want to use. ![](/img/images1/auto-annotation-2.png) If you don't already have a dataset, click the **Create new Dataset with all inputs** button. Then, provide the name for the new dataset in the drop-down that appears. All the inputs in your app will be added to this new dataset. ![](/img/images1/auto-annotation-2-1.png) ## Step 3: Enable Auto-Annotation Select the **Yes** button to enable auto-annotation. ![](/img/images1/auto-annotation-3.png) ## Step 4: Select a Workflow Select a model or workflow you want to use for the auto-annotation task. You can select your own models or workflows, or choose from those publicly available in the Community. For this example, let’s select the [General-Detection](https://clarifai.com/clarifai/main/workflows/General-Detection) workflow that detects a variety of common objects in images by drawing bounding boxes around them. :::caution Use only workflows or models that output concepts; otherwise, they will not be suitable for the auto-annotation task. ::: ![](/img/images1/auto-annotation-4.png) Next, select the concepts you want to be automatically assigned to your inputs. Note that the concepts you choose must be those already included in the workflow or model. For this example, let’s select these concepts: `pants`, `shirt`, `shoe`, `short`, and `skirt`. ## Step 5: Set Decision Threshold Set the decision threshold to specify the limit at which predictions will be automatically approved or flagged for human review. You can configure the threshold in two ways: - **Globally**: This applies to all concepts. Use the slider to set the threshold values, as described below. ![](/img/images1/auto-annotation-5.png) - **Per concept**: This applies to each concept individually. If you choose this option, click the **Set thresholds** button. A pop-up will appear, allowing you to set the threshold values for each concept individually, as described below. ![](/img/images1/auto-annotation-7.png) ![](/img/images1/auto-annotation-6.png) After setting the individual values, click the **Save Changes** button. #### Using the Slider The slider allows you to set the threshold either by moving it or by typing the value manually in the provided field. - **Upper limit**: Sets the value for automatically approving predictions. For example, if set to 0.90, any prediction between 0.90 and 1.0 (marked in green) will be automatically approved and used for annotation. - **Lower limit**: Sets the value for automatically flagging predictions for further human review. For example, if set to 0.50, any prediction between 0.50 and 0.90 (marked in orange) will be flagged for human review before being accepted as an annotation. In this case, the predictions below the lower limit; that is, below 0.50, will be discarded, as they do not meet the set threshold. ## Step 6: Set Reviewers Adding human review adds a critical layer of verification and validation, ensuring that the annotations are both accurate and reliable. The annotations that do not meet the set decision threshold will be passed automatically for human review. So, you can assign yourself as the reviewer or specify if there will be any other app collaborator(s) working on the reviewing task. If you click the **New collaborator** button, a pop-up will appear, allowing you to add new collaborators to the app as well as to the reviewing task. You can learn how to add a collaborator [here](https://docs.clarifai.com/clarifai-basics/applications/collaboration/). ![](/img/images1/auto-annotation-8.png) :::warning Review Partitioning Distributing inputs across multiple reviewers allows you to automatically assign labeled assets to multiple reviewers. This approach improves the speed of the review process and reduces the risk of a single point of dependence. The labeled inputs will be evenly partitioned among all reviewers without duplication of the reviewing work. ::: ## Step 7: Set Task Name Specify the name of the task to facilitate easy identification. ![](/img/images1/auto-annotation-9.png) ## Step 8: Configure Advanced Settings You can optionally expand the **Advanced Settings** section to configure additional options for the auto-annotation task. Within the settings, you can assign one of the following priority levels: - **Low** — The task is of low priority. - **Medium (default)** — The task is of medium priority. - **High** — The task is of high priority. ![](/img/images1/auto-annotation-9-1.png) ## Step 9: Create Labeling Task Click the **Create Labeling Task** button to create the auto-annotation task. Or, click the **Cancel** button to cancel the process. Once the task is created, the dataset will be processed through the workflow, and annotations will be generated automatically. You’ll be redirected to the **Labeling Tasks** page, where you can see the newly created task listed in a table. You can monitor the progress of the auto-annotation in the “Labeling Status” column, where the status percentage will be displayed. Once the process is complete, the status will change to "Complete." ![](/img/images1/auto-annotation-10.png) You can also view other details of the auto-annotation exercise, including: - The reviewing status; - The dataset used; - Priority level for the task; - The concepts applied; - The labelers involved. If you click the ellipsis icon at the end of the row for your selected task, a drop-down menu will appear. This menu allows you to perform various task management activities, such as: - Copying the task ID; - Creating a new dataset version; - Editing the task; - Deleting the task. If you check the Input-Viewer page, you’ll notice that the bounding box annotations that met the set threshold were automatically created, without any manual intervention. ![](/img/images1/auto-annotation-11.png) ## Step 10: Review the Annotations As mentioned earlier, manual review ensures that the annotations created automatically meet the required quality standards and are accurate. To review and edit the annotations that did not meet the set decision threshold, click the **Review** button on the **Labeling Tasks** page. :::note - After completing the review process, the **Review** button will become inactive, preventing further clicks. - If there are remaining inputs requiring labeling, the **Label** button will remain active, allowing you to click it to proceed with labeling. ::: You’ll be redirected to the tasks reviewing page, where you can start reviewing the annotations. ![](/img/images1/auto-annotation-15-1.png) [Click here](https://docs.clarifai.com/portal-guide/annotate/review) to learn how to review the annotations created automatically. ## Step 11: Manage the Annotations You can carry out various management actions on the created annotations on the Input-Manager page. To do so, select the **Inputs** option from the collapsible left sidebar, then click the **Object Mode** button at the upper-right section of the page. The **Object Mode** is a gallery that shows the objects that have been labeled on your inputs. ![](/img/images1/auto-annotation-18.png) If you hover over an input and select it by clicking the small empty box in its upper-left corner, several options will appear, allowing you to perform various management tasks. ![](/img/images1/auto-annotation-19.png) - The search icon appearing on the left side of the input allows you to use it to perform [Smart Object Search](https://docs.clarifai.com/portal-guide/psearch/prank) and retrieve annotated objects within other images based on their content and similarity. - The delete icon allows you to delete the input from your app. - The **Add Concepts** button that appears at the bottom of the page allows you to create new object annotations with the selected concepts. - The **Remove Annotations** button that appears at the bottom of the page allows you to remove the annotation from the input. Additionally, clicking the ellipsis next to the **Labels** section will reveal various options for filtering the display of your inputs on the page. ![](/img/images1/auto-annotation-20.png) You can filter the display by: - Operator — `or` or `and` - Labeler - [Status](https://docs.clarifai.com/api-guide/advanced-topics/status-codes#annotation-related-codes-24xxx) — `Success`, `Awaiting Review`, `Awaiting Consensus Review`, or `Pending` --- ## Create a Labeling Task # Create a Labeling Task **Label a batch of data manually or with AI assistance** Let's demonstrate how you can label a batch of data manually or with AI assistance. :::tip auto-annotation [Click here](https://docs.clarifai.com/portal-guide/annotate/auto-annotation) to learn how to label inputs automatically using the auto-annotation technique. ::: ## Prerequisites - [An application](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/) - [A dataset](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete/) containing the inputs you want to label. For this example, we’ll create a dataset containing a few images of pants, shirts, shoes, shorts, and skirts sourced from this [clothing dataset](https://www.kaggle.com/datasets/agrigorev/clothing-dataset-full/data). - [Concepts](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete#create-concepts) that you want to use for the labeling task. For this example, we'll add the following concepts to our app: `pants`, `shirt`, `shoe`, `short`, and `skirt`. ## Step 1: Start the Labeling Task To create a task, select **Labeling Tasks** from the collapsible left sidebar, then click the **New Labeling Task** button located in the upper-right corner of the **Labeling Tasks** page. ![](/img/community/tasks/task_1.png) ## Step 2: Select Dataset You'll be redirected to the **New Labeling Task** page, where you can provide the details for creating a new task. Start by selecting a dataset from which inputs will be assigned for the labeling task. To do so, click the **Select a dataset** search box and choose the dataset you want to use from the list that drops down. ![](/img/community/tasks/task_4.png) If you don't already have a dataset, click the **Create new Dataset with all inputs** button. Then, provide the name for the new dataset in the drop-down that appears. All the inputs in your app will be added to this new dataset. ![](/img/community/tasks/task_21.png) ## Step 3: Turn Off Auto-Annotation To label the inputs manually or with AI assistance, select the **No** button. This will turn off the [auto-annotation](https://docs.clarifai.com/portal-guide/annotate/auto-annotation) process. ![](/img/community/tasks/task_4-1.png) ## Step 4: Select Task Type Choose the objective you want to achieve with your labeling task. You can choose among the following label types: - Classification — categorizes images, videos, and texts into categories; - Detection — detects where an object of interest is and draws a bounding box around it; - Segmentation (polygons for segmentation) — outlines the exact shape or contour of an object. ![](/img/community/tasks/task_11.png) For this example, let's select the detection label type. ## Step 5: Turn Off or Enable AI-Assist :::warning manual labeling Manual labeling of data is crucial for ensuring high-quality training data, handling complex and ambiguous cases, and providing accurate validation benchmarks. It also ensures data diversity and reduces bias, especially in sensitive applications. It's best for new domains with scarce pre-labeled data and niche datasets where the team is building collective concept definitions and libraries. ::: To label a batch of data manually, select the **No** option when asked if you want to enable AI-assist. This will turn off AI-assist and enable you to label your inputs manually. Next, use the **Search concepts** field to find and add existing concepts from your app that you want to use for the labeling exercise. The concepts you've selected will appear underneath the search box. For this example, we'll add these concepts: `pants`, `shirt`, `shoe`, `short`, and `skirt`. ![](/img/community/tasks/task_11-2.png) You can also create new concepts, and select them. Type the concept name in the **Search concepts** field. Then, click the **Add new concept** button that appears underneath the search box to create the new concept and also add it to your app. ![](/img/community/tasks/task_5.png) :::warning AI-Assisted labeling AI-assisted labeling is a semi-automated approach with human-in-the-loop that combines the power of existing models and workflows with the flexibility and accuracy of manual labeling. This method speeds up the labeling process by automating straightforward tasks while allowing human labelers to focus on complex or ambiguous cases. The AI model provides initial labels that humans can review and refine, ensuring high-quality and consistent annotations. This combines the strengths of both automated systems and human oversight. ::: To label using AI-assist, select the **Yes** option. ![](/img/community/tasks/task_13.png) Then, choose a model or workflow from the available options — either one you own or one publicly available in the Community. For this example, let’s select the [General-Detection](https://clarifai.com/clarifai/main/workflows/General-Detection) workflow that detects a variety of common objects in images by drawing bounding boxes around them. Next, use the **Search Concepts** field to select the concepts you want to use. Note that you can choose concepts already included in the workflow or model, or create new ones, as illustrated earlier. For this example, let’s select these concepts already existing in the workflow: `pants`, `shirt`, `shoe`, `short`, and `skirt`. The concepts you've selected will appear underneath the search box. Notice that there is a small AI-assist robot icon on the left side of each concept you've selected. The presence of the robot implies that the labeling task is going to be AI-assisted. Otherwise, the icon does not appear. :::caution segmentation AI-assist is not available for segmentation labeling tasks. ::: ## Step 6: Add Collaborators Delegating labeling tasks to a team provides several benefits, such as enhancing the overall quality of the exercise, expediting the process, and leveraging a diverse range of skills from various backgrounds. If you do not want to add any collaborator to help you with the task, select the **No** option. ![](/img/community/tasks/task_17.png) If you want to add a collaborator(s), select the **Yes** option. ![](/img/community/tasks/task_18.png) To assign the labeling work to a worker or a group of workers, click the **Select Labelers** search box and select a worker(s) you've already added as a collaborator(s) within your app. You can also assign the task to yourself. The inputs to be labeled will be evenly distributed among all the workers, ensuring that each input is assigned to only one worker and there is no duplication of work. To assign the reviewing work to a worker or a group of workers, click the **Select Reviewers** search box and select a reviewer(s) for the task. Adding human review adds a critical layer of verification and validation, ensuring that the annotations are both accurate and reliable. - Clicking the **New collaborator** button will open a pop-up, enabling you to add new collaborators to the app and assign them to the labeling or reviewing task. You can learn how to add a collaborator [here](https://docs.clarifai.com/clarifai-basics/applications/collaboration/). - Clicking the **Add all collaborators** button will assign the task to all collaborators in your app. On the other hand, clicking the **Remove all collaborators** button will unassign the task from all collaborators in your app. :::warning Review Partitioning Distributing inputs across multiple reviewers allows you to automatically assign labeled assets to multiple reviewers. This approach improves the speed of the review process and reduces the risk of a single point of dependence. The labeled inputs will be evenly partitioned among all reviewers without duplication of the reviewing work. ::: ## Step 7: Set Task Name Specify the name of the task to facilitate easy identification. ![](/img/community/tasks/task_1_1.png) ## Step 8: Add Instructions Optionally, you can provide instructions for your labelers to help them with the labeling task. ![](/img/community/tasks/task_1_2.png) ## Step 9: Configure Advanced Settings You can optionally expand the **Advanced Settings** section to configure additional options for the labeling task. Within the settings, you can assign one of the following priority levels: - **Low** — The task is of low priority. - **Medium** (_default_) — The task is of medium priority. - **High** — The task is of high priority. ![](/img/community/tasks/task_22.png) Also, you can choose the type of review method to be applied for the task: - **Evenly distributed review** (_default_) — Each reviewer will get an even split of the inputs to review. Note that each reviewer will get a batch of 10 inputs to review at a time. - **Stacked review** — Each reviewer will review every input. ## Step 10: Create Labeling Task Click the **Create Labeling Task** button to create the labeling task. Or, click the **Cancel** button to cancel the process. You’ll be redirected to the **Labeling Tasks** page, where you can see the newly created task listed in a table. For this example, you can see the two tasks we created listed in the table: one for manual labeling and another for AI-assisted labeling. ![](/img/community/tasks/task_10.png) You can monitor the progress of the annotation in the "Labeling Status" column, where the current status percentage will be displayed. Once the process is finished, the status will update to "Complete." You can also view other details of the annotation exercise, including: - The reviewing status; - The dataset used; - Priority level for the task; - The concepts applied; - The labelers involved. If you click the ellipsis icon at the end of the row for your selected task, a drop-down menu will appear. This menu allows you to perform various task management activities, such as: - Copying the task ID; - Creating a new dataset version; - Editing the task; - Deleting the task. ## Step 11: Start Labeling To start labeling your inputs, click the **Label** button on the **Labeling Tasks** page. :::note - After completing the labeling process, the **Label** button will become inactive, preventing further clicks. - After labeling the inputs, the **Review** button will become active, allowing you to click it to proceed with reviewing. ::: You'll be redirected to the **Labeling Tasks** tool, where you can begin annotating your inputs. If you enabled AI-assist when you created a new labeling task, you'll notice that capability enabled in the tool. ![](/img/community/tasks/task_14.png) If you did not enable AI-assist, you'll be able to label the inputs manually. ![](/img/community/tasks/task_15.png) [Click here](https://docs.clarifai.com/portal-guide/annotate/labeling-tools/#detection-labeling) to learn how to use the **Labeling Tasks** tool to add bounding box annotations to your batch of inputs. ## Step 12: Start Reviewing To start reviewing your inputs, click the **Review** button on the **Labeling Tasks** page. You’ll be redirected to the tasks reviewing page, where you can start reviewing the annotations for quality control purposes. ![](/img/community/tasks/task_19.png) [Click here](https://docs.clarifai.com/portal-guide/annotate/review) to learn how to review the annotations created manually or with AI assistance. :::tip info - [Members of an organization](https://docs.clarifai.com/portal-guide/clarifai-organizations/members-teams) can create new labeling tasks and utilize the Labeling Tasks tool. They can add collaborators who already exist within the app as well as other organization members. These can be assigned roles as either task workers or task reviewers. Furthermore, there are convenient options available for quickly adding all collaborators or all organization members. - Similarly, [app collaborators](https://docs.clarifai.com/clarifai-basics/applications/collaboration) can also create labeling tasks and assign them to other collaborators who already exist within the app. ::: --- ## Positive and Negative Annotations # Positive and Negative Annotations **Learn how to make positive and negative annotations** Positive and negative annotations play a critical role in training machine learning models. Positive annotations help the model identify and learn the features associated with a particular input. Negative annotations, on the other hand, help the model distinguish between similar inputs by highlighting what the input is not. Some inputs may also have overlapping features, especially in complex datasets. Providing both positive and negative annotations reduces the risk of the model misclassifying similar but distinct inputs, leading to more precise predictions. By annotating your data with both positive and negative labels, you provide your model with the comprehensive information it needs to learn effectively, which results in better performance and more reliable outcomes. Let's talk about how a model could interpret these annotations for a classification labeling task. :::tip You can use the [Labeling Tasks](https://docs.clarifai.com/portal-guide/annotate/labeling-tools) tool to assign both positive and negative labels. ::: ## Positive Annotation If an input is tagged with a positive annotation, it is considered a positive label for it. For example, if you positively annotate your input with the `burger` concept, your custom model will see it as having that label. ![](/img/annotation_i.jpg) If an input is tagged with a positive annotation, a negative label is implicitly assigned to all the other concepts that are not tagged as positive. For example, if an input is positively annotated with the `burger` concept, the model will implicitly treat `pizza` and `taco` as negative labels for that input. In that case, here's how the model could see the labels: ![](/img/annotation_ii.jpg) ## Negative Annotation If an input is tagged with a negative annotation, it is treated as a negative label for that concept. It emphasizes that the input does not belong to that concept. For example, if you negatively annotate your input with the `pizza` concept, your model will recognize that the input does not represent `pizza`. ![](/img/annotation_iii.jpg) If there is no positive annotation for an input, then if any concept is tagged with a negative annotation, it is treated as a negative example for all concepts related to that input. In that case, here's how the model could see the labels: ![](/img/annotation_iv.jpg) ## No Annotations If an input lacks either positive or negative annotations, the model ignores the example. ![](/img/annotation_v.jpg) --- ## Review Annotations # Review Annotations **Review the work performed by your labelers** Reviewing annotations ensures data accuracy and quality, validates model performance, and identifies biases. It helps handle complex cases, enhances training data, builds trust, and facilitates continuous improvement in both the annotation process and AI models. Clarifai offers a valuable tool to help you review the work of your labelers and ensure it meets your quality standards. After labeling a batch of data — whether [manually](https://docs.clarifai.com/portal-guide/annotate/create-a-task), with [AI assistance](https://docs.clarifai.com/portal-guide/annotate/create-a-task), or through [auto-annotation](https://docs.clarifai.com/portal-guide/annotate/auto-annotation) — you can conveniently review the results. Let's demonstrate how to review the annotations from a detection labeling exercise. To review and edit the annotations, click the **Review** button on the **Labeling Tasks** page. ![](/img/images1/review-1.png) :::note After completing the review process, the **Review** button will become inactive, preventing further clicks. ::: You’ll be redirected to the tasks reviewing tool. Click the **Start Reviewing** button in the pop-up window to begin reviewing the task. ![](/img/images1/review-2.png) At the top of the page, you can click the **Review** button to quickly switch to another labeling task without leaving the current interface. You can click the **Approve all** button to approve all the annotations from the specified labeler or the **Reject all** button to reject all the annotations. You can also view the labeler responsible for the annotation exercise. ![](/img/images1/review-3.png) If you click the ellipsis icon at the upper-right corner of the page, a drop-down menu will appear. This menu allows you to view the keyboard shortcuts available to enhance the review experience. You can also click **Open in legacy review mode** to revert to the legacy labeling interface. ![](/img/images1/review-4.png) These are the keyboard shortcuts you can use: - Brightness — `alt` - Saturation — `shift` - Invert — `control` - Pan — `shift + P` - Search — ` / ` The percentage carousel at the lower-left section of the page displays the progress of reviewed inputs as a percentage of the total. ![](/img/images1/review-5.png) The right sidebar allows you to edit the annotations: - The green tick indicates that the annotation is active. Clicking it will delete the annotation, but you must click the **Approve Selected** button for the change to be permanently reflected. :::note When you hover over the active annotation on the right sidebar, you'll notice a change in the intensity of the background color of the image. ::: - The eye icon allows you to blind the annotation. - The delete button allows you to delete the annotation. - The plus (+) button allows you to add another annotation using the same concept. For example, you can click it to add an additional bounding box to the image with the same concept. - The pencil edit button allows you to move the active annotation to a different concept. Clicking it will open a small menu, enabling you to reassign the annotation to another concept. ![](/img/images1/review-6.png) :::tip You can easily resize or move existing bounding box annotations. Just click inside the bounding box you wish to modify, then drag to move or resize. ::: You can click the **Approve Selected** button to approve the created annotation. Or, you can click the **Reject Input** button to reject the created annotation. --- ## Labeling Tasks Tools # Labeling Tasks Tools **Learn about the labeling tools available in Scribe** The Scribe Labeling Tasks platform provides special tools for working with images, videos, and texts for classification, detection, and segmentation labeling tasks. With the tools, you can annotate your inputs faster and more conveniently. After successfully [creating a labeling task](https://docs.clarifai.com/portal-guide/annotate/create-a-task), it will appear on the **Tasks** listing page. To begin working on the task, click the blue **LABEL** button. ![label button](/img/annotate/labeling_tools_1.png) You’ll be redirected to the **Labeling Tasks** screen, where you can begin annotating your inputs. At the upper-right section of the screen, you'll find a variety of tools you can use to simplify the annotation process. On the right sidebar, you can find a list of the [concepts](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete) you specified when you created a new labeling task. You can use them to label your inputs. ![labeling tools](/img/annotate/labeling_tools_2.png) ## Labeling Tools Let's talk about each of the tools. :::warning make the most of the tools - After selecting a labeling tool and using the slider that pops up to adjust your input, you can simply click the **Reset** button next to the slider to revert to its initial version. - You can combine multiple tools as appropriate to make adjustments to your input. - Clicking a button for the second time will deactivate or remove the highlight effect associated with that button. ::: ### View Metadata The metadata button allows you to toggle and view the metadata available in your inputs. ### Brightness The brightness button allows you to adjust the visibility of your inputs. ### Saturation The saturation button allows you to adjust the hue or saturation intensity of your inputs. ### Invert Color The invert color button allows you to apply transformation to your inputs. When clicked, it reverses the colors within the input, turning light areas dark, and vice versa. ### Zoom The zoom button allows you to change the level of magnification or the size of the view for your inputs. ### Pan The pan button allows you to move the visible areas of your inputs so that you can inspect specific regions closely. ### Show Hotkeys The show hotkeys button allows you to display the keyboard shortcuts available for each annotation tool. These are the hotkeys you can use: ![display hotkeys](/img/annotate/labeling_tools_3.png) - **B** — brightness - **S** — saturation - **I** — invert color - **P** — pan - **/** — filter concepts - **Enter** — submit a label ### View and Edit Labels You can view and edit previously submitted labels while working on a task. There is an input carousel tool at the bottom of the labeler screen that allows you to go back and review inputs after you have submitted them. This functionality provides a convenient mechanism to revisit and edit previously submitted labeled inputs. If you realize you made a mistake while labeling, you can easily go back and review your own work. ![View and Edit Labels](/img/annotate/labeling_tools_5.png) ## How to Label Image Inputs ### Classification Labeling :::note To perform classification labeling, you'll need to specify **Classification** as the [label type](https://docs.clarifai.com/portal-guide/annotate/create-a-task#modeling-objective) when setting up your labeling task. ::: With the classification label type, you can provide annotations for an entire image or a single frame of video. On the right sidebar of the Scribe Labeling Tasks platform, you can find a list of the [concepts](https://docs.clarifai.com/portal-guide/annotate/create-a-task#concepts) you specified when you created a new labeling task. To classify an image, just click the tick button next to the concept you want to label the image with. ![classify an image](/img/others/label_types_5.png) You can also click the cross button to explicitly label the image as not having the concept. ![label with cross button](/img/others/label_types_6.png) :::note Clicking an already-clicked tick or cross button removes the label. ::: You can also apply multiple labels to the same image. Let's say you have two classes that might be easily confused by your model, such as cats and dogs, you can specifically indicate that a cat is present and a dog is not present. This can improve the performance of your model, but also result in longer labeling times. In the following screenshot, the tick button has been clicked, indicating that the image has been labeled with the "cat" concept. Also, the cross button has been clicked, indicating that the image has been expressly labeled as not having the "dog" concept. ![cat and dog label](/img/others/label_types_4.png) If you enabled [AI Assist](https://docs.clarifai.com/portal-guide/annotate/ai-assist/) when you created a new labeling task, you'll notice the concepts that the Clarifai AI has suggested, which could be present in the image you want to label — alongside their probabilities. You can accept the highlighted suggestions to help you label your images quickly and fast. You can also adjust the slider to a prediction probability threshold you desire. ![label with AI assist](/img/others/label_types_7.png) Finally, click the **Submit** button on the lower-right corner of the page to submit the labeled input. ### Detection Labeling :::note - To perform detection labeling, you'll need to specify **Detection** as the [label type](https://docs.clarifai.com/portal-guide/annotate/create-a-task#modeling-objective) when setting up your labeling task. - You also need to select a workflow that offers detection capabilities when creating a new labeling task. ::: With the detection label type, you can provide annotation within a single box-shaped region of an image or a video. To begin labeling, click the plus (**+**) button next to the concept you want to use. This action will highlight the button, enabling you to begin drawing a bounding box around the image. You also have the option to use keyboard shortcuts provided alongside the concepts. For example, in the image below, pressing the number "2" on the keyboard will highlight the "vehicle" concept. Once you've highlighted a concept, position your cursor over the input and draw a bounding box around it as closely as possible. Next, release your mouse, and you'll see the annotation appearing under the selected concept. ![draw bounding box](/img/annotate/labeling_tools_4.png) :::tip You can easily resize, move, or relabel existing bounding boxes. Just click inside the bounding box you wish to modify, then drag to move or resize. If you need to change the label, just click the "edit" icon located next to the existing label in the right sidebar. ::: If you enabled AI Assist when you created a new labeling task, you can accept or modify the detected regions that have been suggested for labeling. ![detection for still images](/img/others/label_types_8.png) ### Segmentation Labeling Using Polygons Support for segmentation labeling is coming soon. ## How to Label Video Inputs Support for video labeling is coming soon. ## How to Label Text Inputs Scribe makes it easy to classify your text data. You can classify text inputs in the same way that you would classify images. ![Label text data in Scribe](/img/label-text.jpg) --- ## Create and Train Models # Create and Train Models **Learn how to easily create and train new models** The Clarifai platform simplifies the entire process of creating and training AI models, making it incredibly easy and efficient. With just a few clicks, your model is not only trained but also deployed, ready to enhance your business solutions instantly. You can either build custom models tailored to your specific needs or jumpstart your projects with Clarifai's pre-optimized models, which are designed for immediate use. ![](/img/enlight.svg) import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## Custom Models When you train a custom model, you are telling the system to look at all the inputs with concepts you've provided and learn from them. Then, when the model encounters new inputs, it could correctly generate predictions by applying the learned knowledge. The train operation is asynchronous. It may take some time for your model to be fully trained and ready. Your model will be trained on all inputs that have been processed, and a new version will be created. ## Clarifai Models Clarifai models are designed to be fast, scalable, and highly flexible, providing the ability to quickly deploy solutions that can adapt to your specific business needs. They can simplify complex tasks, reduce development time, and deliver reliable, accurate results. Our [model library](https://clarifai.com/explore/models) is continually expanding and evolving. You can explore it to find a model that best fits your needs. We offer models across a wide range of categories, including generative models like large language models (LLMs), as well as classification, detection, and segmentation models. :::info Try our Hosted Models Before training your own model, we recommend trying the models hosted on our platform to see if they meet your needs. ::: ## Model Types Whether you build a custom model or use one of Clarifai’s pre-built models, selecting the right model type is essential for your specific use case. Different model types are optimized for different tasks and produce distinct outputs based on your input data and desired AI functionality. Choosing the appropriate model type enables you to fully leverage the Clarifai platform and power your business with advanced AI capabilities. :::note tip You can use the [`List Model Types`](https://docs.clarifai.com/api-guide/model/create-get-update-and-delete#get) method to view a complete list of available model types suited to your needs. ::: ### List of Model Types | Model ID | Title | Description | | :--- | :--- | :--- | | `embedding-classifier` | **Transfer Learning Classifier** | Classify images or texts based on the embedding model that has indexed them in your app. Transfer learning leverages feature representations from a pre-trained model based on massive amounts of data, so you don’t have to train a new model from scratch and can learn new things very quickly with minimal training data. | | `audio-embedder` | **Audio Embedder** | Embed audio signal into a vector representing a high level understanding from our AI models. These embeddings enable similarity search and training on top of them. | | `visual-detector-embedder` | **Visual Detector + Embedder** | Detect bounding box regions in images or video frames where things occur and then embed them into a high level understanding from our AI models to enable visual search and training on top of them. | | `optical-character-recognizer` | **Optical Character Recognizer (OCR)** | Detect bounding box regions in images or video frames where text is present and then output the text read with the score. | | `image-to-image` | **Image to Image** | Given an image, apply a transformation on the input and return the post-processed image as output. | | `image-to-text` | **Image To Text** | Takes in cropped regions with text in them and returns the text it sees. | | `text-to-image` | **Text To Image** | Takes in a prompt and generates an image. | | `clusterer` | **Clusterer** | Cluster semantically similar images and video frames together in embedding space. This is the basis for good visual search within your app at scale or for grouping your data together without the need for annotated concepts. | | `image-color-recognizer` | **Image Color Recognizer** | Recognize standard color formats and the proportion each color that covers an image. | | `concept-thresholder` | **Concept Thresholder** | Threshold input concepts according to both a threshold and an operator (\>, \>=, =, \, \>=, =, \ rand()` where `keep_fraction` is the fraction to allow through on average. | | `visual-keypointer` | **Visual Keypoint** | This model detects keypoints in images or video frames. | | `email` | **Email Alert** | Email alert model will send an email if there are any data fields input to this model. | | `sms` | **SMS Alert** | SMS alert model will send a SMS if there are any data fields input to this model. | | `object-counter` | **Object Counter** | Count number of regions that match this model's active concepts frame by frame. | | `image-align` | **Image Align** | Aligns images using keypoints. | | `input-searcher` | **Cross-App Input Searcher** | Triggers a visual search in another app based on the model configs if concept(s) are found in images and returns the matched search hits as regions. | | `input-filter` | **Input Filter** | If the input going through this model does not match those we are filtering for, it will not be passed on in the workflow branch. | | `text-to-audio` | **Text to Audio** | Given text input, this model produces an audio file containing the spoken version of the input. | | `regex-based-classifier` | **Regex Based Classifier** | Classifies text using regex. If the regex matches, the text is classified as the provided concepts. | | `prompter` | **Prompter** | Prompt template where inputted text will be inserted into placeholders marked with `{data.text.raw}`. | | `rag-prompter` | **RAG Prompter** | A prompt template where we will perform a semantic search in the app with the incoming text. | | `image-prompter` | **Image Prompter** | A prompter model, that helps create a MultiModal input with the inputted image and text. | | `concept-to-text-mapper` | **Concept To Text Mapper** | Maps concepts to text. | | `mcp` | **MCP** | Process MCP messages with any input and output. | | `openai` | **OpenAI** | Process Clarifai models with OPENAI Format messages. | | `any-to-any` | **Any To Any** | Process any input and output with any data type. | | `image-tiling-operator` | **Image Tiling Operator** | Operator for tiling images into a fixed number of equal sized images. | | `isolation-operator` | **Isolation Operator** | Operator that computes distance between detections and assigns isolation label. | | `language-id-operator` | **Language Identification Operator** | Operator for language identification using the langdetect library. | | `text-aggregation-operator` | **Text Aggregation Operator** | Operator that combines text detections into text body for the whole image. | | `tiling-region-aggregator-operator` | **Tiling Region Aggregator Operator** | Operator to be used as a follow up to the image-tiling-operator and visual detector. This operator will transform the detections on each of tiles back to the original image and perform non-maximum suppression. | | `barcode-operator` | **Barcode Operator** | Operator that detects and recognizes barcodes from the image. It assigns regions with barcode text for each detected barcode. | | `keyword-filter-operator` | **Keyword Filter Operator** | This operator is initialized with a set of words, and then determines which are found in the input text. | | `raft-operator` | **RAFT Operator** | Calls an LLM to generate questions and answers based on a text input chunk. The output is the chat-formatted instruction for finetuning. | | `tokens-to-entity-operator` | **Tokens to Entity Operator** | Operator that combines text tokens into entities, e.g. `New` + `York` -\> `New York`. | | `track-representation-operator` | **Track Representation Operator** | The operator takes embedding of each track frame and aggregate them to form a track embedding. | | `byte-tracker` | **BYTE Track** | A multi-object tracker that aims to keep track of all boxes per frame forming them into tracklets. | | `centroid-tracker` | **Centroid Tracker** | Relies on the Euclidean distance between centroids of regions in different video frames to assign the same track ID to detections of the same object. | | `kalman-filter-tracker` | **Kalman Filter Hungarian Tracker** | Relies on the Kalman Filter algorithm to estimate the next position of an object, matched to detections using the Hungarian algorithm. | | `text-classifier` | **Text Classifier** | Classify text into a set of concepts. | | `visual-detector` | **Visual Detector** | Detect bounding box regions in images or video frames where things and then classify objects, descriptive words or topics within the boxes. | | `multimodal-to-text` | **Multimodal To Text** | Generate text from either text or images or both as input, allowing it to understand and respond to questions about those images. | | `text-embedder` | **Text Embedder** | Embed text into a vector representing a high level understanding from our AI models. These embeddings enable similarity search and training on top of them. | | `visual-embedder` | **Visual Embedder** | Embed images and videos frames into a vector representing a high level understanding from our AI models. These embeddings enable visual search and training on top of them. | | `visual-segmenter` | **Visual Segmenter** | Segment a per-pixel mask in images where things are and then classify objects, descriptive words or topics within the masks. | | `zero-shot-text-classifier` | **Zero Shot Text Classifier** | Classify text into a set of concepts provided by user using a pretrained model. | | `multimodal-embedder` | **Multimodal Embedder** | Embed text or image into a vector representing a high level understanding from our AI models, e.g. CLIP. These embeddings enable similarity search and training on top of them. | | `text-to-text` | **Text Generator** | Generate or convert text based on text input, e.g. prompt completion, translation or summarization. | | `visual-anomaly-heatmap` | **Visual Anomaly** | Visual anomaly detection with image-level score and anomaly heatmap. | | `zero-shot-image-classifier` | **Zero Shot Image Classifier** | Classify image into a set of concepts provided by user using a pretrained model. | | `audio-classifier` | **Audio Classifier** | Classify audio into a set of concepts. | | `text-token-classifier` | **Text Token Classifier** | Classify tokens from a set of entity classes. | | `visual-classifier` | **Visual Classifier** | Classify images and videos frames into set of concepts. | | `zero-shot-image-segmenter` | **Zero Shot Image Segmenter** | Dynamically segment a per-pixel mask in images where things are and then classify objects, descriptive words or topics within the masks. | | `audio-to-text` | **Audio To Text** | Classify audio signal into string of text. | import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Manage Models # Manage Models **Learn how to get, update, search, and delete models** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonUpdateConfiguration from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/update_model_name_configuration.py"; import PythonUpdateConfiguration2 from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/update_model_name_configuration_2.py"; import PythonListModelTypes from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/list_model_types.py"; import PythonGetModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/get_models.py"; import PythonGetModelID from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/get_model_by_id.py"; import PythonGetModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/get_model_output_info_by_id.py"; import PythonListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/list_model_concepts.py"; import PythonListModelVersions from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/list_model_versions.py"; import PythonGetModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/get_model_version_by_id.py"; import PythonDeleteModel from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/delete_model.py"; import PythonDeleteModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/delete_model_version.py"; import PythonDeleteAllModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/delete_all_models.py"; import PythonSearchModelsName from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/search_models_name_type.py"; import PythonSDKListModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/sdk_list_models.py"; import PythonCLIListModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/cli_list_models.sh"; import PythonSDKModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/py/list_model_output.txt"; import JSUpdateConfiguration from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/update_model_name_configuration.html"; import JSUpdateConfiguration2 from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/update_model_name_configuration_2.html"; import JSListModelTypes from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/list_model_types.html"; import JSGetModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/get_models.html"; import JSGetModelID from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/get_model_by_id.html"; import JSGetModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/get_model_output_info_by_id.html"; import JSListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/list_model_concepts.html"; import JSListModelVersions from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/list_model_versions.html"; import JSGetModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/get_model_version_by_id.html"; import JSDeleteModel from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/delete_model.html"; import JSDeleteModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/delete_model_version.html"; import JSDeleteAllModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/delete_all_models.html"; import JSSearchModelsName from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/js/search_models_name_type.html"; import NodeUpdateConfiguration from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/update_model_name_configuration.js"; import NodeUpdateConfiguration2 from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/update_model_name_configuration_2.js"; import NodeListModelTypes from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/list_model_types.js"; import NodeGetModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/get_models.js"; import NodeGetModelID from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/get_model_by_id.js"; import NodeGetModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/get_model_output_info_by_id.js"; import NodeListModelVersions from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/list_model_versions.js"; import NodeListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/list_model_concepts.js" import NodeGetModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/get_model_version_by_id.js"; import NodeDeleteModel from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/delete_model.js"; import NodeDeleteModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/delete_model_version.js"; import NodeDeleteAllModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/delete_all_models.js"; import NodeSearchModelsName from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/node/search_models_name_type.js"; import JavaUpdateConfiguration from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/update_model_name_configuration.java"; import JavaUpdateConfiguration2 from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/update_model_name_configuration_2.java"; import JavaListModelTypes from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/list_model_types.java"; import JavaListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/list_model_concepts.java"; import JavaGetModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/get_models.java"; import JavaGetModelID from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/get_model_by_id.java"; import JavaGetModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/get_model_output_info_by_id.java"; import JavaListModelVersions from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/list_model_versions.java"; import JavaGetModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/get_model_version_by_id.java"; import JavaDeleteModel from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/delete_model.java"; import JavaDeleteModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/delete_model_version.java"; import JavaDeleteAllModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/delete_all_models.java"; import JavaSearchModelsName from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/java/search_models_name_type.java"; import PHPUpdateConfiguration from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/update_model_name_configuration.php"; import PHPUpdateConfiguration2 from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/update_model_name_configuration_2.php"; import PHPListModelTypes from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/list_model_types.php"; import PHPGetModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/get_models.php"; import PHPGetModelID from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/get_model_by_id.php"; import PHPGetModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/get_model_output_info_by_id.php"; import PHPListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/list_model_concepts.php"; import PHPListModelVersions from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/list_model_versions.php"; import PHPGetModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/get_model_version_by_id.php"; import PHPDeleteModel from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/delete_model.php"; import PHPDeleteModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/delete_model_version.php"; import PHPDeleteAllModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/delete_all_models.php"; import PHPSearchModelsName from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/php/search_models_name_type.php"; import CurlUpdateConfiguration from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/update_model_name_configuration.sh"; import CurlUpdateConfiguration2 from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/update_model_name_configuration_2.sh"; import CurlListModelTypes from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/list_model_types.sh"; import CurlGetModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/get_models.sh"; import CurlGetModelID from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/get_model_by_id.sh"; import CurlGetModelOutput from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/get_model_output_info_by_id.sh"; import CurlListModelVersions from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/list_model_versions.sh"; import CurlListModelConcepts from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/list_model_concepts.sh"; import CurlGetModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/get_model_version_by_id.sh"; import CurlDeleteModel from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/delete_model.sh"; import CurlDeleteModelVersion from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/delete_model_version.sh"; import CurlDeleteAllModels from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/delete_all_models.sh"; import CurlSearchModelsName from "!!raw-loader!../../../code_snippets/api-guide/model/create_get_update_delete/curl/search_models_name_type.sh"; import SDKPatchModel from "!!raw-loader!../../../code_snippets/python-sdk/model_training/clusterer/patch_model.py"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [CLI](https://docs.clarifai.com/resources/api-overview/cli), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ## Get ### List Model Types Learn about the available model types and their hyperparameters. This endpoint lets you list all the possible models that are creatable (when `creatable=true`) or generally in the platform (the other ones have `creatable=false`). {PythonListModelTypes} {JSListModelTypes} {NodeListModelTypes} {JavaListModelTypes} {PHPListModelTypes} {CurlListModelTypes} ### List Models Below is an example of how to get a list of all the models you've created in your app. :::info The **ListModels** endpoint will return details of all the models in your app, alongside the details of their latest versions. ::: :::tip If you want to get a list of the models not within the scope of your app, you need to use your PAT while specifying the `user_id` of their owner and the `app_id` of the application that you’re accessing. For example, to get Clarifai's models in the `main` app, you need to use your PAT while specifying Clarifai's `user_id` as "clarifai" and `app_id` as "main" in the request. ::: {PythonSDKListModels} {PythonCLIListModels} {PythonGetModels} {JSGetModels} {NodeGetModels} {JavaGetModels} {PHPGetModels} {CurlGetModels} Output Example {PythonSDKModelOutput} ### Get Model by ID All models have unique IDs. You can get a specific model by its ID. {PythonGetModelID} {JSGetModelID} {NodeGetModelID} {JavaGetModelID} {PHPGetModelID} {CurlGetModelID} ### Get Model Output Info by ID The output info of a model lists the concepts contained in the latest version of the model—unless a particular version is specified. :::note The `GetModelOutputInfo` endpoint does not support pagination. If you want to split your results into pages, use the below-described `ListModelConcepts` endpoint. ::: {PythonGetModelOutput} {JSGetModelOutput} {NodeGetModelOutput} {JavaGetModelOutput} {PHPGetModelOutput} {CurlGetModelOutput} ### List Model Concepts Apart from the `GetModelOutputInfo` endpoint, you can also use the `ListModelConcepts` endpoint to list the concepts in your model. A major difference between the two is that the `ListModelConcepts` endpoint supports [pagination](https://docs.clarifai.com/api-guide/advanced-topics/pagination/), which lets you easily list concepts instead of displaying all of them at once. :::info If you are using any of the Clarifai gRPC clients, the `ListModelConcepts` endpoint is only available from release 8.10.0. ::: {PythonListModelConcepts} {JSListModelConcepts} {NodeListModelConcepts} {JavaListModelConcepts} {PHPListModelConcepts} {CurlListModelConcepts} ### List Model Versions Every time you train a model, it creates a new version. You can list all the versions created. {PythonListModelVersions} {JSListModelVersions} {NodeListModelVersions} {JavaListModelVersions} {PHPListModelVersions} {CurlListModelVersions} ### Get Model Version by ID To get the details of a specific model version, you must provide the `model_id` as well as the `version_id` parameters. You can inspect the model version status to determine if your model is trained or still training. {PythonGetModelVersion} {JSGetModelVersion} {NodeGetModelVersion} {JavaGetModelVersion} {PHPGetModelVersion} {CurlGetModelVersion} ## Update After creating a model, you can perform patch operations on it by merging, removing, or overwriting data. By default, all actions support overwriting, but they handle lists of objects in specific ways. - The `merge` action updates a `key:value` pair with `key:new_value` or appends to an existing list. For dictionaries, it merges entries that share the same `id` field. - The `remove` action is only used to delete the model's cover image on the platform UI. - The `overwrite` action completely replaces an existing object with a new one. :::info - The **PatchModels** endpoint allows you to patch only the model level fields, and nothing in the model version. It only updates things like name, description, notes, and other metadata type information field you may have. - The **PatchModelVersions** endpoint allows you to change most of the model version fields like gettable, metadata, license, description, notes, and `output_info` (not including concepts). ::: ### Update Model Name Let's use the **PatchModels** endpoint to change the model name to `newname`. {PythonUpdateConfiguration} {JSUpdateConfiguration} {NodeUpdateConfiguration} {JavaUpdateConfiguration} {PHPUpdateConfiguration} {CurlUpdateConfiguration} ### Update Model Configuration Let's use the **PatchModelVersions** endpoint to configure the minimum probability threshold for the outputs we want to view from the model. We can modify the `min_value` parameter available for this model type to filter the outputs to see only the concepts with a probability score of 0.95 or higher. :::tip For embedding-classifiers, the `min_value` parameter falls within the range of 0 to 1, with a step size of 0.01. For most of the other model types, it falls within the range of 0 to 100, with a step size of 0.1. ::: {PythonUpdateConfiguration2} {JSUpdateConfiguration2} {NodeUpdateConfiguration2} {JavaUpdateConfiguration2} {PHPUpdateConfiguration2} {CurlUpdateConfiguration2} ### Other Patch Operations Below is an example of performing other patch operations on a model, such as updating its description and notes. {SDKPatchModel} ## Search ### Search Models by Name and Type You can search all your models by name and type of model. {PythonSearchModelsName} {JSSearchModelsName} {NodeSearchModelsName} {JavaSearchModelsName} {PHPSearchModelsName} {CurlSearchModelsName} ## Delete ### Delete a Model You can delete a model by using the `model_id` parameter. {PythonDeleteModel} {JSDeleteModel} {NodeDeleteModel} {JavaDeleteModel} {PHPDeleteModel} {CurlDeleteModel} ### Delete a Model Version You can also delete a specific version of a model with the `model_id` and `version_id` parameters. {PythonDeleteModelVersion} {JSDeleteModelVersion} {NodeDeleteModelVersion} {JavaDeleteModelVersion} {PHPDeleteModelVersion} {CurlDeleteModelVersion} ### Delete all Models If you would like to delete all models associated with an application, you can also do that. :::caution Please proceed with extreme caution as deleted models cannot be recovered. ::: {PythonDeleteAllModels} {JSDeleteAllModels} {NodeDeleteAllModels} {JavaDeleteAllModels} {PHPDeleteAllModels} {CurlDeleteAllModels} --- ## Visual Classifier # Visual Classifier **Train an image classification model using a pipeline template** **Input**: Images and videos **Output**: [Concepts](https://docs.clarifai.com/create/concepts/) A visual classifier is a deep fine-tuned model that categorizes images and video frames into a set of predefined concepts. It answers the question *"What is in this image?"* or *"Who is in this image?"* For example, it can be used to categorize images into concepts such as "cat", "dog", or "vehicle". ![Image classification example](/img/others-2/image_classification_example.webp) *Image classification example* Use a visual classifier when you have a labeled dataset and want a custom model fine-tuned on your own classes. :::tip Visual classifiers are optimized for classification tasks. If you need to locate *where* objects appear in an image, consider a [Visual Detector](visual-detector.md) instead. ::: You can train a visual classifier two ways: - **[Via the CLI](#via-the-cli)** — Scriptable and reproducible. Recommended for engineering workflows. Train an end-to-end model in three commands. - **[Via the UI](#via-the-ui)** — Click-through training from the Clarifai web app. Recommended for quick experiments without writing code. ## **Via the CLI** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ### Prerequisites Install the Clarifai CLI and authenticate: {`pip install --upgrade clarifai clarifai login`} `clarifai login` auto-detects your user ID and saves your [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat/) locally. ### Train a Classifier (Quick Demo) The fastest way to see a working classifier end-to-end is the [`classifier-pipeline-resnet-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/classifier-pipeline-resnet-quick-start) template. It uses a Clarifai-hosted Food-101 subset as the training dataset, with sensible defaults for every hyperparameter — so no dataset setup or `--set` flags are required. {`clarifai pipeline init --template classifier-pipeline-resnet-quick-start cd classifier-pipeline-resnet-quick-start clarifai pipeline upload clarifai pipeline run --instance=g6e.xlarge`} That's it. `--instance=g6e.xlarge` auto-provisions a compute cluster and nodepool — no separate setup required. The pipeline trains a ResNet-50 image classifier on the public dataset and registers the trained model in your Clarifai model registry. ### Train on Your Own Data When you want to train a classifier on your own concepts, use the [`classifier-pipeline-resnet`](https://github.com/Clarifai/pipeline-examples/tree/main/classifier-pipeline-resnet) template with `--set` flags pointing at your uploaded dataset: ```bash clarifai pipeline init --template classifier-pipeline-resnet \ --set dataset_id=your_dataset_id \ --set dataset_version_id=your_dataset_version_id \ --set concepts='["class1","class2","..."]' cd classifier-pipeline-resnet clarifai pipeline upload clarifai pipeline run --instance=g6e.xlarge ``` Where: | Parameter | Description | |---|---| | `--set dataset_id` | The ID of the dataset to train on | | `--set dataset_version_id` | The specific version of the dataset | | `--set concepts` | A JSON array of the concept labels | To upload a dataset first, see the [Datasets documentation](https://docs.clarifai.com/create/datasets/). For all other init-time overrides (hyperparameters, base model, etc.), see the [Pipeline Templates reference](https://docs.clarifai.com/compute/pipelines/templates). ### Monitor and Use the Trained Model * **Monitor the run** — see [Manage Pipeline Runs](https://docs.clarifai.com/compute/pipelines/manage-run). * **Deploy and run inference on the trained model** — see [Inference](https://docs.clarifai.com/compute/inference/). * **Author a custom pipeline from scratch in Python** — see the [Pipeline DSL reference](https://docs.clarifai.com/compute/pipelines/dsl-reference). ## **Via the UI** Let’s walk through how to create and train a visual classifier model using the UI. ### Step 1: Create an App Create an application to store and manage your model and its associated resources (such as datasets, pipelines, and deployments). You can follow [this guide](https://docs.clarifai.com/create/applications/create/) to set one up. > **Note:** When creating the application, select the default **Image/Video** option as the primary input type. ### Step 2: Prepare Training Data Preparing your data is a critical step in training a model. High-quality, well-structured data helps your model learn effectively, generalize to new inputs, and produce reliable predictions. Make sure your dataset is: * **Clean and accurate** — free from labeling errors * **Diverse** — covers different variations of your target classes * **Sufficient in size** — enough examples for the model to learn meaningful patterns > **Tip:** You can organize your dataset using any spreadsheet tool. [Download a CSV template](https://docs.clarifai.com/portal-guide/advanced-topics/csv-and-tsv) to get started. For this example, we’ll use the [Beans Dataset](https://huggingface.co/datasets/AI-Lab-Makerere/beans) from Hugging Face, which contains images of healthy and diseased bean leaves. :::info objective Based on the selected dataset, we will train a model to classify leaf images into three categories: `Angular Leaf Spot`, `Bean Rust`, or `Healthy`. ::: ### Step 3: Add and Annotate Inputs To [add inputs](https://docs.clarifai.com/create/inputs/upload/ui) to your app, open the collapsible left sidebar and select the **Inputs** option. Click the **Upload Inputs** button in the upper-right corner, then use the uploader pop-up to select and upload your data. As you upload, assign each input category to a [dataset](https://docs.clarifai.com/create/datasets/create/) and [label](https://docs.clarifai.com/create/labeling/ui/create) them with their appropriate concepts. Ensure that all labeled inputs are added to the same dataset. ![](/img/community_2/my-deep-training-1.png) Once done, click the **Upload Inputs** button to add the annotated images to your app. > **Note:** For this tutorial, upload the three image categories to the same dataset, labeling each with its corresponding concept: > > * `Angular Leaf Spot` > * `Bean Rust` > * `Healthy` :::caution refresh your dataset After uploading all the inputs, [refresh your dataset](https://docs.clarifai.com/create/datasets/create/#create-dataset-version) and create a new version to reflect the changes. ::: ### Step 4: Create a Cluster and Nodepool To run and train your model, you’ll need to set up a cluster and nodepool with the appropriate compute resources. Start by [creating a cluster](https://docs.clarifai.com/compute/deployments/clusters-nodepools) that supports GPU-enabled workloads, as GPUs are required for efficient training and inference of vision models. Next, create a nodepool within the cluster and select a GPU-backed instance that matches your performance and budget needs. > **Note:** GPU support is essential for this tutorial. Ensure that the selected nodepool is configured with a compatible GPU instance to avoid performance issues or failed training runs. ### Step 5: Choose a Training Template Select the **Models** option in your app’s collapsible left sidebar. On the ensuing page for listing models, click the **Add a Model** button. ![](/img/community_2/my-deep-training-3.png) In the window that pops up, select the **Train a Model** option. ![](/img/community_2/my-deep-training-3-1.png) You’ll be redirected to a page listing available pipeline training templates. These templates provide pre-configured workflows to help you quickly get started with different types of models. ![](/img/community_2/my-deep-training-4.png) Select the `classifier-pipeline-resnet` template. This is a ResNet-based image classification pipeline designed for training models on labeled image datasets. ### Step 6: Configure Training Settings The ensuing page allows you to review the model training configuration and begin the training process. #### Select Training Template The training template you selected previously will be displayed for you. Otherwise, you can click the **Change** button to change to another training pipeline. ![](/img/community_2/my-deep-training-5.png) #### Select Nodepool Instance Choose the nodepool that will be used to train your model. Select the **Choose an instance** option to open a selection window, where you can pick from existing or recommended nodepools based on your training requirements. Choose your preferred nodepool, then click **Save Changes** to apply your selection. ![](/img/community_2/my-deep-training-6.png) The selected nodepool will be displayed for you. ![](/img/community_2/my-deep-training-7.png) Learn more about selecting a nodepool instance [here](https://docs.clarifai.com/compute/deployments/deploy-model#step-3-select-a-nodepool). #### Set Training Settings Configure the training settings for your model: ![](/img/community_2/my-deep-training-7-1.png) - **Model ID** — Set a unique ID for the model that will be created after it is trained. - **Dataset** — Select the dataset from which inputs will be used for this pipeline. For this tutorial, let's select the dataset we previously created containing the bean leaf images. - **Dataset Version** — Select which version of the dataset to use for training. You must select a dataset first before this option becomes available. - **Training Concepts** — Select the list of concepts you want the model to predict from the existing concepts labeled with your inputs. For this tutorial, let's pick these concepts: `Angular Leaf Spot`, `Bean Rust`, and `Healthy`. - **Training Epochs** — Set how many times the model will see the entire dataset. More epochs can lead to better accuracy but take longer. The default value is `25`. #### Configure Template Each training template includes a set of configurable hyperparameters that control how the model is trained. You can adjust these settings based on your dataset and performance goals. However, for this tutorial, we’ll use the default values provided by the `classifier-pipeline-resnet` template. ![](/img/community_2/my-deep-training-8.png) These are the settings you can configure: * **Batch Size** — Number of samples processed per training step. Default: `64`. * **Image Size** — Size (in pixels) to which input images are resized (square). Default: `224`. * **Per Item Lrate** — Learning rate applied per training sample. Default: `0.00001953125`. * **Weight Decay** — Regularization factor to prevent overfitting. Default: `0.01`. * **Per Item Min Lrate** — Minimum learning rate per sample during training. Default: `1.5625e-8`. * **Warmup Iters** — Number of initial iterations used to gradually increase the learning rate. Default: `5`. * **Warmup Ratio** — Starting ratio of the learning rate during warmup. Default: `0.0001`. * **Flip Probability** — Chance of randomly flipping images during training (data augmentation). Default: `0.5`. * **Flip Direction** — Direction used when flipping images. Default: `horizontal`. * **Concepts Mutually Exclusive** — Whether each input can belong to only one concept (mutually exclusive) or multiple concepts at the same time. Default: `disabled`. * **Pretrained Weights** — Source of initial model weights for transfer learning. Default: `ImageNet-1k`. * **Seed** — Random seed used to initialize training (set `-1` for random behavior). Default: `-1`. ### Step 7: Train the Model​ After configuring the training settings, click the **Train Model** button to start training your model using the selected pipeline. You’ll be redirected to the **Pipeline Version Runs** page, where you can monitor the training job in real time and track how the pipeline executes. ![](/img/community_2/my-deep-training-9-1.png) On this page, you can: * **Monitor run status** — Track the current state of the pipeline: * `RUNNING`: The training job is in progress. While the job is running, you can pause or stop it. * `COMPLETED`: The training finished successfully * `FAILED`: The training did not complete successfully (check logs for details) * **View run details** — See key information such as the start time and total run duration. * **Inspect infrastructure** — View where the job is running, including the cloud provider, region, compute instance type, and allocated resources. * **Follow pipeline execution** — The training runs as an [Argo Workflow](https://docs.clarifai.com/compute/pipelines/create-api#argo-workflow-definition), which breaks the process into steps. You can track the step-by-step execution of the pipeline in real time. * **Explore logs and nodes** — The logs panel displays detailed, JSON-like output, including a list of nodes (pipeline steps such as data loading, training, and evaluation). Each node includes metadata like its ID, type (e.g., `Steps`, `Pod`), and current status. * **Reload logs** — Click the **Reload** button to refresh and view the latest logs. * **Run a new job** — Click **Run Pipeline Version** to launch another training run. You’ll be prompted to select a cluster and nodepool before starting. ### Step 8: Use the Model​ Once your model has been trained successfully, you can start using it for predictions. To access it, select the **Models** option from the collapsible left sidebar. This opens the models listing page. ![](/img/community_2/my-deep-training-10.png) Click the listed model to open its individual page. ![](/img/community_2/my-deep-training-11.png) Next, click the **Deploy Model** button to [create a deployment](https://docs.clarifai.com/compute/deployments/deploy-model). This sets up the compute resources needed to run inference. After deployment, click the **Try Model** button in the upper-right corner to open the [Playground](https://docs.clarifai.com/getting-started/playground), where you can submit inputs and get predictions. ![](/img/community_2/my-deep-training-12.png) For this tutorial, uploading an image of a bean leaf will return classifications such as `Angular Leaf Spot`, `Bean Rust`, or `Healthy`, along with their prediction probabilities. That’s it! {CodeMT} Output {CodeOutputMT} Next, let's select the `visual-classifier` model type and use it to create a model. --> {CodeMC} {PythonCreateModel} {JSCreateModel} {NodeCreateModel} {JavaCreateModel} {PHPCreateModel} {CurlCreateModel} --> {CodeTS} Output {CodeOutputTS} --> {CodeS} Output {CodeOutputS} --> {CodeS2} Output {CodeOutputS2} --> {CodeIMT} {PythonTrainModel} {JSTrainModel} --> {NodeTrainModel} {JavaTrainModel} {PHPTrainModel} {CurlTrainModel} Output {CodeOutputIMT} --> {CodeMP} Output {CodeOutputMP} --> {CodeTrEv} Output {CodeOutputTrEv} --> {CodeTeEv} Output {CodeOutputTeEv} --> {CodeCMP} Output {CodeOutputCMP} --> --- ## Visual Detector # Visual Detector **Train an object detection model using a pipeline template** **Input**: Images and videos **Output**: Regions A visual detector is a deep fine-tuned model that identifies and localizes objects within images and video frames. It goes beyond image classification by answering not only *"What is in this image?"* but also *"Where is it?"* For example, it can be used to detect and locate objects such as "car", "person", or "dog" within an image, drawing a bounding box around each identified instance. Use a visual detector when you have a labeled dataset with bounding-box annotations and want a custom model fine-tuned on your own object classes. :::tip Visual detectors are optimized for detection and localization tasks. If you only need to identify *what* is in an image without locating specific objects, consider a [Visual Classifier](visual-classifier.md) instead. ::: You can train a visual detector two ways: - **[Via the CLI](#via-the-cli)** — Scriptable and reproducible. Recommended for engineering workflows. Train an end-to-end model in three commands. - **[Via the UI](#via-the-ui)** — Click-through training from the Clarifai web app. Recommended for quick experiments without writing code. ## **Via the CLI** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ### Prerequisites Install the Clarifai CLI and authenticate: {`pip install --upgrade clarifai clarifai login`} `clarifai login` auto-detects your user ID and saves your [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat/) locally. ### Train a Detector (Quick Demo) The fastest way to see a working detector end-to-end is the [`detector-pipeline-yolof-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/detector-pipeline-yolof-quick-start) template. It uses a Clarifai-hosted public dataset as the training data, with sensible defaults for every hyperparameter — so no dataset setup or `--set` flags are required. {`clarifai pipeline init --template detector-pipeline-yolof-quick-start cd detector-pipeline-yolof-quick-start clarifai pipeline upload clarifai pipeline run --instance=g6e.xlarge`} That's it. `--instance=g6e.xlarge` auto-provisions a compute cluster and nodepool — no separate setup required. The pipeline trains a YOLOF object detector on the public dataset and registers the trained model in your Clarifai model registry. ### Train on Your Own Data When you want to train a detector on your own object classes, use the [`detector-pipeline-yolof`](https://github.com/Clarifai/pipeline-examples/tree/main/detector-pipeline-yolof) template with `--set` flags pointing at your uploaded dataset: ```bash clarifai pipeline init --template detector-pipeline-yolof \ --set dataset_id=your_dataset_id \ --set dataset_version_id=your_dataset_version_id \ --set concepts='["class1","class2","..."]' cd detector-pipeline-yolof clarifai pipeline upload clarifai pipeline run --instance=g6e.xlarge ``` Where: | Parameter | Description | |---|---| | `--set dataset_id` | The ID of the dataset to train on (with bounding-box annotations) | | `--set dataset_version_id` | The specific version of the dataset | | `--set concepts` | A JSON array of the object class labels | To upload a dataset first, see the [Datasets documentation](https://docs.clarifai.com/create/datasets/). For all other init-time overrides (hyperparameters, base model, etc.), see the [Pipeline Templates reference](https://docs.clarifai.com/compute/pipelines/templates). ### Monitor and Use the Trained Model * **Monitor the run** — see [Manage Pipeline Runs](https://docs.clarifai.com/compute/pipelines/manage-run). * **Deploy and run inference on the trained model** — see [Inference](https://docs.clarifai.com/compute/inference/). * **Author a custom pipeline from scratch in Python** — see the [Pipeline DSL reference](https://docs.clarifai.com/compute/pipelines/dsl-reference). ## **Via the UI** Let's walk through how to create and train a visual detector model using the UI. ### Step 1: Create an App Create an application to store and manage your model and its associated resources (such as datasets, pipelines, and deployments). You can follow [this guide](https://docs.clarifai.com/create/applications/create/) to set one up. > **Note:** When creating the application, select the default **Image/Video** option as the primary input type. ### Step 2: Prepare Training Data Preparing your data is a critical step in training a model. For object detection, your dataset must include images with bounding box annotations that mark where each object of interest appears. Make sure your dataset is: * **Clean and accurate** — free from labeling errors and imprecise bounding boxes * **Diverse** — covers different variations, scales, orientations, and backgrounds of your target objects * **Sufficient in size** — enough annotated examples for the model to learn meaningful patterns For this example, we'll use the [CPPE-5 (Consumer Protective Personal Equipment)](https://huggingface.co/datasets/rishitdagli/cppe-5) dataset from Hugging Face, which contains images annotated with bounding boxes for medical personal protective equipment. You can download the dataset inputs to your local machine for use in this tutorial. :::info objective Based on the selected dataset, we will train a model to detect five categories of medical PPE: `Coverall`, `Face_Shield`, `Gloves`, `Goggles`, and `Mask`. ::: ### Step 3: Add and Annotate Inputs To [add inputs](https://docs.clarifai.com/create/inputs/upload/ui) to your app, open the collapsible left sidebar and select the **Inputs** option. Click the **Upload Inputs** button in the upper-right corner of the Inputs-Manager page. Then, use the uploader pop-up to select and upload your images. As you upload, assign them to a [dataset](https://docs.clarifai.com/create/datasets/create/). ![](/img/community_2/visual-detector-1.png) > **Note:** For this tutorial, we’ll upload 5 images from each category in the PPE dataset. Once the images are uploaded, you’ll need to annotate them with bounding boxes. To do this, click an image on the Inputs-Manager page to open it in the single Input-Viewer page, where annotations can be added. Next, select the **Bounding Box** tool from the navigation bar. For each image, draw a bounding box around every object of interest and assign the correct concept label: * `Coverall` * `Face_Shield` * `Gloves` * `Goggles` * `Mask` ![](/img/community_2/visual-detector-2.png) To learn more about adding detection labels in the Input-Viewer, go [here](https://docs.clarifai.com/create/labeling/ui/create#detection-for-still-images). :::caution refresh your dataset After completing all annotations, [refresh your dataset](https://docs.clarifai.com/create/datasets/create/#create-dataset-version) and create a new version to reflect the changes. ::: ### Step 4: Create a Cluster and Nodepool To run and train your model, you'll need to set up a cluster and nodepool with the appropriate compute resources. Start by [creating a cluster](https://docs.clarifai.com/compute/deployments/clusters-nodepools) that supports GPU-enabled workloads, as GPUs are required for efficient training and inference of detection models. Next, create a nodepool within the cluster and select a GPU-backed instance that matches your performance and budget needs. > **Note:** GPU support is essential for this tutorial. Ensure that the selected nodepool is configured with a compatible GPU instance to avoid performance issues or failed training runs. ### Step 5: Choose a Training Template Select the **Models** option in your app's collapsible left sidebar. On the ensuing page for listing models, click the **Add a Model** button. ![](/img/community_2/visual-detector-3.png) In the window that pops up, select the **Train a Model** option. ![](/img/community_2/visual-detector-3-1.png) You'll be redirected to a page listing available pipeline training templates. These templates provide pre-configured workflows to help you quickly get started with different types of models. ![](/img/community_2/visual-detector-4.png) Select the `detector-pipeline-yolof` template. This is a YOLOF-based object detection pipeline designed for training models on images with bounding box annotations. ### Step 6: Configure Training Settings The ensuing page allows you to review the model training configuration and begin the training process. #### Select Training Template The training template you selected previously will be displayed for you. Otherwise, you can click the **Change** button to change to another training pipeline. ![](/img/community_2/visual-detector-5.png) #### Select Nodepool Instance Choose the nodepool that will be used to train your model. Select the **Choose an instance** option to open a selection window, where you can pick from existing or recommended nodepools based on your training requirements. Choose your preferred nodepool, then click **Save Changes** to apply your selection. ![](/img/community_2/visual-detector-6.png) The selected nodepool will be displayed for you. ![](/img/community_2/visual-detector-7.png) Learn more about selecting a nodepool instance [here](https://docs.clarifai.com/compute/deployments/deploy-model#step-3-select-a-nodepool). #### Set Training Settings Configure the training settings for your model: ![](/img/community_2/visual-detector-7-1.png) - **Model ID** — Set a unique ID for the model that will be created after it is trained. - **Dataset** — Select the dataset from which inputs will be used for this pipeline. For this tutorial, let's select the dataset we previously created containing the annotated PPE images. - **Dataset Version** — Select which version of the dataset to use for training. You must select a dataset first before this option becomes available. - **Training Concepts** — Select the list of concepts you want the model to detect from the existing concepts labeled with your inputs. For this tutorial, let's pick these concepts: `Coverall`, `Face_Shield`, `Gloves`, `Goggles`, and `Mask`. - **Training Epochs** — Set how many times the model will see the entire dataset. More epochs can lead to better accuracy but take longer. The default value is `25`. #### Configure Template Each training template includes a set of configurable hyperparameters that control how the model is trained. You can adjust these settings based on your dataset and performance goals. However, for this tutorial, we'll use the default values provided by the `detector-pipeline-yolof` template. ![](/img/community_2/visual-detector-8.png) These are the settings you can configure: * **Seed** — Random seed used to initialize training. Set to `-1` for non-deterministic behavior, or provide a fixed integer to make results reproducible across runs. Default: `-1`. * **Image Size** — The size (in pixels) to which input images are resized before being fed into the model. A larger size may improve detection of small objects but increases memory usage and training time. Default: `[512]`. * **Max Aspect Ratio** — The maximum allowed aspect ratio of an image when aspect ratio preservation is enabled. Images with a higher aspect ratio will be clamped to this value to prevent extreme distortions. Default: `1.5`. * **Keep Aspect Ratio** — Whether to preserve the original aspect ratio of input images during resizing. When enabled, images are resized without distortion; padding is added as needed to fill the target dimensions. Default: enabled. * **Batch Size** — The number of training samples processed in each iteration. Larger batch sizes can stabilize training but require more GPU memory. Default: `16`. * **Min Samples Per Epoch** — The minimum number of samples that must be seen per epoch during training. This ensures a consistent training duration even when the dataset is small. Default: `300`. * **Per Item Lrate** — The learning rate applied per individual training sample. It controls how much the model's weights are adjusted after processing each item. Default: `0.001875`. * **Pretrained Weights** — The source of the initial model weights used for transfer learning. Starting from pretrained weights (such as those trained on COCO) helps the model converge faster and perform better with less data. Default: `coco`. * **Frozen Stages** — The number of backbone stages whose weights are frozen (not updated) during training. Freezing early stages preserves low-level feature representations learned during pretraining and can speed up training. Default: `1`. * **Inference Max Batch Size** — The maximum number of images processed in a single batch during inference. Adjust this based on available GPU memory and throughput requirements. Default: `2`. ### Step 7: Train the Model​ After configuring the training settings, click the **Train Model** button to start training your model using the selected pipeline. You'll be redirected to the **Pipeline Version Runs** page, where you can monitor the training job in real time and track how the pipeline executes. ![](/img/community_2/visual-detector-9-1.png) On this page, you can: * **Monitor run status** — Track the current state of the pipeline: * `RUNNING`: The training job is in progress. While the job is running, you can pause or stop it. * `COMPLETED`: The training finished successfully * `FAILED`: The training did not complete successfully (check logs for details) * **View run details** — See key information such as the start time and total run duration. * **Inspect infrastructure** — View where the job is running, including the cloud provider, region, compute instance type, and allocated resources. * **Follow pipeline execution** — The training runs as an [Argo Workflow](https://docs.clarifai.com/compute/pipelines/create-api#argo-workflow-definition), which breaks the process into steps. You can track the step-by-step execution of the pipeline in real time. * **Explore logs and nodes** — The logs panel displays detailed, JSON-like output, including a list of nodes (pipeline steps such as data loading, training, and evaluation). Each node includes metadata like its ID, type (e.g., `Steps`, `Pod`), and current status. * **Reload logs** — Click the **Reload** button to refresh and view the latest logs. * **Run a new job** — Click **Run Pipeline Version** to launch another training run. You'll be prompted to select a cluster and nodepool before starting. ### Step 8: Use the Model​ Once your model has been trained successfully, you can start using it for predictions. To access it, select the **Models** option from the collapsible left sidebar. This opens the models listing page. ![](/img/community_2/visual-detector-10.png) Click the listed model to open its individual page. ![](/img/community_2/visual-detector-11.png) Next, click the **Deploy Model** button to [create a deployment](https://docs.clarifai.com/compute/deployments/deploy-model). This sets up the compute resources needed to run inference. After deployment, click the **Try Model** button in the upper-right corner to open the [Playground](https://docs.clarifai.com/getting-started/playground), where you can submit inputs and get predictions. ![](/img/community_2/visual-detector-12.png) For this tutorial, uploading an image will return detected objects with bounding boxes drawn around each identified item — labeled with their concept such as `Coverall`, `Gloves`, or `Mask` — along with their prediction probabilities. That's it! --- ## Vector Search # Vector Search **Leverage vector search capabilities to find similar items in a dataset** Vector search is a technique that enables searching and retrieving unstructured data — like text, images, and videos — based on meaning rather than exact matches. At its core are vector embeddings — numerical representations of data that capture their semantic or visual essence. These embeddings allow data to be mathematically processed and compared, making it possible to find similar items even when they don’t share exact words or features. Unlike traditional keyword-based search, which relies on exact or fuzzy text matches, [vector search](https://www.clarifai.com/blog/finding-what-you-need-a-comprehensive-guide-to-vector-search) compares the distances between embeddings to find results that are contextually or visually similar. This allows for much more intelligent and intuitive retrieval. :::note When you post inputs to our platform, either via the UI or API, your app's [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/) automatically indexes those inputs using the outputs from your models. This index powers vector search capabilities — so you can search using concepts, annotations, or advanced parameters to find inputs that align with the meanings you're interested in. ::: ![Image illustrating how to search by images using Clarifai's concepts](/img/illustration-search.png) ## Powered by a Vector Database Our vector search engine uses deep learning embedding models to first analyze the visual features of each input, such as color, shape, and texture. This process, known as feature extraction, generates a corresponding vector representation for each piece of unstructured data. The embedding models then index these vector representations and store them in our [vector database](https://www.clarifai.com/blog/using-clarifais-native-vector-database) (also called a vector store or a semantic search engine). When a user performs a search, their query is also converted into a vector representation. The vector DB then searches for the vector representations that are most similar to the query vector representation. The results are then displayed to the user. By using our vector search as a service, you can get more relevant search results, faster search times, and scalable performance. ## Types of Vector Search - **Rank** — Leverage your model’s understanding of concepts to rank search results by relevance. This can be based on how confident the model is that a specific concept (like “cat” or “tree”) is present in an input, or how similar one input is to another. This is especially powerful for semantic and visual similarity use cases. - **Filter** — Use filters to narrow down search results and focus only on the data that matters to you. For example, you might want to see only inputs that a specific collaborator has labeled with the word “dog,” or filter by metadata, such as limiting your results to inputs captured in a certain geographic region. - **'AND'** — Create more precise queries by combining multiple conditions. This makes your searches even more targeted. For instance, you might look for all inputs within a specific geographic area that also contain the concept of a “weapon,” or find annotations assigned to a particular user like “Joe.” import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Search via the API # Search via the API **Use vector search to find, sort, and filter what you need in your unstructured data** The API gives you full access to vector search capabilities, allowing you to programmatically search using AI-generated embeddings. You can retrieve inputs based on semantic similarity, filter results using metadata, and combine multiple conditions for more precise queries. Whether you're working with images, text, or other data types, vector search via the API helps you surface the most relevant results quickly and at scale — making it ideal for building intelligent, customized search experiences into your own applications. ![](/img/spacetime.svg) import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Combine or Negate # Combine or Negate **Group or separate items in your search** You can add together multiple search parameters to expand your search. You can even combine negated search terms for more advanced tasks. In annotation search, `Filter` and `Rank` is a list of `Annotation` objects. Filtered annotations will be ANDed. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonCombineNegate from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate.py"; import NodeCombineNegate from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate.js"; import JSCombineNegate from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate.html"; import JavaCombineNegate from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate.java"; import PHPCombineNegate from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate.php"; import CurlCombineNegate from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate.sh"; import PythonCombineNegate_2 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_2.py"; import NodeCombineNegate_2 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_2.js"; import JSCombineNegate_2 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_2.html"; import JavaCombineNegate_2 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_2.java"; import PHPCombineNegate_2 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_2.php"; import CurlCombineNegate_2 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_2.sh"; import PythonCombineNegate_3 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_3.py"; import NodeCombineNegate_3 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_3.js"; import JSCombineNegate_3 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_3.html"; import JavaCombineNegate_3 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_3.java"; import PHPCombineNegate_3 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_3.php"; import CurlCombineNegate_3 from "!!raw-loader!../../../../code_snippets/api-guide/search/combine_or_negate_3.sh"; import curlDatasetSearches from "!!raw-loader!../../../../code_snippets/api-guide/data/datasets/dataset_searches.sh"; import CodeCC from "!!raw-loader!../../../../code_snippets/python-sdk/Search/as/cc.py"; import CodeCCTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/as/customConcepts.ts"; import CodeOR from "!!raw-loader!../../../../code_snippets/python-sdk/Search/as/or.py"; import CodeORTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/as/or.ts"; import CodeAND from "!!raw-loader!../../../../code_snippets/python-sdk/Search/as/and.py"; import CodeANDTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/as/and.ts"; ## OR Search Operation The **OR** filter lets you expand your search to include results that match any of the specified conditions, rather than requiring all of them to be met. {CodeOR} {CodeORTS} ## AND Search Operation The **AND** operation enables you to refine search results by specifying multiple conditions that must all be satisfied at the same time. For example, if a user searches for images containing both the concepts "dog" and "deer," only those images labeled with both concepts will be returned. {CodeAND} {CodeANDTS} ## Combine Filter and Rank When you combine both `Filter` and `Rank`, filter will be applied before ranking annotations. This is important because limiting the results set on large applications can speed up the overall query drastically when doing a ranking. {CodeCC} {CodeCCTS} {PythonCombineNegate} {JSCombineNegate} {NodeCombineNegate} {JavaCombineNegate} {PHPCombineNegate} {CurlCombineNegate} ## Combine Filter by Metadata and Rank You can also [search over custom metadata](https://docs.clarifai.com/api-guide/search/filter#by-custom-metadata) and easily rank the results. {PythonCombineNegate_3} {JSCombineNegate_3} {NodeCombineNegate_3} {JavaCombineNegate_3} {PHPCombineNegate_3} {CurlCombineNegate_3} ## Negate Search Criteria You can find all the data that is NOT similar to a given criteria. {PythonCombineNegate_2} {JSCombineNegate_2} {NodeCombineNegate_2} {JavaCombineNegate_2} {PHPCombineNegate_2} {CurlCombineNegate_2} ## Find Duplicate Images in Dataset Here's how you can use the `PostInputsSearches` endpoint to identify near-duplicate images within a given [dataset](https://docs.clarifai.com/create-manage/datasets/). You can also use the `min_value` threshold parameter to refine the search results, ensuring that only images surpassing a specified minimum probability resemblance score are included in the output. . {curlDatasetSearches} --- ## Filter # Filter **Select a subset of your data based on useful filters** You can filter and customize your search results to find exactly what you want. Filtering helps you reduce the amount of data returned in search results by removing irrelevant content, or by allowing you to select a specific subset of your data. In annotation search, `Filter` is a list of `Annotation` objects. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: :::tip You can learn how to paginate your API requests results [here](https://docs.clarifai.com/additional-resources/api-overview/pagination). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/by_custom_concepts.py"; import PythonUserID from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/by_user_id.py"; import PythonAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/by_annotation_status.py"; import PythonInputsLongitude from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/add_inputs_longitude_latitude.py"; import PythonOneGeoPoint from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/perform_search_one_geo_point.py"; import PythonTwoGeoPoints from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/perform_search_two_geo_point.py"; import PythonCustomAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/by_custom_annotation_info.py"; import PythonAnnotationInfo from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/py/by_annotation_info.py"; import JSCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/by_custom_concepts.html"; import JSUserID from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/by_user_id.html"; import JSAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/by_annotation_status.html"; import JSInputsLongitude from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/add_inputs_longitude_latitude.html"; import JSOneGeoPoint from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/perform_search_one_geo_point.html"; import JSTwoGeoPoints from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/perform_search_two_geo_point.html"; import JSCustomAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/by_custom_annotation_info.html"; import JSAnnotationInfo from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/js/by_annotation_info.html"; import NodeCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/by_custom_concepts.js"; import NodeUserID from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/by_user_id.js"; import NodeAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/by_annotation_status.js"; import NodeInputsLongitude from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/add_inputs_longitude_latitude.js"; import NodeOneGeoPoint from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/perform_search_one_geo_point.js"; import NodeTwoGeoPoints from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/perform_search_two_geo_point.js"; import NodeCustomAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/by_custom_annotation_info.js"; import NodeAnnotationInfo from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/node/by_annotation_info.js"; import JavaCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/by_custom_concepts.java"; import JavaUserID from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/by_user_id.java"; import JavaAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/by_annotation_status.java"; import JavaInputsLongitude from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/add_inputs_longitude_latitude.java"; import JavaOneGeoPoint from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/perform_search_one_geo_point.java"; import JavaTwoGeoPoints from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/perform_search_two_geo_point.java"; import JavaCustomAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/by_custom_annotation_info.java"; import JavaAnnotationInfo from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/java/by_annotation_info.java"; import PHPCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/by_custom_concepts.php"; import PHPUserID from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/by_user_id.php"; import PHPAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/by_annotation_status.php"; import PHPInputsLongitude from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/add_inputs_longitude_latitude.php"; import PHPOneGeoPoint from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/perform_search_one_geo_point.php"; import PHPTwoGeoPoints from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/perform_search_two_geo_point.php"; import PHPCustomAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/by_custom_annotation_info.php"; import PHPAnnotationInfo from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/php/by_annotation_info.php"; import CurlCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/by_custom_concepts.sh"; import CurlUserID from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/by_user_id.sh"; import CurlAnnotationStatus from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/by_annotation_status.sh"; import CurlInputsLongitude from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/add_inputs_longitude_latitude.sh"; import CurlOneGeoPoint from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/perform_search_one_geo_point.sh"; import CurlTwoGeoPoints from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/perform_search_two_geo_point.sh"; import CurlCustomAnnotation from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/by_custom_annotation_info.sh"; import CurlAnnotationInfo from "!!raw-loader!../../../../code_snippets/api-guide/search/filter/curl/by_annotation_info.sh"; import CodeIT from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/fl_it.py"; import CodeITTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/filterWithInputType.ts"; import CodeMD from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/fl_md.py"; import CodeMDTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/filterWithMetadata.ts"; import CodeGP from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/fl_gp.py"; import CodeGPTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/filterWithGeoPoint.ts"; import CodeIS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/fl_is.py"; import CodeISTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/filterWithInputStatusCode.ts"; import CodeData from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/fl_data.py"; import CodeDataTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/filterWithDatasetID.ts"; import CodeMM from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/fl_mm.py"; import CodeMMTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/filters/multiModalFiltering.ts"; ## By Input Type You can specify that search results should only include inputs of a particular data type, such as images, videos, or text. {CodeIT} {CodeITTS} ## By Custom Concepts After you annotate inputs with custom concepts, you can filter by concepts. :::caution train a model When performing a search with custom concepts, ensure that these concepts are first trained using an `embedding-classifier` model ([transfer-learning](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning/) model). Without this training, the search query will result in an error. Training a model generates embeddings for each custom concept. These concept embeddings are then utilized in the search process. ::: {PythonCustomConcepts} {JSCustomConcepts} {NodeCustomConcepts} {JavaCustomConcepts} {PHPCustomConcepts} {CurlCustomConcepts} ## By User ID If you have collaborators in your app and they helped you annotate your inputs, you can also filter annotations by their user ID. {PythonUserID} {JSUserID} {NodeUserID} {JavaUserID} {PHPUserID} {CurlUserID} ## By Dataset ID You can include a dataset ID in the filter to narrow the search results to only inputs or data entries within that specific dataset. {CodeData} {CodeDataTS} ## By Status Code You can refine search results based on the [status](https://docs.clarifai.com/additional-resources/api-overview/status-codes) of the input data, such as whether it has been successfully processed, is pending processing, or has encountered errors. You can also filter the annotations by their status. {CodeIS} {CodeISTS} {PythonAnnotationStatus} {JSAnnotationStatus} {NodeAnnotationStatus} {JavaAnnotationStatus} {PHPAnnotationStatus} {CurlAnnotationStatus} ## By Geo Location Search by geo location allows you to restrict your search results to a bounding box based on longitude and latitude points. There are two ways you can provide longitude/latitude points. You can provide one point and a radius or you can provide two points. It is important to note that a search by geo location acts as a filter and returns results ranked by any other provided search criteria, whether that is a visual search, concept search, or something else. If no other criterion is provided, results will return in the order the inputs were created, NOT by their distance to the center of the search area. If you are providing one point and a radius, the radius can be in "mile", "kilometer", "degree", or "radian", marked by keywords `withinMiles`, `withinKilometers`, `withinDegrees`, or `withinRadians` respectively. If you are providing two points, a box will be drawn from the uppermost point to the lowermost point, and the leftmost point to the rightmost point. Before you perform a search by geo location, make sure you have added inputs with longitude and latitude points. ### Add Inputs With Longitude and Latitude Points Provide a geo point to an input. The geo point is a JSON object consisting of a longitude and a latitude in GPS coordinate system \(SRID 4326\). There can be at most one single geo point associated with each input. {CodeGP} {CodeGPTS} {PythonInputsLongitude} {JSInputsLongitude} {NodeInputsLongitude} {JavaInputsLongitude} {PHPInputsLongitude} {CurlInputsLongitude} ### Perform a Search With One Geo Point and Radius in Kilometers {PythonOneGeoPoint} {JSOneGeoPoint} {NodeOneGeoPoint} {JavaOneGeoPoint} {PHPOneGeoPoint} {CurlOneGeoPoint} ### Perform a Search With Two Geo Points {PythonTwoGeoPoints} {JSTwoGeoPoints} {NodeTwoGeoPoints} {JavaTwoGeoPoints} {PHPTwoGeoPoints} {CurlTwoGeoPoints} ## By Custom Metadata After you have added inputs with custom metadata, you can search by that metadata. Below is an example of searching over custom metadata. You can exact match any `key`: `value` pair no matter how nested it is. For example, if the metadata on an input is: ```javascript { "keyname": "value1", "somelist": [1,2,3], "somenesting": { "keyname2":"value2", "list2":[4,5] } } ``` Then the following searches will find this: ```javascript { "keyname": "value1" } ``` ```javascript { "somelist": [1,2,3] } ``` ```javascript { "somelist": [1,2] } ``` ```javascript { "somenesting": {"keyname2":"value2"} } ``` ```javascript { "somenesting": {"list2":[5]} } ``` How to perform searches: {CodeMD} {CodeMMTS} {PythonCustomAnnotation} {JSCustomAnnotation} {NodeCustomAnnotation} {JavaCustomAnnotation} {PHPCustomAnnotation} {CurlCustomAnnotation} ## By Annotation Info Each annotation has annotation info. Similar to metadata, you have full control of this field and can be any arbitrary JSON. {PythonAnnotationInfo} {JSAnnotationInfo} {NodeAnnotationInfo} {JavaAnnotationInfo} {PHPAnnotationInfo} {CurlAnnotationInfo} ## Multimodal Filtering Multimodal filtering allows you to refine search results based on a combination of visual and textual criteria. Unlike unimodal filtering, which limits filtering to a single data type — such as images or text — multimodal filtering enables more comprehensive queries by considering multiple modalities at once. {CodeMM} {CodeMMTS} --- ## Rank # Rank **Search your data based on concepts or visual similarity** You can rank order your search results with the intuitive insights of an AI. Your model can identify concepts in your data and rank search results by how confident it is that a given concept is present. You can even rank search results by how similar one input is to another input or region of the input model detected. The search results will return the input and also the annotation, which includes the region. In annotation search, `Rank` is a list of `Annotation` objects. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: :::tip You can learn how to paginate your API requests results [here](https://docs.clarifai.com/additional-resources/api-overview/pagination). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonAppConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/by_clarifaimain_app_concepts.py"; import PythonCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/by_custom_concepts.py"; import PythonClarifaiCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/by_clarifaimain_custom_concepts.py"; import PythonConceptLanguage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/by_concept_another_language.py"; import PythonSearchImage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/search_by_image_url.py"; import PythonImageBytes from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/search_by_image_bytes.py"; import PythonInputID from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/by_input_id.py"; import PythonSearchText from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/py/search_by_text.py"; import JSAppConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/by_clarifaimain_app_concepts.html"; import JSCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/by_custom_concepts.html"; import JSClarifaiCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/by_clarifaimain_custom_concepts.html"; import JSConceptLanguage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/by_concept_another_language.html"; import JSSearchImage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/search_by_image_url.html"; import JSImageBytes from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/search_by_image_bytes.html"; import JSInputID from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/by_input_id.html"; import JSSearchText from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/js/search_by_text.html"; import NodeAppConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/by_clarifaimain_app_concepts.js"; import NodeCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/by_custom_concepts.js"; import NodeClarifaiCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/by_clarifaimain_custom_concepts.js"; import NodeConceptLanguage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/by_concept_another_language.js"; import NodeSearchImage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/search_by_image_url.js"; import NodeImageBytes from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/search_by_image_bytes.js"; import NodeInputID from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/by_input_id.js"; import NodeSearchText from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/node/search_by_text.js"; import JavaAppConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/by_clarifaimain_app_concepts.java"; import JavaCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/by_custom_concepts.java"; import JavaClarifaiCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/by_clarifaimain_custom_concepts.java"; import JavaConceptLanguage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/by_concept_another_language.java"; import JavaSearchImage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/search_by_image_url.java"; import JavaImageBytes from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/search_by_image_bytes.java"; import JavaInputID from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/by_input_id.java"; import JavaSearchText from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/java/search_by_text.java"; import PHPAppConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/by_clarifaimain_app_concepts.php"; import PHPCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/by_custom_concepts.php"; import PHPClarifaiCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/by_clarifaimain_custom_concepts.php"; import PHPConceptLanguage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/by_concept_another_language.php"; import PHPSearchImage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/search_by_image_url.php"; import PHPImageBytes from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/search_by_image_bytes.php"; import PHPInputID from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/by_input_id.php"; import PHPSearchText from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/php/search_by_text.php"; import CurlAppConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/by_clarifaimain_app_concepts.sh"; import CurlCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/by_custom_concepts.sh"; import CurlClarifaiCustomConcepts from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/by_clarifaimain_custom_concepts.sh"; import CurlConceptLanguage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/by_concept_another_language.sh"; import CurlSearchImage from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/search_by_image_url.sh"; import CurlImageBytes from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/search_by_image_bytes.sh"; import CurlInputID from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/by_input_id.sh"; import CurlSearchText from "!!raw-loader!../../../../code_snippets/api-guide/search/rank/curl/search_by_text.sh"; import CodeRKTXT from "!!raw-loader!../../../../code_snippets/python-sdk/Search/ranks/rk_text.py"; import CodeRKTXTTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/ranks/rankWithText.ts"; import CodeIMURL from "!!raw-loader!../../../../code_snippets/python-sdk/Search/ranks/rk_imurl.py"; import CodeIMURLTS from "!!raw-loader!../../../../code_snippets/python-sdk/Search/ranks/rankWithImageUrl.ts"; import PythonSearchByConcept from "!!raw-loader!../../../../code_snippets/api-guide/concepts/python/search_by_concept.py"; import JavaScriptSearchByConcept from "!!raw-loader!../../../../code_snippets/api-guide/concepts/js/search_by_concept.html"; import NodeJSSearchByConcept from "!!raw-loader!../../../../code_snippets/api-guide/concepts/node/search_by_concept.js"; import JavaSearchByConcept from "!!raw-loader!../../../../code_snippets/api-guide/concepts/java/search_by_concept.java"; import PHPSearchByConcept from "!!raw-loader!../../../../code_snippets/api-guide/concepts/php/search_by_concept.php"; import CurlSearchByConcept from "!!raw-loader!../../../../code_snippets/api-guide/concepts/curl/search_by_concept.sh"; import CodeOutputExample from "!!raw-loader!../../../../code_snippets/api-guide/concepts/code_output_examples/search_by_concept.txt"; import JSONOutputExample from "!!raw-loader!../../../../code_snippets/api-guide/concepts/code_output_examples/search_by_concept.js"; ## Search by Concepts Once your inputs are indexed, you can search for them by concepts. ### By Clarifai/main App Concepts When you add an input, it automatically gets predictions from the workflow in your [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows), which is typically from the [`clarifai/main`](https://clarifai.com/clarifai/main) app, such as the [Universal](https://clarifai.com/clarifai/main/workflows/Universal) workflow. You can search by those predictions. [Click here](https://docs.clarifai.com/api-guide/concepts/create-get-update#list-concepts) to learn how to get a list of concepts available in the app. {PythonAppConcepts} {JSAppConcepts} {NodeAppConcepts} {JavaAppConcepts} {PHPAppConcepts} {CurlAppConcepts} ### By Custom Concepts After you have added inputs, annotated the inputs, and trained a custom model, you can search by those concepts. :::caution train a model When performing a search with custom concepts, ensure that these concepts are first trained using an `embedding-classifier` model ([transfer-learning](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning/) model). Without this training, the search query will result in an error. Training a model generates embeddings for each custom concept. These concept embeddings are then utilized in the search process. ::: {PythonCustomConcepts} {JSCustomConcepts} {NodeCustomConcepts} {JavaCustomConcepts} {PHPCustomConcepts} {CurlCustomConcepts} ### By Clarifai/main and Custom Concepts You can combine a search to find inputs that have concepts you have supplied as well as predictions from your model. {PythonClarifaiCustomConcepts} {JSClarifaiCustomConcepts} {NodeClarifaiCustomConcepts} {JavaClarifaiCustomConcepts} {PHPClarifaiCustomConcepts} {CurlClarifaiCustomConcepts} ### By Concept in Another Language Concepts that have a translation into another language can be searched for in that language, even without having the default language for your app being in that language. This uses the Clarifai's knowledge graph to lookup the translation and then perform the search. For example, if your app is in English and you want to search for "dog" in Japanese, then you could search with `language="ja"` and `name="犬"`. {PythonConceptLanguage} {JSConceptLanguage} {NodeConceptLanguage} {JavaConceptLanguage} {PHPConceptLanguage} {CurlConceptLanguage} ### By Using ConceptSearches Endpoint You can search for concepts by `name`, even across different languages, using the `ConceptSearches` endpoint. {PythonSearchByConcept} {JavaScriptSearchByConcept} {NodeJSSearchByConcept} {JavaSearchByConcept} {PHPSearchByConcept} {CurlSearchByConcept} Raw Output Example {CodeOutputExample} JSON Output Example {JSONOutputExample} ## Search by Visual Similarity You can use images to search through your collection. The API will return ranked results based on how similar the results are to the image you provided in your query. ### Search by Image URL {CodeIMURL} {CodeIMURLTS} {PythonSearchImage} {JSSearchImage} {NodeSearchImage} {JavaSearchImage} {PHPSearchImage} {CurlSearchImage} ### Search by Image Bytes You can also search for an input by bytes, with the bytes being from local storage. {PythonImageBytes} {JSImageBytes} {NodeImageBytes} {JavaImageBytes} {PHPImageBytes} {CurlImageBytes} ### By Input ID If the input has been indexed, we can use the input ID. If there are multiple embeddings \(for example multiple regions\), we will average the embeddings. {PythonInputID} {JSInputID} {NodeInputID} {JavaInputID} {PHPInputID} {CurlInputID} ## Search by Text Similarity You can use texts to search through your collection of texts. The text-to-text search will return ranked results based on how similar the results are to the text you provided in your query. :::tip - To perform text-to-text searches, you could choose a workflow that includes a text embedder and a clusterer, such as the [**Text**](https://clarifai.com/clarifai/main/workflows/Text) workflow, as the [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/) for your application. - To perform text-to-image searches, you could choose [**Universal**](https://clarifai.com/clarifai/main/workflows/Universal) as the base workflow, which allows you to use texts to search through your collection of images. ::: {CodeRKTXT} {CodeRKTXTTS} {PythonSearchText} {JSSearchText} {NodeSearchText} {JavaSearchText} {PHPSearchText} {CurlSearchText} --- ## Search via the UI # Search via the UI **Use vector search to find, sort, and filter what you need in your unstructured data** Our Smart Search feature leverages vector search capabilities to power the search experience. Available directly on the Input-Manager screen, it enables you to seamlessly search through both structured and unstructured data by leveraging AI to surface the most relevant results with ease. ![](/img/spacetime.svg) > **Note**: _The Inputs-Manager screen is the main page showcasing all the inputs in your app. You can access it by clicking the **Inputs** option in your app's collapsible left sidebar. The screen has [two modes](https://docs.clarifai.com/create-manage/inputs/#inputs-manager-screen): Input Mode (default) and Object Mode._ > ![](/img/community/search/input-manager.png) :::info When performing a Smart Search with custom concepts, ensure that these concepts are first trained using an embedding-classifier model ([transfer-learning](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning/) model). Without this training, the search query may not work. On the Inputs-Manager screen, concepts that have already been trained with a model are marked with a blue circle, while untrained concepts are marked with a grey circle. ::: ## Types of Smart Search We offer the following types of Smart Search options in our platform. - [**Smart Image Search**](https://docs.clarifai.com/create-manage/search/ui/visual/) — Allows you to retrieve images sorted by their visual relevance to a query in the form of: - **Image** — Provide a reference image of interest to compare inputs against. - **Concept** — Provide a trained concept to compare input predictions against. - **Caption** — Provide a full-text description to compare inputs against. - [**Smart Object Search**](https://docs.clarifai.com/create-manage/search/ui/object) — Allows you to retrieve annotated objects (bounding boxes within images) sorted by their visual relevance to a query in the form of: - **Image** — Provide a reference image of interest to compare inputs against. - **Concept** — Provide a trained concept to compare input predictions against. - **Caption** — Provide a full-text description to compare inputs against. - [**Smart Text Search**](https://docs.clarifai.com/create-manage/search/ui/text-search) — Allows you to retrieve text data sorted by their content and semantic similarity to a query in the form of: - **Text** — Provide a text description to compare input predictions against. - **Concept** — Provide a trained concept to compare input predictions against. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Combine or Negate # Combine or Negate **Add or subtract items from your search query** Let's demonstrate how you can perform combined and negative searches on the Input-Manager screen. Go to your app and select the **Inputs** option on the collapsible left sidebar. You’ll be redirected to the Input-Manager page, where you can see the inputs available in your app. ## Negate Search Criteria You can find all the data that is NOT similar to a given criteria. On the **Labels** section on the Input-Manager screen, identify the concept you want to find its negative values and click the checkbox next to it. For example, let's search for all inputs that are NOT annotated as “dog”. After selecting the concept, hover over it. Double arrows will appear with the following tooltip: `Filter for inputs that do NOT match this value`. Click the double arrows to perform the negative search. ![smart image search results](/img/smart-search/search_8.png) You’ll notice that the word **NOT** will be added to the concept, and the search results will be populated on the page. ![smart image search results](/img/smart-search/search_9.png) ## Combine Search Criteria You can add multiple search parameters that expand your search criteria. You can even combine negated search terms for more advanced tasks. Combining search criteria always uses the "AND" operation. "OR" is not an option. For example, you can combine two concepts and search by them. ![smart image search results](/img/smart-search/search_13.png) --- ## Filter # Filter **Reduce noise by filtering out unwanted data** You can use the **Input Filters** section on the Input-Manager screen to customize how inputs are displayed. By default, inputs are sorted by creation date, with the newest at the top. You can filter inputs by dataset, labels, type, or metadata. You can also combine multiple filter parameters to expand your search criteria and get more refined results. ## Filter by Datasets You can filter the display of inputs based on their datasets. To do so, identify the dataset you want to display inputs for and click the checkbox next to it. Note that the number of inputs matching your criteria will be displayed at the top of the display area. ![](/img/smart-search/search_24.png) If you want to filter out inputs that do not belong to a specific dataset, select the dataset and hover over it. Double arrows will appear with a tooltip: `Filter for inputs that do NOT match this value.` Click the double arrows to apply the negative filter. You’ll notice that the word **NOT** will be added next to the dataset's name, and the filter results will be populated on the page. Additionally, you can use the "Unassigned" parameter to filter all inputs that have not been assigned to any dataset. :::tip To reset any applied filters, click the **Clear** button that appears after applying the filters. ::: ## Filter by Labels You can filter the display of inputs based on their labeled concepts. To do so, identify the concept you want to display inputs for and click the checkbox next to it. You can also filter out inputs that do not belong to a specific concept. Start by selecting the label and hovering over it. Double arrows will appear with a tooltip: `Filter for inputs that do NOT match this value.` Click the double arrows to apply the negative filter. You’ll notice that the word **NOT** will be added next to the concept’s name, and the filter results will be populated on the page. Additionally, you can use the "Unlabeled" parameter to filter all inputs that have not been annotated with any concept. ![](/img/smart-search/search_25.png) You can also click the ellipsis next to the **Labels** section to reveal additional options for filtering the display of your inputs. You can filter the display by these parameters: - **Operator** — You can specify either the `or` (default) or `and` operator to determine how this filter should be applied when multiple values are selected. - Choosing `or` will filter for inputs that match _any_ of the selected values. - Choosing `and` will return only the inputs that match _all_ the selected values. Note that this setting does not affect how negated values are interpreted. Inputs matching any negated selections are always removed after those matching the positively selected values have been identified. - **[Labeler](https://docs.clarifai.com/portal-guide/annotate/create-a-task/#step-6-add-collaborators)** — You can filter the display by the labeler who annotated the inputs. By default, inputs from all labelers are shown. - **[Status](https://docs.clarifai.com/api-guide/advanced-topics/status-codes/#annotation-related-codes-24xxx)** — You can filter the display based on the status of the annotation exercise. The available options are `Success` (default), `Awaiting Review`, `Awaiting Consensus Review`, or `Pending`. ## Filter by Input Type You can filter the display of inputs based on their types — audio, image, text, and video. ![](/img/smart-search/search_26.png) ## Filter by Metadata You can filter the display of inputs based on their custom metadata. Note that the metadata should be a valid JSON object. Metadata are additional pieces of information you attach to your inputs when uploading them to the Clarifai platform. This can include product IDs, user IDs, or any other relevant details necessary for achieving specific business outcomes. You can use metadata to quickly filter your data and combine it with other ranking criteria. ![](/img/smart-search/search_27.png) --- ## Object Search # Object Search **Use vector search to sort, rank, and retrieve annotated objects within images** Smart Object Search, also called localized search, is our proprietary feature that uses deep learning techniques to sort, rank, and retrieve annotated objects (bounding boxes) within images based on their content and similarity. It extends the capabilities of traditional object search, which typically relies on object labels or metadata, by leveraging advanced techniques such as object embeddings and vector search. Here's how our Smart Object Search feature works: - **Vector generation**: We transform each annotated object (bounding box) within an image into a high-dimensional vector representation known as an object embedding, and store it in our vector database. Object embeddings are learned using deep learning techniques that capture the visual features and characteristics of the objects. - **Vector search**: When a user performs an object search or provides an example object (bounding box), we convert that object into an object embedding vector. Then, we perform a similarity search in the object vector store using the query object's embedding vector. - **Similarity ranking**: We retrieve the annotated objects that are most visually similar to the query object based on their object embeddings. The ranking of the retrieved objects is determined by the similarity scores between the query object and each of the retrieved objects. Our vector search engine ranks objects with higher similarity scores higher and considers them more visually similar to the query. Let’s demonstrate how you can carry out different types of Smart Object Searches. Start by going to your app and selecting the **Inputs** option in the collapsible left sidebar. You’ll be redirected to the Input-Manager page. Then, click the **Object Mode** button in the upper-right section of the Input-Manager screen. This will display a gallery of objects that have been labeled on your inputs. ![](/img/smart-search/search_17.png) You can then perform various object search actions on the screen. ## Smart Object Search by Visual Similarity This feature is also called Object Similarity Search or Reverse Object Search. It allows you to rank, sort, and retrieve objects based on their visual similarity to a provided query image. There are two ways you can use to perform this type of search. ### Search by Uploading an Image To upload an image to be used for search, click the **Visual Search** button at the right end of the inputs search bar. In the menu that drops down, upload the image you want to use. You can upload the image from your local storage, drag and drop it, or paste its publicly accessible URL. > The image should be in one of the following supported formats: SVG, PNG, JPG, or GIF, with a maximum size of 800 pixels by 400 pixels. ![](/img/smart-search/search_18.png) A pop-up will appear, which allows you to crop the area of interest in the image. The search results will include annotated objects from within images that resemble the cropped area of your provided image. After cropping the image, select the **Find visually similar Object** option, and click the **Search inputs** button. ![](/img/smart-search/search_19.png) A thumbnail of the cropped image will be added to the inputs search bar and your search results will be displayed in a ranking fashion — from the most visually similar to the least visually similar. ![](/img/smart-search/search_20.png) ### Search by an Already Annotated Image If you hover over an already annotated image you want to use to perform the visual similarity search, some icons will appear on the left side of the image. Click the magnifying glass icon. ![smart object search by visual similarity](/img/smart-search/search_21.png) A thumbnail of the image will be added to the inputs search bar and your annotated search results will be displayed in a ranking fashion — from the most visually similar to the least visually similar. ![smart object search results](/img/smart-search/search_22.png) ## Smart Object Search by Concept You can rank, sort, and retrieve objects based on the [concepts](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete) used for annotating them. Just provide a query concept and the most relevant matches associated with that concept will be displayed. Go to the inputs search bar, add an hashtag (#), and type the concept you want to search for. After performing the search, the annotated objects will be displayed within the page based on their content and similarity. ![smart object search results](/img/smart-search/search_23.png) --- ## Text Search # Text Search **Use vector search to sort, rank, and retrieve texts** Smart Text Search is our proprietary feature that uses deep learning techniques to sort, rank, and retrieve text data based on their content and semantic similarity. It goes beyond traditional text search methods that rely solely on exact keyword matching and incorporates advanced techniques like text embeddings and vector search. Here's how our Smart Text Search feature works: - **Vector generation**: We transform each piece of text, such as a sentence or paragraph, into a numerical vector representation known as a text embedding, and store it in our vector database. Text embeddings are learned using natural language processing techniques that capture the semantic meaning and contextual information of the text. - **Vector search**: When a user performs a text search or provides a query text, we convert that query into a text embedding vector. Then, we perform a similarity search against the text vector store using the query text's embedding vector. - **Similarity ranking**: We retrieve text data that is most similar to the query text based on their text embeddings. The ranking of the retrieved text documents is typically determined by the similarity scores between the query text and each of the retrieved texts. Our vector search engine ranks texts with higher similarity scores higher and considers them more semantically similar to the query. Let’s demonstrate how you can carry out different types of Smart Text Searches. Start by going to your app and selecting the **Inputs** option in the collapsible left sidebar. You’ll be redirected to the Input-Manager page, where you can see the text inputs available on your app. You can then perform various text search actions on them. :::tip To perform text searches successfully, you should choose a workflow that includes a text embedder and a clusterer, such as the [Text](https://clarifai.com/clarifai/main/workflows/Text) workflow, as the [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/) for your application. ::: ## Smart Text Search by Semantic Similarity This feature is also called Semantic Search. It allows you to rank, sort, and retrieve texts based on their semantic similarity to a provided query text. If you hover over the text you want to use to perform the semantic similarity search, some icons will appear on the left side of the text. Click the magnifying glass icon. ![smart image search by visual similarity](/img/smart-search/search_5.png) The text you selected will be added to the inputs search bar and your search results will be displayed in a ranking fashion—from the most semantically similar to the least semantically similar. ![smart image search by visual similarity](/img/smart-search/search_6.png) ## Smart Text Search by Predicted Concept You can rank, sort, and retrieve texts based on the [concepts]( https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete) predicted by AI models. Just provide a query concept and the most relevant matches associated with that concept will be displayed. Go to the inputs search bar and start typing the concept you want to search for. You’ll notice that a small drop-down list will appear having the concepts you’ve trained with your model as well as the concepts present in the base workflow of your app. Your model’s concepts will appear first in the list. Choose the concept you want to search for, and your results will be displayed on the page. The search results will be ranked based on the inputs with the highest predicted values for the given concept. ![smart image search results](/img/smart-search/search_7.png) --- ## Image Search # Image Search **Use vector search to sort, rank, and retrieve images** Smart Image Search is our proprietary feature that uses deep learning techniques to sort, rank, and retrieve images based on their content and visual similarity. It goes beyond traditional image search methods that rely solely on image metadata or textual annotations. Here's how our Smart Image Search feature works: - **Vector generation**: We transform each image in your app into a high-dimensional vector representation known as an image embedding, and store it in our vector database. Image embeddings are learned using deep learning techniques that capture the visual features and patterns present in the images. - **Vector search**: When a user performs an image search or provides an example image, we convert that image into an image embedding vector. Then, we perform a similarity search against the image vector store using the query image's embedding vector. - **Similarity ranking**: We retrieve the images that are most similar to the query image based on their image embeddings. The ranking of the retrieved images is typically determined by the similarity scores between the query image and each of the retrieved images. Our vector search engine ranks images with higher similarity scores higher and considers them more visually similar to the query. Let’s demonstrate how you can carry out different types of Smart Image Searches. Start by going to your app and selecting the **Inputs** option in the collapsible left sidebar. You’ll be redirected to the Input-Manager page, where you can see the inputs available on your app. You can then perform various image search actions on them. ## Smart Image Search by Visual Similarity This feature is also called Image Similarity Search or Reverse Image Search. It allows you to rank, sort, and retrieve images based on their visual similarity to a provided query image. There are two ways you can use to perform this type of search. ### Search by Uploading an Image To upload an image to be used for search, click the **Visual Search** button at the right end of the inputs search bar. In the menu that drops down, upload the image you want to use. You can upload the image from your local storage, drag and drop it, or paste its publicly accessible URL. The image should be in one of the following supported formats: SVG, PNG, JPG, or GIF, with a maximum size of 800 pixels by 400 pixels. ![](/img/smart-search/search_14.png) A pop-up will appear, which allows you to crop the area of interest in the image. The search results will include images that resemble the cropped area of your provided image. After cropping the image, select the **Find visually similar Images** option (which is the default), and click the **Search inputs** button. ![](/img/smart-search/search_15.png) A thumbnail of the cropped image will be added to the inputs search bar and your search results will be displayed in a ranking fashion — from the most visually similar to the least visually similar. ![](/img/smart-search/search_16.png) ### Search by an Already Uploaded Image If you hover over an already uploaded image you want to use to perform the visual similarity search, some icons will appear on the left side of the image. Click the magnifying glass icon. ![smart image search by visual similarity](/img/smart-search/search_1.png) A thumbnail of the image will be added to the inputs search bar and your search results will be displayed in a ranking fashion — from the most visually similar to the least visually similar. ![smart image search results](/img/smart-search/search_2.png) ## Smart Image Search by Predicted Concept You can rank, sort, and retrieve images based on the [concepts](https://docs.clarifai.com/portal-guide/concepts/create-get-update-delete) predicted by AI models. Just provide a query concept and the most relevant matches associated with that concept will be displayed. Go to the inputs search bar, add an hashtag (#), and start typing the concept you want to search for. ![smart image search results](/img/smart-search/search_3.png) You’ll notice that a small drop-down list will appear having the concepts you’ve trained with your model as well as the concepts present in the [base workflow](https://docs.clarifai.com/clarifai-basics/applications/application-settings#base-workflow) of your app. Your model’s concepts will appear first in the list. Choose the concept you want to search for, and your results will be displayed on the page. The search results will be ranked based on the inputs with the highest predicted values for the given concept. ![smart image search results](/img/smart-search/search_4.png) ## Smart Image Search by Caption You can rank, sort, and retrieve images based on a predicted match to a query caption text. You just need to provide a caption text that best describes the images you want to search for, and the most relevant matches associated with that query will be displayed. Performing searches using full texts allow you to provide a much more in-depth context and retrieve more relevant results — as compared to other types of searches. :::info You need to choose the **Universal** [base workflow](https://docs.clarifai.com/clarifai-basics/applications/application-settings#base-workflow) for the Smart Image Search by Caption feature to work on your app. ::: Here is an example of how you can use this feature. ![smart image search results](/img/smart-search/search_10.png) You can also get more specific and mention numbers in your query caption text. ![smart image search results](/img/smart-search/search_12.png) You can even get search results with OCR-like (optical character recognition) capabilities. ![smart image search results](/img/smart-search/search_11.png) --- ## Workflows # Workflows **Combine multiple models and perform complex operations with workflows** Workflows is a powerful and innovative Clarifai feature that enables you to seamlessly combine multiple models and [agent system operators](https://docs.clarifai.com/portal-guide/agent-system-operators/) into a unified pipeline on our platform. With workflows, you can create a powerful multi-model system that meets various use cases in a single call — instead of relying only on one model. Workflows provide a modular architecture for your inference pipeline. They provide the building blocks for advanced machine learning ensemble modeling and business logic. Use workflows to process complex logic, gain insights, and build targeted solutions. :::note With workflows, you can work with your machine learning models like they are nodes in a graph. ::: Clarifai makes it easy to connect together models of different kinds so that you can perform complex operations on your data, and build solutions that target your specific business needs. ![](/img/mesh.svg) --- ## Base Workflows # Base Workflows **Index your data and provide your app with a default knowledge base** The base workflow acts as the default knowledge base for your app and provides the basic structure for indexing your data. It gives you a "head start" when working with your data — by pre-indexing your inputs for search and by providing a default embedding for your custom models. ## How it Works Your base workflow makes your data sortable and searchable as soon as it is uploaded. When you add an input to your app, the base workflow of your app computes the outputs from the model(s) in that workflow and indexes those outputs. A base workflow usually contains [an embedding model](https://docs.clarifai.com/api-guide/predict/embeddings), which indexes the inputs. Those indexed outputs enable search and [transfer-learn](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning/) based training on top of the base workflow model(s). ## How to Choose a Base Workflow You can choose a base workflow when [creating a new application](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/). After selecting the primary input type for your app, the base workflow will be automatically selected for you — within the collapsible **Advanced Settings** field. Alternatively, you can select the one that aligns most with your specific use case from the drop-down list accessible when you click the workflows search box. :::tip default base workflow - If you select **Image/Video** as the primary input type when creating an application, [Universal](https://clarifai.com/clarifai/main/workflows/Universal) will automatically be set as the default base workflow. If you select **Text/Document** as the input type, [Text](https://clarifai.com/clarifai/main/workflows/Text) will be set as the default base workflow. - If you create an application via the API, Universal will be set as the default base workflow unless you [specify a different one](https://docs.clarifai.com/api-guide/workflows/base-workflows/#update-your-base-workflow). ::: ![choose base workflow](/img/community_2/base_workflow_choose.png) ## How to Change a Base Workflow You can change the base workflow once your app is created. You can choose a public workflow available in the Clarifai app [here](https://clarifai.com/clarifai/main/workflows) or a workflow you've created in your app. ### Update via the UI To change it, go to your [**App Settings**](https://docs.clarifai.com/clarifai-basics/applications/application-settings#base-workflow) page and select the one you prefer from the list that drops down when you click the workflows search box. After selecting your preferred base workflow, click the **Change Base Workflow** button. :::caution reindexing - You can update the base workflow and choose to reindex existing inputs in your app. Reindexing will process all inputs through the new base workflow. This may take some time, and could incur costs. To avoid reindexing, do not click the **Reindex existing inputs** checkbox. - During reindexing, existing data from the old base workflow isn't deleted. This means you can easily revert back to the old base workflow if needed. ::: ![change base workflow](/img/community_2/base_workflow_change.png) ### Update via the API import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonUpdateWorkflow from "!!raw-loader!../../../code_snippets/api-guide/model/deep_training/update_default_workflow.py"; import JSUpdateWorkflow from "!!raw-loader!../../../code_snippets/api-guide/model/deep_training/update_default_workflow.html"; import NodeUpdateWorkflow from "!!raw-loader!../../../code_snippets/api-guide/model/deep_training/update_default_workflow.js"; import JavaUpdateWorkflow from "!!raw-loader!../../../code_snippets/api-guide/model/deep_training/update_default_workflow.java"; import PHPUpdateWorkflow from "!!raw-loader!../../../code_snippets/api-guide/model/deep_training/update_default_workflow.php"; import CurlUpdateWorkflow from "!!raw-loader!../../../code_snippets/api-guide/model/deep_training/update_default_workflow.sh"; Note that, as previously mentioned, updating the base workflow will re-index your app, processing all inputs through the new base workflow. This may take some time, and could incur costs. You could avoid the costs by deleting all your inputs before updating the base workflow. {PythonUpdateWorkflow} {JSUpdateWorkflow} --> {NodeUpdateWorkflow} {JavaUpdateWorkflow} {PHPUpdateWorkflow} --> {CurlUpdateWorkflow} ## Base Workflows Use Cases You should go for a workflow that optimizes your custom model performance. For example, if you're training a custom model around food, you could choose the **Food** model as your **Base Workflow** to yield better results. Below is a table outlining the potential use cases for various base workflows. You can refer to this table to determine the workflow that best suits your needs. ### Empty |Workflow |Use Case | |--------------------------------------|------------------------| | [Empty](https://clarifai.com/clarifai/main/workflows/Empty) | Empty workflow for apps that do not need inputs to be indexed for search and/or transfer learning. For example, if the app is used for deep training and/or labeling, you can set the base workflow as **Empty** to speed up the input upload process; later, you can change to another workflow to index your app. | ### Text Workflows |Workflow |Use Case | |--------------------------------------|------------------------| | [Text](https://clarifai.com/clarifai/main/workflows/Text) | A single-model workflow of text embedding model for general English text | | [BAAI-General-Embedding-Base-en](https://clarifai.com/clarifai/main/workflows/baai-general-embedding-base-en) | A workflow for text search | | [Universal](https://clarifai.com/clarifai/main/workflows/Universal) | A universal workflow for text and image search | | [Roberta-embedder](https://clarifai.com/clarifai/main/workflows/Roberta-embedder) | | | [MSMARCO-Distilbert-Text-Search](https://clarifai.com/clarifai/main/workflows/msmarco-distilbert-text-search) | Text search workflow for TREC (Text REtrieval Conference) 2022 | | [Text-Moderation](https://clarifai.com/clarifai/main/workflows/Text-Moderation)| A single-model text moderation workflow that combines classification and embedding functions to classify harmful text content | ### Image Workflows |Workflow |Use Case | |--------------------------------------|------------------------| |[Demographics](https://clarifai.com/clarifai/main/workflows/Demographics) | Multi-model workflow that detects, crops, and recognizes demographic characteristics of faces. Visually classifies age, gender, and multi-culture | | [Universal](https://clarifai.com/clarifai/main/workflows/Universal) | A universal workflow for text and image search | | [Apparel](https://clarifai.com/clarifai/main/workflows/Apparel) | | |[Face-V4](https://clarifai.com/clarifai/main/workflows/Face-V4) | | | [General-Detection](https://clarifai.com/clarifai/main/workflows/General-Detection) | A general image detection workflow that detects a variety of common objects, and enables visual search using general embeddings on detected regions | | [OCR-Translation-English-Spanish](https://clarifai.com/clarifai/main/workflows/ocr-translation-english-spanish)| | | [Face-Sentiment](https://clarifai.com/clarifai/main/workflows/Face-Sentiment)| Multi-model workflow that combines face detection and sentiment classification of 7 concepts: anger, disgust, fear, neutral, happiness, sadness, contempt, and surprise | | [Wedding](https://clarifai.com/clarifai/main/workflows/Wedding) | Visually classify wedding-related content and enable visual search using specific embeddings | | [Food](https://clarifai.com/clarifai/main/workflows/Food)| A workflow that combines detection, classification, and embedding functions to visually classify food items and enable visual search using embeddings | | [Moderation](https://clarifai.com/clarifai/main/workflows/Moderation) | An image moderation workflow that combines detection, classification, and embedding to classify harmful content and enable visual search using embeddings | | [Travel](https://clarifai.com/clarifai/main/workflows/Travel) | A workflow that combines detection, classification, and embedding to classify travel-related properties and items, and enable visual search using embeddings | | [Face](https://clarifai.com/clarifai/main/workflows/Face) | A workflow that combines detection, recognition, and embedding to generate face landmarks and enable visual search using detected faces' embeddings | | [General](https://clarifai.com/clarifai/main/workflows/General) | A general image workflow that combines detection, classification, and embedding to identify general concepts, including objects, themes, moods, etc. | | [Visual-Segmenter](https://clarifai.com/clarifai/main/workflows/Visual-Segmenter) | Generates image segmentation masks for concepts in an image | | [Segmentation](https://clarifai.com/clarifai/main/workflows/segmentation) | | | [Person-Tracking](https://clarifai.com/clarifai/main/workflows/person-tracking) | | | [Person-Tracking-Lite](https://clarifai.com/clarifai/main/workflows/person-tracking-lite) | | | [Vehicle-Detection0](https://clarifai.com/clarifai/main/workflows/Vehicle-Detection0) | | ### Audio Workflows |Workflow |Use Case | |--------------------------------------|------------------------| | [ASR-Sentiment](https://clarifai.com/clarifai/main/workflows/asr-sentiment)| Obtains the sentiment of an audio. Takes audio as input, runs an Audio Speech Recognition (ASR) model, and uses the resulting text as input to a text sentiment model | |[Audio-Sentiment](https://clarifai.com/clarifai/main/workflows/audio-sentiment) | Obtains the sentiment of an audio. Takes audio as input, runs an Audio Speech Recognition (ASR) model, and uses the resulting text as input to a text sentiment model | | [ASR-Summarization](https://clarifai.com/clarifai/main/workflows/asr-summarization)| Summarize text from English audio | --- ## Create Workflows # Create Workflows **Create computational graphs that include one or more models** You can build workflows using any Clarifai models or custom models you've created in your app. Outputs from one model can serve as inputs to another — provided the receiving model supports the input type. The input and output types supported by your custom workflows depend on the capabilities of the individual models used to construct them. This model-linking approach enables you to form a graph of interconnected models, allowing you to build sophisticated AI solutions tailored to specific use cases. ## Supported Input and Output Types Different models support different input and output formats. Here are some common examples: - **Inputs** — Concepts, embeddings, images, videos, and regions. - **Outputs** — Concepts, clusters, and regions. ## Create via the UI Let’s demonstrate how to create workflows with a simple example that first extracts text from an image and then translates the extracted text to Spanish. We'll connect the following two Clarifai models to achieve our objective: * The [ocr-scene-english-paddleocr](https://clarifai.com/clarifai/main/models/ocr-scene-english-paddleocr) model, which detects and recognizes English texts in images; * The [text-translation-english-spanish](https://clarifai.com/helsinkinlp/translation/models/text-translation-english-spanish) model translates texts from English to Spanish. We'll specify the IDs of the models and their versions — since a model can have several versions. :::note You can add up to 20 models to a single workflow. ::: ### Step 1: Create Application Let's begin by creating an application that will act as the container for all the related models and workflows for this particular project. [Click here](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/#create-an-application-on-the-portal) to learn how to create an application on the Clarifai platform. :::note When creating the application, select the Text/Document option as the primary input type. ::: ### Step 2: Create a New Workflow To create a new workflow, select the Workflows option in the collapsible left sidebar. Next, click the **Create Workflow** button in the upper-right corner of the page. ![alt_text](/img/community_2/input_nodes_create_new_workflow.png) ### Step 3: Create Your Nodes You'll be redirected to a simple, no-code, drag-and-drop interface that allows you to connect your models together. You'll need to connect the input nodes in your workflow. You can link your nodes to any nodes that precede them in the visual graph. #### Name Your Workflow Let's start by clicking the input field in the upper section of the page and providing a name for the custom workflow. #### Search for First Node Next, in the left sidebar, search for the optical-character-recognizer node. This allows you to configure a model that enables the extraction of texts from images, such as scans of printed pages or photos of street signs. After finding the node, drag and drop it on the empty workspace pane and connect it to the IN element. ![alt_text](/img/community_2/input_nodes_empty_pane.png) :::tip You can use the tools on the left side of the workspace pane to manage the workflow creation process. These tools enable you to zoom in and out, fit the view, arrange the workflow, reset the workspace, and perform other actions to help you efficiently design and organize your workflow. ::: #### Search for the Second Node Next, search for the text-to-text node. This allows you to configure a model that enables the transformation of one kind of text into another. After finding the second node, drag and drop it on the workspace and draw a line that connects it to the first node. This shows the flow of information from one node to another. ![alt_text](/img/community_2/input_nodes_add_another_model.png) ### Step 4: Search for Models Click the optical-character-recognizer node. And on the search box that appears on the right side of the page, specify the ocr-scene-english-paddleocr model as the one to use for optical character recognition. Also, select the version of the model you want to use. ![alt_text](/img/community_2/input_nodes_ocr_model.png) Similarly, click the text-to-text node and specify the text-translation-english-spanish model as the one to use for translating the extracted text from English to Spanish. Also, select its version. ![alt_text](/img/community_2/input_nodes_text_to_text.png) ### Step 5: Save Workflow Finally, click the **Save Workflow** button to save the workflow. This will save the state of your workflow. Now, you are ready to [predict](https://docs.clarifai.com/portal-guide/workflows/working-with-workflows#using-a-workflow) using your brand-new workflow. ![alt_text](/img/community_2/input_nodes_save_workflow.png) ## Create via the API import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeCW from "!!raw-loader!../../../code_snippets/python-sdk/workflows/create_workflow.py"; import PythonSampleNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/sample_workflow_multiple_nodes.py"; import PythonSuppressNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/suppress_output_from_nodes.py"; import JSSampleNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/sample_workflow_multiple_nodes.html"; import JSSuppressNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/suppress_output_from_nodes.html"; import NodeSampleNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/sample_workflow_multiple_nodes.js"; import NodeSuppressNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/suppress_output_from_nodes.js"; import JavaSampleNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/sample_workflow_multiple_nodes.java"; import JavaSuppressNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/suppress_output_from_nodes.java"; import PHPSampleNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/sample_workflow_multiple_nodes.php"; import PHPSuppressNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/suppress_output_from_nodes.php"; import CurlSampleNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/sample_workflow_multiple_nodes.sh"; import CurlSuppressNodes from "!!raw-loader!../../../code_snippets/api-guide/workflows/input_nodes/suppress_output_from_nodes.sh"; import PythonCreate from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/create.py"; import JSCreate from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/create.html"; import NodeCreate from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/create.js"; import JavaCreate from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/create.java"; import PHPCreate from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/create.php"; import CurlCreate from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/create.sh"; import PythonWorkflowPredict from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/workflow_predict.py"; import JSWorkflowPredict from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/workflow_predict.html"; import NodeWorkflowPredict from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/workflow_predict.js"; import JavaWorkflowPredict from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/workflow_predict.java"; import PHPWorkflowPredict from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/workflow_predict.php"; import CurlWorkflowPredict from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/workflow_predict.sh"; import CodeOutputExample1 from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/sample_workflow_predict.txt"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ### Create Workflow In this example, we'll create a simple custom workflow that first extracts text from an image and then translates the extracted text to Spanish. We'll connect the following two models to achieve our objective: - The [ocr-scene-english-paddleocr](https://clarifai.com/clarifai/main/models/ocr-scene-english-paddleocr) model, which detects and recognizes English texts in images; - The [text-translation-english-spanish](https://clarifai.com/helsinkinlp/translation/models/text-translation-english-spanish) model, which translates texts from English to Spanish. We'll specify the IDs of the models and their versions — since a model can have several versions. {PythonCreate} {JSCreate} {NodeCreate} {JavaCreate} {PHPCreate} {CurlCreate} #### Predict With Workflow After creating the workflow, let's now use it to extract texts from [this image](https://samples.clarifai.com/featured-models/ocr-woman-holding-sold-sign.jpg) and translate them into Spanish. The response will contain the predictions each model in the workflow returns for the input. {PythonWorkflowPredict} {JSWorkflowPredict} {NodeWorkflowPredict} {JavaWorkflowPredict} {PHPWorkflowPredict} {CurlWorkflowPredict} ```csharp // Insert here the initialization code as outlined on this page: // https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions var response = client.PostModelOutputs( new PostModelOutputsRequest() { UserAppId = new UserAppIDSet() { UserId = "excaliburne", AppId = "moderation-test" }, ModelId = "aaa03c23b3724a16a56b629203edc62c", // () { new Input() { Data = new Data() { Image = new Image() { Url = "https://samples.clarifai.com/dog2.jpeg" } } } } } }, metadata ); if (response.Status.Code != StatusCode.Success) throw new Exception("Request failed, response: " + response); Console.WriteLine("Predicted concepts:"); foreach (var concept in response.Outputs[0].Data.Concepts) { Console.WriteLine($"{concept.Name, 15} {concept.Value:0.00}"); } ``` --> Text Output Example {CodeOutputExample1} ### Create With Multiple Connected Nodes The following is an example of how to build a workflow with multiple connected nodes. {PythonSampleNodes} {JSSampleNodes} {NodeSampleNodes} {JavaSampleNodes} {PHPSampleNodes} {CurlSampleNodes} ### Suppress Output From Nodes It is possible to turn the outputs from given nodes in your workflow on and off with the `suppress_output` endpoint. This can be helpful when you want to hide outputs for expensive return values like image crops or embedding. By default, this endpoint will be set to false, meaning that we do not suppress any model's output. {PythonSuppressNodes} {JSSuppressNodes} {NodeSuppressNodes} {JavaSuppressNodes} {PHPSuppressNodes} {CurlSuppressNodes} ### Create Using YAML File Note that when creating a workflow using the Python SDK, you need to provide its YAML specification. {CodeCW} Output --- ## Workflows Examples # Workflows Examples **Learn how to build some common workflows on our platform** Clarifai Workflows provide you with the ability to assemble multiple models and model operators together. Learn some of the most popular workflow patterns. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Auto-Annotation via API # Auto-Annotation via API **Use AI to help you build AI. Auto annotation uses your model predictions to label your training data** This tutorial demonstrates how auto-annotation workflows can be configured in the Clarifai API. With auto-annotation, you can use model predictions to label your inputs. Auto-annotation can help you prepare training data or assign other useful labels and metadata to your inputs. Since models are doing most of the work of annotating your data, this enables you to speed-up and scale-up your annotation process while ensuring quality standards, typically reducing human effort of labeling data by orders of magnitude. And since this is built into our APIs, it seamlessly integrates with all the search, training, and prediction functionality of the Clarifai platform. When a concept is predicted by a model, it is predicted with a confidence score of between 0 and 1. In this walkthrough, we will leverage that score in our workflow so that when your model predictions are confident \(close to 1\), you can have your data automatically labeled with that concept. When your predictions are less-than-confident, you can have your input sent to a human reviewer. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: :::tip - [Click here](https://docs.clarifai.com/api-guide/annotate/tasks#auto-annotation-task) to learn how to create an auto-annotation task via the API and automatically label the inputs in your dataset. - [Click here](https://docs.clarifai.com/portal-guide/annotate/auto-annotation/) to learn how to carry out auto-annotation via the UI. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonCreateConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_concepts.py"; import PythonLinkConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/link_concepts.py"; import PythonCreateConceptMapperModel from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_concept_mapper_model.py"; import PythonCreateGreaterThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_greater_than_concept_thresholder.py"; import PythonCreateLessThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_less_than_concept-thresholder.py"; import PythonCreateWriteSuccess from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_write_success_asme_annotation.py"; import PythonCreateWritePending from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_write_pending_asme_annotation.py"; import PythonCreateWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/create_the_workflow.py"; import PythonMakeWorkflowDefault from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/make_new_workflow_apps_default.py"; import PythonAddImage from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/add_an_image.py"; import PythonListAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/list_annotations.py"; import NodeCreateConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_concepts.js"; import NodeLinkConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/link_concepts.js"; import NodeCreateConceptMapperModel from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_concept_mapper_model.js"; import NodeCreateGreaterThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_greater_than_concept_thresholder.js"; import NodeCreateLessThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_less_than_concept-thresholder.js"; import NodeCreateWriteSuccess from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_write_success_asme_annotation.js"; import NodeCreateWritePending from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_write_pending_asme_annotation.js"; import NodeCreateWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/create_the_workflow.js"; import NodeMakeWorkflowDefault from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/make_new_workflow_apps_default.js"; import NodeAddImage from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/add_an_image.js"; import NodeListAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/node/list_annotations.js"; import JavaCreateConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_concepts.java"; import JavaLinkConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/link_concepts.java"; import JavaCreateConceptMapperModel from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_concept_mapper_model.java"; import JavaCreateGreaterThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_greater_than_concept_thresholder.java"; import JavaCreateLessThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_less_than_concept-thresholder.java"; import JavaCreateWriteSuccess from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_write_success_asme_annotation.java"; import JavaCreateWritePending from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_write_pending_asme_annotation.java"; import JavaCreateWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/create_the_workflow.java"; import JavaMakeWorkflowDefault from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/make_new_workflow_apps_default.java"; import JavaAddImage from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/add_an_image.java"; import JavaListAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/java/list_annotations.java"; import CurlCreateConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_concepts.sh"; import CurlLinkConcepts from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/link_concepts.sh"; import CurlCreateConceptMapperModel from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_concept_mapper_model.sh"; import CurlCreateGreaterThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_greater_than_concept_thresholder.sh"; import CurlCreateLessThan from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_less_than_concept-thresholder.sh"; import CurlCreateWriteSuccess from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_write_success_asme_annotation.sh"; import CurlCreateWritePending from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_write_pending_asme_annotation.sh"; import CurlCreateWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/create_the_workflow.sh"; import CurlMakeWorkflowDefault from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/make_new_workflow_apps_default.sh"; import CurlAddImage from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/add_an_image.sh"; import CurlListAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/curl/list_annotations.sh"; ## Create Concepts Let's start by creating the concepts we'll use in our model. We'll create the following concepts: `people`, `man` and `adult`. {PythonCreateConcepts} {NodeCreateConcepts} {JavaCreateConcepts} {CurlCreateConcepts} ## Link Concepts Link the newly created concepts with concepts in the clarifai/main General model. Run the code below three times; once for each concept you created previously. The concept IDs of the clarifai/main General models are as follows: * `ai_l8TKp2h5` - the people concept; * `ai_dxSG2s86` - the man concept; * `ai_VPmHr5bm` - the adult concept. Your model's concept IDs are the ones you created in the previous step: `peopleID`, `manID`, and `adultID`. {PythonLinkConcepts} {NodeLinkConcepts} {JavaLinkConcepts} {CurlLinkConcepts} ## Create a Concept Mapper Model We're going to create a concept mapper model that translates the concepts from the General model to our new concepts. The model will map the concepts as synonyms. Hypernyms and hyponyms are supported as well. We'll be setting the `knowledge_graph_id` value to be empty. If you want to define a subset of relationships in your app to be related to each other, you can provide the `knowledge_graph_id` to each concept relation and then provide that `knowledge_graph_id` as input to this model as well, which will only follow relationships in that subset of your app's knowledge graph. {PythonCreateConceptMapperModel} {NodeCreateConceptMapperModel} {JavaCreateConceptMapperModel} {CurlCreateConceptMapperModel} ## Create a "Greater Than" Concept Thresholder Model This model will allow any predictions >= the concept values defined in the model to be outputted from the model. {PythonCreateGreaterThan} {NodeCreateGreaterThan} {JavaCreateGreaterThan} {CurlCreateGreaterThan} ## Create a "Less Than" Concept Thresholder Model This model will allow any predictions < the concept values defined in the model to be outputted from the model. {PythonCreateLessThan} {NodeCreateLessThan} {JavaCreateLessThan} {CurlCreateLessThan} ## Create a "Write Success as Me" Annotation Writer Model Any incoming Data object full of concepts, regions, etc. will be written by this model to the database as an annotation with ANNOTATION\_SUCCESS status as if the app owner did the work themself. {PythonCreateWriteSuccess} {NodeCreateWriteSuccess} {JavaCreateWriteSuccess} {CurlCreateWriteSuccess} ## Create a "Write Pending as Me" Annotation Writer Model Any incoming Data object full of concepts, regions, etc. will be written by this model to the database as an annotation with ANNOTATION\_PENDING status as if the app owner did the work themself, but needs further review. So, it is marked as pending. {PythonCreateWritePending} {NodeCreateWritePending} {JavaCreateWritePending} {CurlCreateWritePending} ## Create the Workflow We will now connect all the models together into a single workflow. Every input will be predicted by the General Embed model to generate embeddings. The output of the embed model \(embeddings\) will be sent to the general concept to predict concept and cluster model. Then, the concept model's output \(a list of concepts with prediction values\) will be sent to the concept mapper model, which maps Clarifai's concepts to the concepts within your app—`people`, `man` and `adult` in this case. Then, the mapped concepts will be sent to both concept threshold models—\(`GREATER THAN` and `LESS THAN`\). The `GREATER THAN` model will filter out the concepts that are lower than the corresponding value you defined in the model and send the remaining concept list to `write success as me` model, which labels the input with these concepts \(your app concepts only\) as you with `success` status. You can train or search on these concepts immediately. The `LESS THAN` model will filter out concepts that are higher than the corresponding value you defined in the model and send the remaining concept list to `write pending as me` model, which labels the input with these concepts \(your app concepts only\) as you with `pending` status. The model IDs and model version IDs from the public `clarifai/main` application are fixed to the latest version at the time of this writing \(use the `GET /models` endpoint to get an up to date list of the available models\), so they are already hard-coded in the code examples below. It's possible to use other public models or model version IDs. {PythonCreateWorkflow} {NodeCreateWorkflow} {JavaCreateWorkflow} {CurlCreateWorkflow} ## Make the New Workflow Your App's Default Make this the default workflow in the app. So, it will run every time we add an input and execute the auto annotation process. If the workflow is not the default workflow of your app, you can still use `PostWorkflowResults` on new inputs to check that you configured the workflow graph and your models properly, but the data will not be written to the DB. This is recommended before making it your default workflow and adding inputs to your app. {PythonMakeWorkflowDefault} {NodeMakeWorkflowDefault} {JavaMakeWorkflowDefault} {CurlMakeWorkflowDefault} ## Add an Image Adding an image will trigger the default workflow. {PythonAddImage} {NodeAddImage} {JavaAddImage} {CurlAddImage} ## List Annotations You can now list annotations with your user ID and see the annotations created by your workflow. {PythonListAnnotations} {NodeListAnnotations} {JavaListAnnotations} {CurlListAnnotations} --- ## Auto-Annotation via UI # Auto-Annotation via UI **Use AI to help you build AI. Auto annotation uses your model predictions to label your training data** This tutorial demonstrates how auto-annotation workflows can be configured within the Clarifai platform. With auto-annotation, you can use model predictions to label your inputs. Auto-annotation can help you to prepare training data, or assign other useful labels and metadata to your inputs. When a concept is predicted by a model, it is predicted with a confidence score between 0 and 1. For example, when your model predictions are confident \(close to 1\), you can have your data automatically labeled with that concept. When your predictions are less-than-confident, you can have your input sent to a human being for review. This enables you to speed-up and scale-up your annotation process while ensuring quality standards. :::tip Workflows In the Clarifai platform, the outputs from one model can be used as inputs to another model. This forms a workflow. Different models accept and produce different types of inputs and outputs. ::: ![](/img/community/auto-annotation/auto_annotation.jpg) :::warning objective In this tutorial, we'll create a workflow that detects bounding box regions in images of cats and dogs. Once a certain threshold is met, the workflow will automatically generate annotations for these detected regions. If the threshold is not met, the annotation will be marked as pending review. ::: Here's what our final workflow will look like: ![](/img/community/auto-annotation/auto-annotation-9.png) ## Prerequisites - Create an [application](https://docs.clarifai.com/clarifai-basics/applications/create-an-application), add images of cats and dogs to a [dataset](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete) in the app, and add [bounding box labels](https://docs.clarifai.com/portal-guide/annotate/label-types#detection) of `cat` and `dog` to the images, respectively. You could use the following images: ```text https://samples.clarifai.com/dog1.jpeg https://samples.clarifai.com/dog2.jpeg https://samples.clarifai.com/dog3.jpeg https://samples.clarifai.com/cat1.jpeg https://samples.clarifai.com/cat2.jpeg https://samples.clarifai.com/cat3.jpeg ``` - Create a [visual detector](https://docs.clarifai.com/portal-guide/model/model-types/visual-detector) model and train it with the `cat` and `dog` concepts. - Create a [labeling task](https://docs.clarifai.com/portal-guide/annotate/create-a-task/). Remember to choose `Detection` as the modeling objective. Then, go to the **Tasks** listing page and copy the ID of the task. ## Step 1: Set up Visual Detector :::caution visual detector **Input**: `image` **Output**: `regions[…].data.concepts`, `regions[…].region_info.bounding_box` The visual detector model will detect bounding box regions in images and then classify objects within the boxes. ::: **a.** Go to the [workflow builder](https://docs.clarifai.com/portal-guide/workflows/working-with-workflows/) page. **b.** Search for the `visual-detector` node in the left sidebar of the page. **c.** Drag it onto the empty workspace and connect it to the `IN` element. **d.** Use the pop-up that appears in the right sidebar to search for the detection model you created and add it to the node. After selecting the model, we'll use the default settings for the other output configuration options for the purpose of this illustration. :::note You can use the tools on the left side of the workspace pane to manage the workflow creation process. These tools allow you to zoom in and out, fit the view, arrange the workflow, reset the workspace, and more. ::: ![](/img/community/auto-annotation/auto-annotation-1.png) ## Step 2: Set up Region Thresholder :::caution Region Thresholder **Input**: `regions[…].data.concepts` **Output**: `regions[…].data.concepts` This operator allows you to filter regions based on the concepts they contain using specific thresholds for each concept and an overall operator (>, >=, =, \ " (`GREATER_THAN`) threshold type. Thus, only the `cat` and `dog` concepts outputted by the visual detector model above a certain threshold will be processed further downstream. ::: **a.** Search for the `region-thresholder` node in the left sidebar and drag it onto the workspace. Connect it to the `visual-detector` node. ![](/img/community/auto-annotation/auto-annotation-2.png) **b.** Click the **SELECT CONCEPTS** button in the right sidebar. In the window that appears, select the relevant concepts already existing in your application. For this example, let's select the `cat` and `dog` concepts, and use the slider to set their threshold values to 0.95 each. This threshold will determine which annotations are accepted and which are set aside for follow-up review. ![](/img/community/auto-annotation/auto-annotation-3.png) Click the **OK** button to save the changes. You'll see the selected concepts highlighted in the right sidebar, along with their threshold values. ![](/img/community/auto-annotation/auto-annotation-4.png) **c.** Select the `concept_threshold_type` as `GREATER_THAN`. Keep the other configuration options set to their default values. ## Step 3: Set up Annotation Writer :::caution Annotation Writer **Input**: `Any` **Output**: `Any` Annotation writer allows you to write the input data to the database in the form of an annotation with a specified status as if a specific user created the annotation. ::: **a.** Search for the `annotation-writer` node in the left sidebar and drag it onto the workspace. Connect it to the `region-thresholder` node. **b.** In the right sidebar, set up the following output configurations: - Set the `annotation_status` as `ANNOTATION_SUCCESS`. This will write the annotations with the [`ANNOTATION_SUCCESS`](https://docs.clarifai.com/api-guide/advanced-topics/status-codes#annotation-related-codes-24xxx) status. - Set the `annotation_user_id`. This is the `user_id`for which to write the annotations on their behalf as if they manually did the work themselves. You could also choose your own `user_id`. - In this example, let's leave the `annotation_info` field empty. - Set the `task_id`. This is the ID the task annotation work belongs to. You can retrieve it from the **Tasks** listing page — as earlier mentioned. ![](/img/community/auto-annotation/auto-annotation-5.png) ## Step 4: Set up Low Confidence Score Flow In the flow described above, the visual detector model identifies bounding box regions for the provided inputs. These regions are then sent to the region thresholder node. The `GREATER_THAN` operator filters out regions that do not meet the threshold and forwards the remaining regions to the annotation writer model. Bounding boxes that pass this filter are written to the input with an `ANNOTATION_SUCCESS` status. To capture predictions with low prediction scores, we'll set up a separate region thresholder node that uses the `LESS_THAN` operator. This operator will filter out regions with prediction scores below the threshold and write these regions to the input with an `ANNOTATION_AWAITING_REVIEW` status. **a.** Search for the `region-thresholder` node in the left sidebar and drag it onto the workspace. Connect it to the `visual-detector` node. Also, click the **SELECT CONCEPTS** button in the right sidebar, and set the threshold values for the `cat` and `dog`concepts to 0.95 each. Lastly, set the `concept_threshold_type` as `LESS_THAN_OR_EQUAL`. ![](/img/community/auto-annotation/auto-annotation-6.png) :::caution LESS_THAN_OR_EQUAL The `LESS_THAN_OR_EQUAL` (\<=) operator ensures that all values that are either less than or exactly equal to the specified threshold are included in the result. In this case, if we set a threshold of 0.95 using the operator, any value that is 0.95 or lower will meet the condition. ::: **b.** Search for another `region-thresholder` node in the left sidebar and drag it onto the workspace. Connect it to the previous `region-thresholder` node. Also, click the **SELECT CONCEPTS** button in the right sidebar, and set the threshold values for the `cat` and `dog`concepts to 0.50 each. Lastly, set the `concept_threshold_type` as `GREATER_THAN`. This ensures that only values that are higher than the 0.50 threshold are included in the result. ![](/img/community/auto-annotation/auto-annotation-7.png) **c.** Search for an `annotation-writer` node in the left sidebar and drag it onto the workspace. Connect it to the previous `region-thresholder` node. In the right sidebar, set up `annotation_status` as `ANNOTATION_AWAITING_REVIEW`. This will write the annotations with a pending review status. Also, set the values for `annotation_user_id` and `task_id` — as earlier described. ![](/img/community/auto-annotation/auto-annotation-8.png) ## Step 5: Save Workflow Lastly, click the **Save Workflow** button at the upper-right corner of the page. ![](/img/community/auto-annotation/auto-annotation-9.png) Your workflow is now ready for use! ## Step 6: Use Workflow Go to your app's settings page and [change the Base Workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/#how-to-change-a-base-workflow) to the workflow you just created. This ensures that the workflow runs every time you add an input to your app. For example, adding a new image of a dog will trigger the auto-annotation process Then, a bounding box label will be added to the image and it will be assigned with the `dog` concept. You can view the annotation in the Input-Viewer page: ![](/img/community/auto-annotation/auto-annotation-10.png) That's it! --- ## Custom KNN Face Classifier # Custom KNN Face Classifier **Use facial recognition to identify individual people** Let's say you want to build a face recognition system that is able to differentiate between persons of whom you only have a few samples \(per person\). Machine learning models generally require a large inputs dataset to be able to classify the inputs well. When a large dataset is the luxury you do not have, we recommend using our **KNN Classifier Model**, which uses k-nearest neighbor search and plurality voting amongst the nearest neighbors to classify new instances. It's recommended when you only have a small dataset like one input per concept. In this walkthrough, you'll learn how to create a KNN classifier that's going to work based off the Clarifai's base Face model. The whole process below is going to be done programmatically, using the Clarifai's powerful API. :::tip Tip Each of the steps below can also be done manually on the Clarifai Platform. ::: :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonAddImages from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_add_images.py"; import PythonWaitUploadMap from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_wait_upload_map_ids_urls.py"; import PythonListAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_list_the_annotations.py"; import PythonPostNewAnnotations from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_post_new_annotations.py"; import PythonCreateKnnModel from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_create_knn_model.py"; import PythonCreateWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_create_workflow.py"; import PythonPredict from "!!raw-loader!../../../../code_snippets/api-guide/workflows/common_workflows/py/knn_predict.py"; ## Create a New Application [Create a new application](https://docs.clarifai.com/create/applications/create) with **Face** specified as the Base Workflow. ## Add Images Add images that contain the faces you want to use as a training set. Since the application's base model is Face, after adding an image, faces are automatically located and are available to be annotated. {PythonAddImages} ## Wait for Upload & Map IDs to URLs Now we'll wait for all the images to finish uploading, and then create a dictionary mapping from an input ID to the URL. This will help us to annotate the proper image in the next step. {PythonWaitUploadMap} ## List the Annotations Let's now print all the regions that the Face base model detected on our images. The code below prints the annotations together with the model version ID and region ID. These two IDs will be needed in the next step to annotate using our custom concepts. We'll also need the base Face model ID, which is the one where `model_version_id` equals to `embedding_model_version_id`. {PythonListAnnotations} ## Post New Annotations Let's use the above information to add annotations, in the form of a concept, to the detected face regions. Input below the IDs from the previous call, and choose your concept ID and name that you want to annotate the region with \(you may want to use e.g. person's name\). {PythonPostNewAnnotations} ## Create a KNN Model Let's now create a KNN model using the concept IDs that were added above. The model type ID should be set to `knn-concept`. {PythonCreateKnnModel} ## Create a Workflow One last step before making predictions: let's create a workflow that's going to map from the base Face model to our custom KNN model. {PythonCreateWorkflow} ## Predict We're going to run a prediction on the workflow created above. {PythonPredict} --- ## Custom Prompter Model # Custom Prompter Model **Integrate a prompter model into an LLM workflow** A [prompter model](https://docs.clarifai.com/portal-guide/agent-system-operators/prompter) is a type of language model specifically designed to craft instructions that guide the output of large language models (LLMs). It helps in prompt engineering, focusing on optimizing the responses of LLMs to prompts. Let's demonstrate how you can create your own prompter model and connect it to an LLM in a [workflow](https://docs.clarifai.com/api-guide/workflows/). :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonCreatePromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/py/create_prompt_model.py"; import JSCreatePromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/js/create_prompt_model.html"; import NodeCreatePromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/node/create_prompt_model.js"; import JavaCreatePromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/java/create_prompt_model.java"; import CurlCreatePromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/curl/create_prompt_model.sh"; import PHPCreatePromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/php/create_prompt_model.php"; import PythonTrainPromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/py/train_prompt_model.py"; import JSTrainPromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/js/train_prompt_model.html"; import NodeTrainPromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/node/train_prompt_model.js"; import JavaTrainPromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/java/train_prompt_model.java"; import CurlTrainPromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/curl/train_prompt_model.sh"; import PHPTrainPromptModel from "!!raw-loader!../../../../code_snippets/api-guide/model/php/train_prompt_model.php"; import PythonPromptWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/model/py/prompter_workflow.py"; import JSPromptWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/model/js/prompter_workflow.html"; import NodePromptWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/model/node/prompter_workflow.js"; import JavaPromptWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/model/java/prompter_workflow.java"; import CurlPromptWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/model/curl/prompter_workflow.sh"; import PHPPromptWorkflow from "!!raw-loader!../../../../code_snippets/api-guide/model/php/prompter_workflow.php"; import PythonPromptWorkflowPredict from "!!raw-loader!../../../../code_snippets/api-guide/model/py/prompter_workflow_predict.py"; import JSPromptWorkflowPredict from "!!raw-loader!../../../../code_snippets/api-guide/model/js/prompter_workflow_predict.html"; import NodePromptWorkflowPredict from "!!raw-loader!../../../../code_snippets/api-guide/model/node/prompter_workflow_predict.js"; import JavaPromptWorkflowPredict from "!!raw-loader!../../../../code_snippets/api-guide/model/java/prompter_workflow_predict.java"; import CurlPromptWorkflowPredict from "!!raw-loader!../../../../code_snippets/api-guide/model/curl/prompter_workflow_predict.sh"; import PHPPromptWorkflowPredict from "!!raw-loader!../../../../code_snippets/api-guide/model/php/prompter_workflow_predict.php"; import ExampleOutput1 from "!!raw-loader!../../../../code_snippets/api-guide/model/code_output_examples/prompter_model_workflow.js"; ## Create a Prompter Model {PythonCreatePromptModel} {JSCreatePromptModel} {NodeCreatePromptModel} {JavaCreatePromptModel} {PHPCreatePromptModel} {CurlCreatePromptModel} Text Output Example {ExampleOutput1} ## Train a Prompter Model When training a prompter model, you need to provide a prompt template, which serves as a pre-configured piece of text for instructing an LLM. Note that your prompt template should include at least one instance of the placeholder `{data.text.raw}`. When you input your text data at inference time, all occurrences of `{data.text.raw}` within the template will be replaced with the provided text. {PythonTrainPromptModel} {JSTrainPromptModel} {NodeTrainPromptModel} --> {JavaTrainPromptModel} {PHPTrainPromptModel} --> {CurlTrainPromptModel} ## Add to a Workflow After training your prompter model, you can now put it to work by integrating it into an LLM workflow and using it to accomplish various tasks. Below is an example of how to connect a prompter model to an LLM like [GPT-4](https://clarifai.com/openai/chat-completion/models/GPT-4) for text-to-text tasks. {PythonPromptWorkflow} {JSPromptWorkflow} {NodePromptWorkflow} {JavaPromptWorkflow} {PHPPromptWorkflow} {CurlPromptWorkflow} ## Workflow Predict After creating the workflow, let's now use it to perform a text sentiment prediction task. {PythonPromptWorkflowPredict} {JSPromptWorkflowPredict} {NodePromptWorkflowPredict} {JavaPromptWorkflowPredict} {PHPPromptWorkflowPredict} {CurlPromptWorkflowPredict} Text Output Example {ExampleOutput1} As you can see on the output above, the response contains the predictions of each model in the workflow. The prompt text starts with the earlier provided template text, and the `{data.text.raw}` placeholder is substituted with the provided input text. That is what is used as a prompt for the GPT-4 model. And the model correctly predicts the sentiment of the provided input text. --- ## YAML-based Examples # YAML-based Examples **Simple examples of workflows defined in YAML** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeOutputASRYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/asr_yaml.txt"; import CodeOutputDemoYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/demo_yaml.txt"; import CodeOutputFaceSYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/face_search_yaml.txt"; import CodeOutputFaceSentiYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/face_senti_yaml.txt"; import CodeOutputGenYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/general_yaml.txt"; import CodeOutputLangAYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/lang_aware_yaml.txt"; import CodeOutputPrompterYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/prompter_llm_yaml.txt"; import CodeOutputRAGYaml from "!!raw-loader!../../../../code_snippets/python-sdk/workflows/outputs/rag_yaml.txt"; ## Assorted Examples | Node Name | Input & Output | Description | Example Usage | | --- | --- | --- | :----: | | audio-to-text | Audio -> Text | Classify audio signal into string of text. | | | barcode-operator | Image -> Text | Operator that detects and recognizes barcodes from the image. It assigns regions with barcode text for each detected barcode. Supports EAN/UPC, Code 128, Code 39, Interleaved 2 of 5 and QR Code. | | | Centroid Tracker | Frames -> Track ID | Centroid trackers rely on the Euclidean distance between centroids of regions in different video frames to assign the same track ID to detections of the same object. | | | Clusterer | Embeddings -> Clusters | Cluster semantically similar images and video frames together in embedding space. This is the basis for good visual search within your app at scale or for grouping your data together without the need for annotated concepts | | | embeddings-classifier | Embeddings -> Concepts | Classify images or texts based on the embedding model that has indexed them in your app. Transfer learning leverages feature representations from a pre-trained model based on massive amounts of data, so you don't have to train a new model from scratch and can learn new things very quickly with minimal training data | | | image-color-recognizer | Image -> Colors | Recognize standard color formats and the proportion each color that covers an image | | | image-to-text | Image -> Text | Takes in cropped regions with text in them and returns the text it sees. | | | kalman-filter-tracker | Frames -> Track ID | Kalman Filter trackers rely on the Kalman Filter algorithm to estimate the next position of an object based on its position and velocity in previous frames. Then detections are matched to predictions by using the Hungarian algorithm | | | kalman-reid-tracker | Frames -> Track ID | Kalman reid tracker is a kalman filter tracker that expects the Embedding proto field to be populated for detections, and reassigns track IDs based off of embedding distance | | | neural-lite-tracker | Frames -> Track ID | Neural Lite Tracker uses light-weight trainable graphical models to infer states of tracks and perform associations using hybrid similairty of lou and centroid distance | | | neural-tracker | Frames -> Track ID | Neural Tracker uses neural probabilistic models to perform filtering and association | | | optical-character-recognizer | Image -> Text | Detect bounding box regions in images or video frames where text is present and then output the text read with the score | | | tesseract-operator | Image -> Text | Operator for Optical Character Recognition using the Tesseract libraries | | | text-classifier | Text -> Concepts | Classify text into a set of concepts | | | text-embedder | Text -> Embeddings | Embed text into a vector representing a high level understanding from our Al models. These embeddings enable similarity search and training on top of them | | | text-token-classifier | Text -> Concepts | Classify tokens from a set of entity classes | | | visual-classifier | Image -> Concepts | Classify images and videos frames into set of concepts | | | visual-detector | Image -> Bounding Box | Detect bounding box regions in images or video frames where things and then classify objects, descriptive words or topics within the boxes | | | visual-embedder | Image -> Embeddings | Embed images and videos frames into a vector representing a high level understanding from our Al models. These embeddings enable visual search and training on top of them | | | visual-segmenter | Image -> Concepts | Segment a per-pixel mask in images where things are and then classify objects, descriptive words or topics within the masks | | | concept-thresholder | Concepts -> Concpets | Threshold input concepts according to both a threshold and an operator (>, >=, =, \ " threshold type is set for the model, then if the input concept value is greater than the threshold for that concept, the input concept will be output from this model, otherwise it will not be output by the model | | | random-sample | Any -> Any | Randomly sample allowing the input to pass to the output. This is done with the conditional keep_fraction > rand() where keep_fraction is the fraction to allow through on average | | | region-thresholder | Concepts -> Concepts | Threshold regions based on the concepts that they contain using a threshold per concept and an overall operator (>, >=, =, \ " threshold type is set for the model, then if the input regions[...].data.concepts.value is greater than the threshold for that concept, the input concept will be output from this model, otherwise it will not be output by the model. If the entire list of concepts at regions[...].data.concepts is filtered out then the overall region will also be removed | | | byte-tracker | Frame -> Track ID | Uses byte tracking algorithm for tracking objects | | | concept-synonym-mapper | Concept -> Concept | Map the input concepts to output concepts by following synonym concept relations in the knowledge graph of your app | | | image-align | Image -> Image | Aligns images using keypoints | | | image-crop | Image -> Image | Crop the input image according to each input region that is present in the input. When used in a workflow this model can look back along the graph of the workflow to find the input image if the preceding model does not output an image itself so that you can do image -> detector -> cropper type of workflow easily | | | image-tiling-operator | Image -> Image | Operator for tiling images into a fixed number of equal sized images | | | image-to-image | Image -> Image | Given an image, apply a transformation on the input and return the post-processed image as output | | | input-filter | Any -> Any | If the input going through this model does not match those we are filtering for, it will not be passed on in the workflow branch | | | input-searcher | Concepts,Images,Text -> Hits | Triggers a visual search in another app based on the model configs if concept(s) are found in images and returns the matched search hits as regions. | | | keyword-filter-operator | Text -> Concepts | This operator is initialized with a set of words, and then determines which are found in the input text. | | | language-id-operator | Text -> Concepts | Operator for language identification using the langdetect library | | | multimodal-embedder | Any -> Embeddings | Embed text or image into a vector representing a high level understanding from our Al models, e.g. CLIP. These embeddings enable similarity search and training on top of them. | | | multimodal-to-text | Any -> Text | Generate text from either text or images or both as input, allowing it to understand and respond to questions about those images | | | prompter | Text -> Text | Prompt template where inputted text will be inserted into placeholders marked with (data.text.raw). | | | rag-prompter | Text -> Text | A prompt template where we will perform a semantic search in the app with the incoming text. The inputted text will be inserted into placeholders marked with '(data.text.raw)' and search results will be inserted into placeholders with '`{data.hits}`', which will be new line separated | | | regex-based-classifier | Text -> Concepts | Classifies text using regex. If the regex matches, the text is classified as the provided concepts. | | | text-to-audio | Text -> Audio | Given text input, this model produces an audio file containing the spoken version of the input | | | text-to-image | Text -> Image | Takes in a prompt and generates an image | | | tiling-region-aggregator-operator | Frames -> Concepts,Bounding Box | Operator to be used as a follow up to the image-tiling-operator and visual detector. This operator will transform the detections on each of tiles back to the original image and perform non-maximum suppression. Only the top class prediction for each box is considered | | | visual-keypointer | Image -> Keypoints | This model detects keypoints in images or video frames. | | | isolation-operator | Concepts,BoundingBox -> Concepts,BoundingBox | Operator that computes distance between detections and assigns isolation label | | | object-counter | Concepts -> Metadata | count number of regions that match this model's active concepts frame by frame | | | text-aggregation-operator | Text -> Text | Operator that combines text detections into text body for the whole image. Detections are sorted from left to right first and then top to bottom, using the top-left corner of the bounding box as reference | | | tokens-to-entity-operator | Text,Concepts -> Text,Concepts | Operator that combines text tokens into entities, e.g. New' + 'York' -> New York | | | annotation-writer | Any -> Any | Write the input data to the database in the form of an annotation with a specified status as if a specific user created the annotation | | | aws-lambda | Any -> Any | This model sends data to an AWS lambda function so you can implement any arbitrary logic to be handled within a model predict or workflow. The request our API sends is a PostModelOutputsRequest in the 'request' field and the response we expect is a MultiOutputResponse response in the 'response' field | | | email | Any -> Any | Email alert model will send an email if there are any data fields input to this model | | | results-push | Any -> Any | This model pushes clarifai prediction results in an external format | | | sms | Any -> Any | SMS alert model will send a SMS if there are any data fields input to this model | | | status-push | Any -> Any | This model pushes processing status of a batch of inputs ingested through vendor/inputs endpoint in one request | | ## ASR Sentiment Automatic Speech Recognition (ASR) sentiment analysis is the process of detecting the emotional tone or sentiment in spoken language by first transcribing speech using an ASR model and then analyzing the resulting text. {CodeOutputASRYaml} ## Demographics This is a multi-model workflow designed to detect faces, crop them, and recognize key demographic characteristics. It visually classifies attributes such as age, gender, and cultural appearance. {CodeOutputDemoYaml} ## Face Search A workflow that combines face detection, recognition, and embedding to generate facial landmarks and enable visual search based on the embeddings of detected faces. {CodeOutputFaceSYaml} ## Face Sentiment A multi-model workflow that combines face detection with sentiment classification to recognize seven emotional expressions: anger, disgust, fear, neutral, happiness, sadness, and contempt. {CodeOutputFaceSentiYaml} ## General A general-purpose image detection workflow that identifies a wide range of common objects and enables visual search using embeddings generated from the detected regions. {CodeOutputGenYaml} ## Language Aware OCR A workflow that performs Optical Character Recognition (OCR) across multiple languages, automatically adapting to the language present in the input text. {CodeOutputLangAYaml} ## Prompter LLM A workflow that utilizes a prompt template to interact with a Large Language Model (LLM), enabling dynamic and context-aware text generation based on input data. {CodeOutputPrompterYaml} ## RAG Prompter LLM This workflow combines a Large Language Model (LLM) with a Retrieval-Augmented Generation (RAG) prompter template to generate responses informed by relevant external knowledge. {CodeOutputRAGYaml} :::tip Click [here](https://github.com/Clarifai/examples/tree/main/workflows/configs) to view more YAML-based workflows examples. ::: --- ## Workflow Inferences # Workflow Inferences **Make predictions with your workflows** You can make predictions using all the models in a workflow with a single API call. The cost of making predictions with a workflow is the same as calling individual models. The request body follows the same structure as a standard prediction call. In the [response](https://docs.clarifai.com/resources/api-overview/api-outputs), a `results` object will contain the outputs from each model in the workflow, presented in the order they were defined. :::note The maximum number of inputs that can be processed at once with any given workflow is 32. ::: ## Predict via the UI You can use a workflow to make predictions by following these steps: 1. Go to the workflow's overview page; 2. Select a workflow version and upload an image or video; 3. View the predictions returned by each model in the workflow. For example, let's use the custom workflow we [created previously](create.md#create-via-the-ui) to extract text from [this image](https://samples.clarifai.com/featured-models/ocr-woman-holding-sold-sign.jpg), and then translate the extracted text into Spanish. To do so, go to the workflow's overview page, select the version you want to use, and click the blue **"+"** button. Next, select the **Try your own image or video** option on the modal that appears. The small window that pops up allows you to upload the image. ![alt_text](/img/community_2/workflow_predict_try_your_own_image.png) After the image has been uploaded and processed, the output will contain the predictions each model in the workflow returns. You can see in the screenshot below that the text was successfully extracted from the image and then translated into Spanish. ![alt_text](/img/community_2/workflow_prediction_output.png) ## Predict via the API import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeTC from "!!raw-loader!../../../code_snippets/python-sdk/workflows/text_classifier.py"; import CodeTCTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/textClassifier.ts"; import CodeSumm from "!!raw-loader!../../../code_snippets/python-sdk/workflows/summary.py"; import CodeSummTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/textToTextSummarization.ts"; import CodeTGen from "!!raw-loader!../../../code_snippets/python-sdk/workflows/text_generation.py"; import CodeTGenTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/textToTextGeneration.ts"; import CodeVC from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visual_class.py"; import CodeVCTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visualClassifier.ts"; import CodeVS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visual_seg.py"; import CodeVSTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visualSegmenter.ts"; import CodeIT from "!!raw-loader!../../../code_snippets/python-sdk/workflows/image_to_text.py"; import CodeITTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/imageToText.ts"; import CodeTI from "!!raw-loader!../../../code_snippets/python-sdk/workflows/text_to_image.py"; import CodeTITS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/textToImage.ts"; import CodeII from "!!raw-loader!../../../code_snippets/python-sdk/workflows/image_to_image.py"; import CodeIITS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/imageToImage.ts"; import CodeTA from "!!raw-loader!../../../code_snippets/python-sdk/workflows/text_to_audio.py"; import CodeTATS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/textToAudio.ts"; import CodeAT from "!!raw-loader!../../../code_snippets/python-sdk/workflows/audio_to_text.py"; import CodeATTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/audioToText.ts"; import CodeVDF from "!!raw-loader!../../../code_snippets/python-sdk/workflows/vd_face.py"; import CodeVDFTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visualDetectorFace.ts"; import CodeVDO from "!!raw-loader!../../../code_snippets/python-sdk/workflows/vd_object.py"; import CodeVDOTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visualDetectorObject.ts"; import CodeTE from "!!raw-loader!../../../code_snippets/python-sdk/workflows/te.py"; import CodeTETS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/textToEmbedding.ts"; import CodeMM from "!!raw-loader!../../../code_snippets/python-sdk/workflows/multimodal.py"; import CodeMMTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/multiModal.ts"; import CodeVD from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visual_det.py"; import CodeVDTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/visualDetector.ts"; import CodeBP from "!!raw-loader!../../../code_snippets/python-sdk/workflows/batch_predict.py"; import CodeBPTS from "!!raw-loader!../../../code_snippets/python-sdk/workflows/batchPredict.ts"; import CodeOutputTC from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/text_classifier.txt"; import CodeOutputSumm from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/summary.txt"; import CodeOutputTGen from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/text_generation.txt"; import CodeOutputVC from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/visual_class.txt"; import CodeOutputIT from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/image_to_text.txt"; import CodeOutputAT from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/audio_to_text.txt"; import CodeOutputVDF from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/vd_face.txt"; import CodeOutputVDO from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/vd_object.txt"; import CodeOutputTE from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/te.txt"; import CodeOutputMM from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/multimodal.txt"; import CodeOutputVD from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/visual_det.txt"; import CodeOutputBP from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/batch_predict.txt"; import PythonWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/py/workflow_predict_images.py"; import JSWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/js/workflow_predict_images.html"; import NodeWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/node/workflow_predict_images.js"; import JavaWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/java/workflow_predict_images.java"; import CurlWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/curl/workflow_predict_images.sh"; import PHPWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/php/workflow_predict_images.php"; import ExampleCodeWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/output_examples/example_workflow_predict_images.txt"; import ExampleJSONWorkflowPredictImage from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/output_examples/example_workflow_predict_images.js"; import PythonWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/py/workflow_predict_text.py"; import JSWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/js/workflow_predict_text.html"; import NodeWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/node/workflow_predict_text.js"; import JavaWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/java/workflow_predict_text.java"; import CurlWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/curl/workflow_predict_text.sh"; import PHPWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/php/workflow_predict_text.php"; import ExampleCodeWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/output_examples/example_workflow_predict_text.txt"; import ExampleJSONWorkflowPredictText from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/output_examples/example_workflow_predict_text.js"; import PythonWorkflowPredictVideo from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/py/workflow_predict_video.py"; import JSWorkflowPredictVideo from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/js/workflow_predict_video.html"; import NodeWorkflowPredictVideo from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/node/workflow_predict_video.js"; import JavaWorkflowPredictVideo from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/java/workflow_predict_video.java"; import CurlWorkflowPredictVideo from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/curl/workflow_predict_video.sh"; import PHPWorkflowPredictVideo from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/php/workflow_predict_video.php"; import PythonWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/py/workflow_predict_audio.py"; import JSWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/js/workflow_predict_audio.html"; import NodeWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/node/workflow_predict_audio.js"; import JavaWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/java/workflow_predict_audio.java"; import CurlWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/curl/workflow_predict_audio.sh"; import PHPWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/php/workflow_predict_audio.php"; import ExampleCodeWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/output_examples/example_workflow_predict_audio.txt"; import ExampleJSONWorkflowPredictAudio from "!!raw-loader!../../../code_snippets/api-guide/workflows/common_workflows/output_examples/example_workflow_predict_audio.js"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: :::tip If you want to make a predict call with an external workflow that is outside the scope of your app, you need to use a PAT while specifying the `app_id` and the `user_id` associated with the workflow you want to use. ::: ### Text Classification You can use a workflow to categorize and analyze text data. {CodeTC} Output {CodeOutputTC} {CodeTCTS} ### Text-to-Text - Summarization You can use a workflow to generate concise summaries by extracting key insights from lengthy or complex textual content. {CodeSumm} Output {CodeOutputSumm} {CodeSummTS} ### Text-to-Text - Generation You can use a workflow to generate textual content based on a given prompt. This enables dynamic text generation for a wide range of use cases. {CodeTGen} Output {CodeOutputTGen} {CodeTGenTS} ### Text-to-Image You can use a workflow generate images from textual prompts. This enables creative visual content generation based on descriptive input. {CodeTI} Image Output {CodeTITS} ### Text-to-Audio You can use a workflow to convert written text into high-quality audio. Simply submit text as input to generate natural-sounding speech. {CodeTA} {CodeTATS} ### Text-to-Embeddings #### Example 1 You can use a workflow that combines embeddings and clustering to process, organize, and analyze diverse text data. This enables advanced language understanding and categorization. {CodeTE} Output {CodeOutputTE} {CodeTETS} #### Example 2 Let's illustrate how you would produce embeddings and clusters from text inputs using Clarifai's [Language-Understanding](https://clarifai.com/clarifai/main/workflows/Language-Understanding) text workflow. {PythonWorkflowPredictText} {JSWorkflowPredictText} {NodeWorkflowPredictText} {JavaWorkflowPredictText} {PHPWorkflowPredictText} {CurlWorkflowPredictText} Text Output Example {ExampleCodeWorkflowPredictText} Raw Output Example {ExampleJSONWorkflowPredictText} ### Visual Classification You can use a workflow for visual classification tasks. This enables accurate and efficient categorization of images based on learned visual patterns. #### Example 1 Let's illustrate how you would get predictions from a visual classifier workflow. {CodeVC} Output {CodeOutputVC} {CodeVCTS} #### Example 2 Let's illustrate how you would get predictions from image inputs using Clarifai's [Face-Sentiment](https://clarifai.com/clarifai/main/workflows/Face-Sentiment) workflow. The workflow combines these three models: - A visual detector model that detects bounding box regions in an image; - An image cropper model that extracts the specific region of interest from an image; - A visual classifier model that classifies an image into a set of concepts. Note that the `base64` output representation of the image in bytes is already in binary format. It is not encoded, so you do not need to decode it for further downstream tasks. {PythonWorkflowPredictImage} {JSWorkflowPredictImage} {NodeWorkflowPredictImage} {JavaWorkflowPredictImage} {PHPWorkflowPredictImage} {CurlWorkflowPredictImage} Text Output Example {ExampleCodeWorkflowPredictImage} Raw Output Example {ExampleJSONWorkflowPredictImage} ### Visual Segmentation You can use a workflow for visual segmentation tasks. This enables precise categorization of distinct regions within an image. {CodeVS} Image Output {CodeVSTS} ### Image-to-Text You can use a workflow to extract and interpret text from images. This enables OCR applications to efficiently recognize, process, and convert visual text into machine-readable format. {CodeIT} Output {CodeOutputIT} {CodeITTS} ### Image-to-Image You can use a workflow to enhance and transform images, including upscaling for higher resolution and improved visual quality. This delivers a superior visual experience for end users. {CodeII} Image Output {CodeIITS} ### Audio-to-Text #### Example 1 You can use a workflow to convert audio into text by transcribing the provided spoken content. {CodeAT} Output {CodeOutputAT} {CodeATTS} #### Example 2 Let's illustrate how you would get the sentiment of an audio input using Clarifai's [asr-sentiment](https://clarifai.com/clarifai/main/workflows/asr-sentiment) workflow. {PythonWorkflowPredictAudio} {JSWorkflowPredictAudio} {NodeWorkflowPredictAudio} {JavaWorkflowPredictAudio} {PHPWorkflowPredictAudio} {CurlWorkflowPredictAudio} Text Output Example {ExampleCodeWorkflowPredictAudio} Raw Output Example {ExampleJSONWorkflowPredictAudio} ### Visual Detection - Object Search You can use a workflow to detect common objects in images and generate embeddings to enable fast, accurate visual search based on each object’s unique features. :::note When you input a video into the Workflow Predict API, the response includes a list of predicted concepts for each frame of the video. By default, the video is processed at 1 frame per second (FPS), but this rate can be customized in the predict request. This means you’ll receive a set of concepts for every second (1000 milliseconds) of your video. To adjust the FPS rate, use the `sample_ms` parameter in your predict request. The `sample_ms` value specifies the time interval (in milliseconds) between frames selected for inference, determining how frequently frames are processed. The valid range for `sample_ms` is between 100 and 60,000 milliseconds. > **Note:** FPS is calculated as: FPS = 1000 / sample_ms For example, if `sample_ms` is set to 1000, the FPS rate will be 1 (the default value). The Workflow Predict API has size and duration limitations for [video inputs](https://docs.clarifai.com/create/inputs/upload/#videos): - Videos uploaded via URL can be up to 300MB in size or 10 minutes in length. - Videos sent as byte data are limited to 128MB in size. If your video exceeds these limits, you can split it into smaller segments for processing. Exceeding these limits may cause the process to time out and result in an error response. ::: {CodeVDO} Output {CodeOutputVDO} {CodeVDOTS} {PythonWorkflowPredictVideo} {JSWorkflowPredictVideo} {NodeWorkflowPredictVideo} {JavaWorkflowPredictVideo} {PHPWorkflowPredictVideo} {CurlWorkflowPredictVideo} ### Visual Detection - Face Search You can use a workflow that combines face detection, recognition, and embedding to accurately identify facial landmarks. This approach not only detects faces but also generates distinctive embeddings, enabling efficient visual searches based on the unique features of each face. {CodeVDF} Output {CodeOutputVDF} {CodeVDFTS} ### Visual Detection - Custom Workflow You can use a workflow to detect a wide range of common objects within a video. It processes each frame to identify and localize objects, generating regions of interest that outline where the detected objects appear. {CodeVD} Output {CodeOutputVD} {CodeVDTS} ### Multimodal Inputs You can provide data in multiple formats — such as text, images, or a combination of both — to a workflow and receive prediction results tailored to your input. {CodeMM} Output {CodeOutputMM} {CodeMMTS} ### Batch Prediction - Workflows You can use a workflow to process multiple inputs in a single request. You can submit a batch of data and receive comprehensive predictions for all inputs. :::info The batch size should not exceed 32. ::: {CodeBP} Output {CodeOutputBP} {CodeBPTS} --- ## Manage Workflows # Manage Workflows **Manage workflows** ## Manage via the UI ### Edit Workflow After creating your workflow, you can edit it at any time by navigating to its individual page and clicking the **Edit workflow** button in the upper-right section. This allows you to make changes easily whenever needed. ![alt_text](/img/community_2/input_nodes_edit_workflow_2.png) For example, to add a text-to-audio node to your workflow, first locate it in the left sidebar. Drag the node and connect it to the preceding text-to-text node. Next, use the search box on the right side of the page to select the specific model you want for the text-to-audio conversion. ![alt_text](/img/community_2/input_nodes_edit_workflow_1.png) Once you've made your changes, click the **Save as new version** button to save the updated workflow under a new version — without exiting the workflow editor. :::note * You can easily switch between different versions by selecting the respective version ID from the left sidebar in the workflow editor. * Clicking the **Update Workflow** button creates a new workflow version and exits the workflow editor, redirecting to the workflow's main page. ::: ![alt_text](/img/community_2/input_nodes_edit_workflow_3.png) :::tip * You can add a maximum of 20 nodes in a single workflow. * Ensure all connections between nodes are correctly set to avoid errors during execution. ::: ## Manage via the API import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PatchWorkflow from "!!raw-loader!../../../code_snippets/python-sdk/workflows/patch_workflow.py"; import CodeLW from "!!raw-loader!../../../code_snippets/python-sdk/workflows/list_workflows.py"; import CodeDW from "!!raw-loader!../../../code_snippets/python-sdk/workflows/delete_workflow.py"; import CodeExport from "!!raw-loader!../../../code_snippets/python-sdk/workflows/export.py"; import CodeOutputLW from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/list_workflows.txt"; import CodeOutputDW from "!!raw-loader!../../../code_snippets/python-sdk/workflows/outputs/delete_workflow.txt"; import PythonGetWorkflowsApp from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflows_in_app.py"; import PythonGetWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflow_specific_id.py"; import PythonPatchWorkflow from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/patch_workflow.py"; import PythonDeleteWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_workflow_id.py"; import PythonDeleteAllWorkflows from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_all_workflows.py"; import JSGetWorkflowsApp from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflows_in_app.html"; import JSGetWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflow_specific_id.html"; import JSPatchWorkflow from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/patch_workflow.html"; import JSDeleteWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_workflow_id.html"; import JSDeleteAllWorkflows from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_all_workflows.html"; import NodeGetWorkflowsApp from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflows_in_app.js"; import NodeGetWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflow_specific_id.js"; import NodePatchWorkflow from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/patch_workflow.js"; import NodeDeleteWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_workflow_id.js"; import NodeDeleteAllWorkflows from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_all_workflows.js"; import JavaGetWorkflowsApp from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflows_in_app.java"; import JavaGetWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflow_specific_id.java"; import JavaPatchWorkflow from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/patch_workflow.java"; import JavaDeleteWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_workflow_id.java"; import JavaDeleteAllWorkflows from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_all_workflows.java"; import PHPGetWorkflowsApp from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflows_in_app.php"; import PHPGetWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflow_specific_id.php"; import PHPPatchWorkflow from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/patch_workflow.php"; import PHPDeleteWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_workflow_id.php"; import PHPDeleteAllWorkflows from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_all_workflows.php"; import CurlGetWorkflowsApp from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflows_in_app.sh"; import CurlGetWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/get_workflow_specific_id.sh"; import CurlPatchWorkflow from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/patch_workflow.sh"; import CurlDeleteWorkflowID from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_workflow_id.sh"; import CurlDeleteAllWorkflows from "!!raw-loader!../../../code_snippets/api-guide/workflows/create_get_update_delete/delete_all_workflows.sh"; ### Get #### Get all Workflows in an App You can return all custom workflows in your app. :::tip If you want to get a list of the workflows not within the scope of your app, you need to use your PAT while specifying the `user_id` of their owner and the `app_id` of the application that you’re accessing. For example, to get Clarifai's workflows in the `main` app, you need to use your PAT while specifying Clarifai's `user_id` as "clarifai" and `app_id` as "main" in the request. ::: {CodeLW} {PythonGetWorkflowsApp} {JSGetWorkflowsApp} {NodeGetWorkflowsApp} {JavaGetWorkflowsApp} {PHPGetWorkflowsApp} {CurlGetWorkflowsApp} Output {CodeOutputLW} #### Get a Workflow by a Specific ID You can return information about a specific workflow. {PythonGetWorkflowID} {JSGetWorkflowID} {NodeGetWorkflowID} {JavaGetWorkflowID} {PHPGetWorkflowID} {CurlGetWorkflowID} ### Update After creating a workflow, you can perform patch operations on it by merging, removing, or overwriting data. By default, all actions support overwriting, with specific behaviors for lists of objects. - The `merge` action updates an existing `key:value` pair with `key:new_value` or appends to an existing list. For dictionaries, it merges objects that share a matching `id` field. - The `remove` action is only used to delete the workflow's cover image on the platform UI. - The `overwrite` action fully replaces an existing object with a new one. #### Patch Workflow's Models You can change a workflow; that is, change the models of which the workflow consists. {PythonPatchWorkflow} {JSPatchWorkflow} {NodePatchWorkflow} {JavaPatchWorkflow} {PHPPatchWorkflow} {CurlPatchWorkflow} #### Patch YAML Configuration, Description, etc Below is an example of performing patch operations on a workflow, where the YAML configuration is updated, and changes are made to its description, notes, and image URL. {PatchWorkflow} ### Delete :::caution Be certain that you want to delete a particular workflow as the operation cannot be undone. ::: #### Delete Workflow by ID You can delete a specific workflow. {CodeDW} {PythonDeleteWorkflowID} {JSDeleteWorkflowID} {NodeDeleteWorkflowID} {JavaDeleteWorkflowID} {PHPDeleteWorkflowID} {CurlDeleteWorkflowID} Output {CodeOutputDW} #### Delete all Workflows You can delete all custom workflows. :::tip Instead of `delete_all`, you can specify a list of workflow IDs to be deleted, using the `ids` field. ::: {PythonDeleteAllWorkflows} {JSDeleteAllWorkflows} {NodeDeleteAllWorkflows} {JavaDeleteAllWorkflows} {PHPDeleteAllWorkflows} {CurlDeleteAllWorkflows} ### Export Workflow You can easily export your entire workflow as a YAML file. This local copy enables convenient editing and offers the flexibility to create, reuse, or manage workflows with ease. {CodeExport} --- ## Fine-Tune Your First LLM # Fine-Tune Your First LLM **Fine-tune an LLM with LoRA in three commands using a pipeline template** The fastest way to fine-tune an LLM on Clarifai is to scaffold a project from the [`lora-pipeline-unsloth-quick-start`](https://github.com/Clarifai/pipeline-examples/tree/main/lora-pipeline-unsloth-quick-start) pipeline template, then upload and run it. The template uses public data, sensible defaults, and auto-provisions compute — no dataset setup or hyperparameter tuning required for the first run. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## Prerequisites Install the Clarifai CLI and authenticate: {`pip install --upgrade clarifai clarifai login`} `clarifai login` auto-detects your user ID and saves your [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat) locally. ## Fine-Tune the Model Scaffold a project from the LoRA quick-start template, then upload and run it: {`clarifai pipeline init --template lora-pipeline-unsloth-quick-start cd lora-pipeline-unsloth-quick-start clarifai pipeline upload clarifai pipeline run --instance=g5.xlarge`} This trains a LoRA fine-tune of [`unsloth/Qwen3-0.6B`](https://huggingface.co/unsloth/Qwen3-0.6B) on the [`mlabonne/FineTome-100k`](https://huggingface.co/datasets/mlabonne/FineTome-100k) dataset using [Unsloth](https://github.com/unslothai/unsloth). `--instance=g5.xlarge` auto-provisions a compute cluster and nodepool — no separate cluster setup required, and the same nodepool can later serve inference on the fine-tuned model. When training completes, the fine-tuned model is registered in your Clarifai model registry, ready to serve, evaluate, or refine further. ## Customize Before Running To override defaults at init time — a different base model, more epochs, custom LoRA rank — pass `--set key=value` flags. For example, to fine-tune Llama 3.2 1B for three epochs: {`clarifai pipeline init --template lora-pipeline-unsloth-quick-start \\ --set base_model_name="unsloth/Llama-3.2-1B-Instruct" \\ --set num_epochs=3`} See the [Pipeline Templates reference](https://docs.clarifai.com/compute/pipelines/templates) for all customizable parameters. ## What to Explore Next - **[Train a Visual Classifier](https://docs.clarifai.com/create/models/visual-classifier)** — Train an image classifier (ResNet) using a different pipeline template. - **[Train a Visual Detector](https://docs.clarifai.com/create/models/visual-detector)** — Train an object detector (YOLOF) using a different pipeline template. - **[Clarifai Pipelines](https://docs.clarifai.com/compute/pipelines/)** — The Pythonic-first pipeline authoring story, including the DSL for custom workflows. - **[Run inference on the fine-tuned model](https://docs.clarifai.com/compute/inference/)** — Use the trained LLM for predictions. --- ## Deploy Your First Model via UI # Deploy Your First Model via UI **Quickly set up infrastructure for inference** Clarifai provides an intuitive interface that makes it easy to provision compute infrastructure for running your models. [Deployment](https://docs.clarifai.com/compute/deployments/deploy-model) allows you to configure and activate the infrastructure needed to serve model predictions. With just a few simple steps, you can deploy a trained model and start generating predictions. ## Step 1: Sign Up or Log In Start by [logging in to](https://clarifai.com/login) your Clarifai account, or [sign up](https://clarifai.com/signup) for a new one to unlock access to the platform’s powerful AI capabilities. ## Step 2: Choose a Model Clarifai’s [Model Library](https://clarifai.com/explore/model-library) provides a curated collection of the latest AI models ready for testing and integration. Browse the library to quickly find a model by capability (such as text, vision, or multimodal), provider, or any other use case. For this guide, we’ll choose the [Llama-3.2-3B-Instruct](https://clarifai.com/meta/Llama-3/models/Llama-3_2-3B-Instruct) model. After finding a model, click the **Deploy Model** button in the upper-right corner of the page. ![](/img/new-docs/deploy-1.png) ## Step 3: Configure Your Deployment You’ll be redirected to the deployment creation page, where the fields are pre-filled for you — such as the Deployment ID, [PAT](https://docs.clarifai.com/control/authentication/pat) (for authentication), and Nodepool. You can adjust these settings as needed. ![](/img/new-docs/deploy-2.png) > **Note:** A [cloud instance and hardware](https://docs.clarifai.com/compute/cloud-instances) to host your deployment will be assigned automatically. If no compatible nodepool exists in your account, one will be created as part of the deployment process and assigned a `Pending` status. If a compatible nodepool already exists, it will be selected automatically and assigned a `Provisioned` status — meaning the instance is set up and ready to use. Finally, click **Create Deployment** in the bottom-right corner of the page to launch your deployment. A toast notification will appear at the top of the page confirming that the cluster and nodepool have been successfully created using the provided settings, and the model has been deployed within this infrastructure. You’ll then be automatically redirected to the newly created deployment page, where you can view your compute settings and the deployed model. ![](/img/new-docs/deploy-8.png) ## Step 4: Run Inferences To run a prediction with your deployed model, click the **Try Model** button — either on the deployment page or in the upper-right corner of the model's individual page. ![](/img/new-docs/deploy-9-a.png) You’ll be taken to the [Playground](https://docs.clarifai.com/getting-started/playground/) interface, where you can enter your prompt in the message box to run inferences using your deployed model. ![](/img/new-docs/deploy-9-b.png) :::tip Learn more See [how to deploy your models on Clarifai using the UI](https://docs.clarifai.com/compute/deployments/deploy-model#via-the-ui) for more details. ::: --- ## Quick Start With Playground # Quick Start With Playground **Start running inferences in just a few simple steps** Clarifai's AI Playground is an intuitive interface that allows you to experience powerful AI models in action without requiring additional setup. This interactive battleground brings together a curated selection of cutting-edge models, spanning vision, language, and multimodal tasks. You can quickly test performance, experiment with real inputs, and explore Clarifai’s full capabilities — directly from your browser. ## Step 1: Sign Up or Log In Start by [logging in to](https://clarifai.com/login) your existing Clarifai account, or [sign up](https://clarifai.com/signup) for a new one to unlock access to the platform’s powerful AI capabilities. New users receive a welcome bonus to kickstart their exploration. ## Step 2: Get a Model Clarifai’s [Model Library](https://clarifai.com/explore/model-library) provides a curated collection of the latest AI models ready for testing and integration. To get started, browse the library to discover models by provider, capability, use case, or any other parameter. Once you’ve found a suitable model, click the **Try Model** button on the far right of its row. For this guide, we’ll use the [Kimi-K2_5](https://clarifai.com/moonshotai/chat-completion/models/Kimi-K2_5) model. > **Alternatively:** You can select the **Playground** option in the collapsible left sidebar. ![](/img/new-docs/playground-2.png) ## Step 3: Run Your Inference You'll be taken to the AI Playground interface. Once the interface opens, you’ll see an information card that gives you a quick overview of the model, including input and output pricing, and links to the model page and pricing details. You’ll also find predefined prompt examples you can use right away. ![](/img/new-docs/playground-2-1.png) :::note Alternatively In the upper-left section of the Playground, you can use the search bar to find a model for inference. Clicking the search bar opens a selection modal. ![](/img/new-docs/playground-2-2.png) In the upper-right corner of the modal, there’s an account selector that lets you switch between different sources — such as featured models, your personal workspace, any organizations you belong to, or the public Clarifai model library. Selecting a different source updates the list to show the available models from that account or library. > **Note:** You can use the keyboard shortcuts shown at the bottom of the modal to navigate and select models: > - `↑ / ↓ (Navigate)` — Use the up and down arrow keys to move through the model list without using your mouse. > - `↵ (Select)` — Press Enter to select the currently highlighted model. > - `Esc (Close)` — Press Escape to close the dialog without making a selection. ::: Next, in the message box at the bottom of the Playground, enter your desired prompt to generate a response with the selected model. > **Note:** If your chosen model supports image or video inputs as prompts, you can also upload them directly into the interface. Then, submit your request by clicking the arrow icon in the message box or pressing Enter on your keyboard. :::tip For this example, we're using the default settings for inferencing — [`Clarifai Serverless deployment`](https://docs.clarifai.com/compute/overview#deployment-options) and default [inference parameters](#playground-settings). You can customize these settings as needed for more advanced use cases. ::: ![](/img/new-docs/playground-3.png) The results will stream directly in the interface, allowing you to see the generated output token by token, in real time. Note that the model’s chain-of-thought preview is shown to help you understand how the model interpreted your request. At the bottom of the response, you’ll also see the following token and performance metrics: - TTFT (Time to First Token) - IN (input token count) - OUT (output token count) - TOKENS/S (generation speed) ## Additional Playground Features ### Playground Settings The Playground's right sidebar lets you adjust settings and inference parameters to customize model behavior. - **Model Version** — Choose the specific version of the model you want to run. - **Model Method** — Select which model function to call: - **OpenAI Fetch** — Sends an OpenAI-compatible request to the appropriate OpenAI endpoint. Accepts a JSON string with request parameters (including `openai_endpoint`) and returns a JSON string containing the response or error. - **OpenAI Stream (_default_)** — Sends an OpenAI-compatible request and returns a streaming response with cancellation support. - **Predict** — Runs a single prompt completion using the OpenAI client. - **Generate** — Streams a completion response using the OpenAI client. - **Max Completion Tokens** — Sets the maximum number of tokens the model can generate for a completion. This includes both the visible output tokens and any reasoning tokens produced. - **Temperature** — Adjusts the randomness of the output, with values ranging from 0 to 2. Higher temperatures (e.g., 0.8) produce more diverse, creative responses, while lower values (e.g., 0.2) produce more focused, deterministic outputs. We generally recommend adjusting either this or `top_p`, but not both. - **Top P** — Controls nucleus sampling, an alternative to temperature-based sampling. The model selects from the smallest set of tokens whose cumulative probability reaches `top_p`. For example, a value of 0.1 limits choices to the top 10% most probable tokens. As with temperature, we recommend modifying either `top_p` or temperature, but not both. ### Compare Models The AI Playground includes a side-by-side comparison view, making it easy to test different models — or even the same model across multiple versions and instances — to identify the best fit for your needs. You can directly compare factors such as speed, accuracy, and overall performance. Click the **Compare** button in the upper-right corner of the Playground to launch the side-by-side comparison. ![](/img/new-docs/playground-5.png) Next, choose a model for the left panel and another for the right. When you’re done, close the comparison by clicking the **X** button in the upper-right corner of the right panel. ![](/img/new-docs/playground-6.png) > **Note:** If you enable the **Sync Prompts** option, the prompt you enter for the first model will automatically be mirrored to the second one, removing the need to manually copy prompts between models. ### Get Code Snippets You can toggle the button in the upper-left section of the Playground to display ready-to-use API code snippets in various programming languages. Simply copy and use them in your project. ![](/img/new-docs/playground-4.png) --- ## Quick Start With API # Quick Start With API **Start quickly with Clarifai API in a few simple steps** Clarifai provides a robust API designed to get you up and running quickly. With just a few lines of code, you can bring your AI projects to life within minutes. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import InstallJava from "!!raw-loader!../../code_snippets/new-docs/assorted/install-java.java"; import PythonSDKRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/python-sdk-request.py"; import OpenAIRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/open-ai-request.py"; import NodeSDKRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/node-sdk-request.js"; import CLIRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/cli-request.sh"; import CurlRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/curl-request.sh"; import JSRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/js-request.html"; import PyGRPCRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/py-grpc-request.py"; import NodeGRPCRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/node-grpc-request.js"; import JavaGRPCRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/java-grpc-request.java"; import PHPGRPCRequest from "!!raw-loader!../../code_snippets/new-docs/assorted/php-grpc-request.php"; import OutputExample from "!!raw-loader!../../code_snippets/new-docs/assorted/output-request.txt"; ## Step 1: Sign Up or Log In Start by [logging in to](https://clarifai.com/login) your existing Clarifai account, or [sign up](https://clarifai.com/signup) for a new one to unlock access to the platform’s powerful AI capabilities. New users receive a welcome bonus to kickstart their exploration. ## Step 2: Get a Model Clarifai’s [Community platform](https://clarifai.com/explore) offers a wide range of latest models to help you make your first API call. You can easily find a model to use by heading to the Community homepage and exploring the **Trending Models** section, which showcases popular and ready-to-use options. > **Note:** Once you’ve found a model you'd like to use, copy its full model URL — you’ll need this when making prediction requests via the API. ## Step 3: Get a PAT Key To authenticate your connection to Clarifai, you’ll need a [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat). You can get one by navigating to **Settings** in the collapsible left sidebar, selecting **Secrets**, and creating or copying an existing token from there. Then, set the PAT as an environment variable using `CLARIFAI_PAT`. This will authenticate your session when using the SDKs, including the [Clarifai CLI](https://docs.clarifai.com/resources/api-overview/cli). export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ## Step 4: Install Your Preferred SDK You can connect to the Clarifai API using the method that best fits your development environment: - [Python SDK](https://docs.clarifai.com/resources/api-overview/python-sdk) – Seamlessly integrate with Clarifai using our Python client. See the minimum system requirements [here](https://docs.clarifai.com/resources/api-overview/python-sdk#minimum-system-requirements). Note that installing the Python SDK also installs the Clarifai CLI. - [Node.js SDK](https://docs.clarifai.com/resources/api-overview/nodejs-sdk) – Use our SDK for integration in your JavaScript or TypeScript projects. See the minimum system requirements [here](https://docs.clarifai.com/resources/api-overview/nodejs-sdk/#minimum-system-requirements). - [OpenAI client](https://docs.clarifai.com/compute/providers/open-ai) – Leverage Clarifai’s OpenAI-compatible endpoint to run inferences using the OpenAI client library. Here's how to install your preferred package: pip install --upgrade clarifai npm install clarifai-nodejs pip install openai ## Step 5: Send an API Request For this example, let's use the [gpt-oss-120b](https://clarifai.com/openai/chat-completion/models/gpt-oss-120b) model to generate text based on a given prompt. {PythonSDKRequest} {CLIRequest} {NodeSDKRequest} {OpenAIRequest} {CurlRequest} {JSRequest} {PyGRPCRequest} --> Output Example {OutputExample} Congratulations — you've just gotten started with the Clarifai platform! :::tip Learn more [Click here](https://docs.clarifai.com/compute/models/inference/api/) to learn more about how to make inference requests using our API. You'll discover how to list all the available inference methods defined in a model's configuration, generate example code, leverage our Compute Orchestration capabilities for various types of inference requests, and more. ::: ## What to Explore Next You've made your first inference call. Here are some directions to explore from here: - **[Deploy your own model](https://docs.clarifai.com/compute/deployments/)** — Run inference on dedicated GPUs with your own model. - **[Train a model](https://docs.clarifai.com/create/models/)** — Fine-tune a classifier, detector, or LLM on your data using a pipeline template. - **[Build orchestration pipelines](https://docs.clarifai.com/compute/pipelines/)** — Define multi-step, long-running workflows in Python — training, evaluation, agent orchestration, batch processing. - **[Use the OpenAI-compatible API](https://docs.clarifai.com/compute/inference/open-ai/)** — Same endpoint, OpenAI client library. --- ## Deploy Your First Model via CLI # Deploy Your First Model via CLI **Quickly build and deploy your first custom model to the Clarifai platform** The Clarifai platform lets you deploy models to production in just three commands. This guide walks you through the complete workflow — from scaffolding to running predictions in the cloud. We'll show two paths: - **Path A** (recommended): Deploy a pre-trained LLM from HuggingFace using a toolkit - **Path B**: Deploy a custom Python model you write from scratch import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import ModelPyFile from "!!raw-loader!../../code_snippets/python-sdk/model-upload/upload-first-model.py"; import ConfigFile from "!!raw-loader!../../code_snippets/python-sdk/model-upload/upload-first-model.yaml"; import RequirementsFile from "!!raw-loader!../../code_snippets/python-sdk/model-upload/upload-first-model.txt"; import PythonSDKRequest from "!!raw-loader!../../code_snippets/python-sdk/model-upload/predict-first-model.py"; ## Step 1: Install and Log In ### Install Clarifai pip install --upgrade clarifai This installs both the Python SDK and the [CLI](https://docs.clarifai.com/resources/api-overview/cli). ### Log In You need a Clarifai account and a Personal Access Token (PAT). If you don't have one: 1. [Sign up](https://clarifai.com/signup) or [log in](https://clarifai.com/login) 2. Go to **Settings** > **Secrets** in the left sidebar 3. Create a new PAT and copy it Then authenticate: clarifai login The CLI will prompt for your PAT, validate it, auto-detect your user ID, and save everything locally. Verify it worked: clarifai whoami You should see your user ID and the active context name. ## Path A: Deploy a Pre-Trained LLM (Recommended) {#path-a} This is the fastest way to get a model running on Clarifai. We'll use [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) — a small, ungated model that doesn't require a HuggingFace token. ### Step 2A: Scaffold clarifai model init --toolkit vllm --model-name Qwen/Qwen3-0.6B This creates a `Qwen3-0.6B/` directory with everything pre-configured: ```text Qwen3-0.6B/ ├── 1/ │ └── model.py # vLLM inference logic (ready to use) ├── config.yaml # Config with auto-selected GPU instance └── requirements.txt # Dependencies (clarifai, openai — vLLM comes from Docker image) ``` The generated `config.yaml` is minimal — no placeholders to fill in: ```yaml build_info: image: vllm/vllm-openai:latest checkpoints: repo_id: Qwen/Qwen3-0.6B type: huggingface when: runtime # Downloads weights at startup, not during build compute: instance: g4dn.xlarge # Auto-selected based on model VRAM needs model: id: qwen3-06b # Sanitized from Qwen/Qwen3-0.6B ``` > **What gets auto-filled:** `user_id` and `app_id` are resolved from your login context at deploy time. `model.id` is sanitized from the HuggingFace model name (e.g., `Qwen/Qwen3-0.6B` → `qwen3-06b`). `compute.instance` is auto-selected based on the model's estimated VRAM requirements — the CLI fetches the model's architecture from HuggingFace and calculates the exact memory needed for weights, KV cache, and framework overhead. `build_info.image` specifies the Docker base image (vLLM comes pre-installed in it, so `requirements.txt` only lists lightweight dependencies). ### Step 3A: Deploy clarifai model deploy ./Qwen3-0.6B That's it. The CLI will: 1. **Validate** — check `config.yaml` and verify the HuggingFace repo is accessible 2. **Upload** — build a Docker image and push it to Clarifai 3. **Deploy** — auto-create a compute cluster (a group of machines), a nodepool (the specific GPU instance), and a deployment (your model running on that hardware) 4. **Monitor** — stream pod events until the model is ready When it finishes, you'll see output like this: ```text ── Ready ────────────────────────────────────────────── Model deployed successfully! Model: https://clarifai.com/your-user/main/models/qwen3-06b Version: abc12345 Deployment: deploy-qwen3-06b-dd8481 Instance: g4dn.xlarge Cloud: AWS / us-east-1 ── Next Steps ───────────────────────────────────────── Predict: clarifai model predict your-user/main/models/qwen3-06b "Hello" Logs: clarifai model logs --deployment "deploy-qwen3-06b-dd8481" Status: clarifai model status --deployment "deploy-qwen3-06b-dd8481" Undeploy: clarifai model undeploy --deployment "deploy-qwen3-06b-dd8481" ``` > **Copy the predict command from the output** — it contains your actual user ID and deployment ID, so you can paste it directly. ### Step 4A: Predict Copy the predict command from the deploy output, or construct it using your user ID: clarifai model predict your-user/main/models/qwen3-06b "Explain quantum computing in one sentence" > Replace `your-user` with your actual Clarifai user ID (the one shown by `clarifai whoami`). The `main` is the default app that's auto-created for you. The response streams in real-time. You can also test in the [Playground](https://docs.clarifai.com/getting-started/quickstart-playground) by clicking the model URL from the deploy output. ### Step 5A: Manage ```bash # Check deployment status clarifai model status --deployment # Stream live logs (useful for debugging startup issues) clarifai model logs --deployment # View Kubernetes scheduling events (useful if pod won't start) clarifai model logs --deployment --log-type events # Remove deployment when done (stops billing) clarifai model undeploy --deployment ``` > Replace `` with the ID from the deploy output (e.g., `deploy-qwen3-06b-dd8481`). --- ## Path B: Deploy a Custom Python Model {#path-b} Use this path when you're writing your own model logic (not wrapping a pre-trained model). ### Step 2B: Scaffold clarifai model init my-first-model This creates a blank model project. Replace the generated files with your model logic: #### `1/model.py` {ModelPyFile} #### `requirements.txt` {RequirementsFile} #### `config.yaml` {ConfigFile} > `user_id` and `app_id` are auto-filled from your CLI context at deploy time. You don't need to add them. ### Step 3B: Test Locally (Optional) Before deploying, verify your model works locally: clarifai model serve ./my-first-model This starts the model and connects it through a Clarifai-managed API endpoint for testing. Press Ctrl+C to stop. For offline development without a Clarifai login: clarifai model serve ./my-first-model --grpc ### Step 4B: Deploy clarifai model deploy ./my-first-model This CPU-only model doesn't need `--instance` — the CLI uses the `inference_compute_info` from `config.yaml`. For GPU models, you'd add `--instance g5.xlarge` (run `clarifai list-instances` to see all available GPU instances). ### Step 5B: Predict Copy the predict command from the deploy output, or construct it: clarifai model predict your-user/main/models/my-first-model "Yes, I uploaded it!" > Replace `your-user` with your Clarifai user ID (shown by `clarifai whoami`). Output Example Yes, I uploaded it! Hello World 0 Yes, I uploaded it! Hello World 1 Yes, I uploaded it! Hello World 2 Yes, I uploaded it! Hello World 3 Yes, I uploaded it! Hello World 4 Or use the Python SDK: {PythonSDKRequest} --- ## Troubleshooting ### Deploy is stuck on "Monitor" phase The model pod is starting up. For large LLMs, this can take 5–10 minutes as weights download at runtime. Check progress with: ```bash clarifai model logs --deployment ``` If the pod is failing to schedule, check Kubernetes events: ```bash clarifai model logs --deployment --log-type events ``` ### HuggingFace token error Some models (Llama, Gemma, etc.) are **gated** and require a HuggingFace access token. The CLI catches this early during the Validate phase: ```text UserError: HuggingFace repo 'meta-llama/Llama-3.1-8B-Instruct' requires authentication. Set HF_TOKEN in your environment: export HF_TOKEN=hf_... ``` To fix: [create a HuggingFace token](https://huggingface.co/docs/hub/en/security-tokens), request access to the model, then set the token: ```bash export HF_TOKEN=hf_your_token_here ``` The CLI will automatically include it in the build. ### Model predict returns an error Make sure the deployment is fully ready before predicting: ```bash clarifai model status --deployment ``` Wait until the status shows the deployment is active. Runtime checkpoint downloads can take several minutes for large models. --- ## What's Next? - Browse all [available GPU instances](https://docs.clarifai.com/resources/api-overview/cli#clarifai-list-instances) for deployment (`clarifai list-instances`) - Learn about [toolkits](https://docs.clarifai.com/compute/toolkits) (vLLM, SGLang, Ollama, and more) - Explore the full [CLI reference](https://docs.clarifai.com/resources/api-overview/cli) - Set up [autoscaling](https://docs.clarifai.com/compute/deployments/deploy-model#set-autoscaling) for production workloads **Congratulations!** You've successfully deployed a model to the Clarifai platform and run inference with it. --- ## DSPy-Clarifai Integration # DSPy-Clarifai Integration **Learn how to use DSPy along with Clarifai Python SDK** [DSPy](https://dspy-docs.vercel.app/docs/intro) is a framework designed to address the fragility issue encountered in language model based applications by emphasizing programming over prompting. It will optimize the entire pipeline for your specific task. This approach eliminates the need for repetitive manual adjustments to prompts whenever a component is modified. By integrating Clarifai into DSPy, users gain access to Clarifai's capabilities, particularly its ability to call language models (LLM) from the Clarifai platform. This integration empowers users to utilize Clarifai's application as a retriever for their vector search use cases. Clarifai, as a platform, offers a suite of tools making it a valuable addition to DSPy. Its integration allows for the seamless incorporation of LLM models into the DSPy framework, enhancing the efficiency and effectiveness of tasks such as vector search. Why integrating with Clarifai is beneficial? * Access to Advanced Features: Clarifai provides advanced capabilities that can be leveraged within DSPy to enhance various tasks. * Improved Retrieval for Vector Search: By utilizing Clarifai as a retriever for vector search use cases, users can benefit from its robust search capabilities, potentially improving the accuracy and speed of search operations. * Reduced Manual Adjustments: Since DSPy emphasizes programming over prompting, the integration with Clarifai helps minimize the need for repetitive manual adjustments to prompts, thereby enhancing overall efficiency and reducing the risk of errors. 0:02 - Intro to DSPy 1:09 - DSPy Concepts 2:27 - Why Clarifai Integration is useful 2:51 - Code Demo 3:04 - Create Clarifai PAT 3:25 - Get Clarifai User ID 3:42 - Create an App on the Clarifai Platform 4:43 - Choose LLM from Clarifai 6:20 - Create Clarifai Vector Store 6:40 - Initialize LLM 6:55 - Create Clarifai Retriever 7:25 - First Example Run 8:50 - Using dspy.Retrieve method 9:50 - Building a RAG Module 11:09 - Creating Custom Signature Class 12:11 - Final Output import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## DSPy Modules & Signatures # DSPy Modules & Signatures **Learn about Modules & Signatures in DSPy** ## Modules DSPy introduces a novel approach to programming large language models (LLMs) by leveraging modular components called modules. These modules act as reusable building blocks that encapsulate specific functionalities or sub-tasks within a larger LLM program. Unlike pre-built functions with fixed purposes, DSPy modules are designed to be adaptable. They can be combined in various ways to tackle complex tasks and can be applied across different use cases. The ability to create custom modules extends DSPy's capabilities. Developers can tailor modules to address specific needs or domain-specific tasks, making DSPy a highly versatile framework. Some core modules built in DSPy are as follows, * ```Retrieve``` Module: This module facilitates information retrieval from the LLM. You can provide the LLM with a prompt or query, and the Retrieve module will extract the relevant information. * ```ChainOfThought``` Module: This module empowers the LLM to engage in chain-of-thought reasoning. It prompts the LLM to break down complex tasks into a series of logical steps, providing transparency into its reasoning process. * ```Predict``` Module: This module focuses on making predictions based on provided data. It can be used for tasks like sentiment analysis or image classification. In the following example ```dspy.Predict``` is the module we are using for performing sentiment classification. ```python sentence = "Fuel pump is broken" classify = dspy.Predict('sentence -> sentiment') print(classify(sentence=sentence).sentiment) ``` Click [here](https://dspy-docs.vercel.app/docs/building-blocks/modules) to learn more about Modules in DSPy. ## Signatures DSPy signatures guide the LLM through each task with clear instructions. Unlike pre-programmed commands that dictate every step, DSPy signatures emphasize the desired outcome. They tell the LLM what needs to be achieved (mission objective) without prescribing the exact approach. This allows the LLM to leverage its capabilities creatively to fulfil the mission. A DSPy signature includes the following fields, * Input Fields: This could be text, data, code, or anything relevant to the objective. Think of them as the reference materials provided to the LLM. * Output Fields: These define the answer, prediction, or generated text the LLM should produce. In the following example the string `sentence -> sentiment` is a signature for performing sentiment classification. :::tip The field names used in signatures should be meaningful. ::: ```python sentence = "Fuel pump is broken" classify = dspy.Predict('sentence -> sentiment') print(classify(sentence=sentence).sentiment) ``` Click [here](https://dspy-docs.vercel.app/docs/building-blocks/signatures) to learn more about Signatures in DSPy. --- ## Multistage RAG Pipeline With DSPy import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeOutputMR from "!!raw-loader!../../../code_snippets/python-sdk/dspy/outputs/multi_rag.txt"; import CodeOutputRC from "!!raw-loader!../../../code_snippets/python-sdk/dspy/outputs/rag_compare.txt"; # Multistage RAG Pipeline With DSPy **A tutorial by Mogith PN** Information retrieval techniques are growing at such a high pace that the quest for ever-more efficient and accurate systems continues. One such advanced methodology is called RAG. But what if we could push the boundaries of RAG even further? Today, we'll explore the exciting potential of multi-stage, multi-model RAG systems built with DSPy. ## What is RAG? Imagine a system that can sift through mountains of information, identify the most relevant bits, and then craft a clear and concise response to your query. That's the essence of RAG. It operates in three key stages: * Retrieval: A `retriever` module dives into a vast corpus of documents, seeking information related to the user's query. * Augmentation: The retrieved information changes at the `augmenter` stage. This might involve filtering, summarising, or enriching the data with additional context. * Generation: Finally, the `generator` creates a high-quality response that effectively addresses the user's query. Visit [this](https://docs.clarifai.com/sdk/rag) page to learn more about RAG. But how will DSPy help RAG? Well, DSPy supercharges the process inside RAG. Traditional large language model applications often suffer from "fragility," requiring constant tweaking and adjustments when components change. DSPy framework optimizes the entire pipeline for your specific task, eliminating the need for repetitive manual fine-tuning whenever you modify a component. By integrating Clarifai with DSPy, you unlock access to Clarifai's powerful capabilities, including its ability to seamlessly call language models (LLMs) directly from the Clarifai platform. This integration empowers you to leverage Clarifai's application as a retriever specifically suited for vector search use cases. Before we move on, install and setup some packages, ``` !pip install dspy-ai !pip install clarifai ``` ```python import os #Replace your PAT os.environ['CLARIFAI_PAT'] ="YOUR_PAT" ``` Now let's explore a possible implementation of a multi-stage, multi-model RAG system using DSPy. The motive of this experiment is to test the functionality of DSPy in building multi-stage systems in our RAG pipelines. Also, the compatibility to call and use multi models at different stages of the pipeline. You can choose any LLM’s from [this](https://clarifai.com/explore/models?searchQuery=llm&page=1&perPage=24) page. ```python import dspy from dspy.retrieve.clarifai_rm import ClarifaiRM MODEL_URL1 = "https://clarifai.com/mistralai/completion/models/mistral-7B-Instruct" MODEL_URL2 = "https://clarifai.com/meta/Llama-2/models/llama2-7b-chat" PAT = "**********" USER_ID = "USER_ID" APP_ID = "APP_ID" mistral_llm=dspy.Clarifai(model=MODEL_URL1, api_key=PAT, n=1, inference_params={'temperature':0.6}) retriever_clarifai=ClarifaiRM(clarifai_user_id=USER_ID, clarfiai_app_id=APP_ID, clarifai_pat=PAT, k=5) ``` The `GenerateAnswer` class acts as a module within the DSPy pipeline. It takes the user's question and a relevant summary passage as inputs and then utilizes these to generate a concise answer. ```python class GenerateAnswer(dspy.Signature): """Think and Answer the questions based on the context provided.""" context = dspy.InputField(desc="may contain relevant summary passage about user query") question = dspy.InputField(desc="User query") answer = dspy.OutputField(desc="Generate a brief answer") ``` The `GenerateKeywords` class acts as a module within the DSPy pipeline. It takes the user's query as input and then generates a set of keywords that are closely related to the user's original query. ```python class GenerateKeywords(dspy.Signature): """Generate Key words for search, which will be related to user's query""" query = dspy.InputField(desc="User query") answer = dspy.OutputField(desc="3 Search Key words related to user query") ``` ## Why do we need a Re-ranker node? We are retrieving contexts in 2 ways for our pipeline, starting with query search and keyword search, so our retrieved contexts will be quite huge for the model to handle. The `Reranker` class acts as a quality control step within the DSPy pipeline. It takes a single retrieved document and the user's question as input. It then analyzes the document to assess how well it aligns with the user's query and provides a factual answer. The assigned rating (between 1 and 5) serves as an indicator of the document's relevance, potentially influencing how the retrieved information is used in subsequent stages of the pipeline. ### DSPY Typed Predictors In DSPy, typed predictors are a mechanism to enforce type constraints on the inputs and outputs of modules within a pipeline. Click [here](https://dspy-docs.vercel.app/docs/building-blocks/typed_predictors) to learn more about typed predictors. First, let's define the output for the re-ranker model to be float using `dspy typed predictors`. And then we will create the `Reranker` class. ```python from pydantic import BaseModel, Field class Reranker_Output(BaseModel): score: float = Field(desc="A rating between 1 to 10 based on relevance and semantic match, IMPORTANT ! `float` . Nothing else return only float value as output.") ``` ```python class Reranker(dspy.Signature): """Evaluate and rate the retrieved contexts based on their relevance in providing factual answers to user questions.""" context = dspy.InputField(desc="The context retrieved for answering question") question = dspy.InputField(desc="User query") rating = dspy.OutputField(desc="A rating between 1 to 5, IMPORTANT ! `float` . Nothing else return only float value as output.") ``` The `Multi_RAG` module constructs a multi-stage information retrieval system using DSPy. It extracts keywords to broaden the search, retrieves information using both the query and keywords, and then employs a separate language model to evaluate and rank the retrieved passages based on their relevance to the user's question. This multi-stage approach with distinct components allows for a more comprehensive and potentially more accurate information retrieval process. ```python import re import functools class Multi_RAG(dspy.Module): def __init__(self): super().__init__() self.retrieve = dspy.Retrieve(k=5) self.generate_kv = dspy.ChainOfThought(GenerateKeywords) self.generate_answer = dspy.ChainOfThought(GenerateAnswer) self.generate_ratings = dspy.functional.TypedPredictor(Reranker) def forward(self, question): with dspy.context(lm=llama_llm): kv = self.generate_kv(query=question).answer print(f"Keywords extracted : {(kv)}") context=[] for search_query in [question,kv]: results = self.retrieve(search_query).passages context.extend(results) context_rating = [] counter=0 for retrieved_context in context: _rating={} with dspy.context(lm=mistral_llm): rating = self.generate_ratings(context=retrieved_context, question=question).rating _rating['id'] = counter _rating['score'] = rating _rating['context'] = retrieved_context counter+=1 print(f"Rating score for context {counter}: {rating}") context_rating.append(_rating) sorted_data = sorted(context_rating, key=lambda x: float(x['score'].score)) ranked_context = [passage["context"] for passage in sorted_data[:5]] with dspy.context(lm=solar_llm): prediction = self.generate_answer(context=str(ranked_context), question=question) return dspy.Prediction(context=str(ranked_context), answer=prediction.answer) ``` And finally, we run a query on the `Multi_Rag` class, ```python my_question = "What are the potential risks associated with large language models (LLMs) according to the context information?" # Get the prediction. This contains `pred.context` and `pred.answer`. Rag_obj= Multi_RAG() predict_response = Rag_obj(my_question) # Print the contexts and the answer. print(f"Question: {my_question}") print(f"Predicted Answer: {predict_response.answer}") ``` Output {CodeOutputMR} Now we will compare the results obtained from Multistage DSPy RAG with a Vanilla RAG. ```python from clarifai.rag import RAG rag_agent = RAG.setup(app_url="YOUR_APP_URL", llm_url="https://clarifai.com/mistralai/completion/models/mistral-large", max_results=5) n_rag_response = rag_agent.chat(messages=[{"role": "human", "content": "What are the potential risks associated with large language models (LLMs) according to the context information?"}]) print(n_rag_response[0]["content"]) ``` Output {CodeOutputRC} Then we can evaluate the performance of each RAG with the help of `ragas` library. Ragas is a valuable tool for assessing the performance of RAG pipelines. Evaluation of Multistage DSPy RAG, ```python from datasets import Dataset from ragas.metrics import faithfulness, answer_correctness from ragas import evaluate data_samples = { 'question': [benchmark_files["examples"][0]["query"]], 'answer': [predict_response.answer], 'ground_truth': [benchmark_files["examples"][0]["reference_answer"]] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset,metrics=[answer_correctness]) score.to_pandas() ``` Output ```python from datasets import Dataset from ragas.metrics import faithfulness, answer_correctness from ragas import evaluate data_samples = { 'question': [benchmark_files["examples"][0]["query"]], 'answer': [n_rag_response[0]["content"]], 'ground_truth': [benchmark_files["examples"][0]["reference_answer"]] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset,metrics=[answer_correctness]) score.to_pandas() ``` Output If you observe the values of `answer_correctness` for both RAGs, it can be seen that Multistage DSPy RAG outperforms Naive RAG. This Multi-Stage Multi-model RAG is a testament to the ability of DSPy framework and its modular nature of building different sets of stages within our LLM application. --- ## Building RAG with DSPy import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeInit from "!!raw-loader!../../../code_snippets/python-sdk/dspy/init.py"; import CodeData from "!!raw-loader!../../../code_snippets/python-sdk/dspy/data.py"; import CodeDsp from "!!raw-loader!../../../code_snippets/python-sdk/dspy/dspy.py"; import CodeTrun from "!!raw-loader!../../../code_snippets/python-sdk/dspy/trun.py"; import CodeGA from "!!raw-loader!../../../code_snippets/python-sdk/dspy/ga.py"; import CodeRAG from "!!raw-loader!../../../code_snippets/python-sdk/dspy/rag.py"; import CodeChat from "!!raw-loader!../../../code_snippets/python-sdk/dspy/chat.py"; import CodeOutputData from "!!raw-loader!../../../code_snippets/python-sdk/dspy/outputs/data.txt"; import CodeOutputTrun from "!!raw-loader!../../../code_snippets/python-sdk/dspy/outputs/trun.txt"; import CodeOutputChat from "!!raw-loader!../../../code_snippets/python-sdk/dspy/outputs/chat.txt"; # Building RAG with DSPy **Learn how to build RAG with DSPy and Clarifai Python SDK** RAG systems combine two key functionalities: information retrieval and text generation. When you ask a question, RAG first retrieves relevant information (context) from a source like a document or database. Then, it uses that context to generate a well-informed and accurate answer. DSPy acts as your mission control for building RAG systems. It provides essential tools like modules and signatures to design and execute your program efficiently. Click [here](https://www.clarifai.com/blog/what-is-rag-retrieval-augmented-generation) to learn more about RAG. ## Prerequisites * Setting up the Clarifai Python SDK along with PAT. Refer to the installation and configuration with the PAT token [here](https://docs.clarifai.com/python-sdk/sdk-overview/). :::note Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) ::: * Clone the Clarifai Examples repository to get the data files required for this example. ``` !git clone https://github.com/Clarifai/examples.git %cd /content/examples/ ``` * Install the required packages. ``` !pip install clarifai !pip install langchain !pip install dspy-ai ``` ## Initialization The first part of creating a DSPy-Clarifai application is to set some initial fields. The variables should be configured correctly for authentication and model access. {CodeInit} Here we are opting for llama2-70b-chat as the LLM Model. You can choose different LLM Models for the DSPy from Clarifai Community [Models](https://clarifai.com/explore/models?filterData=%5B%7B%22field%22%3A%22use_cases%22%2C%22value%22%3A%5B%22llm%22%5D%7D%5D&page=1&perPage=24). ## Data Ingestion To use Clarifai as a retriever, the straightforward approach involves ingesting documents directly into the Clarifai app, which functions as your vector database. This enables easy retrieval of similar documents based on their vectors. To streamline this ingestion process, we've integrated the Clarifai vector database into our workflow. For this task, we will be using the Vehicle Repair Manual as data for DSPy. :::info Langchain is required only for the data ingestion step, which you can skip if data has been already ingested through alternate methods. ::: {CodeData} Output {CodeOutputData} ## Setup DSPy In the next step we are going to initialize DSPy with an LLM model from the Clarifai platform, this showcases the flexibility Clarifai offers. {CodeDsp} Before we move on to the next section let’s do a test run, {CodeTrun} Output {CodeOutputTrun} ## RAG with DSPy To construct a RAG module in DSPy effectively, you first need to define its signature. The signature explains the input and output fields succinctly, mapping from "question" to "answer" in a clear and intuitive manner. Once the signature is established, you proceed to create the module itself. A module in DSPy is where you put the signature into action, defining a specific functionality that compiles and generates responses based on the given queries. To begin, you construct a signature class, detailing the required input fields and the corresponding output fields. It's essential to provide comprehensive docstrings and descriptions within the class to ensure that the DSPy signature understands the context thoroughly and can compile the best prompt for the given use case. By following these steps, you can create robust and effective modules within DSPy, enabling seamless processing and response generation for various natural language processing tasks. The ```GenerateAnswer``` class is given below, {CodeGA} The ```RAG``` class is given below, {CodeRAG} ## Chat In the final step, we are going to perform information retrieval using a Clarifai retriever based on factual evidence. {CodeChat} Output {CodeOutputChat} --- ## Integrating Clarifai With Databricks # Integrating Clarifai With Databricks **Improve your data processing and analytics capabilities** [Databricks](https://docs.databricks.com/en/introduction/index.html) is a cloud-based data platform for big data analytics and machine learning. You can use its unified interface and tools to process, store, clean, share, analyze, model, and monetize your datasets and AI solutions at scale. Databricks is built on top of [Apache Spark](https://spark.apache.org/), an open-source, fast, and general-purpose cluster-computing framework. Spark allows for distributed data processing, making it suitable for handling large-scale data and complex analytics tasks. Databricks provides a wide range of features that make it easy to use Spark, including: - **Managed clusters**: Databricks automatically manages Spark clusters for you, so you don't have to worry about the hassle of setting them up and maintaining them. - **Interactive notebooks**: Databricks supports interactive notebooks, similar to Jupyter notebooks, which allow you to create and share documents that contain live code, equations, visualizations, and narrative text. - **Collaboration tools**: Databricks provides collaboration tools that make it easy for your teams to work together in a shared workspace. Integrating Clarifai with Databricks allows you to streamline and manage your machine learning journey. By combining Clarifai's capabilities with Databricks, you can gain advanced analytical insights from both structured and unstructured data. Integrating Clarifai into the Databricks environment allows you to: - Upload your datasets into a Clarifai application. This simplifies transferring data from Databricks into Clarifai. - Fetch datasets from a Clarifai application to Databricks. This facilitates further data exploration and analysis within Databricks. - Fetch annotations from a Clarifai application to Databricks. This enriches your training data, improving the accuracy and performance of your machine learning models. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Databricks-Connect Module # Databricks-Connect Module **Use a no-code interface to move data across Databricks and Clarifai** The Clarifai’s [Databricks-Connect](https://clarifai.com/clarifai/Databricks-Connect-App/modules/Databricks-Connect) module seamlessly integrates Clarifai and Databricks, offering a user-friendly, no-code solution for bi-directional data movement. You can use the module to effortlessly transfer data across both platforms at the click of the button. :::tip Modules are custom plugins with UI elements that let you extend the capabilities of the Clarifai platform. You can learn more about them [here](https://docs.clarifai.com/portal-guide/modules/). ::: The Databricks-Connect module allows you to: - Authenticate a Databricks connection and connect with its compute clusters - Import data from Databricks volume into Clarifai app and dataset - Export data and annotations from a Clarifai app into Databricks volume and table - Update input and annotation information within the chosen Delta table for the linked Clarifai app Let’s demonstrate how you can use the module. ## Prerequisites - A [Clarifai application](https://docs.clarifai.com/clarifai-basics/applications/create-an-application) - A [dataset](https://docs.clarifai.com/portal-guide/datasets/create-get-update-delete) within a Clarifai application - Set up your Clarifai [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) inside your Databricks workspace. [Click here](https://docs.databricks.com/en/security/secrets/secrets.html#secrets) to learn how to create and set secrets in Databricks. Also, use the following naming convention when setting secrets, as the module interprets secrets according to the specified scope and key name. - `scope-name = "clarifai"` - `key-name = "CLARIFAI_PAT"` ## Install the Module ### Option 1 Go to the Databricks-Connect module [page](https://clarifai.com/clarifai/Databricks-Connect-App/modules/Databricks-Connect) and click the **Install Module** button at the upper-right corner. Next, on the small window that pops up, select a desired user and destination app for installing the module. Click the **Confirm** button. ![](/img/modules/databricks-connect-1.png) On the ensuing page, choose a destination URL for installing the module, select the module visibility, and click the **Install to this App!** button. After the module has been successfully installed, click the **Click to use installed module version** button to start using it. You can also refresh the page for your installed module to appear on the left sidebar under the **Installed Modules** category.. ![](/img/modules/databricks-connect-2.png) If you click the button, you’ll be redirected to a page that requires you to authorize the module to access data on your behalf. You will see the permissions the module requires. ![](/img/modules/databricks-connect-3.png) Click the **Authorize** button. ### Option 2 Alternatively, you can use the module’s URL to install it in your app. To do so, click the **Manage Installed Modules** option at the bottom of the collapsible left sidebar. Grab the URL of the Databricks-Connect module and paste it into the **Module URL** input field. ![](/img/modules/databricks-connect-4.png) You can then follow the ensuing prompts to complete installing the module in your app. ## Authenticate a Databricks Connection After installing the module and authorizing it to access data on your behalf, you’ll be redirected to a page that requires you to provide configuration information to connect to your Databricks workspace. Provide the following credentials: ![](/img/modules/databricks-connect-5.png) - **Databricks Host** — Specify the hostname. Example is `https://.cloud.databricks.com`. - **Databricks Token** — Specify the personal access token. Learn how to get it [here](https://docs.databricks.com/en/dev-tools/auth/pat.html#databricks-personal-access-token-authentication). - **Databricks Cluster ID** — Specify the `cluster_id`. Example is `1234-12345-ab1cd2ef`. Next, click the **Authenticate** button. After successfully authenticating a Databricks connection, a section will appear on the same page that allows you to carry out various data ingestion tasks with the module. :::info We currently only support transferring image data between Clarifai and Databricks. ::: ## Import Data From Databricks to Clarifai You can use the module to import data into a Clarifai app and a dataset. While importing the data, a Delta table will be created in Databricks that tracks the exported input information. Under the **Import** tab, you can either import data from a **Databricks Unity Catalog** or an Amazon **S3** bucket. ### a) Import from a Databricks volume To import data from a Databricks volume, select the **Databricks Unity Catalog** tab and provide the needed information. :::note In Databricks, [a catalog](https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html) contains schemas, and a schema acts like a database that contains tables (for storing structured data) and volumes (for storing unstructured data). ::: ![](/img/modules/databricks-connect-6.png) - **List of catalogs available** — Select your desired source catalog. - **List of schemas available** — Select your desired source schema. - **List of volumes available** — Select your desired source volume. - **Inputs Delta table name** — Provide the name of the new table that will be created in Databricks. The table will be created inside `selected_catalog.selected_schema.table_name`. :::note The image names in the Databricks platform will be used as `input_id` names on the Clarifai platform. So, ensure the names are unique to prevent any potential errors during the import process. ::: Lastly, specify the destination app and dataset on the Clarifai platform where you intend the data to be imported. Click the **Upload** button. ![](/img/modules/databricks-connect-7.png) If the import process is successful, you’ll see the exported data tracked inside the created `selected_catalog.selected_schema.table_name`. ![](/img/modules/databricks-connect-17.png) ### b) Import from an S3 bucket To import data from an S3 bucket, select the **S3** tab and provide the needed information. :::note To enable the ingestion of inputs from an S3 bucket, ensure that the chosen cluster for the operation has the required permissions and roles to access it. For detailed instructions on configuring Databricks S3 access with the instance profile, refer to the documentation [here](https://docs.databricks.com/en/connect/storage/tutorial-s3-instance-profile.html#tutorial-configure-s3-access-with-an-instance-profile). ::: ![](/img/modules/databricks-connect-8.png) You have the flexibility to use any of the following file formats for data ingestion: - **CSV** — Refer to the explanation below on the correct format to use. - **Delta format** (Parquet format) — Ensure the format follows the correct data schema as explained below. After choosing your desired file format type, enter the S3 URL of the file you want to import. Here is an example: `s3://my-s3-bucket/my-folder/my-file.csv`. Lastly, specify the destination app and dataset on the Clarifai platform where you intend the data to be imported. Click the **Upload** button. ### File Format The CSV or the Delta file you want to import should be in the format that Clarifai accepts. Here are the columns you can include in the file: - **inputid** (mandatory) — A unique identifier for the input. - **input** (mandatory) — A publicly accessible URL that will be used to download the image. - **concepts** (optional) — The concepts contained in images. - **metadata** (optional) — Additional information in inputs. - **geopoints** (optional) — Geopoints applicable to the inputs. Here is an example: | inputid | input | concepts | metadata |geopoints | |-----------|----------|---------|---------|-------| | image01 | https://samples.clarifai.com/featured-models/color-colored-houses-boats-on-canal.jpg | image | `{'filename': 'canal.jpg', 'split': 'train'}` | 102,70 | | image02 | https://samples.clarifai.com/metro-north.jpg | image | | | Here are more details of the annotation structures Clarifai support: | Annotation Type | Structure | |------------- | ------------- | | Bounding box annotation | Regions[...].region_info.bounding_box`{top_row,left_col,bottom_row,right_col}`, Regions[...].data.concepts | | Polygon annotation | `Regions[...].region_info.polygon.points{row[…],col[…]}, Regions[...].data.concepts` | | Image segmentation |Regions[...].region_info.mask, Regions[...].data.concepts | | Video prediction | Frame.frame_info.time[…], Frame.frame_info.data.concepts | | Text/image classification| Annotations.data.concepts | ## Export Data From Clarifai to Databricks You can use the module to export data from a Clarifai app to a Databricks table. This allows you to seamlessly transfer pre-annotated, ready-to-use data for building AI pipelines within the Databricks environment. Under the **Export** tab, start by specifying the app and dataset on the Clarifai platform from which you want to export the data. ![](/img/modules/databricks-connect-9.png) Next, on the **Databricks** section, you have the option to select **Export images**. If left unchecked, the specified annotations and their associated information will be exported to a Databricks table. On the other hand, checking it will result in the export of both the specified annotations and their corresponding images to a Databricks table. ### Option 1 If you leave the **Export images** option unchecked, you need to provide the following information: ![](/img/modules/databricks-connect-10.png) - **List of catalogs available** — Select your destined, pre-existing catalog. - **List of schemas available** — Select your destined, pre-existing schema. - **Inputs Delta table name** — Provide the name of a new annotations Delta table to be created. After providing the required information, click the **Export** button. It will create a Delta table that contains both annotations and inputs information. A preview of the newly created table data will also be displayed below the **Export** button. ![](/img/modules/databricks-connect-14.png) :::info The inputs table encompasses details about each input, including input ID, image information, date created, and more. The annotations table holds information on each annotation, such as annotation ID, annotation user ID, date created, and more. The common link between these two tables is the **input ID**, which is the same in both of them. Here is a sample database structure of inputs table and annotations table: ![](/img/modules/databricks-connect-18.png) ::: ### Option 2 On the other hand, checking the **Export images** option enables you to export the annotations alongside their associated images in a Databricks volume. Note that this action will activate an extra selection box, allowing you to choose the specific volume within the schema where you wish to store the images. ![](/img/modules/databricks-connect-11.png) :::note The `input_id` of an input will serve as its filename when stored within the designated volume. ::: This is how the exported data appears on Databricks: ![](/img/modules/databricks-connect-16.png) ## Update Databricks With Inputs and Annotations You can use the module to update Databricks Delta tables with new inputs and annotations. This allows you to synchronize your Databricks workspace whenever you've added new inputs or annotations in your Clarifai app, modified existing ones, or made any changes to them. Under the **Update** tab, start by specifying the app and dataset on the Clarifai platform which have your updated data. ![](/img/modules/databricks-connect-12.png) Next, on the **Databricks** section, provide the following information: ![](/img/modules/databricks-connect-13.png) - **List of catalogs available** — Select your destined, pre-existing catalog. - **List of schemas available** — Select your destined, pre-existing schema. - **Annotations Delta table** — Select your destined, pre-existing annotations table. It cannot be the same as the inputs Delta table. - **Inputs Delta table** — Select your destined, pre-existing inputs table. It cannot be the same as the annotations Delta table. After providing the required information, click the **Update** button. The upsert process will begin in the background, seamlessly updating the corresponding tables. A preview of the table data will also be displayed below the button. ![](/img/modules/databricks-connect-15.png) --- ## Fetch Annotations # Fetch Annotations **Seamlessly retrieve annotations from Clarifai into Databricks** The Clarifai platform allows you to [annotate your inputs](https://docs.clarifai.com/portal-guide/annotate/create-get-update-delete), enriching them with valuable labels and metadata. You can effortlessly fetch annotations from a Clarifai application to Databricks. This integration is particularly valuable for machine learning workflows in Databricks, as it introduces annotated data from Clarifai into the platform. Annotated data enhances the quality of training data, a crucial factor in improving the accuracy and performance of machine learning models. Let’s illustrate how you can seamlessly transfer annotations from Clarifai into the Databricks environment. ## Prerequisites - Databricks notebook development environment - Get your PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get your Clarifai user ID - Get the ID of the Clarifai app where you want to fetch the annotations from - Get the ID of the dataset having the annotations within your app - Install the [Clarifai PySpark](https://github.com/Clarifai/clarifai-pyspark) package by running `pip install clarifai-pyspark ` - Install Protocol Buffers by running `pip install protobuf==4.24.2 `. It’s a cross-platform, serialization protocol that describes the structure of the data to be sent :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Databricks1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_fetch_annotations_JSON.py"; import Databricks2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_fetch_annotations_dataframe.py"; import Databricks3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_fetch_annotations_dataframe_2.py"; ## Retrieve Annotations in JSON Format You can retrieve detailed information about the annotations in your Clarifai app’s dataset. You’ll get a JSON response containing comprehensive details about the annotations. Optionally, you can specify a list of input IDs for which you want to fetch their annotations. {Databricks1} ## Retrieve Annotations as a Dataframe You can retrieve detailed information about your annotations in a structured dataframe format. The dataframe includes key columns like `annotation_id`, `annotation`, `annotation_user_id`, `input_id`, `annotation_created_at` and `annotation_modified_at`. Note that the JSON response may include supplementary attributes, offering comprehensive insights beyond the specified columns in the dataframe. Optionally, you can specify a list of input IDs for which you want to fetch their annotations. {Databricks2} ### Retrieve Inputs With Annotations as a Dataframe You can obtain inputs along with their corresponding annotations in a structured dataframe. This capability allows for the simultaneous retrieval of input details and their associated annotations. The resulting dataframe consolidates information seamlessly from both the annotations and inputs dataframes, as outlined in the previously mentioned functions. {Databricks3} :::info You can get examples for integrating Clarifai with Databricks [here](https://github.com/Clarifai/clarifai-pyspark/tree/main/examples). ::: --- ## Fetch Data # Fetch Data **Seamlessly retrieve your data from Clarifai into Databricks** You may use Clarifai for tasks like image recognition and analysis. Then, you may want to bring the results or the processed data into Databricks for more in-depth exploration, analysis, or integration with other data sources. Let’s illustrate how you can effortlessly transfer data from Clarifai into the Databricks environment. ## Prerequisites - Databricks notebook development environment - Get your PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get your Clarifai user ID - Get the ID of the Clarifai app where you want to fetch the data from - Get the ID of the dataset having the data within your app - Install the [Clarifai PySpark](https://github.com/Clarifai/clarifai-pyspark) package by running `pip install clarifai-pyspark ` - Install Protocol Buffers by running `pip install protobuf==4.24.2 `. It’s a cross-platform, serialization protocol that describes the structure of the data to be sent :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Databricks1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_fetch_data_JSON.py"; import Databricks2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_fetch_data_dataframe.py"; import Databricks3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_fetch_data_volume.py"; ## Retrieve Data Files in JSON Format You can retrieve detailed information about the input data in your Clarifai app’s dataset. You’ll get a JSON response containing comprehensive details about the dataset files. Ensure you use the `input_type` parameter for targeted retrieval based on the data file types. You can specify the desired type, such as "image", "video", "audio", or "text", to obtain specific details relevant to that file type. {Databricks1} ## Retrieve Data Files as a Dataframe You can retrieve detailed information about your data files in a structured dataframe format. The dataframe includes key columns like `input_id`, `image_url/text_url`, `image_info/text_info`, `input_created_at`, and `input_modified_at`. Ensure to specify the `input_type` parameter to tailor the results to a specific type, such as "image", or "text". Note that the JSON response may include additional attributes, offering comprehensive insights beyond the specified columns in the dataframe. {Databricks2} ## Retrieve Data Files to Databricks Volume You can effortlessly download image and text files from your Clarifai app’s dataset to your Databricks volume. You need to specify the path where the retrieved data will be stored in the volume and utilize the response obtained from the `list_inputs()` function as the parameter. {Databricks3} :::info You can get examples for integrating Clarifai with Databricks [here](https://github.com/Clarifai/clarifai-pyspark/tree/main/examples). ::: --- ## Upload Data # Upload Data **Seamlessly upload your data from Databricks into Clarifai** You can ingest datasets from Databricks into your Clarifai environment. This allows you to easily take advantage of Clarifai's AI capabilities to analyze and extract insights from your data, without having to manually move it between the two platforms. Once your data has been uploaded to Clarifai, you can leverage Clarifai’s AI capabilities to get the most out of your data. For example, you can use Clarifai to identify and classify [objects and scenes](https://docs.clarifai.com/portal-guide/annotate/create-get-update-delete/) in images. Let’s illustrate how you can seamlessly transfer data from a Databricks [volume](https://docs.databricks.com/en/sql/language-manual/sql-ref-volumes.html) to Clarifai. ## Prerequisites - Databricks notebook development environment. Also, ensure your Databricks workspace is enabled to work with [Unity Catalog](https://docs.databricks.com/en/data-governance/unity-catalog/get-started.html) - Get the path URL of the Databricks location having your data - Get your PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get your Clarifai user ID - Get the ID of the Clarifai app where you want to upload the data - Get the ID of a dataset within your app - Install the [Clarifai PySpark](https://github.com/Clarifai/clarifai-pyspark) package by running `pip install clarifai-pyspark ` - Install Protocol Buffers by running `pip install protobuf==4.24.2 `. It’s a cross-platform, serialization protocol that describes the structure of the data to be sent :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Databricks1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_upload_from_volume_folder.py"; import Databricks2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_upload_from_csv.py"; import Databricks3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_upload_from_delta_table.py"; import Databricks4 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_upload_from_dataframe.py"; import Databricks5 from "!!raw-loader!../../../code_snippets/api-guide/integrations/databricks_upload_with_dataloader.py"; ## Upload Data From a Volume Folder You can upload images or text files stored in a Databricks volume to your Clarifai application. It’s important to ensure that the folder exclusively contains either images or text files. {Databricks1} ## Upload Data From CSV You can upload data from a CSV file stored in a Databricks volume to your Clarifai application. The CSV file must include two essential columns: `inputid` and `input`. You can also include additional supported columns such as `concepts`, `metadata`, and `geopoints`. The `input` column within the CSV is a versatile field, capable of accommodating either a file URL or path, or it can contain raw text. {Databricks2} ## Upload From Delta Table You can upload data from a delta table in a Databricks volume to your Clarifai application. The table must include two essential columns: `inputid` and `input`. You can also include additional supported columns such as `concepts`, `metadata`, and `geopoints`. The `input` column within the table is a versatile field, capable of accommodating either a file URL or path, or it can contain raw text. {Databricks3} ## Upload From Dataframe You can upload data from a PySpark dataframe in a Databricks volume to your Clarifai application. The dataframe must include two essential columns: `inputid` and `input`. You can also include additional supported columns such as `concepts`, `metadata`, and `geopoints`. The `input` column within the table is a versatile field, capable of accommodating either a file URL or path, or it can contain raw text. {Databricks4} ## Upload With Custom Dataloader You can utilize the custom data loader option if your dataset is stored in an alternative format or necessitates preprocessing. This grants you the flexibility to provide a specialized data loader tailored to your specific requirements. For reference, you can explore a variety of data loader examples [here](https://github.com/Clarifai/examples/tree/main/datasets/upload). Ensure that the necessary files and folders for the dataloader are stored in Databricks volume storage to facilitate seamless integration and accessibility. {Databricks5} :::info You can get examples for integrating Clarifai with Databricks [here](https://github.com/Clarifai/clarifai-pyspark/tree/main/examples). ::: --- ## DeepEval # DeepEval **Evaluate your LLM applications using DeepEval and Clarifai** [DeepEval](https://docs.confident-ai.com/) is an open-source evaluation framework for LLM applications. It provides a rich set of metrics to evaluate hallucination, answer relevancy, contextual precision, faithfulness, toxicity, and other qualities that matter in production LLM systems. By integrating Clarifai with DeepEval, you can use Clarifai-hosted models as both the models being evaluated and as judge models that score evaluation metrics — giving you a fully self-contained evaluation pipeline on the Clarifai platform. ## Prerequisites Before getting started, make sure you have the following: - A [Clarifai account](https://clarifai.com/signup) with a valid [Personal Access Token (PAT)](https://docs.clarifai.com/control/authentication/pat) - Python 3.8 or later ## Installation You can install the following necessary packages: - `deepeval` — The DeepEval framework, which provides evaluation metrics and test case utilities. - `openai` — The OpenAI client library used for the OpenAI-compatible connection to Clarifai. This is the combined command for installing them: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import DeepEvalWrapper from "!!raw-loader!../../code_snippets/python-sdk/model-train-eval/deepeval_wrapper.py"; import DeepEvalEval from "!!raw-loader!../../code_snippets/python-sdk/model-train-eval/deepeval_eval.py"; import DeepEvalOutput from "!!raw-loader!../../code_snippets/python-sdk/model-train-eval/outputs/deepeval_output.txt"; pip install deepeval openai ## Create a Clarifai LLM Wrapper DeepEval requires a custom model class that conforms to its `DeepEvalBaseLLM` interface. The following wrapper connects to any Clarifai-hosted model via the [OpenAI-compatible endpoint](https://docs.clarifai.com/compute/inference/open-ai): {DeepEvalWrapper} :::tip You can use any model from the [Clarifai Community](https://clarifai.com/explore) as the judge model by changing the `model_url` parameter to any publicly accessible Clarifai model URL. ::: ## Run an Evaluation Once the wrapper is in place, you can define evaluation metrics and run them against test cases. The example below uses `GEval` — a flexible, criteria-based metric — to evaluate the factual correctness of a model's response: {DeepEvalEval} Example Output {DeepEvalOutput} The `metric.score` is a float between 0 and 1, and `metric.reason` provides a natural-language explanation of the score produced by the judge model. ## Supported Metrics The following DeepEval metrics are compatible with Clarifai models as the judge: | Metric | What It Measures | |---|---| | **Answer Relevancy** | How relevant the generated answer is to the input question | | **Faithfulness** | Whether the response is grounded in the provided context without hallucinating | | **Contextual Precision** | Whether the retrieved context ranks relevant items higher than irrelevant ones | | **Contextual Recall** | How much of the expected output is supported by the retrieved context | | **Toxicity** | Presence of harmful, offensive, or inappropriate content in responses | | **Bias** | Whether the response reflects unfair bias toward a group or viewpoint | | **GEval** | A customizable metric scored against user-defined criteria and evaluation steps | :::info For a complete walkthrough including knowledge bases and RAG evaluation, visit the [Clarifai Examples Repository](https://github.com/Clarifai/examples/tree/main/Integrations/DeepEval). ::: --- ## Integrating Clarifai With Embedchain # Integrating Clarifai With Embedchain **Learn how to use Clarifai models in Embedchain** [Embedchain](https://embedchain.ai/) is an open-source framework designed to facilitate the integration of Large Language Models (LLMs) with custom knowledge bases. This framework allows users to create bots that can answer questions based on the content added to them. It aims to make the process of embedding data and using it with LLMs straightforward and efficient. By integrating it with Clarifai, users can now use LLMs and embedders from the Clarifai Platform for their tasks. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Using Clarifai LLMs And Models In Embedchain import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeInit from "!!raw-loader!../../../code_snippets/python-sdk/embedchain/init.py"; import CodeDI from "!!raw-loader!../../../code_snippets/python-sdk/embedchain/di.py"; import CodeQuery from "!!raw-loader!../../../code_snippets/python-sdk/embedchain/query.py"; import CodeOutputDI from "!!raw-loader!../../../code_snippets/python-sdk/embedchain/outputs/di.txt"; import CodeOutputQuery from "!!raw-loader!../../../code_snippets/python-sdk/embedchain/outputs/query.txt"; # Using Clarifai LLMs And Models In Embedchain **Using Clarifai models in Embedchain** In this example, we will explore ways to create a RAG agent with LLMs and embedders from Clarifai using Embedchain. ## Prerequisites * Setting up the Clarifai Python SDK along with PAT. Refer to the installation and configuration with the PAT token [here](https://docs.clarifai.com/python-sdk/sdk-overview/). :::note Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) ::: * Install the required packages. ``` !pip install embedchain[clarifai] ``` ## Initialisation Embedchain allows users to create personalised AI apps using the `App` method. It also allows users to create and manage instances of bots (applications) that leverage custom knowledge bases for answering queries. You can also add data from different sources such as text, documents, websites, images, and videos into your app. Click [here](https://docs.embedchain.ai/api-reference/app/overview) to learn more about apps in embedchain. {CodeInit} This code initializes an application instance from the EmbedChain framework with specific configurations for a large language model (LLM) and an embedder, both sourced from Clarifai. The `App.from_config` method is used to set up this instance. The LLM configuration specifies the use of the `mistral-7B-Instruct` model hosted by Clarifai, with additional parameters such as a temperature setting of 0.5 to control response randomness and a maximum token limit of 1000 for generated outputs. The embedder configuration uses the `text-embedding-ada` model from Clarifai for embedding text data. ## Data Ingestion In EmbedChain you can add data and its embedding from various sources into the application's knowledge base, making it accessible for querying by the bots. This process involves extracting content from different data types, converting it into embeddings using specified models, and storing these embeddings in the knowledge base. For our example, we are going to use an Image URL. Click [here](https://docs.embedchain.ai/api-reference/app/add) to learn more about data ingestion in embedchain. :::info By default, embedchain uses `chromadb` as vectorstore for your app. ::: {CodeDI} Output {CodeOutputDI} ## Query Users can now run queries on the data they ingested using the `app.query()` method. Refer to [this](https://docs.embedchain.ai/api-reference/app/query) page to know more about `app.query()`. {CodeQuery} Output {CodeOutputQuery} --- ## Third-Party Integrations # Third-Party Integrations **Seamlessly integrate Clarifai with your favorite tools and frameworks** Clarifai offers integrations with a variety of popular third-party tools and frameworks, enabling you to leverage Clarifai's powerful AI capabilities within your existing workflows and applications. These integrations help you build more sophisticated AI solutions by combining Clarifai's comprehensive platform with specialized tools for data processing, language models, and more. ## Available Integrations Clarifai integrates with the following platforms and frameworks: ### Language Model Frameworks - **[LangChain](./langchain/README.mdx)** - Build applications with large language models using Clarifai's LLMs, embeddings, and vector store capabilities. - **[LlamaIndex](./llamaindex/README.mdx)** - Use Clarifai models for LLM inference and text embeddings within LlamaIndex applications. - **[DSPy](./DSPy/README.mdx)** - Program with language models using Clarifai's LLM capabilities and vector search within the DSPy framework. - **[Embedchain](./embedchain/README.mdx)** - Create AI-powered bots with custom knowledge bases using Clarifai's LLMs and embedding models. ### Data Processing & ETL - **[Unstructured.io](./unstructured/README.mdx)** - Transform and ingest unstructured data into Clarifai's vector database for RAG applications. - **[Databricks](./databricks/README.mdx)** - Combine Clarifai's AI capabilities with Databricks for large-scale data processing and analytics. ### Agent Frameworks - **[OpenAI Agents SDK](https://docs.clarifai.com/compute/agents/build#openai)** - Build agentic AI apps with the OpenAI Agents SDK using Clarifai-hosted models via LiteLLM. - **[CrewAI](https://docs.clarifai.com/compute/agents/build#crewai)** - Build collaborative AI agent teams that work together to accomplish complex tasks using Clarifai's LLM models. - **[Google ADK](https://docs.clarifai.com/compute/agents/build/#google-adk)** - Create sophisticated AI agents with Google's Agent Development Kit powered by Clarifai models. - **[Vercel](https://docs.clarifai.com/compute/agents/build/#vercel)** - Use the Vercel AI SDK, a TypeScript toolkit for working with OpenAI-compatible APIs, to build agentic AI apps that make use of Clarifai models. - **[Agno](https://docs.clarifai.com/compute/agents/build/#agno)** - Build intelligent autonomous agents with advanced capabilities using Clarifai's diverse model library. ### Evaluation & Testing - **[DeepEval](https://docs.clarifai.com/create/models/evaluate/deepeval)** - Evaluate and test your LLM applications using Clarifai models with the DeepEval framework. ## Why Integrate with Clarifai? Integrating Clarifai into your workflow provides several key benefits: - **Production-Ready AI Platform**: Access a comprehensive platform covering the full AI lifecycle from data exploration to inference. - **Multiple Capabilities in One Place**: Use LLMs, embeddings, and vector storage within a single, unified platform. - **Easy Integration**: Simple SDK-based integration with popular frameworks and tools. - **Scalability**: Built to handle production-scale workloads efficiently. - **Flexibility**: Choose from a wide variety of models and customize them for your specific use case. ## Getting Started Each integration guide provides detailed instructions on: - Setting up the integration - Authentication and configuration - Common use cases and examples - Best practices for production deployment Select an integration from the list above to get started with your preferred framework or tool. --- ## Integrating Clarifai With LangChain # Integrating Clarifai With LangChain **Unlock the full potential of large language models** [LangChain](https://langchain.com) is an open-source framework that simplifies the creation of applications using large language models (LLMs). It is designed to help you in crafting cutting-edge applications that seamlessly blend the prowess of LLMs with diverse computational resources and knowledge bases. Clarifai offers an all-encompassing platform that covers the complete AI lifecycle—from data exploration to inferencing. Notably, Clarifai offers support for LLMs, embeddings, and a vector store within a single, production-scale platform. LLMs are a transformative technology. However, relying on LLMs in isolation, while impressive, often falls short of unleashing their full potential. Their true power shines when they are combined with other sources of computation and knowledge. You can take advantage of LangChain’s ability to streamline the complex processes involved in creating LLM applications and Clarifai’s rich AI infrastructure to take your natural language-based applications to unprecedented heights. [Integrating Clarifai with LangChain](https://www.clarifai.com/blog/linking-up-clarifai-with-langchain-integration) is an excellent choice to operationalize your LangChain implementations and advance the frontiers of what’s possible with LLMs. Your integrations might involve building prompt templates, text output processors, creating chains within tasks, developing agents, retrievers, and many more. :::info You can learn more about integrating Clarifai with LangChain [here](https://python.langchain.com/docs/integrations/providers/clarifai). ::: import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## LLM Models # LLM Models **Use LangChain to interact with Clarifai LLMs** Let’s illustrate how you can use LangChain to interact with Clarifai LLMs (large language models) and complete various tasks, such as text classification, sentiment analysis, text generation, text summarisation, question answering, and many more. ### Prerequisites - Python development environment - Get a PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get the ID of the user owning the model you want to use - Get the ID of the app where the model is found - Get the ID of the model you want to use. Large language models can be found [here](https://clarifai.com/explore/models?filterData=%5B%7B%22field%22%3A%22use_cases%22%2C%22value%22%3A%5B%22llm%22%5D%7D%5D&page=1&perPage=24) - Install the [Clarifai Python SDK](https://docs.clarifai.com/python-sdk/sdk-overview) by running `pip install clarifai` - Install LangChain by running `pip install langchain` :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Langchain1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/langchain_1.py"; import Langchain2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/langchain_2.py"; import OutputExample1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/outputexample_1.txt"; import OutputExample2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/outputexample_2.txt"; Here is an example of how to use a Clarifai model and LangChain for a question answering task. {Langchain1} Output Example {OutputExample1} :::info You can explore the [LangChain documentation](https://api.python.langchain.com/en/latest/llms/langchain.llms.clarifai.Clarifai.html#langchain.llms.clarifai.Clarifai) to learn more on how to use the framework to interact with Clarifai’s LLMs. ::: ## Prompt template Prompt templates are pre-defined recipes for generating prompts for large language models. With a prompt template, you can provide instructions that guide a large language model in understanding the context of the input so that it can generate relevant and coherent output. LangChain provides the necessary tooling that lets you create and work with [prompt templates](https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/). Here is an example of how you can use a prompt template for a question answering task. {Langchain2} Output Example {OutputExample2} --- ## Text Embeddings # Text Embeddings **Use Clarifai and LangChain to create text embeddings** Embeddings create a vector representation of textual content. This is beneficial because it implies we can conceptualize text within a vector space, and facilitate tasks such as semantic search where we look for pieces of text that exhibit the highest similarity within that vector space. Let’s illustrate how you can use LangChain to interact with Clarifai models and create text embeddings. ### Prerequisites - Python development environment - Get a PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get the ID of the user owning the model you want to use - Get the ID of the app where the model is found - Get the ID of the model you want to use. Text embedding models can be found [here](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22text-embedder%22%5D%7D%5D) - Install the [Clarifai Python SDK](https://docs.clarifai.com/python-sdk/sdk-overview) by running `pip install clarifai` - Install LangChain by running `pip install langchain` :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Langchain3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/langchain_3.py"; import OutputExample3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/outputexample_3.txt"; Here is how you can create text embeddings. {Langchain3} Output Example {OutputExample3} :::info You can explore the [LangChain documentation](https://python.langchain.com/docs/integrations/text_embedding/clarifai) to learn more on how to use the framework with Clarifai for text embedding tasks. ::: --- ## Vector Store # Vector Store **Use Clarifai and LangChain to create a vector store and perform searches** Clarifai offers a powerful, built-in [vector database](https://www.clarifai.com/blog/using-clarifais-native-vector-database) within its AI platform. It is designed to store and retrieve large amounts of unstructured data quickly and accurately. The Clarifai platform uses an embedding model to automatically index any piece of data uploaded to it. It creates high-dimensional vectors that are kept in the vector store, from where you can efficiently search for and retrieve data based on these vectors. Let’s illustrate how you can use Clarifai and LangChain to [retrieve text data](https://docs.clarifai.com/portal-guide/psearch/text-search) based on their content and semantic similarity. ### Prerequisites - Python development environment - Get a PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get your Clarifai’s user ID - Get the ID of the text app where the text data will be uploaded. _Ensure your text app has the appropriate base workflow for indexing your text documents, such as the Language-Understanding workflow_ - Install the [Clarifai Python SDK](https://docs.clarifai.com/python-sdk/sdk-overview) by running `pip install clarifai` - Install LangChain by running `pip install langchain` :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Langchain4 from "!!raw-loader!../../../code_snippets/api-guide/integrations/langchain_4.py"; import Langchain5 from "!!raw-loader!../../../code_snippets/api-guide/integrations/langchain_5.py"; import Langchain6 from "!!raw-loader!../../../code_snippets/api-guide/integrations/langchain_6.py"; import OutputExample4 from "!!raw-loader!../../../code_snippets/api-guide/integrations/outputexample_4.txt"; import OutputExample5 from "!!raw-loader!../../../code_snippets/api-guide/integrations/outputexample_5.txt"; import OutputExample6 from "!!raw-loader!../../../code_snippets/api-guide/integrations/outputexample_6.txt"; ## Similarity Search on Uploaded Texts You can create a Clarifai vector store from a list of texts. You can upload each text with its respective metadata to a Clarifai application. You can then use your Clarifai application to perform a semantic search and find contextually relevant texts. Here is an example. {Langchain4} Output Example {OutputExample4} You can also leverage metadata filters to enhance the filtering capabilities within an app. ```python # This one will limit the similarity query to only the texts that have key of "source" matching value of "book 1" book1_similar_docs = clarifai_vector_db.similarity_search("I would love to see you", filter={"source": "book 1"}) # You can also use lists in the input's metadata and then select things that match an item in the list. This is useful for categories like below: book_category_similar_docs = clarifai_vector_db.similarity_search("I would love to see you", filter={"category": ["books"]}) ``` ## Similarity Search on Uploaded Web Documents You can create a Clarifai vector store from a list of web documents. You can upload each document with its respective metadata to a Clarifai application. You can then use your Clarifai application to perform a semantic search and find contextually relevant documents. Here is an example. :::note We’ll use the [`unstructured` library](https://python.langchain.com/docs/integrations/providers/unstructured) to extract clean text and prepare the raw source documents for downstream tasks. You can install it by running `pip install unstructured`. ::: {Langchain5} Output Example {OutputExample5} ## Similarity Search on Existing Applications Most users are likely to have already added data to their Clarifai applications before engaging with LangChain. In this example, we will leverage the existing data within an application to perform a semantic search and find relevant documents. {Langchain6} Output Example {OutputExample6} :::info You can explore the [LangChain documentation](https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.clarifai.Clarifai.html#langchain.vectorstores.clarifai.Clarifai) to learn more on how to use the framework with the Clarifai vector store. ::: --- ## Integrating Clarifai With LlamaIndex # Integrating Clarifai With LlamaIndex **Unlock the full potential of large language models** [LlamaIndex](https://www.llamaindex.ai/) is an open-source framework designed to help you unleash the power of large language models (LLMs) over your data. Instead of using LLMs in isolation, you can unlock their full potential as a cutting-edge technology by combining them with other sources of computation and knowledge. LLMs are often pre-trained on publicly available data, but not on your own private or specific data. Therefore, they may not perform well on a specific problem you’re trying to solve. LlamaIndex lets you augment your LLM applications with your private or domain-specific data and boost their accuracy. You can use LlamaIndex for various use cases, including: - Ingesting, structuring, and accessing external data to use with LLM applications. - Building an embeddings query interface that accepts any input prompt and leverages your data to provide knowledge-augmented responses. - Storing and indexing your data for various purposes, including integrating with downstream vector search and database services. Integrating Clarifai with LlamaIndex offers an exceptional opportunity to tap into Clarifai’s rich AI infrastructure and build accurate and powerful LLM applications. Through this integration, you can use LlamaIndex to interact with Clarifai LLM models for a wide range of tasks, such as getting answers over unstructured data and building data augmented chatbots. You can also create text embeddings that can facilitate tasks such as semantic search. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## LLM Models # LLM Models **Use LlamaIndex to interact with Clarifai LLMs** Let’s illustrate how you can use LlamaIndex to interact with Clarifai LLMs (large language models) and complete various tasks, such as text classification, sentiment analysis, text generation, text summarisation, question answering, and many more. ## Prerequisites - Python development environment - Get a PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get the URL of the model you want to use. Large language models can be found [here](https://clarifai.com/explore/models?filterData=%5B%7B%22field%22%3A%22use_cases%22%2C%22value%22%3A%5B%22llm%22%5D%7D%5D&page=1&perPage=24) - Alternatively, get the ID of the user owning the model you want to use, the ID of the app where the model is found, and the name of the model - Install LlamaIndex and [Clarifai Python SDK](https://docs.clarifai.com/python-sdk/sdk-overview) by running `pip install llama-index-llms-clarifai` :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: :::note Clarifai models can be referenced either through their full URL or by using the combination of user ID, app ID, and model name. If the model version is not specified, the latest version will be used by default. ::: ## Text Completion Here is an example of how to use a Clarifai LLM and LlamaIndex for a text completion task. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import LlamaIndex1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/llamaindex_1.py"; import LlamaIndex2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/llamaindex_2.py"; import OutputExample1 from "!!raw-loader!../../../code_snippets/api-guide/integrations/llamaindex_output_1.txt"; import OutputExample2 from "!!raw-loader!../../../code_snippets/api-guide/integrations/llamaindex_output_2.txt"; {LlamaIndex1} Output Example {OutputExample1} ## Chat Here is an example of how to use a Clarifai LLM and LlamaIndex for a chatting task. {LlamaIndex2} Output Example {OutputExample2} :::info You can explore the [LlamaIndex documentation](https://docs.llamaindex.ai/en/stable/examples/llm/clarifai.html) to learn more on how to use the framework to interact with Clarifai’s LLMs. ::: --- ## Text Embeddings # Text Embeddings **Use Clarifai and LlamaIndex to create text embeddings** Embeddings create a vector representation of textual content. This is beneficial because it implies we can conceptualize text within a vector space, and facilitate tasks such as semantic search where we look for pieces of text that exhibit the highest similarity within that vector space. Let’s illustrate how you can use LlamaIndex to interact with Clarifai models and create text embeddings. ## Prerequisites - Python development environment - Get a PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](https://clarifai.com/settings/security) section - Get the URL of the model you want to use. Text embedding models can be found [here](https://clarifai.com/explore/models?page=1&perPage=24&filterData=%5B%7B%22field%22%3A%22model_type_id%22%2C%22value%22%3A%5B%22text-embedder%22%5D%7D%5D) - Alternatively, get the ID of the user owning the model you want to use, the ID of the app where the model is found, and the name of the model - Install LlamaIndex and [Clarifai Python SDK](https://docs.clarifai.com/python-sdk/sdk-overview) by running `pip install llama-index-embeddings-clarifai` :::info You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ::: :::note Clarifai models can be referenced either through their full URL or by using the combination of user ID, app ID, and model name. If the model version is not specified, the latest version will be used by default. ::: Here is how you can create text embeddings. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import LlamaIndex3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/llamaindex_3.py"; import OutputExample3 from "!!raw-loader!../../../code_snippets/api-guide/integrations/llamaindex_output_3.txt"; {LlamaIndex3} Output Example {OutputExample3} :::info You can explore the [LlamaIndex documentation](https://docs.llamaindex.ai/en/stable/examples/embeddings/clarifai.html) to learn more on how to use the framework with Clarifai for text embedding tasks. ::: --- ## Integrating Unstructred.io with Clarifai # Integrating Unstructred.io with Clarifai **Learn how to use Unstructured along with Clarifai SDKs** [Unstructured.io](https://unstructured.io) tackles a common challenge in large language models (LLMs): transforming raw data into a format LLMs can understand and process effectively. It is a data ETL (Extract, Transform, Load) platform designed for LLMs. Unstructured.io is a data ingestion powerhouse as it can ingest various data formats. It acts as a central hub, regardless of where your data resides ie. local machines, cloud storage, or databases. It also provides connectors and libraries to simplify data access from various sources. You can configure it to retrieve the data your LLMs need seamlessly. Unstructured.io goes beyond simply moving data. It offers tools and libraries to transform raw data into a structured format suitable for LLM consumption. By integrating Clarifai into Unstructred.io, users gain access to Clarifai's capabilities ie. it can ingest and store unstructured data into Clarifai vectorDB and call LLMs from the Clarifai platform. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Data Ingestion Using Unstructured.io # Data Ingestion Using Unstructured.io **Learn about the data ingestion process in Unstructured.io** Unstructured.io provides a powerful platform for handling the ingestion of unstructured data. Central to this process are the source and destination connectors, which facilitate the movement of data from its origin to a storage or processing system. ## Source Connectors Source connectors are designed to interface with various unstructured data sources, allowing you to seamlessly ingest data into the Clarifai platform. Click [here](https://docs.unstructured.io/api-reference/ingest/source-connectors/overview) to learn more about source connectors. Below is an example of using S3 as a source connector. ```python # Importing necessary modules from the 'os' library import os # Importing necessary configurations and classes from unstructured.ingest.connector.fsspec.s3 from unstructured.ingest.connector.fsspec.s3 import S3AccessConfig, SimpleS3Config # Importing configuration classes from unstructured.ingest.interfaces from unstructured.ingest.interfaces import ( PartitionConfig, ProcessorConfig, ChunkingConfig, ReadConfig, ) # Importing the S3Runner class from unstructured.ingest.runner from unstructured.ingest.runner import S3Runner # Importing necessary configurations and classes from unstructured.ingest.connector.clarifai from unstructured.ingest.connector.clarifai import ( ClarifaiAccessConfig, ClarifaiWriteConfig, SimpleClarifaiConfig, ) # Importing base writer and ClarifaiWriter from unstructured.ingest.runner.writers.clarifai from unstructured.ingest.runner.writers.base_writer import Writer from unstructured.ingest.runner.writers.clarifai import ( ClarifaiWriter, ) if __name__ == "__main__": # Creating an instance of ClarifaiWriter writer = clarifai_writer() # Creating an instance of S3Runner with various configurations runner = S3Runner( processor_config=ProcessorConfig( verbose=True, # Enable verbose output output_dir="s3-output-local", # Directory to store output locally num_processes=2, # Number of processes to use ), read_config=ReadConfig(), # Configuration for reading data partition_config=PartitionConfig(), # Configuration for partitioning data connector_config=SimpleS3Config( access_config=S3AccessConfig( key=access_key, # S3 access key secret=secret_access, # S3 secret access key ), remote_url="s3 URL", # URL of the S3 bucket ), writer=writer, # Writer to use for output writer_kwargs={}, # Additional arguments for the writer ) # Running the S3Runner runner.run() ``` ## Destination Connector Destination connectors enable the processed data to be loaded into various storage or processing systems where it can be further analyzed, queried, or used in applications. Click [here](https://docs.unstructured.io/api-reference/ingest/destination-connector/overview) to learn more about destination connectors. Below is an example of using Clarifai as a destination connector. ```python import os # Importing necessary configurations and classes from unstructured.ingest.connector.clarifai from unstructured.ingest.connector.clarifai import ( ClarifaiAccessConfig, ClarifaiWriteConfig, SimpleClarifaiConfig, ) # Importing local configuration class from unstructured.ingest.connector.local from unstructured.ingest.connector.local import SimpleLocalConfig # Importing configuration classes from unstructured.ingest.interfaces from unstructured.ingest.interfaces import ( ChunkingConfig, PartitionConfig, ProcessorConfig, ReadConfig, ) # Importing the LocalRunner class from unstructured.ingest.runner from unstructured.ingest.runner import LocalRunner # Importing base writer and ClarifaiWriter from unstructured.ingest.runner.writers.clarifai from unstructured.ingest.runner.writers.base_writer import Writer from unstructured.ingest.runner.writers.clarifai import ( ClarifaiWriter, ) # Function to create and return a ClarifaiWriter instance def get_writer() -> Writer: return ClarifaiWriter( connector_config=SimpleClarifaiConfig( access_config=ClarifaiAccessConfig(api_key=os.getenv("CLARIFAI_PAT_KEY")), # Access config using environment variables app_id=os.getenv("CLARIFAI_APP_ID"), # Clarifai app ID from environment variables user_id=os.getenv("CLARIFAI_USER_ID"), # Clarifai user ID from environment variables ), write_config=ClarifaiWriteConfig(), # Write config for Clarifai ) if __name__ == "__main__": # Creating a writer instance using the get_writer function writer = get_writer() # Creating an instance of LocalRunner with various configurations runner = LocalRunner( processor_config=ProcessorConfig( verbose=True, # Enable verbose output output_dir="local-output-to-clarifai-app", # Directory to store output locally num_processes=2, # Number of processes to use ), connector_config=SimpleLocalConfig( input_path="example-docs/book-war-and-peace-1225p.txt", # Path to the input file ), read_config=ReadConfig(), # Configuration for reading data partition_config=PartitionConfig(), # Configuration for partitioning data chunking_config=ChunkingConfig(chunk_elements=True), # Configuration for chunking data writer=writer, # Writer to use for output writer_kwargs={}, # Additional arguments for the writer ) # Running the LocalRunner runner.run() ``` ## Ingestion Configuration Ingestion configuration in Unstructured.io allows you to customize the data ingestion process to suit your specific requirements. Key configuration aspects include: * **Batch Size**: Specifies the number of records to process in each batch, balancing throughput and performance. * **Schedule**: Defines the frequency of data ingestion, such as hourly, daily, or real-time ingestion. * **Error Handling**: Configures how the system handles errors during ingestion, including retry policies and logging mechanisms. Click [here](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/overview) to learn more about ingestion configuration. --- ## Chat With Dropbox Using Unstructured.io import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeInit1 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/dropbox/init1.py"; import CodeInit2 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/dropbox/init2.py"; import CodeDI from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/dropbox/di.py"; import CodeChat from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/dropbox/chat.py"; import CodeOutputDI from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/dropbox/outputs/di.txt"; import CodeOutputChat from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/dropbox/outputs/chat.txt"; # Chat With Dropbox Using Unstructured.io **Learn how to chat with data from Dropbox** [Dropbox](https://www.dropbox.com/official-teams-page?_tk=paid_sem_goog_biz_b&_camp=1018334849&_kw=dropbox|e&_ad=666300362351||c&gad_source=1&gclid=CjwKCAjw1K-zBhBIEiwAWeCOFw939qj3aTgFRDagZZw0ugD4sLAMy1AeGE_ReqKsEyd5dgMn96KX8RoCzJ4QAvD_BwE) is a cloud storage service that allows users to store, sync, and share files online. It provides seamless file synchronization across devices, enabling access to updated files from anywhere with an internet connection. Users can easily share files and folders with others, even if they don't have a Dropbox account. Using Dropbox as a source connector you can now ingest data to a Clarifai app and then leverage all of Clarifai platform's abilities. In this example, we are going to chat with our data ingested into the Clarifai app using RAG. ## Prerequisites * Setting up the Clarifai Python SDK along with PAT. Refer to the installation and configuration with the PAT token [here](https://docs.clarifai.com/python-sdk/sdk-overview/). :::note Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) ::: ```python import os os.environ['CLARIFAI_PAT'] ="YOUR_PAT" ``` * Install the required packages. ``` ! pip install "unstructured[clarifai]" ! pip install "unstructured[dropbox]" ``` ## Initialization First, let us setup the data we are going to ingest into the app. The data we are going to use will be stored in Dropbox. To access the data using Unstructured.io, we have to provide Dropbox access token. :::info Setup dropbox access token. Refer [this](https://developers.dropbox.com/oauth-guide) page for instructions. ::: ```python DROPBOX_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" ``` After setting up the access tokens, let’s import some necessary libraries. {CodeInit1} Next, we will have to write a function to set up the ingestion configurations required to upload the data into our app in the Clarifai platform. {CodeInit2} ## Data Ingestion In data ingestion, there are two important concepts Source Connector and Destination Connector. For our use case the Source Connector will fetch the data from Dropbox and the Destination Connector will send the transformed data to the Clarifai app. Click [here](https://unstructured-io.github.io/unstructured/ingest/index.html) to learn more about Ingestion. {CodeDI} Output {CodeOutputDI} ## Chat In the final step, we are going to perform information retrieval using RAG based on the data we ingested from Dropbox to the Clarifai app using Unstructured.io. You can use a workflow with a RAG prompter for initialising RAG. After successfully creating a workflow, you can get the URL from the Clarifai portal. After creating the rag object using workflow URL you can start retrieving text from the data we ingested using Unstructured.io. {CodeChat} Output {CodeOutputChat} --- ## Use RAG With Unstructured.io import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeInit1 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/s3/init1.py"; import CodeInit2 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/s3/init2.py"; import CodeDI from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/s3/di.py"; import CodeChat from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/s3/chat.py"; import CodeOutputDI from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/s3/outputs/di.txt"; import CodeOutputChat from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/s3/outputs/chat.txt"; # Use RAG With Unstructured.io **Learn how to use RAG with Unstructured.io** RAG systems are a powerful combination of two techniques: information retrieval and text generation. When you ask a question, the system searches for related details (context) and then leverages that context to generate a response using text generation methods. Using Unstructured.io we can transform the data into a format suitable for RAG. The Clarifai platform provides various LLMs that can be used for text generation inside RAG. Hence by integrating Clarifai with Unstructured.io, you can build RAG applications with ease. ## Prerequisites * Setting up the Clarifai Python SDK along with PAT. Refer to the installation and configuration with the PAT token [here](https://docs.clarifai.com/python-sdk/sdk-overview/). :::note Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) ::: ```python import os os.environ['CLARIFAI_PAT'] ="YOUR_PAT" ``` * Install the required packages. ``` ! pip install "unstructured[clarifai]" #make sure the unstructured version is 0.13 or above ! pip install "unstructured[s3]" #since our source is S3 ! pip install httpx ``` ## Initialization The first part of creating an app based on Unstructured.io is to set up the data we are going to ingest into the app. The data we are going to use will be stored in the s3 bucket. To access the data using Unstructured.io, we have to provide some AWS access keys. :::info Click [here](https://medium.com/@shamnad.p.s/how-to-create-an-s3-bucket-and-aws-access-key-id-and-secret-access-key-for-accessing-it-5653b6e54337) to learn how to get the s3 access keys. ::: ```python access_key='YOUR_S3_ACCESS_KEYS' secret_access='YOUR_S3_SECRET_ACCESS_KEYS' ``` After setting up the access keys for the s3 bucket, let’s import some necessary libraries. {CodeInit1} Next, we have to write a function that will configure the target Clarifai app where the ingested documents will be loaded, {CodeInit2} ## Data Ingestion In data ingestion, there are two important concepts Source Connector and Destination Connector. For our use case the Source Connector will fetch the data from the S3 bucket and the Destination Connector will send the transformed data to the Clarifai app. Click [here](https://unstructured-io.github.io/unstructured/ingest/index.html) to learn more about Ingestion. {CodeDI} Output {CodeOutputDI} ## Chat In the final step, we are going to perform information retrieval using RAG based on the data we ingested from S3 to the Clarifai app using Unstructured.io. You can use a workflow with a RAG prompter for initialising RAG. After successfully creating a workflow, you can get the URL from the Clarifai portal. After creating the rag object using workflow URL you can start retrieving text from the data we ingested using Unstructured.io. {CodeChat} Output {CodeOutputChat} --- ## Ingest Email Messages From Salesforce Using Unstructured.io import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeInit1 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/init1.py"; import CodeInit2 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/init2.py"; import CodeDI from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/di.py"; import CodeChat1 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/chat1.py"; import CodeChat2 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/chat2.py"; import CodeOutputDI from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/outputs/di.txt"; import CodeOutputChat1 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/outputs/chat1.txt"; import CodeOutputChat2 from "!!raw-loader!../../../code_snippets/python-sdk/unstructured/salesforce/outputs/chat2.txt"; # Ingest Email Messages From Salesforce Using Unstructured.io **Learn how to ingest email messages from Salesforce** [Salesforce](https://www.salesforce.com/in/) is a cloud-based customer relationship management (CRM) platform that assists businesses in managing their relationships and interactions with customers and prospects. It offers a comprehensive suite of tools including Sales Cloud, Service Cloud, Marketing Cloud, Commerce Cloud, and Analytics Cloud, among others, to streamline various business processes. In Salesforce, email messages play a crucial role in customer communication and interaction tracking. The platform integrates seamlessly with popular email services like Outlook and Gmail through Salesforce Inbox, allowing users to log emails directly into Salesforce. Additionally, Salesforce supports the creation of email templates, enabling consistent and personalized communication using merge fields from Salesforce records. In this tutorial, we are going to ingest the email messages from Salesforce to the Clarifai app and then use LLM to classify and summarize emails. ## Prerequisites * Setting up the Clarifai Python SDK along with PAT. Refer to the installation and configuration with the PAT token [here](https://docs.clarifai.com/python-sdk/sdk-overview/). :::note Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) ::: ```python import os os.environ['CLARIFAI_PAT'] ="YOUR_PAT" ``` * Install the required packages. ``` ! pip install "unstructured[clarifai]" ! pip install "unstructured[salesforce]" ``` * Setup JWT authorization in Salesforce. Refer [this](https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_auth_key_and_cert.htm) page for instructions. ## Initialization The data we are going to ingest into our app is the body of the email messages in Salesforce. Since we have already set the JWT authorization for Salesforce let’s import some required libraries, {CodeInit1} Next we will have to write a function to setup the ingestion configurations required to upload the data into our app in the Clarifai platform. {CodeInit2} ## Data Ingestion In data ingestion, there are two important concepts, Source Connector and Destination Connector. For our use case the Source Connector will fetch the data from Salesforce and the Destination Connector will send the transformed data to the Clarifai app. Click [here](https://unstructured-io.github.io/unstructured/ingest/index.html) to learn more about Ingestion. :::info In `SalesforceAccessConfig`, the category is set as `EmailMessage` for this task. Unstructured.io also supports other fields like `Account`, `Case`, `Campaign`, and `Lead` . ::: {CodeDI} Output {CodeOutputDI} ## Chat In the final step, we are going to chat with the data using RAG. You can use a workflow with a RAG prompter for initialising RAG. After successfully creating a workflow, you can get the URL from the Clarifai portal. After creating the rag object using workflow URL you can start retrieving text from the data we ingested using Unstructured.io. {CodeChat1} Output {CodeOutputChat1} {CodeChat2} Output {CodeOutputChat2} --- ## Welcome import Head from '@docusaurus/Head'; import CardGrid from '@site/src/components/CardGrid'; import { startHereCards, exploreCards, popularTopics } from '@site/src/data/homepageCards'; # Welcome **Orchestrate and customize AI workloads on any infrastructure, at the scale you need** Clarifai lets you build, deploy, and manage AI across any infrastructure — from shared cloud to your own on-premise or edge environments. It works with models from any source — Clarifai-built, third-party, or self-hosted. Teams deploy on Clarifai's shared or dedicated cloud, within their own VPC, on-premise, or at the edge — all managed from a single control plane. Everything in Clarifai flows through three core areas: - **Compute** – Orchestrate AI models with cost-efficient deployment, scaling, and optimization across environments. - **Control & Governance** – Manage AI resources, monitor performance, and track costs through a unified control plane. - **Create & Manage** – Customize AI workloads with tools for labelling, training, evaluating, and deploying models. ## See Clarifai in Action ## Start Here ## Explore the Platform ## Popular Topics --- ## Clarifai Releases and Update Types # Clarifai Releases and Update Types **Learn about our release types and monthly updates** We continually enhance our platform with regular releases and updates. Understanding the types of updates and their frequency helps users make the most of the platform's capabilities. Whether you're seeking cutting-edge features, stability improvements, or security enhancements, staying informed about Clarifai's release cycles ensures you can leverage the latest advancements in AI. ## Release Types Clarifai release editions refer to the different stages or types of releases that our software product can go through — from development to deployment. These options help us manage the introduction of new features, bug fixes, and updates while ensuring quality and stability. We can issue a release at any of the following phases, based on the degree of maturity: :::caution Attention - Non-GA features are provided primarily for evaluation and feedback purposes and should NEVER be used for production workloads. - We could issue a release as an Enterprise-only feature, meaning it will be accessible exclusively to users on the Enterprise [pricing plan](https://www.clarifai.com/pricing). ::: | Release Type | Description | Purpose | Who can use? | For Production/SLA? | Interface Stability | Support | |--------------|----------------|-----------------|--------------------------|---------------------------|---------------------------|---------------------------| | Private Preview | Available to a select group of design partners, often by invitation. It's of Alpha quality. | Gather feedback from a controlled user base before a wider release. Public documentation may not be available. | Invite only | No | No | Engineering team | | Public Preview | Available to all users, but still considered pre-release. The goal of public preview releases is to provide users with an opportunity to test new features. Public releases may be feature-complete but still have areas of instability. | Gather feedback from a broader user base, ensure compatibility, and identify any integration issues prior to general availability. Documentation is usually available. | Everyone | No | Yes | Support team | | General Availability (GA) | The official release of the product to everyone. It is considered stable, feature-complete, and ready for production use. | The final product is available for use by any user based on their pricing plan. | Everyone | Yes | Yes | Support team | ## Monthly Update Types Clarifai releases major platform updates monthly on the first Tuesday. The update versioning follows a **Version X.Y.Z** format: - **X**: Indicates the release year since Clarifai’s founding in 2014. For example, updates in 2024 follow a 10.Y.Z format because 2024 is ten years since 2014. - **Y**: Represents the month of the major release. - **Z**: Denotes the minor release within that month's major release. **Note**: - Clarifai supports Generally Available (GA) releases of active products for up to two years. - Eligible code-fixes and hot-fixes are provided via new minor releases (Z) on top of the latest major release branch, for up to two releases from the most current major release. - A major release is identified by a change in the first (X) or second (Y) digit in the versioning nomenclature. The following table details expectations for each update type: | Update Type | Description| Can include new GA features? | Can include breaking API changes?|Can include new preview features?|Can include new security and stability fixes?| |--------------|----------------|-----------------|--------------------------|---------------------------|---------------------------| |Major Releases|Significant updates that may include new features, major changes, improvements, bug fixes, and breaking changes | Yes | Yes | Yes | Yes | |Minor Releases| Minor updates to customer impacting bugs, security issues, or other critical problems| No | No | Yes | Yes | --- ## Release 10.0 # Release 10.0 **Release Date:** January 9th, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Text Generation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Introduced a UI for [text generation](https://docs.clarifai.com/portal-guide/model/model-types/text-to-text) models|The Model-Viewer screen of text generation models now has a revamped UI that lets you effortlessly generate or convert text based on a given text input prompt. For third-party wrapped models, like those provided by OpenAI, you can choose to utilize their API keys as an option, in addition to using the default Clarifai keys.Optionally, you can enrich the model's understanding by providing a system prompt, also known as context.Optionally, inference parameters are available for configuration. They are hidden by default. The revamped UI provides users with versatile options to manage the generated output. You can regenerate, copy, and share output. | |![new-feature](/img/new_feature.jpg)|Added more training templates for text-to-text generative tasks|You can now use Llama2 7/13B and Mistral templates as a foundation for fine-tuning text-to-text models.There are also additional configuration options, allowing for more nuanced control over the training process. Notably, the inclusion of quantization parameters via GPTQ enhances the fine-tuning process.| ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Introduced the RAG-Prompter operator model| You can now use the RAG-Prompter operator (agent system operator) to perform [retrieval-augmented generation](https://www.clarifai.com/blog/what-is-rag-retrieval-augmented-generation) (RAG) tasks. | |![new-feature](/img/new_feature.jpg)|Improved the process of making predictions on the Model-Viewer screen|To make a prediction using a model, navigate to the model’s viewer screen and click the **Try your own input** button. A modal will pop up, providing a convenient interface for adding input data and examining predictions. The modal now provides you with three distinct options for making predictions: **Batch Predict on App Inputs**—allows you to select an app and a dataset. Subsequently, you’ll be redirected to the Input-Viewer screen with the default mode set to **Predict**, allowing you to see the predictions on inputs based on your selections.**Try Uploading an Input**—allows you to add an input and see its predictions without leaving the Model-Viewer screen.**Add Public Preview Examples**—allows model owners to add public preview examples.| |![improvement](/img/improvement.jpg)|Improved the UI/UX of the models’ [evaluation leaderboard](https://docs.clarifai.com/product-updates/changelog/release98/#evaluation-leaderboard)| Replaced the *Context based classifier* wording with **Transfer learn**.Added a dataset filter functionality that only lists datasets that were successfully evaluated.A full URL is now displayed when hovering over the table cells.Replaced "-" of empty table cells in training and evaluation dataset columns with `"{{appName}}-all-app-inputs"` | |![improvement](/img/improvement.jpg) | Added support for inference settings |All models have undergone updates to incorporate new versions that now support inference hyperparameters like temperature, top_k, etc. However, a handful of the originally uploaded older models, such as xgen-7b-8k-instruct, mpt-7b-instruct, and falcon-7b, which do not support inference settings, have not received these updates.| | ![bug](/img/bug.jpg)|Fixed an issue where detection and segmentation models generated the wrong code| Previously, if you clicked the “Use Model” button, the code generated by detection models, such as [General Detection](https://clarifai.com/clarifai/main/models/general-image-detection) and [Face Detection](https://clarifai.com/clarifai/main/models/face-detection), and segmentation models, such as [General Segmentation](https://clarifai.com/clarifai/main/models/image-general-segmentation), could only work with classification models. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with the model version dropdown|Previously, the dropdown on the model's page displayed the latest model version, irrespective of whether that version was marked as trained and ready for use in the versions table. We’ve fixed the issue, and the dropdown now accurately reflects the trained and ready-to-use model versions. | |![bug](/img/bug.jpg)|Fixed an issue that prevented model training with default settings|Previously, attempts to train certain models, particularly visual classifiers and visual detectors, using the recommended default settings resulted in numerous errors. We fixed the issue. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published several new, ground-breaking models|Wrapped [Fuyu-8B](https://clarifai.com/adept/fuyu/models/fuyu-8b), an open-source, simplified multimodal architecture with a decoder-only transformer, supporting arbitrary image resolutions, and excelling in diverse applications, including question answering and complex visual understanding.Wrapped [Cybertron 7B v2]( https://clarifai.com/fblgit/una-cybertron/models/una-cybertron-7b-v2), a MistralAI-based language model (llm) excelling in mathematics, logic, and reasoning. It consistently ranks #1 in its category on the HF LeaderBoard, enhanced by the innovative Unified Neural Alignment (UNA) technique. Wrapped [Llama Guard](https://clarifai.com/meta/Llama-2/models/llamaGuard-7b), a content moderation, llm-based input-output safeguard, excelling in classifying safety risks in Human-AI conversations and outperforming other models on diverse benchmarks.Wrapped [StripedHyena-Nous-7B](https://clarifai.com/togethercomputer/stripedHyena/models/stripedHyena-Nous-7B), an innovative hybrid chat llm, featuring multi-head attention and gated convolutions, outperforms Transformers in long-context summarization with notable efficiency improvements in training and inference. Wrapped [Imagen 2](https://clarifai.com/gcp/generate/models/imagen-2), a cutting-edge text-to-image llm, offering high-quality, multilingual image generation with advanced features, including improved text rendering, logo generation, and safety measures.Wrapped [Gemini Pro](https://clarifai.com/gcp/generate/models/gemini-pro), a state-of-the-art, llm designed for diverse tasks, showcasing advanced reasoning capabilities and superior performance across diverse benchmarks.Wrapped [Mixtral 8x7B](https://clarifai.com/mistralai/completion/models/mixtral-8x7B-Instruct-v0_1), a high-quality, Sparse Mixture-of-Experts (SMoE) llm model, excelling in efficiency, multilingual support, and competitive performance across diverse benchmarks. Wrapped [OpenChat-3.5](https://clarifai.com/openchat/openchat/models/openchat-3_5-1210), a versatile 7B LLM, fine-tuned with C-RLFT, excelling in benchmarks with competitive scores, supporting diverse use cases from general chat to mathematical problem-solving. Wrapped [DiscoLM Mixtral 8x7b alpha](https://clarifai.com/discoresearch/discoLM/models/discoLM-mixtral-8x7b-v2), an experimental 8x7b MoE language model, based on Mistral AI's Mixtral 8x7b architecture, fine-tuned on diverse datasets. Wrapped [SOLAR-10.7B-Instruct](https://clarifai.com/upstage/solar/models/solar-10_7b-instruct), a powerful 10.7 billion-parameter LLM with a unique depth up-scaling architecture, excelling in single-turn conversation tasks through advanced instruction fine-tuning methods. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Introduced the [Databricks-Connect UI](https://clarifai.com/clarifai/Databricks-Connect-App/modules/Databricks-Connect) module for integrating Clarifai with Databricks|You can use the module to: Authenticate a Databricks connection and connect with its compute clusters.Export data and annotations from a Clarifai app into Databricks volume and table.Import data from Databricks volume into Clarifai app and dataset.Update annotation information within the chosen Delta table for the Clarifai app whenever annotations are getting updated.| ## Base Workflow |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Changed the default base workflow of a default first application|Previously, for new users who skipped the onboarding flow, a default first application was generated having "General" as the base workflow. We’ve replaced it with the "Universal" base workflow. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added support for [Secure Data Hosting](https://docs.clarifai.com/product-updates/upcoming-api-changes/secure-data-hosting/#what-is-secure-data-hosting) (SDH) | The SDK now supports the SDH feature for uploading and downloading user inputs. | |![improvement](/img/improvement.jpg)| Added vLLM template for model upload to the SDK|This additional template expands the range of available templates, providing users with a versatile toolset for seamless deployment of models within their SDK environments. | | ![bug](/img/bug.jpg) | Fixed an issue with the CocoDetection loader| Previously, the CocoDetection loader had a bug that caused it to convert bounding box coordinates from the xywh format (center coordinates, width, height) to the xyxy format (top-left, bottom-right coordinates) using the wrong divisor. Instead of dividing by the bounding box's height, it was incorrectly dividing by its width. We fixed it. | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Improved pagination handling during multiple input deletion in the Input-Manager| Previously, there was an issue of pagination fetch inconsistencies after deleting multiple inputs or annotations. Now, when you delete a selection of inputs or annotations, the pagination mechanism resets to ensure a more accurate and streamlined retrieval of pages. | |![bug](/img/bug.jpg)|Fixed an issue with performing visual search on the Input-Manager| Previously, if you performed a face similarity search in an app with "Face" as the base workflow, it could return no outputs. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to exit the **Upload Inputs** pop-up window|Previously, there was no way to exit the pop-up if there were no inputs to upload—users were forced to refresh the page. We fixed the issue. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added ability to view and edit previously submitted inputs while working on a task|We have added an input carousel to the labeler screen that allows users to go back and review inputs after they have been submitted. This functionality provides a convenient mechanism to revisit and edit previously submitted labeled inputs. | |![bug](/img/bug.jpg)|Fixed an issue with deleting a task on the Tasks page |Previously, deleting the first task in a list of multiple tasks did not work as expected. While the deletion request was processed, the row representing the deleted task remained visible, resulting in an unusual and weird state. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with using AI-Assist on the Labeler screen, where predictions were sometimes prone to becoming "stuck" | Previously, when labeling using the AI-Assist feature, and you tried to approve or reject an image by clicking the checkmark or the x button, an unintended consequence arose—multiple concept IDs were generated with each action. Subsequently, the bounding box annotations associated with these additional concept IDs persisted on the screen for the remaining images in the task, rendering them immovable. We fixed the issue. | ## Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Made enhancements to the App Settings page|Added a new collaborators table component for improved functionality.Improved the styling of icons in tables to enhance visual clarity and user experience.Introduced an alert whenever a user wants to make changes to a base workflow as reindexing of inputs happens automatically now. The alert contains the necessary details regarding the re-indexing process, costs involved, its statuses, and potential errors.| |![improvement](/img/improvement.jpg)|Enhanced the inputs count display on the App Overview page|The tooltip (`?`) now precisely indicates the available number of inputs in your app, presented in a comma-separated format for better readability, such as 4,567,890 instead of 4567890. The display now accommodates large numbers without wrapping issues. The suffix 'K' is only added to the count if the number exceeds 10,000. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Added “Last Updated” date in resources alterations| We’ve replaced "Date Created" with "Last Updated" in the sidebar of apps, models, workflows, and modules (these are called resources). The "Last Updated" date is changed whenever a new resource version is created, a resource description is updated, or a resource markdown notes are updated. | |![improvement](/img/improvement.jpg)|Added a "No Starred Resources" screen for models/apps empty state | We’ve introduced a dedicated screen that communicates the absence of any starred models or apps in the current filter when none is present. | |![improvement](/img/improvement.jpg)|Enhanced the resource overview page with larger images where possible| We now include rehosting of large versions of resource cover images alongside small ones. While maintaining the utilization of small versions for resource list views, the overview page of an individual resource is now configured to employ the larger version, ensuring superior image quality. Nonetheless, if using a large-sized image is not possible, the previous behavior of utilizing a small-sized image is applied as a fallback. | |![improvement](/img/improvement.jpg)|Enhanced image handling in listing view| Fixed an issue where cover images were not being correctly picked up in the listing view. Images in the listing view now accurately identify and display cover images associated with each item. | |![improvement](/img/improvement.jpg)|Enhanced search queries by including dashes between text and numbers| For instance, if the search query is "llama70b" or "gpt4," we also consider "llama-70-b" or "gpt-4" in the search results. This provides a more comprehensive search experience. | |![improvement](/img/improvement.jpg)|Revamped code snippets presentation|We’ve updated the code snippet theme to a darker and more visually appealing scheme. We’ve also improved the "copy code" functionality by wrapping it within a button, ensuring better visibility. | |![bug](/img/bug.jpg)|Fixed an issue with the "Back to Community" button | When attempting to access an unavailable resource in the Clarifai Community portal, users are directed to a page indicating the non-existence of the resource. And a button is provided that users can click and return to the portal. Previously, the button could not redirect users. Clicking the button now correctly redirects users back to the Community portal. | |![bug](/img/bug.jpg)|Fixed an issue that prevented deleting a cover image| You can now remove a cover image from any resource—apps, models, workflows, datasets, and modules—without any problems. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Allowed members of an organization to work with the Labeler Tasks functionality | The previous implementation of the Labeler Tasks functionality allowed users to add collaborators for working on tasks. However, this proved insufficient for Enterprise and Public Sector users utilizing the Orgs/Teams feature, as it lacked the capability for team members to work on tasks associated with apps they had access to. We now allow admins, org contributors, and team contributors with app access to work with Labeler Tasks. | ## On-Premise |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Disabled the "Please Verify Your Email" popup | We deactivated the popup, as all accounts within on-premises deployments are already being automatically verified. Furthermore, email does not exist for on-premises deployments. | --- ## Release 10.1 # Release 10.1 **Release Date:** February 6th, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) **[API-only]**|Introduced incremental training of model versions | You can now update existing models with new data without retraining from scratch. After training a model version, a checkpoint file is automatically saved. And you can initiate incremental training from that previously trained version checkpoint. Alternatively, you provide the URL of a checkpoint file from a supported 3rd party toolkit like HuggingFace or MMCV. | |![new-feature](/img/new_feature.jpg) **[API-only]** |Introduced ability to add inputs through cloud storage URLs|You can now provide URLs from cloud storage platforms such as S3, GCP, and Azure, accompanied by the requisite access credentials. This functionality simplifies the process of adding inputs to our platform, offering a more efficient alternative to the conventional method of utilizing PostInputs for individual inputs.| ## Evaluation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Enhanced the evaluation process for detector models|Enriched the metrics by introducing additional fields, namely "Total Predicted," "True Positives," "False Negatives," and "False Positives." These additional metrics provide a more comprehensive and detailed assessment of a detector performance.Previously, a multi-selector was used for the selection of an Intersection over Union (IoU). We replaced that confusing selection with a radio button format, emphasizing a single, mutually exclusive choice for IoU selection. We also made other minor UI/UX improvements to ensure consistency with the evaluation process for classification models.| ## Devtools Integrations |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Integrated Clarifai into DSPy|This integration, now part of the recently released DSPy version 2.1.7, empowers DSPy users to consume Clarifai's LLM models and utilize Clarifai's apps to retrieve Clarifai's vector search engine. Notably, Clarifai is the only provider enabling users to harness multiple LLM models. You can get started on how to use DSPy with Clarifai [here](https://github.com/stanfordnlp/dspy/tree/main/examples/integrations). | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Made improvements to LLM fine-tuning|Added support for CSV upload for streamlined data integration.Added more training templates to tailor the fine-tuning process to diverse use cases.Added advanced configuration options, including quantization parameters via GPTQ, which further empowers users to fine-tune models with heightened precision and efficiency.| |![improvement](/img/improvement.jpg)|Improved the Model-Viewer's version table|Cross-app evaluation is now supported in the model version tab to have a more cohesive experience with the Leaderboard. Users, and collaborators with access permissions, can also select datasets or dataset versions from org apps, ensuring a comprehensive evaluation across various contexts. This improvement allows users to view both training and evaluation data across different model versions in a centralized location, enhancing the overall version tracking experience. | |![improvement](/img/improvement.jpg)|Improved the management of model annotations and associated assets |Previously, when a model annotation was deleted, the corresponding model assets remained unaffected. If you now delete a model annotation, a simultaneous action would mark the associated model assets as deleted. This ensures that the deletion process is comprehensive, avoiding any lingering or orphaned assets. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published several new, ground-breaking models|Published [Phi-2]( https://clarifai.com/microsoft/text-generation/models/phi-2), a Clarifai-hosted, 2.7 billion-parameter large language model (LLM) achieving state-of-the-art performance in QA, chat, and code tasks. It is focused on high-quality training data and has demonstrated improved behavior in toxicity and bias. Wrapped [Deepgram Nova-2](https://clarifai.com/deepgram/transcribe/models/audio-trascription). It sets a new benchmark in speech-to-text with 30% lower error rates, and unmatched speed, making it the superior choice in automatic speech recognition. Wrapped [Deepgram Audio Summarization](https://clarifai.com/deepgram/transcribe/models/audio-summarization). It offers efficient and accurate summarization of audio content, automating call notes, meeting summaries, and podcast previews with superior transcription capabilities. Wrapped [Text-Embedding-3-Large](https://clarifai.com/openai/embed/models/text-embedding-3-large), a high-performance, flexible text embedding model with up to 3072 dimensions, outperforming its predecessor. Wrapped [Text-Embedding-3-Small](https://clarifai.com/openai/embed/models/text-embedding-3-small), a highly efficient, flexible model with improved performance over its predecessor, Text-Embedding-ADA-002, in various natural language processing tasks.Wrapped [CodeLlama-70b-Instruct](https://clarifai.com/meta/Llama-2/models/codeLlama-70b-Instruct), a state-of-the-art AI model specialized in code generation and understanding based on natural language instructions.Wrapped [CodeLlama-70b-Python](https://clarifai.com/meta/Llama-2/models/codeLlama-70b-Python), a state-of-the-art AI model specialized in Python code generation and understanding, excelling in accuracy and efficiency.| ## Mobile |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved the mobile version of the onboarding flow| Updated the "create an app" guided tour modal for mobile platforms.Made other improvements such as updating the "Add a Model" modal and the "Find a Pre-Trained model" modal for mobile platforms. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Integrated the RAG-Prompter operator model | You can now use the RAG-Prompter operator (agent system operator) in the SDK to perform [retrieval-augmented generation](https://www.clarifai.com/blog/what-is-rag-retrieval-augmented-generation) (RAG) tasks. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added ability to minimally review existing image mask annotations on the Input-Viewer|You can view your image mask annotations uploaded via the API.You can delete an entire image mask annotation on an inputYou can view the mask annotation items displayed on the Input-Viewer sidebar.| |![bug](/img/bug.jpg)|Fixed an issue with annotating polygons on the Input-Viewer|Previously, when zooming in, the polygon editing points and the associated tag would scale up, resulting in undesired effects. We fixed the issue, and both elements now adhere to a consistent 100% scale, irrespective of the zoom level. We also made these improvements: Polygon editing points (also known as polygon handles) now match the size of the polygon line itself, both in edit and drawing mode.The size of the polygon line is now adjusted to adhere to a fixed 100% scale, irrespective of the zoom level. This provides a clear and well-defined boundary for the polygon. The text box containing the concept name now remains at a 100% scale consistently, unaffected by the scale of the polygon or bounding box. This ensures readability and avoids text becoming too small or large at different magnifications. | |![bug](/img/bug.jpg)|Fixed an issue that caused unintended zooming of the browser window when using the zoom in / zoom out hotkeys (CMD + / CMD -) on the Input-Viewer screen|Previously, executing these hotkeys not only affected the canvas area but also triggered a zoom in / zoom out action on the entire browser window. We fixed the issue, and pressing them now exclusively zooms the canvas area while avoiding any impact on the overall browser window. | |![bug](/img/bug.jpg)|Fixed an issue with an undesired cursor pointer behavior in drawing mode, specifically in bounding box mode|Previously, if you selected the “select” tool and clicked the border of a bounding box, the cursor remained in the default state when moved inside the bounding box. This was confusing as it failed to indicate the ability to drag the bounding box. We fixed the issue, and the cursor now appropriately changes to a hand icon when hovering inside the bounding box.| ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg)|Fixed an issue with selecting a text in the smart search field during a visual search|Previously, if you provided an invalid text in the smart search field and clicked CTRL / CMD + A, it could undesirably select all inputs. We fixed the issue, and clicking CTRL / CMD + A when the search field is in a focused state successfully selects the intended text. | | ![bug](/img/bug.jpg)|Fixed an issue with an infinite cycle occurrence when performing a smart rank search within the Input-Manager |Previously, an infinite loading icon could appear when consecutively performing a smart rank search using the first listed input. This resulted in the backend not receiving the expected search request. We fixed the issue. | | ![bug](/img/bug.jpg) |Fixed an issue with the upload job monitor in the Input-Manager|Previously, there was an issue wherein the percentage displayed in the job monitor occasionally exceeded 100%. We fixed the issue, ensuring that the percentage displayed in the job monitor aligns precisely with the actual progress of the job, eliminating any misleading overestimations.| |![bug](/img/bug.jpg) |Fixed an issue that resulted in the closure of the inputs upload pop-up window | Previously, when attempting to upload inputs through the pop-up window and selecting a concept, users sometimes experienced an unexpected closure of the window. We fixed it. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) |Made minor enhancements to the Workflow builder UI|Rectified the alignment discrepancy in some left-side models to ensure uniform left alignment. Introduced an X or Close/Cancel button for improved user interaction and clarity. Ensured that users can easily straighten the line connecting two nodes.| |![bug](/img/bug.jpg)|Fixed an issue with adjusting concept thresholder node levels|Previously, when creating workflows, it was not possible to adjust the concept thresholder node in some cases. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with the [Demographics](https://clarifai.com/clarifai/main/workflows/Demographics) workflow | Previously, if you were logged in, the Demographics workflow page could crash, and failed to give predictions. The workflow now works as intended for logged-in users. | |![bug](/img/bug.jpg)|Fixed an issue where the RAG-Prompter model template failed to update in the workflow UI|Previously, when a custom template was created for the RAG-Prompter model and subsequently assigned in a workflow through the PostWorkflow API call, the API call indicated a successful update of the prompter model template. However, the workflow UI did not reflect this updated template. We fixed the issue, ensuring synchronization between the API call results and the workflow UI display. | ## Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Added ability to copy an app to an organization|Previously, in the **Copy / Duplicate App** modal, the dropdown for selecting users lacked an option for organizations. You can now select an organization directly from the dropdown list of potential destinations when copying or duplicating an app.| |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to perform face searches|Previously, attempting visual search or face search in an app with the base workflow set as “Face” resulted in an inability to retrieve inputs, yielding no search results. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with the selection of the “Text/Document” input type during the creation of a new app|The selection button is by default enabled with the “Image/Video” option chosen. Previously, changing it to the “Text/Document” option led to a malfunction, requiring users to cancel and retry the selection process. We fixed the issue, ensuring an error-free experience when opting for the “Text/Document” input type during app creation. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with editing a bounding box | Previously, there was an issue that hindered editing bounding box sizes or modifying concepts within the labeler screen. We fixed it. | |![bug](/img/bug.jpg) |Fixed an issue where inputs were incorrectly assigned to the task creator instead of the designated labeler(s) |Previously, when a user created a labeling task and assigned it to another labeler, the task creator sometimes observed the "Label" button being enabled for themselves, despite not being assigned as a labeler. This led to inadvertent assignment of inputs to the task creator, causing an obstruction for the assigned labeler in accessing the tasks. We fixed the issue. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved the search behavior within of the `use_cases` field|Previously, the `use_cases` field within the ListModels feature was configured as an AND search, unlike other fields such as `input_fields` and `output_fields`. We improved the `use_cases` attribute to operate with an OR logic, just like the other fields. This adjustment broadens the scope of search results, accommodating scenarios where models may be applicable to diverse use cases. | |![improvement](/img/improvement.jpg)|Changed the thumbnails for listing resources to use small versions of cover images|Previously, the thumbnails for listing resources used the large versions of cover images. We changed them to use the small versions—just like for other resources like Apps, Models, Workflows, Modules, and Datasets. We also made the change to the left sidebars. | |![improvement](/img/improvement.jpg)|Implemented a modification to facilitate a more user-friendly experience for non-logged-in users interacting with text-to-image models|Clicking the “Generate” button now triggers a login/sign-up pop-up modal. This guides users who are not currently logged in through the necessary authentication steps, ensuring a smoother transition into utilizing the model's functionality. | |![bug](/img/bug.jpg) |Fixed an issue where altering a user ID resulted in a disruption of model access| Previously, when a user modified their profile user ID, this change failed to reflect in model responses where the `user_id` was referenced. Consequently, the model responses retained the previous user ID, triggering a 'Resource had been moved' error. We fixed it. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) |Fixed an issue where a user could get added multiple times to the same organization|We implemented safeguards against the unintended duplication of users within an organization. Previously, if a user clicked the "Accept" button on the organization invitation page multiple times, they could be redundantly registered within the same organization. Consequently, the user interface exhibited multiple instances of the same organization. | |![bug](/img/bug.jpg)| Fixed an issue with selecting the focus organization on the left sidebar |Previously, attempting to switch the organizational focus in the left sidebar did not effectively update the corresponding organization details on the rest of the page. The details of the previous organization persisted, causing a misalignment. We fixed the issue, ensuring that changing the organization in the left sidebar now accurately populates the corresponding details throughout the rest of the page. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved the module installation process |The modal has been refined to use app IDs, eliminating reliance on deprecated app names. Previously, the pop-up modal for installing a module into an app retained the usage of deprecated app names. | |![improvement](/img/improvement.jpg)|Improved the relevance of the link to GitHub on the module page| Previously, there was a small GitHub button at the top of any module’s overview page. We relocated it to the right-hand side, aligning it with other metadata such as description, thereby improving its clarity as a clickable link. | --- ## Release 10.10 # Release 10.10 **Release Date:** November 7th, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | :::info Clarifai Product Roadmap We have a public roadmap that gives you a view of our upcoming plans and helps us gather your valuable feedback. You can explore the features we're considering, vote for the ones you want most, or submit new ideas. You can check it [here](https://portal.productboard.com/bd1rxfuyfbu6vqnmkva3mprx/tabs/1-under-consideration). ::: ## Compute Orchestration |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) **(Private Preview)** | Released Compute Orchestration | Clarifai’s new Compute Orchestration system provides you with an efficient way to deploy inference workloads on any compute, in any bare metal, or cloud environment. This feature is currently in [Private Preview](https://docs.clarifai.com/product-updates/changelog/release-types). To request access, please contact us or sign up [here](https://forms.gle/Cj9r8wgeYk5HQ67Y9). Learn more about it [here](https://docs.clarifai.com/portal-guide/compute-orchestration/). | ## PATs and API Keys​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Breaking Change** | Important changes to the use of PATs and API keys | This change will be rolled out this month in November. Learn more about it [here](https://docs.clarifai.com/product-updates/upcoming-api-changes/pat-api-keys/). Previously, you could use API keys to access any model, concept, or workflow owned by the app scoped to the API key, as well as those owned by the user `clarifai` in the application `main`. Now, accessing models or workflows owned by `clarifai` in the application `main` can only be done with a PAT tied to your account. | ## New Released Models​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Released new models | Released [Pixtral 12B](https://clarifai.com/mistralai/completion/models/pixtral-12b), a natively multimodal model excelling in multimodal reasoning, instruction following, and text benchmarks with a 12B parameter architecture supporting variable image sizes and long context inputs. Released [Granite-3.0-8B-Instruct](https://clarifai.com/ibm/granite/models/granite-3_0-8b-instruct), a versatile, enterprise-ready language model optimized for multilingual understanding, coding, and instruction-following across diverse tasks and constrained environments. Released [Granite-3.0-2B-Instruct SLM](https://clarifai.com/ibm/granite/models/granite-3_0-2b-instruct), a lightweight, multilingual, enterprise-ready language model optimized for instruction-following and code understanding, ideal for versatile applications under Apache 2.0. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Added a [Costs & Budget tab](https://docs.clarifai.com/portal-guide/control-center/costs-budget) for finances in the Control Center | Added a new tab to display users' financial data. It displays the costs of your billed operations, making it easier to track and manage expenses on the platform. | | ![improvement](/img/improvement.jpg) | Made other improvements | Restricted the 'Upgrade your plan' button to display only for users on the Community plan with personal accounts. Added comma delimiters to numbers in the thousands for increased readability, e.g., displaying 78,755,432 instead of 78755432. | |![bug](/img/bug.jpg) | Fixed various bugs in the Control Center| Fixed an issue where filtering data for the last 6 months using the calendar widget triggered a toast notification error: “Failed to retrieve model versions.” Fixed an issue that led to this error: “Unable to find app while trying to locate embed model version from workflow.” Fixed an issue where changing the chart type would reset any applied filters. Selected filters now remain intact when switching chart types. Fixed an issue on the Report Details page where clicking the 'Back' button would always redirect to the Overview section. It now correctly navigates to the user's previous page. Fixed an issue where the selected period in a chart would start a day earlier than the specified date. A displayed chart now accurately reflects the selected start date. | ## Data Ingestion |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Released text-based, ready-to-use data pre-processing pipelines | These pipelines streamline data ingestion into our platform. With our Data Utils Library and Python SDK, users can now easily upload, transform, and chunk text files. These pipelines support various document types, including PDFs and DOCs, for seamless ingestion. | ## UI Modules​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |**Public Preview** | Released a UI module for evaluating OCR (Optical Character Recognition) workflows | You can use the module to evaluate the performance of workflows for OCR tasks directly within our platform. Learn more about it [here](https://docs.clarifai.com/portal-guide/modules/ocr-evaluation-module). | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added granular scopes for organization's team members | Users can now assign granular access levels for applications within an organization. When adding team members to an application, you can define specific scopes to limit their access, rather than granting full access to the app. This role-based access control improves security and ensures that users stay focused on their designated responsibilities. Initial roles include `Labeler Worker`, `Labeler Lead`, and `Model Trainer`. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made various improvements to labeling tasks | Enabled all assigned reviewers to review each input before the labeling task is marked as complete. Enabled collaborators to set task priority levels (High, Medium, Low) for each task. | ## Smart Text Search |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made various improvements to the [Smart Text Search](https://docs.clarifai.com/portal-guide/psearch/text-search) feature | These improvements include replacing the deprecated Language-Understanding workflow with a new text workflow, preventing billing actions when a base workflow is forcefully reindexed, and additional improvements for optimized performance and functionality. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made several improvements to the Python SDK | Added a functionality for merging datasets. Fixed an issue where duplicate results were returned in the SDK’s visual search. Added input uploaders to the client repository. Fixed a rotation issue where exported dataset images appeared rotated by 90 degrees. Added tests for CRUD operations on Compute Orchestration for deployment of models. Introduced `request-id-prefix` header to SDK requests for enhanced monitoring. Implemented CLI support for model uploads. Fixed model service issues related to model uploads. Added Python version badges to `README.md`. Temporarily removed stream tests until the Stream API is fixed. Removed unnecessary prefixes from concept IDs added via SDK. Upgraded `llama-index-core` library version for security enhancements. Included metadata in exported dataset annotation files. Upgraded `clarifai-grpc` to version 10.9.11. Enhanced user experience for model uploads and fixed runner tests. Resolved bugs affecting model uploads. Fixed `download_checkpoints` and adjusted functionality for running models locally. Improved handling for missing `huggingface_hub` package. Implemented script to test and run a runner's model locally. Enhanced model upload experience for computer vision models. Increased test coverage for Dataloaders and Evaluation modules in the SDK. | ## Platform Bugs |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue with Google OAuth sign-up flow | Previously, when signing up for a Clarifai account using Google OAuth, the modal to complete profile registration did not appear automatically. Users had to manually refresh the sign-up modal to proceed. We fixed the issue, and the modal refreshes automatically as intended. | |![bug](/img/bug.jpg) | Fixed issues with the account selector | To select an account, click your profile icon in the upper-right corner and choose the desired account from the dropdown list. Previously, in each listed account, the user's ID was hidden beneath the user's name, which caused the name to appear lower than the avatar icon. We fixed it, and the names and IDs now align correctly. Also, in some cases, the account selector dropdown would abruptly shift to the right and get cut off whenever a user switched accounts. We fixed it, and the selector now remains properly aligned during account changes. | | ![bug](/img/bug.jpg) | Fixed an issue with the collaborator feature | Previously, organization accounts could be added as collaborators to an app. We fixed it, and they can no longer be added as collaborators. | --- ## Release 10.11 # Release 10.11 **Release Date:** December 3rd, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | :::info Clarifai Product Roadmap We have a public roadmap that gives you a view of our upcoming plans and helps us gather your valuable feedback. You can explore the features we're considering, vote for the ones you want most, or submit new ideas. You can check it [here](https://portal.productboard.com/bd1rxfuyfbu6vqnmkva3mprx/tabs/1-under-consideration). ::: ## Compute Orchestration |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) **(Public Preview)** | Released Compute Orchestration in [Public Preview](https://docs.clarifai.com/product-updates/changelog/release-types) | Clarifai’s new Compute Orchestration capabilities provides you with an efficient way to deploy inference workloads on any compute, in any bare metal, or cloud environment. Learn more about it [here](https://docs.clarifai.com/portal-guide/compute-orchestration/). To request access, please contact us [here](https://www.clarifai.com/explore/contact-us-co). | ## Concepts |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved concept capabilities with additional features | You can now create extensive concept hierarchies, accommodating thousands of concepts across up to 10 levels. Concepts can be named using period delimiters (e.g., `parent.child.subchild.subsubchild.subsubsubchild`), such as `vehicle.tracked_vehicle.tank.sherman.M4A3`, for intuitive navigation through the hierarchical layers. Added a dedicated page that allows you to list, view, and manage concepts, including adding metadata like descriptions, images, and URLs. You can expand each concept to view its hierarchy, displaying all child concepts under each parent in an organized structure. | ## Single Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made various improvements to the [single Input-Viewer](https://docs.clarifai.com/glossary/general-ai#input-viewer) screen | Added ability for users to rotate images and adjust their brightness, sharpness, contrast, and saturation. Improved the autocomplete functionality in the concept search field, making the displayed suggestions to be more relevant when users search for concepts. Relocated the "Add new concept" option to the bottom of the autocomplete table, which appears when users use the search field to annotate inputs with concepts. You can also now use keyboard shortcuts to move from one option to another. Added more keyboard shortcuts for labeling inputs in the Input-Viewer. Improved the visual design of classification and object regions in the Input-Viewer, enhancing clarity and user experience. | ## Control Center​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | Implemented locale-based date formatting for charts. This ensures the user's browser locale is used to adjust the date format automatically based on their location (e.g., US or EU date formats). Previously, selecting a range like the last 7 days included the current day, even though data for the current date was often unavailable. This has been fixed so that the current date is now excluded when selecting the last "x" days. We’ve updated Org access for Control Center pages. Previously, only Org Admins could access any of these pages. Now, Org Admins, Org Contributors, and Org Users can access the Overview and Usage pages, while the Costs page remains accessible only to Org Admins. | |![bug](/img/bug.jpg) | Fixed bugs on the Control Center | Fixed an issue where pinning or unpinning charts on the Usage and Operations tab did not correctly update their visibility on the Overview tab as intended. Fixed an issue where selecting the 6-month option in the Date Range selector caused an error. Previously, when a date range was selected and you navigated to a different tab or section, returning to the Control Center page would reset the date range to the default "Last 7 Days". We’ve fixed this issue, ensuring the selected date range persists. Fixed an issue with header styling in the Control Center. | ## Agent System Operators |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced a new agent system operator (node type) for character detection | We developed a new operator that transforms Visual Detector model outputs into an OCR-compatible format. This operator assigns detected concept names as text for each region, enabling seamless integration with the Text Aggregator, particularly for foreign language character detection. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to the Python SDK | Fixed a bug in runners selection when using `Deployment`. Fixed an ID issue in the multimodal loader during the deletion of failed inputs. Added functionality to list inputs in the `Dataset` class. Added the ability to delete annotations in the `Input` class. Added a Dockerfile template based on new base images by parsing `requirements`. Added validation for the `base_url` parameter. Added CLI support for Compute Orchestration resources, including Compute Cluster, Nodepool, and Deployment. | ## Platform Bugs |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue with member invitations in [Organizations](https://docs.clarifai.com/portal-guide/clarifai-organizations/)| We’ve fixed an issue where it was possible to invite someone to an organization they were already a member of. Invitations now correctly prevent duplicate membership. | | ![bug](/img/bug.jpg) | Fixed an issue with sorting apps | We’ve fixed an issue where the A-Z sorting functionality in the app selector was not working as expected. Apps now sort correctly in alphabetical order. | | ![bug](/img/bug.jpg) | Fixed an issue with app duplication | We’ve fixed an issue where inputs were not copied when duplicating an application. Now, duplicating an application will correctly copy all associated inputs. | | ![bug](/img/bug.jpg)|Fixed an issue with spacing below headers| We’ve fixed an issue where content directly below headers in some places appeared cramped due to insufficient spacing. Now, there is appropriate padding and spacing below headers, ensuring improved visual clarity and consistent alignment throughout the UI. | --- ## Release 10.2 # Release 10.2 **Release Date:** March 5th, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Evaluation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) **[Developer Preview]**|Introduced a module for evaluating large language models (LLMs)|You can use the module to evaluate the performance of LLMs against standardized benchmarks alongside custom criteria to gain deep insights into their strengths and weaknesses. Here are some of its key features: Evaluate across 100+ tasks covering diverse use cases like RAG, classification, casual chat, content summarization, and more. Each use case provides the flexibility to choose from relevant evaluation classes like Helpfulness, Relevance, Accuracy, Depth, and Creativity. You can further enhance the customization by assigning user-defined weights to each class. Define weights on each evaluation class to create custom weighted scoring functions. This lets you measure business-specific metrics and store them for consistent use. For example, for RAG-related evaluation, you may want to give zero weight to Creativity and more weights for Accuracy, Helpfulness, and Relevance. Save the best performing prompt-model combinations as a workflow with a single click for future reference.| ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published new models _(Clarifai-hosted models are the ones we host within our Clarifai Cloud. Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys.)_| Wrapped [Google Gemini Pro Vision](https://clarifai.com/gcp/generate/models/gemini-pro-vision), which was created from the ground up to be multimodal (text, images, videos) and scale across a wide range of tasks.Wrapped [Claude 3 Opus](https://clarifai.com/anthropic/completion/models/claude-3-opus), a state-of-the-art, multimodal language model (LLM) with superior performance in reasoning, math, coding, and multilingual understanding.Wrapped [Claude 3 Sonnet](https://clarifai.com/anthropic/completion/models/claude-3-sonnet), a multimodal LLM balancing skills and speed, excelling in reasoning, multilingual tasks, and visual interpretation.Wrapped [Qwen1.5-72B-Chat](https://clarifai.com/qwen/qwenLM/models/qwen1_5-72B-chat), which leads in language understanding, generation, and alignment, setting new standards in conversational AI and multilingual capabilities, outperforming GPT-4, GPT-3.5, Mixtral-8x7B, and Llama2-70B on many benchmarks.Wrapped [DeepSeek-Coder-33B-Instruct](https://clarifai.com/deepseek-ai/code-generation/models/deepseek-coder-33b-instruct), a SOTA 33 billion parameter code generation model, fine-tuned on 2 billion tokens of instruction data, offering superior performance in code completion and infilling tasks across more than 80 programming languages. Clarifai-hosted [Gemma-2b-it](https://clarifai.com/gcp/generate/models/gemma-2b-it), a part of Google DeepMind's lightweight, Gemma family LLM, offering exceptional AI performance on diverse tasks by leveraging a training dataset of 6 trillion tokens, focusing on safety and responsible output. Clarifai-hosted [Gemma-7b-it](https://clarifai.com/gcp/generate/models/gemma-7b-it), an instruction fine-tuned LLM, lightweight, open model from Google DeepMind that offers state-of-the-art performance for natural language processing tasks, trained on a diverse dataset with rigorous safety and bias mitigation measures. Clarifai-hosted [DeciLM-7B-Instruct](https://clarifai.com/deci/decilm/models/deciLM-7B-instruct), a state-of-the-art, efficient, and highly accurate 7 billion parameter LLM, setting new standards in AI text generation. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Made enhancements to the Python SDK | Updated and cleaned the requirements.txt file for the SDK.Fixed an issue where a failed training job led to a bug when loading a model in the Clarifai-Python client library, and concepts were replicated when their IDs did not match. | |![improvement](/img/improvement.jpg)|Made enhancements to the RAG (Retrieval Augmented Generation) feature|Enhanced the RAG SDK's `upload()` function to accept the `dataset_id` parameter. Enabled custom workflow names to be specified in the RAG SDK's `setup()` function. Fixed scope errors related to the `user` and `now_ts` variables in the RAG SDK by correcting their definition placement, which was previously inside an `if` statement.Added support for chunk sequence numbers in the metadata when uploading chunked documents via the RAG SDK. | ## Community UI |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added a notification for remaining time for free deep training | Added a notification at the upper-right corner of the **Select a model type** page about the number of hours left for deep training your models for free. | |![new-feature](/img/new_feature.jpg)|Added feedback form | Added feedback form links to the header and listings pages of models, workflows, and modules. This enables registered users to provide general feedback or request a specific model.| |![new-feature](/img/new_feature.jpg)|Added a display of inference pricing per request| The model and workflow pages now display the price per request for both logged-in and non-logged-in users.| |![improvement](/img/improvement.jpg)|Implemented progressive image loading for images|Progressive image loading displays low-resolution versions of images initially, gradually replacing them with higher-resolution versions as they become available. It solves page load issues and preserves image sharpness.| |![improvement](/img/improvement.jpg)| Replaced spaces with dashes in IDs|When updating User, App, or any other resource IDs, spaces will be replaced with dashes.| |![improvement](/img/improvement.jpg)|Updated links |Updated the text and link for the Slack community in the navbar's info popover to 'Join our Discord Channel.' Similarly, updated the link similar to it at the bottom of the landing page to direct to Discord.Removed the "Where's Legacy Portal?" text. | |![improvement](/img/improvement.jpg)|Display name in PAT toast notification|We've updated the account security page to display a PAT name instead of PAT characters in the toast notification.| |![bug](/img/bug.jpg)|Fixed an issue with the **Create an App** modal| Previously, errors displayed in the App ID section could overlap, making them difficult to read. We fixed it. | ## Mobile |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved the mobile onboarding flow|Made minor updates to mobile onboarding.| |![improvement](/img/improvement.jpg)|Improved sidebar appearance| Enhanced sidebar appearance when folded in mobile view. | ## Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Added an option to edit the scopes of a collaborator|You can now edit and customize the scopes associated with a collaborator’s role on the App Settings page. | ## Leaderboard |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with inconsistency in the Leaderboard sorting|Previously, there was an issue where the leaderboard filtering was not functioning correctly under certain situations. We fixed it. | |![bug](/img/bug.jpg)|Fixed an issue with the Leaderboard where duplicate items caused confusion and usability concerns|Previously, duplicate items could appear in the dataset sidebar, leading to uncertainty when selecting them and confusion with the hover state. We’ve rectified this to enhance clarity and user experience.| ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed issues with uploading bulk inputs| We fixed an issue where it was not possible to upload multiple files with the same name.We fixed an issue where it was not possible to reupload failed or canceled uploads. | |![bug](/img/bug.jpg)|Improvements made to text preview on text tiles in the Input-Manager|Previously, when a single string (a single word) was excessively long, it remained unbroken and did not wrap onto separate lines. Additionally, adjusting the zoom level of the grid did not affect the amount of text displayed. We fixed the issues, and the texts are now displayed as desired. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with renaming a concept in the Input-Viewer| Previously, if you added a concept and renamed the concept, you could not use the highlighted input box to “filter by concept” for the new relation that was created. We fixed the issue, and you can now filter by concept using the original name as well as the new relation that was created for it. | |![bug](/img/bug.jpg)|Fixed an issue with the gearbox icon in the Input-Viewer|Previously, opening the gearbox popover, selecting a model or workflow, and then closing the popover resulted in the gearbox icon remaining gray, indicating that nothing was selected inside the popover. We fixed the issue, and now the gearbox icon turns blue when an item is selected within its popover.| |![bug](/img/bug.jpg)| Fixed an issue with entering the same name of a concept|For example, previously, if you generated predictions from classifications — let’s say one of the predicted concepts is “sky.” If you edited “sky” to “wall,” a new relation was created. However, if you edited the concept name again to “sky,” a new relation was erroneously created. We fixed the issue, and now since the relation and the originally predicted concept are the same, we remove the “wall” relation. | |![bug](/img/bug.jpg)|Fixed an issue where deleting generated annotations was possible in the Input-Viewer|Previously, users could delete AI-assisted annotation suggestions made using the General-detection workflow, regardless of whether they were their own or not. We fixed the issue, and now users can only delete their own annotations.| |![bug](/img/bug.jpg)|Fixed an issue that triggered an 'unsaved' warning when navigating away from the Input-Viewer|This issue caused confusion as the browser mistakenly indicated unsaved changes when attempting to reload the page. We fixed it. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with AI-assisted bounding box annotations in the Labeling Tasks screen|Previously, when multiple AI-assisted annotations were present for the same concept in an input, clicking the check mark to accept one of them could result in the name of the selected annotation changing unexpectedly. We fixed it. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with the cancel button functionality when deleting an organization| Previously, the cancel button on the dialog for deleting an organization did not function as expected. We fixed the issue. Additionally, we have added the organization name to the dialog, ensuring users know precisely what they are deleting. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Enabled deletion of associated model assets when removing a model annotation|Now, when deleting a model annotation, the associated model assets are also marked as deleted.| ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved model selection|Made improvements to the model selection drop-down list on the workflow builder.| |![bug](/img/bug.jpg)|Fixed an issue with creating workflows | Previously, users experienced an issue where the input node would move unexpectedly when models were dropped into the graph during the initial workflow creation process. This required users to manually adjust the location of the input node before proceeding with workflow creation. We have fixed this issue to ensure a smoother workflow creation experience. | |![bug](/img/bug.jpg)|Fixed an issue where underlines showed up for some buttons|Previously, certain buttons, such as those found on the workflow page, were displaying underlines unnecessarily. We fixed it. | |![bug](/img/bug.jpg)|Fixed an issue with the RAG Prompter model template|Previously, when creating a RAG Prompter model with a custom template and assigning it to a workflow via an API call, the API call result indicated that the Prompter model template was updated. However, the updated template was not visible in the workflow UI. We fixed the issue. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed issues with some modules |Fixed an issue that caused the [Cluster Labeler](https://clarifai.com/clarifai/data/modules/cluster-labeler) module and the [Low-dimensional visualization](https://clarifai.com/clarifai/visualize/modules/low_dimensional_viz) module not to work ## Others |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with the pending email verification modal|Previously, when an email address was long, it could extend beyond the boundary of the 'Please verify your email' modal. We fixed it.| |![bug](/img/bug.jpg)|Fixed an issue with SSO authentication|You can log in to the Clarifai platform via an SSO method (either GitHub or Google), complete the SSO authentication steps, and get redirected to Clarifai seamlessly.Previously, if you failed to complete the SSO login within 500 seconds, the login session could expire, resulting in an 'Invalid or expired oauth request state' error. To address this, we have extended the expiration time to allow users enough time to complete any authentication steps with their SSO provider. | --- ## Release 10.3 # Release 10.3 **Release Date:** April 2nd, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## App Creation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Introduced [app templates](https://docs.clarifai.com/clarifai-basics/app-templates) for streamlined app creation | We now provide pre-built, ready-to-use templates that expedite the app creation process. Each template comes with a range of resources, such as datasets, models, workflows, and modules, allowing you to quickly hit the ground running with your app creation process. | ## Node SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) **(Developer Preview)** | Released a new [Node SDK](https://github.com/Clarifai/clarifai-nodejs) | We released the first open source version (for developer preview) of a Node SDK for JavaScript/TypeScript developers focused on creating web services and web apps consuming AI models. It is designed to offer a simple, fast, and efficient way to experience the power of Clarifai’s AI platform — all with just a few lines of code.You can check its documentation [here](https://docs.clarifai.com/nodejs-sdk/installation-guide/).| ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published new models _(Clarifai-hosted models are the ones we host within our Clarifai Cloud. Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys)_|Clarifai-hosted [Mxbai-embed-large-v1](https://clarifai.com/mixedbread-ai/embed/models/mxbai-embed-large-v1), a state-of-the-art, versatile, sentence embedding model trained on a unique dataset for superior performance across a wide range of NLP tasks. It also tops the [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard). Clarifai-hosted [Genstruct 7B]( https://clarifai.com/nousresearch/instruction-generation/models/genstruct-7b ), an instruction-generation LLM, designed to create valid instructions given a raw text corpus. It enables the creation of new, partially synthetic instruction fine-tuning datasets from any raw-text corpus. Wrapped [Deepgram’s Aura Text-to-Speech](https://clarifai.com/deepgram/tts/models/aura-tts) model, which offers rapid, high-quality, and efficient speech synthesis, enabling lifelike voices for AI agents across various applications. Wrapped [Mistral-Large](https://clarifai.com/mistralai/completion/models/mistral-large), a flagship LLM developed by Mistral AI, and renowned for its robust multilingual capabilities, advanced reasoning skills, mathematical prowess, and proficient code generation abilities.Wrapped [Mistral-Medium](https://clarifai.com/mistralai/completion/models/mistral-medium), Mistral AI's medium-sized model. It supports a context window of 32k tokens (around 24000 words) and outperforms Mixtral 8x7B and Mistral-7b on benchmarks across the board.Wrapped [Mistral-Small](https://clarifai.com/mistralai/completion/models/mistral-small), a balanced, efficient large language model offering high performance across various tasks with lower latency and broad application potential. Wrapped [DBRX-Instruct](https://clarifai.com/databricks/drbx/models/dbrx-instruct), a state-of-the-art, efficient, open LLM by Databricks. It’s capable of handling input length of up to 32K tokens. The model excels at a broad set of natural language tasks, such as text summarization, question-answering, extraction, and coding. | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Added ability to import datasets via archive files with ease | Within the Input-Manager, users can now seamlessly upload archive or zipped files containing diverse data types such as texts, images, and more. | ## Devtools Integrations |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Integrated the `unstructured` Python library with Clarifai as a target destination connector | [The `unstructured` library](https://pypi.org/project/unstructured/) provides open-source components for ingesting and pre-processing images and text documents. We’ve integrated it with Clarifai to allow our users streamline and optimize the data processing pipelines for LLMs. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![enterprise](/img/enterprise.jpg) **[Enterprise-only]** | Added support for exporting your own trained models | You can now export the models you own from our platform to a pre-signed URL. Upon export, you'll receive model files accessible via pre-signed URLs or private cloud buckets, along with access credentials. Please note that we only support exporting trainable model types. Models such as `embedding-classifiers`, `clusterers`, and `agent system operators` are not eligible for export. | |![improvement](/img/improvement.jpg) |Improved the Model-Viewer UI of multimodal models | For multimodal models like [GPT4-V](https://clarifai.com/openai/chat-completion/models/gpt-4-vision), users can provide input text prompts, include images, and optionally adjust inference settings. The output consists of generated text. They also support the use of [3rd party API keys](https://docs.clarifai.com/api-guide/predict/text#use-third-party-api-keys) (for Enterprise Customers). | |![improvement](/img/improvement.jpg)|Made improvements to the [local model upload](https://docs.clarifai.com/portal-guide/model/local-model-upload) functionality |We now provide users with a pre-signed URL for uploading models.We added educational materials and tooltips to the local model upload UI. We made other improvements to make the process of uploading models simple and intuitive. | |![improvement](/img/improvement.jpg)|Enhanced the functionality of the **Actions** column within a model’s versions table|We refactored the column into an intuitive context menu. Now, when a user clicks on the three dots, a dropdown menu presents various options, optimizing user experience and accessibility.| |![improvement](/img/improvement.jpg)|Enabled deletion of associated model assets when removing a model annotation|Now, when deleting a model annotation, the associated model assets are also marked as deleted.| ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved the functionality of the [Face workflow](http://clarifai.com/clarifai/main/workflows/Face)|You can now use the Face workflow to effectively generate face landmarks and perform face visual searches within your applications. | |![bug](/img/bug.jpg)|Fixed an issue with changing an app’s base workflow for users subscribed to the Community pricing plan | Previously, users on the Community plan encountered errors when changing an app's base workflow to trigger re-indexing of the app’s inputs. We fixed the issue.| ## Community UI |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added Python SDK code snippets to the **Use Model / Workflow** modal window | If you want to use a model or a workflow for making API calls, you need to click the **Use Model / Workflow** button at the upper right corner of the individual page of a model or workflow. The modal that pops up has snippets in various programming languages, which you can copy and use. We introduced Python SDK code snippets as a primary tab. Users can now conveniently access and copy the Python SDK code snippets directly from the modal. | |![improvement](/img/improvement.jpg)| Revamped the resource filtering experience on desktop devices | We relocated the filtering sidebar from the right to the left side of the screen, optimizing accessibility and user flow.We also made other improvements to the filtering feature, such as using chevrons to mark the collapsible sections, enhancing the alignment of the clear button, and enhancing the appearance of the divider line. We also added `Multimodal-to-text`, `Multimodal-embedder`, and `text-to-audio` filtering options. | |![improvement](/img/improvement.jpg)|Revamped mobile resource filters with a fresh design | Implemented a new and improved design for resource filters on mobile platforms.| |![improvement](/img/improvement.jpg)| Added ability to sort apps listed on the collapsible left sidebar of your individual app page | You can now sort the apps alphabetically (from A to Z) or by "Last Updated." This lets you find the apps you need quickly and efficiently. | |![improvement](/img/improvement.jpg)|Enhanced markdown template functionality with custom variables|We have introduced a feature that allows users to insert custom variables such as `{{user_id}}` and `{{app_id}}` into markdown templates, particularly in sections like the Notes section of a model. These variables are dynamically replaced with the corresponding `user_id` and `app_id` extracted from the URL, allowing you to personalize content within your templates. For example, within the Notes section of a model, you can now add `{{user_id}}` to dynamically display the user who created the model. | |![improvement](/img/improvement.jpg)|Improved responsiveness for 13-inch MacBooks|We improved responsiveness issues to ensure an optimal viewing experience for 13-inch MacBook devices with a viewport of 1440px × 900px dimensions. | |![bug](/img/bug.jpg)|Fixed an issue with repeated consent requests for users with Google accounts| Previously, users logging into the Community platform using their Google accounts encountered an issue where they were repeatedly prompted to give consent upon each login. The user consent is now stored and persisted, eliminating the need for repetitive requests. | |![bug](/img/bug.jpg)|Fixed an issue where an undesirable background appeared when a login session expired on a public page|Now, when a session expires on a public page, a modal login window will appear while you're still on the same page. This allows you to easily log back in and continue where you left off. Also, closing the modal by clicking the "X" button or successfully logging in will keep you on the same public page, eliminating unnecessary navigation. | | ![bug](/img/bug.jpg) |Fixed an issue with editing the scopes of a collaborator on the App Settings page| Previously, it was not possible to deselect scopes when editing a collaborator’s role on the App Settings page. We fixed the issue.We also fixed an issue where some essential scopes were missing for collaborators. | | ![bug](/img/bug.jpg) |Fixed an image flicker behavior issue on the Feature page| Previously, users experienced disruptive flickering when navigating the page, impacting the overall user experience negatively. The images now load smoothly and without any flickering.| | ![bug](/img/bug.jpg)| Fixed an issue that made bullets or numbers to be invisible | Previously, the bullets or numbers in a list were not getting rendered properly when applying them in the Notes section of a resource. We fixed the issue. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) |Added support for exporting models| You can now use the Python SDK to [export](https://github.com/Clarifai/clarifai-python/pull/304) your own trained models to an external environment. | |![new-feature](/img/new_feature.jpg)|Introduced improvements to the dataloader module | We added [retry mechanisms](https://github.com/Clarifai/clarifai-python/pull/307) for failed uploads and introduced systematic handling of failed inputs. These improvements optimize the data import process and minimize errors within the dataloader module. | |![new-feature](/img/new_feature.jpg)|Added support for dataset version ID| Previously, it was not possible to access or interact with specific versions of a dataset within the Python SDK. This update introduces support for dataset versions in several key areas as detailed [here](https://github.com/Clarifai/clarifai-python/pull/315). | |![improvement](/img/improvement.jpg)|Made enhancements to the RAG (Retrieval Augmented Generation) feature|Enhanced the RAG SDK's `upload()` function to accept the `dataset_id` parameter. Enabled custom workflow names to be specified in the RAG SDK's `setup()` function. Added support for chunk sequence numbers in the metadata when uploading chunked documents via the RAG SDK. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg) |Fixed an issue where it was not possible to end a labeling task in the Labeler tool |Previously, it wasn't possible to end and track the progress of a labeling task in the Labeler. This sometimes led to duplicate labeling efforts and difficulty managing large datasets. We fixed the issue and now provide a way to gauge the progress of a labeling task. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg) |Cleaned up Organization pages for improved layout and readability| We removed excess white spaces that were appearing in various sections across the Organization pages, specifically on the Member and Team pages. This update optimizes the presentation and readability of the content. | --- ## Release 10.4 # Release 10.4 **Release Date:** May 7th, 2024 Release 10.4 Added symmetric and asymmetric text search capabilities within the Input-Manager Start for free | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Added symmetric and asymmetric text search capabilities within the Input-Manager | Symmetric search allows users to find text inputs that closely resemble other text inputs, facilitating easy identification of similar content. On the other hand, asymmetric search empowers users to search for text inputs that effectively answer specific query questions, enhancing the precision and relevance of search results. | |![bug](/img/bug.jpg)|Fixed an issue with visual searches| Previously, visual searches for inputs on the Input-Manager were sometimes slow. We fixed it. | |![bug](/img/bug.jpg)|Fixed issues with concepts used for searching inputs on the Input-Manager|The tags or concepts used for performing searches had undergone visual changes. Additionally, it was not possible to delete tags individually; the only way was to remove the entire string in the search field. We fixed these issues.| ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published new models _(Clarifai-hosted models are the ones we host within our Clarifai Cloud. Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys)_| Wrapped [Command-R+](https://clarifai.com/cohere/generate/models/command-r-plus), a highly efficient, multilingual, enterprise-grade LLM optimized for real-world business applications, boasting advanced RAG capabilities and a 128k-token context window. Wrapped [Gemini 1.5 Pro](https://clarifai.com/gcp/generate/models/gemini-1_5-pro), a powerful, efficient LLM with 1 million long-context window, enabling advanced reasoning and comprehension across various data types. Wrapped [Mixtral-8x22B](https://clarifai.com/mistralai/completion/models/mixtral-8x22B), the latest and largest Mixture of Expert LLM from Mistral AI. It's a state-of-the-art machine learning model using Mixture 8 of Experts (MoE) 22b models. Wrapped [Qwen1.5-32B](https://clarifai.com/qwen/qwenLM/models/qwen1_5-32B-chat), a model that provides competitive AI performance with optimized efficiency, making it a strong choice for diverse, multilingual applications. Clarifai-hosted [Gemma-1.1-7b-it](https://clarifai.com/gcp/generate/models/gemma-1_1-7b-it), a lightweight, decoder-only LLM trained on 6 trillion tokens of diverse text data, suitable for various text generation tasks with improved quality and safety measures. Clarifai-hosted [Llama-3-8B-Instruct](https://clarifai.com/meta/Llama-3/models/Llama-3-8B-Instruct). Llama 3 instruction-tuned models are optimized for dialogue use cases and outperform many of the available open source chat LLMs on common industry benchmarks. Wrapped [Llama-3-70B-Instruct](https://clarifai.com/meta/Llama-3/models/llama-3-70B-Instruct), an advanced, scalable language model designed for diverse applications, offering state-of-the-art performance in coding, reasoning, and multi-use conversational capabilities. Wrapped [Llama-3-70B](https://clarifai.com/meta/Llama-3/models/llama-3-70B). Llama-3 is a state-of-the-art large language model designed for enhanced reasoning and coding. It has a broad application across multiple languages and tasks. Clarifai-hosted [LLaVA-v1.6-Mistral-7B](https://clarifai.com/liuhaotian/llava/models/llava-v1_6-mistral-7b), a high-performance, efficient, and cross-lingual large multimodal model, boasting state-of-the-art capabilities in visual reasoning, OCR, and zero-shot Chinese multimodal understanding. Clarifai-hosted [LLaVA-1.5-7b](https://clarifai.com/liuhaotian/llava/models/llava-1_5-7b), a state-of-the-art language vision model that represents a significant advancement in the field of multimodal artificial intelligence. | ## New Published Templates |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new [app templates](https://docs.clarifai.com/clarifai-basics/app-templates) | Published [Text Moderation Template](https://clarifai.com/clarifai/text-moderation), which provides ready-to-use workflows and models, leveraging NLP Models and LLMs to automatically monitor and detect inappropriate or harmful text content.Published [Sentimental Analysis Template](https://clarifai.com/clarifai/sentiment-analysis), which provides a guide for sentimental analysis and comes with several ready-to-use sentimental analysis workflows and models dealing with different use cases, leveraging different NLP models and LLMs.Published [Chatbot Template](https://clarifai.com/clarifai/chatbot-template), which allows you to develop AI chatbots swiftly using Clarifai LLMs, offering personalized assistance and integrating external data with RAG framework for enhanced capabilities. Published [Image Moderation Template](https://clarifai.com/clarifai/image-moderation), which provides diverse AI-powered workflows for automatically filtering and categorizing inappropriate or harmful images based on various criteria. Published [Content Generation Template](https://clarifai.com/clarifai/content-generation-template), which empowers users to efficiently create diverse, tailored text, from emails and blogs to social media content and stories, enhancing communication and creativity. Published [Document Summarization Template](https://clarifai.com/clarifai/document-summarization), which is an app template for document summarization — supports 3 levels that start with Novice and end up with Expert. Published [RAG Template](https://clarifai.com/clarifai/rag-template), which streamlines the creation of Retrieval-Augmented Generation (RAG) applications with Clarifai, enhancing LLMs with external knowledge for accurate, up-to-date information generation. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added new features. Learn more about them [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Added a flag to download a model. If the `export_dir` parameter is provided in `Model().export()`, the exported model will be saved in the specified directory. Otherwise, the export status will be shown. Added support for Label IDs in Dataloaders (using the `label_ids` parameter) and `get_proto` functions in the Inputs class.Implemented pagination feature in Search. Added `pagination` parameter in the `Search()` class and included `per_page` and `page_no` parameters in the `Search().query()` method.Added algorithm parameter to `Search()`. Added root certificate support to establish secure gRPC connections by including a `root_certificates_path` parameter in all relevant classes and authentication helpers. Also, updated gRPC to the latest version.Added missing VERSION and `requirements.txt` files to `setup.py.` | | ![improvement](/img/improvement.jpg) | Made some improvements | Upgraded to the latest version of the clarifai-grpc package (version 10.3.4) to leverage the newer gRPC functionality.RAG apps, workflows, and other resources are now automatically set up using UUIDs in their IDs instead of timestamps. This change helps avoid race conditions. Changed logger for `Inputs().upload_annotations` to display full details of failed annotations. Updated the model upload CLI documentation.Limited the maximum upload batch size for `Inputs().upload_inputs()` function. Also addressed the inconsistency in the model version ID parameter between `App.model()` and `Model()`.Removed runners from the SDK. | | ![bug](/img/bug.jpg) | Fixed some bugs | Fixed an issue where UUIDs were used in tests to avoid race conditions with timestamps. Hardcoded the schema package to version 0.7.5, as it introduced breaking changes. Fixed RAG upload bug by updating llama-index-core version to 0.10.24 in the `ImportError` message. Fixed a `RAG.setup()` bug where if a specific workflow was deleted and another workflow with the same ID was created, by adding a timestamp while creating a new prompter model. Fixed `RAG.upload()` to support uploading a folder of text files. Fixed the training status bug by removing the constraint of users specifying `model_type_id` for training_logs and instead using `load_info()` to get model version details. Fixed a create workflow bug that occurred due to the change in the model version ID parameter. Fixed unnecessary infra alerts by adding a wait time before deleting a model in model training tests. | ## Node SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added some features | Added a new `Dataset` class with several methods for handling datasets, including creating and deleting versions, listing versions, and uploading data from a folder or a CSV file. Added several new methods to the `Input` class for handling inputs, including methods for bulk uploading, waiting for inputs, deleting failed inputs, and retrying uploads.Added several new methods and properties to the `Search` class, including support for different search algorithms and metrics, and improved handling of queries and pagination. | | ![improvement](/img/improvement.jpg) | Made an improvement| Since the `uuidv4` package has been deprecated in npm, we replaced it with the recommended `uuid` package. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Added ability to browse the resources in a template | You can now access a modal popup on a template’s overview page that allows you to easily view the resources available in it, including inputs, datasets, models, workflows, and modules.| | ![improvement](/img/improvement.jpg)| Introduced an easier way to add collaborators |We added an option to add app collaborators from the App Overview screen. | | ![improvement](/img/improvement.jpg) | Introduced underscores for creating an app ID | You can now use underscores when creating an app ID. | |![improvement](/img/improvement.jpg) | Moved app templates and starred apps to their own paths | Previously, we implemented this as parameter filters. We relocated templates and starred filters to separate URLs. You can now access templates at `apps/templates` and starred apps at `apps/starred`.| | ![improvement](/img/improvement.jpg) | Disabled **Use Model** button for models with no versions |As it's not possible to send API calls to models with no versions, we've disabled the **Use Model** button if no version exists. Therefore, you'll need to create a version first before using a model. | | ![improvement](/img/improvement.jpg) | Increased font size for various texts in mobile view | We enhanced the readability of texts in various areas, such as app IDs, user IDs, and field placeholders, by increasing their font size in mobile view.| | ![improvement](/img/improvement.jpg) | Added resource pricing on the Model-Viewer page for some types of models | We added pricing information for these model types: `text-to-text`, `image-to-text`, and `multimodal-to-text`. | |![bug](/img/bug.jpg)|Fixed an issue with filtering [templates](https://docs.clarifai.com/clarifai-basics/app-templates/) | Previously, when selecting a template from the app creation modal, toggling between the "All" and "Starred" menu bar options didn't function correctly. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with excessively truncated model names|The long model names are now truncated less aggressively, particularly on desktop devices.| |![bug](/img/bug.jpg)|Fixed an issue with the app switcher feature for collaborators|The app switcher feature, located in the collapsible left sidebar, allows you to effortlessly switch between different apps. Clicking it reveals a drop-down menu displaying your own apps as well as those you have access to. Previously, the app switcher feature produced an error for users in a collaborated app. We fixed this issue, and it now works as intended.| |![bug](/img/bug.jpg)|Fixed an issue with the browser's "Back" button during user onboarding | Previously, during user onboarding, the "Back" button didn't allow users to navigate from the Model-Viewer page back to the Community listing page. We fixed this issue. | |![bug](/img/bug.jpg)|Fixed an issue where modules’ IDs appeared in the collapsed sidebar| Previously, when you collapsed the left sidebar, the IDs of installed modules in your app could be visible there. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue where log-in steps sometimes resulted in a "page not found" error |Users can now log in without encountering any errors. | |![bug](/img/bug.jpg) | Fixed an issue with “Use Model” modal during the onboarding flow | If you click the “Use Model” button on a Model-Viewer page, a modal will pop up with code snippets for using the model in a workflow. Previously, creating a workflow this way was not working properly during a user onboarding; it resulted in multiple redirects. We fixed the issue. | | ![bug](/img/bug.jpg) | Fixed an issue where some essential scopes were missing for collaborators | We fixed the issue where essential scopes were missing for collaborators, ensuring proper access to apps. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Improved organization invitation emails | We improved organization invitation emails by adding details of the person who sent the invitation. The email now includes their user ID and user name, which enhances the legitimacy of the invitation.| |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to patch an app after transferring its ownership to an organization|Previously, if you transferred an app to an organization, you couldn't patch it. For example, you could not edit the org app’s description. We fixed it. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed issues with playing a video input within the Input-Viewer| Previously, users experienced limitations when trying to control video playback, such as navigating to specific points, skipping forward or backward, or performing any interactive actions. Any such attempt to scroll through a video could result in it snapping back to the starting point unexpectedly. We fixed these issues. | |![bug](/img/bug.jpg)|Fixed issues with renaming concepts in the Input-Viewer| While on the Input-Viewer screen, you can rename a concept listed in the right sidebar, which also establishes a relation for it.Previously, renaming a concept prevented you from using the **Filter by concept** field to search for the new name. We have fixed this issue, and you can now search for both the original concept and the new relation created for it.Previously, when creating a new relation for a concept that already had a previous relation, the new relation could be added without removing the old one. We fixed this issue, and now, before creating new relations, old ones are removed automatically. We also fixed an issue with entering the same name of a concept. For example, previously, if you generated predictions from classifications — let’s say one of the predicted concepts is “sky.” If you edited “sky” to “wall,” a new relation was created. However, if you edited the concept name again to “sky,” a new relation was erroneously created. We fixed the issue, and now since the relation and the originally predicted concept are the same, we remove the “wall” relation. | |![bug](/img/bug.jpg)|Fixed an issue with the gearbox icon in the Input-Viewer|Previously, opening the gearbox popover, selecting a model or workflow, and then closing the popover resulted in the gearbox icon remaining gray, indicating that nothing was selected inside the popover. We fixed the issue, and now the gearbox icon turns blue when an item is selected within its popover.| |![bug](/img/bug.jpg)|Fixed an issue with managing object-track annotations for video inputs on the Input-Viewer|Previously, the timeline next to the left of the scrollbar of a video input represented seconds instead of frames. We fixed the issue, and it now accurately represents frames, as intended. Additionally, we addressed an issue that prevented the slider from reaching the last frame of a video input.| | ![bug](/img/bug.jpg) |Fixed an issue with the canvas Zoom In/Out functionality on the Input-Viewer | Previously, it was extremely laggy and unusable when using the CMD/CTRL button with the mouse wheel up/down or when using a trackpad. Now, when the user presses the CMD/CTRL button and scrolls the mouse wheel up/down, the action is performed at a normal speed, depending on how the user is scrolling. | ## Labeling Tasks​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with displaying more concepts when creating a new labeling task| Previously, clicking “show 50 more” in the concept select list resulted in nothing being loaded, preventing users from seeing the full list of concepts. We fixed this issue, and now the full list of concepts is loaded after clicking “show 50 more”. | |![bug](/img/bug.jpg)|Fixed an issue where changing the dataset while creating a new task did not update the task name |Now, on the task creation page, switching between different datasets correctly updates the task name field to match the currently selected dataset, unless the user has manually edited the field before.| --- ## Release 10.5 # Release 10.5 **Release Date:** June 4th, 2024 Release 10.5 Introduced a new auto-annotation experience for labeling already existing inputs in datasets Introduced the ability to manually review labeled inputs by partitioning them Start for free | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Data Labeling​ :::warning Private Preview These data labeling features have been released as [**Private Preview**](https://docs.clarifai.com/product-updates/changelog/release-types). ::: |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) |Introduced a new [auto-annotation](https://docs.clarifai.com/portal-guide/annotate/auto-annotation) experience for labeling already existing inputs in datasets| You can now set up a labeling task and enable automatic annotation, streamlining the process of annotating your text and image inputs quickly and efficiently. You can select a model or workflow you own or choose one published in the Community for the auto-annotation task.You can also review and edit the annotations from auto-annotation tasks. | | ![new-feature](/img/new_feature.jpg) |Introduced the ability to manually [review](https://docs.clarifai.com/portal-guide/annotate/review) labeled inputs by partitioning them | You can now distribute inputs to be reviewed across multiple reviewers in the Labeling Tasks tool. This lets you automatically distribute labeled assets across multiple reviewers to improve the speed of review mode and reduce chances for a single point of dependence. | ## Devtools Integrations |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Integrated LiteLLM with Clarifai | [LiteLLM](https://docs.litellm.ai/docs/) is an open-source Python library that provides a unified interface to call 100+ Large Language Models (LLMs) using the same Input/Output format. It allows you to seamlessly interact with LLM APIs using the standardized OpenAI format. This integration aims to provide users with more powerful, efficient, and versatile tools for their Natural Language Processing (NLP) tasks. | ## Deep Fine-Tuning |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Made improvements to deep fine-tuned templates | We made significant improvements to various [deep fine-tuned templates](https://docs.clarifai.com/portal-guide/model/deep-training/#template-types), enhancing the capabilities available for training your models. The updates include:MMClassification visual classification template: Updated from version 1.5.0 to 2.1.0, offering improved features and performance for visual classification tasks.MMDetection visual detection template: Updated from version 2.24.1 to 3.3.0, providing advanced capabilities and optimizations for visual detection tasks.[YOLOX](https://github.com/open-mmlab/mmdetection/tree/main/configs/yolox) support added: We introduced support for YOLOX, a state-of-the-art object detection training template, further expanding the tools available for high-performance object detection. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published new models _(Clarifai-hosted models are the ones we host within our Clarifai Cloud. Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys)_ | Wrapped [Snowflake Arctic-Instruct](https://clarifai.com/snowflake/arctic/models/snowflake-arctic-instruct) model, a cost-effective, enterprise-focused Large Language Model (LLM) that excels in SQL, coding, and instruction-following tasks, offering open access to its advanced AI capabilities. Clarifai-hosted [Qwen-VL-Chat](https://clarifai.com/qwen/qwen-VL/models/qwen-VL-Chat), a high-performing Large Vision Language Model (LVLM) by Alibaba Cloud for text-image dialogue tasks, excelling in zero-shot captioning, VQA, and referring expression comprehension while supporting multilingual dialogue. Clarifai-hosted [CogVLM-Chat](https://clarifai.com/thudm/cogvlm/models/cogvlm-chat), a state-of-the-art visual language model that excels in generating context-aware, conversational responses by integrating advanced visual and textual understanding. Wrapped [GPT-4o](https://clarifai.com/openai/chat-completion/models/gpt-4o), a multimodal AI model that excels in processing and generating text, audio, and images, offering rapid response times and improved performance across languages and tasks, while incorporating advanced safety features. Wrapped [Gemini 1.5 Flash](https://clarifai.com/gcp/generate/models/gemini-1_5-flash), a cost-efficient, high-speed foundation LLM optimized for multimodal tasks, ideal for applications requiring rapid processing and scalability. Wrapped [WizardLM-2 8x22B](https://clarifai.com/wizardlm/generate/models/wizardLM-2-8x22B), a state-of-the-art open-source LLM, excelling in complex tasks like chat, reasoning, and multilingual understanding, competing closely with leading proprietary models. Wrapped [Qwen1.5-110B-Chat](https://clarifai.com/qwen/qwenLM/models/qwen1_5-110B-chat) LLM, with over 100 billion parameters, demonstrates competitive performance in base language tasks, shows significant improvements in chatbot evaluations, and boasts of multilinguality. Wrapped [Mixtral-8x22B-Instruct]( https://clarifai.com/mistralai/completion/models/mixtral-8x22B-Instruct-v0_1), the latest and largest mixture of expert LLM from Mistral AI with state-of-the-art machine learning model using a mixture 8 of experts (MoE) 22b models. Wrapped [DeepSeek-V2-Chat](https://clarifai.com/deepseek-ai/deepseek-chat/models/deepseek-V2-Chat), a high-performing, cost-effective 236 billion MoE LLM, excelling in diverse tasks such as chat, code generation, and math reasoning. Clarifai-hosted [MiniCPM-Llama3-V 2.5](https://clarifai.com/openbmb/miniCPM/models/miniCPM-Llama3-V-2_5), a high-performance, efficient 8B parameter multimodal model excelling in OCR, multilingual support, and multimodal tasks. Wrapped [Codestral-22B-v0.1](https://clarifai.com/mistralai/completion/models/codestral-22b-instruct), an advanced generative LLM designed for versatile and efficient code generation across 80+ programming languages.Clarifai-hosted [Phi-3-Vision-128K-Instruct](https://clarifai.com/microsoft/text-generation/models/phi-3-vision-128k-instruct), a high-performance, cost-effective multimodal model for advanced text and image understanding tasks.Clarifai-hosted [Openchat-3.6-8b](https://clarifai.com/openchat/openchat/models/openchat-3_6-8b-20240522) model, a high-performance, open-source LLM fine-tuned from Llama3-8b with C-RLFT, delivering ChatGPT-level performance across various tasks. | ## New Published App Templates |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published new [app templates](https://docs.clarifai.com/clarifai-basics/app-templates)|Published [Coding Template](https://clarifai.com/clarifai/coding-template), which helps to streamline the development process by facilitating efficient code completion, bug detection, refactoring, and more. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added TypeScript (Node.js) SDK code snippets to the **Use Model / Workflow** modal window | If you want to use a model or a workflow for making API calls, you need to click the **Use Model / Workflow** button at the upper right corner of the individual page of a model or workflow. The modal that pops up has snippets in various programming languages, which you can copy and use. We introduced TypeScript SDK code snippets as one of the tabs. Users can now conveniently access and copy the code snippets directly from the modal. | | ![improvement](/img/improvement.jpg) | Added a feature that redirects users to their previous page after logging in | If you are logged out, you will now be taken back to the page you were on after logging in. | | ![improvement](/img/improvement.jpg) | Introduced the ability to add users who are not organization members to teams | This enhancement allows for greater flexibility in team composition and collaboration. | |![improvement](/img/improvement.jpg)|Redesigned the user activation form for new Clarifai accounts|After signing up, users receive a verification email. Clicking the link in the email redirects them to the Clarifai platform to complete their profile details.| |![bug](/img/bug.jpg)| Fixed an issue with sorting resources on mobile view | We fixed an issue where the **Sort by** feature was not visible. You can now easily sort resources using these criteria on your mobile device.| |![bug](/img/bug.jpg)|Fixed delayed loading of page contents| Previously, the notes section on Community pages did not appear immediately on the first render. We fixed the issue, ensuring the contents now load promptly as expected. | |![bug](/img/bug.jpg)| Fixed a UI issue with modals | We fixed a UI issue where modal buttons would overflow the borders if the window height was too small. With this fix, modals are now fully responsive, ensuring that buttons remain within the borders regardless of the window height. | | ![bug](/img/bug.jpg) | Fixed an issue with user onboarding | Fixed an issue that prevented the **Create Application** modal from opening during the user onboarding process. | | ![bug](/img/bug.jpg) | Fixed an issue with the placeholder search text on the Community page| We fixed an issue where the placeholder text in the main page's search bar was partially hidden on smaller screen sizes or when the browser was resized. The placeholder text now displays properly across all devices and screen sizes. | | ![bug](/img/bug.jpg) |Fixed an issue with the table showing the current members of an organization | We fixed an issue with the organization members table where long names in the "Name" column would overflow the table elements. Long names now display correctly within the table. | ## Apps / Templates |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved the handling of unauthenticated users|If a user is not logged in or lacks access to the app:We hide the Inputs block and its values. So, the user cannot see the inputs details on the App Overview page. We direct the user to available public resources (e.g. models) without prompting for a login.| | ![improvement](/img/improvement.jpg) | Added a "Reindex" checkbox in the "Change Base Workflow" section | This option allows users to decide whether they want to reindex their app's inputs before [changing the app's base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows#how-to-change-a-base-workflow). | | ![improvement](/img/improvement.jpg) | Made improvements in the app settings page | Removed extra spaces in the tables within the API Keys and Collaborators sections on the app settings page for a cleaner and more streamlined layout. | | ![improvement](/img/improvement.jpg) | Expanded the list of available workflows when creating a new application | You can now choose from a broader selection of base workflows to better suit your needs. | |![bug](/img/bug.jpg)| Fixed an issue with deleting an app | Previously, when deleting an app from the App Overview page, the pop-up dialog would remain open even though the app was successfully deleted. You had to refresh the page to close it. We fixed the issue, and now, upon deleting an app, the dialog closes, and you are redirected to the **My Apps** or **Org Apps** page. | |![bug](/img/bug.jpg)|Fixed the position of the collapse button in the app sidebar|Previously, the button for collapsing the app sidebar was misaligned. We fixed the issue, and the button is now properly positioned. | |![bug](/img/bug.jpg)|Fixed an issue with filtering app templates| To view available templates, visit the **Apps / Templates** screen and select the ["Templates"](https://clarifai.com/explore/apps/templates) option.Previously, when sorting templates by "Last Updated," the list mistakenly included both apps and templates. To view only templates, users had to manually select "All" and then "Templates" again.We fixed the issue. Now, when you sort by "Last Updated" on the Templates page, only templates are displayed as intended. | | ![bug](/img/bug.jpg) | Fixed an issue with truncated template names|Previously, in the template selection screen for creating an app, long template names were truncated and unreadable. We fixed the issue. For example, now, if template names don't fit in the available space, they are omitted, and a tooltip saying "Clarifai's Templates" appears on hover. | | ![bug](/img/bug.jpg) | Added the ability to quickly delete the auto-completed App ID field | We introduced a cross (X) icon to the App ID and Short Description fields, allowing users to quickly erase contents when setting up an app using a template.| ## Platform Resource Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Improved ability to select a version of a resource | We introduced a standardized version selection feature across the platform, now available on both the Model-Viewer and Dataset-Viewer pages. This feature allows you to select and view specific versions of your resources using truncated digits and dates for easy identification. | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Fixed an issue with the left sidebar | We fixed an issue where the left sidebar on the Input-Manager and Input-Viewer was not scrollable. Previously, this made it difficult to access all sidebar content. With this improvement, the left sidebar is now scrollable, allowing you to easily bring all parts into view. | | ![bug](/img/bug.jpg) | Fixed an issue with the scrolling behavior in the Input-Manager | Previously, when the first batch of Input-Manager search results was fully visible on the screen without requiring scrolling, the page entered a broken state. This state made it difficult to scroll down to view more inputs and resulted in failure to fetch the next batch of inputs. We fixed the issue, and the scrolling behavior now works as desired. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | Deprecation | Deprecated some tracker agent system operators | We deprecated the Neural Tracker and Neural Lite Tracker operators. These operators were previously used to chain together with other models in a workflow for object-tracking tasks in computer vision. We no longer support them. | | ![bug](/img/bug.jpg) | Fixed an issue with a redundant close button | We fixed an issue where a redundant close button appeared in the modal for updating text-to-text workflow nodes. Previously, this caused two close buttons to be displayed instead of one. With this fix, we removed the redundant close button, ensuring only one close button is displayed in the modal. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue where `get_upload_status` would override `log_warnings` in the `dataset.upload_dataset` function | The issue occurred because `self.logger` was shared between both loggers, causing the upload status table to overwrite the log file when both `get_upload_status` and `log_warnings` arguments were used simultaneously. We fixed the issue. | ## Planning for Future Releases |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Planned future breaking changes |An upcoming release of Clarifai’s API, 10.7, will significantly change how Personal Access Tokens (PATs) and API keys work. We plan to implement this change on August 6th, 2024. Learn more [here](https://docs.clarifai.com/product-updates/upcoming-api-changes/pat-api-keys). | --- ## Release 10.6 # Release 10.6 **Release Date:** July 2nd, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Data Labeling​ :::warning GA These data labeling features have been released as [**Generally Available (GA)**](https://docs.clarifai.com/product-updates/changelog/release-types). ::: |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) |Introduced a new [auto-annotation](https://docs.clarifai.com/portal-guide/annotate/auto-annotation) experience for labeling already existing inputs in datasets| You can now set up a labeling task and enable automatic annotation, streamlining the process of annotating your text and image inputs quickly and efficiently. You can select a model or workflow you own or choose one published in the Community for the auto-annotation task.You can also review and edit the annotations from auto-annotation tasks. | | ![new-feature](/img/new_feature.jpg) |Introduced the ability to manually [review](https://docs.clarifai.com/portal-guide/annotate/review) labeled inputs by partitioning them | You can now distribute inputs to be reviewed across multiple reviewers in the Labeling Tasks tool. This lets you automatically distribute labeled assets across multiple reviewers to improve the speed of review mode and reduce chances for a single point of dependence. | ## Base Workflow |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Changed the default [base workflow](https://docs.clarifai.com/portal-guide/workflows/base-workflows/) to Universal | Previously, when an application was created via the API without specifying a base workflow, the [General](https://clarifai.com/clarifai/main/workflows/General) workflow was used by default. Now, the default has been updated to the [Universal](https://clarifai.com/clarifai/main/workflows/Universal) workflow, ensuring consistency with applications created via the UI. If you prefer not to use the default behavior, you can manually specify your desired workflow using an [API parameter](https://docs.clarifai.com/api-guide/workflows/base-workflows#update-your-base-workflow) or through the UI drop-down menu. | ## Devtools Integrations​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Integrated the Embedchain framework with Clarifai | [Embedchain](https://embedchain.ai/) is an open-source framework that makes it easy to develop and deploy personalized AI-powered applications. We’ve integrated it with the Clarifai platform to streamline the creation of personalized LLM applications and offer a seamless process for managing various types of unstructured data.You can check the integration documentation [here](https://docs.clarifai.com/integrations/embedchain/). | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models _(Clarifai-hosted models are the ones we host within our Clarifai Cloud. Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys)_ | Clarifai-hosted [Florence-2-large](https://clarifai.com/microsoft/florence/models/florence-2-large), a lightweight, versatile vision-language model by Microsoft, excelling in multiple tasks using a unified representation and the extensive FLD-5B dataset. Wrapped [Claude 3.5 Sonnet](https://clarifai.com/anthropic/completion/models/claude-3_5-sonnet ), a high-speed, advanced AI model excelling in reasoning, knowledge, coding, and visual tasks; ideal for complex applications. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) |Extended Organization functionality for non-Enterprise Plans| Previously exclusive to Enterprise subscribers, the [Organization](https://docs.clarifai.com/portal-guide/clarifai-organizations/) functionality is now available for Community and PAYG billing plans. This provides more options for collaboration. However, advanced Org management features, such as roles and teams, will still be available to Enterprise plan users only. | |![improvement](/img/improvement.jpg)| Improved the form validation process on the "Create Your Profile" screen | We improved it by ensuring that all fields are validated before moving to the next screen, turning off the "Create Profile" button if the form is not valid, and greying out placeholder values in select fields, just like in input fields. | | ![improvement](/img/improvement.jpg) |Updated the UI design of several organization pages and tables| We improved the design of several pages, such as added a new notification method for invitees to an organization, redesigned tables for members, and introduced new tables for teams and their members or apps. | | ![improvement](/img/improvement.jpg) | Fixed issues with pre-defined role scopes to enhance the functionality and management of Collaborators and Org/Teams members | These improvements ensure more accurate and flexible role assignments, leading to better access control, collaboration, and productivity within the organization. | | ![bug](/img/bug.jpg) | Fixed license type filtering on **Models** listing page | We fixed an issue where sorting models by their license type on the **Models** listing page was not functioning correctly. You can now accurately filter models according to their license type. | | ![bug](/img/bug.jpg) | Fixed an issue with Python (SDK) and NodeJS (SDK) code samples | If you want to use a model or a workflow for making API calls, you need to click the **Use Model / Workflow** button at the upper right corner of the individual page of a model or workflow. The modal that pops up has snippets in various programming languages, which you can copy and use. Previously, the Python (SDK) and NodeJS (SDK) code samples for image classification models displayed incorrect snippets. We fixed this issue, and they now show the correct code samples. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Added ability to edit a workflow name|Previously, the workflow builder only allowed you to set workflow names when creating a workflow. The styling was also outdated, and you could not edit the workflow name while in edit mode. We improved this experience. Now, when in edit mode, the workflow name will be highlighted, allowing you to easily start typing a new name.| |![bug](/img/bug.jpg) | Fixed an issue with the visual-detector node for creating workflows | Previously, the slider for configuring the threshold for detected concepts on the visual-detector node was not working. We fixed the issue, and the slider now works as intended. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)|Made some improvements. Learn more about them [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | To prevent disk writes in environments where the file system might be protected, certain convenience features have been moved exclusively to the CLI. These features are only necessary for CLI usage and have been relocated to that section of the package. Added the ability to generate a random ID for uploaded text inputs if an input ID is not provided in the Dataloader. Previously, uploading text data to a dataset that already contained inputs would fail. Generating a random ID for new inputs resolves this issue. Introduced `BaseClient.from_env()` and added some new endpoints. Upgraded to the latest version of the Clarifai-gRPC package (version 10.5.0) to leverage the newer gRPC functionality. | ## Node SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Released Node.js SDK v0.0.3. Learn more about it [here](https://github.com/Clarifai/clarifai-nodejs/blob/main/CHANGELOG.md). | Added custom certificate support to enable the use of root SSL certificates for secure gRPC connections. Added new RAG capabilities. Introduced unified documentation for Python and Node.js SDKs.| --- ## Release 10.7 # Release 10.7 **Release Date:** August 6th, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Llama 3.1 |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Introduced the Llama 3.1 template | You can now use the Llama 3.1 training template to fine-tune text-to-text models for classification and generation tasks. We provide the 8B version, which you can use for extended context, instruction-following, and advanced applications. Learn more about it [here](https://docs.clarifai.com/portal-guide/model/deep-training/text-templates#llama-31). | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Published new models _(Clarifai-hosted models are the ones we host within our Clarifai Cloud. Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys)_ | Published [Qwen1.5-7B-Chat](https://clarifai.com/qwen/qwenLM/models/qwen1_5-7b-chat), an open-source, multilingual LLM with 32K token support, excelling in language understanding, alignment with human preferences, and competitive tool-use capabilities.Published [Qwen2-7B-Instruct](https://clarifai.com/qwen/qwenLM/models/qwen2-7b-chat), a state-of-the-art multilingual language model with 7.07 billion parameters, excelling in language understanding, generation, coding, and mathematics, and supporting up to 128,000 tokens. Published [Whisper-Large-v3](https://clarifai.com/openai/whisper/models/whisper-large-v3), a Transformer-based speech-to-text model showing 10-20% error reduction compared to Whisper-Large-v2, trained on 1 million hours of weakly labeled audio, and can be used for translation and transcription tasks. Published [Llama-3-8b-Instruct-4bit](https://clarifai.com/meta/Llama-3/models/llama-3-8b-instruct-4bit), an instruction-tuned LLM optimized for dialogue use cases. It can outperform many of the available open source chat LLMs on common industry benchmarks. Published [Mistral-Nemo-Instruct](https://clarifai.com/mistralai/completion/models/mistral-nemo-instruct-2407), a state-of-the-art 12B multilingual LLM with a 128k token context length, optimized for reasoning, code generation, and global applications. Published [Phi-3-Mini-4K-Instruct](https://clarifai.com/microsoft/text-generation/models/phi-3-mini-4k), a 3.8B parameter small language model offering state-of-the-art performance in reasoning and instruction-following tasks. It outperforms larger models with its high-quality data training.Published [GPT-4o-mini]( https://clarifai.com/openai/chat-completion/models/gpt-4o-mini), an affordable, high-performing small model excelling in text and vision tasks with extensive context support. Published [Llama 3.1-8b-Instruct](https://clarifai.com/meta/Llama-3/models/llama-3_1-8b-instruct), a multilingual, highly capable LLM optimized for extended context, instruction-following, and advanced applications. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added patch operations | Introduced patch operations for input annotations and concepts. Introduced patch operations for apps and datasets. | | ![improvement](/img/improvement.jpg) | Improved the RAG SDK | We enabled the RAG SDK to use environment variables for enhanced security, flexibility, and simplified configuration management. | | ![improvement](/img/improvement.jpg) | Improved the logging experience | Enhanced the logging experience by adding a constant width value to rich logging. | | ![bug](/img/bug.jpg)| Fixed an issue with the dataset export feature | We fixed an issue with the feature using a class input session with an authentication header when downloading the dataset export. | |![bug](/img/bug.jpg) |Fixed an issue with Python SDK's visual search functionality | Previously, the Python SDK's visual search functionality returned duplicate results. We fixed the issue, and now the search and ranking results for image searches are returned correctly without duplicates. | ## Organization Settings and Management​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Introduced a new **Organization User** role | This role has access privileges similar to those of an **Organization Contributor** for all apps and scopes. However, it comes with view-only permissions without create, update, or delete privileges. | |![improvement](/img/improvement.jpg) |Implemented restrictions on the ability to add new organizations based on the user's current organization count and feature access |If a user has created one organization and does not have access to the multiple organizations feature, the "Add an organization" button is now disabled. We also display an appropriate tooltip to them. If a user has access to the multiple organizations feature but has reached the maximum creation limit of 20 organizations, the "Add an organization" button is disabled. We also display an appropriate tooltip to them. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Improved the functionality of the [Hyperparamater Sweeps](https://clarifai.com/clarifai/ml/modules/module-hyperparameter_sweeps) module | You can now use the module to effectively train your model on a range and combinations of hyperparameter values. | ## Docs Refresh |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Made significant improvements to our documentation [site](https://docs.clarifai.com/) | Upgraded the site to use Docusaurus version 3.4. Other enhancements include aesthetic updates, a more intuitive menu-based navigation, and a new comprehensive API reference guide. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Enabled deletion of associated model assets when removing a model annotation| Now, when deleting a model annotation, the associated model assets are also marked as deleted. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the functionality of the [Face workflow](https://clarifai.com/clarifai/main/workflows/Face) | You can now use the Face workflow to effectively generate face landmarks and perform face visual searches within your applications. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Fixed issues with Python and Node.js SDK code snippets |If you click the “Use Model” button on an individual model’s page, the “Call by API / Use in a Workflow” modal appears. You can then integrate the displayed code snippets in various programming languages into your own use case. Previously, the code snippets for Python and Node.js SDKs for image-to-text models incorrectly outputted concepts instead of the expected text. We fixed the issue to ensure the output is now correctly provided as text. | |![improvement](/img/improvement.jpg)|Added support for non-ASCII characters| Previously, non-ASCII characters were entirely filtered out from the UI when creating concepts. We fixed this issue, and you can now use non-ASCII characters across all components. | | ![improvement](/img/improvement.jpg) | Improved the display of concept relations | Concept relations are now shown next to their respective concept names, providing clearer and more immediate context. | | ![bug](/img/bug.jpg) | Fixed an issue with delay and redirection during login | Previously, a modal login form would appear when attempting to log in at https://clarifai.com/explore. However, a second login form would also appear, causing confusion with the presence of two forms. We fixed the issue, ensuring a smoother and more intuitive login experience. | | ![bug](/img/bug.jpg) |Fixed an issue with "View inputs" link on smaller screens | Previously, the "View inputs" link on the App Overview page could exceed its boundary on smaller screens, particularly if the number of inputs in the app exceeded 1,000. We fixed this issue to ensure the link now properly adjusts to screen size, maintaining its alignment and readability. | ## Auto-Annotation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed some bugs related to the auto-annotation feature | Fixed an issue where some inputs were not labeled during the auto-annotation process. Fixed an issue where auto-generated concept IDs were incorrectly displayed instead of concept names. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with the labeling tasks table | Previously, the date column was not appearing in the table for listing labeling tasks. The date column now correctly displays in the table, providing the necessary date information for each labeling task. | | ![bug](/img/bug.jpg) | Fixed an issue where an unnecessary modal popup appeared on the tasks review screen | Previously, when you created a new task, labeled it with appropriate concepts, and clicked the **Review** button to navigate to the review screen, a modal stating "No task assignments" could immediately appear. We fixed this issue, and the modal no longer appears, ensuring a smoother review process. | |![bug](/img/bug.jpg) | Fixed an issue with annotation display on the labeling tasks list | Previously, annotations were displayed with annotation IDs instead of names on the labeling tasks list. We fixed this issue, and annotations are now correctly displayed with their names. | | ![bug](/img/bug.jpg) |Fixed an issue with creating a dataset version from task page| Previously, when creating a dataset version from the task page for an auto-annotation task that had been reviewed, the version included labels rejected during the review process. We fixed this issue. Now, when creating a dataset version from the task page, the current state of inputs is accurately considered, and any rejected labels or inputs are excluded from the version. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with bounding box labeling on the Input-Viewer screen | Previously, during labeling with a bounding box on the single Input-Viewer screen, the bounding box could sometimes start from the wrong position, requiring users to adjust it manually. We fixed this issue, ensuring a smoother labeling experience. | --- ## Release 10.8 # Release 10.8 **Release Date:** September 3rd, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Workflows - Agent System Operators |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Deprecation** | Deprecated some agent system operators | We have deprecated the following agent system operators `Custom Code Operator`, `AWS Lambda`, `Isolation Operator`, `Tesseract Operator`, `Neural Tracker`, and `Neural Lite Tracker`. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [Prompt-Guard-86M](https://clarifai.com/meta/text-classifier/models/prompt-guard-86m), a multilingual classifier model designed to detect and prevent prompt injection and jailbreak attacks in LLM-powered applications. | ## Concept Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced advanced concept management via the Python SDK | This feature enables associating multiple labels with an image or text chunk. It supports the creation of hierarchical or relationship-based structures, enhancing both fine-tuning and search capabilities. It also enriches metadata and labeling exercises, which significantly improves data preparation processes. Learn more about it [here](https://github.com/Clarifai/examples/blob/main/concepts/concept_management.ipynb). | ## Secure Data Hosting (SDH) |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Enabled [SDH](https://docs.clarifai.com/product-updates/upcoming-api-changes/secure-data-hosting/) for some model features | We now serve the `model version train logs` and `model version export` capabilities via `secure-data-hosting` instead of `pre-signed urls`. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue with dataset export | We fixed an issue where exporting a dataset through the Python SDK completed successfully, but the exported file lacked annotations. | | ![bug](/img/bug.jpg) | Fixed an issue with model export | We fixed an issue where exporting a model via the Python SDK would result in failures. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Improved error messages for visibility patching requests | Previously, the error messages returned when patching the visibility field of a resource or user were not very helpful. We improved these messages, making them more descriptive and informative to better assist users in troubleshooting visibility patching requests. | | ![improvement](/img/improvement.jpg) | Upgraded to TanStack Table v8 | We’ve updated to the latest version of TanStack Table, v8, to enhance our table and datagrid functionalities across the platform. This headless UI library offers a range of new features and enhanced capabilities, providing a more robust and efficient experience. This upgrade also includes a fix for searching date fields. You can now easily search and filter dates within a table’s date column. | | ![improvement](/img/improvement.jpg) |Enabled sticky table headers| We’ve implemented sticky headers for all tables on our platform. Now, when scrolling through a table, the header remains fixed, making it easier to reference column titles as you navigate through the data. | | ![improvement](/img/improvement.jpg)| Removed scrollbars on smaller screens | We’ve removed horizontal and vertical scrollbars from the left menu on smaller screen sizes. This adjustment ensures a cleaner, more user-friendly interface when the window height is small, which improves the overall user experience. | |![improvement](/img/improvement.jpg) | Ensured full visibility of app sidebar components on mobile and desktop devices | We’ve fixed issues with the app’s sidebar to ensure all components are fully visible and functional on both mobile and desktop screens. This includes resolving problems with cropped lists, overlapping scrollbars, and smoother behavior when interacting with the sidebar. | |![bug](/img/bug.jpg) | Fixed an issue where some operations were still allowed for an Organization User | Previously, Organization Users were mistakenly allowed to perform certain CRUD operations, such as uploading and deleting inputs. We fixed the issue to enable Organization Users to only have view-only permissions, without the ability to create, update, or delete resources. | |![bug](/img/bug.jpg) | Corrected alignment of the "Add Email" field | Previously, the "Add Email" field on the "Account Settings" page was misaligned and did not display correctly. We fixed the issue, ensuring the field now appears as intended. | | ![bug](/img/bug.jpg) | Fixed an issue with disabled "Create Org" button | Previously, the "Create Org" button was incorrectly disabled for certain users who hadn't joined or created an organization, preventing them from creating one. This issue has now been fixed, allowing users to create an organization even if they haven't been part of one before. | ## RAG Prompter |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the [RAG Prompter](https://docs.clarifai.com/portal-guide/agent-system-operators/rag-prompter/) operator | Previously, the RAG Prompter searched for queries across all inputs from the app, spanning across multiple datasets. This occasionally caused context searches to become mixed between different datasets, resulting in jumbled or inaccurate context hits. To resolve this, we now offer enhanced filtering options that allow context searches to be narrowed down based on specific datasets and metadata. These options include metadata filtering, dataset filtering, and dynamic dataset filtering. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Added missing buttons in reviewer field for labeling tasks | We’ve improved the reviewer selection field by adding two new buttons: “Add/Remove all collaborators” and “New collaborator.” These options streamline the process of managing reviewers when creating a new labeling task. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Introduced automatic resizing of bounding boxes/polygons | When annotating and drawing a bounding box or polygon, if the shape is stretched beyond the main canvas area on any side, it will now automatically resize and adjust to fit within the canvas edge on the same side. | | ![improvement](/img/improvement.jpg) | Enhanced annotation identification in Input-Viewer | When hovering over or selecting an annotated object on the canvas of the Input-Viewer, the corresponding annotation in the right sidebar is now highlighted. This makes it easier for users to quickly identify the annotation for deletion, editing, or other purposes. | |![improvement](/img/improvement.jpg) | Added option to download original asset from Input-Viewer | You can now download the original asset directly from the Input-Viewer page. If you right-click on the canvas area, a button will appear, which allows you to download the original input. Previously, only masked versions could be downloaded. | | ![bug](/img/bug.jpg) | Fixed various issues with playing a video input within the Input-Viewer | For example, users previously experienced limitations when trying to control video playback, such as navigating to specific points, skipping forward or backward, or performing any interactive actions. Any such attempt to scroll through a video could result in it snapping back to the starting point unexpectedly. We fixed such issues. | | ![bug](/img/bug.jpg) | Enabled editing of bounding box annotations | Previously, once you drew a bounding box and clicked outside of it, you couldn't edit (move or resize) the annotation by clicking on it. We've fixed this issue, allowing for seamless editing of bounding box annotations. | | ![bug](/img/bug.jpg) | Fixed unexpected deletion of bounding box or polygon annotations | We fixed an issue where clicking the checkmark on the right sidebar of the Input-Viewer page unexpectedly deleted a bounding box or polygon annotation. This behavior was unintended, as annotations can be rightfully deleted by clicking the three vertical dots next to the annotation and clicking the “Delete annotation” button that appears. | | ![bug](/img/bug.jpg) | Fixed an issue with using keyboard shortcuts when working with bounding boxes on the Input-Viewer | Previously, creating, selecting, or removing a bounding box would disable keyboard shortcuts. Now, keyboard shortcuts remain active and functional during all bounding box operations. | | ![bug](/img/bug.jpg) | Fixed an issue with editing bounding boxes | We fixed an issue where editing a bounding box by resizing or moving it caused unintended changes in the coordinates on the opposite side(s). We fixed this "jiggle" of the coordinates. Now, only the edited coordinates are adjusted, while the others remain unchanged, ensuring precise and accurate bounding box modifications. | | ![bug](/img/bug.jpg) | Fixed an issue with creating bounding boxes |You can make bounding box predictions on the Input-Viewer screen by selecting the “Predict” mode and choosing a model, such as “General-detection,” or a workflow to view the predictions on your inputs. You can click on the suggested bounding boxes to edit and accept the annotations. Previously, an error could occur after completing the editing process, preventing the creation of a bounding box. We fixed this issue. Now, after finishing the editing process, a new bounding box is successfully created without any errors. | |![bug](/img/bug.jpg) | Fixed resizing issue for close-proximity bounding boxes | Previously, when two bounding boxes were positioned close to each other, it could sometimes be impossible to resize one of them. We fixed the issue, allowing for accurate resizing even when bounding boxes are in close proximity. | | ![bug](/img/bug.jpg) | Fixed incorrect starting position for bounding box annotations | Previously, when creating a bounding box annotation, it could start at the wrong position, which required a user to constantly adjust the box. We’ve fixed the issue, ensuring that bounding boxes start exactly where intended. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg) | Fixed an issue with filtering concepts in the workflow editor | Previously, if you added a new model to a workflow, and tried filtering the concepts in the model using the “SELECT CONCEPTS” modal, nothing was filtering. We fixed the issue, and the filtering now works as intended. | --- ## Release 10.9 # Release 10.9 **Release Date:** October 4th, 2024 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Clarifai Product Roadmap |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced new public roadmap | We've published a public roadmap to give you a view of our upcoming plans and to gather your valuable feedback. You can explore the features we're considering, vote for the ones you want most, or submit new ideas For a sneak peek at what's coming, such as Compute Orchestration, check out our public roadmap [here](https://portal.productboard.com/bd1rxfuyfbu6vqnmkva3mprx/tabs/1-under-consideration). | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced Clarifai’s Control Center | It is an all-in-one interface that provides centralized visibility into your utilization of our platform during a selected period. Learn more about it [here](https://docs.clarifai.com/portal-guide/control-center). | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [OpenAI o1-preview](https://clarifai.com/openai/chat-completion/models/o1-preview), a reasoning-focused AI model designed for complex problem-solving in math, coding, and science. Published [OpenAI o1-mini](https://clarifai.com/openai/chat-completion/models/o1-mini), a reasoning-focused LLM designed for complex problem-solving in math, coding, and science. Published [Llama-3.2-11B-Vision-Instruct]( https://clarifai.com/meta/Llama-3/models/llama-3_2-11b-vision-instruct), a multimodal LLM by Meta designed for visual reasoning, image captioning, and VQA tasks, supporting text and image inputs with 11B parameters. Published [Llama-3.2-3B-Instruct](https://clarifai.com/meta/Llama-3/models/llama-3_2-3b-instruct), a multilingual, instruction-tuned LLM optimized for dialogue and text generation tasks. | ## Python SDK​ |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | [**Private Preview**](https://docs.clarifai.com/product-updates/changelog/release-types) | Introduced enhanced model upload capabilities | Introduced a new method for model uploads to our platform. This integration includes: A new, more efficient model upload process utilizing Clarifai's runners. Migration of relevant files from runners-python to appropriate locations within the clarifai-python repository. Learn more about it [here](https://github.com/Clarifai/examples/tree/main/models/model_upload). | | ![improvement](/img/improvement.jpg) | Enhanced model upload capabilities with HuggingFace integration and concept handling | We've expanded our model upload functionality to improve integration with HuggingFace and enhance concept handling: Introduced a HuggingFaceLoader utility for seamless checkpoint downloads and concept fetching from HuggingFace for classification and detection models. Implemented new methods in `model_upload` for efficient checkpoint downloads and concept handling. Updated the model upload process to use the new HuggingFaceLoader for HuggingFace-related models. Changed the command-line interface to use `--model_path` instead of `-folder` for improved clarity. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Added "Collaborations" as a top-level link | We’ve introduced "Collaborations" as a top-level link, similar to "My Apps" and "Community," making it easier for users to access apps they’ve been invited to collaborate on. | | ![improvement](/img/improvement.jpg) | Made some visual enhancements | We fixed CSS artifacting issues in the app creation input fields, improving the visual experience. We fixed an issue where the password field in the sign-up modal was not displaying correctly. We improved tables across the platform, including updating header names to use Capital Case and resolving table height issues for a more consistent and polished appearance. We fixed a visual issue where the input field border in the "Create an App from a template" modal was displayed incorrectly. We fixed an issue where the icon for an app's cover image was displayed incorrectly in the app's sidebar. We fixed an issue where the "Template" tag, which identifies template apps on the "My Apps" listing page, was not displaying correctly. We fixed an issue where the validation text, which is shown when a user provides incorrect input values in the sign-up form, overlapped and obstructed other fields in the form. | | ![bug](/img/bug.jpg) | Fixed an issue with saving users' contact information | Previously, a bug during account creation caused contact information to be captured incorrectly. Even attempts to update the information on the account settings page would fail. We fixed it. | |![bug](/img/bug.jpg) | Fixed issues with the user-guided tour | We fixed some issues with the application guided tour flow for onboarding new users. Primarily, the guided tour is now only shown to logged-in users who have never used our platform before. Also, Org Users are not shown the guided tour because they already went through this process via their personal accounts. | | ![bug](/img/bug.jpg) | Fixed an issue with listing invited members to an organization | Previously, Org Contributors were able to view the pending members tab. Since inviting members is an Admin-only function, we now restrict Org Contributors, Org Users, and Team Contributors from accessing the pending invitations list. | | ![bug](/img/bug.jpg) | Fixed an issue with inviting users to an organization | Previously, users could not accept invitations to join an organization if the invitation was sent to an email with a different case (upper or lower) than the one associated with their account. We’ve now made email addresses case-insensitive when sending organization invitations. | | ![bug](/img/bug.jpg) | Fixed an issue with pricing display | We fixed an issue where the pricing information for base workflows was missing. The pricing information is now displayed in the upper-right section of a workflow page. | |![bug](/img/bug.jpg) | Fixed an issue with user registration | Previously, some users were able to create an account without consenting to the terms of service. Now, the "Create account" button remains disabled until the user consents by checking the terms of the service box. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced workflow versioning | You can now create workflow versions that track the changes of models, operators, agents, prompts, and their configurations within the pipeline nodes. | | ![bug](/img/bug.jpg) | Fixed RAG Prompter modification issues in public workflows | Previously, modifying a RAG Prompter model in a public workflow would unexpectedly create a new private model and version, causing save failures due to the incompatibility of private models in public workflows. This also led to workflow execution errors. We've resolved this by ensuring that modifications to existing nodes reuse the original model, either creating a new version or updating the existing one as appropriate. This fix maintains workflow integrity, prevents unintended privatization, and eliminates related execution errors. | | ![bug](/img/bug.jpg) | Fixed compatibility issue between Image Cropper and Multimodal-to-Text models in Workflows | Previously, when using an Image Cropper with a Multimodal-to-Text model in a workflow, the cropped image regions were not being properly processed. This was due to the Multimodal-to-Text model being incorrectly configured with a single 'Any' input type, causing the workflow runner to skip the cropped regions. We've addressed this by updating the model's input configuration to correctly specify [image, text] input fields. This fix ensures that all cropped image regions are now properly processed by the Multimodal-to-Text model, improving the accuracy and functionality of workflows using these components. | | ![bug](/img/bug.jpg) | Fixed an issue with the Concept thresholder | We fixed an issue where the concept thresholder was displaying concept ID while selecting concepts instead of concept names. Now, this issue is fixed, and it shows concept names correctly. | | ![bug](/img/bug.jpg) | Fixed an issue with the RAG-prompter metadata field update | Previously, when updating metadata, the old metadata was merged with the new metadata instead of being overwritten. We fixed the issue, now metadata gets correctly overwritten instead of being merged. | | ![bug](/img/bug.jpg) | Fixed issues with _max_results_ and _min_score_ sliders and metadata input in the workflow builder page | Previously, the _max_results_ slider allowed decimal inputs and had an inconsistent maximum value, while the _min_score_ slider's range was incorrect. Additionally, the metadata field was rejecting valid JSON input. We fixed these issues as follows: **Max Results:** The slider now only accepts whole numbers and correctly allows values up to 128, instead of being limited to 100. **Min Score:** We've adjusted the range to be between 0 and 1, with a step size of 0.01, ensuring more precise and appropriate score selection. **Metadata:** The system now properly accepts and processes valid JSON input in the metadata field, allowing users to input structured data as intended. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue with the model builder page | Previously, when users attempted to expand the Output Settings section while viewing model version details, a client-side exception would occur, showing the error: _"Application error: a client-side exception has occurred (see the browser console for more information)."_ This would crash the page. Now, the issue has been resolved, and users can seamlessly expand the _Output Settings_ section. | --- ## Release 11.0 # Release 11.0 **Release Date:** January 8th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | :::info Clarifai Product Roadmap We have a public roadmap that gives you a view of our upcoming plans and helps us gather your valuable feedback. You can explore the features we're considering, vote for the ones you want most, or submit new ideas. You can check it [here](https://portal.productboard.com/bd1rxfuyfbu6vqnmkva3mprx/tabs/1-under-consideration). ::: ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced Financial Manager role for [organizations](https://docs.clarifai.com/portal-guide/clarifai-organizations/) | This role provides access to key financial data, including current spending based on the organization's pricing plan and budget settings, enabling better financial oversight and management. | |![improvement](/img/improvement.jpg)| Expanded permissions for the [Labeler Lead](https://docs.clarifai.com/portal-guide/clarifai-organizations/members-teams/#3-labeler-lead) role | Labeler Leads can now create, modify, and delete concepts, giving them greater control over the labeling process. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) |Made improvements to the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | We've hidden the totals column in the following charts within the Control Center: `Total Model Predictions by ID` and `Total Model Predictions by Type`. This update streamlines the display for better clarity and focus. We've fixed an issue where the corresponding tab in the left sidebar would not be highlighted when viewing [a detailed report page](https://docs.clarifai.com/portal-guide/control-center/usage-dashboard#view-report-details) of a chart. The sidebar now properly reflects the selected section for a smoother navigation experience. We've fixed an issue where dates were not appearing on some charts in the Control Center. Pinning or unpinning [charts](https://docs.clarifai.com/portal-guide/control-center/usage-dashboard/#charts) now triggers a notification at the top of the screen, displaying either "Chart pinned to the Overview" or "Chart unpinned from the Overview," depending on the action. We've made improvements to the [calendar feature](https://docs.clarifai.com/portal-guide/control-center/#date-ranges) for a smoother and more flexible user experience: Users can now manually input dates to select a custom date range. Clicking the arrow in the dropdown window for selecting a custom date range now navigates through months one at a time, instead of two. Users can now select a single day, removing the previous limitation of a minimum two-day range. The current day is now highlighted with a non-filled circle for better visibility. The ability to select future dates has been disabled. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to the Python SDK _Learn more about them [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md)_. | Changed labels to be optional in Dataloaders to better support data ingestion pipelines in the `clarifai-datautils` library, enabling greater flexibility during data processing. Added model building logs to improve tracking and debugging. Added `user_id` field to the RAG class for better user identification. Added support for testing and running a model locally within a container. Added CLI support for the Model Predict functionality. Updated Dockerfile for `Sglang` to improve compatibility and performance. Updated available Torch images, along with code refactoring for better maintainability. Fixed an issue with model local testing to ensure smoother functionality. Removed `protobuf` from requirements to resolve conflicts with `clarifai-grpc`. Fixed edge case issues with bounding box information. Added support for downloading `data.parts` as bytes for increased flexibility. Changed default environment to `prod` for model uploads. Added tests for all stream and generate methods to enhance reliability. Added code coverage test reports to PRs for improved quality control. | ## Platform Improvements |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved navigation highlighting for top-level resources | Now, when users visit resource pages under the "My Apps" or "Community" sections, the corresponding navigation items are correctly highlighted. For example, visiting `/user-id/app-id/models/model-id` highlights the "My Apps" navigation item, visiting `/explore/apps` highlights the "Community" navigation item, and visiting `/settings/billing` results in no navigation item being highlighted, as it falls outside these sections. | ## Platform Bugs |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed team name update issue | We've fixed an issue that prevented users from updating a team's name on the organization settings page. You can now easily rename your teams without any problems. | |![bug](/img/bug.jpg) | Fixed default permissions selection issue | We've fixed an issue where the radio button for selecting default permissions for a team was not highlighting upon selection. You can now easily choose the default permissions when creating a new team. | | ![bug](/img/bug.jpg) | Fixed permission issue for organization ownership | We've fixed an issue where users without the necessary permissions could be added as owners of another organization. Previously, users who were not allowed to create multiple organizations could still be invited as owners. This is no longer possible, ensuring proper permission handling. | ## Workflow Bugs |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue with updating a workflow ID | Previously, attempting to edit a workflow and update its ID resulted in an error message stating, "ID already exists," even when no matching IDs existed in the system. This error would trigger immediately after entering the ID and persist even if the ID was partially removed. We fixed it, and workflow IDs can now be updated without errors. | --- ## Release 11.1 # Release 11.1 **Release Date:** February 5th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Audit Logging |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Clarifai's Audit Logging feature helps you monitor platform activities for better visibility, security, and governance. It captures detailed logs of operations so you can track what has been done, who has initiated it, and what has been the result of this action. | You can perform audit tracking via the Teams & Tabs tab within the Control Center. Learn more [here](https://docs.clarifai.com/portal-guide/control-center/teams-logs/). You can also perform audit tracking programmatically via our API. Learn more [here](https://docs.clarifai.com/api-guide/audit-log/). | ## Compute Orchestration (_Public Preview_) |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | [Compute Orchestration](https://docs.clarifai.com/portal-guide/compute-orchestration/) now supports Google Cloud Platform (GCP) | Compute Orchestration preview customers can now deploy dedicated compute clusters in GCP's us-east4 region. Additional regions will be added over time. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published DeepSeek models, which are free for a limited time on our Community platform | Released [DeepSeek-R1-Distill-Qwen-1_5B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-1_5B), a 1.5B-parameter dense model distilled from DeepSeek-R1 based on Qwen-1.5B. Released [DeepSeek-R1-Distill-Qwen-7B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-7B), a 7B-parameter dense model distilled from DeepSeek-R1 based on Qwen-7B. Released [DeepSeek-R1-Distill-Qwen-14B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-14B), a 14B-parameter dense model distilled from DeepSeek-R1 based on Qwen-14B. Released [DeepSeek-R1-Distill-Qwen-32B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-Distill-Qwen-32B), a 32B-parameter dense model distilled from DeepSeek-R1 based on Qwen-32B. | ## Data Utils |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Released [Data Utils](https://github.com/Clarifai/clarifai-python-datautils) as an open source Python library | Get a range of multimedia data utilities designed to streamline your data management and processing operations. [Image annotation loader](https://github.com/Clarifai/examples/blob/main/Data_Utils/Image%20Annotation/image_annotation_loader.ipynb) — A comprehensive framework for loading, exporting, and analyzing different annotated datasets. [Data ingestion pipelines](https://github.com/Clarifai/examples/blob/main/Data_Utils/Ingestion%20pipelines/README.md) — Load text files (PDF, DOC, etc.), transform and chunk the content, and upload it to the Clarifai platform. [Multimodal dataloader](https://github.com/Clarifai/examples/blob/main/Data_Utils/Ingestion%20pipelines/Multimodal_dataloader.ipynb) [Ready-to-use foundational pipelines](https://github.com/Clarifai/examples/blob/main/Data_Utils/Ingestion%20pipelines/Ready_to_use_foundational_pipelines.ipynb) | ## Platform Improvements |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made platform improvements | We've enhanced several elements of the top navigation bar for a more seamless experience. Notably, the drop-down menu for switching between an organization workspace and a personal workspace has moved from the upper right to the upper left. We've introduced the ability to automatically generate a cover image for a resource based on its short description. This capability is available only for users on [paid plans](https://www.clarifai.com/pricing). | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) |Improved the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | We've improved the Control Center experience on mobile devices for smoother navigation and better usability. We've improved the Overview page by adding animations that suggest charts users can pin. This allows users to easily access the charts they want to analyze. We've improved table columns to now sort items based on their data type: dates (oldest to newest), words (A to Z), and numbers (lowest to highest). We've added links in Usage & Operations charts that direct you to related charts in the Costs & Budget page, and vice versa. For example, the Total inputs chart now has a link in the lower right corner to view its associated cost chart. | | ![bug](/img/bug.jpg) | Fixed bugs in the Control Center | Previously, when you selected a date range in the Control Center and navigated away, the selection would not persist upon returning. Instead, it would revert to the default "Last 7 Days" setting. We fixed the issue to ensure the selection persists. Previously, the toast notification that appears when a chart is pinned could sometimes be hidden under the fixed header. We've fixed this issue to ensure the notification remains visible at the top. Previously, when selecting a date range in the calendar using options like ‘This week’, ‘This month’, or ‘This quarter’, the apply button would be disabled because future dates were inadvertently included in the selection. We fixed the issue. Additionally, if you click on a single date, the selection period will automatically reset to the chosen date. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK _Learn more about them [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md)_. | Enhanced model upload experience with a more streamlined process.Introduced model upload tests to ensure reliability.Updated Torch version in images and now delete the `.tar` file after every upload. Added local model run tests for better validation. Included `CLARIFAI_API_BASE` in test container for flexibility. Removed Triton requirements to simplify dependencies. Expanded tests for downloads and made various improvements. Made API validation optional for greater configurability. Added environment variable for logging control to enhance debugging. Updated base images for better performance and security. Optimized downloads from Hugging Face (HF) for efficiency. Now fetching `user_id` from an environment variable for easier configuration. Added HF token validation to improve security. Fixed model prediction methods when using `compute_cluster_id` and `nodepool_id`. Resolved model upload issues for smoother deployment. Enhanced error logging for better troubleshooting. | ## Base Workflow |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Changed the default [base workflow](https://docs.clarifai.com/api-guide/workflows/base-workflows/) for apps created via the API | Previously, when an application was created via the API without specifying a base workflow, the [Universal](https://clarifai.com/clarifai/main/workflows/Universal) workflow was used by default. Now, the default has been updated to the [General](https://clarifai.com/clarifai/main/workflows/General) workflow for better performance. | --- ## Release 11.10 # Release 11.10 **Release Date:** November 7th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Single-Click Deployments |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Introduced one-click deployments | Previously, users had to manually configure clusters and nodepools before [deploying a model](https://docs.clarifai.com/compute/deployments/deploy-model), with limited setup guidance. Now, the platform automatically recommends suitable instance types based on model requirements and creates clusters or nodepools if none exist — enabling seamless, one-click model deployment. | ## Environment Secrets |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Introduced environmental secrets | Environment secrets are encrypted values that can be used as environment variables in your workflows. Learn more about them [here](https://docs.clarifai.com/control/authentication/environment-secrets/). | ## Structured Outputs |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Added support for structured JSON outputs | You can now generate structured JSON outputs directly from Clarifai-hosted OpenAI-compatible models using Pydantic schemas. This feature ensures that model responses adhere to a defined data structure, making it easier to integrate outputs into downstream applications safely and reliably. Learn more about them [here](https://docs.clarifai.com/compute/inference/open-ai/#structured-outputs). | ## Toolkits |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Added support for [toolkits](https://docs.clarifai.com/compute/toolkits/) | Added SGLang toolkit support to the CLI `init` command — `clarifai model init --toolkit sglang`. Added Python toolkit support to the CLI `init` command — `clarifai model init --toolkit python`. | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [DeepSeek-OCR](https://clarifai.com/deepseek-ai/deepseek-ocr/models/DeepSeek-OCR), which achieves SOTA 96%+ OCR precision at 9-10× compression ratio, ~90% accuracy at 10-12× compression ratio, and massive production scalability with 200,000+ pages per day with a single A100-40G GPU. Published [GLM-4.6](https://clarifai.com/zai/completion/models/GLM_4_6), which unifies reasoning, coding, and agentic capabilities into a single model. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md) | Fixed missing `user_id` parameter issue in the CLI `local-runner` command. Added Model Deployment Workflow step after Model Upload in the CLI.Added optional protobuf response details in Pythonic models with parameter validation.Added `USER_ID` field to the configuration in CLI Model Init.Added user input prompt for OpenAI local runner initialization.Fixed `async_client` initialization issue.Disabled `async_stub` during `ModelClient` initialization.Fixed `UnboundLocalError` in model init when using `--model-type-id` without a toolkit.Added `config-lock.yaml` support to `clarifai pipeline upload`. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Updated the Control Center to provide unified reporting for all models, regardless of billing method | Previously, usage stats depended on how models were billed — ops, tokens, or compute time. Now, all models report operations, and LLMs also report tokens, ensuring consistent and transparent usage metrics across the platform. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Community search to deliver more relevant results | Previously, all search criteria, such as model ID, user ID, and descriptions, were weighted equally. Now, model IDs (for example, `gpt-oss-120b`) carry greater weight, making searches more accurate and relevant to specific models. | --- ## Release 11.11 # Release 11.11 **Release Date:** December 3rd, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | [**Breaking Change**](https://docs.clarifai.com/product-updates/changelog/release-types#release-types) | Decommissioning legacy models | We are upgrading our infrastructure, and as part of this transition, we will decommission most of our legacy models on December 31, 2025. Learn more [here](https://docs.clarifai.com/product-updates/upcoming-api-changes/decommission-legacy-models/). | | ![improvement](/img/improvement.jpg) | Made improvements| Only a curated set of featured models can now be run using the Clarifai Shared SaaS (Serverless) deployment option. To use any other models, users must deploy them to their own cluster and nodepool. | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Published new models | Published [Trinity Mini](https://clarifai.com/arcee_ai/AFM/models/trinity-mini), a 26B-parameter (3B active) sparse mixture-of-experts language model, engineered by Arcee AI for efficient inference over long contexts with robust function calling and multi-step agent workflows. Published [GLM 4.6](https://clarifai.com/zai/completion/models/GLM_4_6), which brings unified reasoning, coding, and agentic capabilities in a single model — with a 200K context window and stronger coding performance on real-world tools. | ## Playground |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements| Added the ability to sync prompts in the Playground's compare mode, eliminating the need to copy prompts between models. This makes it easy to compare different models by speed and performance. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements | Implemented a new design for account navigation and reordered menu items to improve usability and overall user experience. Added the ability for users to upload a custom profile picture, which replaces the default initials and appears across the Community (e.g., on shared model or app details). Updated code snippets to mask the user's PAT by default. The full value remains visible when the snippet is copied. Added the ability to mark all activity-related notifications as read in the notification icon at the upper-right corner of the navigation bar. Each notification also now displays its creation date and time. Added the ability to add and remove credit card details. The previous edit option has been removed. Added a quick guide on uploading models using the Python SDK. | | ![bug](/img/bug.jpg) | Fixed bugs | Fixed an issue where an unclear error message appeared when an access token was missing. A meaningful message is now provided. Fixed an issue where some invoices displayed missing or incorrect category names. Invoices now correctly show all category items. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) |Made improvements | Added an option to export the data of each chart as a CSV file. | | ![bug](/img/bug.jpg) | Fixed bugs | Fixed an issue where reloading the Control Center in an organization account redirected users away instead of refreshing the intended content. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md) | Added platform specification support to ```config.yaml``` for model versions. Added a ```--platform``` CLI option for model uploads. Added support for the new ```struct_value``` field in runner data utilities. Added support for including the deployment user ID. Removed model proto caching from ```ModelRunner```, ```ModelServicer```, and the server. Updated the Dockerfile base image Git hash. Refactored the ```Dockerfile.template``` for building Clarifai model runner images by introducing a multi-stage build that separates model asset downloading from final image creation, resulting in cleaner and more efficient builds. Fixed an issue by ensuring the model proto containing secrets is loaded once during server initialization and made available to all prediction requests. Added full support for the OpenAI Responses API (streaming and non-streaming) to the dummy model implementation, improved token usage accounting for both ```chat.completions``` and ```responses``` endpoints, and introduced comprehensive tests for this functionality. Added a validation mechanism to the model loading process in the ```Model``` class, improving reliability during initialization. Improved parsing of package names and versions from requirement lines, including support for dependencies defined with ```@``` and more consistent whitespace handling. Centralized and streamlined logic for reading environment variables and passing them to the ```ClarifaiAuthHelper```, improving maintainability and flexibility. Added ```visual-keypointer``` to the list of concepts-required model types. Improved the robustness of the ```clarifai model local-runner``` command by loading and validating model configuration earlier and adding stricter checks for model type consistency. Optimized model runner performance by loading the model proto once at initialization rather than requiring it for every prediction request. Added comprehensive environment validation to provide immediate feedback when users attempt to run model tests in unsupported environments, helping prevent confusion when tests fail. | --- ## Release 11.2 # Release 11.2 **Release Date:** March 4th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) **Public Preview** | Upgrading the [Labeling Tasks](https://docs.clarifai.com/portal-guide/labeling-tasks/create-a-task) UI tool | We are upgrading the task labeling experience. This upgrade unifies task labeling with the same underlying components from our single [Input-Viewer](https://docs.clarifai.com/portal-guide/input-viewer/) tool, ensuring a seamless and unified experience for users. This upgraded feature is currently in [Public Preview](https://docs.clarifai.com/product-updates/changelog/release-types/#release-types). To request access, please [contact us](https://www.clarifai.com/explore/contact-us). | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Improved the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | We have updated access privileges for different roles: [Organization Contributors](https://docs.clarifai.com/portal-guide/clarifai-organizations/security#scopes-and-access-levels-of-organization-members) can now access the Overview tab pages, Usage & Operations tab pages, and [detailed report pages](https://docs.clarifai.com/portal-guide/control-center/usage-dashboard#view-report-details) of their charts. Organization Users can now access the Overview tab pages, Usage & Operations tab pages, and detailed report pages of their charts. Financial Managers can now access the Overview tab pages, Usage & Operations tab pages, Costs & Budget tab pages, and detailed report pages of their charts. We have added an empty state for the Overview tab. If all data is hidden, empty visuals will be shown, and no charts will be displayed. We now display two decimal places for financial values. For example, $20.1 is shown as $20.10. We have fixed an issue with table sorting where the number 0 was not being handled correctly. Now, numerical sorting works as expected, ensuring that 0 is properly ordered along with other values. We have added cross-navigation links between the detailed report charts in the Usage & Operations tab and the Costs & Budget tab. These links allow users to seamlessly access related charts. For example, the detailed report page for the Total Number of Operations chart now includes a link in the lower right corner, directing users to the Cost of Operations chart. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to the Python SDK. Learn more about them [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Added support for local dev runners from the CLI. Used the non-runtime path for tests. Fixed local tests. Caught additional codes that models have at startup. Introduced three instances when checkpoints can be downloaded. Fixed dependency parsing. Used new base images and fixed Clarifai version. Disabled API validation in server.py. Fixed Docker test locally. Fixed Hugging Face (HF) checkpoints error. Fixed deployment tests. Fixed issue with filename being recognized as an invalid input ID. Updated Model Predict CLI. Set tests health port to None. Refactored model class and runners to be more independent. Added storage request inference based on tar and checkpoint size. | ## Data Utils |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Made improvements to the [Data Utils](https://docs.clarifai.com/sdk/data-utils/) library | Added support for DOCX and Markdown file formats. Introduced batch prediction for the ImageSummarizer pipeline. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the navigation bar | We have made further adjustments to the navigation bar and links for improved usability and accessibility. | | ![bug](/img/bug.jpg) | Fixed platform bugs | Fixed a sporadic JSON syntax error in the platform that occasionally disrupted the user experience by displaying an error screen with a “Back to Community” button. Fixed an issue where the “Last Updated” sorting did not reflect the most recent changes after actions like deleting an item in an app. Fixed an issue that prevented users from scrolling to the bottom of code snippets on the platform, causing some sections of the code to remain hidden. Fixed multiple errors when fetching organization roles, where the network tab and browser URL displayed an issue incorrectly stating that the user ID is not an organization. Fixed an issue where the input boxes in the pop-up window for creating an app displayed with reduced height. Fixed an issue where the tooltip text for the feedback icon incorrectly showed “Seed Feedback” instead of “Send Feedback.” Fixed an issue with icon styling in the app’s models listing table, where the search icon in the Model ID column was not displaying correctly. Fixed an issue where users were unable to access their apps after selecting the “My Resources” tab in the top navigation bar. | --- ## Release 11.3 # Release 11.3 **Release Date:** April 3rd, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Compute Orchestration |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced [Compute Orchestration](https://www.clarifai.com/products/compute-orchestration) | You can now easily deploy any model, on any compute, at any scale. These capabilities allow you to optimize your AI compute, avoid vendor lock-in, and control spend more efficiently. Introduced the "compute time" billable item to support this functionality. Learn more about it [here](https://docs.clarifai.com/compute/overview). | ## AI Playground |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced the AI Playground | The Playground interface is a testing battleground that allows you to quickly interact with powerful AI models without additional setup. Try it [here](https://clarifai.com/playground?model=Qwen2_5-VL-7B-Instruct). | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Upgraded the [Labeling Tasks](https://docs.clarifai.com/portal-guide/labeling-tasks/create-a-task) UI tool | We upgraded the task labeling experience. This upgrade unifies task labeling with the same underlying components from our single [Input-Viewer](https://docs.clarifai.com/portal-guide/input-viewer/) tool, ensuring a seamless and unified experience for users. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced a redesigned [homepage](https://clarifai.com/home) to help you [get started](https://docs.clarifai.com/getting-started/quickstart) with the Clarifai platform quickly| You can easily find trending AI models, test them in a playground, set up your [computing infrastructure](https://docs.clarifai.com/getting-started/first-deployment), browse your apps, and more — all in one place. You can also customize your homepage layout using the "Configure Home" button in the upper-right corner. | | ![improvement](/img/improvement.jpg) | Made platform improvements | Introduced a new `Infrastructure Manager` role, allowing users to create, modify, and delete clusters and nodepools within an organization. Removed the automatic app onboarding flow for new users upon signup. Apps are no longer required for certain actions, such as making [deployments](https://docs.clarifai.com/getting-started/first-deployment) with Compute Orchestration. We now automatically fetch a user's full name after signing up with SSO via Google or GitHub, eliminating the need to manually enter it during the signup flow. | |![bug](/img/bug.jpg) | Fixed platform bugs | Fixed an issue where users experienced an error when logging into the Clarifai platform due to a broken login state. Fixed an issue that caused certain app overview pages to crash. Fixed an issue where a disruptive top header appeared during the onboarding signup flow for new users. Fixed an issue in the onboarding signup flow where the job role and country dropdowns were appearing behind the "Complete your profile" modal. Fixed an issue where some model prices were displayed correctly when logged in but appeared as 0 when logged out. Fixed an issue where clicking the "Select All Scopes" button when creating a PAT or adding an app contributor did not select the "Deployment" package. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | Refined table layouts for improved clarity and readability. Introduced a new multi-column tooltip for charts, enhancing data visibility and readability. Fixed a color inconsistency where the tooltip card displayed an incorrect color when hovering over chart elements. The tooltip now accurately reflects the corresponding chart segment color. Fixed an issue where the usage dashboard failed to correctly report costs for deleted models. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK. Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Removed HF loader `config.json` validation for all Clarifai model type IDs. Added regex patterns to filter checkpoint files for download. Implemented validation for CLI configuration. Fixed Docker image name and introduced the `skip_dockerfile` option in the `test-locally` subcommand of the Model CLI. Enhanced the CLI login module for better authentication. Updated the CLI to enable local model testing without requiring remote access. Modified the default value of the `num_threads` field for optimized performance. Dropped support for Python 3.8, 3.9, and 3.10 in testing. Updated deployment testing configurations. Removed the `model_path` argument from the CLI. Added configuration for multi-threaded runners to improve execution efficiency. | --- ## Release 11.4 # Release 11.4 **Release Date:** May 9th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Developer Experience |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced a new Python-based method for model uploading and inference | Built with a Python-first, user-centric design, this flexible approach simplifies the process of working with models — enabling users to focus more on building and iterating, and less on navigating API mechanics. It streamlines inference, accelerates development, and enhances overall usability. Learn more about it [here](https://docs.clarifai.com/compute/models/inference/api) and [here](https://docs.clarifai.com/compute/models/upload/). | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [Llama-4-Scout-17B-16E-Instruct](https://clarifai.com/meta/Llama-4/models/Llama-4-Scout-17B-16E-Instruct), a powerful model in the Llama 4 series featuring 17 billion parameters and 16 experts for advanced instruction tuning. Published [Qwen3-30B-A3B-GGUF](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-GGUF), the newest in the Qwen series, featuring dense and MoE models with major improvements in reasoning, instruction-following, agent tasks, and multilingual support. Published [o4-mini]( https://clarifai.com/openai/chat-completion/models/o4-mini) model from OpenAI. It’s a smaller model optimized for fast, cost-efficient reasoning — it achieves remarkable performance for its size and cost, particularly in math, coding, and visual tasks. Published [o3](https://clarifai.com/openai/chat-completion/models/o3) model from OpenAI. It’s a well-rounded and powerful model across domains. It sets a new standard for math, science, coding, and visual reasoning tasks. | ## Playground |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Enhanced the model [Playground](https://docs.clarifai.com/getting-started/quickstart#step-2-run-your-inference-in-playground) experience | Added automatic mode detection based on the selected model — now intelligently switches between Chat and Vision modes for predictions. Improved model search and identification for a faster, more accurate selection experience.Introduced a Personal Access Token (PAT) dropdown, enabling users to easily insert their PAT keys into code snippets.Implemented dynamic pricing display that updates based on the selected deployment. The selected deployment ID is now automatically injected into the inference code. | ## Home Page |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Enhanced the Platform's [Home page](https://clarifai.com/home) | The Home page is now accessible to all users, with sections requiring login automatically hidden for non-logged-in users. Added a "Recent Activity" section that lets users view their most recent actions and operations performed on the platform. Made additional improvements to the Home page to enhance usability, performance, and overall user experience. | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the [Community](https://clarifai.com/explore) platform | Revamped the Explore page with refreshed visual designs, a featured models showcase, and categorized use cases such as LLMs and VLMs.Updated the individual model viewer page with an improved UI, direct access to the Playground, deployment listings, and additional enhancements. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Enhanced the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | Added Compute time metrics for Compute Orchestration to the Control Center: Added Compute Hours in the Overview tab. Added Compute Hours costs in the Costs tab. Added Compute Hours usage details in the Usage tab. Added Compute Orchestration operations to audit logging: Operations related to clusters, nodepools, and model deployments are now tracked and visible in the [Teams & Logs](https://docs.clarifai.com/portal-guide/control-center/teams-logs) tab within the Control Center. Introduced new, more efficient and stable chart types with improved tooltips for better data visualization and user experience. Enhanced the design of the ["Total Model Predictions by ID"](https://docs.clarifai.com/portal-guide/control-center/usage-dashboard#total-model-predictions-by-id) chart by making the chart clickable, allowing users to navigate directly to the corresponding model. Also introduced other UI refinements for a more intuitive experience. Adjusted hover cards on charts to stay within the viewport by dynamically lowering their position and adding scrollbars when content exceeds the visible area. | ## Org Feature |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Enhanced the [Org feature](https://docs.clarifai.com/portal-guide/clarifai-organizations/) | Previously, new organization accounts inherited the [plan](https://www.clarifai.com/pricing) of the user's personal account. Now, all new organization accounts are created on the Community plan by default, giving users the flexibility to upgrade as needed. This update applies to users on the Community, Essential, and Professional plans. Enterprise users are not affected. We improved the "Member Since" column in the organization members list table. It now displays when a member joined the organization, rather than when they assumed their current role. Restricted visibility of Settings pages by hiding those that users do not have permission to access, | ## Billing |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Enhanced the billing section | Redesigned the credit card management UI for a more intuitive user experience. Implemented validation to prevent adding duplicate credit card numbers. Added support for setting and changing the default credit card. | ## Sign-up flow |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the sign-up flow | Refined the sign-up flow to improve clarity and user experience across key steps. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)|Improved the Python SDK. Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Added support for [Pythonic models](#developer-experience) for a more native development experience.Fixed failing tests to improve overall stability.Improved CLI performance, now ~20x faster for most operations.Introduced config contexts in the CLI, with more enhancements on the way.Enhanced error messages for missing arguments and edge cases.Fixed return arguments in the model builder for correct outputs. | ## Platform Bugs |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg) | Fixed some bugs | Fixed some bugs in the top navigation bar to improve user experience across the platform. Fixed a bug on the Community Models page that caused flickering and disabled scrolling after applying a filter. Fixed a bug in the Control Center where pie chart labels overlapped, enhancing readability and visual clarity. Fixed a bug that prevented team members with the Model Trainer role from accessing applications in Organizations. | --- ## Release 11.5 # Release 11.5 **Release Date:** June 5th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Agentic Frameworks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Introduced support for [agentic frameworks](https://docs.clarifai.com/compute/agents/) | This signifies a major advancement, enabling the development and orchestration of multi-step, goal-directed AI agents that can autonomously reason, plan, and take actions across Clarifai's platform. Additionally, Clarifai now integrates the Model Context Protocol (MCP) and enables building [MCP servers](https://docs.clarifai.com/compute/agents/mcp). This allows Clarifai users to bridge their AI agents with enterprise data and external APIs, providing dynamic, real-time context. | ## OpenAI Inferences |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Introduced an [OpenAI-compatible](https://docs.clarifai.com/compute/models/inference/open-ai) API endpoint | This endpoint allows you to leverage your existing OpenAI API code and workloads to make inferences with Clarifai models. | ## Token-Based Billing |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Added initial support for token-based [billing](https://www.clarifai.com/pricing) for some large language models (LLMs) and image generation models | The token-based billing will align pricing with industry standards and better reflect the costs associated with these models. The token-based pricing will be gradually applied only to inference on the Community using models deployed with Clarifai's default Shared compute. [Dedicated compute instances](https://docs.clarifai.com/compute/deployments/deploy-model/) will continue to use per-compute-time billing, regardless of the type of model deployed. Also, the per-request billing will continue for legacy vision models on the Community. | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [DeepSeek-R1-0528-Qwen3-8B](https://clarifai.com/deepseek-ai/deepseek-chat/models/DeepSeek-R1-0528-Qwen3-8B), which improves reasoning and logic via better computation and optimization, nearing the performance of top models like O3 and Gemini 2.5 Pro. Published [Qwen2_5-Coder-7B-Instruct](https://clarifai.com/qwen/qwenCoder/models/Qwen2_5-Coder-7B-Instruct), a code-specific LLM series (0.5B–32B) with improved code generation, reasoning, and fixing. Trained on 5.5T tokens, the 32B model rivals GPT-4o in coding capabilities. Published [Claude Opus 4]( https://clarifai.com/anthropic/completion/models/claude-opus-4), a state-of-the-art large language model from Anthropic. It supports text and multimodal inputs and can generate high-quality, context-aware text completions, summaries, and more. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK. Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Restored support for pretrained model configuration files. Added `clarifai model init` CLI command to generate default files for model upload. Resolved an issue with model upload reliability. Improved handling of Clarifai config in URL construction. Updated code snippets for MCP and OpenAI integrations. Fixed issues related to model upload workflows. Resolved a bug in `MCPModelClass` that affected notifications. Enhanced `OpenAIModelClass` to streamline request processing, improve modularity, and simplify parameter extraction and validation. Fixed a bug in `OpenAIModelClass` to ensure full JSON responses are returned. Cleaned up `fastmcp` implementation for better maintainability. Added `OpenAIModelClass` to support integration with OpenAI-compatible API endpoints. Fixed utility functions for OpenAI messages and code snippets. Simplified OpenAI client wrapper functions. Added MCP integration, expanded CLI support, and improved handling of environment variables. Resolved various Python-specific bugs and syntax issues. Introduced a base class for visual classifier models. Now prints script path after model upload for easier reference. Added AMD-related enhancements. Removed redundant model downloads and improved error logging for gated Hugging Face repositories. Introduced a base class for visual detector models. Removed `rich` from requirements for a leaner dependency set. Added parameter support for inference configuration in both `model.py` and the frontend. Fixed query parameter retrieval in `ClarifaiAuthHelper` for Streamlit apps. Corrected `pyproject.toml` configuration. Resolved issues in local development runners. Fixed runner ID misassignment in local dev runner setup. Switched to `uv` and `ruff` to speed up testing, formatting, and linting. Replaced inappropriate use of `==` with `is` for Pythonic comparisons. Simplified local dev runner setup via CLI commands. Fixed indirect inheritance issue from `ModelClass`.| ## Node.js SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Node.js SDK. Learn more [here](https://github.com/Clarifai/clarifai-nodejs/blob/main/CHANGELOG.md). | Introduced a more efficient model inference technique for better performance. Enabled support for OpenAI-compatible API endpoints in Node.js environments.Fixed a minor issue with type exports for improved developer experience.Ensured package compatibility with both ESM and CommonJS (CJS) environments.Resolved security vulnerabilities by updating affected package versions.| ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to [Clarifai Organizations](https://docs.clarifai.com/control/clarifai-organizations/) | The create button is no longer visible to users who lack the necessary permissions to create resources within an organization. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) | The colors used in bar and line charts within the Control Center are now more consistent across different charts and when viewing detailed reports. This resolves a previous issue where colors would change unexpectedly, improving visual coherence. The Control Center now displays a "no data available" message instead of zeros when there is no data to show. This provides a clearer indication of the data status. [The Teams and Logs](https://docs.clarifai.com/control/control-center/teams-logs) tab in the Control Center is now hidden for users who do not have audit logging access. Previously, the tab was visible but empty or showed an error message. | ## Platform Improvements |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made platform improvements | Added a "Remember Me" option to the homepage login page, in addition to the one already existing in the login popup window. This provides users with a consistent login experience. Improved the reliability of cover image generation for resources, resulting in a better user experience. Implemented pagination for personal access token (PAT) keys. This will allow users with a large number of keys to navigate them more easily across multiple pages. Enhanced the homepage to improve overall usability. Hid the Recent Activity section on the homepage for users or organizations without access permissions. Made visual improvements to the billing interface, including a more appealing display of credit cards and a smoother plan change process. | --- ## Release 11.6 # Release 11.6 **Release Date:** July 8th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Local Runners |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Introduced Local Runners | Local Runners are a powerful feature of the Clarifai platform that lets you develop, test, and execute models on your local machine. Learn more about the feature [here](https://docs.clarifai.com/compute/models/upload/run-locally). | ## AMD and Oracle Support |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Added support for AMD and Oracle [instances](https://docs.clarifai.com/compute/cloud-instances) | You can now deploy models using AMD-based compute and Oracle Cloud infrastructure, giving you more flexibility in performance and cost options. | ## Token-Based Billing |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Started rolling out token-based [billing](https://www.clarifai.com/pricing) for some models | We’ve begun applying [token-based](https://docs.clarifai.com/product-updates/changelog/release115#token-based-billing) pricing to certain models on our Community platform as part of a gradual rollout. | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [Gemma-3n-E2B](https://clarifai.com/gcp/generate/models/gemma-3n-E2B-it-GGUF-4bit-text) and [Gemma-3n-E4B ](https://clarifai.com/gcp/generate/models/gemma-3n-E4B-it-GGUF-4bit-text) models. Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. Gemma 3n models are designed for efficient execution on low-resource devices. | ## Workflow Pipelines |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to pipelines _(pipeline steps are reusable components that can be used in Clarifai workflows)_. | Refined pipeline execution logic and validation to better support complex, multi-step workflows. Enhanced access control (RBAC) to improve management of execution context and user permissions. Introduced CLI support for uploading pipeline steps, with support for multiple steps per version — enabling greater reuse and faster iteration. Added step scoping and feature-flag-controlled RPCs to enable safer rollouts across shared infrastructure. Resolved image URL resolution issues to ensure accurate references in pipeline step configurations. | ## Playground |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Made improvements to the [Clarifai Playground](https://docs.clarifai.com/getting-started/quickstart-playground) | The Playground page is now publicly accessible — no login required. However, certain features remain available only to logged-in users. Added model descriptions and predefined prompt examples to the Playground, making it easier for users to understand model capabilities and get started quickly. Added Pythonic support in the Playground for consuming the new model specification. Additionally, improved the Playground user experience with enhanced inference parameter controls, restored model version selectors, and clearer error feedback. | ## Compute UI |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Made improvements to the Compute UI | Completely rebuilt the Compute UI for a more modern and intuitive user experience. Finalized key components, including redesigned list views, an improved create flow, updated tables, and overall usability enhancements. Enhanced alignment between frontend state and backend orchestration logic for a smoother, more predictable experience. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the Python SDK. Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Added per-output token context tracking for batch operations. Introduced a new `set_output_context()` method for models to specify token usage per output. Improved token usage tracking in ModelClass using thread-local storage. Enhanced batch processing support with an ordered token context queue. Fixed token context ordering in batch operations using a FIFO queue approach. Temporarily disabled test_client_batch_generate while implementing token tracking improvements. Fixed legacy proto support for improved compatibility. Added authentication support to the URL fetcher for SDH-protected URLs. Fixed and validated code snippets; added automated tests for snippet accuracy. Included model listing functionality in both CLI and method interfaces. Implemented terminal prompt to ask users whether to create a new app if the specified app doesn’t exist. Introduced asynchronous `predict` endpoints (v2). Added Model Utils module to the SDK for shared logic. Enabled model-level authentication when setting the runner. Improved local development tooling and URL helper utilities. Added proactive validation of code and requirements before upload to catch issues early. Integrated `uv` into the build process for faster and more reliable builds. Removed an unused parameter in the `VisualClassifier` class to clean up the API. Added support for `/responses`, `/embeddings`, and `/images/generations` endpoints in the OpenAI class. Fixed data display issues and updated OpenAI parameter handling for better control. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Made platform improvements | Improved compute time billing by increasing billing granularity beyond the millicent level, enabling more accurate usage billing. Introduced dynamic code snippets in the UI to showcase the latest ways to invoke models. Improved robustness by resolving sorting issues in pricing and enhancing handling of incomplete or missing context IDs. Improved the Community Home page by showing recent activity from the past 7 days, adding a "Cluster" grouping to organize compute-related operations, and making the deployments table more compact when displaying few entries. Users are now set to private by default immediately after sign-up, enhancing privacy over profile visibility. Improved the [Control Center](https://docs.clarifai.com/portal-guide/control-center/) by adding a visible timestamp showing when the dashboard was last refreshed, making it easier for users to confirm their billing data is up to date. Also optimized the response size of dashboard items for faster loading and better performance. | | ![bug](/img/bug.jpg)| Fixed platform bugs | Fixed an issue that prevented users from editing the description or details of a model. Fixed an issue where the sorting selection in the apps dropdown on the Task Labeling page did not persist when switching between apps. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Made improvements to [Clarifai Organizations](https://docs.clarifai.com/control/clarifai-organizations/) | Personal access tokens (PATs) are now shown in the PAT dropdown menu, regardless of whether the user is operating under their personal account or within an organization context. The Join Org notification in the navbar has been updated to match the design of the Join Org modal/page, ensuring a consistent behavior. Any user with a verified email address that matches the one used in an organization invitation can now accept or decline the invitation. Users receiving organization invites will now see a persistent popup upon login until they accept or decline the invitation. | --- ## Release 11.7 # Release 11.7 **Release Date:** August 7th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Developer Plan |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Introduced a Developer Plan | This plan provides access to [Local Runners](https://docs.clarifai.com/compute/local-runners/) and open-source models at a promotional price of $1 per month. Learn more [here](https://www.clarifai.com/pricing). | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [GPT‑OSS-120B and GPT‑OSS-20B](https://clarifai.com/openai/chat-completion/models/gpt-oss-120b), OpenAI's latest state-of-the-art open-weight language models, designed for real-world reasoning, tool use, and on-device deployment. Published GPT-5 family of models, which spans three tiers: [GPT-5](https://clarifai.com/openai/chat-completion/models/gpt-5) for advanced reasoning and generative tasks, [GPT-5 Mini](https://clarifai.com/openai/chat-completion/models/gpt-5-mini) for fast, cost-effective real-time use, and [GPT-5 Nano](https://clarifai.com/openai/chat-completion/models/gpt-5-nano) for ultra-low-latency edge and budget-sensitive deployments. Published [Qwen3-Coder-30B-A3B-Instruct](https://clarifai.com/qwen/qwenCoder/models/Qwen3-Coder-30B-A3B-Instruct), a high-performing, efficient model with strong agentic coding abilities, long-context support, and broad platform compatibility. | ## Ollama Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Added support for running Ollama models locally | You can now download and run Ollama models locally with Clarifai’s Local Runners. Learn more [here](https://docs.clarifai.com/compute/local-runners/ollama). | ## Playground |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the [AI Playground](https://docs.clarifai.com/getting-started/quickstart-playground) with a side-by-side comparison view | This allows users to easily test and compare different models, or the same model on different instances, to select the most suitable option. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the Python SDK. Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Added a quick fix for local runner signature handling Added a check to skip code generation when context is not set Ensured pipeline_steps are properly used in templates Fixed issues related to nodepool creation Improved pipeline status code checks for better reliability Implemented various bug fixes and enhancements for pipelines Added list / ls CLI commands for pipelines and pipeline steps Fixed the broken link to the PAT account settings pageAdded support for verbose logging in OllamaEnhanced error messages for Pythonic models Improved logging experience during login Refined logging output for the Local Runner Added CLI config context support to `BaseClient` authentication for easier environment switching. Introduced live logging functionality for model runners to monitor outputs in real-time. Unified context management under a single `config` command for consistency and simplicity. Added a utility function to return both gRPC stub and channel in one call. Added validation checks for Local Runner requirements during initialization. Improved error handling for failed URL downloads to provide clearer feedback. Included Playground URLs in Local Runner logs for easier navigation and debugging. Added unit tests for toolkits to ensure reliability and code quality. Improved logging in the Local Runner CLI for better clarity and troubleshooting. Formatted client scripts using `black` for consistent and clean code style. Added support for downloading GitHub folders and specifying toolkits in `model init`. Improved handling of `PAT` and `USER_ID` for better authentication reliability. Fixed thread flag handling in Local Runner and added validation for invalid users. Implemented PAT token validation during the `clarifai login` command. Standardized Local Runner naming across the SDK for consistency. Fixed the `pipelinestep upload` command to properly parse all `compute-info` parameters and preserve custom Dockerfiles. Resolved base model template import and return issues in the SDK. Set the default `pool_size` flag to 1 for local development runner threads. Updated constants used by the Local Runner for better configuration control. Added support for the `--version` flag in the Clarifai CLI for easier version tracking. Improved handling of `model_type_id` and enhanced configuration management. Enabled support for specifying `deployment_user_id` in the `Model` class to refine runner selection. Enhanced the `model init` command by allowing initialization from a GitHub repository for greater flexibility. Fixed CLI `PATH` handling issues on Windows systems. Corrected issues in the code generation script for improved output consistency. Added an alias for the `pipelinestep` CLI command and significantly increased test coverage for the `clarifai.runners.pipeline_steps` module. Improved CLI documentation and added detailed help messages for various model-related commands. Set the default number of gRPC server threads to `CLARIFAI_NUM_THREADS` or 32, if unspecified. Integrated configuration contexts into the `model upload` CLI command for environment-specific behavior. Introduced a `pipeline run` CLI command similar to `model predict` for easier pipeline execution. Updated `requirements.txt` to reflect the latest supported protocol version. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Made some platform improvements | Token-based models now clearly show accurate input and output pricing, including when used within workflows. Workflow pricing now displays multiple pricing tiers for each model included in the workflow. Updated the Settings sidebar to support URL-based account switching. Access to any Settings page now requires this URL format: `/userOrOrgId/settings/{resource-type}`. Added support for redirecting users back to their original source page after login or sign-up, where applicable. Added the "Create Organization" flow as a follow-up step after the new user onboarding process. | |![bug](/img/bug.jpg) | Fixed some bugs | Fixed an issue where a renamed app continued to show its old ID in the Settings page and deletion toast notification. Fixed an issue where accessing unauthorized resources triggered multiple error pop-ups; it now shows a single, graceful notification. Fixed an issue where both My Resources and Community tabs were highlighted when opening an app under My Resources in Org accounts; now only My Resources is correctly highlighted. Fixed an issue where reordering Home page card layouts via ‘Configure Home’ did not save the intended order. Fixed an issue where the recent activity section on the Home page displayed incorrect information when users performed operations across multiple apps. Fixed an issue where the Sign Up and verify email pages did not redirect after successful authentication, or triggered the organization onboarding modal. | --- ## Release 11.8 # Release 11.8 **Release Date:** September 3rd, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Unified Navigation Experience |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Made unified navigation enhancements | We’ve moved the main navigation elements into the collapsible left sidebar. This streamlines the UI, frees up valuable screen real estate, and delivers a discoverable navigation experience across the entire platform. | ## Billing |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Made changes to billing system | Introduced monthly spending limits of $100 for Developer and Essential plans, and $500 for the Professional plan. You can contact us if you require higher limits. Added a new credit card pre-authorization process. A temporary charge of $50 for Developer, $100 for Essential, and $500 for Professional plans is applied to verify card validity and available funds. The amount is automatically refunded within seven days. | ## Control Center |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made improvements to [Control Center](https://docs.clarifai.com/control/control-center/) | Added half-size widgets in the Control Center configure page, allowing users to resize charts to half their original size. This enhancement makes it easier to rearrange charts and optimize the layout for side-by-side comparisons. Fixed the Stored Inputs Cost chart to correctly display the average cost for the selected period instead of showing an incorrect total. Enhanced the date grouping for charts, where periods longer than one month now display weekly aggregated data for better readability. Empty states have been added to charts to display meaningful messages when no data is available, rather than just showing zero. Added cross-links between compute cost and usage charts, allowing users to navigate easily between these views. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Improved the Python SDK. Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md). | Fixed Local Runner CLI commandUpdated protocol and gRPC versionsPrevented downloading of original checkpointsIntegrated secrets into SDKFixed num_threads settingFixed pip checks when cache is brokenCorrected usage setting on OpenAI responsesAdded stream_options validation for internal streaming model uploadAdded missing packaging dependencyEnsured JSON errors are always returned on OpenAI callsSet default to 32 threadsFixed ruff and dependency-related issuesFixed model upload deploymentAdopted method signature for Local RunnerAdded user confirmation to prevent Dockerfile overwrite during model upload | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Made improvements to platform| Added a public resource filter button that lets users quickly view resources shared with the Community. This feature makes it easy to filter and explore resources based on their public visibility. Improved the playground to show a clear error message when users exceed streaming prediction limits. Extended the login session duration for users logging in via Google or GitHub SSO to seven days, which was previously much shorter. | |![bug](/img/bug.jpg) | Fixed some bugs | Fixed an issue where the toast notification displayed the old app ID when deleting a renamed app, instead of showing the updated ID. Fixed an issue where users were not redirected to the intended page after login, causing them to get stuck. Fixed an issue where the model page layout expanded incorrectly in width. Fixed issues in the Recent Activity section of the Community home page where resource ID links were not visible on smaller screens, and where audit logs displayed an incorrect target app ID. | --- ## Release 11.9 # Release 11.9 **Release Date:** October 13th, 2025 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Reasoning Engine |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Clarifai has launched a Reasoning Engine optimized for agentic AI inference | In independent benchmarks conducted by Artificial Analysis on GPT-OSS 120B, the Clarifai Reasoning Engine set new records on standard GPUs: 544 tokens/sec throughput, 3.6s time-to-first-token, and $0.16 per million tokens. Learn more [here](https://www.clarifai.com/press-release-clarifai-launches-reasoning-engine-optimized-for-agentic-ai-inference). | ## Toolkits |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Added support for toolkits | Added support for initializing models with the [`vLLM`](https://docs.clarifai.com/compute/local-runners/vllm), [`LMStudio`](https://docs.clarifai.com/compute/local-runners/lmstudio), and [`Hugging Face`](https://docs.clarifai.com/compute/local-runners/hf) toolkits for local runners. | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Published new models | Published [Qwen3-Next-80B-A3B-Thinking](https://clarifai.com/qwen/qwen3/models/qwen3-next-80B-A3B-Thinking), a 80B-parameter, sparsely activated reasoning-optimized LLM that delivers near-flagship performance on complex reasoning tasks with extreme efficiency in training and ultra-long context inference (up to 256K tokens). Published [Qwen3-30B-A3B-Instruct-2507](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-Instruct-2507), which improves comprehension, coding, multilingual knowledge, user alignment, and has 256K long-context handling. Published [Qwen3-30B-A3B-Thinking-2507](https://clarifai.com/qwen/qwenLM/models/Qwen3-30B-A3B-Thinking-2507), which is an enhanced version with significantly improved reasoning, general capabilities, user alignment, and a long-context understanding. | ## B200s and GH200s |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Added new Vultr [cloud instances](https://docs.clarifai.com/compute/cloud-instances) | Introduced competitively priced B200 instances operating from Seattle and GH200 instances powered by Vultr. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Made some platform improvements | Added short [role descriptions](https://docs.clarifai.com/control/clarifai-organizations/members-teams#roles-for-members) in the Invite Member modal to clarify each member’s permissions when inviting them to an organization or modifying their access. In the [Control Center](https://docs.clarifai.com/control/control-center/), charts for computer vision models (displaying predictions) and for LLMs (displaying tokens) are now presented separately, each with its own usage and cost information. | | ![bug](/img/bug.jpg) | Fixed some bugs| Fixed an issue that prevented the signup/login modal from appearing on the Compute page ([clarifai.com/compute](https://clarifai.com/compute)). | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md) | Introduced a new `patch_version` method in the Model class and integrated it into local runner workflows. Improved logging by highlighting example code scripts printed during local runner workflows.Changed the default local development model type from `text-to-text` to `any-to-any`. Fixed an issue with converting gRPC response enums to integers during runner creation. Fixed a `TypeError` when parsing checkpoint size from an environment variable. Enabled health probe support, allowing `ModelClass` implementations to define liveness/readiness checks. Improved interactive pipeline initialization with user prompts that replace placeholder TODO values. Implemented Git registry metadata capture during model upload with model-scoped change detection. Local runner now automatically uses the latest local-dev model version. Reduced friction by continuing to leverage a single prebuilt AMD base image. Updated type hints and docstring descriptions across all major files in the `clarifai/client` folder for better code quality, maintainability, and developer experience. Improved overall Model CLI UX with consolidated flags, clearer help text, and better error surfacing. Updated the `clarifai model predict` CLI to align with recent Pythonic model changes. Updated the local runner default API base URL. Refined logging in model and pipeline step builders for clearer diagnostics. Added pagination support to pipeline log monitoring, returning all entries beyond the first 50. | --- ## Release 12.0 # Release 12.0 **Release Date:** January 8th, 2026 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Pay-As-You-Go Plan |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced a simple, flexible Pay-As-You-Go plan | The new plan has no minimum monthly commitments and far fewer feature gates. Just prepay credits, build, and go. Additionally, the new plan comes with a one-time $5 welcome credit after verifying your phone number and auto-recharge capability. Learn more [here](https://docs.clarifai.com/control/account-billing#billing). | ## Pipelines |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | [**Public Preview**](https://docs.clarifai.com/product-updates/changelog/release-types#release-types) | Introduced the ability to create, run, and manage pipelines via the Clarifai CLI | Clarifai Pipelines let you design and run asynchronous, long-running, multi-step processes for complex AI and MLOps tasks. Learn more [here](https://docs.clarifai.com/compute/pipelines/). | ## Multi-Nodepool Deployment |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced model routing capabilities | We support universal model routing to provide high availability and optimal load distribution across all model services. This allows intelligent routing logic across multiple model-serving nodepools, including spillover handling and routing across different data centers. Learn more [here](https://docs.clarifai.com/compute/deployments/clusters-nodepools#multi-nodepool-deployment). | ## Vercel AI SDK Integration |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Clarifai is now available as an inference provider in the Vercel AI SDK | You can use Clarifai-hosted models directly through the OpenAI-compatible interface in `@ai-sdk/openai-compatible`, with no changes to your application logic. Learn more [here](https://docs.clarifai.com/compute/inference/vercel). | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published two new reasoning models from the Ministral 3 family| Published [Ministral-3-3B-Reasoning-2512](https://clarifai.com/mistralai/completion/models/Ministral-3-3B-Reasoning-2512), a compact model in the "Ministral 3" family. It delivers high-performance, open-weight capabilities while remaining efficient and deployable in realistic hardware settings. Published [Ministral-3-14B-Reasoning-2512](https://clarifai.com/mistralai/completion/models/Ministral-3-14B-Reasoning-2512), the largest model in the “Ministral 3” family. It delivers high-performance, open-weight capabilities close to much larger models while remaining efficient and deployable in realistic hardware settings. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md) | Fixed issues with local model runnersFixed checkpoint download failures when `hf_transfer` was not installedResolved compatibility conflicts with the latest vLLMAdded a comprehensive artifact management system for both the SDK and CLIIntroduced interactive `config.yaml` creation during the model upload processAdded Container and Env model support for local runnersAdded comprehensive test coverage for the `MCPConnectionPool` connection lifecycleUpdated status codes and descriptions for model runner failure casesPrevented a `TypeError` during model version creationFixed a runner ID bug affecting local runnersFixed user verification issues in the development environmentUpgraded `urllib3` to version `>= 2.6.2`Added retry logic to OpenAI API calls and fixed related test mocksFixed a `TypeError` when `accelerator_type` is `None` in `config.yaml`Improved local runner handling of duplicate runner ID errorsAdded `CLARIFAI_HF_TOKEN` to the CLI contextAdded tests for the local runner CLI commandRemoved automatic file generation during model operations | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Made some improvements | Added cleaner filters in the Control Center, making charts more intuitive and easier to navigate. Improved the Team & Logs tab in the Control Center to ensure that today’s audit logs are included when selecting the last 7 days. Enabled stopping responses in the right panel when using the Playground’s [Compare mode](https://docs.clarifai.com/getting-started/playground/#compare-models). | | ![bug](/img/bug.jpg)| Fixed some bugs | Fixed various signup and login bugs, resulting in a more reliable and smoother user experience. Fixed an issue in the mobile view of the Configure Home settings where enabling one option unexpectedly disabled another, causing inconsistent behavior. Fixed an issue where Predict and Generate methods produced an error in the Playground. Fixed an issue with the 6-month date range in the Control Center. | --- ## Release 12.1 # Release 12.1 **Release Date:** February 3rd, 2026 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Clarifai Artifacts |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced a new artifact management system for Clarifai apps| You can now programmatically manage binary and file-based assets — such as model weights, checkpoints, and configuration files — using the Python SDK or the CLI. You can create, upload, download, list, inspect, and delete artifacts and their versions. | ## Cessation of the Community Plan |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Retired the Community Plan | We previously offered a freemium Community Plan that included a limited number of monthly operations at no cost. All Community Plan users have now been migrated to our new Pay-As-You-Go plan, which provides a more sustainable and competitive pricing model. All users who verify their phone number receive a $5 free welcome bonus. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the Python SDK Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md) |Added the `load_concepts_from_config()` method to `VisualDetectorClass` and `VisualClassifierClass` to load concepts from `config.yaml`.Added a Dockerfile template that conditionally installs packages required for video streaming.Fixed deployment cleanup logic to ensure it targets only failed model deployments.Implemented an automatic retry mechanism for OpenAI API calls to gracefully handle transient `httpx.ConnectError` exceptions.Fixed attribute access for OpenAI response objects in agentic transport by using `hasattr()` checks instead of dictionary `.get()` methods. | ## SMS Verification |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Introduced SMS verification | We implemented SMS verification during the account registration process to reduce bots usage and multi-account abuse. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the home page| Previously, the Quick Compute section on the home page offered two deployment options — Basic and Advanced. We have simplified it to a single quick deployment option. | | ![improvement](/img/improvement.jpg)| Made Pay-As-You-Go UI updates| We styled the credit section to make it clear that it is clickable and added new columns to show both the original credit grant amount and the remaining balance. We fixed a bug that prevented users from marking a different credit card as default. | | ![bug](/img/bug.jpg) | Fixed some bugs| Fixed a bug where redirection did not work correctly after deleting a team. Removed empty lines from example code snippets that caused issues when copying and pasting them elsewhere. Fixed an issue where the “Mark as read” button in the notification bar remained clickable when it should have been disabled. | --- ## Release 12.2 # Release 12.2 **Release Date:** March 5th, 2026 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Clarifai CLI |Status |Change |Details| |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Streamlined model-to-production workflow | Introduced a streamlined 3-command workflow (`model init` → `model serve` → `model deploy`) to take models from scaffold to production in minutes. Learn how to get started quickly [here](https://docs.clarifai.com/getting-started/upload-model).Unified deployment into a single `model deploy` command with automatic infrastructure provisioning.Added structured deployment phases with progress indicators and next steps.| | ![improvement](/img/improvement.jpg) | Deployment lifecycle management | Added `model status`, `model logs`, and `model undeploy` commands.Introduced direct `deployment` commands for managing deployments by ID.Maintained backward compatibility with legacy flags and configs.| | ![improvement](/img/improvement.jpg) | Intelligent infrastructure & instance handling | Added GPU auto-selection based on model VRAM and toolkit requirements.Enabled multi-cloud instance discovery with GPU shorthands and legacy normalization.Added custom Docker base image support for faster toolkit builds.| | ![improvement](/img/improvement.jpg) | Enhanced local development & runners | Enhanced `model serve` with env, Docker, and standalone gRPC modes.Added concurrency controls and optional Docker image retention.Added health-check configuration for the `clarifai model local-runner` via `--health-check-port`, `--disable-health-check`, and `--auto-find-health-check-port` flags.| | ![improvement](/img/improvement.jpg) | Authentication & context Management | Improved authentication with interactive/non-interactive [`login`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-login) and flexible [`logout`](https://docs.clarifai.com/resources/api-overview/cli#clarifai-logout) options. Added named CLI contexts with `--context` overrides.Expanded `whoami` with profile inspection, org listing, and JSON output.| | ![improvement](/img/improvement.jpg) | Model & app management enhancements | Enhanced `model predict` with streaming auto-detection, method inference, JSON output, and deployment routing.Implemented early HuggingFace gated-model validation with actionable errors.Added full CRUD support for apps with convenient command aliases.| | ![improvement](/img/improvement.jpg) | Configuration & performance improvements | Simplified `config.yaml` with automatic normalization and smart defaults.Improved CLI startup performance through lazy module loading.Improved output formatting with clickable URLs, structured sections, and command hints.| ## Pipelines |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | [Public Preview](https://docs.clarifai.com/product-updates/changelog/release-types#release-types) | Introduced Training on Pipelines to align training workflows with production-grade deployment and scalable infrastructure | Introduced support for initializing pipelines using templates — including training templates — directly via the CLI. Launched a new UI experience for training models seamlessly within pipelines. _**Note:** Training on Pipelines enables training of models fully compatible with the Clarifai Compute Orchestration platform. It also allows training to leverage dedicated provisioned compute resources._| | ![improvement](/img/improvement.jpg) | Made improvements to pipeline steps | Enhanced the `pipeline step init` command to include a `visibility` field. Users can now set pipeline steps to be publicly visible during initialization through both the CLI and builder APIs. By default, pipelines and pipeline step templates are created with `PRIVATE` visibility. | ## Artifacts |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Made improvements to provide users with full control over artifact lifecycle management| Artifacts no longer expire automatically by default. Artifacts are only deleted if an `expires_at` value is explicitly set at upload time. The CLI now displays the `latest-version-id` alongside artifact visibility for easier version management and referencing. | ## Playground |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Made Playground improvements| Enhanced the Playground with major upgrades to the Universal Search experience, including multi-panel (compare mode) support, improved workspace handling, and smarter model auto-selection. Model selections are now panel-aware to prevent cross-panel conflicts, workspace context is more robust and explicit, and the UI can optionally display simplified model names for a cleaner experience. | ## Deployments |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made various improvements to deployments | Dynamic nodepool routing now allows multiple nodepools to be attached to a single deployment, with configurable scheduling strategies. Deployment visibility has been improved with the introduction of status chips and enhanced list views across Deployments, Nodepools, and Clusters. DigitalOcean and Azure have been added as supported cloud instance providers. Support has been added for explicitly starting and stopping deployments, enabling users to preserve configurations without deleting and recreating deployments. A redesigned Deployment details page has been introduced, with expanded status visibility across tables and the Playground. | ## Platform |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made various improvements to the platform | A new Model Library UI has been launched to provide a more streamlined experience for browsing and exploring models. Several platform UX enhancements were introduced, including adding Universal Search to the navbar and refreshing user listing pages. Introduced a new account experience. Rolled out a new Home 3.0 interface. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Deprecation** | Removed modules | Modules previously extended Clarifai’s UIs and enabled customized backend processing. Support for Modules has now been fully dropped, and all associated module components have been removed from the platform. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Introduced new features and enhancements to improve model management| Added OpenAI handler support for fetching featured models directly by model ID. Introduced stream cancellation support. Added model admission control hooks. Enabled graceful rollovers between model versions during deployment. Fixed an issue where `PatchNodepools` overwrote the maximum instance count. Fixed an issue where deployments reused outdated compute information. Updated the API to report live deployment replica counts and nodepool node counts. Improved error responses for clearer diagnostics. Delivered general performance improvements. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)|Improved the Python SDK Learn more [here](https://github.com/Clarifai/clarifai-python/blob/master/CHANGELOG.md) | Prevented environment variable clobbering in tests by using `@patch.dict` and correcting patch paths after the lazy-loading refactor.Fixed the ModelRunner health server starting twice, which could cause “Address already in use” errors; added support to disable the health server and automatically select an available port.Reduced overhead in the ModelRunner admission-control polling loop to improve runner efficiency. Fixed a serializer regression affecting runner serialization utilities.Added admission-control support for model runners.Added `openai` as a core dependency. Removed the `inference_compute_info` requirement for local model runners. Relaxed the `clarifai-protocol` version constraint from `==0.0.35` to `>=0.0.35, Resolved dependency version-constraint issues.Fixed authentication issues in the model deployment CLI.Added support for passing `num_threads` from `config.yaml` into the model version protobuf.Switched Dockerfile templates to use `tini` as the default entrypoint to improve signal handling and zombie process reaping in runner containers.Refactored the Stdio Model Context Protocol (MCP) server to improve logging clarity and remove unused legacy code.| --- ## Release 12.3 # Release 12.3 **Release Date:** April 8th, 2026 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Agent Skills |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Released Clarifai Skills| Clarifai Skills are specialized prompt templates that transform AI coding assistants — Claude Code, Cursor, Codex, and more — into Clarifai platform experts. Learn more about them [here](https://docs.clarifai.com/compute/agents/agent-skills). | ## Model Training |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added model pipeline training from templates | You can now train models using pipeline templates, enabling a streamlined, configuration-driven training workflow. | | Deprecation | Deprecated legacy model training (Triton + Kubeflow) | Legacy model training methods using Triton and Kubeflow have been deprecated.Note: Transfer Learn training remains available. | ## Request Routing |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg)| Improved how Clarifai routes prediction requests for optimal performance| Added KV cache affinity to route requests to replicas with relevant cache state.Added session-aware routing to keep user requests on the same replica. Reduced cold starts with automatic pre-warming of popular instances. Added prediction caching for identical input + model + version combinations. Learn more about them [here](https://docs.clarifai.com/compute/inference/routing/). | ## UI Updates |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Updated Compute List and View pages | Refreshed the List and View pages for deployments, nodepools, and clusters with improved layouts and information display. | | ![improvement](/img/improvement.jpg) | Updated the log viewer component | Improved the log viewer UI for better readability and navigation of model and runner logs. | | ![improvement](/img/improvement.jpg) | Updated the Home experience | Refreshed the Clarifai platform Home page with an improved experience for navigating resources and getting started. | | ![improvement](/img/improvement.jpg) | Updated the model page UI | Redesigned model page with an improved layout and user experience. | ## Python SDK ### Model Serving & Deployment |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added `clarifai model deploy` command and simplified `clarifai model init` | New `clarifai model deploy` command with multi-cloud GPU discovery and a zero-prompt deployment flow.Simplified `config.yaml` structure for model initialization. | | ![improvement](/img/improvement.jpg) | Smart resource reuse and private-by-default for `clarifai model serve` | Model serve now reuses existing resources when available instead of creating new ones.Served models are private by default. | | ![improvement](/img/improvement.jpg) | Added `--keep` flag to `clarifai model serve` | Use `--keep` to preserve the build directory after serving, useful for debugging and inspecting build artifacts. | | ![improvement](/img/improvement.jpg) | Local Runner is now public by default | Models launched via the local runner are now publicly accessible by default, removing the need to manually set visibility. | ### Model Runner |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added `VLLMOpenAIModelClass` | New `VLLMOpenAIModelClass` parent class with built-in cancellation support and health probes for vLLM-backed models. | | ![improvement](/img/improvement.jpg) | Optimized model runner memory and latency | Reduced memory footprint and improved response latency in the model runner.Streamlined overhead in SSE (Server-Sent Events) streaming. | | ![improvement](/img/improvement.jpg) | Auto-detect and clamp `max_tokens` | The runner now automatically detects the backend's `max_seq_len` and clamps `max_tokens` to that value, preventing out-of-range errors. | ### Bug Fixes |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed reasoning model token tracking and streaming in agentic class | Fixed token tracking for reasoning models to correctly account for reasoning tokens.Fixed event-loop safety, streaming, and tool call passthrough in the agentic class. | | ![bug](/img/bug.jpg) | Fixed user/app context conflicts in CLI | Resolved conflicts between `user_id` and `app_id` when using named contexts in CLI commands. | | ![bug](/img/bug.jpg) | Fixed `clarifai model init` directory handling | `clarifai model init` now correctly updates an existing model directory instead of creating a subdirectory. | --- ## Release 12.4 # Release 12.4 **Release Date:** May 7th, 2026 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Published new models | Published [Nemotron Nano V3 Omni](https://clarifai.com/nvidia/chat-completion/models/nemotron-nano-v3-omni) (30B total / 3B active — MoE). It’s NVIDIA's multimodal LLM that unifies video, audio, image, and text understanding in a single model, with integrated reasoning support. Published [Qwen3.6-35B-A3B](https://clarifai.com/qwen/qwen-VL/models/Qwen3_6-35B-A3B-FP8), an efficient MoE LLM with 35B total but only 3B active parameters, delivering near-large-model performance at much lower compute cost.| ## Cached Prompt Tokens |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced cached prompt tokens in model responses | Model responses now include cached prompt token counts, providing accurate token usage reporting when prompt caching is active. Learn more [here](https://docs.clarifai.com/compute/inference/routing/#prediction-caching). | ## Pipelines |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced code-first Pipeline DSL with CLI support | Introduced a code-first Pipeline DSL that lets you define pipelines programmatically in Python and generate configs via CLI.Use `clarifai pipeline init` to scaffold the pipeline config and `clarifai pipeline upload` to deploy it to the platform. Learn more [here](https://docs.clarifai.com/create/models/). | | ![new-feature](/img/new_feature.jpg) | `clarifai pipeline run --dev` for local pipeline development | New `--dev` flag on `clarifai pipeline run` enables a local development loop, letting you iterate on pipeline steps without deploying to the cloud.See [PR #1012](https://github.com/Clarifai/clarifai-python/pull/1012) for details. | | ![new-feature](/img/new_feature.jpg) | `clarifai pipeline local-run` for Docker-based step testing | New `clarifai pipeline local-run` command runs individual pipeline steps locally inside Docker containers, matching the production runtime environment before you deploy. | | ![new-feature](/img/new_feature.jpg) | Auto-create compute resources via `--instance` flag | `clarifai pipeline run` now accepts an `--instance` flag that automatically creates the required compute cluster and nodepool if they don't already exist. | | ![improvement](/img/improvement.jpg) | Improved `clarifai pipeline init` UX and help text | Clearer help text, improved prompts, and a post-init next-steps message now guide users through the full pipeline setup flow after running `clarifai pipeline init`. | ## Model Deployment |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Disabled `deploy_latest_version` for `clarifai model serve` | `clarifai model serve` no longer automatically promotes a newly uploaded version as the live deployed version, giving you explicit control over which version is active. | ## Local Runners |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Local runner defaults to PRIVATE; `--public` flag patches all visibilities | Models and resources created via the local runner are now private by default.Pass the `--public` flag to make all associated resources public in a single command. | ## Bug Fixes (Python SDK) |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed deployment worker pinning on `clarifai model serve` | Re-pins the deployment's `desired_worker` to the current model version when running `clarifai model serve`, preventing stale version references after serving. | | ![bug](/img/bug.jpg) | Fixed Hugging Face private repo access validation | Corrected access validation for private Hugging Face repos that return a `not_found` response to anonymous requests, eliminating false access errors on valid private repos. | | ![bug](/img/bug.jpg) | Loosened pinned requirements and fixed Clarifai package detection | Relaxed overly strict version pins in the SDK's dependencies and fixed a bug in detecting the installed Clarifai package version. | | ![bug](/img/bug.jpg) | Fixed `User.app()` returning empty values | `User.app()` now correctly returns actual server-side app data instead of empty placeholder values. | --- ## Release 5.10 # Release 5.10 ## Changelog 5.10 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Accounts | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Create delete email endpoints in v2 to finally get off old internal endpoints to streamline operations | | ![](/img/new_feature.jpg) | Create Patch, Delete, Get CreditCards endpoint in v2 APIs to finally get off old internal endpoints to streamline operations | | ![](/img/improvement.jpg) | Improved billing for collaborators | | ![](/img/bug.jpg) | PostVerifyEmail error causing some issues not being able to verify their email addresses upon sign-up. Fixed | | ![](/img/bug.jpg) | Fixed flaky email verification integration test to provide more stability to sign-up process | | ![](/img/bug.jpg) | Fixed a link to a non-public version of our API used for development purposes which led to a lot of login issues for users who landed there | ### Applications | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Created display for scopes on collaborator invitations, allowing users to easily understand and control the scope of access allowed for app collaborators | | ![](/img/new_feature.jpg) | Introduced Collaborators and Collaborations endpoints in API and UIs in Portal | | ![](/img/new_feature.jpg) | Add ability to upload inputs from App Details screen in Portal | | ![](/img/improvement.jpg) | Created collaboration tab in Portal, making it easy to add collaborators to apps | | ![](/img/improvement.jpg) | Created display to show the user who invited you to collaborate on an app | | ![](/img/improvement.jpg) | Update email phrases for collaborator invitations. After successful sign-up, the user is now redirected to the app's dashboard in Portal | | ![](/img/bug.jpg) | Fixed issue with concept counts in some apps | | ![](/img/bug.jpg) | Clicking pencil icon to edit an API Key in Portal crashed apps. Fixed | ### Data Management | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | PATCH /inputs needs to check status of asset before patching | | ![](/img/improvement.jpg) | Removed sync DELETE /inputs after runtime config tested | | ![](/img/improvement.jpg) | Changed POST /inputs to be async always to simplify processing of workflows after API client tests updated | | ![](/img/improvement.jpg) | Added pagination to clusters making for easier data management | | ![](/img/bug.jpg) | Sporadic inability to delete any inputs via Portal or in bulk via the API | | ![](/img/bug.jpg) | Numerous third party security fixes under the hood during ongoing upgrades | | ![](/img/bug.jpg) | Fix 40012 status caused by parallel deletes and adds having a race condition | | ![](/img/bug.jpg) | Update status\_changed\_at when deleting inputs so we can better track changes | | ![](/img/bug.jpg) | Cache the input counts so that apps can display them in Portal efficiently | | ![](/img/bug.jpg) | Handle killing URL downloading if it is processing for more than 60s. This will make URL processing much more reliable | | ![](/img/bug.jpg) | Return an error if a user sends YouTube video URL as that is not a valid URL to a video we can download | | ![](/img/bug.jpg) | Prevent PostInputs from creating inputs with a user-provided Input.ID that contains a colon | | ![](/img/bug.jpg) | Video calls failed if URLs contain parameters after the file type. Fixed | | ![](/img/bug.jpg) | Failed to resolve DNS MX record in URL down-loader which effected some downloads. Fixed | | ![](/img/bug.jpg) | Investigate why some re-hosted s3 links are no longer working | | ![](/img/bug.jpg) | Getting input counts was broken in some apps, reporting zero, which caused Portal to add an input view to display always | | ![](/img/bug.jpg) | Debug UnicodeErrors in URL downloading to fix URLs with Unicode characters | | ![](/img/bug.jpg) | Fix the poor handling of video too large error message | | ![](/img/bug.jpg) | Unable to batch delete inputs from time to time has been fixed | | ![](/img/bug.jpg) | Media processor video handling was having errors with decoding some videos | | ![](/img/bug.jpg) | Delete Image Button doesn't work in some scenarios | | ![](/img/bug.jpg) | Fixed support for webp image format so it is available again | ### Annotate | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Deploy General Detection Beta Model to recognize multiple objects with bounding boxes. | | ![](/img/new_feature.jpg) | Deployed new face detector for improved face detection performance over images and video | | ![](/img/new_feature.jpg) | Created custom training enhancements that handle negatives better for improved model performance | | ![](/img/new_feature.jpg) | Created evaluation metrics for custom facial recognition in backend for improved facial recognition performance | | ![](/img/improvement.jpg) | Topological sort for workflows for scheduling a sequence based on dependencies | | ![](/img/improvement.jpg) | Cleaned up duplicate models in workflow model list | | ![](/img/improvement.jpg) | Deployed clarifai/main general v1.5 in concept model | | ![](/img/improvement.jpg) | Create Pixel Training Hyperparameter Help Guide | | ![](/img/improvement.jpg) | Improved accuracy of annotation counts, improving the user experience when annotating inputs | | ![](/img/bug.jpg) | If an image is tagged with a concept that is not in the model, training fails due to KeyError, this is fixed | | ![](/img/bug.jpg) | Fix detection labeling bug where previous images image ratio is used which would cause display issues | | ![](/img/bug.jpg) | We have updated Portal to scale to a large number of concepts with much lower resource usage | | ![](/img/bug.jpg) | Investigate face bounding box probabilities consistency to improve user experience | | ![](/img/bug.jpg) | Bounding box creation canvas in Portal was breaking on resize of the window | | ![](/img/bug.jpg) | Model | | ![](/img/bug.jpg) | Cleaned up duplicate models in the workflow model list, so that you no longer see two General models | | ![](/img/bug.jpg) | Unintended behavior for private model version IDs for certain customers has been fixed | | ![](/img/bug.jpg) | Models referencing deleted backends should be marked as deleted | | ![](/img/bug.jpg) | The latest version of our general model wasn't always default, now it is | | ![](/img/bug.jpg) | Fixed a bug with face recognition evaluations. | | ![](/img/bug.jpg) | Deleted Concepts Persisted in face recognition models. Not anymore! | | ![](/img/bug.jpg) | Inability to see whether a large model is training and making progress, or hung has been addressed to better support our customers | | ![](/img/bug.jpg) | Model won't train in some apps with no positive examples issue has been resolved | | ![](/img/bug.jpg) | Fixed issues with color models failing for a short period of time | | ![](/img/bug.jpg) | Fixed list of models available to workflows to only show a single General model | ### Predict | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Return custom detection evaluations through the GO API | | ![](/img/improvement.jpg) | Improved cluster page performance | | ![](/img/bug.jpg) | Investigate health checks killing a prediction backend service, which could affect some predictions in the API | | ![](/img/bug.jpg) | Workflow predict sometimes was failing with 98012 status code. Many fixes here should reduce that | | ![](/img/bug.jpg) | Workflow Predict called the wrong model sometimes. Not any more! | | ![](/img/bug.jpg) | Video playback out of sync with detections in our demos | | ![](/img/bug.jpg) | Fixed issues with regions predicted on inputs would be carried over between inputs in Portal | | ![](/img/bug.jpg) | Fixed the flaky face recognition tests to ensure stability of our face recognition product | | ![](/img/bug.jpg) | Face Detection backends were running out of memory for some predictions, this has been resolved | | ![](/img/bug.jpg) | Return more descriptive error msg for post metric endpoint | ### Search | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Added helper text/suggestions to improve Portal user experience | | ![](/img/improvement.jpg) | Header Search return app\_owner's user info in collaboration endpoints | | ![](/img/bug.jpg) | Explorer Search Bar - Clicking the green/red circle icons didn't reliably detect click, now it does! | | ![](/img/bug.jpg) | Portal not showing the correct number of results in concept search. Fixed. | | ![](/img/bug.jpg) | Left/right arrows in single image view don't switch between images with regions. Fixed | | ![](/img/bug.jpg) | Fixed carousel thumbnail clicks wiping query params / trigger new search | --- ## Release 5.11 # Release 5.11 ## Changelog 5.11 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Accounts | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Create a UI for personal access tokens making it easier for users to access their own apps and any apps where they have been added as collaborators | | ![](/img/new_feature.jpg) | Updated /keys to work with PATs so that app-specific keys can be created programmatically. | | ![](/img/bug.jpg) | Login \(user/PW\) has no rate limit/max attempts. Fixed | | ![](/img/bug.jpg) | Remove all instances of worker\_id from explorer | | ![](/img/bug.jpg) | When email link to verify my email address clicked, still see "verify your email" banner. Fixed | | ![](/img/bug.jpg) ![](/img/enterprise.jpg) | API services do not function once Queue goes down and comes back up. Fixed. This makes on premise deployments more resilient to power failures. | ### Applications | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Add apps and keys scopes so they can be created with personal access tokens | | ![](/img/improvement.jpg) ![](/img/enterprise.jpg) | Copy app count and last\_inputs added in app duplication | | ![](/img/bug.jpg) | Fixed demo font syntax | | ![](/img/bug.jpg) | Fixed details page header missing description | | ![](/img/bug.jpg) | Added favicon for Portal | | ![](/img/bug.jpg) | Unable to copy an app that has been shared via Collaborators. Fixed | | ![](/img/bug.jpg) | Setting useCustomConfig isn't checked at login. Fixed | | ![](/img/bug.jpg) | Collaboration apps have race condition where wrong user id is used | | ![](/img/bug.jpg) | Stopped loading of collaborations for search demo/logged-out users | | ![](/img/bug.jpg) | Return “All” scopes when listing available scopes so that you have that option when creating new keys. | | ![](/img/bug.jpg) | Collaborators can not see workers. Fixed | | ![](/img/bug.jpg) | Missing `Apps_Get` scope in session token auth caused creation of keys to fail temporarily. Fixed | | ![](/img/bug.jpg) | List of missing scopes is not correct in error messages. Fixed | ### Data Management | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Optimize video detection frame rate on Front end | | ![](/img/improvement.jpg) | Improve JSON serialization performance in our servers by using an optimized third party library | | ![](/img/improvement.jpg) | Able to overwrite default max conn for Citus | | ![](/img/improvement.jpg) | Rewrite input counting in the API to be more scalable and robust | | ![](/img/bug.jpg) | Allow RegionInfo from SpireDetectEmbedResponse to contain Point when saving to DB | | ![](/img/bug.jpg) | Unable to upload same file\(s\) through browse files. Fixed | | ![](/img/bug.jpg) | ffmpeg can produce no frames for very short videos | | ![](/img/bug.jpg) | Add Inputs/View Explorer does not display in new app anymore. Fixed | | ![](/img/bug.jpg) | Clicking video thumbs in detail view does not reload a video. Fixed | | ![](/img/bug.jpg) | Keyboard navigation in image details view highlights incorrect thumb | | ![](/img/bug.jpg) | No Prompt when uploading an image to Explorer through URL. Fixed | | ![](/img/bug.jpg) | Properly return error if `AddAssets` failed to insert into database | ### Annotate | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Remove classification/detection toggle in image details view | | ![](/img/bug.jpg) | Improved adding negatives to regions | | ![](/img/bug.jpg) | Create one annotation for each bbox | | ![](/img/bug.jpg) | Log capability added for annotation/search request/response | | ![](/img/bug.jpg) | Eliminated error if no annotation to be deleted | | ![](/img/bug.jpg) | Last concept used for bounding boxes is retained between apps. Fixed | | ![](/img/bug.jpg) | The Add Positives / Add Negatives buttons on a Concept details view breaks portal | | ![](/img/bug.jpg) | Custom facial recognition bboxes on grid view do not correlate. Fixed | ### Model | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Ability to keep concepts sorted by alpha in Portal | | ![](/img/new_feature.jpg) | Implement image crop model to make it possible to work in subregions of an image | | ![](/img/new_feature.jpg) | Implement random sample model type, adding to fixed function feature set | | ![](/img/improvement.jpg) | Update training templates to have more straightforward names and more friendly defaults | | ![](/img/improvement.jpg) | Fix the WorkflowInput field name in proto to workflow\_input | | ![](/img/improvement.jpg) | Allow models that need outputs from previous nodes in a workflow to have access to those outputs to support chaining complex graphs of models | | ![](/img/bug.jpg) | Confusion matrix predicted/true are swapped in evaluation results. Fixed | | ![](/img/bug.jpg) | Fixed generalModel imports and optimize video click handlers with useCallback hooks | | ![](/img/bug.jpg) | Fix for selectEmbedModelVersionId in detection apps | | ![](/img/bug.jpg) | Drawing annotations: wrong embed model version id | | ![](/img/bug.jpg) | Made custom training evaluations for large models stable. | | ![](/img/bug.jpg) | Training progress is saved too frequently, causing very slow training | | ![](/img/bug.jpg) | Return friendlier errors for incorrect parameters passed to templates | | ![](/img/bug.jpg) | Fixed a bug in tracing setup for custom trainer and evaluator | | ![](/img/bug.jpg) | Some models were operating slowly because of lack of resources. Fixed | | ![](/img/bug.jpg) | Training System failed to train some layers. Fixed | | ![](/img/bug.jpg) | Prevent users from evaluating models that are not trainable | | ![](/img/bug.jpg) | Fixed node ID validation logic in Bug in workflows | ### Predict | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Add colors to differentiate region results | | ![](/img/bug.jpg) | Cannot view workflow results in a face app. Fixed | | ![](/img/bug.jpg) | Video spire tests are not running correctly. Fixed | | ![](/img/bug.jpg) | Video processing fails with 'caseids' error. fixed | ### Search | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Add click to search metadata attributes in image details sidebar | | ![](/img/new_feature.jpg) | Implement visual search in another app as a model type you can add to a workflow | | ![](/img/bug.jpg) | Search bar missing in some cases. Fixed | | ![](/img/bug.jpg) | Region Searches within Search Bar still use crop coordinates instead of base64 bytes. Fixed | | ![](/img/bug.jpg) | Click Search button icons on Thumbs not working for localized search. Fixed | | ![](/img/bug.jpg) | Disable all search by click handlers in Portal for Text Apps | | ![](/img/bug.jpg) | Disable "hide all positively labeled" inputs button for NLP until search works | | ![](/img/bug.jpg) | Scroll active thumb into view in image details carousel | | ![](/img/bug.jpg) | Render Video Assets in Search Bar | | ![](/img/bug.jpg) | Editing geo/json search items no longer work after adding the search bar tooltip. Fixed | | ![](/img/bug.jpg) | TypeError: Cannot read 'get' of undefined when clicking image thumbnails in Explorer search bar. Fixed | | ![](/img/bug.jpg) | Explorer Visibility in small resolution screen improved | --- ## Release 6.0 # Release 6.0 ## Changelog 6.0 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Accounts | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Remove country field from signup form, simplifying new customer signups | | ![](/img/bug.jpg) | Essential Plan User can't add collaborators. Fixed | ### API | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Introduce new [Python gRPC API client](https://docs.clarifai.com/api-guide/api-overview), enabling new features and performance enhancements across API | | ![](/img/new_feature.jpg) | Introduce new [Java gRPC API client](https://docs.clarifai.com/api-guide/api-overview), enabling new features and performance enhancements across API | | ![](/img/improvement.jpg) | Update API key type for "app\_specific" for app-specific keys to be more clear to users | ### Applications | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Allow Personal Access Tokens when calling /users/me \(GetUsers\) | | ![](/img/new_feature.jpg) | \[Frontend\] Enable "Copy Application" from collaborated apps, making it easy to duplicate and build upon existing applications | | ![](/img/bug.jpg) | Program to clean internal apps crashing. Fixed | ### Data Management | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Added the ability to accept b64 Gifs | | ![](/img/bug.jpg) | Functionality to upload pre-tagged images missing. Fixed | | ![](/img/bug.jpg) | Images pre-tagged with concepts do not successfully upload into Clarifai UI On doing bulk uploads \(>20-30 urls\). Fixed | | ![](/img/bug.jpg) | Bulk image upload issue. Fixed | | ![](/img/bug.jpg) | "Download Failed" error when uploading images. Fixed | | ![](/img/bug.jpg) | Issue with post inputs key being a PAT in a collector. Fixed | ### Annotate | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Skip aligning landmarks if landmark points are out of range to avoid errors and unexpected behavior | | ![](/img/bug.jpg) | Bounding Boxes and Cropped Regions aren't displaying on Videos with default runtime config. Fixed | | ![](/img/bug.jpg) | Insert annotations and related data in batch to improve performance | ### Model | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Evaluate new face embedding model workflow end to end for optimal performance | | ![](/img/improvement.jpg) | Validate that concept.app\_id shouldn't be set when creating/patching models | | ![](/img/improvement.jpg) | Add new predicate to knowledge graph for "relates\_to" to represent synonyms | | ![](/img/bug.jpg) | Model training lag. Fixed | | ![](/img/bug.jpg) | Model has missing inputs. Fixed | | ![](/img/bug.jpg) | Submitted models becoming stuck in queue. Fixed | | ![](/img/bug.jpg) | Custom training models when uploaded images are not fully pre-processed. Fixed | | ![](/img/bug.jpg) | Custom facial recognition bboxes do not correspond with detection boxes/ Custom facial recognition prediction interval for video is still 1000ms for apps supporting 100ms runtime config. Fixed | | ![](/img/bug.jpg) | frame\_info time off by a factor of 10 for general detection model. Fixed | | ![](/img/bug.jpg) | Detection Models throw error at end of video due to invalid index lookup. Fixed | ### Workflow | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Deleting a workflow should clear or update localStorage. Fixed | ### Portal | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Clean up app overflow UI, improving user experience | | ![](/img/improvement.jpg) | Improve Error boundary screen, improving user experience | | ![](/img/improvement.jpg) | Add sentry error Id to Error Screen | | ![](/img/bug.jpg) | Images not loading. Fixed | | ![](/img/bug.jpg) | Label and prediction on the right side under Custom Model Predictions section no longer shows up automatically. Fixed | | ![](/img/bug.jpg) | Provide a way for user.metadata to be updated from portal when there are failing apps stuck in there. Fixed | | ![](/img/bug.jpg) | Predictions for a detection model don't show properly in portal. Fixed | | ![](/img/bug.jpg) | Custom facial recognition Predict Boxes not displaying. Fixed | | ![](/img/bug.jpg) | Adding inputs in explorer redirects to explorer view with flashing images. Fixed | ### Predict | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Run prediction by ID in small batch, improving performance | | ![](/img/bug.jpg) | Custom model predictions not displaying. Fixed | | ![](/img/bug.jpg) | Custom model detections not displaying. Fixed | ### Search | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Add file upload input button to explorer search bar, simplifying the UX for file uploads | | ![](/img/new_feature.jpg) | Filter custom facial recognition bboxes using a sliding bar, adding easy thresholding to custom facial recognition models | | ![](/img/improvement.jpg) | Search Bar allows file upload | | ![](/img/improvement.jpg) | Remove Explorer App Overflow Menu for improved UX | --- ## Release 6.1 # Release 6.1 ## Changelog 6.1 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Clients | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Remove Feedback endpoints from Python client | | ![](/img/improvement.jpg) | Remove Feedback endpoints from Java client | | ![](/img/improvement.jpg) | Remove Feedback endpoints from Javascript client | | ![](/img/improvement.jpg) | Remove Feedback endpoints from Portal/demo | | ![](/img/improvement.jpg) | Remove image.crop field from Python API client | | ![](/img/improvement.jpg) | Remove image.crop field from Java API client | | ![](/img/improvement.jpg) | Remove image.crop field from Javascript API client | ### Model | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Added detection evaluation in platform | | ![](/img/new_feature.jpg) | Introduce concept mapping model that uses the knowledge graph relations, creating a path for users to eventually benefit from pool of networked data | | ![](/img/bug.jpg) | Fix a bug that caused the new face predictions to have a huge performance drop | | ![](/img/bug.jpg) | Train and eval worker didn't invalidate model related cache. Fixed | | ![](/img/bug.jpg) | Fix bug in deleting a concept relation by ID | ### Portal | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Bulk labelling can now be done from Explorer mode grid view. | | ![](/img/improvement.jpg) | Show Check/X on custom detection model predictions in Portal | | ![](/img/improvement.jpg) | Allow multi concepts per bbox | | ![](/img/bug.jpg) | Negative tags not visible in Portal. Fixed | ### Predict | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Remove extra round trip to storage in predict pathway | | ![](/img/improvement.jpg) | Remove the image.crop argument during predict and POST /inputs calls to simplify the API | | ![](/img/improvement.jpg) | Add region predictions from custom models to detections in videos | ### Search | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Implement search by annotation.status in backend | | ![](/img/improvement.jpg) | Connect saved searches and annotation status | --- ## Release 6.10 # Release 6.10 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Scribe | Status | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | When a reviewer opens the "For Review" tab, determine whether there is work to be reviewed | | ![improvement](/img/improvement.jpg) | Remove Labeler V1 | | ![bug](/img/bug.jpg) | Keybindings not working when buttons are clicked | | ![bug](/img/bug.jpg) | Interpolation transforms not working | | ![bug](/img/bug.jpg) | Update Search API Endpoint | | ![bug](/img/bug.jpg) | Prediction bounding box shows for concepts not included in task | | ![bug](/img/bug.jpg) | Image filters don’t work | | ![bug](/img/bug.jpg) | AI Assist should not create predictions with 0% confidence | | ![bug](/img/bug.jpg) | AI Assist is creating annotation but is not auto marking the green check box or the red "X" | | ![bug](/img/bug.jpg) | AI Assist sends incorrect app ID when fetching workflows | | ![bug](/img/bug.jpg) | Full assignment strategy results in error during review | | ![improvement](/img/improvement.jpg) | Tests: Labeler Interpolation Sagas | | ![improvement](/img/improvement.jpg) | Tests: Labeler Inputs Sagas | | ![improvement](/img/improvement.jpg) | Tests: Labeler Canvas Sagas | | ![improvement](/img/improvement.jpg) | Show Sentry Event Id for Saga Errors | | ![bug](/img/bug.jpg) | Labeler AI Assist Breaking for custom User ID | | ![bug](/img/bug.jpg) | Unable to load/access "Assigned to me" labeler tasks | | ![bug](/img/bug.jpg) | Only able to draw one bounding box while using interpolation | | ![bug](/img/bug.jpg) | Brightness, Saturation and Invert doesn't change | | ![bug](/img/bug.jpg) | Task Edit form doesn't show the correct FPS value from the task object | | ![bug](/img/bug.jpg) | Task Form shows warning when creating a new task | | ![bug](/img/bug.jpg) | Place Order button remains on page when switching back from LaaS option to self-labeling | | ![improvement](/img/improvement.jpg) | Tests: Labeler UI Sagas | | ![improvement](/img/improvement.jpg) | When "My own Labelers" is clicked. All fields are not shown back | | ![improvement](/img/improvement.jpg) | Testing LabelOrder reducer - user journey | | ![improvement](/img/improvement.jpg) | Change cancel order copy | | ![improvement](/img/improvement.jpg) | Unit test LabelOrder reducer. | | ![improvement](/img/improvement.jpg) | Testing LabelOrder reducer | | ![improvement](/img/improvement.jpg) | Feature gate Labelling suggestions | | ![bug](/img/bug.jpg) | Pinning apps breaking search app functionality | | ![bug](/img/bug.jpg) | Unable to create a new Task in Labeler | | ![new-feature](/img/new_feature.jpg) | Better Video Features | | ![improvement](/img/improvement.jpg) | Opening a task should ONLY fetch annotations for the task that was opened | | ![bug](/img/bug.jpg) | Task Edit form doesn't pre-populate worker strategy selection | | ![bug](/img/bug.jpg) | Submit for Review logic incorrectly assumes the last input is last | | ![new-feature](/img/new_feature.jpg) | On the Task creation form Add section to Review strategy block for Consensus type | | ![bug](/img/bug.jpg) | Fix key issue in ExplorerAnnotationStatusList component in Explorer | | ![new-feature](/img/new_feature.jpg) | Handle Noisy Labels | | ![bug](/img/bug.jpg) | Optimistically update "don't show again" in onboarding modal of Task Form | | ![bug](/img/bug.jpg) | Flipkart would like better formatting of markdown for instructions | | ![bug](/img/bug.jpg) | Show onboarding modals for each persona: Task creator, Reviewer, Worker | | ![bug](/img/bug.jpg) | Onboarding tooltips for /apps doesn't go away when I click "awesome" | | ![bug](/img/bug.jpg) | Pressing ESC patches my metadata 4 times in a row | | ![bug](/img/bug.jpg) | Move Hubspot chat icon into the navbar | | ![bug](/img/bug.jpg) | Task Form Inputs and Buttons are Misaligned | | ![bug](/img/bug.jpg) | Task Creation submit not redirecting | | ![bug](/img/bug.jpg) | In Labeler, “Skip” and "Reject" buttons are hidden by the questions bot blob | | ![bug](/img/bug.jpg) | Toggling panning makes the image blurry | | ![bug](/img/bug.jpg) | Clicking "Create Task" doesn't force you to exit the task creation screen. | | ![new-feature](/img/new_feature.jpg) | App Details - Tool Tip only shows 1/3 \(there doesn't seem to be a 2/3 or 3/3\) | | ![improvement](/img/improvement.jpg) | Display onboarding modal for Reviewer | | ![improvement](/img/improvement.jpg) | Display onboarding modal for Task Creator | | ![improvement](/img/improvement.jpg) | Review Strategy for TaskForm | | ![improvement](/img/improvement.jpg) | AI Assist for TaskForm | | ![improvement](/img/improvement.jpg) | Manual reviewers for TaskForm | | ![improvement](/img/improvement.jpg) | Worker strategy input for TaskForm | | ![improvement](/img/improvement.jpg) | Worker input for TaskForm | | ![improvement](/img/improvement.jpg) | Create ProposerWorkflow input for TaskForm | | ![improvement](/img/improvement.jpg) | Create VideoFps input for TaskForm | | ![improvement](/img/improvement.jpg) | Create Concepts input for TaskForm | | ![improvement](/img/improvement.jpg) | Create InputSource input for TaskForm | | ![improvement](/img/improvement.jpg) | Create TaskType input for TaskForm | | ![improvement](/img/improvement.jpg) | Create separate styled component file for FormTypeSection | | ![bug](/img/bug.jpg) | Onboarding modal points to 2 videos and they link deeper into the same video. | ## Enlight | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Implement track association module | | ![improvement](/img/improvement.jpg) | implement log probability using kalman state as input | | ![bug](/img/bug.jpg) | cannot save model details in portal with lrate < 0.001 | | ![improvement](/img/improvement.jpg) | \[OCR\] Scene Text Reco & Text Classify Integration | | ![new-feature](/img/new_feature.jpg) | Splice Concepts Together | | ![new-feature](/img/new_feature.jpg) | New Layers / Arch | | ![new-feature](/img/new_feature.jpg) | Models on New Data Types | | ![new-feature](/img/new_feature.jpg) | Neural Net Toolkit | | ![new-feature](/img/new_feature.jpg) | Recurrent Nets Up and Running | | ![improvement](/img/improvement.jpg) | YOLO dataset script | | ![improvement](/img/improvement.jpg) | deploy landmark model | | ![improvement](/img/improvement.jpg) | Update roll up for task\_input\_counts table to have worker\_id information | | ![improvement](/img/improvement.jpg) | Implement annotation count | | ![improvement](/img/improvement.jpg) | Integrate filter, status, association into a single tracker model | | ![improvement](/img/improvement.jpg) | Implement aggregator | | ![improvement](/img/improvement.jpg) | Design simulated data that mimic low confidence / high recall detections | | ![improvement](/img/improvement.jpg) | Design training losses | | ![improvement](/img/improvement.jpg) | Try box in x y a r format \(SORT format\) | | ![improvement](/img/improvement.jpg) | Try box in x y w h format | | ![bug](/img/bug.jpg) | Checkpoint for test eval can be deleted before eval is started | | ![improvement](/img/improvement.jpg) | Improve face orientation bucketing of LandmarkAlignTransform in DataProvider | | ![improvement](/img/improvement.jpg) | Use tracker state in TrackerPredict calls | | ![improvement](/img/improvement.jpg) | Make Text Aggregator Operator public | | ![improvement](/img/improvement.jpg) | Remove transformers dependency | | ![improvement](/img/improvement.jpg) | Simple visual tracker example | | ![improvement](/img/improvement.jpg) | Ingest VGGFACE2 | | ![improvement](/img/improvement.jpg) | Explore multi-language support | | ![improvement](/img/improvement.jpg) | Rewrite Export Scripts | | ![new-feature](/img/new_feature.jpg) | Introduce region threshold model | | ![improvement](/img/improvement.jpg) | Add filter\_other\_concepts param to concept threshold model | | ![improvement](/img/improvement.jpg) | Remove the unnecessary requires\_sequential\_frames field of ModelType | | ![improvement](/img/improvement.jpg) | Add support for image crop model to work on video | | ![bug](/img/bug.jpg) | Fix duplicate\_app.go restriction on number of annotations | | ![improvement](/img/improvement.jpg) | Notebook to demo tracker evals | | ![improvement](/img/improvement.jpg) | RNN vs Model-based tracking: Real data | ## Armada | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Triton GRPC Client Upgrade | | ![improvement](/img/improvement.jpg) | Migrate Spire to use TextTokenizer Transform | | ![improvement](/img/improvement.jpg) | Implement TextTokenizer Transform | | ![improvement](/img/improvement.jpg) | Export model to TorchScript | | ![improvement](/img/improvement.jpg) | HDFS for EID storage | | ![improvement](/img/improvement.jpg) | Experiment tracking improvements | | ![improvement](/img/improvement.jpg) | Trace from EID | | ![improvement](/img/improvement.jpg) | EID management | | ![improvement](/img/improvement.jpg) | Ingest A Video dataset | | ![improvement](/img/improvement.jpg) | Resolve TRT saved model Issues Across GPU compute Capabilities | | ![improvement](/img/improvement.jpg) | People/Vehicle Detector TRT conversion and inference benchmarking | ## Portal | Status | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | CSS on toggles within model mode was cutting off the toggle | | ![bug](/img/bug.jpg) | Model creation broken | | ![new-feature](/img/new_feature.jpg) | Create Hubspot dashboard with funnels and charts | | ![new-feature](/img/new_feature.jpg) | Capture customer behavior analytics | | ![bug](/img/bug.jpg) | CORS error prevents uploaded images from rendering | | ![bug](/img/bug.jpg) | Properly configure S3 bucket for inputs to allow portal \(prod and dev\) | | ![bug](/img/bug.jpg) | Investigate why the vendors bundle is 12MB | | ![new-feature](/img/new_feature.jpg) | Render relation-based parent-child relationship in Proposals | | ![new-feature](/img/new_feature.jpg) | Proposers UI with a new annotations reducer | | ![new-feature](/img/new_feature.jpg) | Create annotations module in store/explorer for Proposers UI | | ![improvement](/img/improvement.jpg) | Move Proposers data transformation to a saga+web worker | | ![improvement](/img/improvement.jpg) | Integrate Proposers with Classification apps | | ![improvement](/img/improvement.jpg) | Show App Concepts alongside model concepts in Proposers UI | | ![bug](/img/bug.jpg) | Explorer does exponentially increasing duplicate calls to GET inputs endpoint | | ![new-feature](/img/new_feature.jpg) | Introduce knowledge graph link creation when check/X is clicked on clarifai/main models | | ![improvement](/img/improvement.jpg) | update created-at timestamp format to: hours/minutes/seconds for model-input | | ![bug](/img/bug.jpg) | data-mode - uploaded-images multiple elements get re-rendered | | ![improvement](/img/improvement.jpg) | Zoom-in & zoom-out controls for explorer geo | | ![improvement](/img/improvement.jpg) | Allow users to upload the same url twice from data mode | | ![improvement](/img/improvement.jpg) | Set app-id instead of the app-name on new app-creation for readable urls | | ![improvement](/img/improvement.jpg) | Fully qualified url for app-listing | | ![bug](/img/bug.jpg) | NaN being displayed in model-versions | | ![bug](/img/bug.jpg) | Portal does not parse links properly | | ![bug](/img/bug.jpg) | Unable to view versions or eval metrics page as a collaborator | | ![improvement](/img/improvement.jpg) | Data Ingestions | | ![improvement](/img/improvement.jpg) | Create initial eval script to test cascade approach | | ![improvement](/img/improvement.jpg) | Work with BE to nurse input uploads | | ![improvement](/img/improvement.jpg) | Get Input Embeddings and Cluster over All | | ![improvement](/img/improvement.jpg) | Get Input Embeddings and Cluster over Concepts | | ![bug](/img/bug.jpg) | Data Mode page performance issues when pasting 60 inputs in production | | ![bug](/img/bug.jpg) | Bulk unlabeling doesn't work | | ![new-feature](/img/new_feature.jpg) | Change model type to model type ID everywhere | | ![improvement](/img/improvement.jpg) | Slider for concept threshold should allow for full precision floats with box to set the value | | ![improvement](/img/improvement.jpg) | Upload third party models through Portal | | ![improvement](/img/improvement.jpg) | Develop the model predict view for portal for existing models. | | ![new-feature](/img/new_feature.jpg) | Add dropdown for model\_type\_id | | ![new-feature](/img/new_feature.jpg) | Allow users to upload pre-trained models | | ![improvement](/img/improvement.jpg) | Make versions tab as default in model details page | | ![improvement](/img/improvement.jpg) | Update `ModelPredictions` component to make use of `reselect` & sagas | | ![bug](/img/bug.jpg) | Create collectors always selects user as clarifai | | ![bug](/img/bug.jpg) | AnimatedList caches the children nodes causing improper renders | | ![bug](/img/bug.jpg) | Sending invalid fields during model creation | | ![bug](/img/bug.jpg) | Workflow page crashes | | ![bug](/img/bug.jpg) | Improve performance of explorer | | ![bug](/img/bug.jpg) | Remove console errors in various parts of portal | | ![bug](/img/bug.jpg) | GetInputCount\(\) is called on loop after exiting data mode | | ![bug](/img/bug.jpg) | Create new Collector redirects to apps page | | ![bug](/img/bug.jpg) | Collectors Adding app model bug | | ![bug](/img/bug.jpg) | copy public workflow do not populate workflow model | | ![bug](/img/bug.jpg) | In eval metrics page, "Concept by Concept" section shows counts that are incorrect | | ![bug](/img/bug.jpg) | In eval metrics page, "1 split" section is incorrect | | ![bug](/img/bug.jpg) | Prevent patching model annotations | | ![bug](/img/bug.jpg) | Deprecate old search endpoint from FE | | ![bug](/img/bug.jpg) | Add Embed Model Version Id to Bulk Labeler Classification Annotations | | ![bug](/img/bug.jpg) | Range Slider on Eval Metrics Page causes app crash when moving | | ![improvement](/img/improvement.jpg) | Allow copying public workflows into existing workflows | ## API | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Overwrite user annotation info with annotation writer model info | | ![bug](/img/bug.jpg) | Unable to add polygon annotation with embed\_model\_version\_id | | ![improvement](/img/improvement.jpg) | Experiment with docs.clarifai.com using API versions | | ![new-feature](/img/new_feature.jpg) | Make gRPC Swift client | | ![bug](/img/bug.jpg) | Can't assign a task to myself if I have a custom user\_id | | ![bug](/img/bug.jpg) | Annotation writer model & concept thresholder create incorrect additional annotations | | ![bug](/img/bug.jpg) | App copy doesn't stop after timeout | | ![new-feature](/img/new_feature.jpg) | Asset search by status | | ![improvement](/img/improvement.jpg) | Monitor media processor predict err rate | | ![improvement](/img/improvement.jpg) | Log spire name | | ![bug](/img/bug.jpg) | Clean idle/pending asset publish video asset to pipeline separately | --- ## Release 6.11 # Release 6.11 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Scribe | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Update annotations count for tasks with consensus review | | ![bug](/img/bug.jpg) | App Collaborators are unable to view model versions. Fixed | | ![improvement](/img/improvement.jpg) | Refactor CustomEval worker | | ![improvement](/img/improvement.jpg) | Trigger consensus review when non-anchor annotation is updated | | ![new-feature](/img/new_feature.jpg) | Collaborator annotation success by default | ## Account | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Signup form should show some visual feedback when submitting | ## Spacetime Search | Status | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | New Saved Searches not working. Fixed | | ![bug](/img/bug.jpg) | Unable to save search. Fixed | ## Portal | Status | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | Frontend merge requests failing. Fixed | | ![new-feature](/img/new_feature.jpg) | Create a 'People Tracker' in Portal with specific parameters from centroid tracker | | ![new-feature](/img/new_feature.jpg) | Proposers - add filtering by status | | ![new-feature](/img/new_feature.jpg) | Proposers - switch workflows | | ![improvement](/img/improvement.jpg) | Show concepts from workflows in proposals | | ![improvement](/img/improvement.jpg) | Natural language form for Proposals relation type chooser | | ![new-feature](/img/new_feature.jpg) | Allow users to drag & reorder selected models during workflow creation | | ![improvement](/img/improvement.jpg) | Add filters to Proposers UI | | ![improvement](/img/improvement.jpg) | Add rendering of Proposers UI bounding boxes | | ![new-feature](/img/new_feature.jpg) | Make predictions, versions and concept show for clarifai/main models | | ![improvement](/img/improvement.jpg) | No loader in proposals causing user confusion | | ![bug](/img/bug.jpg) | Proposers - no name shown for collaborators. Fixed | | ![bug](/img/bug.jpg) | Proposers - classification parent-child hit-or-miss if modelOutputs don't load. Fixed | | ![bug](/img/bug.jpg) | Button displays in Model Mode for text workflows are off. Fixed | | ![improvement](/img/improvement.jpg) | Allow users to bulk-unlabel previously labelled inputs | | ![improvement](/img/improvement.jpg) | Adopt sortable table everywhere in portal | | ![improvement](/img/improvement.jpg) | Model Mode > OUTPUT\_INFO.DATA.CONCEPTS > trashcan icon too far away from the concept name | | ![improvement](/img/improvement.jpg) | Display concepts in upload trained model page | | ![bug](/img/bug.jpg) | Omits being added as Rank instead of Filters. Fixed | | ![bug](/img/bug.jpg) | All model versions are not being displayed. Fixed | | ![improvement](/img/improvement.jpg) | Support Secure Data Hosting for Text Assets | | ![bug](/img/bug.jpg) | Saved Searches not working in Explorer. Fixed | | ![bug](/img/bug.jpg) | Omit Negative/Positive and Filter Positive/Negative requests malformed. Fixed | | ![bug](/img/bug.jpg) | Workflow Range Slider not working for all workflows. Fixed | | ![bug](/img/bug.jpg) | Bulk labeling should read labels from selected annotations/inputs. Fixed | | ![improvement](/img/improvement.jpg) | Disable fetching annotation in bulk-labeling | | ![bug](/img/bug.jpg) | Undefined is not an object \(evaluating 'Ua\[o\].videos'\). Fixed | | ![bug](/img/bug.jpg) | Failed prop type: Invalid prop `sm` supplied to `Col`. Fixed | | ![improvement](/img/improvement.jpg) | Remove Create Workflow modal from Labeler | | ![bug](/img/bug.jpg) | console.log data printed in Taskform. FIxed | | ![bug](/img/bug.jpg) | Uploading iPhone photos in Portal results in the inputs rotating incorrectly \(inconsistent\). Fixed | | ![bug](/img/bug.jpg) | Explorer console error in predictions reducer fixed | | ![bug](/img/bug.jpg) | Broken thumbnail when uploading a video via data mode page. Fixed | | ![bug](/img/bug.jpg) | Switch collectors to use new labeler whitelist flag | ## Armada | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Re-process assets faster | ## API | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Update error status codes in daily\_error\_code\_count | | ![improvement](/img/improvement.jpg) | Add req\_id to logged error | | ![new-feature](/img/new_feature.jpg) | Finish gRPC Rust client | | ![new-feature](/img/new_feature.jpg) | Move the python API client tests to gRPC Python client. | | ![improvement](/img/improvement.jpg) | Update & release the clients with the secure gRPC channel | | ![improvement](/img/improvement.jpg) | Improved workflow logging | | ![improvement](/img/improvement.jpg) | Add PostModelOutputs retries to the gRPC Python test suite for dev & staging | | ![improvement](/img/improvement.jpg) | Validate new app id when duplicate app | | ![improvement](/img/improvement.jpg) | Workflow improvements | | ![bug](/img/bug.jpg) | Fix undefined variable | ## Enlight Train | Staus | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | Model training only uses available inputs, not waiting for all inputs to be processed. Fixed | | ![new-feature](/img/new_feature.jpg) | Add support for landmark extraction model type | | ![improvement](/img/improvement.jpg) | Pre-trained model creation field improvements | | ![bug](/img/bug.jpg) | Fix new deep training: missing platform in benchmark config + torch import affecting TF benchmarking | | ![improvement](/img/improvement.jpg) | Allow public model versions if they are used in public workflow | | ![bug](/img/bug.jpg) | Deep training Bug with small amount of inputs fixed | --- ## Release 6.2 # Release 6.2 ## Changelog 6.2 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Accounts | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Updated Privacy Policy URL | | ![](/img/bug.jpg) | Fixed panic error in Signup | ### Applications | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Ensured collectors are deleted when apps are deleted | | ![](/img/bug.jpg) | View In Explorer button missing in app details. Fixed | | ![](/img/bug.jpg) | Fixed failed to generate thumbnail | | ![](/img/bug.jpg) | Fixed app duplication error when getting worker | | ![](/img/bug.jpg) | Deleted collaborator should also mark application\_worker to deleted. Fixed | ### Inputs | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Inputs count stuck at > 0 after delete all, with all inputs seemingly deleted | ### Label | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Task view UI for workers | | ![](/img/new_feature.jpg) | Create task manager page and task creation page | | ![](/img/new_feature.jpg) | New Icon for Task Manager/Task Viewer | | ![](/img/bug.jpg) | Fixed POST annotations call on frontend to use correct embed model | | ![](/img/bug.jpg) | Post annotations should include embed\_model\_version\_id. Fixed | ### Model | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Added Apparel Detection to Demo App | | ![](/img/new_feature.jpg) | Created UI for creating knowledge graph concept relations relations | | ![](/img/new_feature.jpg) | Create annotation writer model to write annotations to DB | | ![](/img/improvement.jpg) | Pass and use test and train data queries to trainer | | ![](/img/improvement.jpg) | Added migration to upgrade old model\_type in DB | | ![](/img/improvement.jpg) | Depredated Face from Python client | | ![](/img/improvement.jpg) | Unified the TypeExt and Type fields in model object. | | ![](/img/improvement.jpg) | Deprecated facedetect\* model types. | | ![](/img/improvement.jpg) | Unified FaceEmbedModel and DetectEmbedModel | | ![](/img/improvement.jpg) | Converted face.Identity responses to concepts like other detection models to be consistent | | ![](/img/bug.jpg) | Fixed demo font syntax | | ![](/img/bug.jpg) | Video Timeline does not display on the demo app | | ![](/img/bug.jpg) | Fixed Range Slider Value/Text in Apparel Detection Demo | | ![](/img/bug.jpg) | Fixed demographics model to return embeddings and work with auto-annotate | | ![](/img/bug.jpg) | Adding collaborator model counter-intuitively requires ENTER in order to enable the submit button. Fixed | ### Predict | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Validated that concept relation doesn't already exist on POST relations | | ![](/img/bug.jpg) | Prediction requests are being fired too frequently instead of using cache. Fixed | | ![](/img/bug.jpg) | postModelOutputs is not called for newly labeled assets without a manual refresh | ### Search | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Return annotations posted by user in search results | | ![](/img/bug.jpg) | Search by region not working for face detection. Fixed | | ![](/img/bug.jpg) | Make “save” search button internal only | | ![](/img/bug.jpg) | Saved Searches in Portal use the incorrect user ID | | ![](/img/bug.jpg) | Fix crop search from single image view for faces/detections | ### Portal | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | UI for collector crud | | ![](/img/improvement.jpg) | Deprecate Face from Portal | | ![](/img/improvement.jpg) | Improve tabs UI | | ![](/img/bug.jpg) | Video Predictions are failing in Portal | | ![](/img/bug.jpg) | Fixed broken font syntax | ### Workflow | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Video detection workflow prediction support | | ![](/img/new_feature.jpg) | Public general v1.5 workflow | | ![](/img/bug.jpg) | Allow Patching to existing public workflow | | ![](/img/bug.jpg) | Can not train LOPQ if app base workflow is face. Fixed | --- ## Release 6.3 # Release 6.3 ## Changelog 6.3 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Applications | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Send collaborator emails asynchronously | | ![](/img/bug.jpg) | NLP bug fixes for non-text apps | ### Inputs | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Consolidated input related status codes | | ![](/img/improvement.jpg) | Add frame.id to API as well as region.track\_id | | ![](/img/bug.jpg) | Granted select permission to clarifairead | ### Annotate | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Added list annotations filter status | | ![](/img/improvement.jpg) | Added concept selection for tasks | | ![](/img/improvement.jpg) | Post/Patch annotations request now allow setting status | | ![](/img/improvement.jpg) | Changed task form options | | ![](/img/improvement.jpg) | Set annotation status to awaiting for review if the authorized user is not app owner | | ![](/img/improvement.jpg) | Return only input\_level annotation in input.data | | ![](/img/bug.jpg) | Drawing annotations: wrong embed model version id. Fixed | | ![](/img/bug.jpg) | Eliminated error if no annotation to be deleted | | ![](/img/bug.jpg) | Create one annotation for each bbox | ### Model | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Added support for adding and training on text in the platform | | ![](/img/new_feature.jpg) | Created a NLP mock prediction endpoint | | ![](/img/improvement.jpg) | Created test set to evaluate quick trained models or k-fold if no test search is specified | | ![](/img/improvement.jpg) | Added vocab\_id for demographics model concepts | | ![](/img/improvement.jpg) | Fixed sorting of A.G.E. concepts in golang for demographics model so we don't chop off sets of them | | ![](/img/improvement.jpg) | Deprecated Face from javascript Client | | ![](/img/improvement.jpg) | Deprecated Face from Java Client | | ![](/img/bug.jpg) | Confusion matrix predicted/true are swapped in evaluation results. Fixed | | ![](/img/bug.jpg) | Explorer Image/Text Joint embedding | | ![](/img/bug.jpg) | Fixed selectEmbedModelVersionId in detection apps | | ![](/img/bug.jpg) | Fixed generalModel imports and optimized video click handlers with useCallback hooks | ### Search | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Persisted the saved search used in train a model version | | ![](/img/bug.jpg) | Created log for annotation/search request/response | | ![](/img/bug.jpg) | Region Searches within Search Bar still use crop coordinates instead of base64 bytes. Fixed | ### Portal | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Created new Single Image View and Image Tools | | ![](/img/new_feature.jpg) | Enabled Display Text Thumbnails in App Grid View and App Details View | | ![](/img/new_feature.jpg) | Text Thumbnails display in Portal/Search Bar disabled | | ![](/img/new_feature.jpg) | Enabled View Text Assets in Portal's Image View | | ![](/img/new_feature.jpg) | Added Text Inputs To Explorer Apps | | ![](/img/new_feature.jpg) | Imported new icons for Labeler Image Tools into the style guide | | ![](/img/improvement.jpg) | Added login tracking to analytics package in Portal | | ![](/img/improvement.jpg) | Allowed pasting into the add inputs text area and clear the text box after clicking submit | | ![](/img/bug.jpg) | Search bar not visible. Fixed | | ![](/img/bug.jpg) | Removed all instances of worker\_id from Explorer | | ![](/img/bug.jpg) | Fixed popover left/right overflow | | ![](/img/bug.jpg) | Disabled all search by click handlers in Portal for Text Apps | | ![](/img/bug.jpg) | Click Search button icons on Thumbs not working for localized search. Fixed | | ![](/img/bug.jpg) | Fixed details page header missing description | | ![](/img/bug.jpg) | Fixed demo font syntax | ### Workflow | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Added a Range Slider to filter Workflow Predictions by Value | | ![](/img/improvement.jpg) | Updated Face workflow to include the detect faces as concepts for search | --- ## Release 6.4 # Release 6.4 ## Changelog 6.4 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Applications | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Improved responsiveness of collaborations tab in /apps | | ![](/img/improvement.jpg) | Enabled list collaborators to list deleted collaborators | | ![](/img/bug.jpg) | Login Form breaks app. Fixed | | ![](/img/bug.jpg) | Deleting an app no longer redirects to /apps | | ![](/img/bug.jpg) | Can’t create models in new app. Fixed | ### Inputs | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Fixed .webp files not working when sent as base64 | ### Annotate | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | MVP of labeler single image view functionality | | ![](/img/new_feature.jpg) | Support detection tasks | | ![](/img/improvement.jpg) | Detection Labeler: Color Coded Concepts | | ![](/img/improvement.jpg) | Add workflow\_id to task creation and show AI predictions to verify in labeler mode | | ![](/img/new_feature.jpg) | Split tasks admin view into tabs | | ![](/img/improvement.jpg) | Add visual sections to task form | | ![](/img/improvement.jpg) | Add default queries for "all inputs" and "all unlabelled inputs" in task create view | | ![](/img/improvement.jpg) | Autocomplete annotation user | | ![](/img/new_feature.jpg) | Implement Classification Task Review Logic | | ![](/img/new_feature.jpg) | Implement Review Process into tasks | | ![](/img/improvement.jpg) | Introduce stats collection APIs for worker stats | | ![](/img/new_feature.jpg) | Implement APIs for polygon region support | | ![](/img/new_feature.jpg) | Incorporate image filters for labelling | | ![](/img/improvement.jpg) | Update image tool icons | | ![](/img/new_feature.jpg) | Ability to zoom in on images | | ![](/img/improvement.jpg) | Remove ‘alt’ from hotkeys, just use letters and arrows | | ![](/img/new_feature.jpg) | Label to draw box in video frame using frame bytes | | ![](/img/improvement.jpg) | Display videos in labeler | | ![](/img/improvement.jpg) | Add video fps field for tasks | | ![](/img/improvement.jpg) | Draw Bounding Boxes in Labeler Detection Videos | | ![](/img/improvement.jpg) | Add video controls for video in labeler | | ![](/img/bug.jpg) | Fix Classification Annotation | | ![](/img/bug.jpg) | Video annotation deletion. Fixed | | ![](/img/bug.jpg) | AI Assist Predictions did not show for General workflow classification task. Fixed | | ![](/img/bug.jpg) | Fix Classification video annotation | | ![](/img/bug.jpg) | Display video in classification tasks | | ![](/img/bug.jpg) | Fix Labeler input urls | | ![](/img/bug.jpg) | Fix Annotation creation for video | | ![](/img/bug.jpg) | Fix Labeler post calls | | ![](/img/bug.jpg) | Detection Labeler: fix zoom | | ![](/img/bug.jpg) | Fixed image cropper task description | | ![](/img/bug.jpg) | Fix concept threshold creation | | ![](/img/bug.jpg) | Set annotation status ‘success’ | | ![](/img/bug.jpg) | Restrict tasks to only the assigned users | | ![](/img/bug.jpg) | Add validation to TaskForm’s concept field | | ![](/img/bug.jpg) | Display human tags for human box as child | ### Model | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Convert Deep Trained Model to Embedding Model for Use as "Base Workflow" | | ![](/img/new_feature.jpg) | Classification predictions for AI assistance | | ![](/img/new_feature.jpg) | Merge this detection and custom model prediction sections for detection models | | ![](/img/improvement.jpg) | Video labelling UI for classification. | | ![](/img/improvement.jpg) | Remove the non-creatable types from model mode | | ![](/img/improvement.jpg) | Improve the create classifier / detector view options in model mode | | ![](/img/improvement.jpg) | Add deep training options in model mode | | ![](/img/improvement.jpg) | Update random sampling model to have a slider | ### Predict | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Fixed public concept rank | ### Search | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Support detection evaluations in PostAnnotationSearchMetrics | | ![](/img/new_feature.jpg) | Support nlp search \(only filtering\) | | ![](/img/new_feature.jpg) | Add evaluations between two saved search label sets | | ![](/img/improvement.jpg) | Fix labeler search amount | | ![](/img/bug.jpg) | Error: "Cannot search over `annotations`" when clicking a general app. Fixed | | ![](/img/bug.jpg) | Model name and details is not populated upon model creation in model mode | | ![](/img/bug.jpg) | Fix annotation search when accessing the LabelerPage | | ![](/img/bug.jpg) | Search by annotation\_info should not return the embed annotation. Fixed | | ![](/img/bug.jpg) | Search for metadata in detection apps doesn't work. Fixed | ### Portal | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Create Dual Range Slider | | ![](/img/new_feature.jpg) | Enable patching the default workflow from Portal and error if needs reindex | | ![](/img/new_feature.jpg) | Allow drawing bounding boxes on paused video frames | | ![](/img/improvement.jpg) | Add scopes for collaborators and metrics to Portal | | ![](/img/improvement.jpg) | Allow up to 15-20x zoom level for really large images. | | ![](/img/improvement.jpg) | Allow selection the embed\_model\_version\_id from Portal when creating a model | | ![](/img/bug.jpg) | Fix inconsistent fps between uploading video and predicting video | | ![](/img/bug.jpg) | Missing frame time. Fixed | | ![](/img/bug.jpg) | Model annotations not appearing in Explorer. Fixed | | ![](/img/bug.jpg) | When creating the auto annotation workflow editing the workflow crashes Portal | | ![](/img/bug.jpg) | Fix image tools state | ### Workflow | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | CreateWorkflow model improvements | | ![](/img/improvement.jpg) | Allow custom concept models in the default app workflows | | ![](/img/improvement.jpg) | Add a "Make a Copy" or "Copy to New Workflow" button for each workflow | | ![](/img/improvement.jpg) | Allow patching the default workflow in Portal | | ![](/img/improvement.jpg) | Show the default workflow in the list of workflows for the app | | ![](/img/improvement.jpg) | Validate patching of default workflow is compatible in backend | | ![](/img/bug.jpg) | Large workflow name causes overlap in app details view. Fixed | | ![](/img/bug.jpg) | Portal crashes if page reloads during workflow add/edit. Fixed | | ![](/img/bug.jpg) | Detection workflow recompute also predict detect-concept | | ![](/img/bug.jpg) | Allow detect-concept models to be added to workflows | | ![](/img/bug.jpg) | Patch workflow create worker | | ![](/img/bug.jpg) | Fix validation of inputs in workflows | | ![](/img/bug.jpg) | Fix workflow embed\_join\_annotation\_id issue | ### Clients | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Clean up private API client repos | | ![](/img/bug.jpg) | Remove public workflows from Python client | --- ## Release 6.5 # Release 6.5 ## Changelog 6.5 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Applications | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Can't Access Main Apps Page with invalid collaborators. Fixed. | | ![](/img/bug.jpg) | Unable to create new Application \(General Detection\). Fixed. | | ![](/img/bug.jpg) | `application_sharing` scopes field should be `json` instead of `jsonb`. Fixed. | ### Inputs | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Pasting long text makes Uploader unusable due to lack of scrolling. Fixed. | | ![](/img/improvement.jpg) | Support uploading text containing emojis. | ### Annotate | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Integrate and Implement task deletion using new endpoints. | | ![](/img/improvement.jpg) | Implement /tasks CRUD in API. | | ![](/img/improvement.jpg) | Allow annotation writer model to set the `task_id` in `annotation_info`. | | ![](/img/improvement.jpg) | Make polygons a separate task type. | | ![](/img/improvement.jpg) | Add empty CRUD endpoints for tasks. | ### Model | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | `vocab_id` doesn't appear in the returned object for demographics model. Fixed. | | ![](/img/bug.jpg) | `segment-concept` model types are no longer returning the segmentation mask. Fixed. | | ![](/img/bug.jpg) | NLP text input does not scroll when longer than viewport height. Fixed. | | ![](/img/bug.jpg) | Clear text inputs after upload. | | ![](/img/improvement.jpg) | Improve the "TextFile" React Component for NLP. | | ![](/img/improvement.jpg) | Make existing model details view configurable by model type. | | ![](/img/bug.jpg) | Edit model should only contain the fields related to the selected model. Fixed | | ![](/img/bug.jpg) | NLP frontend text input is covered entirely blue when selected. Fixed. | | ![](/img/bug.jpg) | Enforce fields in post/patch models to adhere to model types. Fixed. | ### Predict | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Fixed fps issue for video predictions. | | ![](/img/bug.jpg) | Validate `stat_value_agg_type`. | ### Search | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Dropdown Search Help Menu no longer displays in the search bar. Fixed. | | ![](/img/new_feature.jpg) | Video thumbs have relevant timestamp in search. | | ![](/img/improvement.jpg) | Added adjustable search results threshold. | | ![](/img/new_feature.jpg) | Search over multi-embed workflows. | | ![](/img/new_feature.jpg) | Added search on input level. | | ![](/img/improvement.jpg) | Improved search query by using multi join. | | ![](/img/bug.jpg) | Fixed panic in list saved searches endpoint. | | ![](/img/bug.jpg) | Input metadata search from table not working. Fixed. | ### Portal | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | "Return to Log in " doesn't redirect to login page. Fixed. | | ![](/img/bug.jpg) | Clicking on image, or Explorer Mode with images that contain geo coordinates crashed the app. | | ![](/img/improvement.jpg) | Portal model predicts use hosted URL when available instead of normal URL. | | ![](/img/bug.jpg) | When selecting a concept and going to the next image the concept checkbox won't stay selected. Fixed | | ![](/img/improvement.jpg) | Allow multi-select from explorer grid view and add metadata. | | ![](/img/improvement.jpg) | Integrate and utilize new CRUD endpoints in Portal. | | ![](/img/new_feature.jpg) | Allow for pasted text to keep formatting in the text box. | | ![](/img/bug.jpg) | Prediction threshold slider custom model predicts without base workflow annotations. Fixed | | ![](/img/improvement.jpg) | Strings without spacing format properly in Explorer's Asset Grid View | | ![](/img/bug.jpg) | `annotation_info` should be a valid JSON in Model Mode. Fixed. | ### Workflow | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Hide the "add text" section of the add inputs modal for non text workflows. | | ![](/img/improvement.jpg) | Validate that all nodes in workflows list their inputs based on type. | | ![](/img/new_feature.jpg) | Add NLP to Workflows List | | ![](/img/improvement.jpg) | Generalize the iterations over regions/frames in workflow code. | | ![](/img/new_feature.jpg) | Add ability to "make a copy" of public\_workflows. | | ![](/img/improvement.jpg) | Allow indexing embedding from detect -> crop -> embed style workflows. | | ![](/img/improvement.jpg) | Allow setting input nodes for all users, not just @clarifai.com users. | | ![](/img/improvement.jpg) | Allow non-internal users setting input node when creating workflows. | | ![](/img/bug.jpg) | Create/Patch workflow uncaught exception. | ### Clients | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Update docs.clarifai.com to reflect our current API clients including grpc clients. | --- ## Release 6.6 # Release 6.6 ## Changelog 6.6 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### API | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Integrate Python functions service with API | | ![](/img/improvement.jpg) | List available model types through API | ### Model | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Fix video error from new face cluster model in staging env | | ![](/img/bug.jpg) | 21312 Ground truth data caseids must be nonempty and unique. Fixed | | ![](/img/improvement.jpg) | Update deep training to list the ModelTypes | | ![](/img/improvement.jpg) | Move model\_metadata to better place in protos. | | ![](/img/improvement.jpg) | Generalize the domex-visual-searcher model type | | ![](/img/improvement.jpg) | List available model types from backend services that provide models. | ### Portal | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Fixed bug in submitting finished Labeler Task | | ![](/img/bug.jpg) | Select all concepts checkbox can be de-synced from actual concepts badges. Fixed | | ![](/img/bug.jpg) | Disable Create Task button if not app owner | | ![](/img/bug.jpg) | Search by task\_id returns incorrect data | | ![](/img/bug.jpg) | Removed model creation from concept creation action in portal | | ![](/img/bug.jpg) | Video scrubber cannot be moved. Fixed | | ![](/img/bug.jpg) | Fixed ability to delete interpolation tracks \(you can only delete frames at this time\). | | ![](/img/bug.jpg) | Worker ids used instead of names in report overview in stats view. Fixed | | ![](/img/bug.jpg) | Y-axis on labels created stats page is wrong. Fixed | | ![](/img/bug.jpg) | In task creation, adding concepts should be simple to click all the options right away. Fixed | | ![](/img/bug.jpg) | Toggling concept visibility doesn't affect previously hidden child region. Fixed | | ![](/img/bug.jpg) | Leftover | | ![](/img/bug.jpg) | If reviewer is not a collaborator, UI sends empty reviewer id back instead of raising error. Fixed | | ![](/img/bug.jpg) | Bounding box disappears on resizing. Fixed | | ![](/img/bug.jpg) | While adjusting bounding box, it creates an additional bounding box over no object. Fixed | | ![](/img/bug.jpg) | Using Play button brings up "Oops" page. Fixed | | ![](/img/bug.jpg) | Error on opening Video Labeler. Fixed | | ![](/img/bug.jpg) | Labeler sidebar interaction bugs and unresponsiveness \(due to lack of optimistic UI\). Fixed | | ![](/img/bug.jpg) | Newly drawn object disappears from canvas after drawing, and reappears after API response. Fixed | | ![](/img/bug.jpg) | Infinite loading in Labeler Mode for app without any inputs. Fixed | | ![](/img/bug.jpg) | Use name field for tasks in Labeler admin. Fixed | | ![](/img/bug.jpg) | Clicking labeler icon crashes. Fixed | | ![](/img/bug.jpg) | Moving bounding box around repeatedly creates a race condition, shows error notification and duplicate box. Fixed | | ![](/img/bug.jpg) | Explorer inputs stale state. Fixed | | ![](/img/bug.jpg) | Diagnose issues affecting overall hanging/speed/performance of Labeler | | ![](/img/bug.jpg) | Carousel thumbnails not showing up in Labeler. Fixed | | ![](/img/bug.jpg) | Not able to create overlapping bounding boxes. Fixed | | ![](/img/bug.jpg) | When user adds mass metadata in Explorer, the UI says success but metadata does not persist. Fixed | | ![](/img/bug.jpg) | Create annotations while creating task. Fixed | | ![](/img/bug.jpg) | Labeler board showing wrong task type. Fixed | | ![](/img/bug.jpg) | Polygon annotations break Explorer. Fixed | | ![](/img/bug.jpg) | Concept autocomplete in Labeler task creation is showing clarifai/main concepts. Fixed | | ![](/img/bug.jpg) | Should not be allowed to create a task with no concepts if my app has no concepts. Fixed | | ![](/img/bug.jpg) | Fixed image tools state | | ![](/img/bug.jpg) | Restricted tasks to only the assigned users | | ![](/img/bug.jpg) | Add validation to TaskForm’s concept field | | ![](/img/bug.jpg) | Removed all instances of worker\_id from Explorer | | ![](/img/bug.jpg) | Create one annotation for each bbox | | ![](/img/bug.jpg) | 98011 panic on ListTasks. Fixed | | ![](/img/bug.jpg) | App names no longer display in Explorer. Fixed | | ![](/img/bug.jpg) | Empty Annotations are not displaying after drawing a new bounding box until after refreshing the page. Fixed | | ![](/img/bug.jpg) | Drawing a new bounding box in Explorer after previously labeling a region display an error. Fixed | | ![](/img/bug.jpg) | Video search results do not play at the most relevant video time. Fixed | | ![](/img/bug.jpg) | Change text upload UI to support moderation workflow | | ![](/img/bug.jpg) | Change object key lookup in boundingBoxContainer to use lodash/get | | ![](/img/bug.jpg) | Submitting Task for Review break Portal. Fixed | | ![](/img/bug.jpg) | Labeling a region on an asset with multiple detected regions will put the child annotation in the wrong group in Explorer's sidebar. Fixed | | ![](/img/bug.jpg) | ConvertToBoundingBoxRegion function breaks Explorer when annotation information has not loaded at time of render. Fixed | | ![](/img/bug.jpg) | Video Frame Annotating in Explorer throws errors. Fixed | | ![](/img/bug.jpg) | When drawing a new bounding box, Base64 string for video annotations shows the wrong regions. Fixed | | ![](/img/bug.jpg) | Fixe 10MB issue with video uploads | | ![](/img/bug.jpg) | Detection Regions and Indexes are thrown off on video assets. Fixed | | ![](/img/bug.jpg) | Video Search Results still on showing Inputs. Fixed | | ![](/img/bug.jpg) | DetailsPageHeader adds 2.25rem margin to the DetailsPageBody. Fixed | | ![](/img/bug.jpg) | Video Interpolation in Labeler breaks dev. Fixed | | ![](/img/bug.jpg) | Redux is no longer calculating the sample\_ms rate, preventing bounding boxes from rendering. Fixed | | ![](/img/bug.jpg) | Fixed video pause error when navigating between videos | | ![](/img/bug.jpg) | ImagePile in Labeler Task View does not display image thumbs due to extraneous object nesting. Fixed | | ![](/img/bug.jpg) | Remove warning error from console for immutable passed in props to SearchGrid.js | | ![](/img/bug.jpg) | Post annotation to detection region should use region id in portal. Fixed | | ![](/img/bug.jpg) | Delete app button in app details takes you to blank page. Fixed | | ![](/img/bug.jpg) | Incorrect bbox/label numbers displayed in image. Fixed | | ![](/img/bug.jpg) | Multiple video thumbs selected in search results when selecting one thumb. Fixed | | ![](/img/improvement.jpg) | Improve Labeler mode window resizing. | | ![](/img/improvement.jpg) | W and E hotkeys for image labelling to go left/right. | | ![](/img/bug.jpg) | Polygon annotations break Explorer. Fixed. | | ![](/img/bug.jpg) | Polygons regions don’t appear when panning and zooming. Fixed. | | ![](/img/improvement.jpg) | Allow users to create concepts on task create view. | | ![](/img/bug.jpg) | Fixed task list item count query. | | ![](/img/improvement.jpg) | Utilising new task endpoints to Create tasks and integrate to show tasks in Portal. | | ![](/img/bug.jpg) | LabelerPage refresh error. Fixed. | | ![](/img/bug.jpg) | Should not be allowed to create a task with no concepts if my app has no concepts. Fixed. | | ![](/img/bug.jpg) | Unknown page Error. Complete interpolation of an object doesn't show bbox. Complete tracking of a box will disappear from the video. Fixed. | | ![](/img/bug.jpg) | Fixed carousel padding. | | ![](/img/bug.jpg) | Labeler board showing wrong task type. Fixed. | | ![](/img/improvement.jpg) | Integrate worker/reviewer side of Labeler. | | ![](/img/improvement.jpg) | Add 'name' field to new Tasks. | | ![](/img/bug.jpg) | Create annotations while creating task. | | ![](/img/bug.jpg) | Carousel thumbnails not showing up in Labeler. Fixed. | | ![](/img/improvement.jpg) | No image clearing/loading indicator in Labeler. Fixed. | | ![](/img/bug.jpg) | Use name field for tasks in Labeler administration. | | ![](/img/bug.jpg) | Bulk labeling value does not update in store upon labeling. Fixed. | | ![](/img/improvement.jpg) | Record time per annotation and per input to /stats/values in Labeler mode of Portal. | | ![](/img/improvement.jpg) | Query and display stats across workers per task for time and count of annotations. | | ![](/img/bug.jpg) | Concept autocomplete in Labeler task creation is showing clarifai/main concepts. Fixed. | | ![](/img/new_feature.jpg) | Implement polygon task type in Labeler. | | ![](/img/bug.jpg) | Applying filters in Portal breaks bulk labeling / unlabeling. Fixed. | | ![](/img/bug.jpg) | Unable to bulk-label annotations. Fixed. | | ![](/img/improvement.jpg) | Change submit to "Complete Task" in Labeler page and add progress bar as it's working. | | ![](/img/improvement.jpg) | Allow Patching region annotations in Labeler mode. | | ![](/img/improvement.jpg) | Add AI assist thresholding. | | ![](/img/improvement.jpg) | Add ability to set annotation\_info in the annotation writer | | ![](/img/bug.jpg) | `annotation_info` should be a valid JSON in Model Mode. Fixed. | | ![](/img/improvement.jpg) | Upgrade gulp and node to latest version for testing-library support | | ![](/img/improvement.jpg) | Write PropType declarations for componens/ConceptListTable | | ![](/img/improvement.jpg) | Enable collapse behavior in sidebar concepts | | ![](/img/improvement.jpg) | Create atomic reusable sidebar components | | ![](/img/improvement.jpg) | Integrate React Testing Library | | ![](/img/improvement.jpg) | Konva: Image centering, zooming, panning | | ![](/img/improvement.jpg) | Move toolbar logic to react context | | ![](/img/improvement.jpg) | Refactor TaskForm related thunks to sagas | | ![](/img/improvement.jpg) | \[Rearch\]Scaffold Labeler Redux in a new nested state slice & Implement Sagas | | ![](/img/improvement.jpg) | \[P2\] Task id is used in dropdown of stats tasks rather than task.name | | ![](/img/improvement.jpg) | \[P0\]Show taskId at task list | | ![](/img/improvement.jpg) | Move region visibility state to its own React Context | | ![](/img/improvement.jpg) | Get sidebar list data directly from redux | | ![](/img/improvement.jpg) | \[P1\] Don't hide task form if error occurs | | ![](/img/improvement.jpg) | Remove delay of annotation request | | ![](/img/improvement.jpg) | Remove animation for showing concepts on right side | | ![](/img/improvement.jpg) | Perf: only fetch input predictions/annotations if user stays on image, not while navigating | | ![](/img/improvement.jpg) | Get Labeler internal features ready for internal users | | ![](/img/improvement.jpg) | Offload annotation creation to backend | | ![](/img/improvement.jpg) | allow reviewers update annotations | | ![](/img/improvement.jpg) | No image clearing/loading indicator in Labeler | | ![](/img/improvement.jpg) | Controls for resizing bounding boxes need to be more visible | | ![](/img/improvement.jpg) | Fabric rendering to be real-time; sync from API in background | | ![](/img/improvement.jpg) | make tasks endpoint public | | ![](/img/improvement.jpg) | list task by worker id/reviewer id | | ![](/img/improvement.jpg) | Add 'name' field to new Tasks | | ![](/img/improvement.jpg) | Integrate worker/reviewer side of labeller | | ![](/img/improvement.jpg) | Integrate and Implement task deletion using new endpoints | | ![](/img/improvement.jpg) | Utilising new task endpoints to Create tasks and integrate to show tasks at portal | | ![](/img/improvement.jpg) | Integrate and utilise new CRUD endpoints in portal | | ![](/img/improvement.jpg) | Allow users to create concepts on task create view | | ![](/img/improvement.jpg) | Add empty CRUD endpoints for tasks | | ![](/img/improvement.jpg) | Make polygons a separate task type | | ![](/img/improvement.jpg) | Add AI assist thresholding | | ![](/img/improvement.jpg) | W and E hotkeys for image labelling to go left/right. | | ![](/img/improvement.jpg) | Improve labeler mode window resizing. | | ![](/img/improvement.jpg) | Allow Patching region annotations in labeler mode. | | ![](/img/improvement.jpg) | crank up internal message size to handle larger videos with more outputs | | ![](/img/improvement.jpg) | Introduce stats collection APIs for worker stats. | | ![](/img/improvement.jpg) | Implement /tasks CRUD in API. | | ![](/img/improvement.jpg) | Add frame.id to API as well as track id. | | ![](/img/improvement.jpg) | Remove ‘alt’ from hotkeys, just use letters and arrows straight up | | ![](/img/improvement.jpg) | Update image tool icons | | ![](/img/improvement.jpg) | Add support for fields under ENUM values during model creation | | ![](/img/improvement.jpg) | Implement Dynamic model types | | ![](/img/improvement.jpg) | Ungrouped Annotations/New Annotation Regions should display at the top of Explorer's Detections List | | ![](/img/improvement.jpg) | Change explorer to use sample\_ms instead from network response instead of deducing the value | | ![](/img/improvement.jpg) | Hide Workflow List Elements if below Range Slider value | | ![](/img/improvement.jpg) | Modify Annotating from Custom Model Predictions to post new annotations. | | ![](/img/improvement.jpg) | Update model mode with new designs | | ![](/img/improvement.jpg) | Refactor ImageUtils.js file to individual functions instead of one object | | ![](/img/improvement.jpg) | Added threshold search result in Portal | | ![](/img/improvement.jpg) | Update media player icons | | ![](/img/improvement.jpg) | Display timestamp bar in Explorer grid view for video results | | ![](/img/improvement.jpg) | Concept relation should autocomplete concept name | | ![](/img/new_feature.jpg) | Send email to workers when they are added to task | | ![](/img/new_feature.jpg) | Assigning a worker or reviewer to a task sends an email | | ![](/img/new_feature.jpg) | Apps with empty workflow should allow all task types \(concepts, bounding box, polygon\) during task creation | | ![](/img/new_feature.jpg) | Edit Task feature | | ![](/img/new_feature.jpg) | Support consensus review settings | | ![](/img/new_feature.jpg) | Support detection tasks | | ![](/img/new_feature.jpg) | Create new Single Image View and Image Tools | | ![](/img/new_feature.jpg) | Task view UI for workers | | ![](/img/new_feature.jpg) | Introduce AWAITING\_REVIEW status for annotations | | ![](/img/new_feature.jpg) | Split tasks admin view into tabs | | ![](/img/new_feature.jpg) | Test out idea behind tasks as saved searches and POST /annotations iterations | | ![](/img/new_feature.jpg) | Implement search by annotation.status in backend | | ![](/img/new_feature.jpg) | Search by images or video type in the right hand side bar of explorer's grid view | | ![](/img/new_feature.jpg) | Video Crop Region Search | | ![](/img/new_feature.jpg) | video thumbs display relevant frame in search | ### Workflows | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Display workflow detection predictions on the main/large image in Portal | --- ## Release 6.7 # Release 6.7 ## Changelog 6.7 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### API | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Don't allow updating task workers | | ![](/img/improvement.jpg) | Don't create duplicated task annotations | | ![](/img/improvement.jpg) | Fix detection evals showing and too many metrics calls | | ![](/img/improvement.jpg) | Fix pillow installs for webp | | ![](/img/improvement.jpg) | Add enum for embed model version id field type. | | ![](/img/improvement.jpg) | Don't show model types for backends that aren't responding. | | ![](/img/improvement.jpg) | Clean up output\_info.data path | | ![](/img/improvement.jpg) | Add model\_type\_id to Model protos. | | ![](/img/improvement.jpg) | Add `/models/types/{model\_type\_id}` endpoint | | ![](/img/improvement.jpg) | Prevent models\_versions.is\_public from every being null. | | ![](/img/bug.jpg) | model mode types that are internal only are being returned. | | ![](/img/improvement.jpg) | Create Labeling Order Object and send email to datalabeling@clarifai.com each time backend receives an Labeling Order Object & makes datalabeling a super user | | ![](/img/improvement.jpg) | App reindex | | ![](/img/improvement.jpg) | Patchable multi-embed workflows with re-index | ### Model | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Demographics model is now broken in model gallery. | | ![](/img/improvement.jpg) | Deprecate model.type from model mode | | ![](/img/improvement.jpg) | Update model gallery design | | ![](/img/improvement.jpg) | embed\_model\_version\_id should be a dropdown | ### Portal | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Add versioning to repo, redux, on screen | | ![](/img/improvement.jpg) | Allow reviewer to modify annotations during review process. | | ![](/img/new_feature.jpg) | Store entire canvas state in redux/context, and drive canvas updates by central store | | ![](/img/improvement.jpg) | batch v2 shape events by only 1 PATCH/DELETE request | | ![](/img/improvement.jpg) | Delete functionality v2 | | ![](/img/improvement.jpg) | Completely detach labeler rendering from server syncing process to enable background syncing | | ![](/img/improvement.jpg) | Input navigating functionality in v2 | | ![](/img/improvement.jpg) | Set new regionId as "selected" shape if user has selected a transient shape during async updates | | ![](/img/improvement.jpg) | Region edit + delete API sync | | ![](/img/improvement.jpg) | Fix annotation denormalizer to rehydrate actual concept value | | ![](/img/improvement.jpg) | Create new region in labeler v2 | | ![](/img/improvement.jpg) | Logic to reduce batched drawing events to least number of API operations | | ![](/img/improvement.jpg) | Implement new selectors for regions in v2 | | ![](/img/improvement.jpg) | Normalize Annotations & Regions data for redux storage | | ![](/img/improvement.jpg) | Nest labelerTasks reducer inside labeler reducer | | ![](/img/improvement.jpg) | Konva: Concept Region drawing implementation | | ![](/img/improvement.jpg) | Konva: Implement Rect Transformation | | ![](/img/improvement.jpg) | Implement a futureproof schema for labeler interaction events | | ![](/img/improvement.jpg) | Move to event-driven design & have the ability to batch updates using custom logic | | ![](/img/improvement.jpg) | Create a single Sidebar component for all Labeling types, make children configurable | | ![](/img/improvement.jpg) | Remove all props unnecessarily passed from LabelerPage to deep children and make components get props from Redux only | | ![](/img/improvement.jpg) | Remove all logic from components to sagas for higher level orchestration of features | | ![](/img/bug.jpg) | Video selector improvements & test updation | | ![](/img/bug.jpg) | Can only save 50 annotations on an image `{Usability}` | | ![](/img/bug.jpg) | Konva: resizing BBox below minimum size and "crossing over" makes things awry | | ![](/img/bug.jpg) | LabelerPage complete re-render of all components on mouseHover, mousMove \(img attached\) | | ![](/img/bug.jpg) | Cypress script doesn't terminate webpack-dev-server child process | | ![](/img/bug.jpg) | Cypress pre-run script doesn't check if dev server is already running | | ![](/img/bug.jpg) | Create Unit+Integration testing framework | | ![](/img/bug.jpg) | Integrate headless Cypress with build testing | | ![](/img/new_feature.jpg) | Switch to react-konva for performant canvas rendering | | ![](/img/improvement.jpg) | Add task id to task list | | ![](/img/improvement.jpg) | Panning functionality improvements | | ![](/img/bug.jpg) | Lock video playback and interpolation to fps | | ![](/img/bug.jpg) | Annotations created with interpolation seem to have incorrect frame indices | | ![](/img/bug.jpg) | Bounding Boxes and Concepts inconsistent during video playback `{Usability}` | | ![](/img/bug.jpg) | While annotating video, interpolation freezes and all annotations disappear | | ![](/img/bug.jpg) | Boxes/Interpolation objects are not saving after task submission | | ![](/img/improvement.jpg) | Display task instructions to workers in labeler mode | | ![](/img/improvement.jpg) | Add infinite scroll loading to labeler carousel | | ![](/img/bug.jpg) | Not incrementing onNext and onPrev pages in Labeler Carousel | | ![](/img/improvement.jpg) | Display only minimal log in Portal react app | | ![](/img/improvement.jpg) | Update Model mode to use the GET /models/types endpoint | | ![](/img/improvement.jpg) | Add list/grid toggle in model mode on all view | | ![](/img/improvement.jpg) | Collectors UI should use the layout similar to ModellingMode/LabellerMode | | ![](/img/bug.jpg) | Bulk add concepts to region annotations in app with multi-embed base workflow | | ![](/img/bug.jpg) | Profile page crashes on load | | ![](/img/bug.jpg) | Model mode array of concepts should be unique | | ![](/img/bug.jpg) | Model creation/edit bugs | | ![](/img/bug.jpg) | New Collector page not scrollable | | ![](/img/bug.jpg) | App Workflows - Unable to update model version for custom models | | ![](/img/new_feature.jpg) | Display Created At Date in App Grid View | | ![](/img/new_feature.jpg) | Support .txt files from local file browser | | ![](/img/new_feature.jpg) | Support uploading of multiple video assets as well as image and video assets within the same CSV file | | ![](/img/new_feature.jpg) | Add better user feedback for uploading text assets | | ![](/img/new_feature.jpg) | Upload Text by CSV for NLP | | ![](/img/improvement.jpg) | Final NLP MVP Feature Changes | | ![](/img/improvement.jpg) | Remove 0 area detection filtering from frontend code | | ![](/img/improvement.jpg) | Modify accepted CSV upload format so every column corresponds to a network request field | | ![](/img/improvement.jpg) | Resolve final bugs with bounding box indexes | | ![](/img/improvement.jpg) | Prevent uploading image and video asset types to Text apps | | ![](/img/improvement.jpg) | No Visual Feedback for Text input Upload | | ![](/img/improvement.jpg) | Support Uploading Files through the OS File Browser for NLP | | ![](/img/bug.jpg) | Custom Model Prediction Bounding Boxes are misaligned from the Detections Bar | | ![](/img/bug.jpg) | Workflow Tab should display and load on initial view for text apps | | ![](/img/bug.jpg) | Unable to navigate between text assets within explorer's asset detail view | | ![](/img/bug.jpg) | CSV uploads not parsing metadata and concepts | | ![](/img/bug.jpg) | Explorer's Advanced Search does not support searching by concepts | | ![](/img/bug.jpg) | Training a classification model no longer display anything within the Custom Model Predictions tab | | ![](/img/improvement.jpg) | create annotation CUD sagas for labeler v2 | | ![](/img/improvement.jpg) | Don't create task annotations in frontend | | ![](/img/bug.jpg) | Form: Input Source showing auto complete options from other apps | | ![](/img/bug.jpg) | Rich text instructions icon bugs | | ![](/img/bug.jpg) | Task create form doesnt force you to set a reviewer if you specify manual review | | ![](/img/bug.jpg) | TypeError: val.add is not a function | | ![](/img/bug.jpg) | Fix autocomplete when user selects "All inputs" for selecting inputs in task creation | | ![](/img/bug.jpg) | Error pops up when collaborator tries to edit task | | ![](/img/bug.jpg) | When I attempt to edit an existing labeling task t... | | ![](/img/bug.jpg) | Page not responding \[Usability\] | | ![](/img/bug.jpg) | \[Explorer\] concept thumbnails aren't displaying from model details view | | ![](/img/bug.jpg) | Concept Detail View displays incorrect assets | | ![](/img/improvement.jpg) | Add all concepts button to model mode forms | | ![](/img/bug.jpg) | Concept Autocomplete in Model Mode doesn't always display | ### Workflows | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Allow run workflow and search embedding from embed model in workflow | ### Applications | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Metadata Namespacing for Clarifai Apps | | ![](/img/bug.jpg) | App details page should send a user to models page to create models rather than using modal | --- ## Release 6.8 # Release 6.8 ## Changelog 6.8 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Labeler | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Use single LabelerToolbar for all labelers, make shared using Context | | ![](/img/improvement.jpg) | Remove all computation from components, move to selectors for perf | | ![](/img/bug.jpg) | Fixed carousel scroll behavior | | ![](/img/bug.jpg) | Added button to add collaborator when adding reviewer | | ![](/img/bug.jpg) | Post incorrect bounding box. Fixed | | ![](/img/bug.jpg) | Refreshing in Labeler Crashes Portal. Fixed | | ![](/img/new_feature.jpg) | Add Grid Review UI to Review Page | | ![](/img/bug.jpg) | Panning while playing a video renders rects incorrectly. Fixed | | ![](/img/bug.jpg) | Navigation from the task creation page if task creation fails. Fixed | | ![](/img/improvement.jpg) | Konva: Image filters | | ![](/img/improvement.jpg) | Konva: Drawing rects | | ![](/img/improvement.jpg) | Konva: Drawing polygons | | ![](/img/improvement.jpg) | Smaller shapes should supersede zIndex values if they are engulfed by larger ones | | ![](/img/bug.jpg) | Region selectors inefficient and running on each call, bypassing reselect memoization. Fixed | | ![](/img/bug.jpg) | v2 interpolation: app crash on reload. Fixed | | ![](/img/bug.jpg) | CSS issue causing VideoControls to be inaccessible to mouse. Fixed | | ![](/img/bug.jpg) | v2 video: no thumbails in carousel. Fixed | | ![](/img/bug.jpg) | v2 keyboard hint showing weird characters. Fixed | | ![](/img/bug.jpg) | Selected Shape becomes unselected on playing video \(make selection persist across track\). Fixed | | ![](/img/bug.jpg) | Ron is unable to create a task with AI assist in prod. Fixed | | ![](/img/bug.jpg) | Write and Preview Tab style. Fixed | | ![](/img/bug.jpg) | Task create form shows name as "undefined undefined" when a user has not filled in profile details. Fixed | | ![](/img/bug.jpg) | Input source that was selected should be shown when task selected. Fixed | | ![](/img/bug.jpg) | Create order fails if I'm a clarifai user. Fixed | | ![](/img/bug.jpg) | Layout in order admin form has some issues. Fixed | | ![](/img/bug.jpg) | Multiple errors when creating bounding boxes. Fixed | | ![](/img/bug.jpg) | Cannot see annotations from a collaborator in v2 linear review. Fixed | | ![](/img/bug.jpg) | Keyboard shortcuts dont work in labeler v2. Fixed | | ![](/img/bug.jpg) | When a worker opens labeler, display the instructions by default. Fixed | | ![](/img/bug.jpg) | Non-clarifai users should have v2 only, clarifai accounts should have v2 by default with option to switch to v1. Fixed | | ![](/img/bug.jpg) | Cursor should change to a crosshair when drawing a bounding box. Fixed | | ![](/img/bug.jpg) | Cursor should change to an open hand when panning is selected and closed hand when grabbing/panning. Fixed | | ![](/img/bug.jpg) | Reviewer cannot see annotations by collaborators. Fixed | | ![](/img/bug.jpg) | If the reviewer is NOT the app owner, clicking review takes them to explorer. Fixed | | ![](/img/bug.jpg) | Tooltip for labeler nav icon should be uppercase. Fixed | | ![](/img/bug.jpg) | As a worker, if I return to Labeler, I should be able to continue from where I was previously. Fixed | | ![](/img/bug.jpg) | LaaS orders can't assign inputs which block the workers. Fixed | | ![](/img/bug.jpg) | Community users should have LaaS option grayed out with an explanation. Fixed | | ![](/img/bug.jpg) | Make the "order admin view" text larger and prominent as a section header. Fixed | | ![](/img/bug.jpg) | In labeler UI \(worker\), submit button should say "Submit Input for Review" to make it clear what the button does. Fixed | | ![](/img/bug.jpg) | \[P3\] In all tasks view, only app owner should see edit/delete icons | | ![](/img/bug.jpg) | Carousel blocks input visibility \(not just video controls\). Fixed | | ![](/img/bug.jpg) | In labeler UI carousel, show a check for any input that was submitted, and gray it out slightly. Fixed | | ![](/img/improvement.jpg) | Separate annotation sagas + standardise request batching code \(for v2 store\) | | ![](/img/bug.jpg) | When creating a task in an app w/ no concepts, “Select all concepts” should not be checked by default. There are no concepts created yet. Fixed | | ![](/img/new_feature.jpg) | Update task status on task list | | ![](/img/bug.jpg) | Deleting an annotation in reviewer deletes all annotations. Fixed | | ![](/img/bug.jpg) | Too many scrollbars in sidebar. Fixed | | ![](/img/bug.jpg) | Partition worker strategy Error. Fixed | | ![](/img/improvement.jpg) | Integrate feature gating with LaaS. | | ![](/img/bug.jpg) | Panning state not in sync with drawing/moving. Fixed | | ![](/img/bug.jpg) | Have to click the + button 2 times to make it work. Fixed | | ![](/img/bug.jpg) | Dragging mouse outside of the canvas while drawing leaves the drawing in inconsistent state. Fixed | | ![](/img/bug.jpg) | Delete icon on v2 sidebar deletes all annotations on the input. Fixed | | ![](/img/bug.jpg) | Resizing shapes near the right edge of the frame causes weird resize behavior. Fixed | | ![](/img/bug.jpg) | Entering date manually in Order control modal fixed | | ![](/img/bug.jpg) | Carousel Thumbnail animation not working; images looking weird in aspect-ratio due to incorrect CSS | | ![](/img/bug.jpg) | Task Form console errors. Fixed | | ![](/img/bug.jpg) | Add video icon to carousel for video inputs. Fixed | | ![](/img/bug.jpg) | When we use keyboard shortcuts to activate a concept for bounding boxes, show visual feedback | | ![](/img/bug.jpg) | Cannot read property '0' of undefined. Fixed | | ![](/img/bug.jpg) | Review tab shows new tasks that have no work ready to review. Fixed | | ![](/img/bug.jpg) | Video Interpolation doesn't work. Fixed | | ![](/img/bug.jpg) | Labeler UI sees last input even after submitting everything. Fixed | | ![](/img/bug.jpg) | Box disappears for a second while drawing on video. Fixed | | ![](/img/bug.jpg) | Disable worker input when editing a task. Fixed | | ![](/img/bug.jpg) | Input data stops being fetched if labeler is exited once and revisited. Fixed | | ![](/img/bug.jpg) | Make v2 annotations state flatter. Fixed | | ![](/img/bug.jpg) | Darker colors poorly visible in sidebar region items. Fixed | | ![](/img/bug.jpg) | Mysterious Phantom Boxes Appearing. Fixed | | ![](/img/bug.jpg) | Video not loading. Fixed | | ![](/img/bug.jpg) | Multiple boxes appearing. Fixed | | ![](/img/bug.jpg) | Misaligned Boxes. Fixed | | ![](/img/bug.jpg) | Bounding Boxes and Concepts inconsistent during video playback `{Usability}`. Fixed | | ![](/img/bug.jpg) | Change Labeler to use getHostedAssetUrl. Fixed | | ![](/img/bug.jpg) | Enable drawing even if annotations haven't loaded. Fixed | | ![](/img/improvement.jpg) | Use new feature flags at frontend & Labeler for all | | ![](/img/bug.jpg) | Can't add Iris workers to LaaS order. Fixed | | ![](/img/bug.jpg) | Instructions shouldn't be false while editing. Fixed | | ![](/img/bug.jpg) | Labelers/Reviewers should not see "--" when the task does not have AI Assist enabled. Fixed | | ![](/img/improvement.jpg) | Carousel should show some visual feedback when an input has been rejected | | ![](/img/bug.jpg) | Carousel flickers and re-renders images when submitting annotations. Fixed | | ![](/img/improvement.jpg) | Add loading indicator to labeler view when fetching data | | ![](/img/bug.jpg) | Moving a polygon to the edge of the input causes it to patch outside the allowed range. Fixed | | ![](/img/bug.jpg) | Wrong worker\_per\_input field. Fixed | | ![](/img/bug.jpg) | Hide other regions during interpolation. Fixed | | ![](/img/bug.jpg) | Concepts Tasks: Cannot read property 'id' of undefined. Fixed | | ![](/img/improvement.jpg) | Carousel should show some visual feedback when an input has been skipped | | ![](/img/bug.jpg) | Polygon points are sometimes too small to click. fixed | | ![](/img/bug.jpg) | Send embed model id for image annotations. Fixed | | ![](/img/bug.jpg) | Skipping/Submitting annotations causes unnecessary rerenders of the entire carousel \(all thumbs\). Fixed | | ![](/img/bug.jpg) | Get an error when submitting an input in a classification task. Fixed | | ![](/img/improvement.jpg) | Improve concept creation process for new apps that you want to label | | ![](/img/bug.jpg) | Labeler Reviewer No longer renders assets. Fixed | | ![](/img/new_feature.jpg) | Add "Orders" section to task list admin view | | ![](/img/new_feature.jpg) | Add checkbox to task creation for LaaS Orders | | ![](/img/bug.jpg) | List name field instead of id fields in task lists. Fixed | | ![](/img/improvement.jpg) | Set task error code and error description if task annotations pipeline fails | | ![](/img/bug.jpg) | Label task submit error: Malformed or invalid request. Fixed | | ![](/img/bug.jpg) | Label video - playback control issue fixed | | ![](/img/improvement.jpg) | Hovering annotations in sidebar of Labeler, should highlight the region in the image. | | ![](/img/bug.jpg) | Jumping Boxes during video interpolation. Fixed | | ![](/img/improvement.jpg) | Polygon rendering in Labeler v2 | | ![](/img/improvement.jpg) | Virtual scrolling input carousel | | ![](/img/bug.jpg) | LabelOrders not fetched when refresh at /labeler page. Fixed | | ![](/img/improvement.jpg) | Lock Edit feature for LaasOrders other than pending orders | | ![](/img/improvement.jpg) | Account for system states \(inputId, taskID\) between heartbeats and account for them in canvas interaction manager | | ![](/img/improvement.jpg) | Cleanup labelerv2 state on unmount | | ![](/img/improvement.jpg) | V2 Rendering Video Regions | | ![](/img/improvement.jpg) | V2 Video Interpolation | | ![](/img/improvement.jpg) | Labeler saga to process all remaining actions on input change & before user exits | | ![](/img/bug.jpg) | Sometimes, bounding box values on Transformer go in the negative, Fixed | | ![](/img/bug.jpg) | Task Form: Convert fps -> sample\_ms | | ![](/img/improvement.jpg) | Let Clarifai user permissions for status & ETA change | | ![](/img/improvement.jpg) | Implement clarifai user journey for LaaS | | ![](/img/improvement.jpg) | Seperate LaaS order tasks from simple labeling tasks. | | ![](/img/improvement.jpg) | Edit task functionality for clarifai user | | ![](/img/improvement.jpg) | Include Order Task in "assigned to me" and "for review" | | ![](/img/bug.jpg) | Regions disappeared in sidebar. Fixed | | ![](/img/improvement.jpg) | Implement a way for Clarifai users to review Order tasks | | ![](/img/improvement.jpg) | Video Rendering Sync with FPS | | ![](/img/improvement.jpg) | Reconcile V1 and V2 video frame index | | ![](/img/improvement.jpg) | Convert incorrectly created fps to sampleMs | | ![](/img/improvement.jpg) | Better signposting of task instruction preview panel | | ![](/img/bug.jpg) | Reset Button doesn't work. Fixed | | ![](/img/bug.jpg) | Can't go back from Labeler UI. Fixed | | ![](/img/bug.jpg) | Collaborators can not add collaborators. Fixed | | ![](/img/improvement.jpg) | Labeler: Add both index and time to all video annotations | | ![](/img/new_feature.jpg) | Implement polygon drawing | | ![](/img/bug.jpg) | Toolbar Next & Previous button issue fixed | | ![](/img/bug.jpg) | Worker filters don't work in review grid sidebar. Fixed | | ![](/img/bug.jpg) | Fixed styling/layout of progress bar in the grid review page | | ![](/img/improvement.jpg) | Add "select all" link next to each concept heading in the grid | | ![](/img/new_feature.jpg) | Integrate order task with current implementation for reviewer and worker | | ![](/img/bug.jpg) | Modify Labelerv2 sagas to be compatible with listening to polygon events | | ![](/img/bug.jpg) | Instructions editor should not show toolbar toggle, when in preview mode. Fixed | | ![](/img/bug.jpg) | Worker strategy should be included while adding workers. Fixed | | ![](/img/bug.jpg) | Task creation form concept field should correctly handle paginated response. Fixed | | ![](/img/bug.jpg) | Partition worker strategy should only be selectable if you have more than 1 worker. Fixed | | ![](/img/new_feature.jpg) | Labeler v2 submit functionality | | ![](/img/bug.jpg) | GridReview: app crash due to code for getting reviewer name. Fixed | ### API | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Patch annotation req failed. Fixed | | ![](/img/new_feature.jpg) | Allow any type of task when the app default workflow is empty workflow | | ![](/img/new_feature.jpg) | LaaS billing | | ![](/img/bug.jpg) | Undo the delete of cvat persistent volumes | | ![](/img/bug.jpg) | Copier failed in workflow prediction and causing 99009. Fixed | | ![](/img/new_feature.jpg) | Make gRPC C\# client | | ![](/img/new_feature.jpg) | Make gRPC PHP client | | ![](/img/bug.jpg) | Feedback for malinformend CSV formats | | ![](/img/improvement.jpg) | Make PostKeys and PatchKeys support apps->user\_id set to "me" | | ![](/img/new_feature.jpg) | Add automated testing of documentation code examples | | ![](/img/improvement.jpg) | change to getHostedAssetUrl to support returning both video thumbnails and video urls | | ![](/img/improvement.jpg) | Prepare clients for the secure gRPC channel | | ![](/img/improvement.jpg) | Update the gRPC copying code with C\#, PHP | | ![](/img/improvement.jpg) | Use sendgrid template for email | ### Model | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Add AWS Lambda to model mode | | ![](/img/new_feature.jpg) | Add AWS Lambda model type to API | | ![](/img/new_feature.jpg) | Put Fairface model in production | | ![](/img/improvement.jpg) | Append landmark and pose annotations to Fairface dataset | | ![](/img/bug.jpg) | Fix empty status response | | ![](/img/bug.jpg) | Miscellaneous Fixes on Object Counter and KNN | | ![](/img/improvement.jpg) | Allow empty statusCallbackURL and entityStatusCallbackURL | | ![](/img/new_feature.jpg) | Smart Reply | | ![](/img/new_feature.jpg) | Remove isInternalUser Selector from Text Features | ### Workflow | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Editing the Empty workflow throws an error in portal. Fixed | | ![](/img/improvement.jpg) | Add Filtering By Concepts for Text workflows | | ![](/img/improvement.jpg) | Add supress\_output field option to each workflow node in create workflow view | | ![](/img/improvement.jpg) | Add workflows tab to model gallery | | ![](/img/bug.jpg) | Allow reindexing to different workflow without having a shared workflow node \(with the old one\) | | ![](/img/bug.jpg) | No response when "Update workflow" button is pressed. Fixed | ### Portal | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Refactor Sidebar ✅/❌ functionality to sagas | | ![](/img/improvement.jpg) | Combine Tool components | | ![](/img/bug.jpg) | Cannot add card. Something went wrong. Fixed | | ![](/img/bug.jpg) | DOMEX face app using face detect. Clicking on any image causes portal to crash. Fixed | | ![](/img/bug.jpg) | App in staging, crashing when using pause/play with video. Fixed | | ![](/img/improvement.jpg) | Improve algorithm for grouping annotations and predictions in explorer. | | ![](/img/bug.jpg) | N "Predicted Bounding Boxes" toggle button only works after clicking twice | | ![](/img/improvement.jpg) | Show track ID for videos in explorer | | ![](/img/improvement.jpg) | Use ModelType to validate args and persist default values with model versions. | | ![](/img/improvement.jpg) | Update create workflows page design | | ![](/img/improvement.jpg) | Add sortable columns when in list view of model mode. Fixed | | ![](/img/improvement.jpg) | Add pagination to the list of collaborations on app list page of Portal. | | ![](/img/improvement.jpg) | Adopt same tabs everywhere in portal | | ![](/img/improvement.jpg) | Display user\_id in user's profile page of portal. | | ![](/img/improvement.jpg) | Use fully qualified urls throughout portal | | ![](/img/improvement.jpg) | Expose the delete button in explorer single input view | | ![](/img/bug.jpg) | Adding new concepts to classification apps disappear from Single Image View until refresh. Fixed | | ![](/img/improvement.jpg) | Fix CSS styling of Text Assets for Single Image View | | ![](/img/improvement.jpg) | Image terminology in eval page | | ![](/img/bug.jpg) | Empty workflow breaks explorer workflow dropdown. Fixed | | ![](/img/bug.jpg) | Fix create model range selector min/max values | | ![](/img/bug.jpg) | Model gallery in model mode fails when you click on any concept model with a concept not found message. Fixed | | ![](/img/bug.jpg) | Classification Prediction Scores still disappear for previously created apps. Fixed | | ![](/img/bug.jpg) | Disable "Train" button on pre-trained models | | ![](/img/bug.jpg) | Video times offset by 50ms | | ![](/img/bug.jpg) | Detection Tab of Image Details Sidebar does not always display in Face apps. Sometimes it shows classification equivalent. Fixed | | ![](/img/bug.jpg) | Model details page crashes while displaying concepts. Fixed | | ![](/img/bug.jpg) | Video thumbnails not displaying in search results. Fixed | | ![](/img/bug.jpg) | Listing collaborators models in collector view doesn't work. Fixed | | ![](/img/bug.jpg) | Fix API error while listing collaborators' models in collectors UI. Fixed | | ![](/img/bug.jpg) | Remove unnecessary field from model details page. Fixed | | ![](/img/bug.jpg) | Slider for Explorer prediction confidence doesn't apply to all the workflow nodes. Fixed | | ![](/img/bug.jpg) | Sending embed\_model\_version\_id on all model types but that's not valid. Fixed | | ![](/img/improvement.jpg) | Memoize sorted detection annotations and custom model predictions to prevent UI lag | | ![](/img/improvement.jpg) | Modify the way users navigate to the model details page | | ![](/img/improvement.jpg) | Fix collector mode to filter by user, then app, then models, then model versions. | | ![](/img/bug.jpg) | Image carousel does not scroll to the currently selected text input being viewed | --- ## Release 6.9 # Release 6.9 ## Changelog 6.8 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![](/img/new_feature.jpg) | ![](/img/improvement.jpg) | ![](/img/bug.jpg) | ![](/img/enterprise.jpg) | ### Scribe | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Improve email subject for canceled order | | ![](/img/new_feature.jpg) | \[P0\]Cancel feature for Order owner | | ![](/img/improvement.jpg) | Task form should select "All Inputs" by default | | ![](/img/bug.jpg) | Create Task button is incorrectly locked due to missing inputs | | ![](/img/bug.jpg) | Embed Model Version Id Missing in classification annotation requests | | ![](/img/bug.jpg) | Detection Annotations with Predictions display out of order | | ![](/img/bug.jpg) | Can only view app owner annotations when viewing an app as a collaborator | | ![](/img/bug.jpg) | Fix Detection Annotations displayed as collaborator bug | ### Spacetime | Status | Details | | :--- | :--- | | ![](/img/bug.jpg) | Clicking the x on search item in explorer grid view for search across apps does not properly clear the url | | ![](/img/new_feature.jpg) | Add Min Search Score Range Slider to Search Across Apps Tab | | ![](/img/new_feature.jpg) | Clicking a video search result does not seek to the corresponding timestamp | | ![](/img/new_feature.jpg) | Search By Region button on video thunbnails | | ![](/img/new_feature.jpg) | Clicking see all within image details sidebar of the visual search across apps should render all of the search results in explorer's asset grid view | | ![](/img/new_feature.jpg) | clicking see all from image details sidebar opens refine search search bar in explorer's asset grid view | | ![](/img/new_feature.jpg) | Add getSeekedVideoFrame onMouseOver to visual search results of | | ![](/img/new_feature.jpg) | Add video timestamps to video search results within the | | ![](/img/new_feature.jpg) | Add search across apps to Image View righthand sidebar | | ![](/img/new_feature.jpg) | Add select app in "Refine Search" righthand sidebar | | ![](/img/new_feature.jpg) | Search for annotation.status in Explorer search bar. | | ![](/img/new_feature.jpg) | Search for annotation.user\_id in Explorer search bar. | | ![](/img/improvement.jpg) | Change Annotation Search To say "filter by" | | ![](/img/bug.jpg) | Users having the same name as a collaborator causes annotation searching to break. | | ![](/img/bug.jpg) | Manually typing annotation search crashes explorer | | ![](/img/bug.jpg) | Endless error if a search fails in explorer | | ![](/img/bug.jpg) | Search grid view in explorer gets stuck with old results | ### Enlight | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Tracker evals: support for original coordinates | | ![](/img/new_feature.jpg) | Integrate MORSE metrics and AP to tracker eval pipeline | | ![](/img/improvement.jpg) | Cleanup trackers' interface | | ![](/img/improvement.jpg) | Platform-aware triton orchestrator | | ![](/img/improvement.jpg) | Connect tracker evaluations with the servicer | | ![](/img/improvement.jpg) | Implement first version of tracking eval | | ![](/img/improvement.jpg) | Add embed\_model\_version\_id to cluster and KNN model types | | ![](/img/improvement.jpg) | Show ROC AUC even if 0.0 | | ![](/img/bug.jpg) | crop model carriers forward concepts and other things but shouldn’t. | | ![](/img/bug.jpg) | Edit model doesn't work in model details page and app details page | ### Mesh | Status | Details | | :--- | :--- | | ![](/img/new_feature.jpg) | Create pubic Visual Text Recognition workflow | | ![](/img/improvement.jpg) | \[Portal\] Use only\_base parameter when choosing base workflow | | ![](/img/improvement.jpg) | Display model name in Selected model row in workflows edit page | | ![](/img/bug.jpg) | Remove workflows creation discrepancy | | ![](/img/bug.jpg) | Workflows nodes being set as Loading | ### Portal | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | Add Transform, Resize and Drag functionality to canvas rectangles | | ![](/img/improvement.jpg) | Create CollabpsableBox block to provide app-wide reusable accordion boxes with menu items | | ![](/img/new_feature.jpg) | Create a central entities factory for app-wide entities like annotations to be shared across modules like Explorer and Labeler | | ![](/img/new_feature.jpg) | Mouse leave preserves the FE generated thumbnail of a video at a specific time. | | ![](/img/new_feature.jpg) | Set up end to end testing framework & write auth tests | | ![](/img/new_feature.jpg) | Create Data mode design | | ![](/img/new_feature.jpg) | Filter annotations by user id in explorer | | ![](/img/improvement.jpg) | re-position data-mode in sidebar | | ![](/img/improvement.jpg) | Fix Development Environment Crashing on Portal | | ![](/img/improvement.jpg) | Add responsiveness capability to Portal | | ![](/img/improvement.jpg) | user can set a unique user\_id \( username \) in profile | | ![](/img/improvement.jpg) | Enable HTML links for mode-switching icons | | ![](/img/improvement.jpg) | Remove Upload from Explorer in favor of data mode | | ![](/img/improvement.jpg) | Memoize video thumbnail urls | | ![](/img/improvement.jpg) | Fetch Tag icon data for Explorer inputs on hover | | ![](/img/improvement.jpg) | Toast notifications should always be on top | | ![](/img/improvement.jpg) | Refactor Data mode | | ![](/img/improvement.jpg) | Refactor Data mode \(Upload component\) | | ![](/img/improvement.jpg) | Remove Detection options dropdown menu | | ![](/img/improvement.jpg) | Better Toast Notification System | | ![](/img/improvement.jpg) | Update `ModelVersionSelector` component to make use of `reselect` | | ![](/img/improvement.jpg) | Add create\_at date to explorer single input view. | | ![](/img/improvement.jpg) | Expose geo coordinates just like metadata. | | ![](/img/improvement.jpg) | Add input\_fields and output\_fields columns to model selection view \(from the ModelType\) | | ![](/img/improvement.jpg) | Migrate model details page to model mode | | ![](/img/bug.jpg) | Copy button for personal access tokens doesn't work | | ![](/img/bug.jpg) | /models API request being made with appId as undefined | | ![](/img/bug.jpg) | hovering over inputs keeps fetching annotations even if fetched once | | ![](/img/bug.jpg) | Fix incorrect check for 'isDetectionModel' throughout Portal | | ![](/img/bug.jpg) | Broken TypeScript configuration for Cypress and Jest | | ![](/img/bug.jpg) | TypeScript type-checks not running on build/push | | ![](/img/bug.jpg) | Data Mode crashing due to legacy string refs | | ![](/img/bug.jpg) | Predictions don't show up if you reload on Explorer input | | ![](/img/bug.jpg) | Metrics view doesn't work anymore | | ![](/img/bug.jpg) | Classification Annotations not loading | | ![](/img/bug.jpg) | Model filter app name resets when you click ctrl/alt key | ### API | Status | Details | | :--- | :--- | | ![](/img/improvement.jpg) | \[API Client Tests Javascript\] Fail fast when `stage` is invalid | | ![](/img/improvement.jpg) | Don’t treat StatusCode\_FEATUREFLAG\_BLOCKED errors as server errors | --- ## Release 7.0 # Release 7.0 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Scribe | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Return UserIDs for Reject/Approve annotation endpoint and fix bug | | ![bug](/img/bug.jpg) | Integration test timeout when waiting for post input | | ![improvement](/img/improvement.jpg) | Sortable-tables for labeler tasks-list | ## Portal | Status | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | Fixed inconsistency in interpolation & drawing state | | ![bug](/img/bug.jpg) | Fixed ghost transformer | | ![improvement](/img/improvement.jpg) | Migrate Front-End Deployment Jobs | | ![improvement](/img/improvement.jpg) | Enable all features for default user on local env | | ![improvement](/img/improvement.jpg) | Allow adding collaborator by non-primary email address | | ![bug](/img/bug.jpg) | Fixed nil pointer dereference in model validation | | ![bug](/img/bug.jpg) | Fixed nil pointer dereference in workflow validation | | ![new-feature](/img/new_feature.jpg) | Multi search item search support in Portal | | ![new-feature](/img/new_feature.jpg) | Support positive and negative metadata searches | | ![new-feature](/img/new_feature.jpg) | Add geo coordinates in bulk edit in explorer grid view | | ![improvement](/img/improvement.jpg) | display app-id in app-listing card | | ![improvement](/img/improvement.jpg) | Make the Portal header short | | ![improvement](/img/improvement.jpg) | display app re-indexing stats in data-mode \( internal-users \) | | ![improvement](/img/improvement.jpg) | Add Search By Region and Hiding region capabilities to Annotations & Proposals | | ![improvement](/img/improvement.jpg) | better CSV upload error messages | | ![improvement](/img/improvement.jpg) | Clean up Bulk labeling code | | ![bug](/img/bug.jpg) | Fixed modelId is null in Model details page | | ![bug](/img/bug.jpg) | Fixed model-versions not loading | | ![bug](/img/bug.jpg) | Fixed collectors user-selection API-calls fail with invalid-token | | ![bug](/img/bug.jpg) | Collectors pre & post-queue have incorrect labels. Fixed | | ![bug](/img/bug.jpg) | Fixed copy user-id to clipboard in profile | | ![bug](/img/bug.jpg) | Workflow Selection causes app crashes when using an empty workflow | | ![bug](/img/bug.jpg) | Portal sub-pages doesn't load on refresh | | ![bug](/img/bug.jpg) | Endless Concept Relation Calls | | ![bug](/img/bug.jpg) | Annotations Panel shows no annotations in classification app. Fixed | | ![bug](/img/bug.jpg) | User is able to create workflow without nodes \(click grey button\). Fixed | | ![bug](/img/bug.jpg) | Fixed 404 notifs when fetching concept relations in proposals | | ![bug](/img/bug.jpg) | Proposers - no relation type rendered if just model output | | ![bug](/img/bug.jpg) | New workflow model id is incorrectly populated | | ![bug](/img/bug.jpg) | Input Details page isn't confirming/showing the labelled concept | | ![bug](/img/bug.jpg) | Fixed 1 Model version is being displayed in modelversionselector | | ![bug](/img/bug.jpg) | Fixed copying apps | | ![bug](/img/bug.jpg) | Fixed demo app to correctly load fonts | ## Armada | Status | Details | | :--- | :--- | | ![bug](/img/bug.jpg) | When predicting by input ID, fall back to using the model if we fail to retrieve outputs from the DB | | ![bug](/img/bug.jpg) | Publish videos separately when reindexing | | ![improvement](/img/improvement.jpg) | Prefer clarifai rehosted URLs over original urls when predicting | | ![bug](/img/bug.jpg) | Fixed failure of cluster inferencing when no embeddings were received | | ![bug](/img/bug.jpg) | Fix workflow failures in tracker workflows when there is no detection in the first frame | | ![bug](/img/bug.jpg) | Validated training examples have bounding boxes in them during deep training. | ## Enlight | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Revise template parameters | | ![bug](/img/bug.jpg) | Fixed BERT EIDs | | ![bug](/img/bug.jpg) | Fixed centroid tracker bug | | ![new-feature](/img/new_feature.jpg) | Centroid tracker Platform Integration | | Sub-task | Landmark post processing in Python Media Processor | | ![new-feature](/img/new_feature.jpg) | Add `Track` export method to neural lite tracker handler | | ![bug](/img/bug.jpg) | Fixed DST directory\_upload script | | ![new-feature](/img/new_feature.jpg) | Add model version descriptions | | ![new-feature](/img/new_feature.jpg) | Add new multilingual text similarity embed model version and update the Text workflow to use it. | | ![new-feature](/img/new_feature.jpg) | Add support for audio indexing and transfer learning | | ![new-feature](/img/new_feature.jpg) | Thread through audio support in the platform | ## API | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Make gRPC Go client | | ![improvement](/img/improvement.jpg) | Add reading URL from CLARIFAI\_GRPC\_BASE to all clients | | ![improvement](/img/improvement.jpg) | Create endpoint for fetching relations of multiple concepts in one API call | | ![bug](/img/bug.jpg) | Fixed app-description duplication on app-copy | | ![bug](/img/bug.jpg) | Fixed tracker model prediction panic | | ![bug](/img/bug.jpg) | Reclaimed infinite loop | | ![bug](/img/bug.jpg) | Fixed incorrect asset count in pipeline | | ![bug](/img/bug.jpg) | Fixed redis stream msg id to timestamp err | | ![bug](/img/bug.jpg) | Fixed video ingestion using empty workflow | | ![bug](/img/bug.jpg) | Fixed app description during app creation is not being saved. | --- ## Release 7.1 # Release 7.1 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## API | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Add apps and keys scopes so they can be created with personal access tokens. | | ![new-feature](/img/new_feature.jpg) | Change /keys to work with PATs | | ![new-feature](/img/new_feature.jpg) | Change /apps to work with personal access tokens | | ![improvement](/img/improvement.jpg) | Split java proto files into multiple and add package name. | | ![improvement](/img/improvement.jpg) | Write script to prune the proto files into public only rpcs and messages | | ![improvement](/img/improvement.jpg) | Mark the appropriate fields in protos as cl\_private\_rpc to release grpc clients. | | ![improvement](/img/improvement.jpg) | copy app count and last\_inputs added in app duplication | | ![improvement](/img/improvement.jpg) | Rewrite input counting in the API to be more scalable and robust. | | ![bug](/img/bug.jpg) | Fix collector scopes so that predict keys don't need Collectors:Get | | ![bug](/img/bug.jpg) | Check the GetInputsKey of collectors has access to userA's information. | | ![bug](/img/bug.jpg) | Return “All” scopes when listing available scopes so that you have that option when creating new keys. | | ![bug](/img/bug.jpg) | properly return err if `AddAssets` failed to insert into DB | | ![bug](/img/bug.jpg) | POST inputs wasn’t using batch model optimizations correctly. | | ![bug](/img/bug.jpg) | created\_at field in sharing table is incorrect | | ![bug](/img/bug.jpg) | Missing Apps\_Get scope in session token auth caused creation of keys to fail temporarily. | | ![bug](/img/bug.jpg) | API services do not function once Queue goes down and comes back up has been fixed. This makes on premise deployments more resilient to power failures. | | ![bug](/img/bug.jpg) | List of missing scopes is not correct in error messages | | ![bug](/img/bug.jpg) | Fix node ID validation logic in Bug in workflows | ## Armada | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Create endpoint for taking down a spire | | ![new-feature](/img/new_feature.jpg) | Create endpoint to deploy deep training models | | ![improvement](/img/improvement.jpg) | Calculate detection pr and roc curves using score buckets | | ![improvement](/img/improvement.jpg) | Support for pytorch inference in spire | | ![improvement](/img/improvement.jpg) | able to overwrite default max conn for Citus | | ![improvement](/img/improvement.jpg) | Upgrade to go version for performance boost. | | ![improvement](/img/improvement.jpg) | Make runtime config to remote the extra round trip to storage in predict pathway. | | ![improvement](/img/improvement.jpg) | Improve JSON serialization performance in our servers by using an optimized third party library. | | ![improvement](/img/improvement.jpg) | Fix the WorkflowInput field name in proto to workflow\_input | | ![bug](/img/bug.jpg) | When reaching the final page, network request responds with 500 internal service error. Fixed | | ![bug](/img/bug.jpg) | Spire fails to launch in local-k8s-USER with error "persistentvolumeclaim not found". Fixed | | ![bug](/img/bug.jpg) | Video processing fails with 'caseids' error. Fixed | | ![bug](/img/bug.jpg) | Fix a connection issue from Golang backend service to media processing service | | ![bug](/img/bug.jpg) | Allocated resources for faster model performance | ## Enlight | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Return deep training evals through the API | | ![new-feature](/img/new_feature.jpg) | OCR Model | | ![improvement](/img/improvement.jpg) | Update templates to have more straightforward names and more friendly defaults | | ![bug](/img/bug.jpg) | Add 'Face' Default workflow to \`[https://api-dev.clarifai.com](https://api-dev.clarifai.com)' | | ![bug](/img/bug.jpg) | Make custom/transfer training evaluations for large models stable. | | ![bug](/img/bug.jpg) | training progress is saved too frequently, causing very slow training | | ![bug](/img/bug.jpg) | Return friendlier errors for incorrect parameters passed to templates | | ![bug](/img/bug.jpg) | Fix a bug in tracing setup for custom trainer and evaluator | | ![bug](/img/bug.jpg) | Accellerated training for specific cases | | ![bug](/img/bug.jpg) | Training System failed to train some layers | ## Mesh | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Implement image crop model | | ![new-feature](/img/new_feature.jpg) | Implement RandomSample model type | | ![improvement](/img/improvement.jpg) | Allow models that need outputs from previous nodes in a workflow to have access to those outputs to support chaining complex graphs of models. | ## Portal | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | make UI for personal access token | | ![improvement](/img/improvement.jpg) | Send event to Hubspot when a user signs up on portal | | ![improvement](/img/improvement.jpg) | Change HTML tag of ImagePile component from '' to '' | | ![improvement](/img/improvement.jpg) | Optimize Video Detection Frame Rate on Front end | | ![improvement](/img/improvement.jpg) | Remove classification/detection toggle in image details view | | ![improvement](/img/improvement.jpg) | Add colors to differentiate region results | | ![bug](/img/bug.jpg) | Scroll active thumb into view in image details carousel | | ![bug](/img/bug.jpg) | Unable to upload same file\(s\) through browse files. Fixed | | ![bug](/img/bug.jpg) | Added a favicon for Portal | | ![bug](/img/bug.jpg) | Can’t create a new API key or edit the information of API key. Fixed | | ![bug](/img/bug.jpg) | Delete input while having other inputs selected deselects everything. Fixed | | ![bug](/img/bug.jpg) | Cannot view workflow results in a face app. Fixed | | ![bug](/img/bug.jpg) | Setting useCustomConfig isn't checked at login. Fixed | | ![bug](/img/bug.jpg) | ffmpeg can produce no frames for very short videos. Fixed | | ![bug](/img/bug.jpg) | Video spire tests are not running correctly. Fixed | | ![bug](/img/bug.jpg) | Verify your email | | ![bug](/img/bug.jpg) | Last concept used for bounding boxes is retained between apps. Fixed | | ![bug](/img/bug.jpg) | Intercom links to old community site \(and maybe old FAQ page\). Fixed | | ![bug](/img/bug.jpg) | Add Inputs/View Explorer does not display in new app anymore | | ![bug](/img/bug.jpg) | Clicking video thumbs in detail view does not reload a video | | ![bug](/img/bug.jpg) | The Add Positives / Add Negatives buttons on a Concept details view breaks portal | | ![bug](/img/bug.jpg) | Collaboration apps have race condition where wrong user id is used | | ![bug](/img/bug.jpg) | Don't load collaborations for search demo/logged-out users | | ![bug](/img/bug.jpg) | Keyboard navigation in image details view highlights incorrect thumb | | ![bug](/img/bug.jpg) | Render Video Assets in Search Bar | | ![bug](/img/bug.jpg) | AppDetailsPanel add inputs/view in explorer no longer displays in devel | | ![bug](/img/bug.jpg) | Editing geo/json search items no longer work after adding the search bar tooltip | | ![bug](/img/bug.jpg) | TypeError: Cannot read 'get' of undefined when clicking image thumbnails in explorer search bar | | ![bug](/img/bug.jpg) | Explorer Visibility in small resolution screen | | ![bug](/img/bug.jpg) | No Prompt when uploading an image to explorer through url | | ![bug](/img/bug.jpg) | CFR rectangles on grid view do not correlate | | ![bug](/img/bug.jpg) | Prevent users from evaluating models that are not trainable. | ## Scribe | Status | Details | | :--- | :--- | | ![improvement](/img/improvement.jpg) | Mark the /annotation endpoints with cl\_private\_rpc | | ![bug](/img/bug.jpg) | No longer able to copy an app that has been shared with you via Collaborators. Fixed | | ![bug](/img/bug.jpg) | Collaborators can not see workers | ## Spacetime | Status | Details | | :--- | :--- | | ![new-feature](/img/new_feature.jpg) | Implement visual search in another app as a model type you can add to a workflow. | | ![new-feature](/img/new_feature.jpg) | Add click to search metadata attributes in image details sidebar | | ![new-feature](/img/new_feature.jpg) | Ability to keep concepts sorted by alpha. | | ![improvement](/img/improvement.jpg) | Refactor search | | ![improvement](/img/improvement.jpg) | Add metadata to collector added inputs so that you can filter by collector ID | --- ## Release 7.10 # Release 7.10 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish bert-base-multilingual-uncased-sentiment model | | ![new-feature](/img/new_feature.jpg) |Publish informal_to_formal_styletransfer model | | ![new-feature](/img/new_feature.jpg) |Publish Helsinki-NLP/opus-mt-xx-yy Translation Models (xx: source language; yy: target language) | | ![new-feature](/img/new_feature.jpg) |Publish Automatic Speech Recognition models "wav2vec2-base-xx" of 9 Languages (xx: language code) | ## Platform Object |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Annotation: Added support for SearchResult regions | | ![new-feature](/img/new_feature.jpg) |Model-Type: Added Input-Searcher| | ![new-feature](/img/new_feature.jpg) |Model-Type: Added Track-Aggregator | | ![improvement](/img/improvement.jpg) |Input: Extract informative properties during upload| ## Launch |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Community v1 (Perceive) launch | --- ## Release 7.11 # Release 7.11 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish ner-german-large and bert-base-chinese-ner | | ![new-feature](/img/new_feature.jpg) |Publish 18-class NER model "ner-english-ontonotes-fast" | | ![new-feature](/img/new_feature.jpg) |Publish ASR model of noisy telephone data: "wav2vec2-large-robust-ft-swbd-300h" | | ![new-feature](/img/new_feature.jpg) |Publish ASR models of less common languages "wav2vec2-large-xlsr-53" | --- ## Release 7.2 # Release 7.2 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## API |Status |Details | |-----------|---------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Add Rust/Go/Swift/C++ gRPC clients to be built + released| ## Portal |Status |Details | |-----------|---------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Set up NextJS for marketplace | | ![new-feature](/img/new_feature.jpg) |Delete and Deduplication UX for Explorer Sidebar | | ![new-feature](/img/new_feature.jpg) |Grouping/Ranking/Prediction sliders in Explorer sidebar | | ![improvement](/img/improvement.jpg) |Hide predictions tab for models having unsupported output_type| | ![improvement](/img/improvement.jpg) |Add tests for model predictions | | ![improvement](/img/improvement.jpg) |Make error toast persistant (or at least stay longer) | | ![improvement](/img/improvement.jpg) |Add integration tests for model-gallery - sortable-tables| | ![improvement](/img/improvement.jpg) |Design change for model-mode filters into dropdown-filter| | ![improvement](/img/improvement.jpg) |Adopt auto-complete dropdown filters everywhere in Portal| | ![bug](/img/bug.jpg) |Classification apps should always PATCH the input level annotation, not POST| | ![bug](/img/bug.jpg) |Cannot read property 'filterNot' of undefined | ## Scribe |Status |Details | |-----------|---------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |[Portal-Screen] Labeler-Tasks-Overview: Removed Pending-Inputs count from Task-List| | ![improvement](/img/improvement.jpg) |[Portal-Screen] Labeler-View: Added click-to-insert a new point on polygon edge| | ![improvement](/img/improvement.jpg) |[Portal-Screen] Labeler-View: Added Un-submitted Labels Warning| | ![bug](/img/bug.jpg) |AI Assist threshold panel is too large | | ![bug](/img/bug.jpg) |Change Worker mode submit button text from "Submit input for review" to just "Submit input"| | ![bug](/img/bug.jpg) |Keybindings don't work unless the sidebar is "focused" | | ![bug](/img/bug.jpg) |When submitting the last input in the carousel, I am taken back to admin view rather than fetching the next batch of inputs| | ![bug](/img/bug.jpg) |Zoom goes to 510% then to NaN | ## Enlight |Status |Details | |-----------|---------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |[Platform-Object] Training-Runner: Added Clarifai_EfficientDet template for Visual-Detector| --- ## Release 7.3 # Release 7.3 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Scribe |Status |Details | |-----------|---------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) | User-Login: New styling and authentication methods | | ![new-feature](/img/new_feature.jpg) | Task-Reviewer: Added Grid display option | | ![bug](/img/bug.jpg) Task-Manager: Fixed non-responsive behavior of Workers column | | ![improvement](/img/improvement.jpg) | Task-Manager: Added progress bars for Input-Assignment statuses | | ![improvement](/img/improvement.jpg) | Task-Manager: Added Input-Count column to table | | ![improvement](/img/improvement.jpg) | Task-List: Removed unnecessary "Mark Task as Done" button | | ![improvement](/img/improvement.jpg) | Task-List: Redesigned "For-Review" tab | | ![improvement](/img/improvement.jpg) | Task-List: Redesigned "Assigned to Me" tab | | ![new-feature](/img/new_feature.jpg) | Task-Labeler: Added timeline panel for displaying video tracks | | ![new-feature](/img/new_feature.jpg) | Task-Labeler: Added support for Polygon-Interpolation | | ![improvement](/img/improvement.jpg) | Task-Labeler: Added count of pending Inputs | ## Portal |Status |Details | |-----------|---------------------------------------------------------| | ![bug](/img/bug.jpg) Model-Evaluation: Fixed incorrect Precision/Recall metrics for Visual-Detectors | | ![bug](/img/bug.jpg) Explorer: Fixed redundant scrolling issue with Saved-Searches | | ![bug](/img/bug.jpg) Explorer: Fixed issue preventing storage of filter values to Saved-Searches | | ![improvement](/img/improvement.jpg) | Explorer: Added FPS Setting to "App Workflow Prediction" panel | | ![bug](/img/bug.jpg) Text Upload in Explorer Has Weird Styling | | ![improvement](/img/improvement.jpg) | Remove `embed_model_version_id` from POST & PATCH annotations | | ![bug](/img/bug.jpg) Model predictions cors issue | | ![improvement](/img/improvement.jpg) | create-workflow - sortable-tables - linear-view | | ![improvement](/img/improvement.jpg) | Fix annotations approximate count query | | ![improvement](/img/improvement.jpg) | Collectors Edit Function of Existing Task | ## Enlight |Status |Details | |-----------|---------------------------------------------------------| | ![improvement](/img/improvement.jpg) | Task: Added "None" option for Input-Source | | ![new-feature](/img/new_feature.jpg) | Model-Type: Initial release of Text-Token-Classifier | | ![new-feature](/img/new_feature.jpg) | Model-Type: Added support for up to 18 languages for OCR-Scene | | ![new-feature](/img/new_feature.jpg) | Model-Type: Added support for up to 162 languages for OCR-Document | | ![bug](/img/bug.jpg) App slow due to loading 250+ requests in Model Versions selector on Create Workflows page | | ![improvement](/img/improvement.jpg) | Propagate error status code descriptions in deep training | | ![new-feature](/img/new_feature.jpg) | Workflow-Card: Published new version of "face" with Landmark Detect + Align Transform | | ![new-feature](/img/new_feature.jpg) | Model-Card: Published "general" Text-Token-Classifier | | ![new-feature](/img/new_feature.jpg) | Weapon Model V1 | | ![new-feature](/img/new_feature.jpg) | Logo V2 POC | ## API |Status |Details | |-----------|---------------------------------------------------------| | Bug | Make a gRPC Python script new line replacement work in POSIX sed | --- ## Release 7.4 # Release 7.4 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Portal |Status |Details | |-----------|---------------------------------------------------------------------------------------| | ![improvement](/img/improvement.jpg) |App Workflows Panel on Application Details Page | | ![new-feature](/img/new_feature.jpg) |Add annotations to inputs using the annotations tab and prediction tab | | ![bug](/img/bug.jpg) |Cannot delete input level annotations from annotation tab (single user) | | ![bug](/img/bug.jpg) |Negative concept search (PostInputsSearch) isn't working properly | | ![new-feature](/img/new_feature.jpg) |Single Input View For both Detection/Classification Annotations | | ![improvement](/img/improvement.jpg) |Grid view - Fixed Label/un-label toggle button and remove tag on each input| | ![bug](/img/bug.jpg) |Application-Details: Poor performance for Applications with many Annotations| | ![bug](/img/bug.jpg) |Model-Editor: Cannot scroll through long hover-overs | | ![improvement](/img/improvement.jpg) |Model-Editor: Clarified help-text for config options | | ![improvement](/img/improvement.jpg) |Model-Evaluator: Removes FPR / TPR metrics for Visual-Detectors | | ![bug](/img/bug.jpg) |Bbox/region annotations not showing under the Annotations tab in Explorer (Input Details view)| | ![bug](/img/bug.jpg) |Explorer Mode Annotations uses concept_id instead of concept_name | | ![bug](/img/bug.jpg) |Slow UX while labelling through inputs in Explorer | ## Enlight |Status |Details | |-----------|---------------------------------------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Model-Card: Published "product-review-sentiment-multi" Text-Classifier | | ![new-feature](/img/new_feature.jpg) |Model-Card: Published "social-media-sentiment-english" Text-Classifier | | ![new-feature](/img/new_feature.jpg) |Workflow-Card: Published "face-v4" with new AngularMargin Visual-Embedder| | ![bug](/img/bug.jpg) |Model Predictions component fails at performance for videos | | ![bug](/img/bug.jpg) |Model Predictions can not label at 0th frame index | ## Scribe |Status |Details | |-----------|---------------------------------------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Added new Track-Timeline editing tools | | ![improvement](/img/improvement.jpg) |Task-Labeler: Annotations only submitted when Input is submitted | | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Extended hotkeys with Quick-Filter tool | | ![bug](/img/bug.jpg) |Task-Labeler: Fixed Input loading issue | | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Support changing Concepts for B-boxes | | ![bug](/img/bug.jpg) |Task-Manager: Fixed loading issue | | ![bug](/img/bug.jpg) |Task-Manager: Prevented opening empty Tasks | | ![bug](/img/bug.jpg) |Task-Manager: Unable to change Task from "Complete" to "In Progress" | | ![bug](/img/bug.jpg) |Task-Reviewer: Fixed issue preventing Tasks from loading | | ![bug](/img/bug.jpg) |Task-Reviewer: Fixed issue with loading Annotations | | ![bug](/img/bug.jpg) |Unable to Approve in Review Mode | --- ## Release 7.5 # Release 7.5 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Enlight |Status |Details | |-----------|--------------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Model-Card: Published "general" Visual-Segmenter| | ![new-feature](/img/new_feature.jpg) |Model-Card: Published "subject" Visual-Segmenter| | ![new-feature](/img/new_feature.jpg) |Model: Published new Room-Types Model-Version | | ![new-feature](/img/new_feature.jpg) |Publish face-sentiment model | ## Armada |Status |Details | |-----------|--------------------------------------------------------------| | ![improvement](/img/improvement.jpg) |Model-Example: Initial Release | | ![improvement](/img/improvement.jpg) |Model-Link: Initial Release | | ![new-feature](/img/new_feature.jpg) |Model-Type: Added Token-To-Entity | ## Portal |Status |Details | |-----------|--------------------------------------------------------------| | ![bug](/img/bug.jpg) |Got error and error toast msg won't close when selecting face models in proposers| | ![bug](/img/bug.jpg) |[Model-Details: Fixed displaying an evaluation of a Model-Version| | ![bug](/img/bug.jpg) |Model-Editor: Fixed incorrect URL when editing as Collaborator| | ![new-feature](/img/new_feature.jpg) |Workflow-Builder: Initial release of Graph-Editor| | ![bug](/img/bug.jpg) |Can not upload input in data model if the input file does not have extension| | ![bug](/img/bug.jpg) |Collaborator invitation email doesn't have link to the app | ## Scribe |Status |Details | |-----------|--------------------------------------------------------------| | ![improvement](/img/improvement.jpg) |Task-Labeler: Added Input-Metadata display | | ![improvement](/img/improvement.jpg) |Task-Labeler: Added Merge and Split operation for Tracks| | ![improvement](/img/improvement.jpg) |Task-Labeler: Added support for Video-Input-level Annotations| | ![bug](/img/bug.jpg) |Task-Labeler: Fixed inability to draw b-boxes near edges of frames| | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Suggest Tracks for AI-Assist detected Objects| | ![bug](/img/bug.jpg) |Task-Manager: Fixed "Assigned to Me" tab so that counts/progress are individualized| | ![improvement](/img/improvement.jpg) |Task-Reviewer: Improved Annotation loading performance| --- ## Release 7.6 # Release 7.6 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Armada |Status |Details | |-----------|-----------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |[Platform-Object] Annotation: Added support for Image-Masks| | ![new-feature](/img/new_feature.jpg) |[Platform-Object] Workflow-Version: Initial Release | ## Portal |Status |Details | |-----------|-----------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |[Portal] Edit Collaborator Scopes in Portal | | ![new-feature](/img/new_feature.jpg) |New drawing UX for Explorer | | ![bug](/img/bug.jpg) |[Portal-Screen] Explorer: Fixed inconsistent visibility of visual-search button| | ![bug](/img/bug.jpg) |[Portal] Change to invalid user id do not show error | | ![bug](/img/bug.jpg) |Cannot click the edit button on the task | | ![bug](/img/bug.jpg) |Input view not loading workflow panels properly when you scroll through the input drawer| | ![bug](/img/bug.jpg) |Unable to See Annotations Created in Labeler in Explorer Annotations Tab| ## Scribe |Status |Details | |-----------|-----------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |[Portal-Screen] Task-Labeler: Added Quick Image-Mask tool | | ![new-feature](/img/new_feature.jpg) |[Portal-Screen] Task-Reporter: Initial release of Single-Task-Report| | ![improvement](/img/improvement.jpg) |[Portal-Screen] Task-Labeler: Text-wrapping for Input-Metadata| | ![improvement](/img/improvement.jpg) |[Portal-Screen] Task-Manager: Reduce minimum Consensus Threshold to 2| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Labeler: Fixed inconsistency in applying rotation metadata to Image-Inputs| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Labeler: Fixed incorrect URL being fetched for Video-Inputs| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Manager: Added pagination for viewing of more than 128 tasks| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Manager: Fixed inconsistent pagination of Tasks| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Reviewer: Fixed broken "Approve Worker" button| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Reviewer: Fixed inability to manually approve annotations without consensus| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Reviewer: Unable to edit or reject annotations in Consensus Review| | ![bug](/img/bug.jpg) |[Portal-Screen] Task-Reviewer: Video-Input thumbnails fail to load| ## Enlight |Status |Details | |-----------|-----------------------------------------------------------| | ![bug](/img/bug.jpg) |Model eval on negatives for NLP models (should not have negatives)| | ![bug](/img/bug.jpg) |Model Evaluation Not Available in UI | ## Spacetime |Status |Details | |-----------|-----------------------------------------------------------| | ![bug](/img/bug.jpg) |Saved search - on cancelling one search from combination search, the search by concept is defaulting to rank based search instead of staying filter based search| | ![bug](/img/bug.jpg) |Searching in Explorer with a General Model concept adds that concept to your app| --- ## Release 7.7 # Release 7.7 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Models |Status |Details | |-----------|--------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Model: Published "edge-optimized-person" Visual-Detector| | ![new-feature](/img/new_feature.jpg) |Model: Published "hate-symbol" Visual-Detector | ## Platform Objects |Status |Details | |-----------|--------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Annotation: Added support for Image-Masks | | ![improvement](/img/improvement.jpg) |Visual-Search: Added support for Landmark-Align enabled Workflows| | ![new-feature](/img/new_feature.jpg) |Task: Added ability to add/remove Workers after creation| | ![bug](/img/bug.jpg) |Training-Coordinator: Jobs returning "Failed to get data" error| ## Portal |Status |Details | |-----------|--------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Added Quick-Mask tool | | ![new-feature](/img/new_feature.jpg) |Task-Reporter: Initial release of Single-Task-Report | | ![improvement](/img/improvement.jpg) |Input-Details: Add copy image and text files as well as urls| | ![improvement](/img/improvement.jpg) |Model-Evaluation: Click to preview Text-Inputs in table | | ![improvement](/img/improvement.jpg) |Task-Labeler: Allow merging/splitting of saved Track-Annotations| | ![bug](/img/bug.jpg) |Model-Evaluator: Sorting by ACCURACY SCORE (ROC AUC) does not sort| | ![bug](/img/bug.jpg) |Task-Editor: Fixed inaccessible Collaborators tab | | ![bug](/img/bug.jpg) |Task-Reviewer: Not showing all Annotations awaiting review| | ![bug](/img/bug.jpg) |Task-Reviewer: Video-Input thumbnails fail to load | --- ## Release 7.8 # Release 7.8 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Model |Status |Details | |-----------|------------------------------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Model: Published "person-detection-edge" Visual-Detector | | ![new-feature](/img/new_feature.jpg) |Model: Published "person-vehicle-detection-edge" Visual Detector| ## Platform-Object |Status |Details | |-----------|--------------------------------------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Annotation: Added support for Time-Segments | | ![bug](/img/bug.jpg) |Application: Reindexing with no inputs returns invalid request| | ![new-feature](/img/new_feature.jpg) |Model-Type: Added Track-Aggregator | | ![improvement](/img/improvement.jpg) |Model: Added Model.Notes field | | ![improvement](/img/improvement.jpg) |Task: Continuously check Consensus threshold logic | | ![bug](/img/bug.jpg) |Task: Fix incorrect status mapping in Annotation stats | | ![improvement](/img/improvement.jpg) |Task: Ignore Consensus logic on "Empty" annotations | ## Portal |Status |Details | |-----------|----------------------------------------------------------------------------------------------------------| | ![bug](/img/bug.jpg) |Input-Details: Audio doesn't stop when exiting after playing video | | ![improvement](/img/improvement.jpg) |Task-Labeler: Added histogram showing count of objects throughout Video-Inputs | | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Added Time-Segment tool | | ![improvement](/img/improvement.jpg) |Task-Labeler: Added tracklet visualization for Object-Track labels | | ![bug](/img/bug.jpg) |Task-Labeler: AI-Assist suggestions not displaying for Public Visual-Classifiers | | ![bug](/img/bug.jpg) |Task-Labeler: Fix carousel being pushed off-screen by timeline | | ![bug](/img/bug.jpg) |Task-Labeler: Loading animation continues indefinitely after GetTaskInputs returns no hits| | ![improvement](/img/improvement.jpg) |Task-Manager: Reduce minimum Consensus Threshold to 2 | | ![bug](/img/bug.jpg) |Task-Reviewer: Video-Input thumbnails fail to load | | ![improvement](/img/improvement.jpg) |Workflow-Builder: Added new navigation options and tooltips | | ![bug](/img/bug.jpg) |Workflow-Builder: Fixed handling of non-create-able model-types | | ![bug](/img/bug.jpg) |Workflow-Builder: Fixed the logic for showing model/version selector in Node details | | ![bug](/img/bug.jpg) |Improved handling of failed inputs that are not marked as failed | | ![improvement](/img/improvement.jpg) |Onboarding Dialog improvements | --- ## Release 7.9 # Release 7.9 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Platform-Object |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Annotation: Added support for Time-Segments | | ![new-feature](/img/new_feature.jpg) |Dataset-Version: Initial Release | | ![new-feature](/img/new_feature.jpg) |Dataset: Initial Release | | ![improvement](/img/improvement.jpg) |Input: Extract informative properties during upload| | ![improvement](/img/improvement.jpg) |Task: Continuously check Consensus threshold logic | | ![improvement](/img/improvement.jpg) |Task: Ignore Consensus logic on "Empty" annotations| ## Portal-Screen |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Task-Labeler: Added Time-Segment tool | | ![bug](/img/bug.jpg) |Task-Reviewer: Unable to edit or reject annotations in Consensus Review| ## New Model Version |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |(Yolo5x) of model person-detection-yolo | --- ## Release 8.0 # Release 8.0 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish Multilingual OCR "language-aware-multilingual-ocr-multiplex" Model | | ![new-feature](/img/new_feature.jpg) |Publish Multilingual OCR "paddleocr-multilingual" Model | | ![new-feature](/img/new_feature.jpg) |Image-to-text "general-english-image-caption-clip" Model | | ![new-feature](/img/new_feature.jpg) |"CLIP" text-embedder and visual-embedder Models | | ![new-feature](/img/new_feature.jpg) |Handwritten OCR "TROCR-BASE-HANDWRITTEN" | --- ## Release 8.1 # Release 8.1 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "t5-base-finetuned-question-generation-ap" model | | ![new-feature](/img/new_feature.jpg) |Publish "t5-base-en-generate-headline" model | | ![new-feature](/img/new_feature.jpg) |Publish "distilbart-cnn-12-6" model | | ![new-feature](/img/new_feature.jpg) |Publish "gpt2-chinese-poem" model | | ![new-feature](/img/new_feature.jpg) |Publish "codebert-base-ft-detect-insecure-code" model | | ![new-feature](/img/new_feature.jpg) |Publish "natural-language-inference-distilroberta-base" model | | ![new-feature](/img/new_feature.jpg) |Publish "disease-ner-ncbi" NER model | | ![new-feature](/img/new_feature.jpg) |Publish "codebert-base-finetuned-detect-insecure-code" model| --- ## Release 8.10 # Release 8.10 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) |Released the Community Usage dashboard|We publicly released the Usage dashboard that allows you to get measurable insights into your utilization of the Clarifai Community platform during the selected period.We also improved the display of the columns in the dashboard's Training Hours section.| |![new-feature](/img/new_feature.jpg)|Published the OpenVINO face detection model|You can now use the [OpenVINO face detection model](https://clarifai.com/openvino/face-detection/models/face-detection-0200) to detect faces in images.| |![new-feature](/img/new_feature.jpg)|Published the Barcode-QRcode-Reader operator|You can now use the [Barcode-QRcode-Reader](https://clarifai.com/yuchen/workflow-test/models/BARCODE-QRCODE-Reader) operator to detect and recognize barcodes and QR codes from images.| |![new-feature](/img/new_feature.jpg)|Published the moderation-abuse-japanese model|You can now use the [moderation-abuse-japanese](https://clarifai.com/yuchen/text-moderation/models/moderation-abuse-japanese) model to detect harmful texts in the Japanese language. It is a text moderation model finetuned for automatic cyberbullying detection.| |![new-feature](/img/new_feature.jpg)|Published the moderation-abuse-korean model|You can now use the [moderation-abuse-korean](https://clarifai.com/yuchen/text-moderation/models/moderation-abuse-korean) model to detect harmful texts in the Korean language. It is a text moderation model for hate speech/cyberbully detection.| |![improvement](/img/improvement.jpg)|Made changes to the top navigation menu|Changed the naming of “Home” to “My Apps.”Changed the style and copy of the “Create an App” button.Added a display of the current Community version in the dropdown options when you click the **?** button.| |![bug](/img/bug.jpg)|Fixed an issue where the login button could be displayed to logged-in users in rare cases|Previously, a logged-in user navigating the Community platform—such as navigating from Community models to model viewer and selecting any public model or workflow—could see the login button displayed. This occurred occasionally and for a short duration.| |![bug](/img/bug.jpg)|Fixed an issue where editing a workflow via the UI failed to work|You can now successfully edit an existing workflow via the simple drag-and-drop interface.| ## API |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Created **ListModelConcepts** endpoint for listing concepts in a model with pagination|Previously, getting a list of concepts used in a model required fetching the model with **additional_fields=outputs** param or using the **GetModelOutputInfo** endpoint. However, neither of those endpoints supports pagination, especially for models that have a huge number of concepts.The new **ListModelConcepts** endpoint adds pagination for easily listing concepts instead of displaying all concepts at once. | ## Old Portal |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| | ![bug](/img/bug.jpg) |Fixed an issue where TIFF images failed to load when added to the portal|TIFF images now render properly in the old portal. | --- ## Release 8.11 # Release 8.11 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published the BlazeFace face detection model for mobile GPUs|You can now use the [BlazeFace face detection model](https://clarifai.com/paddlepaddle/face/models/general-image-detector-blazeface_ssh-widerface) to detect faces in images. It is a lightweight and well-performing face detector tailored for mobile GPU inference.| |![new-feature](/img/new_feature.jpg)|Published four different models for general image detection|The models are [Detic](https://clarifai.com/explore/models?searchQuery=detic): A Detector with image classes that can use image-level labels to easily train detectors. They can detect twenty-thousand classes using image-level supervision.| |![bug](/img/bug.jpg)|Fixed an issue where markdown notes exceeded page width|Previously, when a code block with large text existed in markdown notes, it exceeded the page width. This led to the model viewer’s canvas expanding beyond the normal width and breaking without loading inputs well.| |![bug](/img/bug.jpg)|Fixed an issue where a user's long app name broke the sidebar |A user's app had a long name that extended beyond the sidebar. The name was shortened.| |![bug](/img/bug.jpg)|Fixed an issue where the synonym mapper model failed when used in a workflow|The synonym mapper model now works as expected when used in a workflow prediction.| |![bug](/img/bug.jpg)|Fixed an issue where changing an existing model's template failed to work|Previously, editing the existing template of a visual-detector model did not work. The existing template and its corresponding parameters stayed the same even after selecting another template.| ## Portal-Screen |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| |![bug](/img/bug.jpg)|Fixed an issue on the Input-Manager when applying label filters for inputs|Previously, selecting a label to see the images associated with that label resulted in an error. The search query took longer than expected| |![bug](/img/bug.jpg)|Fixed an issue on the Input-Manager when using the "Unassigned" option for filtering inputs not assigned to any datasets.|Previously, the "Unassigned" filter option on the front-end sent a filtering request in a format that the back-end did not support. It occurred if an app had multiple datasets.| ## Old Portal |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| |![improvement](/img/improvement.jpg)|Improved the error message generated when a model failed to train|Previously, when a model failed to train, the error message generated was not sufficient for the user to provide a fix.| | ![bug](/img/bug.jpg)|Fixed an issue where thumbnails failed to load after a visual search|After using the visual search feature to search for images, the resulting thumbnails now load properly. | --- ## Release 8.2 # Release 8.2 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "text-summarization-english-pegasus" model | | ![new-feature](/img/new_feature.jpg) |Publish "news_summarization_russian_mbart" model | | ![new-feature](/img/new_feature.jpg) |Publish PaddleOCR models of 28 languages | | ![new-feature](/img/new_feature.jpg) |Publish "text_summarization_spanish_mt5" summarization model | --- ## Release 8.3 # Release 8.3 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "financial-news-sentiment-distilroberta" model | | ![new-feature](/img/new_feature.jpg) |Publish "financial-sentiment-finbert" model | | ![new-feature](/img/new_feature.jpg) |Publish "financial-summarization-pegasus" model | ## Portal-Screen |Status |Details | |-----------|---------------------------------------------------| | ![bug](/img/bug.jpg) |"App Copy" does not duplicate already duplicated apps| --- ## Release 8.4 # Release 8.4 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "logos-yolov5" model | | ![new-feature](/img/new_feature.jpg) |Publish "text-embedder-blip" model | | ![new-feature](/img/new_feature.jpg) |Publish "image-embedder-blip" model | | ![new-feature](/img/new_feature.jpg) |Publish "general-image-embedding-vit" model | | ![new-feature](/img/new_feature.jpg) |Publish "general-image-recognition-vit" model | | ![new-feature](/img/new_feature.jpg) |Publish "general-english-image-caption-blip" model | | ![new-feature](/img/new_feature.jpg) |Publish "ocr-document-english-printed-trocr-large" model | | ![new-feature](/img/new_feature.jpg) |Publish "english-grammar-correction-t5" model | ## Portal-Screen |Status |Details | |-----------|---------------------------------------------------| | ![improvement](/img/improvement.jpg) |Input-Manager: Search Inputs by Dataset| | ![improvement](/img/improvement.jpg) |Input-Manager: Bulk add/remove Inputs from Dataset| --- ## Release 8.5 # Release 8.5 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Workflow Publishing | ## New Portal |Status |Details | |-----------|---------------------------------------------------| | ![improvement](/img/improvement.jpg) |TwoFactor Authentication| ## Marketplace Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "sup-simcse-roberta-large"| ## Portal-Screen |Status |Details | |-----------|---------------------------------------------------| | ![improvement](/img/improvement.jpg) |Dataset-Manager: Browse through Datasets within an Application| | ![improvement](/img/improvement.jpg) |Dataset-Viewer: View info & statistics for a Dataset| | ![improvement](/img/improvement.jpg) |Input-Manager: Apply edits to a selection of Inputs| --- ## Release 8.6 # Release 8.6 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community Workflow |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "asr-sentiment" workflow| ## Community Model |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish ASR Mandarin Chinese model| | ![improvement](/img/improvement.jpg) |New versions of all PaddleOCR models with improved accuracy| ## Portal-Screen |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg)|Input Uploader Setting Initial Release| | ![new-feature](/img/new_feature.jpg)|Dataset-Manager: Create a new Dataset| | ![improvement](/img/improvement.jpg)|Workflow Viewer: Preview Workflow predictions via improved experience| | ![improvement](/img/improvement.jpg) |Task-Manager: View Task workers and status more clearly| | ![improvement](/img/improvement.jpg) |Input-Viewer: View Concepts, Bounding Boxes, and Polygons| --- ## Release 8.7 # Release 8.7 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community Workflow |Status |Details | |-----------|---------------------------------------------------| | ![new-feature](/img/new_feature.jpg) |Publish "asr-moderation" workflow | | ![new-feature](/img/new_feature.jpg) |Language identification -> OCR workflow| ## Portal-Screen |Status |Details | |-----------|---------------------------------------------------| | ![improvement](/img/improvement.jpg) |[Input Manager] Bulk Select and Edit Label, Metadata and Datasets| | ![improvement](/img/improvement.jpg) |[Public Release]: View Existing Org Apps| | ![improvement](/img/improvement.jpg) |Input-Manager: Apply actions to all Inputs matching a search query| | ![improvement](/img/improvement.jpg) |Assignment-Viewer: Complete labeling Assignments in new Portal| | ![improvement](/img/improvement.jpg) |Model-Editor: Edit Models with an improved experience| | ![improvement](/img/improvement.jpg) |Workflow-Editor: Edit Workflows with improved experience| | ![improvement](/img/improvement.jpg) |Input-Manager: Filter Inputs by Dataset| ## Portal-Object |Status |Details | |-----------|---------------------------------------------------| | ![improvement](/img/improvement.jpg) |Dataset: Setup a labeling Task for a Dataset| ## Platform-Object |Status |Details | |-----------|---------------------------------------------------| | ![improvement](/img/improvement.jpg) |Model: Train a Visual-Segmenter model via MMSegmentation| --- ## Release 8.8 # Release 8.8 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community Model |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) |Improved the prediction output UI of visual segmentation models | Changed the default preview image samples for the [image-general-segmentation](https://clarifai.com/clarifai/main/models/image-general-segmentation) model and added model notes.Improved the working of the [image-subject-segmentation](https://clarifai.com/clarifai/main/models/image-subject-segmentation) model. | |![improvement](/img/improvement.jpg)|Hidden unselected tags in the model detail page|Disallowed tags that the model owner did not select to be displayed to all users. Previously, all tags were shown to all users, including the ones that the model owner did not select, which were not relevant to the particular model.Added an edit button to allow model owners to select the tags they need. If the edit button is clicked, a checkbox list is displayed having all the tags; that is, the selected tags and the previously invisible unselected tags.| |![new-feature](/img/new_feature.jpg)|Published the xlm-roberta-base-language-detection model |You can now use the published [xlm-roberta-base-language-detection](https://clarifai.com/erfan/text-classification/models/xlm-roberta-base-language-detection) model to detect the language of a text.| ## Portal-Screen |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| | ![improvement](/img/improvement.jpg) |Added ability to create workflows with models from the different apps you own|You can now use your models from different apps to create workflows in the Portal. Previously, you had to make the models from your different apps public before stringing them together in a workflow. | |![improvement](/img/improvement.jpg)|User disallowed from adding an untrained model version to a workflow graph|When selecting a version for a model, untrained versions now have distinct styling, which makes it clear that they cannot be selected.| |![improvement](/img/improvement.jpg)|Improved the allowed file sizes for prediction inputs|You can now upload your own video inputs of up to 100MB while image, audio, and text inputs are limited to 20MB.| |![improvement](/img/improvement.jpg)|Fixed the absence of video controls on video inputs|Video inputs now have various buttons that a user can interact with and control the video.| |![bug](/img/bug.jpg)|Made it possible to use the text field to adjust a model’s Test_Split_Percent params correctly and set the value to a number|Using the slider to adjust the Test_Split_Percent, Train_Info params when editing a model correctly sets the value to a number. However, previously, typing the value of the Test_Split_Percent in the text field adjusted the corresponding value to a string containing a number. This resulted in failed requests.| |![bug](/img/bug.jpg)|Fixed the invisibility of the “Edit Workflow” button after creating a workflow|Previously, it could take several seconds before the “Edit Workflow” button appeared after creating a workflow. Or, it could appear after refreshing the page. | |![bug](/img/bug.jpg)|Fixed the disappearance of the “Edit Workflow” button when adding or removing examples|The “Edit Workflow” button no longer disappears when adding or removing examples.| |![bug](/img/bug.jpg)|Fixed inability to edit the model ID on the “Edit Model Page” |You can now edit the model ID on the “Edit Model Page”. Previously, editing the model ID was only possible on the “Model Viewer Page”.| |![bug](/img/bug.jpg)|Fixed sample prediction image not showing upon initially opening some models|The default sample images for some models (such as [logos-yolov5](https://clarifai.com/clarifai/main/models/logos-yolov5) and [ocr-scene-english-paddleocr](https://clarifai.com/clarifai/main/models/ocr-scene-english-paddleocr)) were not displayed initially when their model pages were opened—although thumbnails were highlighted and predictions appeared.| |![bug](/img/bug.jpg)|Fixed issues with the update status button on the dataset viewer|The update status button on the dataset viewer now works as expected. | |![bug](/img/bug.jpg)|Fixed crashing of prediction canvas when incorrect data is sent|Prediction canvas no longer crashes. Previously, canvas could go blank due to non-region items sent to it. | |![bug](/img/bug.jpg)|Fixed possibility of a user creating datasets on another app without the required scopes|The “Create Dataset” button is now hidden when a user misses the required “Datasets_Add” scope. This prevents a non-app owner from creating datasets in an app they do not have the necessary permissions. | |![bug](/img/bug.jpg)|Fixed the malfunctioning reloading after searching by input ID|If you now reload a page after clicking on the search icon of an input to search by input ID, the page will reload as desired. Previously, after refreshing the page, the input ID was still in the URL, but it was not showing on the search results.| ## Platform-Object |Status |Change |Details | |-----------|---------------------------------------------------|---------------------------------------------------| |![improvement](/img/improvement.jpg)|Improved the MMSegmentation_SegFormer pre-configured model template|Training two different models on the same dataset using the MMSegmentation_SegFormer now gives consistent results. Previously, if one of the models had a fewer number of concepts, it would not behave as expected.| |![bug](/img/bug.jpg)|Fixed inability to patch non-trainable model|You can now patch a non-trainable model on the platform.| --- ## Release 8.9 # Release 8.9 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Created a tab that lists the concepts for each model|Alongside the *Overview* and *Versions* tabs, each model page now has a *Concepts* tab that lists the concepts it contains. The tab also allows you to search for concepts and sort them by ascending or descending order.| |![improvement](/img/improvement.jpg)|Added ability to embed YouTube videos in model notes |You can now easily embed YouTube videos in your model notes. This allows you to include more instructional content on how to use your models.| |![improvement](/img/improvement.jpg)|Added additional use cases for models|You can now select the use cases for the models you own from a wide variety of options.| |![bug](/img/bug.jpg)|Fixed an issue where updating a password resulted in a user being logged out|Previously, if a user tried updating to a new password, and entered the wrong old password, they got logged out. There is now a descriptive error message stating that the old password does not match the one available in the records.| |![bug](/img/bug.jpg)|Fixed a rare issue where a model failed to get enough training data|In rare cases, the system retrieved inputs with very few annotations, which resulted in a "Failed to get data" error—though the data associated with the concept selected had been labeled for classification operations and concepts had been added in the model details. | ## Portal-Screen |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| | ![improvement](/img/improvement.jpg)|Added ability to pin apps, models, and workflows|You can now pin the apps, models, and workflows you own. After clicking the pin icon within any of your listed apps, models, or workflows, the selected entity will rise to the top of the page, allowing you to access it easily.| |![bug](/img/bug.jpg)|Fixed an issue where the *Create Model* page kept showing *LOADING*|The *Create Model* page now works without any issues.| |![bug](/img/bug.jpg)|Fixed an issue where a user could not create a new workflow|A user can now create a new workflow without experiencing errors.| |![bug](/img/bug.jpg)|Aligned the carousel correctly on the model viewer page|The carousel on the model viewer page is now aligned properly alongside the other buttons.| |![bug](/img/bug.jpg)|Fixed an issue where a user lost all their already added inputs when the Inputs Loader encountered an error|Previously, when the Inputs Loader experienced a problem when uploading new inputs to an application, the user lost all the inputs they had already added. They could also be stuck on the Input Error screen, which made it impossible to carry out any other input management tasks.| |![bug](/img/bug.jpg)|Fixed an issue where the Dataset-Viewer failed to report the correct number of annotations|The Dataset-Viewer now reports the correct number of annotations.| ## Old Portal |Status |Change |Details | |-----------|---------------------------------------------------|---------------------------------------------------| |![bug](/img/bug.jpg)|Fixed an issue where [evaluating](https://docs.clarifai.com/portal-guide/evaluate/) a trained model's performance produced incorrect results|The metrics generated when evaluating a detection model's performance now work as desired.| --- ## Release 9.0 # Release 9.0 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published the HuggingFace text classification template|The HuggingFace text classification template is now available to users. You can use the pre-configured text classifier model template to train with your data and improve the accuracy of your results.| |![new-feature](/img/new_feature.jpg)|Published the silent-face-anti-spoofing model.|You can now use the silent face anti-spoofing detection model to determine if a face in an image is a spoofing attempt or not. It is designed to prevent people from tricking facial identification systems, such as those used for unlocking phones or accessing secure locations.| |![improvement](/img/improvement.jpg)|Made major redesign changes on listing pages|Improved the user experience when navigating the listing pages for models and workflows.Added a new design and colors for models and workflows icons.Added ability to list models and workflows by either grid view or list view.Added mobile-responsive support.| | ![improvement](/img/improvement.jpg)|Made improvements to the Usage Dashboard|Replaced “Stored Inputs” with “Average Stored Inputs.” We now compute it as an average, not as the total.Added the word “Total” to “Training Hours” and “Ops Count.” They are now referred to as “Total Training Hours” and “Total Ops Count” respectively.| |![improvement](/img/improvement.jpg)|Added ability to hover over collapsible sidebar icons|In the collapsed view on the left sidebar, the menu items are hidden and only the icons are left. We have now added a hover pop-up of the same text we hide to help users understand what each menu item is about.| |![improvement](/img/improvement.jpg)|Stopped showing name and surname on public profiles|Public profiles show a user's name if it's their own account, they are a collaborator of your app, or they are part of your organization.| |![bug](/img/bug.jpg)|Fixed an issue where collaborators' names showed as "undefined" in several places|Names of app collaborators are now displayed as desired without any issues.| |![bug](/img/bug.jpg)|Fixed an issue where the collapsible left sidebar sometimes did not display all the menu items unless the page was reloaded|All the items of the sidebar are now displayed as desired.| |![bug](/img/bug.jpg)|Fixed an issue where the header items did not display properly on mobile devices|The top navigation bar items are now displayed properly on small screens.| |![bug](/img/bug.jpg)|Fixed an issue where editing a model's params did not show the proper original params|Previously, if you tried editing a model to see or change its parameters, you could find that some of them were set to default values, even if you set them differently during model creation. This is not the case currently as the proper original params are reflected.| ## Workflow-Editor |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| |![improvement](/img/improvement.jpg)|Made improvements to dynamically render layouts of workflows|The workflow editor is now able to interpret an entire workflow graph definition and dynamically calculate the proper 2-dimensional coordinates for each of the nodes in the graph.| ## API |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| |![new-feature](/img/new_feature.jpg)|Introduced the Python Utils package|You can now use the collection of small Python functions and classes to make common patterns shorter and easier when interacting with our API.| |![bug](/img/bug.jpg)|Fixed an issue when the API received Base64 decoded masks|Previously, if you provided a Base64 mask via an HTTP request, it failed to be created. It now works as expected.| --- ## Release 9.1 # Release 9.1 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Introduced the Clarifai Organizations feature within the Community|You can now use the powerful feature to consolidate multiple Clarifai accounts into an organization and make the most out of your teams.It enhances collaboration, allows you to centralize the management of your Clarifai operations, and more.| |![new-feature](/img/new_feature.jpg)|Published the "Recommended" Resnet template|You can now use the template to yield optimal results with your visual classification models.| |![new-feature](/img/new_feature.jpg)|Published the [moderation-all-resnext-2](https://clarifai.com/datastrategy/MAIN-image-moderation/models/moderation-all-resnext-2) model|You can now use the image classification model to identify inappropriate content—such as nudity, drugs, suggestive, gore, etc.—in images and video.| |![new-feature](/img/new_feature.jpg)|Published the [moderation-abuse-japanese](https://clarifai.com/ptaszynski/text-moderation/models/moderation-abuse-japanese) model|You can now use the text classification/moderation model to detect cyberbullying in Japanese texts. | |![new-feature](/img/new_feature.jpg)|Published the [moderation-abuse-korean](https://clarifai.com/beomi/text-moderation/models/moderation-abuse-korean) model|You can now use the text classification/moderation model to classify Korean text into four concepts: hate speech, offensive language, gender bias, or other bias. | |![improvement](/img/improvement.jpg)|Made "My Apps" text dynamic |The top left menu item, "My Apps", now leads to either own apps (on a user's profile) or org apps (if the user has switched to an organization profile). In the latter case, the text now reads "Org Apps."| |![improvement](/img/improvement.jpg)|Made improvements to the model version table|Added spinning wheel to model version table. It appears during training and shows the progress of the model evaluation.Fixed a slight issue with model version table padding.| |![bug](/img/bug.jpg)|Fixed an issue with duplicating apps|Previously, when copying an app’s resources, the drop-down list to select a destination app contained app names, which had already been deprecated.The issue has now been fixed and a user cannot choose an app name in the drop-down list; there is only an app id to choose from. | |![bug](/img/bug.jpg)|Fixed an issue where the model list page loaded excess models other than the ones owned by the user.|The model list page now fetches the correct number of models owned by the user within that app. This greatly improves the loading speed of the models' page. | |![bug](/img/bug.jpg)|Fixed an issue where the sorting drop-down box could be hidden when applying the listing view on apps, models, or workflows|The **Sort by** functionality is now fully visible and sorts apps, models, and workflows correctly when using the list view or the grid view.| |![bug](/img/bug.jpg)|Fixed an issue where Python code snippets for some models failed to work |The Python code snippets now work as desired when used for calling the models via an API request.| |![bug](/img/bug.jpg)|Fixed an issue where when a member clicked an invite link to join an organization, they got redirected automatically.|Previously, when either an existing Clarifai user or an unregistered user clicked an invite link to join an organization, the page with options to accept the invite or register for an account before accepting the invite could appear—but it could redirect to the organization's account or sign up page within a second without a chance for the user to read the message and react accordingly.The redirect no longer happens automatically.| |![bug](/img/bug.jpg)|Fixed incorrect values displayed in the Usage dashboard for average stored inputs|Previously, there was a discrepancy between the value of the average stored inputs displayed in the Usage highlights section and that displayed in the Average Stored Inputs chart.The value of the average stored inputs is now displayed correctly in both places.| |![bug](/img/bug.jpg)|Fixed an issue with adding a secondary email in the account settings page|Previously, if you added a new email to your account, and clicked the sent verification link in your inbox, you could get an error—although the additional email was correctly verified.| |![bug](/img/bug.jpg)|Fixed an issue with the job monitor not showing when uploading text inputs|The input upload job monitor is now visible for text inputs. It correctly shows the progress status of the uploads.| |![bug](/img/bug.jpg)|Fixed an issue where adding metadata to image inputs was impossible|Adding metadata information to image inputs now works as expected.| |![bug](/img/bug.jpg)|Fixed some issues with displaying polygon points|Polygons labeled in the Input Viewer page are now displayed as desired.| |![bug](/img/bug.jpg)|Fixed an issue with configuring a model template using pre-trained weights|Previously, it was not possible to select a pre-trained weight out of the provided list of options when undertaking a deep train model configuration.You can now select any of the options out of the drop-down list. | |![bug](/img/bug.jpg)|Fixed an issue with loading some models|Previously, there was an application-type error when trying to load some models from the Community. The issue has been fixed.| ## Old Portal |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| |![new-feature](/img/new_feature.jpg)|Added a **SWITCH TO NEW PORTAL** button to allow users to switch to the new portal|You can now simply click the button on the legacy portal to be redirected to the new portal and try out its features.| ## API |Status |Change |Details | |-----------|---------------------------------------------------|----------------------------------------------------| |![improvement](/img/improvement.jpg)|Improved the error codes generated when a request fails|Users now get more specific error codes that tell them why a particular request failed. The codes reflect specific parameter-related errors, which make debugging issues easy and fast.| |![improvement](/img/improvement.jpg)|Improved the handling of retries for throttle and unconnected errors|Added better handling for the throttle error codes returned from API requests and unconnected error codes raised from the gRPC clients.| |![bug](/img/bug.jpg)|Fixed an issue where using the **PostDatasetInputs** endpoint to add existing inputs with ids > 32 characters to a dataset resulted in an error|Input ids length is no longer validated for the **PostDatasetInputs** endpoint. So, the character limit is no longer applied. | --- ## Release 9.10 # Release 9.10 **Release Date:** November 7th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Improved the Platform Tracker performance with a detect-track workflow | Introduced the state-of-the-art [BYTE-Track](https://arxiv.org/abs/2110.06864), an online multi-object tracking system built upon the principles of [Simple Online and Realtime Tracking](https://arxiv.org/abs/1602.00763) (SORT). With BYTE-Track, users can seamlessly integrate it into their detect-track workflows, unlocking advanced capabilities for efficient object tracking. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added model [inference parameters](https://github.com/Clarifai/examples/blob/main/models/model_predict.ipynb) customization| You can now configure inference parameters such as temperature, max tokens, and more, depending on the specific model you are using, for both text-to-text and text-to-image generative tasks. This empowers you to customize and fine-tune your model interactions to better suit your individual needs. | |![new-feature](/img/new_feature.jpg)| Added a robust search interface within the Python SDK for image and text inputs |The SDK now supports vector search (ranking) capabilities and offers advanced filtering options by parameters. You can flexibly refine search results using a variety of criteria, including concepts, image bytes, image URLs, text descriptions, embedded metadata tags, and geo points (longitude and latitude, with radius limits). The search interface also supports AND and OR operators for complex queries.The SDK has also been updated to include schema validation checks to ensure data integrity and search accuracy.You can get examples of how the search functionality works [here](https://github.com/Clarifai/examples/blob/main/search/cross_modal_search.ipynb).| |![bug](/img/bug.jpg)|Fixed SDK model output info parameters bugs related to clarifai-grpc version 9.9.0| This update ensures the SDK resolves the model output info error when invoking the 'postmodel' endpoint with specific parameters. | |![bug](/img/bug.jpg)|Fixed the issue of duplicate input IDs when uploading two similar datasets to the same app|We now append a random value to each input ID during dataset upload. This ensures that input IDs remain unique, allowing users to work with similar inputs across multiple datasets within the same app without encountering conflicts. | ## Integrations |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Introduced Clarifai and Databricks integration | This integration is achieved via the Clarifai Python SDK and it is available [here](https://github.com/Clarifai/clarifai-pyspark).This integration enables developers to efficiently manage unstructured data and computing tasks while leveraging Clarifai's computer vision and natural language capabilities.It facilitates seamless data ingestion and movement between Databricks and Clarifai. | ## PAT |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) |Added ability to automatically generate a Personal Access Token (PAT) when you create an account | Previously, only app-specific keys were automatically generated when you created an app. A PAT will also now be generated for you during account creation. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Published several new, ground-breaking models|Wrapped [Nougat-base](https://clarifai.com/facebook/nougat/models/nougat-base), a Meta AI-developed visual transformer model that converts document images, including complex math equations, into structured text, offering advancements in academic paper parsing.Wrapped [Mistral-7B-OpenOrca](https://clarifai.com/mistralai/completion/models/mistral-7B-OpenOrca), a high-performing large language model achieved by fine-tuning the Mistral-7B base model using the OpenOrca dataset.Wrapped [Zephyr-7B-alpha](https://clarifai.com/huggingface-research/zephyr/models/zephyr-7B-alpha), a 7 billion parameter model, fine-tuned on Mistral-7b and outperformed the Llama2-70B-Chat on MT Bench. Wrapped [OpenHermes-2-mistral-7B](https://clarifai.com/mistralai/completion/models/openHermes-2-mistral-7B), a 7 billion LLM fine-tuned on Mistral with 900,000 entries of primarily GPT-4 generated data from open datasets.Wrapped [Whisper-large-v2](https://clarifai.com/openai/whisper/models/whisper-large-v2), a versatile pre-trained ASR and speech translation model trained on multilingual data without requiring fine-tuning.Wrapped [SSD-1B](https://clarifai.com/segmind/segmind-stable-diffusion/models/ssd-1b), a diffusion-based text-to-image model—it's 50% smaller and 60% faster than SDXL 1.0.Wrapped [Jina-embeddings-v2](https://clarifai.com/jinaai/jina-embeddings/models/jina-embeddings-v2-base-en), an English text embedding model by Jina AI. It’s based on the Bert architecture with an 8192-sequence length, outperforming OpenAI's embedding model in various metrics. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Improved `min_value` range for consistency across all model types | For embedding-classifiers, we’ve standardized `min_value` to have a range of 0 to 1 with a step size of .01. For most of the other model types, we’ve standardized it to have a range of 0 to 100 with a step size of .1. | |![improvement](/img/improvement.jpg)|Made time information changes to the Centroid Tracker model|We’ve made significant improvements to the Centroid Tracker, specifically within the "time_info" section. We added "start_time" and "end_time" to provide precise information regarding when an object was detected and when detection ceased.| |![improvement](/img/improvement.jpg)|Made improvements to the Model-Viewer’s version table|We made the changes to make the table more consistent with the [evaluation leaderboard]( https://docs.clarifai.com/product-updates/changelog/release98#evaluation-leaderboard). It now provides users with a cohesive and familiar interface.We relocated evaluation actions from a separate module to the table to enhance the user experience. | |![improvement](/img/improvement.jpg)|Made significant improvements to enhance the dataset and concept selection process when training models | Model builders who haven't yet created datasets or dataset versions can now conveniently choose the 'app default dataset' in the model training editor screen. This option provides visibility into the labeled input counts, allowing users to verify their data before initiating the training process.The concept selection interface now displays the labeled input count for each concept. This feature helps users prevent training concepts without adequate labeled inputs and simplifies the process of identifying data imbalances, all without the need to navigate away from the screen. | |![bug](/img/bug.jpg)| Fixed a modal reappearance issue | If you click the “Use Model” button on an individual model’s page, the “Call by API / Use in a Workflow” modal appears. You can then click the “Copy Code” button to copy the displayed code for integration into your own use case. Previously, if you clicked “Copy Code” and clicked “Use Model” again, the modal would not reappear. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with selecting a model version on the publish screen| Previously, if you tried choosing a different model version on the publish screen, it resulted in a failure. It occurred when trying to edit the visibility of models with multiple versions. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue where the model page closed unexpectedly|Previously, if you clicked the “View JSON” button to view the prediction output of embedding models in the onboarding flow, the model pop-up could close unexpectedly. We fixed the issue.| ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with processing videos using the Universal workflow| The [Universal](https://clarifai.com/clarifai/main/workflows/Universal) workflow now processes videos as desired. | |![bug](/img/bug.jpg)| Fixed an issue where it was not possible to copy a workflow without first changing its ID | Previously, you could successfully copy a workflow only after changing the copied workflow ID. You can now copy an existing workflow, even if you keep the same workflow name during the copying process, such as from "(workflow name)-copy" to "(workflow name)-copy." | ## Listing Resources |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Added ability to view whether a resource is available publicly or privately|When listing your own resources, such as models, we've added an icon that clearly indicates whether they are private or shared within the Community.| |![improvement](/img/improvement.jpg)|Added starring option to modules|Similar to other resources, you can now mark modules as favorites by using the star icon.| |![improvement](/img/improvement.jpg)|Improved the accessibility of starred resources| Previously, you could only access starred resources by navigating to the top-right profile menu and selecting the “starred” option. You can now easily access both your own and Community resources by choosing either the “All” or “Starred” view on the main screen for listing resources, making it more intuitive to find what you need. | ## License Types |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Added several new license types| If you want to select a license type for your resource, we've expanded your options to provide a diverse range that can cater to your unique preferences. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) |Enhanced searching for organization members| You can now search for organization members using both their first name and last name, individually or in combination. | |![improvement](/img/improvement.jpg)|Adjusted a team's app view of organization apps |We removed 'App name,' added a non-sortable 'App description' with a maximum of two lines, introduced 'Date created,' and optionally included 'Last updated' if the information is available via the API.| ## Search |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Made searchability enhancements on the Community platform | You can now enjoy an upgraded experience when searching by resource ID, user ID, short description, and even [markdown notes](https://docs.clarifai.com/product-updates/changelog/release99#markdown-search). These enhancements ensure that you can find the exact information you need more efficiently and accurately. | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Implemented caching of input thumbnails throughout Input-Manager and Input-Viewer | This caching mechanism significantly enhances the overall efficiency of our system by minimizing the need to repeatedly load or generate thumbnails, resulting in faster response times and smoother interactions for our users. | |![improvement](/img/improvement.jpg)| Enhanced user experience during smart searches | Instead of blocking user actions, we now display a non-intrusive loading overlay. This overlay will be visible during search requests within the Input-Manager, ensuring that the search grid results remain accessible without disruption.Fixed an the issue where `getBaseWorkflowHasMultimodalEmbedder` erroneously returned 'false' when the component was not loaded. We modified the component’s behavior and it now returns 'undefined' when it is not loaded. When the return value is 'undefined,' we now disable the search bar to ensure users can only conduct searches of the correct type. | |![improvement](/img/improvement.jpg)| Improved the behavior of the input upload job monitor in the Input-Manager | If you upload inputs on the Input-Manager, a small sidebar window appears at the bottom-right corner of the screen, providing you with real-time status updates on the upload process. There is also a checkbox in the pop-up window, allowing you to tailor your monitoring preferences to better suit your needs. If the checkbox is checked, the upload monitor will initiate polling. It will also immediately update the input list as new inputs become available.If the checkbox is unchecked, polling will continue. However, the input list will only be updated once ALL jobs have been completed. Previously, there was an issue where unchecking the checkbox would halt polling, preventing updates. | |![improvement](/img/improvement.jpg)|Prevented manual page refresh during input uploads|We now prevent users from refreshing the page while inputs are still uploading. We display a modal that prompts the user to confirm whether they want to reload the page or not. This ensures users are aware of ongoing uploads and helps avoid unintended disruptions caused by manual page refreshes.| |![bug](/img/bug.jpg)|Fixed issues with fetching input counts| We resolved issues related to the REST API request for fetching counts, which previously lacked proper filters as validated by the backend. The problems led to error responses (400/500 errors). | |![bug](/img/bug.jpg)| Fixed an issue that caused a "No inputs found" message to appear | Previously, if you conducted an image search in the Input-Manager, selected "unlabeled" in the Labels menu, and then refreshed the page, you could get the error message. We fixed it. | |![bug](/img/bug.jpg)| Fixed a "Duplicate input ID" error that was preventing the creation of datasets | This issue occurred when attempting to bulk add 25+ inputs to a dataset by selecting all of them in the Input-Manager page using the checkbox at the top, leading to an error message despite no actual duplicates being present. Notably, the error did not occur when you created a dataset before uploading the inputs and subsequently adding the same inputs from the 'upload inputs' modal. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue when using the OR operator to filter inputs in the Input-Manager| The OR operator lets you filter inputs that do not match the given value.Previously, when the operator was selected, the z-index of the menu overlay could disrupt the user interface elements. We fixed the issue.| |![bug](/img/bug.jpg)|Fixed an issue with showing annotations on the Object Mode in the Input-Manager| Previously, after conducting an image search in the Input Mode and subsequently switching to the Object Mode, the annotation tags failed to display as expected. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue where newly created concepts could not be deselected|You can create a new concept in the “Labels” section of the Input-Manager. The newly created concept will be automatically selected. Previously, you could not deselect the newly created concept unless you selected another concept first. We fixed the issue and you can now unselect a concept that you have just created. | |![bug](/img/bug.jpg)|Fixed an issue with showing error message when uploading inputs|If a user is unable to upload an image, we now clearly communicate the cause of the CORS error.| |![bug](/img/bug.jpg)|Fixed an issue with the clear button in the “Labels” section of the Input-Manager|Previously, the clear button could not deselect selected concepts. We fixed the issue and you can now use it to reset and remove all selected concepts.| ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with the Input-Viewer gallery | Previously, if you selected an item with index 0 in the Input-Manager and then accessed the Input-Viewer, scrolling down on the left sidebar failed to load subsequent inputs in the gallery. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue that caused delayed loading of video thumbnails in the Input-Viewer| We've implemented various actions to ensure video thumbnails in the Input Viewer's left-side gallery do not experience significant delays when retrieved from the server. | |![bug](/img/bug.jpg)|Fixed an issue that caused unintended closure of model/workflow selection modal in AI Assist|If you want to choose a model or workflow to assist in input labeling within the Input-Viewer, navigate to the settings for [AI Assist](https://docs.clarifai.com/portal-guide/annotate/ai-assist/). Click the “Select Model or Workflow” search box, and then click the “View all” button. A modal window will open, enabling you to search for your desired model or workflow. Previously, that modal could close inadvertently and the page could be refreshed. We fixed the issue and the modal now remains open without any unexpected closures. You can type in the search bar without triggering unintended events in other components. | |![bug](/img/bug.jpg)|Fixed incorrect overlay display in Predict mode in Input-Viewer |The background overlay is now correctly displayed only when a user is in Annotate mode and in drawing mode without selecting a concept from the top bar. | |![bug](/img/bug.jpg)|Fixed an issue where the tip popover meant for region annotations appeared when you switched to Predict mode|The popover is now shown exclusively when you’re in Annotate mode and you’ve not created any annotations within your application.| ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue that caused duplicated AI-assisted suggestions in the right-hand sidebar of the Labeling Tasks screen|Prediction suggestions in the Labeler’s right-hand sidebar are no longer duplicated.We also fixed an issue that caused discrepancies between the API predictions and those displayed in the right-hand panel.| |![bug](/img/bug.jpg)|Fixed an issue that prevented suggestions with scores exceeding 0.995 from being displayed in the Labeling Tasks screen | High-quality prediction suggestions are now visible within the Labeler, allowing you to make informed and accurate labeling decisions. | |![bug](/img/bug.jpg)|Fixed an issue that caused a mismatch between bounding box predictions and actual labels|Previously, there was an inconsistency between the concepts listed in the right-hand sidebar and the corresponding bounding boxes visible in the Labeler's canvas area. We fixed the issue to ensure that the concepts and bounding boxes are now in perfect synchronization.| |![bug](/img/bug.jpg)|Fixed an issue with the [partitioned worker](https://docs.clarifai.com/portal-guide/annotate/create-a-task/#add-collaborators) labeling strategy|Previously, there were problems with labeling results when tasks were distributed among multiple workers. We fixed the issue, ensuring accurate and consistent labeling outcomes in such scenarios.| |![bug](/img/bug.jpg)|Fixed issues regarding the malfunction of the "APPROVE ALL" button and the uncertainty surrounding the functionality of the "APPROVE USER" button in labeler review-annotation mode| The buttons now operate as expected, and uncertainties regarding the approval status have been eliminated. This enhancement ensures a smoother and more reliable user experience during the process of reviewing annotations. | |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to submit labeled inputs for some labeling tasks | Previously, you could not submit labeled inputs for certain labeled tasks. When working on some labeling tasks, if you loaded the input, selected the relevant concepts, and attempted to submit your labeled input, you could encounter an error. We fixed the issue. | ## Onboarding Flow |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) |Reordered the 'Use Model' and 'Use Workflow' tabs in the onboarding flow|In the 'Use Model' or 'Use Workflow' pop-up, we moved 'Call by API' to the top position and made 'Python' the first choice.We applied the changes within the 'Use Model' pop-up, 'Use Workflow' pop-up, and in the onboarding version of 'Use Model.' | --- ## Release 9.11 # Release 9.11 **Release Date:** December 6th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) |![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added flexible API key selection | For third-party wrapped models, like those provided by OpenAI, Anthropic, Cohere, and others, you can now choose to utilize their API keys as an option, in addition to using the default Clarifai keys. This flexibility allows you to integrate your preferred services and APIs into your workflow, enhancing the versatility of our platform. You can learn how to add them [here](https://docs.clarifai.com/api-guide/predict/text#use-third-party-api-keys). | ## Training Time Estimator |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Introduced a Training Time Estimator for both the API and the Portal| This feature provides users with approximate training time estimates before initiating the training process. The estimate is displayed above the "train" button, rounded down to the nearest hour with 15-minute increments.It offers users transparency in expected training costs. We currently charge $4 per hour. | ## Billing |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Expanded access to the deep fine-tune feature|Previously exclusive to professional and enterprise plans, the deep fine-tune feature is now accessible for all pay-as-you-grow plans.Additionally, to provide more flexibility, all users on pay-as-you-grow plans now receive a monthly free 1-hour quota for deep fine-tuning. | |![new-feature](/img/new_feature.jpg)|Added an invoicing table to the billing section of the user’s profile | This new feature provides you with a comprehensive and organized view of your invoices, allowing you to easily track, manage, and access billing-related information. | |![improvement](/img/improvement.jpg)|Enabled all users (all billing plans) to add collaborators |Previously, users on the Community plan were restricted from adding collaborators to their apps. All users can now add collaborators to their apps.| ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Published several new, ground-breaking models|Wrapped [Cohere Embed-v3](https://clarifai.com/cohere/embed/models/cohere-embed-english-v3_0), a state-of-the-art embedding model that excels in semantic search and retrieval-augmentation generation systems, offering enhanced content quality assessment and efficiency.Wrapped [Cohere Embed-Multilingual-v3](https://clarifai.com/cohere/embed/models/cohere-embed-multilingual-v3_0), a versatile embedding model designed for multilingual applications, offering state-of-the-art performance across various languages.Wrapped [Dalle-3](https://clarifai.com/openai/dall-e/models/dall-e-3), a text-to-image generation model that allows you to easily translate ideas into exceptionally accurate images.Wrapped [OpenAI TTS-1](https://clarifai.com/openai/tts/models/openai-tts-1), a versatile text-to-speech solution with six voices, multilingual support, and applications in real-time audio generation across various use cases.Wrapped [OpenAI TTS-1-HD](https://clarifai.com/openai/tts/models/openai-tts-1-hd), which comes with improved audio quality as compared to OpenAI TTS-1.Wrapped [GPT-4 Turbo](https://clarifai.com/openai/chat-completion/models/gpt-4-turbo), an advanced language model, surpassing GPT-4 with a 128K context window, optimized performance, and knowledge incorporation up to April 2023. Wrapped [GPT-3_5-turbo](https://clarifai.com/openai/chat-completion/models/GPT-3_5-turbo), an OpenAI’s generative language model that provides insightful responses. It’s a new version supporting a default 16K context window with improved instruction following capabilities. Wrapped [GPT-4 Vision](https://clarifai.com/openai/chat-completion/models/openai-gpt-4-vision), which extends GPT-4's capabilities regarding understanding and answering questions about images—expanding its capabilities beyond just processing text.Wrapped [Claude 2.1](https://clarifai.com/anthropic/completion/models/claude-2_1), an advanced language model with a 200K token context window, a 2x decrease in hallucination rates, and improved accuracy. | |![improvement](/img/improvement.jpg)| Improved the [color-recognition](https://clarifai.com/clarifai/main/models/color-recognition) model | We enhanced the UI of the color recognition model for superior performance and accuracy. | ## Multimodal-to-Text |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Introduced multimodal-to-text model type | This model type handles both text and image inputs, and generates text outputs. For example, you can use the [openai-gpt-4-vision](https://clarifai.com/openai/chat-completion/models/openai-gpt-4-vision) model to process both text and image inputs (via the API) and image inputs (via the UI). | ## Text Generation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) **[Developer Preview]** |Added Llama2 and Mistral base models for text generation fine-tuning |We've renamed the text-to-text model type to "Text Generator" and added Llama2 7/13B and Mistral models with GPTQ-Lora, featuring enhanced support for quantized/mixed-precision training techniques.| ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Added model training to the Python SDK| You can now use the SDK to perform model training tasks. Example notebooks for model training and evaluation are available [here](https://github.com/Clarifai/examples/tree/main/models/model_train). | |![improvement](/img/improvement.jpg)|Added CRUD operations for runners| We’ve added CRUD (Create, Read, Update, Delete) operations for runners. Users can now easily manage runners, including creating, listing, and deleting operations, providing a more comprehensive and streamlined experience within the Python SDK. | ## Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Added a section on the App Overview page that shows the number of inputs | Similar to other resource counts, we added a count for the number of inputs in your app. Since the number of inputs could be huge, we round the displayed number to the nearest thousand or nearest decimal. Nonetheless, there is a tooltip that you can hover over to show the exact number of inputs within your app. | |![improvement](/img/improvement.jpg)|Optimized loading time for applications with large inputs| Previously, applications with an extensive number of inputs, such as 1.3 million images, experienced prolonged loading times. Users can now experience faster and more efficient loading of applications even when dealing with substantial amounts of data. | |![improvement](/img/improvement.jpg)|Improved the functionality of the concept selector | We’ve enhanced the concept selector such that pasting a text replaces spaces with hyphens. We’ve also restricted user inputs to alphabetic characters and allowed manual entry of dashes.The changes apply to various locations within an application for consistent and improved behavior. | |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to patch the cover image of a default workflow|You can now successfully update the cover image of an application’s default workflow without any issues. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with the workflow builder | Previously, if you tried to connect models incorrectly within the workflow view, a modal window appeared that was unresponsive to closure. We fixed the issue. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with displaying cover images | We fixed an issue where cover images were not displayed on the sidebar for Text-to-Image and Prompter model types. | | |![bug](/img/bug.jpg)|Fixed an issue with the loss of a selected license on page refresh | Previously, adding a license type on a model and subsequently refreshing the page would result in the selected license being removed, displaying as 'None'. We fixed the issue. | |![bug](/img/bug.jpg)|Removed cover image upload component from the onboarding dialog | In the onboarding dialog for creating a new model by selecting a pre-trained model, the cover image upload component was previously visible for non-owners/non-collaborators. The dialog now only displays the already uploaded image instead of providing the option to upload a new one. | |![bug](/img/bug.jpg)|Fixed a CORS issue that prevented image loading for Paddle OCR model| Previously, there was an issue with the Paddle OCR model where the image was not loading onto the canvas due to a CORS (Cross-Origin Resource Sharing) problem. Although the model correctly displayed predictions, the image itself was not visible on the canvas. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue where switching between deep fine-tuned templates caused the training settings section to disappear| Previously, if you created a deep fine-tuned model—either a visual classifier or detector—and selected a template; and if you chose a different template from the dropdown list, it caused the optional training setting section to disappear. We fixed it. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)|Fixed an issue with editing workflows|You can edit a model in a workflow by selecting the “Select a Model” search box in the sidebar that appears on the right side of the workflow builder. Also, if you click the “VIEW ALL” button, a modal appears, allowing you to select other models for your workflow.Previously, you could not close the modal that pops up. This resulted in the loss of unsaved edits when you went back to editing your workflow or refreshed the page. We fixed the issue.| ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Removed pinning of resources | With the advancement of the starring functionality, pinning is no longer necessary. We removed it. | |![improvement](/img/improvement.jpg)|Added ability to delete a cover image|You can now remove a cover image from any resource—apps, models, workflows, datasets, and modules.| ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Improved bulk labeling notifications in the Input-Manager| Users now receive a prompt toast message pop-up, confirming the successful labeling of selected inputs. This improvement ensures users receive immediate feedback, providing confidence and transparency in the bulk labeling process. | |![improvement](/img/improvement.jpg)|Enabled deletion of annotations directly from [smart search](https://docs.clarifai.com/portal-guide/psearch/visual-search#smart-image-search-by-visual-similarity) results in the Input-Manager| After conducting a ranked search (search by image) and switching to Object Mode, the delete icon is now active on individual tiles. Additionally, for users opting for bulk actions with two or more selected tiles, the delete button is now fully functional. | |![improvement](/img/improvement.jpg)|Added a pop-up toast for successful label addition or removal| Implemented a pop-up toast message to confirm the successful addition or removal of labels when labeling inputs via grid view. The duration of the message has been adjusted for optimal visibility, enhancing user feedback and streamlining the labeling experience. | |![improvement](/img/improvement.jpg)|Allowed users to edit or remove objects directly from smart search results in the user interface (UI)| Previously, users were limited to only viewing annotations from a smart object search, with the ability to edit or remove annotations disabled. Now, users have the capability to both edit and remove annotations directly from smart object search results. Users can now have a consistent and informative editing experience, even when ranking is applied during annotation searches. | |![improvement](/img/improvement.jpg)|Improved the stability of search results in the Input-Manager| Previously, users encountered flaky search results in the Input-Manager, specifically when performing multiple searches and removing search queries. For example, if they searched for terms like #apple and #apple-tree, removed all queries, and then attempted to search for #apple again, it would be missing from the search results. Users can now expect stable and accurate search results even after removing search queries. | |![bug](/img/bug.jpg)|Fixed an issue with missing annotation tags in Object Mode | Previously, after performing an image search in Input Mode and switching to Object Mode, annotation tags were occasionally not displayed. The fix ensures consistent visibility of annotation tags in Object Mode following an image search in Input Mode. | |![bug](/img/bug.jpg)|Fixed an issue with duplicate search results in concept select dropdown after page refresh| Previously, there was an issue where refreshing the page in the Input-Manager led to duplicate search results for the concept select dropdown. For example, users would search for "#apple," refresh the page, and then attempt to search for "#apple" again, but it could result in duplicate select entries. After refreshing the page, users no longer encounter duplicate search results for the concept select dropdown. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![enterprise](/img/enterprise.jpg)|Fixed an issue that caused anomalies with annotations and annotators in the Input-Viewer | Previously, a peculiar issue affected the functionality of the Input-Viewer, causing unexpected behavior related to annotations and annotators. Specifically, when you exported annotations in JSON format from the Datasets page, you could observe some irregularities. The number of annotations was inconsistent. In some instances, the count exceeded the number of inputs in situations where one annotation was expected per input.On the Input-Viewer, there were instances where annotations appeared to be duplicated multiple times when reviewing inputs. Such duplication was inconsistent with the actions of labelers. Also, when selecting "ALL" to view all annotators, that action unexpectedly removed all visible annotations.We fixed all the issues, and the Input-Viewer now works as expected. | |![bug](/img/bug.jpg)|Fixed an issue with the Zoom-In/Zoom-Out functionality in the Input-Viewer| Previously, there was an issue where the shortcuts for zooming in or out propagated to the browser, affecting the entire page instead of the canvas image. We fixed the issue, ensuring that the zoom-in/zoom-out events are now isolated to the canvas, providing a more intuitive and controlled zoom experience. | |![bug](/img/bug.jpg)|Introduced persisted model/workflow selection via localStorage in [AI Assist](https://docs.clarifai.com/portal-guide/annotate/ai-assist/#2-choose-a-model-or-workflow)| When selecting a model or workflow you want to get its predictions, the choice is now persisted in the localStorage. Now, users won't have to repeatedly select the model or workflow after refreshing the page. We also fixed an issue that caused incorrect states in the model/workflow selector component. The fix includes displaying an empty message and ensuring that the component behaves as expected, even when searching rapidly. | |![bug](/img/bug.jpg)|Added a delay for Input-Viewer action bar controls| We have introduced a deliberate delay when hovering over controls in the Input-Viewer action bar. This delay prevents unintentional display of information tooltips, reducing the likelihood of user frustration. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![enterprise](/img/enterprise.jpg)| Added a multi-org membership functionality | Users can now create, join, and engage with multiple organizations. Previously, a user’s membership was limited to only one organization at any given time. | |![improvement](/img/improvement.jpg)|Added Org initials on the icon invites|Organization’s initials are now appearing on the icon for inviting new members to join the organization. We replaced the generic blue icon with the respective organization initials for a more personalized representation—just like in the icons for user/org circles. | |![bug](/img/bug.jpg)| Fixed an issue with abruptly switching to a “user” account | Previously, if you clicked an organizational app, you could be inadvertently switched to a “user” account. Additionally, the list of organizational accounts was not consistently displayed. We fixed the issues. | ## Labeling Tasks |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added ability to fetch the labeling metrics specifically tied to a designated task on a given dataset| To access the metrics for a specific task, simply click on the ellipsis icon located at the end of the row corresponding to that task on the Tasks page. Then, select the "View Task metrics" option. This introduced functionality empowers labeling task managers with a convenient method to gauge task progress and evaluate outcomes. It enables efficient monitoring of label view counts, providing valuable insights into the effectiveness and status of labeling tasks within the broader dataset context. | |![improvement](/img/improvement.jpg)|Improved the [partitioned worker](https://docs.clarifai.com/portal-guide/annotate/create-a-task/#add-collaborators) labeling strategy | In the task creation screen, when a user selects `Worker Strategy = Partitioned`, we now hide the `Review Strategy` dropdown, set `task.review.strategy = CONSENSUS`, and set `task.review.consensus_strategy_info.approval_threshold = 1`.Users now have the flexibility to conduct task consensus reviews with an approval threshold set to 1.We have optimized the assignment logic for partitioned tasks by ensuring that each input is assigned to only one labeler at a time, enhancing the efficiency and organization of the labeling process. | |![improvement](/img/improvement.jpg)|Enhanced submit button functionality for improved user experience |In labeling mode, processing inputs too quickly could lead to problems, and there could also be issues related to poor network performance. Therefore, we’ve made the following improvements to the "Submit" button:Upon clicking the button, it is immediately disabled, accompanied by a visual change in color. The button remains disabled while the initial labels are still loading and while the labeled inputs are still being submitted. In the latter case, the button label dynamically changes to “Submitting.” The button is re-enabled promptly after the submitted labels have been processed and the page is fully prepared for the user's next action. | |![bug](/img/bug.jpg)|Fixed an issue where double-clicking the "Submit" button resulted in duplicated annotations|Previously, in the context of a visual detection task involving bounding box labeling, double-clicking the submit button caused each bounding box to be duplicated, leading to potential input skipping. We fixed it.| |![bug](/img/bug.jpg)|Fixed an issue that triggered an error during label review| When using AI Assist for labeling, a situation arose where, upon submission for review, the reviewer could successfully accept labels meeting the specified threshold. However, an error would occur after approving a certain number of inputs. We fixed it. | |![bug](/img/bug.jpg)|Improved input carousel navigation and canvas display when approving labels| Previously, when approving labels, the carousel input would progress, updating the thumbnail carousel and loading annotations for the current input. However, the main image canvas failed to display the corresponding image, showing the previous one instead. We fixed the issue and incorporated a safety logic into the "Approve" button to ensure a smoother and more accurate approval process. | |![bug](/img/bug.jpg)|Fixed an issue where reviewer changes did not seem to persist | Previously, issues arose where reviewer edits or changes didn't appear to persist reliably. We’ve introduced a loading overlay that temporarily restricts other user actions during the app's loading or submission processes, preventing unintended calls and race conditions. This enhancement ensures that reviewer changes are now consistently applied and successfully retained. | |![bug](/img/bug.jpg)|Fixed an issue with labeling out-of-bounds bounding boxes| Previously, attempting to save a bounding box positioned out of bounds resulted in an error. Specifically, drawing a bounding box beyond the defined bounds while labeling any input triggered an issue upon saving the annotation. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with not pulling new batches of inputs consistently|This problem persisted across labeling task creation scenarios, whether it's a full task for a single labeler or a partitioned task.Previously, after labeling 5-10 inputs, the labeler unexpectedly reverted to the first input instead of fetching new inputs from the stack, leading to repetitive labeling of the same set. Also, after closing the labeler screen, the “LABEL” button on the “Tasks” page could still appear but was no longer clickable. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed issues with annotation of bounding boxes | Previously, there was an issue preventing the successful annotation of bounding boxes when utilizing concepts containing capital letters or dashes. We fixed the issue. We fixed an issue where it was not possible to delete AI-assisted bounding box labels that remain unaccepted or rejected.Previously, users were unable to change the concept of a bounding box within a labeling task. This constraint applied to both manually created bounding boxes and those generated through accepted or rejected AI-assist predictions. We fixed the issue.We fixed an issue where it was not possible to adjust a bounding box touching the edge.We fixed an issue where the confidence threshold for filtering predictions in AI-assist bounding box labeling did not work as intended. | |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to edit and add concepts to an existing labeling task| Previously, if you created a labeling task, began annotating an input, and later revisited the labeling task to include additional concepts, the newly added concepts did not reflect in the list when returning to label inputs. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue with adjusting the threshold of AI-assist suggestions| Previously, there was an issue where moving the threshold beyond the score of an accepted AI-assist suggestion resulted in the hiding of the annotation. This affected both the concept list and the image canvas—though the annotations created were still successfully processed upon input submission. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue that led to an app crashing when clicking the "X" icon on a region| This error occurred specifically when following a sequence of actions: creating an app with the base workflow as Universal, adding images to a dataset, creating concepts, creating a labeling task, and attempting to make a region negative by clicking the "X" button on the first image. We fixed it. | |![bug](/img/bug.jpg)|Fixed an issue where a collaborator could not assign an app owner as a labeler when creating a task|Previously, if an app collaborator attempted to create a task, they could not add the app owner as a worker to the task. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue where adding a collaborator in a labeling task was not accurately reflected|In the process of creating a labeling task, the option to assign a collaborator was functioning correctly—confirmations, such as email notifications, were sent appropriately. However, upon editing the task subsequently, there was an incorrect display of "No" under the collaborator option. We fixed it. | |![bug](/img/bug.jpg)|Fixed an issue where deleting a collaborator completely broke the labeling tasks screen|Previously, removing a collaborator disrupted the functionality of labeler tasks. We fixed the issue. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Introduced automatic retrying on MODEL_DEPLOYING status in LLM modules| This improvement enhances the reliability of predictions in LLM modules. Now, when a MODEL_DEPLOYING status is received, a retry mechanism is automatically initiated for predictions. This ensures a more robust and consistent user experience by handling deployment status dynamically and optimizing the prediction process in LLM modules. | |![improvement](/img/improvement.jpg)|Improved caching in Geoint module using app state hash| We’ve enhanced the overall caching mechanism for the Geoint module for visual searches. | |![improvement](/img/improvement.jpg)|Revamped the [face visual search](https://clarifai.com/mansi_k/VisualSearchPOC/modules/face_search) module|We improved the module for a more refined and enhanced user experience.| |![bug](/img/bug.jpg)|Fixed a bug in the LLM Battleground module | Previously, there was a "Resource does not exist" bug in the module. We fixed it. |![bug](/img/bug.jpg)|Fixed an issue with the [data-augmentation](https://clarifai.com/clarifai/data/modules/data-augmentation) module| Previously, users were unable to create or store inputs when submitting an augmentation job through the module. We fixed it. | --- ## Release 9.2 # Release 9.2 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)|Added the single sign-on (SSO) registration option|You can now use the SSO feature to sign up on the Clarifai platform using your Google account.| |![new-feature](/img/new_feature.jpg)|Published v2 form for creating deep-trained models|You can now use the new form to create your deep-trained models.| |![improvement](/img/improvement.jpg)|Adjusted the titles for resources to enhance user-friendliness and consistency in structure.|Titles for various resources, such as pages for models, workflows, apps, and datasets, are now generated properly and consistently.| |![improvement](/img/improvement.jpg)|Improved the design of the pending verification page|Once a new user signs up for an account, they're redirected to a pending verification page that informs them to verify their email address before they can access our platform.We've improved the design of the page. | |![improvement](/img/improvement.jpg)| Added infinite scroll for displaying concepts, instead of displaying all concepts at once|The concepts tab that lists the concepts available in each model is now paginated. This allows you to easily view the concepts available in each model, especially for models with a lot of concepts. | |![improvement](/img/improvement.jpg)|Added pagination to the collaborations page|A list of collaborations is now paginated.| |![improvement](/img/improvement.jpg)|Added a message to the **Unassigned** datasets filter option tooltip in the Input-Manager|Previously, the tooltip for the **Unassigned** datasets filter option was empty. It now has a message that appears when a cursor is positioned over it.| |![improvement](/img/improvement.jpg)|Renamed some UI fields for creating deep-trained models|We've renamed and re-ordered some UI fields for creating deep-trained models to make them easier to understand. | |![improvement](/img/improvement.jpg)|Made improvements to the model viewer video interface|When you upload a video in the model viewer after clicking the **Try your own image or video** button, you will get a new and updated interface for the video controls, including an option to view the video on a full screen.| |![bug](/img/bug.jpg)|Fixed an issue where some app names appeared instead of app IDs in the **Use in a Workflow** pop-up window|The **Use in a Workflow** pop-up window appears if you want to use a model in a workflow.Previously, the drop-down list that lets you select a destination app for your workflow could populate app names, instead of app IDs. The issue has now been fixed, and there are only app IDs to choose from.| |![bug](/img/bug.jpg)|Fixed an issue where searching for any string executed too quickly|Previously, search results could appear after each character typed in the search box. The unnecessary search results could not disappear, making it difficult to go back and perform a new search after searching for anything.Currently, a search is only executed after a user completes typing the search term.| |![bug](/img/bug.jpg)|Fixed an issue where an empty section appeared on a model's viewer page during prediction|Previously, an empty box would appear on the prediction load after 0.5 seconds. The model viewer page now loads properly.| |![bug](/img/bug.jpg)|Fixed an issue where paginating a page overwrites the initial search results|Pagination no longer overwrites the initial results after performing a search.| |![bug](/img/bug.jpg)|Fixed an issue where text from one column overflowed into another on the models' listing page|Texts on the models' listing page no longer run into multiple columns.| |![bug](/img/bug.jpg)|Fixed an issue where a user did not receive any feedback after adding a new concept label in Input-Viewer|Previously, a user did not get any notification after successfully adding a new concept to an input in the explorer view—only until they tried to add it for the second time, in which case they'd receive an alert that it has a duplicate ID.| |![bug](/img/bug.jpg)|Fixed an issue where deep train configuration resets to default|Previously, when creating a new deep-trained visual detector model, the image size parameter for configuring a template kept resetting to the default value of 512.| ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)|Added ability to overwrite a model's `output_info` in a workflow graph|You can now override the outputs from given nodes in your workflow by passing the `WorkflowNode.OutputInfoOverride` parameter within the definition of each `api.WorkflowNode`. This modifies the `ModelVersion` you specified, and tells us to use the modified `ModelVersion` when you use the workflow for inference. If provided, any `output_info` in the graph would take precedence over the `output_info` of the stored `model_version` at predict time of the workflow. This way, if you don't own the model version, you can change the output settings to fit your specific use cases.This change affects the **PostWorkflow** endpoint (for adding workflows to apps), the **PatchWorkflow** endpoint (for editing workflows), and the **PostWorkflowResultsRequest** endpoint (for predicting with workflows).| --- ## Release 9.3 # Release 9.3 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | | **Apps** | | | ![improvement](/img/improvement.jpg) | Revamped the **App Overview** page | Introduced a new design with easy edit options.The models and collaborators available in the app are now listed on the right sidebar, allowing you to quickly assess the app's contents.There is an option to upload an image and use it as a reference for the app.There are more options to share the app to various popular sites. | | ![improvement](/img/improvement.jpg) | Added ability to star apps | You can now star your apps, and they'll be ranked in the starred section. This could allow you to bookmark them for future references and attach relevance to the starred ones. | | | **Resources Listing Views** | | | ![improvement](/img/improvement.jpg) | Improved the sorting criteria for filtering and listing items | When the list is sorted by "Last Updated," the order for listing items becomes "Descending" by default.When the list is sorted by "Model Name," the order for listing resources becomes "Ascending" by default. | | ![improvement](/img/improvement.jpg) | Improved the display of selected filtered items | Selected filtered items are now displayed in a row on the top of the grid.You can remove a selected filtered item by clicking the **Clear** button.You can remove all selected filtered items by clicking the **Clear all** button. | | | **Model Viewer Improvements** | | | ![improvement](/img/improvement.jpg) | Improved the **Use Model** feature in an app pop-up modal | When you want to get the code snippets for using a model programmatically, you click the **Use Model** button, and on the small window that pops up, select the **Call by API** option.The buttons on the window that pops up have been improved for better display.There is now a drop-down option on the window that pops up that lets you select a PAT (Personal Access Token), which will be automatically inserted into the code snippet. The latest PAT you created is selected and inserted into the snippets by default.There is also a button for **Create a new token**, which you can use to easily create a new PAT. The styling for the code snippets has also been improved. | | ![improvement](/img/improvement.jpg) | Added a non-zero detection_threshold to the [yolov5 logo](https://clarifai.com/clarifai/main/models/logos-yolov5) model | The model now has an updated default detection_threshold (`detection_threshold: 3.5`). | | ![improvement](/img/improvement.jpg) | Improved the output of the OCR model to include scores | You can now use the OCR model correctly and see the scores. | | ![bug](/img/bug.jpg) | Fixed an issue where some existing model configuration fields failed to load as intended when editing a model | Previously, when you tried to update an existing model and create a new version, some of the fields in the editing form, such as model id and some deep-train template configuration settings, did not load properly.The model configuration details are now loaded properly when updating a model. | | ![bug](/img/bug.jpg) | Fixed an issue where the sample image failed to show up on initially opening a model—although the thumbnail was highlighted and predictions appeared | The sample prediction images on models are now displayed as expected. | | | **Input-Manager and Input-Viewer** | | | ![bug](/img/bug.jpg) | Fixed an issue with the Input-Viewer screen | If you go to the **Input Filters** section on the Input-Manager and select a dataset, and then click an image in the dataset, you'll be redirected to the Input-Viewer page.Previously, using the left sidebar to scroll through the film strip of images in a dataset on the Input-Viewer could result in an endless loop.The issue only happened when a dataset was selected. It has now been fixed. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented a user from scrolling past five elements in the concepts drop-down list | When adding annotation tags to an input on the Input-Viewer, you can now scroll past the fifth concept in the drop-down list that pops up with the already available concepts.The option to create a new concept always appears if an exact match of the concept you want to add is not available. | | ![bug](/img/bug.jpg) | Fixed an issue that resulted in an existing **Datasets** filter to be removed after selecting the "Unlabeled" **Labels** filter on the Input-Manager| Previously, selecting any labeled options could correctly display the images with that label in the currently selected dataset; however, selecting the “Unlabeled” option on the **Labels** filter could reset the initial dataset selection. The issue has been fixed. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented canceling a polygon drawing with the Esc key on the Input-Viewer| You can now press the Escape keyboard button to cancel the creation of a polygon. | | ![bug](/img/bug.jpg) | Fixed an issue that caused the negate button to be hidden for the **Datasets** filter option on the Input-Manager | **Datasets** are now using the same styles as **Labels**—they are now unified with matching styles (padding).The negate button now works as expected.| | ![bug](/img/bug.jpg) | Fixed an issue that caused incorrect sizing of the bulk edit toolbar on the Input-Manager | The pop-up now takes the full width.The option for editing items is now correctly spelled as “Edit labels” and not the previous “Edit lables.” | | ![bug](/img/bug.jpg) | Fixed an issue that caused the concept selector bar not to remain active/focused after adding a new concept | Previously, the **Add annotation tags** concept selector bar on the Input-Viewer could not remain active after adding a new concept. The issue has been fixed, and you can keep hitting enter to add new concepts without the bar becoming inactive.| | ![bug](/img/bug.jpg) | Fixed an issue where the sidebar disappeared when hovering over elements | The sidebar is now displayed as desired on the Input-Viewer. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented drawing a bounding box inside an existing one on the Input-Viewer | You can now easily draw a bounding box inside another one. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented new concepts from being created when clicking the plus button. | The plus button is found next to any field for creating new concepts. It now works as expected when creating new concepts.| | ![bug](/img/bug.jpg) | Fixed an issue that prevented custom models from appearing on the prediction drop-down on the Input-Viewer | Custom models are now appearing among the options for predictions on the Input-Viewer. | | ![bug](/img/bug.jpg) | Fixed an issue that caused HTTP 400 / 500 errors when uploading large inputs on the Input-Manager | Uploading large inputs now works without any errors.| | ![bug](/img/bug.jpg) | Fixed an issue that prevented notifications from showing when adding labels on the Input-Viewer | There is now a notification that tells you that an annotation was successfully added. If you want to add the same annotation for the second time, the field now gets cleared. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented images beyond the 50th result from being loaded on the Input-Viewer | If you click on an image on the Input-Manager, you'll be redirected to the Input-Viewer page where you can see the details of the image.Previously, clicking the 51st (or beyond) image in the dataset did not load the main image properly on the Input-Viewer, so you couldn't see its details. Also, the thumbnail scrollbar that gave the position of the clicked image gave a wrong number. It also complicated using the non-loaded image for making predictions.Images #51 (and beyond) are now loaded and displayed correctly on the input details view. | | ![bug](/img/bug.jpg) | Fixed an issue that caused inconsistent results when navigating between Input-Manager and Input-Viewer | If you apply some filters on the Input-Manager view and click on an image, you'll be redirected to the Input-Viewer screen, which has a film strip of images.Previously, that behavior resulted in some of the images not being able to be loaded properly.| | | **Task-Editor** | | | ![bug](/img/bug.jpg) | Fixed an issue that caused semi-transparent pop-ups after creating a task on the Task-Editor | You can add inputs to a dataset by going to the **Datasets** section on the Input-Manager, selecting a dataset, and adding inputs to it.Previously, if you created and labeled a task, approved a user for the task, and selected a dataset to upload your inputs, you could encounter a small semi-transparent pop-up window for uploading inputs at the bottom of the page. | | ![bug](/img/bug.jpg) | Fixed issues with adding new collaborators on the Task-Editor | Fixed an issue causing poor visibility of the add new collaborator pop-up on the Task-Editor. It is now displayed correctly and with full visibility.Fixed an issue that caused requests to fail when adding new collaborators on the Task-Editor. | | ![bug](/img/bug.jpg) | Fixed issues that affected the usability of the **select all concepts** option on the Task-Editor | You can now conveniently select all concepts on the Task-Editor.Fixed an issue causing an empty pop-up below the **Select concepts** search box to remain open once all the concepts have been selected. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented the concept pop-up from closing after adding a new concept on the Task-Editor | Previously, after adding a new concept, the **Add new concept** drop-down that pops up could not be closed after clicking it. It now works properly. | | ![bug](/img/bug.jpg) | Fixed an issue that caused the label button to remain enabled for users that were not assigned any tasks on the Task-Manager | The label button is now enabled appropriately. | | ![bug](/img/bug.jpg) | Fixed an issue that caused user names to appear as duplicates in the drop-down that pops up when assigning labeling tasks to workers on the Task-Editor.| User names assigned to tasks in the **Labeler** section now appear appropriately. | | ![bug](/img/bug.jpg) | Fixed an issue where a reviewer erroneously got a message that they've already reviewed all annotations in their assigned labeling tasks | Previously, when a user wanted to review their assigned tasks, they could get an erroneous message that no annotations were available for review and that they've already reviewed all annotations. It now works properly. | | ![bug](/img/bug.jpg) | Fixed an issue that allowed users to review tasks that do not have any inputs for reviewing | Users can no longer review tasks on the Task-Manager that do not have any inputs.Also, previously, if you clicked the **REVIEW** button on the Task-Manager, you could be redirected to the tasks review page where a **REJECT USER** button could appear and then suddenly disappear after about two seconds. That issue has been fixed. | | ![bug](/img/bug.jpg) | Fixed an issue that caused the **LABEL** and **REVIEW** buttons to change state incorrectly on the Task-Manager | The enable and disable behavior of the buttons has been fixed. | ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Made critical, breaking changes to the use of PATs and API keys | The changes are explained [here.](https://docs.clarifai.com/product-updates/upcoming-api-changes#upcoming-changes) | |![bug](/img/bug.jpg) | Fixed an issue where the API threw a `status.code` error code but referred to a different code in `status.details` within the same request | The API now returns the same status codes in `status.code` and `status.details` fields. | --- ## Release 9.4 # Release 9.4 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Community ### Model-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added the **Use Workflow** feature in an app pop-up modal | You can now navigate to a workflow viewer page and click the **Use Workflow** button at the upper-right corner of the page. A small window will pop up that lets you use the workflow either in an app or use it programmatically via an API. It works just like the **Use Model** feature for models. | | ![new-feature](/img/new_feature.jpg)| Added a model versions drop-down and **See versions table** button | Model version selection is now more prominent than the previous semi-hidden view in the predict pane.If you now select a model version, and make predictions with it, the predict pane will render the predictions for the specific version.If you click the **See versions table** button, a table that lists the different versions of your model will appear. | | ![improvement](/img/improvement.jpg) | Added the possibility to view the concepts of public models without having to log in | You can now view a public model’s concepts without logging in. | | ![improvement](/img/improvement.jpg) | Improved the **Call by API** feature in an app pop-up modal | When you want to get the code snippets for using a model programmatically, you click the **Use Model** button, and on the small window that pops up, select the **Call by API** option.Previously, if you used the **Create a new token** button to create a new PAT token with partial scopes, it could not show up. Creating a new PAT under the **Call by API** tab now works as expected.We've also improved the styling for the placeholders in the code snippets. | | ![bug](/img/bug.jpg) | Fixed an issue that made it impossible to edit a model's visibility| You can now edit a model's visibility from private to public.| ### Model Evaluation |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Added a concept selector for handling a confusion matrix with a large concept list| Previously, confusion matrix items appeared clogged when evaluating models with many concepts in their datasets, which made it impossible for viewing and interpretation. The issue has been fixed. | | ![bug](/img/bug.jpg) | Fixed a "No Page Found" error when the evaluation button was clicked after a model was evaluated | You can now see the results after clicking the evaluation button. You won't encounter any errors. | ### Model Prediction |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue where the prediction sidebar showed an "oops ..." error for non-logged in users | For a non-logged in user, a "Log In To Predict" button now appears when they open a model or a workflow. After clicking the button, they get directed to a log in page.| | ![bug](/img/bug.jpg) | Fixed an issue where the [image-general-segmentation](https://clarifai.com/clarifai/main/models/image-general-segmentation) model showed only masks without any predicted concepts | The model now outputs both masks and concepts. | | ![bug](/img/bug.jpg) | Fixed an issue where incorrect tags showed for classifications in the [Demographics](https://clarifai.com/clarifai/main/workflows/Demographics) workflow | The correct tags are now shown for canvas vs. prediction panel for the workflow. | | ![bug](/img/bug.jpg) | Fixed an issue where the first prediction for text models did not render results | Text models are now working as expected. | ### Model-Editor |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue where it was not possible to edit a model's template name | You can now change the template while updating a model. | | ![bug](/img/bug.jpg) | Fixed some misaligned fields in the model version edit form versus the original model creation form | The text fields are now properly aligned in the model version edit form as in the original model creation form. | ### Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Improved the listing of the organization settings items on the navigation bar | If you click your user profile icon at the upper-right section of the navigation bar, and switch to an organization, you'll notice that we've made the following improvements to the drop-down list that follows:"Personal Settings" has been replaced by "Organization Settings."The "Billing" item has been removed."My Apps" has been replaced by "Org Apps.""Usage" link has been added."Members" and "Teams" have been added for admin users. | | ![bug](/img/bug.jpg) | Fixed an issue where team contributors could not see the models and workflows of apps that they have access to. | Previously, a team contributor could not access models and workflows in the "Org Apps" listing, even if some apps were assigned to their team.Team contributors now have sufficient scopes to see the models and workflows of apps that they have access to. | ### Resources Listing Views |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue that prevented the **My Apps** page from showing a newly created app | Previously, if you created a new app, the **My Apps** page did not show it until you refreshed the page.You can now create a new app, and it will be automatically listed on the **My Apps** page without the need to refresh the page first. | | ![bug](/img/bug.jpg) | Fixed an issue where it was not possible to copy the ID of an app on the Overview page | If you want to copy the ID of an app, you go to the App Overview page, and click the three little dots at the upper-right corner of the page. Then, click the **Copy ID** option on the list that drops down.You can now copy the ID of an app on the Overview page without any issues. | | ![bug](/img/bug.jpg) | Fixed an issue where the workflow page did not load all workflows | The workflow page now loads all workflows. | ### App Overview Page |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Added the possibility of a collaborator to update or upload an app image | Previously, only the owner could upload or update an app image. Instead of limiting the privilege to the app owner only, we've made it possible for collaborators to make the change. It now depends on the user’s scope. | | ![bug](/img/bug.jpg) | Fixed an issue where a black background persisted when adding an app's notes | Previously, if you enabled the dark theme on some systems, such as on macOS, the form on the app overview page persistently had a black background—both when editing notes and after publishing notes. | ### Sidebar Revamp |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Renamed some elements on the collapsible left sidebar | "App Resources" has been renamed to "AI Lake.""Scribe" has been renamed to "Tasks." | ### Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue that caused the concept drop-down to appear behind the **Label as...** modal | The **Label as…** button appears at the bottom of the page when a user selects an input. And if a user clicks the button, the label modal appears. Previously, if you typed a concept in the **Concepts** field within the label modal, the concept results search or add new concept drop-down did not appear. It now works as expected. | | ![bug](/img/bug.jpg) | Fixed some issues that prevented using the **X** icon to clear the Smart Search bar | Previously, the **X** icon sometimes failed to appear when a user selected one or more smart concepts. The **X** icon now appears when a user selects more than one concept.The **X** icon also appears when a user writes an incomplete tag or concept. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented searching by clicking a concept in the auto-complete list | Previously, if you used the Smart Search bar, clicking a concept in the auto-complete list could not trigger an automatic search. You can now click a suggestion from the list of the auto-completed concepts, and it will fill in the concept name and trigger a search automatically. | | ![bug](/img/bug.jpg) | Fixed an issue that caused confusing auto-complete concept suggestions | The Smart Search functionality now works correctly when special characters are used in the search query.The Smart Search functionality now works correctly when searching for a term with uppercase or lowercase letters. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented scrolling through the auto-completed list of concepts | When performing a Smart Search, only five results now show in the suggestion list, while the other suggestions are scrollable. | ### Task-Labeler |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg)| Fixed an issue that prevented applying AI Assist suggestions within a labeling task | The AI Assist tool now works as expected in giving concept suggestions, which a user can apply when labeling inputs. | ### Task-Manager and Task-Editor |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue that caused incorrect results in the concept list on the Task-Editor |If you want to edit a task, and you click on the **Concepts** field, a drop-down list of the created concepts appears. The correct results are now showing in the concepts list on the Task-Editor page.| | ![bug](/img/bug.jpg) | Fixed an issue that caused the label and review buttons to change state incorrectly on the Task-Manager | The enable and disable behavior of the review and label buttons now work as expected. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented displaying some error texts explaining why the **Submit** button was disabled on the Task-Editor | If you try to create a task, it now shows some error texts regarding the fields required to be filed before the **Submit** button is enabled. | ### Input-Uploader |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue that caused input import jobs containing large images to fail | Uploading large images now works without issues.| | ![bug](/img/bug.jpg) | Fixed an issue that caused the import job "Uploading..." status to appear to stop at 50% | Input uploading status now goes to 100%During uploading, input processing now shows the progress in %. | |![bug](/img/bug.jpg) | Fixed an issue that caused the import monitor to suddenly disappear after all jobs became inactive | After jobs are finished, the window is now kept open so that you can see that the import has been successful.| ### Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue that caused failed predictions on the Input-Viewer | Previously, some public models were broken because of incorrectly sending `concepts` and `metadata` data alongside the image. The issue has been fixed. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented viewing videos in full-screen on the Input-Viewer | You can now upload a video and view it in full-screen on the Input-Viewer. | | ![bug](/img/bug.jpg) | Fixed an issue that caused the dragging of the entire canvas while editing polygons on the Input-Viewer| Dragging action is now disabled when editing polygons in cursor mode. | | ![bug](/img/bug.jpg) | Fixed an issue that caused video inputs to appear as images on the Input-Viewer | Video inputs are now processed and handled as expected. | | ![bug](/img/bug.jpg) | Fixed an issue that prevented viewing inputs when secure data hosting was enabled | Images are now loading properly on the Input-Viewer when secure data hosting is enabled in an account. | ### Usage Dashboard |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Updated the start date for the 6-month billing cycle | You can find the date range selection at the top-right corner of the Usage Dashboard page. You can define the output of the billing cycle by selecting your preferred date range. We've updated the start date for the **Last 6 months** selection for better accuracy. | --- ## Release 9.5 # Release 9.5 Date: June 5th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Old Portal |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Deprecation** | Our old portal is officially entering early deprecation and will no longer be actively maintained | The legacy portal will be decommissioned and it will no longer be accessible after July 3rd, 2023.We encourage our users to switch to the new portal for a better experience. | ## Community ### Large Language Models (LLMs) |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added more LLM model options to allow our users to unleash the power of the latest developments in Generative AI | We've wrapped several LLM models from various vendors into our platform. You can use them to perform a wide range of tasks, including text summarization, text generation, text embedding, text language detection, text-to-image conversion, text-to-audio conversion, transcription, translation, and image upscaling. We've wrapped the following models from Cohere: `cohere-summarize`, `cohere-generate`, `cohere-embed`, `cohere-multilingual-embed`, `cohere-detect-language` (text-to-text, text-to-embeddings).We've wrapped the following models from AI21 Labs: `Jurassic2-Large`, `Jurassic2-Grande`, `Jurassic2-Grande-Instruct`,`Jurassic2-Jumbo-Instruct`, `Summarize-API` (text-to-text).We've wrapped the following models from OpenAI: `GPT-3.5`, `GPT-4` (text-to-text), `openai-embed` (text- to-embed). | | ![new-feature](/img/new_feature.jpg) | Unleashed increased capabilities with LLMs | Added ability to perform transfer learning on top of LLMs.Added ability to perform auto-labeling with GPT3.5/4 and large LLM wrappers. | ### Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Published a new model type called Python Regex-Based Classifier | The new model type allows you to classify text using regex. If the regex matches, the text is classified as the provided concepts. If you're a Python coder who needs to do pattern-matching text classification (such as text moderation, LLM-based zero-shot, or auto-labeling), you can provide concepts and tag text inputs with the Regex-Based Classifier. Optionally, you can chain it together with many other models and operators in a workflow. | |![improvement](/img/improvement.jpg)| Updated the default params values for the MMDetection__YoloF deep training template | The deep training template now has updated default parameter values. | |![improvement](/img/improvement.jpg)| Replaced the “Train Model” button with a “Create New Model Version” button on the model viewer screen| The new button now more explicitly points to what you can achieve if you click it. | |![improvement](/img/improvement.jpg)| Added a missing gear icon at the upper-right section of the model viewer page | Previously, there was a missing gear icon on the model viewer page. The icon is used to cache public preview predictions.The gear icon is now available, and you can use it to add public preview examples for non-logged-in users to see. | |![improvement](/img/improvement.jpg)| Added preset inputs to appear on the left side of the Model-Viewer screen | If you open a model, inputs (thumbnails) now appear on the left side. | |![improvement](/img/improvement.jpg)| Improved the design of the model version table and the training log monitor | If you create a model and hit the "train" button, you'll be redirected to the model version screen. You can click the pencil button to edit a model version description and save it on the fly.You can get information about the status of the evaluation—the spinning wheel lets you check the status. You can also view the status message, view the training log monitor (for deep trained models only), retrain with a new version, and cancel the training.We've also added various action buttons for copying, deleting, and viewing a trained model version. | |![improvement](/img/improvement.jpg) | Improved the design of the model versions dropdown list and "See versions table" button on the Model-Viewer screen| Model version selection is now more prominent than the previous semi-hidden view on the predict pane. If you select a model version, the predictions of that version will be rendered on the predict pane. You can also click the "See versions table" button to see the history of a model version. | |![bug](/img/bug.jpg) | Fixed an issue where confusion matrix items appeared clogged in the evaluation results for datasets with many concepts, which complicated their viewing and interpretation | If you go to the evaluation results page to evaluate the performance of a model version, the items in the confusion matrix do not now appear clogged when you select many concepts. | |![bug](/img/bug.jpg)| Fixed an issue where model training log details showed as loading even when training logs were not available | Previously, when the status for training a model showed as trained, the monitor kept showing that the training logs were being loaded. This happened because the embed-classifier model type does not have training logs. Currently, “View Logs” is only shown when logs are available. | |![bug](/img/bug.jpg)| Fixed an issue where the prediction pane in the model viewer page of successfully user-trained models disappeared | The prediction pane of user-trained models now works as expected. The "Add Preview Images" and "Try Your Input" buttons are now working as expected. | |![bug](/img/bug.jpg)| Fixed an issue where the initial prediction results of Clarifai's text models could not be rendered | Clarifai's text models now render first prediction results appropriately. | |![bug](/img/bug.jpg)| Fixed an issue where the segmentation output mask size did not match the input image | If you open a visual segmentation model, the segmentation output mask size now matches the input image. | |![bug](/img/bug.jpg)| Fixed an issue where the "Try your own Input" pop-up modal disappeared immediately | If you navigate to any visual-classifier or visual-detector model, either in your own app or Community, click the blue "+" icon on the left-hand side of the screen, a modal will appear asking you to upload an image to try the model. Previously, the modal could disappear immediately. After fixing the issue, the modal now stays open and waits for the user to choose an image. | ### Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added ability to customize non-user owned model output config in workflows | You can now customize the output config settings of a model belonging to another user, and use that model in a workflow, such as in auto-annotation tasks. | |![bug](/img/bug.jpg)| Fixed an issue where the workflow editor gave an "Invalid Version” error when a community model was selected | Previously, if you selected the "Visual Classifier" model type when editing a workflow, and then chose any image classification model from the Community, an "Invalid Version" error could be displayed. We've fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue where the workflow editor failed to respect the range definition of an operator argument | The range definition of an operator argument now works well when editing a workflow. | |![bug](/img/bug.jpg)| Fixed an issue where the select concepts pop-up modal on the workflow editor failed to disappear | Previously, if you wanted to edit a concept-thresholder model type and clicked the "SELECT CONCEPTS" button, the ensuing pop-up modal could not disappear from the workflow editor screen.The select concepts modal now gets closed if you navigate to a previous page. | |![bug](/img/bug.jpg)| Fixed an issue where editing and updating a workflow wiped out the preview examples | The uploaded preview input examples are now saved and remain public even after editing a workflow. | ### Header and Sidebar |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Improved the design of the navigation bar | We've improved the design of the navigation bar. For example, we've fixed an issue where the styling for all the buttons in the navigation bar appeared to be wrong after refreshing or hard reloading the page. | |![improvement](/img/improvement.jpg) | Made the thumbnail of an app on the collapsible left sidebar to use the app's uploaded image | If a user has uploaded an app's image on the app overview page, it will now also appear as the app's thumbnail on the collapsible left sidebar. | ### Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Published a new type of Base Workflow for apps called "Roberta-embedder" | When creating a new application, you can now choose the new type of Base Workflow for your app. The workflow lets you perform transfer learning on text inputs. | |![improvement](/img/improvement.jpg)| Restricted the visibility of the settings page of public apps| Modified the visibility of the app settings page for non-logged-in users, regular logged-in users, app collaborators, app owners, organization admins, organization members, and team members. The app settings page is now not visible to users without the necessary permissions. | |![improvement](/img/improvement.jpg)| Removed a duplicate language understanding workflow that appeared when a user created a new app | When a user created a new application, a duplicate language understanding workflow appeared in the dropdown list for selecting the app's Base Workflow. It has now been removed. | |![bug](/img/bug.jpg) |Fixed an issue where making a model public caused the app associated with it to crash | Publicizing a model now works as expected. | ### Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Exposed the app settings section to members of an organization | We've removed the API section on the page.All roles—admins, organization contributors, and team contributors—now have access to every item on the collapsible left sidebar.All roles now have access to the “Create a Model” and “Create a Workflow” buttons. | |![improvement](/img/improvement.jpg)| Introduced the use of the logged-in user's PAT (Personal Access Token) when showing the Call by API code samples | Previously, using an organization's PAT in the Call by API code samples gave an error from the backend. Therefore, we now always fetch the logged-in user’s PAT to be used in the Call by API samples. | |![bug](/img/bug.jpg) | Fixed an issue that prevented organization members from assigning themselves to Tasks on the Task-Editor| Previously, an organization member could not assign themself to an organization app as a labeler—without using collaborators.Labeling tasks are now created successfully. | |![bug](/img/bug.jpg) | Fixed an issue that prevented organization members from accessing assigned tasks on the Task-Manager | An organization member can now assign a task to themself and label the task from the listed labeling actions. | ### Datasets |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue that prevented creating a new version of a dataset if the dataset had no inputs | Previously, if you created a new dataset with no inputs, and you tried to create a version for it, that action broke the empty dataset and produced errors. We've fixed the issue. | |![bug](/img/bug.jpg)| Fixed a broken link on the datasets manager page that did not point correctly to the dataset documentation | Previously, the "Learn more" link on the datasets manager page that pointed to the dataset documentation was broken. The link now works correctly. | ### Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Improved the working of the Smart Search feature | If you now perform a Smart Search and then clear out the search input field, and hit the enter button, the search results will be reset.We've replaced the placeholder text in the Smart Search bar with the following help text: `Begin typing to Smart Search (press # to add a Concept) text`. | |![improvement](/img/improvement.jpg) | Added hover-over tooltips for clickable elements on the Input-Manager | Added tooltips to the **Datasets** section and the negate (or invert) labels.Added popovers with descriptions and "Learn More" links to the **Select or add concepts** filter field and the **Metadata** section. | |![improvement](/img/improvement.jpg)| Added support for importing a batch of video inputs on the Input-Manager | You can now easily add a batch of video inputs. | |![improvement](/img/improvement.jpg)| Implemented a toggle functionality when uploading inputs | We've added the ability to toggle and automatically refresh the input grid while actively importing inputs.We've also updated the icon and the styles of the input upload monitor window. | |![bug](/img/bug.jpg)| Fixed some issues with the Smart Search feature| If you type a text search in the input field, spaces in the texts are now not automatically converted into dashes.You can now manually insert dashes into tags to freely search by concepts, without automatic conversion. | |![bug](/img/bug.jpg)| Fixed an issue that caused an incorrect status count for input import jobs | The input upload monitor now displays a correct status count when uploading inputs. | |![bug](/img/bug.jpg)| Fixed an issue that caused input import jobs containing large images to fail | Inputs of large images are now uploaded successfully. | |![bug](/img/bug.jpg)| Fixed an issue that caused the import job "Uploading..." status to appear to stop at 50% | The input upload processing now shows in %, and it goes up to 100%. | |![bug](/img/bug.jpg)| Fixed an issue that caused the import job monitor to suddenly disappear after all jobs are inactive | After the input import jobs are finished, the window is now still kept open so that a user can see that the upload process has succeeded. | |![bug](/img/bug.jpg)| Fixed an issue that caused import jobs containing both images and videos to fail | You can now upload bulk inputs of images and videos without noticing any errors. | |![bug](/img/bug.jpg)| Fixed an issue where trained concepts appeared in the completion list twice | Previously, requests happened in parallel without canceling each other. The issue has been fixed. | |![bug](/img/bug.jpg) | Fixed an issue that prevented the second page of inputs from being loaded when some inputs got a FAILED upload status | The second page of inputs is now loaded as desired. | ### Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Added a help message before a user draws a bounding box or a polygon | Added the "you must select a concept before drawing bbox / polygon" help message to be shown every time a user wants to draw a bounding box or a polygon. | | ![improvement](/img/improvement.jpg)| Added ability to move between inputs on the Input-viewer with hotkey(s) | You can now use the keys to move between inputs: `up/down + left/right`. | | ![improvement](/img/improvement.jpg)| Added hover-over tooltips for clickable elements on the Input-Viewer | We've added popovers to various buttons on the Input-Viewer. This lets you know what each button does before clicking it. | | ![bug](/img/bug.jpg) | Fixed some issues that prevented successfully creating a polygon on an input | Previously, clicking the initial point of a polygon could not close it. You can now finish creating a polygon by clicking the initial point.Previously, creating a polygon could sometimes close or finish abruptly without warning or intent. You now need to deliberately click on the initial point to close the polygon. | |![bug](/img/bug.jpg) | Fixed an issue that prevented SDH (Secure Data Hosting) images from rendering on the input viewer page| SDH refers to an approach we use to store and manage inputs in a secure environment. All the input URLs are backed by an input hosting server with token-based authorization. The inputs are fetched from the hosted URLs only with an authorized token. Canvas image rendering for SDH is now working properly. | | ![bug](/img/bug.jpg) | Fixed an issue that when SDH was enabled, the backend service returned an SDH URL, which could not be processed for predictions | Previously, when SDH was enabled, the backend service returned an SDH URL, which was not the URL of the input. However, the backend does not support making predictions using an SDH URL directly—because if we predict using the URL, we download the user-provided inputs directly.We've fixed the issue by removing the URL from the request whenever an input ID is present in the data block. In case there is no ID in the request, we'll use the URL as a fallback. | | ![bug](/img/bug.jpg) | Fixed an issue that allowed the frontend to send the entire `image` object when running model or workflow predictions | Previously, the frontend sent the entire `image` object returned from the API response, implying that the `image.url` was actually the user provided URL, not the hosted URL. Currently, when we make predictions from a user input, the request only has `image.url`, and it's the hosted origin URL built from the API response, without any other fields. The same rule applies to the other input types. | |![bug](/img/bug.jpg) | Fixed an issue where the concept dropdown list still remained visible even when a user navigated away from it | If you go to the Input-Viewer screen, and select the **Annotate** mode option, you can add concepts to an input from the list that drops down after clicking the concepts search box. Previously, that dropdown list still remained visible even after clicking the outside of it. We've fixed the issue. | ### Task-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue where errors were displayed on the tasks listing table | The tasks listing table now displays labeling tasks correctly with no errors. | ### Search |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an issue that caused the model or the workflow selector to return incorrect search results | If you type in the inputs selector and search for a model or a workflow, it now returns the correct search results.We also fixed an issue that caused incorrect states when using the model or the workflow selector component. | ### Email |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an error shown in the verification link of a secondary email | Previously, when a user added a secondary email to their account, and clicked the verification and login link sent to their inbox, they could get an error. We've fixed the issue. | ## Clarifai-Python-Utils |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added more utilities and examples to the [Clarifai-Python-Utils](https://github.com/Clarifai/clarifai-python-utils) SDK | Clarifai-Python-Utils offers a comprehensive set of utilities and tools that simplifies and enhances the integration of Clarifai's powerful AI capabilities into your Python projects. We've added more utilities and examples for building common tasks so that you can leverage the full potential of Clarifai's AI technology.For example, you can now use the SDK to perform data uploads in xView and ImageNet dataset formats while displaying the updated progress of the upload process. | |![new-feature](/img/new_feature.jpg)|Created a Python script that generates an images archive from an export archive, and added it to the Clarifai-Python-Utils repository | Created a Python class that delivers various functionalities via an SDK to a user, including downloading URLs, unarchiving ZIP files, and iterating over all the inputs in an export archive.The script is useful for users who export dataset versions and want to process them further. | --- ## Release 9.6 # Release 9.6 **Release Date:** July 11th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Old Portal |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Decommissioned** | The old portal has now been decommissioned | The old portal is no longer publicly accessible. You can contact our support department if you still need to access the unmaintained old portal (at your own risk). All visitors to `portal.clarifai.com` will be hard redirected to `clarifai.com/explore`. | ## Community ### Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Introduced modules as a way to extend Clarifai’s UIs and perform customized backend processing | [Modules](https://docs.clarifai.com/portal-guide/modules/) are custom plugins that provide both UI elements and the ability to perform some computation around our API. You can now use modules to extend Clarifai UIs, augment backend processing capabilities, and build customized solutions quickly and easily. | ### Large Language Models (LLMs) and Foundation Models (FMs) |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added more LLM and FM model options to allow our users to unleash the power of the latest developments in Generative AI | We are continuing to wrap more LLM and FM models from various vendors into our platform. We intend to provide you with more options so that you can choose the best Generative AI model based on the task you want to achieve.We've wrapped the following model from Anthropic: `claude-v1` for text-to-text tasks. | |![improvement](/img/improvement.jpg)|Performed LLM fine-tuning on mid-large size models | We performed LLM fine-tuning with various base models from `gpt-neo` to `roberta-xlm-lage` using Parameter-Efficient Fine-Tuning (PEFT). We used the LoRA (low-rank adaptation) technique to freeze the weights of the pre-trained models and fine-tune them with a small model, achieving excellent fine-tuning results. The supported "fine-tunable" tasks include text classification (9.6 released) and text generation, such as content creation (coming soon in 9.7 release). You can now perform text classification with two new templates: `gpt-neo-125m-lora` and `gpt-neo-2.7b-lora`. We also added the ability for users to perform advanced custom model configurations. For example, you can set up PEFT, which allows you to get performance comparable to full fine-tuning while only having a small number of trainable parameters. | |![enterprise](/img/enterprise.jpg) / Developer-Preview | Performed LLM inferencing of mid-large size models | We performed inferencing on the following open source LLMs: `hkunlp_instructor-xl` and `tiiuae-falcon-7b-instruct`. We exposed the trained models to novel data and got predictions (inferences), which allowed us to optimize their performance. | ### Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | We've improved the efficiency and fixed some bugs related to importing large batches of inputs via the input uploader | You'll now get a better experience when importing batches of inputs by uploading files, URLs, or texts. You can also efficiently monitor the status of the inputs during and after the upload exercise.| |![bug](/img/bug.jpg) | Fixed an issue that prevented the unification of Input-Manager and Input-Viewer stores | Previously, if you clicked the spyglass icon on an image on the Input-Manager page to perform a visual or face similarity search, and then clicked on any of the images returned by the search, the right image you clicked could not be populated on the Input-Viewer page. The issue has been fixed. | |![bug](/img/bug.jpg)| Fixed an issue that caused slow visual similarity searches | We've improved the speed and efficiency of visual similarity searches. | |![bug](/img/bug.jpg)| Fixed an issue that prevented new datasets from appearing under the **Input Filters** section on the Input-Manager | If you now use the input uploader pop-up window to add a new dataset, it will be correctly listed under **Datasets** without any issues. | |![bug](/img/bug.jpg)| Fixed some issues that caused incorrect styling of the upload job monitor window on the Input-Manager | If some upload jobs are being processed and you collapse the upload monitor, a blue line is now displayed at the bottom of the upload monitor. It turns to green once the upload is successful. If you select some inputs while the upload monitor is still active, the selected items now overlap the upload monitor at the bottom bar. There is a blue line that shows a percentage of the status of the upload process. | |![bug](/img/bug.jpg)| Fixed an issue that caused the inputs upload process to fail when one upload job failed | Previously, if you uploaded inputs in bulk, and one input failed to upload properly, the entire upload process failed. We've fixed the issue by showing an error message for the failed upload, while continuing uploading the rest of the inputs. | |![bug](/img/bug.jpg)| Fixed an issue that prevented Smart Search results from being loaded after a user pressed the enter button on a keyboard |We've fixed the issue, and Smart Search now works as expected, and with better efficiency. | ### Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added ability to hover in order to reveal helpful tips about the labeling tools on the Input-Viewer | If you now hover over a tool icon, you'll reveal a helpful tooltip, including an illustrative animation, a description, and a hotkey. The enhanced tooltips significantly improve the accessibility and usability of the Input-Viewer. | |![bug](/img/bug.jpg) | Fixed an issue that prevented filtering of annotations on the Input-Viewer | If you now select the filter option, it's immediately applied and works as expected. | |![bug](/img/bug.jpg)| Fixed an issue with drawing a second bounding box or a polygon on the Input-Viewer | To annotate an input with a bounding box, you go to the Input-Viewer screen, switch to the bounding box mode, select a concept, and draw it on the canvas area. Previously, if you wanted to add a second bounding box, the canvas area could become grey without any concept selected.You can now add a second bounding box without experiencing any issues. | |![bug](/img/bug.jpg)| Fixed an issue that prevented editing existing polygon annotation points on the Input-Viewer | Previously, if you tried editing an existing polygon point, a new point was created instead of modifying the old one. You can now click on a polygon shape and edit any point without any issues. | |![bug](/img/bug.jpg) | Fixed an issue that caused unexpected backend search requests while scrolling on the Input-Viewer | Previously, there was an issue with the **PostInputSearches** endpoint that caused improper accumulation of input status filters when performing infinite scrolling. We've fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue that prevented the rendering of text inputs on the Input-Viewer when Secure Data Hosting (SDH) was enabled| Previously, if you created an app with the base workflow set to "Text" and uploaded a CSV file having text inputs, the uploaded data could not be rendered properly. We've fixed the issue. | | ![bug](/img/bug.jpg)| Fixed an issue that prevented the rendering of images on the canvas area on the Input-Viewer when SDH was enabled | Images are now rendering properly on the canvas area when SDH is enabled. | | ![bug](/img/bug.jpg)| Fixed an issue that prevented tool icons from showing an active state on the Input-Viewer | Previously, the tool icons did not render an active, or selected, state style, which made it difficult to know the tool you've selected for performing an action, such as drawing a bounding box or a polygon. We've fixed the issue—there is always one tool with an active, or selected, state styling on the Input-Viewer toolbar. | | ![bug](/img/bug.jpg)| Fixed an issue that prevented using up/down hotkeys to navigate between inputs on the Input-Viewer | You can now use the up/down hotkeys to successfully navigate between inputs. | | ![bug](/img/bug.jpg)| Fixed an issue that caused incorrect responsive styling of sidebar preview items on the Input-Viewer | Vertical scrolling now appears after a user has added a 5th concept. Before the 5th item, vertical scrolling does not appear. If the content fits into the box, the concepts do not have horizontal scrolling. | ### User Onboarding |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue where the onboarding window popped up when a user clicked a non-existent URL | Previously, if you clicked a non-existent URL, you got a "Resource does not exist" error and an onboarding window popped up, requiring you to either choose a guided tour of the platform or explore it on your own. If you wanted to explore the platform on your own, a new “my-first-application” application was created for you, even if you already had one. If you now click a non-existent URL, you'll just get a message that the page does not exist and the onboarding process will not be started. | ### Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg)| Published BLIP-2 image captioning and multi-modal embedding models | We've published the [`general-english-image-caption-blip-2`](https://clarifai.com/salesforce/blip/models/general-english-image-caption-blip-2) model. | | ![new-feature](/img/new_feature.jpg) | Published a generative model | We've wrapped the following model from Stability AI: [`Stable-Diffusion-XL-beta-v2-2-2`](https://clarifai.com/stability-ai/stable-diffusion-2/models/stable-diffusion-xl-beta-v2-2-2) for text-to-image generation tasks. | |![improvement](/img/improvement.jpg) | Introduced the use of the logged-in user's PAT (Personal Access Token) when showing the Call by API code samples | Previously, using an organization's PAT in the Call by API code samples gave an error from the backend. Therefore, we now always fetch the logged-in user’s PAT to be used in the Call by API samples. | |![improvement](/img/improvement.jpg)| Improved the URL structure of model versions | The URL of a model version now has a `/versions/` reference in its path. It makes the URL fully qualified and easy to infer. | |![improvement](/img/improvement.jpg)| Improved the animations of the "Try your own image or video" and "Add Public Preview Examples" buttons| If you now hover over either of the buttons on a model overview page, an improved pop-up will appear with a simple animation. | |![improvement](/img/improvement.jpg) | Added "Try your own inputs" support for transfer learned models | You can now use the "Try your own inputs" feature with transfer learned models. Previously, it wasn’t supported. | | ![improvement](/img/improvement.jpg) | Made some UI/UX improvements on the Model-Viewer page | OCR models now support scores output. Improved the Model ID and Description fields. Moved preview examples to the left side of the Canvas. | |![bug](/img/bug.jpg)| Fixed an issue where it was not possible to hover over a smaller bounding box that was within a larger bounding box | Previously, hovering over bounding box predictions of some models was not working very well. Bounding boxes that were inside other bounding boxes could not detect the hovering effect. We've fixed the issue. | ### Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg)| Fixed an issue that prevented saving workflows | You can now create and save a workflow using the "Save Workflow" button without experiencing any issues. | | ![bug](/img/bug.jpg)| Fixed an issue with using the **Suppress output** field to edit a workflow | Previously, you could check the **Suppress output** field and update a workflow. However, if you tried editing the workflow again, you could find that the checkbox was incorrectly showing as unchecked. We've fixed the issue. | ### Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Added a toggle to make an app private or public | You can now use the toggle button on an App Settings page to update its visibility to either private or public. | ### Email |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg) | Fixed an error shown in the verification link of a secondary email | Previously, when a user added a secondary email to their account, and clicked the verification and login link sent to their inbox, they could get an error. We've fixed the issue. | --- ## Release 9.7 # Release 9.7 **Release Date:** August 15th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Old Portal |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Decommissioned** | The old portal has now been decommissioned | The old portal is no longer publicly accessible. You can contact our support department if you still need to access the unmaintained old portal (at your own risk). All visitors to `portal.clarifai.com` (old portal) will be hard redirected to `clarifai.com/explore` (new portal). | ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Enabling [Secure Data Hosting](https://docs.clarifai.com/product-updates/upcoming-api-changes/secure-data-hosting) (SDH) feature for all users | SDH is now enabled for all users. This is a **critical, breaking change**. See [here](https://docs.clarifai.com/product-updates/upcoming-api-changes/secure-data-hosting#what-you-need-to-do) for more information. | ## Community ### Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Published several new, ground-breaking models |Published [**General-English-Image-Caption-Blip-2**](https://clarifai.com/salesforce/blip/models/general-english-image-caption-blip-2), a scalable multimodal pre-training method that enables any Large Language Models (LLMs) to ingest and understand images. It unlocks the capabilities of zero-shot image-to-text generation.Published [**Falcon-7B-Instruct**](https://clarifai.com/clarifai/LLM-OpenSource-Models-Training-Inference-Test/models/tiiuae-falcon-7b-instruct), a 7B parameters LLM based on Falcon-7B and fine-tuned on instructions and conversational data; they thus lend better to popular assistant-style tasks. Published [**Hkunlp_Instructor-XL**](https://clarifai.com/clarifai/LLM-OpenSource-Models-Training-Inference-Test/models/hkunlp_instructor-xl), an embedding model that can generate text embeddings tailored to any task (e.g., classification, clustering, text evaluation, etc.) and domains (e.g., science, finance, etc.) by simply providing the task instruction, without any fine-tuning. Published [**Llama2-7B-Chat**](https://clarifai.com/meta/Llama-2/models/Llama2-7b-chat), a fine-tuned LLM from Meta that is optimized for dialogue use cases.Published [**Llama2-13B-Chat**](https://clarifai.com/meta/Llama-2/models/llama2-13b-chat), a fine-tuned LLM from Meta that is optimized for dialogue use cases. Published [**Text-Bison**](https://clarifai.com/gcp/generate/models/text-bison), a foundation model from GCP (Google Cloud Platform) that is optimized for a variety of natural language tasks such as sentiment analysis, entity extraction, content creation, document summaries, answers to questions, and labels that classify content. Published [**Code-Gecko**](https://clarifai.com/gcp/generate/models/code-gecko), a foundation model from GCP that supports code completion. It generates new code based on the code that was recently typed by a user.Published [**Code-Bison**](https://clarifai.com/gcp/generate/models/code-bison), a foundation model from GCP that supports code generation. It generates code based on a natural language description. For example, it can create functions, web pages, and unit tests.Published [**Textembedding-Gecko**](https://clarifai.com/gcp/embeddings/models/textembedding-gecko), an embedding model from GCP that generates embeddings from the given text, which can be used for different language-related tasks.Published [**Detr-Resnet-101**](https://clarifai.com/facebook/object-detection/models/detr-resnet-101), a DEtection TRansformer (DETR) object detection model that is trained end-to-end on COCO 2017 dataset (118k annotated images).Published [**General-Image-Recognition-Deit-Base**](https://clarifai.com/facebook/image-classification/models/deit-base-distilled-patch16-224), a Data-Efficient Image Transformer (DeiT) image classification model that is pre-trained and fine-tuned on ImageNet-1k (1 million images, 1,000 classes). Published [**Claude-v2**](https://clarifai.com/anthropic/completion/models/claude-v2), a chat completion model from Anthropic, driven by an LLM, for generating contextually relevant and coherent responses. Published [**General-Image-Recognition-Deit-Base**](https://clarifai.com/facebook/image-classification/models/deit-base-distilled-patch16-224), a Data-Efficient Image Transformer (DeiT), state-of-the-art image classification model that is pre-trained and fine-tuned on ImageNet-1k (1 million images, 1,000 classes). Published [**General-English-Image-Caption-Blip-2-6_7B**]( https://clarifai.com/salesforce/blip/models/general-english-image-caption-blip-2-6_7B), a state-of-the-art image captioning model with 6.7B parameters.Published [**Multimodal-Embedder-Blip-2 **]( https://clarifai.com/salesforce/blip/models/multimodal-embedder-blip-2), a scalable multimodal pre-training method that enables any LLMs to ingest and understand images. It unlocks the capabilities of zero-shot image-to-text generation. Published [**XGen-7B-8K-Instruct**](https://clarifai.com/salesforce/xgen/models/xgen-7b-8k-instruct), a powerful 7-billion parameter LLM trained on up to 8K sequence length with fine-tuning on instructional data, enabling robust long sequence. Published [**MPT-Instruct-7B**]( https://clarifai.com/mosaicml/mpt/models/mpt-7b-instruct), a decoder-style transformer LLM, fine-tuned for efficient short-form instruction with 6.7B parameters. | |![new-feature](/img/new_feature.jpg)| Added ability to customize Hugging Face and MMCV (OpenMMLab Computer Vision) deep training templates using the Python config file format| You can now add your own custom model configuration when creating a text classification model using the Hugging Face deep training template. You can also add custom configurations to MMClassification, MMDetection, and MMSegmentation deep training templates. You can customize their loss functions, backbones, necks, heads, and more.| |![bug](/img/bug.jpg)| Fixed an issue that caused the model evaluation process to break when numerous concepts were used | Model evaluation now works as desired. | | ![bug](/img/bug.jpg) | Fixed an issue with the A21 Jurassic generative model that caused it to cache output prompts, resulting in repetitive responses upon subsequent usage | The A21 Jurassic model now generates new responses, providing different outputs each time the page is refreshed. | | ![bug](/img/bug.jpg)| Fixed an issue where models and workflows ignored new app and user IDs | Previously, any attempts to rename an app or a user ID, or to relocate the app to an organization (equivalent to altering the user ID), resulted in the models and workflows failing to recognize these updated values. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed an issue where it was not possible to update a model's visibility |Previously, if you edited a model's visibility and published the changes, trying to edit the model's visibility again could not work. We fixed the issue. | |**Developer Preview with Request-Only Access**| Added ability to [import models](https://docs.clarifai.com/portal-guide/model/hf-model-importer) from Hugging Face | You can now import models with permissive licenses from Hugging Face and use them on the Clarifai platform. | | **Developer Preview with Request-Only Access** | Added ability to fine-tune text-to-text models | Advanced model builders can now further customize the behavior and output of the text-to-text models for specific text generation tasks. They can train the models on specific datasets to adapt their behavior for particular tasks or domains. | ### Model Page Improvements |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Reduced the width of the PAT ID field | If you click the **Use Model** button on a model's page, a small window will pop up. Next, if you select the **Call by API** tab, you'll be able to see the PAT key injected in your code samples. You can also click the **Create a new token** button to generate a new PAT. We improved the layout of the PAT ID field and the **Create a new token** button to enable them to fit properly in the form. | | ![improvement](/img/improvement.jpg)|Added app name to the model/workflow/modules tiles details|We have incorporated the app's name that corresponds to the displayed resource, offering improved clarity. Now, when you review the tile details on an individual page for a model, workflow, or module, you will notice the app's name specified in the following format: `User ID / App ID`.| | ![improvement](/img/improvement.jpg) | Created optimized UX (User Experience) for the prompter models on the Model-Viewer | We refined the way users interact with and receive output from the prompter models, aiming to make the process more intuitive, efficient, and user-friendly. | |![bug](/img/bug.jpg)| Fixed an issue where the model page for any detection model crashed when swapping between the **Overview** and **Concepts** tabs, then back again—but only if the predicted bounding boxes were rendered previously | Previously, if you navigated to any detection model's page, waited for the model to render bounding box predictions on the **Overview** tab (default), and then switched to the **Concepts** tab, switching back to the **Overview** tab generated an error. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue where the **Create new Model** page displayed a series of broken thumbnails | The thumbnails on the page are now displayed properly. | ### Sorting |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added ability to sort by stars and adjusted "created/updated" behavior | This sorting change affects both individual and Community pages as well as all resources—apps, models, workflows, modules, and datasets. In addition to sorting by **Name** and **Last Updated**, we added two more options: **Star Count** (default option, henceforth) and **Date Created**. If a user selects **Date Created**, **Last Updated**, or **Star Count**, the sorting results will be displayed in **Descending** order (by default)—the newest, more popular items will appear first. If a user selects **Model Name**, the sorting results will be displayed in **Ascending** order (by default)—items will be displayed alphabetically. | ### Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Exposed apps as a new resource in the Community listing | Just like models and workflows, you can now share, sort, and search apps in the Community. | ### Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Introduced the Smart Image Search by Caption feature on the Input-Manager | You can rank, sort, and retrieve images based on a predicted match to a query caption text. You just need to provide a caption text that best describes the images you want to search for, and the most relevant matches associated with that query will be displayed. | |![new-feature](/img/new_feature.jpg)| Fixed an issue that caused infinite polling for inputs after uploading has been completed |You can now upload inputs successfully without experiencing any issues. | |![new-feature](/img/new_feature.jpg)| Added ability to view estimated search result counts on the Input-Manager | You can now view an estimated number of inputs associated with your search results. | |![bug](/img/bug.jpg) | Fixed an issue where uploading a CSV file of text data into a newly created dataset did not work | You can now create a new dataset and upload CSV files with text data without encountering any issues. | |![bug](/img/bug.jpg)| Fixed an issue that prevented the unification of Input-Manager and Input-Viewer stores | The Input-Manager and the Input-Viewer now have the same unified stores. They now display the same search results, and the list of inputs used in the inputs manager grid is the same as those used in the inputs gallery on the Input-Viewer page. | ### User Onboarding |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Previously, when you were using an org account, selecting “Explore on your own” on the onboarding modal created the default first app under your logged-in user's account, and not on your org user's account. | The onboarding modal now creates the default first app under the org user's account. The user is also now able to see their full name displayed on the onboarding modal. | |![bug](/img/bug.jpg) | Fixed an issue where a created PAT did not appear in the list of PATs | Previously, when you created a new PAT on the “Use in API” screen during the onboarding process, the PAT was not automatically populated and could not be used straight away, as compared to the standard "Use in API" flow on a model's page. We fixed the issue. | ### User Account Settings |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added ability to make a user's profile public or private | You can now update the visibility of your user profile to either public or private.You will not be able to keep any resources public if you set your user profile visibility to private. | |![improvement](/img/improvement.jpg)| Added new roles on the **Job Role** drop-down list | On the **Account Settings** page, you will find a form that enables you to update your contact information. We have made updates to the roles listed in the **Job Role** field within the form. The new roles are also reflected in the sign-up form. | ### Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg) | Added “AppAddCollaborators” featured flag for org admins and org members | Org admins and org members now have the “AppAddCollaborators” featured flag, which enables them to add collaborators to an app. | |![bug](/img/bug.jpg)| Fixed an issue where an org contributor was not allowed to authorize or uninstall Installed Module Versions (IMV) | An org contributor role now has sufficient scopes to successfully authorize or uninstall IMVs. | |![bug](/img/bug.jpg)| Fixed an issue where a team member was not allowed to view or use IMVs | A team contributor or member role now has sufficient scopes to successfully view or use IMVs. | ### Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Improved the handling of GitHub repository URLs with trailing slashes |We enhanced the way we handle the importation of modules from Streamlit app repositories on GitHub with trailing slashes ("/") at the end of their URLs. | | ![improvement](/img/improvement.jpg) | Improved the design of the **Install Module** pop-up modal | If you click the **Install Module** button at the upper-right corner of an individual module page, a small modal will pop up. We've improved the design of the modal to allow you to select a destination organization where you want to install the module. Previously, you could solely select a user.| |![improvement](/img/improvement.jpg) | Increased the deployment time for the module manager | Previously, when you created a new module, the deployment timed out after 5 minutes. If the module required a longer time to build, the module deployment failed after 5 minutes.We increased the deployment time for the module manager from 5 minutes to 10 minutes. | |![improvement](/img/improvement.jpg) | Updated the module manager to set secrets that modules can use | You can now set environment variables and secrets for module versions as you create them. | |![improvement](/img/improvement.jpg) | Allowed anonymous usage of the [LLM Battleground](https://clarifai.com/anthropic/completion/installed_module_versions/LLMBattleground/compare) module | You can now anonymously use the module to compare the performance of large language models. You do not need to log in to the Clarifai platform before using it. | |![improvement](/img/improvement.jpg) | Added a warning to be displayed before deleting a module version | We added a warning informing a user that deleting a module version will uninstall each of its install instances. | | ![bug](/img/bug.jpg) | Added a directory to prevent modules from breaking libraries | Previously, we encountered a bug that was hindering modules from writing temporary files, leading to the disruption of certain Python packages during runtime. We fixed the issue. | |![bug](/img/bug.jpg) | Fixed a bug that prevented getting commits from GitHub repository branches | We fixed a Module-Manager bug that caused errors when getting commits from GitHub repository branches. | ### Secure Data Hosting (SDH) |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Previously, when SDH was enabled on the Portal, cross-user app copying did not work | Previously, when SDH was active, duplicating an app you are added as a collaborator resulted in the destination app having broken inputs. On the other hand, duplicating your own apps worked just fine. We fixed the issue. | ### Labeler |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg)| Fixed an issue that prevented annotations from being created while working on a task | You can now successfully add annotations when labeling tasks. | ### Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![bug](/img/bug.jpg)| Fixed an issue where selecting concepts from the workflow visual graph builder resulted in an error | Previously, selecting concepts for a model node in the workflow visual graph builder resulted in an empty list. We fixed the issue. | --- ## Release 9.8 # Release 9.8 **Release Date:** September 13th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) **(Developer Preview)** | Released a new [Python SDK](https://github.com/Clarifai/clarifai-python) | We released the first open source version (for developer preview) of a Python SDK designed to offer you a simple, fast, and efficient way to experience the power of Clarifai’s AI platform—all with just a few lines of code. Here's a glimpse of its key features: **First-Class Object Interaction:** The SDK provides a seamless interface for managing essential elements like apps, inputs, datasets, and models, making it easier than ever to work with these core components. You can list, create, delete (optional as applicable), or update them. **Effortless Predictions:** You can effortlessly perform predictions using both models and workflows, enabling you to leverage the power of AI for various use cases.**Data Upload:** Simplify the process of uploading image and text data, saving you time and effort in preparing your datasets for AI analysis. You can check its documentation [here](https://docs.clarifai.com/python-sdk/sdk-overview) and refer to examples [here](https://github.com/Clarifai/examples). | ## Clarifai-Python-Utils |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Deprecated** | We have deprecated the Clarifai Python utilities project in favor of the Python SDK | Starting from version 9.7.1, Clarifai-Python-Utils is no longer actively maintained or supported. We strongly recommend transitioning to the Python SDK, available from the 9.7.2 release onwards, as it offers improved performance and a wider range of features. | ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Published several new, ground-breaking models | Wrapped [Claude-Instant-1.2](https://clarifai.com/anthropic/completion/models/claude-instant-1_2), a fast, versatile, and cost-effective large language (LLM) model with improved math, coding, reasoning, and safety capabilities.Wrapped [Llama2-70b-Chat](https://clarifai.com/meta/Llama-2/models/llama2-70b-chat), a fine-tuned Llama-2 LLM that is optimized for dialogue use cases.Wrapped [StarCoder](https://clarifai.com/bigcode/code/models/StarCoder), an LLM with 15.5 billion parameters, excelling in code completion, modification, and explanation, specifically focused on Python, while also maintaining strong performance in other programming languages. Wrapped [Stable Diffusion XL](https://clarifai.com/stability-ai/stable-diffusion-2/models/stable-diffusion-xl), a text-to-image generation model that excels in producing highly detailed and photorealistic 1024x1024 images.Wrapped [Dolly-v2-12b](https://clarifai.com/databricks/Dolly-v2/models/dolly-v2-12b), a 12 billion parameter causal LLM created by Databricks that is derived from EleutherAI's Pythia-12b and fine-tuned on a ~15K record instruction corpus generated by Databricks employees. Wrapped [RedPajama-INCITE-7B-Chat](https://clarifai.com/togethercomputer/RedPajama/models/RedPajama-INCITE-7B-Chat), an LLM trained on the RedPajama base dataset, and excels in chat-related tasks. It leverages context and understanding to generate coherent and contextually relevant responses. Wrapped [Whisper](https://clarifai.com/openai/transcription/models/whisper), an audio transcription model for converting speech audio to text. Wrapped [ElevenLabs Speech Synthesis](https://clarifai.com/eleven-labs/audio-generation/models/speech-synthesis), a robust text-to-speech and voice cloning model for creating realistic speech and voices.Wrapped [GCP Chirp ASR](https://clarifai.com/gcp/speech-recognition/models/chirp-asr), a state-of-the-art, speech-to-text, speech recognition model developed by Google Cloud.Wrapped [AssemblyAI](https://clarifai.com/assemblyai/speech-recognition/models/audio-transcription), a speech recognition model that can quickly turn pre-recorded audio into text, achieving human-level accuracy in just seconds. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added the [Chatbot](https://clarifai.com/clarifai/generative-ai/installed_module_versions/module-chatbot) module | It's a conversational chatbot that remembers the conversation as you go. | |![new-feature](/img/new_feature.jpg)| Added the [DocQA](https://clarifai.com/clarifai/geoint-demo-3/installed_module_versions/docQA) module | Allows you to upload documents such as PDFs and chat with your data. Watch this [video](https://www.youtube.com/watch?v=HbuOu9zq2UE) to see it in action. | |![improvement](/img/improvement.jpg)| Improved the usability of the [LLM Battleground](https://clarifai.com/meta/Llama-2/installed_module_versions/LLM_Battleground/compare) module | You can now seamlessly use it to evaluate and compare multiple LLMs simultaneously. Learn how to use it [here](https://www.clarifai.com/blog/compare-top-llms-with-llm-battleground). | ## AI Assist |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Added the innovative AI assist feature on the Input-Viewer screen. You can now use it to generate annotations for your inputs automatically | You can now request suggestions from any model or workflow available to you on a particular input. You can then convert the suggestions into annotations. We fixed the following issues to ensure its proper functioning: Fixed an issue that previously caused the AI assist settings to reset frequently when switching between inputs. Now, the AI assist state remains persistent, ensuring a smoother experience when transitioning between inputs. Fixed an issue that led to app crashes when selecting a model within the AI assist modal. Fixed an issue that previously hindered the organization of generated labels, ensuring they are now sorted in descending order based on their concept values.Fixed an issue where suggestions were initially displayed with one color for concepts, but upon refreshing or accepting them, the color would change.Fixed an issue where the pen icon failed to appear for editing the concept list suggestions. Fixed the suggestion behavior so that when a user unchecks the same checkbox, it returns to being a suggestion instead of being completely removed from the list. We ensured that clicking on the three dots next to each label suggestion consistently opens the correct menu, without any unexpected jumps, and displays the menu's content as intended. | ## Smart Object Search |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Introduced the [smart object search](https://docs.clarifai.com/portal-guide/psearch/prank) (also called localized search) feature | You can now use the feature to sort, rank, and retrieve annotated objects (bounding boxes) within images based on their content and similarity. We fixed the following issues to ensure its proper functioning: Fixed an issue that previously hindered the selection of a bounding box prediction nested within a larger bounding box prediction.Fixed an issue that prevented bounding box annotations from being created while working on a task. | ## Evaluation Leaderboard |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Enterprise-only**| Introduced a new leaderboard feature designed to streamline the process of identifying the top-performing models within a specific model type | This feature organizes models based on their evaluation results, making it effortless to access the best models for your chosen criteria.Organizational teams now have the capability to efficiently discover models tailored to a specific task type and evaluation dataset, allowing them to pinpoint the top-performing models effortlessly. Furthermore, they can delve deeper into dataset specifics, label information, and model details while conducting a comprehensive comparison of model performances. | ## Local Model Upload UI |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **Enterprise-only**| Introduced a valuable UI feature that allows users to upload custom-built models directly from their local development environments | This functionality allows you to share and utilize locally trained models on our platform, converting them into Triton models effortlessly. Our platform supports widely used formats like TensorFlow, PyTorch, and ONNX, ensuring compatibility with your preferred development tools. | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![improvement](/img/improvement.jpg) | Added ability to filter inputs and annotations within the Input-Manager based on the type of data they contain | You can now filter based on whether any text, image, video, and/or audio data is contained.You can now filter based on whether any bounding box, polygon, mask, point, and/or span region_info is contained.You can now filter based on whether any frame_info or time_info is contained. | |![bug](/img/bug.jpg)| Fixed an issue with inconsistencies between concept IDs and concept names, which had been causing disruptions across multiple areas | When creating a new concept, its ID now mirrors its name. For instance, if you add an annotation to a dataset on the Input-Manager, the annotation ID aligns with its annotation name. | |![bug](/img/bug.jpg)| Fixed an issue that prevented the gallery from automatically refreshing when uploading inputs on the Input-Manager | Previously, there was no automatic gallery refresh in an app's Inputs-Manager screen during input uploads, specifically when the upload progress percentage changed or when input processing was completed. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue related to bulk data uploading of different data types| When you now upload a mix of data like 50 images and 5 videos simultaneously, the images are sent as a single request, whereas the videos are sent as separate requests, resulting in 5 individual requests for the 5 videos. Videos are uploaded as one per request. Other input types, including text data, are uploaded in batches of 128 each. | |![bug](/img/bug.jpg)| Fixed an issue with the [visual similarity search](https://docs.clarifai.com/portal-guide/psearch/visual-search#smart-image-search-by-visual-similarity) feature | Previously, when you clicked the magnifying glass icon located on the left side of an image, you could not initiate a visual similarity search. We fixed the issue. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added ability to use hotkeys to switch between annotation tools on the Input-Viewer | We improved the accessibility and usability of the Input-Viewer by adding a new feature that enables the use of hotkeys on the annotation tools. For example, **B** is for the bounding box tool, **P** is for the polygon tool, and is **H** for the hand tool. | |![bug](/img/bug.jpg)| Fixed an issue that prevented a collaborator from creating annotations on the Input-Viewer | Collaborators can now successfully create annotations on the Input-Viewer. | ## Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg) | Added a listing for **Datasets** under the AI Lake on the collapsible left sidebar | Just like models, workflows, and modules, we've also added a datasets option on the collapsible left sidebar for your own and organization apps. | |![improvement](/img/improvement.jpg)|Enhanced the app overview page by introducing a dedicated section that highlights the resources available in your app—datasets, models, workflows, and modules. | Now, at a glance, you can see the number of each resource type available in an app. For quick action, you can click the “add” button to add a desired resource, or click the “view” button to see a listing of the items available in your chosen resource type. | |![improvement](/img/improvement.jpg)|Allowed collaborators to click the three-dot icon located on the upper-right section of the app overview page| Previously, this feature was exclusively accessible to the app owner, but now, collaborators with the necessary permissions can also harness its capabilities. Upon clicking the three-dot icon, a pop-up emerges, offering different app management options. | | ![improvement](/img/improvement.jpg) | Introduced valuable enhancements to the app creation process | Particularly, we added a **Primary Input Type** selector in the modal. This selector offers two distinct choices: you can opt for **Image / Video** as the primary input type or choose **Text / Document** based on the specific workflow requirements of your application. This enables you to choose the most suitable default base workflow for the type of input you intend to utilize in your application. Nonetheless, you can still choose any other workflow manually. | |![improvement](/img/improvement.jpg)| Made minor improvements | Removed the “NEW” tag from the “Labeling Tasks” (previously called "Labeller") option on the collapsible left sidebar. It's also now being listed under the AI Lake.Fixed broken "Learn more..." links scattered across various pages that list resources within an app. | |![bug](/img/bug.jpg)| Fixed an issue where creating apps on the Safari web browser failed | Creating apps on the Safari web browser now works as desired. | |![bug](/img/bug.jpg)| Fixed an issue where the length of a long app name exceeded the provided field | App names of varying lengths can now be accommodated within the specified field without causing any display issues. | |![bug](/img/bug.jpg)| Fixed an issue with the search functionality for organizational apps | Previously, if you searched for specific apps within your organization, no search results were returned. We fixed the issue. | ## Models Improvements |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Transitioned dataset information handling for model version creation|Previously, dataset info was only stored in `train_info.params.dataset_id` and `train_info.params.dataset_version_id` . We included an additional check for `train_info.dataset` and `train_info.dataset.version` in the model type fields, which take precedence if available.We also added two new field types, `(DATASET` and `DATASET_VERSION)`, to replace the older ID-based fields, enabling the use of actual dataset objects and facilitating compatibility with datasets from other applications in the future.| |![improvement](/img/improvement.jpg)| Improved the "Use Model / Use Workflow" modal pop-up | When you click either the "Use Model" or the "Use Workflow" button on the respective model's or workflow's page, a pop-up window will appear. We streamlined the user experience by placing the "Call by API" tab as the initial option within this window. Previously, the "Use in a Workflow / App" tab held this position, but we prioritized the more common "Call by API" functionality for easier access.We also enhanced accessibility by positioning Python as the primary option among the programming languages with code snippets.We also updated the code snippets for text models, setting raw text as the default option for making predictions. Predicting via local files and via URLs are still available as optional alternatives. | |![improvement](/img/improvement.jpg)| Created new model versions for person detector models without cropping, as cropping is causing these models to miss people on the margins | We duplicated the existing model versions and modified the data provider parameters to include downsampling, resizing, and padding only, in alignment with the standard upload process for new visual models. | |![improvement](/img/improvement.jpg)| Improved the presentation of the JSON output generated from model predictions | Previously, the JSON output would extend beyond the borders of the display modal screen, causing inconvenience. We also improved the user experience by making the button for copying all the output contents more user-friendly and intuitive. | | ![bug](/img/bug.jpg) | Improved the prediction area for Community models to show a "sign up or log in" prompt for users who are not currently logged in | Previously, when you were not logged in, the Community model output section showed an "insufficient scopes" error. We now intercept the error and instead prompt users to log in or sign up—while showing the default predictions. | |![bug](/img/bug.jpg)| Fixed an issue that caused an application to crash | Previously, if you clicked the "Use Model" button and then selected the "Call by API" option for certain models, the application crashed. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue where an unexpected pop-up window appeared while carrying out various actions | The rogue pop-up interruption is no longer visible when adding models to workflows, when clicking the "Cancel" button while choosing the model path, or when creating a new app. | |![bug](/img/bug.jpg)| Fixed an issue where it was not possible to view the evaluation metrics of old transfer learned models| Previously, you could not access the evaluation metrics for older transfer learning models, as the drop-down menu lacked the option to select a dataset. That limitation applied to all transfer learning models that were trained and evaluated prior to the implementation of the changes on how the evaluation metrics work. | |![bug](/img/bug.jpg)| Fixed an issue where the `base_model` for transfer learning models did not display a list of the available base models | All the models from the base workflow that produce embeddings are currently listed. | ## Sorting |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Changed the default sorting criteria for resources |We changed the default sorting criteria for the resources you own—apps, models, workflows, modules, and datasets—to **Last Updated**. The default sorting criteria for Community resources is still by **Star Count**. | ## Filters |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Introduced a quick way to filter both large language models (LLMs) and image generation (text-to-image) models, making it easier to find the models you need | You can now streamline your search by simply clicking a checkbox to show exclusively the available LLMs on the Community platform. This same convenient feature applies to image generation models as well. | ## User Account Settings |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Enabled a user's active subscription plan to be visible on the billing page | You can now view the correct subscription plan you're enrolled in directly on the billing page. It's also included in the drop-down options in case you wish to explore or switch to another plan. | --- ## Release 9.9 # Release 9.9 **Release Date:** October 10th, 2023 | New Feature | Improvement | Bug Fix | Enterprise Only | | :---: | :---: | :---: | :---: | | ![new-feature](/img/new_feature.jpg) | ![improvement](/img/improvement.jpg) | ![bug](/img/bug.jpg) | ![enterprise](/img/enterprise.jpg) | ## API |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Added inference parameters customization | You can now configure [inference parameters](https://docs.clarifai.com/api-guide/predict/llms/#use-hyperparameters-to-customize-llms) such as temperature, max tokens, and more, depending on the specific model you are using, for both text-to-text and text-to-image generative tasks. This empowers you to customize and fine-tune your model interactions to better suit your individual needs. | |![bug](/img/bug.jpg)|Fixed an issue where the **ListWorkflows** endpoint fetched workflows from deleted apps| Previously, workflows from deleted applications were fetched in the response of the **ListWorkflows** endpoint. We fixed the issue. | ## Python SDK |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Added ability to upload and download text and image data | The [SDK](https://docs.clarifai.com/python-sdk/sdk-overview) supports the following text data formats: plain text, CSV, and TSV (in roadmap). It supports the following image data formats: JPEG, PNG, TIFF, BMP, WEBP, and GIF. You have the flexibility to upload data from either URLs or local storage, whether it's within a single folder or spread across multiple sub-folders. You can upload text data enriched with metadata and text classification annotations. You can also upload image data enriched with annotations, geoinformation, and metadata information.You can efficiently create apps designed for text mode and image mode use cases.The SDK ensures transparent progress tracking during each upload and promptly reports any encountered errors. | |![improvement](/img/improvement.jpg) |Integrated YAML workflow interfaces with proto endpoints | In the SDK, the workflow interfaces are now represented in YAML format, but they are seamlessly integrated with proto endpoints for efficient communication and interaction. | |![improvement](/img/improvement.jpg)| Enhanced the SDK's foundational structure | The SDK now supports authentication only using Personal Access Tokens (PAT) and UserIDs.We plan to provide helper utilities for rich text formatting and printing to facilitate a user-friendly interaction. We are working to constantly add a robust error-handling system that covers various scenarios.| ## New Published Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Published several new, ground-breaking models | Wrapped [Falcon-40B-Instruct](https://clarifai.com/tiiuae/falcon/models/falcon-40b-instruct), a causal decoder-only Large Language Model (LLM) built by TII based on Falcon-40B and fine-tuned on a mixture of Baize.Wrapped [WizardLM-70B](https://clarifai.com/wizardlm/generate/models/wizardLM-70B-V1_0), a model fine-tuned on the Llama2-70B model using Evol+ methods; it delivers outstanding performance.Wrapped [CodeLlama-13B-Instruct-GPTQ](https://clarifai.com/meta/Llama-2/models/codellama-13b-instruct), [CodeLlama-34B-Instruct-GPTQ](https://clarifai.com/meta/Llama-2/models/codellama-34b-instruct), and [CodeLlama-7B-GPTQ](https://clarifai.com/meta/Llama-2/models/codeLlama-7B-GPTQ). Code Llama is a family of code-focused LLMs, built upon Llama 2. These models excel at filling in code, handling extensive input contexts, and following programming instructions without prior training for various programming tasks. Wrapped [WizardLM-13B](https://clarifai.com/wizardlm/generate/models/wizardLM-13B), an LLM fine-tuned on the Llama-2-13b model using the Evol+ approach; it delivers outstanding performance. Wrapped [WizardCoder-15B](https://clarifai.com/wizardlm/generate/models/wizardCoder-15B), a code-based LLM that has been fine-tuned on Llama 2 using the Evol-Instruct method and has demonstrated superior performance compared to other open-source and closed-source LLMs on prominent code generation benchmarks.Wrapped [WizardCoder-Python-34B](https://clarifai.com/wizardlm/generate/models/wizardCoder-Python-34B), a code-based LLM that has been fine-tuned on Llama 2. It excels in Python code generation tasks and has demonstrated superior performance compared to other open-source and closed-source LLMs on prominent code generation benchmarks.Wrapped [Phi-1](https://clarifai.com/microsoft/text-generation/models/phi-1), a high-performing 1.3 billion-parameter text-to-code language model, excelling in Python code generation tasks while prioritizing high-quality training data. Wrapped [Phi-1.5](https://clarifai.com/microsoft/text-generation/models/phi-1_5), a 1.3 billion parameter LLM that excels at complex reasoning tasks and was trained on a high-quality synthetic dataset.Wrapped [OpenAI's GPT-3.5-Turbo-Instruct](https://clarifai.com/openai/completion/models/gpt-3_5-turbo-instruct), an LLM designed to excel in understanding and executing specific instructions efficiently. It excels at completing tasks and providing direct answers to questions.Wrapped [Mistral-7B-Instruct](https://clarifai.com/mistralai/completion/models/mistral-7B-Instruct), a state-of-the-art 7.3 billion parameter language model, outperforming Llama2-13B in multiple NLP benchmarks, including code-related challenges. | ## LlamaIndex |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added support for [LlamaIndex](https://www.llamaindex.ai/) | We now support integrating with the LlamaIndex data framework for various use cases, including: Ingesting, structuring, and accessing external data for the LLMs you fine-tune on our platform. This allows you to boost the accuracy of your LLM applications with private or domain-specific data.Storing and indexing your data for various purposes, including integrating with our downstream vector search and database services.Building an embeddings query interface that accepts any input prompt and leverages your data to provide knowledge-augmented responses. | ## Models |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | **GA Released**| Added ability to fine-tune text-to-text models| Advanced model builders can now further customize the behavior and output of the text-to-text models for specific text generation tasks. They can train the models on specific datasets to adapt their behavior for particular tasks or domains. | |**Enterprise-Preview**| Fixed an issue with the Leaderboard, which organizes models based on their evaluation results | Previously, for visual detection models, the displayed metrics included classification metrics like ROC, instead of the expected metric, which is Mean Average Precision (MAP). The Leaderboard now correctly displays MAP, just like in the model viewer's version table. | |![bug](/img/bug.jpg)| Introduced multi-label text classification with Hugging Face text classification pipeline | Previously, there was an issue with multi-label text classification using the Hugging Face text classification pipeline. While it worked well for multi-class classification, it did not perform correctly for multi-label classification. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed issues with the model version details page | Fixed an issue that prevented the model version details page from displaying any output fields.Fixed an issue that caused the model version details table page to crash unexpectedly.Fixed an issue where the model version details page displayed incorrect parameter values when switching between different model versions.Fixed an issue where the model version details page displayed fields irrelevant to the specific model type. Unnecessary fields unrelated to the selected model type are now not visible. | |![bug](/img/bug.jpg)| Fixed an issue where the Model Viewer page incorrectly rotated images in specific community apps | Previously, when images with a portrait orientation (height greater than width) were submitted to some apps, the predicted detection boxes appeared misaligned or incorrect. The Model Viewer page now correctly rotates images with portrait orientation, aligning detection boxes accurately for improved user experience and detection accuracy. | |![bug](/img/bug.jpg)| Fixed an issue where the model ID disappeared after selecting the `Base embed_model` in the model creation process | Previously, users encountered an issue where their previously entered model ID would disappear upon selecting a base embed model when creating a new model. We fixed the issue. | ## Modules |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Removed collaborative applications from the apps drop-down list within the **Install Module** modal when an org user is selected | Since organization users do not have collaborative apps or engage in collaborations, we have optimized the interface by removing collaborated apps in the **Install Module** modal specifically for this user group. This also fixed the app failure bug. | |![improvement](/img/improvement.jpg)| Enhanced the default app selection in the **Install Module** modal | The **Install Module** modal now automatically selects your current app as the default destination to install your chosen module. | |![improvement](/img/improvement.jpg) | Improved the search functionality for modules | This refinement in the search feature offers a more streamlined and productive experience when looking for specific modules or related content. | | ![improvement](/img/improvement.jpg)| Introduced a handy addition to the module's viewer page—an easy-to-use copy icon | You can now click the copy icon located in the **Latest Version ID** section to swiftly obtain the complete latest version URL for a module. This URL allows you to expedite the module installation process for your application. | ## Input-Manager |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added support for video inputs in Input-Manager | You can now view and manage video inputs directly within the Input-Manager. This new feature provides you with greater flexibility and control over your video assets, enhancing your overall experience with our platform. | |![new-feature](/img/new_feature.jpg)| Added ability to filter inputs by annotation user and annotation status in Input Manager | We introduced two new filter options in the Input-Manager: Labeler filter —Allows you to filter inputs that were annotated by the selected user.Status filter —Allows you to filter inputs that have annotations that match the selected status. | |![bug](/img/bug.jpg)| Fixed checkbox behavior in the input gallery screen | Previously, when you switched from Object Mode to Input Mode, the checkbox for selecting all items would remain checked, causing confusion. We fixed the issue. | |![bug](/img/bug.jpg)|Fixed clear button functionality for input type filters | In the Input-Manager, you can filter inputs by their types, and the clear button is designed to unselect all filtering options. Previously, the clear button was not functioning correctly after applying filters based on input types. You can now utilize it to reset and remove all applied filters. | |![bug](/img/bug.jpg)| Enabled rank search to be compatible with other filters | Previously, there was an issue that prevented other filters from being applied when conducting a rank search. You can now apply rank search in conjunction with other filters, enhancing the effectiveness of your search experience. | |![bug](/img/bug.jpg)|Fixed query timeout issue with metadata searches| Previously, including the "country_code":"US" metadata in your search query led to timeouts. However, when executing the same query without the "country_code":"US" metadata, it completed successfully in approximately 4 seconds. You can now perform your query, even with the "country_code":"US" metadata included, and achieve successful results without experiencing timeouts. | |![bug](/img/bug.jpg)| Eliminated duplicate requests in the Input-Manager's screen | Previously, an issue caused duplicate or even triple requests. It occurred when loading the page with or without a `searchId` in the URL, which resulted in unnecessary and excessive requests. Extra requests are no longer generated. | |![bug](/img/bug.jpg)| Fixed an issue with automatic refresh of the input gallery in Input-Manager after performing bulk edit actions | Previously, there was an issue that hindered the automatic refresh of the input gallery after performing bulk deleting or labeling actions. These improvements ensure that filters accurately reflect the changes made. Previously, bulk deleting of inputs didn't show live updates in the input gallery. We fixed this issue.Previously, bulk labeling of the last remaining unlabeled inputs didn't yield the expected results. We fixed this issue, ensuring that these inputs are correctly labeled after your bulk labeling action. | ## Input-Viewer |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| Added ability to view and tag text inputs within the Input-Viewer | We have designed a comprehensive UI that allows you to interactively create and manage annotations for text inputs. | |![bug](/img/bug.jpg)| Improved keyboard shortcut functionality in Input-Viewer | In the Input-Viewer, you can use keyboard shortcuts (hotkeys), such as H, V, P, and B, to switch between annotation tools. Previously, when opening the model selector or any other element with a popover while in the Input-Viewer, typing any of the hotkeys would unintentionally change tools in the background. We fixed the issue, and the hotkeys do not now trigger events that affect tools in the background. | |![bug](/img/bug.jpg)| Enabled error-free transition to annotation mode in Input-Viewer| Previously, there was an error immediately after switching to annotation mode in the Input-Viewer. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed issue preventing collaborators from creating annotations in Input-Viewer | Collaborators can now actively participate in the annotation process, contributing to a more collaborative and efficient workflow. | |![bug](/img/bug.jpg)| Improved the Input-Viewer's URL handling to ensure a more seamless navigation experience | Previously, there was an issue where the input ID part of the route parameters was not utilized effectively for rendering, leading to undesired redirects. We fixed the issue. | ## Invitations |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![new-feature](/img/new_feature.jpg)| You can now seamlessly review and accept pending organization invitations directly within the Portal, eliminating the need to rely solely on email for this purpose| These invitations could be for various actions, such as app collaboration, organization enrolment, or task participation. This is valuable for both our SaaS offering and our on-premise deployments, as it caters to scenarios where email support may be limited or unavailable. | ## Organization Settings and Management |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with displaying incorrect menu items in the mobile view | Previously, when you switched from your personal user account to an organization account by clicking your profile icon at the top-right corner of the navigation menu bar, the drop-down list displayed your personal details instead of the organization’s information. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue where an organization contributor could not access public workflows | Organization contributors (organization members) can now fetch a complete list of all public workflows without any hindrances. | |![bug](/img/bug.jpg)| Enabled organization contributors to view available destination apps in **Install Module** Modal | We fixed an issue that previously hindered organization contributors from viewing the list of available destination apps within the **Install Module** modal when attempting to install a module. | |![bug](/img/bug.jpg)| Fixed an issue with searching for organization members | Previously, when you used multiple parameters, such as both first name and last name together, to search for members whose first name or last name matched the query text, it could result in an error within the response. We fixed the issue. | |![bug](/img/bug.jpg)| Fixed an issue where collaborators with the organization contributor role failed to list model evaluations | Previously, collaborators trying to list model evaluations encountered an error message stating "Insufficient scopes", despite the expectation that the role would grant them access to view evaluations. We fixed the issue. | ## Apps |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Enhanced App Settings page to display details of collaborators | Now, you can easily access and view the details of your collaborators, including their ID, first and last name, email, job role, and the date they were added to the app. | |![bug](/img/bug.jpg)| Fixed an issue where the app thumbnails were not cached properly | Previously, there was an issue where the images of uploaded app thumbnails on the App Overview page and on the drop-down sidebar list, which provides quick access to your available apps, occasionally appeared as blank or missing. We fixed the issue. | ## Login/Logout and Signup Flow |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![improvement](/img/improvement.jpg)| Introduced modal pop-ups for login and signup, and for various scenarios where redirects were previously employed | We have retained dedicated login and signup pages while introducing modal windows for these actions. You can now access login and signup functions directly from your active page, providing quick and easy access to these essential features.For operations like "Use Model," "Use Workflow," "Install Modules," and more, we have replaced the redirection process with modal pop-ups. This eliminates the extra step and ensures a smoother user journey. | | ![bug](/img/bug.jpg)| Fixed an issue with the reCAPTCHA system | We improved the reCAPTCHA system to offer users a notably smoother and more user-friendly experience. | ## Workflows |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| |![bug](/img/bug.jpg)| Fixed an issue with processing videos using the Universal workflow | You can now confidently process videos with the Universal workflow without encountering any hindrances or issues. | |![bug](/img/bug.jpg)| Fixed an issue with editing a workflow | Previously, while editing any workflow, the model version displayed "No results found," which was inconsistent with the initial workflow creation experience. The model version behavior now matches what is displayed when initially creating a workflow. | |![bug](/img/bug.jpg)| Removed the default/base workflow from "Use Model" modal | To use a model in a workflow, go to the model’s viewer page, click the “Use Model” button at the upper-right corner of the page, select the “Use in a Workflow” tab, and select a destination app and its base workflow. You’ll be redirected to the workflow editor page. Previously, if you tried to update the workflow in the editor page, you could encounter an error. The issue arises because the app's default/base workflow cannot be edited, but this limitation is communicated to the user late only after they've made changes to the workflow. We fixed the issue by graying out or excluding the base workflow option when users attempt to use a model in an existing base workflow. | ## Markdown Search |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added ability to perform text-based searches within the markdown notes of apps, models, workflows, modules, and datasets | For instance, when you visit https://clarifai.com/explore/models and input "detecting the presence" in the search field, the results will include any relevant resource where the phrase "detecting the presence" is found within the markdown notes. This feature enhances search functionality and allows for more precise and context-aware discovery of resources based on the content within their markdown notes. | ## Cover Image |Status |Change |Details | |--------------------------------------|------------------------|---------------------------| | ![new-feature](/img/new_feature.jpg) | Added cover image support for various types of resources, including apps, datasets, models, workflows, and modules | This feature adds a visually engaging element to your resources, making them more appealing and informative. You can also delete the image if you want to. | --- ## Upcoming Platform Changes # Upcoming Platform Changes **Learn about scheduled changes to the Clarifai platform** This page lists the changes that we want you to be aware of well in advance, as they may affect how you use the Clarifai platform. These changes include scheduled downtime and other improvements in the stability, performance, or functionality of the Clarifai platform, all aimed at better serving you as a customer. Some of these changes may not be backward compatible and may require you to update how you call our APIs. We have created this page with the intention of being as transparent as possible, so that you can plan any corresponding changes in advance and minimize any interruptions to your usage of Clarifai services. :::tip The dates listed in the following tables indicate when we plan to implement the changes. While we may actually make the change in the days following the specified date, it is advisable to update your client-side code before that date to minimize any downtime to your applications. ::: ## Upcoming Changes | Date | Change | Details | | :--- | :--- | :--- | | | | | ## Completed Changes | Date | Change | Details | | :--- | :--- | :--- | | June 11th, 2025 | Changes to the use of PATs and API keys | **(Breaking change)** Important changes to the use of PATs and API keys. Learn more [here](./pat-api-keys) | | July 2nd, 2024 | Changed the default base workflow to Universal for applications created via the API | **(Breaking change)** Learn more [here](https://docs.clarifai.com/product-updates/changelog/release106#base-workflow) | | August 15, 2023 | Enabling Secure Data Hosting (SDH) feature for all users | **(Breaking change)** Learn more [here](./secure-data-hosting) | | July 27th, 2023 | Fixed critical issue with the workflow builder returning old hash-based ID | **(Breaking change)** Learn more [here](./workflow-builder-ids) | | January 26th, 2023 | Deprecation of `closed_environment`| Deprecation of `closed_environment` in favor of `enrich_dataset` for creating embedding-classifier models. Learn more [here](./closed-environment) | | January 20th, 2023 | Updates to model and model version endpoints | Critical updates to model and model version endpoints. Learn more [here](./model-version-endpoints) | | January 4th, 2023 | Changes to **PostModelOutputs** and **PostWorkflowResults** responses | Exclusion of Some Fields From **PostModelOutputs** and **PostWorkflowResults** Prediction Responses. Learn more [here](./postmodeloutputs-responses) | | Varied dates | Other previous changes | Other previous completed changes. Learn more [here](./other-previous-changes) | --- ## Deprecation of `closed_environment` in favor of `enrich_dataset` # Deprecation of `closed_environment` in favor of `enrich_dataset` **Learn about deprecation of `closed_environment`** ## Date January 26th, 2023 ## Change Deprecation of `closed_environment` in favor of `enrich_dataset` for creating embedding-classifier models ## Details When using the **PostModels** endpoint to create a custom embedding-classifier model, you could include the `closed_environment` variable, as part of the `modelVersion.OutputInfo.OutputConfig` struct. The variable accepted a Boolean value and specified whether a pre-stored dataset, of (usually) negative embeddings, should be added to the training process of your model. This generally leads to higher model accuracy without any additional effort on your end. - If `closed_environment` was set to `False`, which was the default action, we would try to use additional negative embeddings during the training process. _However, the default action would fail if the underlying base model did not have negative embeddings._ - If it was set to `True`, it meant that the user wanted a closed environment for the training and therefore we did not add additional negative embeddings. This worked for all embedding models. We plan to replace it with `enrich_dataset` that is specified inside `modelVersion.TrainInfo.Params` when creating embedding-classifiers, which is the only type of model that supports it. The `enrich_dataset` variable will be implemented as an `ENUM` instead of a `BOOL` so that it can have two options: `Automatic` (default) and `Disabled`. - `Automatic` means that if there are negative embeddings for a base model, we will use them—and we won’t use them if they’re not available. _So, the training will not fail if the underlying embeddings do not have negative embeddings._ - `Disabled` means that we should not use the negative embeddings whether they are available or not. That way, `enrich_dataset` fixes the problem with `closed_environment`. Previously, setting the `closed_environment` variable to `False` (the default value) would fail if the base model didn’t have the negatives for it. This change will also affect the **PostModelVersions** endpoint. --- ## Decommissioning Legacy Models # Decommissioning Legacy Models **Learn about the models we'll decommission** - **Type of Change** — Important [breaking change](https://docs.clarifai.com/product-updates/changelog/release-types#release-types) regarding decommissioning of legacy models - **Implementation Date** — December 31st, 2025 - **Reason** — We are upgrading our infrastructure, and as part of this transition, we will decommission most of our legacy models. ## Models Here are some legacy models that will be decommissioned on December 31st: - ocr-scene-english-paddleocr - general-english-image-caption-clip - apparel-classification-v2 - general-image-recognition-vit - travel-recognition - celebrity-face-detection - embed - moderation-english-text-classification - nougat-base - ocr-scene-italian-paddleocr - ocr-scene-french-paddleocr - ocr-scene-cyrillic-paddleocr - ocr-scene-arabic-paddleocr - ocr-scene-spanish-paddleocr - ocr-scene-german-paddleocr ## Impact After December 31, these models will be set to **Private** and will no longer accept public inference requests. ## What You Need To Do Update your applications to use the current models available in the [Community](https://clarifai.com/explore), including LLM and VLM options that support zero-shot image classification, OCR, text understanding, and other related tasks. These models can be accessed via the API and used as direct replacements depending on your workflow. If you are unsure which model to migrate to or need guidance choosing the right replacement, feel free to [contact us](https://www.clarifai.com/explore/contact-us) or reach out on [Discord](https://discord.com/invite/WgUvPK4pVD). We are happy to help. --- ## Updates to Model and Model Version Endpoints # Updates to Model and Model Version Endpoints **Learn about updates to model and model version endpoints** ## Date January 20th, 2023 ## Change Critical updates to model and model version endpoints ## Details #### Old Behavior - Previously, using the **PostModels** endpoint to create a new model also created a placeholder version of the model with user-provided fields. And if the `model_type_id` of the model was trainable, then a new ModelVersion was created with UNTRAINED status by default. Otherwise, if the `model_type_id` was not trainable, then a new ModelVersion was created with TRAINED status. - Modifying a model's config settings requires using the **PatchModels** endpoint. It's how you previously changed the info fields, descriptions, notes, metadata for both models and model versions. If you were only patching fields that are informational about the model, and not the model version, a model version was not created. If you were patching a trainable model where the latest model version was trained, and you were only changing the `output_info`, a new trained model version was created with the new info. Otherwise, if you were patching a trainable model where the latest model version had not been trained, the created model version was marked as untrained by default. If you were patching an untrainable model type, the new created model version was marked as trained. - Previously, using the **PostModelVersions** endpoint automatically, by default, kicked off training the latest untrained model version—even though a user may not intend to train the latest version, which could unnecessarily incur training costs. - Previously, using the **PatchModelVersions** endpoint only patched a model versions' visibility, metadata, license, or description—while maintaining the model version's status. #### New Behavior - **PostModels** will create new models but not create new model versions. This means trainable models that have not yet been trained will require the additional step of calling **PostModelVersions**—while providing the `*_info` fields in the model version—to effect training. - **PostModelVersions** will allow users to give information specific to a model version. All the `*_info` fields—such as `output_info`, `input_info`, `train_info`, and `import_info`—will be migrated to the endpoint. This would minimize the confusion and difficulty of maintaining these endpoints. Users will be able patch model specific fields without worrying about model version fields being affected. - **PatchModels** will allow users to patch only the model level fields, nothing in the model version. Unnecessary model versions will no longer be created. This allows users to easily track persisted versions. - **PatchModelVersions** will be the new way to change most of the model version fields like gettable, metadata, license, description, notes, and `output_info` (not including concepts). - If users used `model.output_info.output_config` when inferencing, they will have to change that to `model.model_version.output_info.output_config`. --- ## Other Previous Completed Changes # Other Previous Completed Changes **Learn about other previous completed changes** ## Details | Date | Change | Details | | :--- | :--- | :--- | | November 22, 2022 |Deprecation of `POST /searches` | The generic search API will be deprecated in favor of `POST /inputs/searches` and `POST /annotations/searches`. `POST /searches` will still be supported for now, but will not receive any feature updates so users are suggested to use the newer search endpoints. | | January 20, 2022 |Deprecation of `name` and `display_name` | To make Clarifai Model IDs more readable and user friendly, we plan to make the following API/UI changes during the week of Jan 17th. Please see user impact and suggestions below and contact [support@clarifai.com](mailto:support@clarifai.com) if you have any questions. The old `user_unique_id` will still be usable in all queries, but the responses will be filled with the new `v2_user_unique_id`. `name` and `display_name` are deprecated in the API and UI, and `user_unique_id` will soon be deprecated as well so users are suggested to use new model id field `v2_user_unique_id`. | | November 24, 2021. 9:00am ET | Deprecation of type option | The `type` option in POST /models and /models/searches reuest will no longer be supported and will be removed from our API after this point in time. `model_type_id` is in use for model type references. | | February 12, 2021. 9:00am ET | Deprecation of `delete_all` option | The `delete_all` option in DELETE /inputs request will no longer be supported and will be removed from our API after this point in time. You can delete inputs by id. Each request can have at most 128 ids. | | October 16, 2020. 9:00am ET | Deprecation of Demographics Model | To reduce the risk of race bias in our own models, we have constructed a new approach to visual recognition of race. We've also divided age, race and gender recognition into separate models, and then packaged the models into a new public Demographics Workflow. This new approach provides much more flexibility, and makes outputs easier to parse. We will be retiring the current demographics model on October 16th, 2020. Please reference this [blog post](https://www.clarifai.com/blog/new-demographics-workflow), and our [API documentation](https://docs.clarifai.com/api-guide/api-overview) for more information about how you can update your code to take advantage of the new workflow. | | October 20, 2020. 9:00am ET | Model Training Do Not Wait For Inputs To Be Processed | Currently, when we train a context-based classifier model, we wait for all inputs to be added to your app before a model version is created and processed, with a 1 hour training timeout. In the future, we will use any available inputs and annotations that are available at the time a model version is created for training. If the input is pending or in progress, those inputs and associated annotations will not be used for training. You can use [https://api.clarifai.com/v2/inputs/status](https://api.clarifai.com/v2/inputs/status) to check input counts for each status. | | February 27, 2020. 9:00am ET | Deprecation of Face object from API | The Face object in our API responses will be deprecated in favor of a list of Concepts that other model types return. This should only effect users of the Celebrity, Demographics, or custom face recognition models where the `data.face` attributes like `data.face.identity`, `data.face.age_appearance`, `data.face.gender_appearance`, and `data.face.multicultural_appearance` will now be returned in the list of `data.concepts` Concept object. The API will return both for a while during the transition to give you time to update your code away from using the `data.face` objects altogether. We are doing this to simplify the API interface and make it more easily compatible for advanced functionality that is coming soon in workflows! The custom face recognition and celebrity models are a simple change to just access the new `data.concepts` field, but the demographics model is a more fundamental change away from having three distinct lists of concept to a single list. In order to cope with this, we have introduced a `vocab_id` field in each `data.concepts` entry that is returned by the demographics model so that you can distinguish `age_appearance`, `gender_appearance` and `multicultural_appearance`.To convert new format to old format, check python example [here.](https://github.com/Clarifai/docs/tree/master/examples/deprecate_face_object). | | February 24, 2020. 9:00am ET | Consolidation of Input Related Status Codes | As we support more media types, it is impractical to have status codes for each. Thus status codes will now be prefixed `INPUT_...` rather than `INPUT_IMAGE_...` or `INPUT_VIDEO_...`. We will maintain the int value for the `INPUT_IMAGE_...` prefixed statuses, but no longer support the int values associated with statuses prefixed `INPUT_VIDEO...`. | | February 12, 2020. 9:00am ET | Deprecation of Face model type names | The `facedetect*` model types will be deprecated in favor of their more general `detect*` counterparts. For example these would be the changes of model type: `facedetect` -> `detect` `facedetect-identity` -> `detect-concept` `facedetect-demographics` -> `detect-concept` `facedetect-embed` -> `detect-embed` This change is to unify the APIs around face products and object detection products so that they are compatible everywhere either is used. | | February 3, 2020. 9:00am ET | `PATCH /inputs` overwrite action change | The overwrite action when patching inputs currently has some inconsistent behavior. If you patch `input.data.metadata` or `input.data.geo` fields on an input that has `input.data.concepts` already added to it, these concepts will remain after the patch even though the patch action was `overwrite`. Going forward, the overwrite behavior will overwrite the entire `data` object with what is included in the `PATCH /inputs` API call. Therefore if concepts are not provided in the patch call, but were originally on that input, they will be erased \(overwritten with an empty list of concepts\). You can maintain the current behvaiour by always sending back the complete `data` object from `GET /input/{input_id}` along with any modification to it if you are using the `overwrite` action. Update: this change has become more complicated than originally expected and we may not undergo it after all, more to come in future. Still a good idea to update your PATCH calls to use the `merge` or `remove` actions instead of `overwrite` due to `overwrite`'s inconsistency. | | February 1, 2020. 9:00am ET | Deprecation of Focus Model | The Focus model will no longer be supported and will be removed from our API after this point in time. If you have requests for recognizing focus and blurry regions within images please contact sales@clarifai.com so that we can help you directly. | | November 20, 2019. 9:00am ET | `image.crop` argument will be deprecated | In some requests we used to allow cropping of images during the request using the `image.crop` field. This was for convenience only, but in reality is was rarely ever used and significantly complicates the processing pipelines under the hood. Therefore, we will no longer support the `image.crop` field in any requests that used to accept it. If you want to have similar behavior please crop the images on the client side and send the cropped bytes as base64 encoded image data. | | September 30, 2019. 5:00pm ET | `DELETE /inputs` will only operate asynchronously | Along the same lines as `POST /inputs` becoming completely asynchronous, we are cleaning up some inconsistent behavior in the API for deleting inputs. Previously, when a single image is deleted with `DELETE /inputs` or `DELETE /inputs/{input_id}` it was a synchronous operation, but when a batch of images were deleted it was asynchronous. We are making both asynchronous. This allows us to provide more advanced functionality with workflows that index your inputs. What this means for your code is if you application relies on the input having been deleted when the `DELETE /inputs` or `DELETE /inputs/{input_id}` calls return, you now need to add a second call to `GET /inputs/{input_id}` in order to check that it fails with a not found error. | | September 24, 2019. 5:00pm ET | `POST /inputs` will only operate asynchronously | We are cleaning up some inconsistent behavior in the API where a single image added with `POST /inputs` was a synchronous operation, but a batch of images was asynchronous. We are making both asynchronous. This allows us to provide more advanced functionality with workflows that index your inputs. What this means for your code is if you application relies on added inputs having already been indexed when the `POST /inputs` call returns, you now need to add a second call to `GET /inputs/{input_id}` in order to check the status of the input you just added to look for 30000 \(INPUT\_IMAGE\_DOWNLOAD\_SUCCESS\) status code. | | September 11, 2019. 9:00am ET | Scheduled Database Downtime | We plan to upgrade our database to make it faster and provide more space for your applications. We expect a few minutes of downtime during this upgrade but you should plan for up to an hour of downtime in case things don't go as expected. This will primarily affect the following uses of our platform: POST/GET/PATCH/DELETE inputs, Search, Custom Training, Model Evaluation | --- ## Changes to Use of PATs and API Keys # Changes to Use of PATs and API Keys **Learn about breaking changes to the use of PATs and API keys** ## Change - **Type of Change** — Important [breaking change](https://docs.clarifai.com/product-updates/changelog/release-types#release-types) regarding the use of PATs and API keys. - **Implementation Date** — June 11th, 2025 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; :::info TL;DR Previously, you could access Clarifai models like this: ```bash curl -X POST "clarifai.com/api/_api/v2/users/$YOUR_USER_ID/apps/$YOUR_APP_ID/models/general-image-recognition/outputs" \ -H "Authorization: Key $YOUR_API_KEY" \ # ... (rest of your request) ``` The above method will now fail. You must now specify Clarifai’s user and app ID instead of your own, and authenticate using a PAT instead of an API key. ```bash curl -X POST "clarifai.com/api/_api/v2/users/clarifai/apps/main/models/general-image-recognition/outputs" \ -H "Authorization: Key $YOUR_PAT_TOKEN" \ # ... (rest of your request) ``` ::: ## Details An upcoming release of Clarifai’s API, **11.6**, will significantly change how [Personal Access Tokens (PATs)](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens/) and [API keys](https://docs.clarifai.com/clarifai-basics/authentication/app-specific-api-keys/) work. We plan to implement this change on June 11th, 2025, providing sufficient notice to change the way your applications authenticate on our platform. #### Terminology If any of the terms used here are unfamiliar, you can check them in our [glossary](https://docs.clarifai.com/clarifai-basics/glossary). Specifically, we mention [models](https://docs.clarifai.com/clarifai-basics/glossary/#model), [workflows](https://docs.clarifai.com/clarifai-basics/glossary/#workflows), [public](https://docs.clarifai.com/clarifai-basics/glossary/#public), [private](https://docs.clarifai.com/clarifai-basics/glossary/#private), [collaboration](https://docs.clarifai.com/clarifai-basics/glossary/#collaboration), [organization](https://docs.clarifai.com/clarifai-basics/glossary/#organization), and [community](https://docs.clarifai.com/clarifai-basics/glossary/#community). #### Why we are making this change With PATs you can access resources for which you’re a collaborator or teammate. You can also access public content shared by any user, in addition to all your private content across all of your apps. This simplifies the use of all the resources you have access to unlike using API keys that are restricted to a single application. PAT provides a consistent, secure, and robust authentication method. Finally, for Enterprise clients, Org functionality is PAT only and this change creates a consistent method of authentication across the platform. #### What is changing Previously, you could use API keys to access any model, concept, or workflow owned by the app scoped to the API key, as well as those owned by the user `clarifai` in the application `main`. Now, accessing models or workflows owned by `clarifai` in the application `main` can only be done with a PAT tied to your account. To be specific: - You must now use PATs to make API calls for resources that are outside the scope of your apps, such as Clarifai’s models and workflows. While using a PAT, you must also specify the `USER_ID` of the application owner, and the `APP_ID` of the application that you’re accessing. The legacy behavior allowed you to use the `USER_ID` and `APP_ID` of any application on the platform to access Clarifai models and workflows in the app "main". This change requires you to specify the `USER_ID` (clarifai) and `APP_ID` (main) associated with the application containing the resource (model, concept, workflow, etc). - You will no longer be able to use API keys to access resources outside the application the API key is created in. With a key, there is no need to specify the user_id or the app_id as they are already part of the key. API keys will function as normal when accessing resources within the application the key is created in, but will no longer allow access to resources owned by the user `clarifai` in the application `main`. - Since workflows are a collection of models, some of which may be references to models that are not in the same application as the workflow itself, you should also use PATs to interact with workflows. While API keys will still work for the time being for workflows in the same app as the API key that contains only models from that same app, this will be a very narrow use of workflows. Therefore, we recommend updating your code to use PATs when using workflows too. - The preferred method for accessing the Clarifai API moving forward is with a PAT. To avoid potential future breakage we recommend using a PAT. Of course, we will provide prior notice if additional behavior is going to change for API keys. We hope and expect that this will not be a significant change for you. In order to implement it, you will need to ensure that you set the `PAT`, `USER_ID`, and `APP_ID` variables appropriately. There are examples using all of our supported languages on [this page](https://docs.clarifai.com/api-guide/predict/images), and we are available at any time if you need assistance or have any questions. The best place to contact us for support questions is our [Discord Server](https://discord.gg/WgUvPK4pVD), which is monitored by many of our support teams and is the fastest way to get help. We do apologize for any inconvenience this causes; however, we are confident that this is a positive change that will simplify the usage of the platform going forward and make it easier to leverage AI created by other people on our platform! Thank you for your understanding and please feel free to reach out for any help. --- ## Changes to PostModelOutputs and PostWorkflowResults Responses # Changes to PostModelOutputs and PostWorkflowResults Responses **Learn about changes to PostModelOutputs and PostWorkflowResults responses** ## Date January 4th, 2023 ## Change Exclusion of some fields from **PostModelOutputs** and **PostWorkflowResults** prediction responses ## Details When using the **PostModelOutputs** endpoint or the **PostWorkflowResults** endpoint to make a prediction call, the entire model information, including all hyperparameters, is included for each output in the response. This is extremely verbose and also unnecessary, as the same information appears repeatedly throughout the response. It also impacts network usage, ease of viewing and processing the results and debugging by the user, and other performances. Model description, notes, and related model info fields are to be excluded from **PostModelOutputs** and **PostWorkflowResults** prediction responses. The model and model version ids are still available in the responses. If you need more model info than that available from any of the responses, you can look up the info by model id using the **GetModel** endpoint. --- ## Enabling Secure Data Hosting # Enabling Secure Data Hosting **Learn about how to use SDH to download inputs securely** **Change**: Enabling Secure Data Hosting (SDH) feature for all users **Date**: August 15, 2023 (we’ll enable it in the 9.7 release) **Who is affected**: API users who want to download inputs securely **Type of change**: Critical, breaking change **What you need to do**: [See below](#what-you-need-to-do) ## What is Secure Data Hosting? At Clarifai, we use the Amazon S3 storage service to host user-provided inputs. Additionally, we’ll be introducing Secure Data Hosting (SDH) as an extra layer of security and access control to the data stored on Amazon S3. Secure Data Hosting simply works as a proxy over the Amazon S3 storage service. It acts as an intermediary or middleman, allowing authorized users to access the content stored on Amazon S3. When a user with the necessary authorization tries to access an SDH URL, it will retrieve the corresponding content from the associated S3 URL, and display it to the user. The SDH service uses a token-based authorization mechanism to grant access. All inputs are fetched from the hosted service only with an authorized token—the tokens are issued after a proper user authentication as part of the Clarifai platform login process. By employing secure data hosting as a proxy over Amazon S3, we enhance the security of users’ data while leveraging the robust storage and infrastructure capabilities provided by Amazon S3. The SDH service helps us to ensure that users’ inputs remain protected and can only be accessed by authorized individuals or systems. :::tip - After enabling SDH in the 9.7 release, all inputs will be accessible through SDH URLs. The new inputs will be uploaded to S3 as private files, which can only be accessible via secure SDH URLs. S3 URLs will not be accessible through the API anymore as they will be replaced by SDH URLs —all our endpoints will now return SDH URLs instead of S3 URLs. - Old inputs that were added before the release will keep having public S3 URLs, However, if a non-logged-in user still wants to access an S3 URL that was added before the transition, they will still be able to access that old S3 URL because it remains public. SDH URLs will only be accessible to logged-in, authorized users. - If you share an SDH URL with someone, they will not be able to download the shared content unless they have access to it; for example, you can add them as a collaborator to the app, or you can add them to the same team as yourself, the app owner. ::: ## Uploading and Downloading Inputs Let’s demonstrate how Clarifai implements the Secure Data Hosting feature for uploading and downloading user inputs. ![secure data hosting](/img/others/sdh-1.png) #### Uploading 1. A user starts by uploading an input, such as a file of an image, to the Clarifai platform. 2. Clarifai hosts the file in a Clarifai Amazon S3 bucket. 3. If SDH is enabled for an account, the input hosted on the S3 storage service will be backed by an SDH service with token-based authorization. #### Retrieving 1. The SDH service will fetch inputs from the S3 storage service via an S3 API, and not the S3 web URLs. So, without an authorized token, a user cannot fetch inputs from the re-hosted URLs. 2. If a user logs in successfully, they are issued with an authorized token, which they can use to fetch inputs via the SDH service. 3. If a user logs out, and they do not have a proper token, they cannot download the SDH URL of the input—even if they have a valid input URL. ## How it Works You can start by using the [`ListInputs`](https://docs.clarifai.com/api-guide/data/create-get-update-delete#list-all-inputs) endpoint to list all the inputs in your app. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import sdh1 from "!!raw-loader!../../../code_snippets/api-guide/others/sdh-1.sh"; import sdh2 from "!!raw-loader!../../../code_snippets/api-guide/others/sdh-2.json"; import sdh3 from "!!raw-loader!../../../code_snippets/api-guide/others/sdh-3.json"; import sdh4 from "!!raw-loader!../../../code_snippets/api-guide/others/sdh-4.sh"; import sdh5 from "!!raw-loader!../../../code_snippets/api-guide/others/sdh-5.sh"; import sdh6 from "!!raw-loader!../../../code_snippets/api-guide/others/sdh-6.py"; {sdh1} If you run the above command, and SDH has not been enabled on your account, you’ll get a response with the hosted information about each input. Here is an example response (it’s truncated for brevity): {sdh2} And if SDH has been enabled on your account, you’ll get the following response: {sdh3} Let’s talk about the components of the `image` object in the response: - **url**—the original URL that was used to upload the image to the Clarifai platform. - **hosted**—an object nested within the `image` object that contains the details for downloading the image. - **prefix**—the initial part of the URL for downloading the image. If SDH has been enabled on your account, it will be `https://data.clarifai.com` Otherwise, it will be an S3 URL such as https://s3.amazonaws.com/clarifai-api/img3/prod. - **suffix**—the last part of the URL for downloading the image. - **sizes**—the different sizes you can specify for downloading the image. To get a downloadable URL for the image, you’ll combine the following: `prefix + sizes (specify any size) + suffix` For example, if SDH is enabled, an image’s download URL could look like this: ```text https://data.clarifai.com/orig/users/alfrick/apps/deep-learning/inputs/image/140c856dc82565d2c4d6ea720fceff78 ``` If you click the above URL to access the image, you’ll get an error message. So, without providing the necessary authorization, you cannot access an SDH URL. However, if you provide the authorization, like in the example below, the request will be successful: {sdh4} On the other hand, if SDH is disabled, you can also get the download URL by combining the three components. Here is an example: ```text https://s3.amazonaws.com/clarifai-api/img3/prod/orig/c272a8b0ea6c47c7b17430b922597aad/140c856dc82565d2c4d6ea720fceff78 ``` If you click the above S3 URL, you’ll notice that you can access the image even without providing the necessary authorization. However, if you provide the authorization, like in the example below, the request will fail: {sdh5} So, while you require authorization to access an SDH URL, the authorization does not work when accessing an S3 URL. ## What You Need To Do If you are an API user downloading inputs programmatically, you must change your code to transition from S3 input URLs to SDH input URLs. In order to download the input URL, you need to first check if the URL is an SDH URL or an S3 URL. - If the input URL is an SDH URL, then provide the authorization token to download the URL; - Otherwise, do not provide the authorization token to download the URL. Here is an example in Python: {sdh6} --- ## Changes to Workflow Builder Model IDs # Changes to Workflow Builder Model IDs **Learn about changes to the workflow builder model IDs** **Change**: Fixed an issue where the workflow builder inconsistently returned an old hash-based ID, instead of a new, user-friendly V2 ID of models **Fix Release Date**: July 27th, 2023 **Who is affected**: Portal and API users who need to match the model IDs returned from workflows **Type of change**: Critical, breaking change **What you need to do**: [See below](#what-you-need-to-do). ## Issue If you pick a model when creating a workflow on the Portal, you will see the ID of the model you’ve selected. This is the V2 ID, which is a human-readable, user-friendly version of the ID of the model. For example, the V2 ID of the following visual classification model is `apparel-classification-v2`. ![apparel classification model id](/img/others/workflow-builder-1.png) However, if you save the workflow and access the model again, you will notice that the ID of the model has changed to an old version, which we had migrated from. That ID is a hash value that may not be user-friendly. ![apparel classification model hash id](/img/others/workflow-builder-2.png) And if you use the [ListWorkflows]( https://docs.clarifai.com/api-guide/workflows/create-get-update-delete#get-all-workflows-in-an-app) API method to get all the custom workflows in your app, you notice that the backend returns the old hash-based ID of the model. ![get custom workflows app](/img/others/workflow-builder-3.png) ## Fix To fix the inconsistency, all workflow graphs will start returning the new V2 ID of models, instead of the old hash-based ID. ## What you need to do - If you’re an API user who depends on programmatically matching the model IDs returned from workflows, you need to ensure that you correctly reference the V2 ID of your models in your code. - If you’re a Portal user who checks the results of your workflow graphs by matching the model IDs, you need to ensure that you correctly check the V2 ID of your models. --- ## Clarifai API Overview # Clarifai API Overview **The Clarifai API helps you incorporate powerful AI technology into your application** The Clarifai API \(Application Programming Interface\) is the interface that allows clients and servers to “talk” to each other. This is for “software to software” communication, and it is useful for programmatically implementing Clarifai's AI technology within your own products and tools. The Clarifai API acts as a sort of “gatekeeper” to our software that translates clients like Python, Node, and Java. These clients are used to make requests, which the API translates to commands that the software can understand. Clarifai’s API allows users to access the Clarifai platform through four request types: * **POST** - Upload inputs and information * **PATCH** - Update or modify existing information * **GET** - Request information * **DELETE** - Delete existing information :::tip important You can access Clarifai's API over HTTPS at `https://api.clarifai.com`. ::: To integrate the API into different programming languages, use the REST API directly. You can refer to the cURL examples in the documentation, which can be easily translated into other languages for seamless implementation. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## API Outputs # API Outputs **Understand Clarifai’s API responses** import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import Output1 from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/api-outputs.json"; import Output2 from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/rest-outputs.json"; Clarifai’s API responses are designed to provide clear and structured feedback to help you easily interpret results and effectively debug your code. Each API response adheres to a well-defined, nested, hierarchical structure, where data is organized into distinct messages and fields. The responses are delivered in different data formats depending on your request, but they can be processed in a similar manner programmatically. - [REST API](https://docs.clarifai.com/api-guide/api-overview/api-clients/grpc-vs-http-channels#when-to-use-the-http-channel) requests deliver responses in JSON, characterized by the `{ "key": "value" }` syntax. - For gRPC requests, typically made through our [Python SDK](https://docs.clarifai.com/sdk/python-installation) or other [client libraries](https://docs.clarifai.com/api-guide/api-overview/api-clients/), responses are provided in Protocol Buffers (protobuf), a binary serialization format optimized for speed and size. Protobuf uses the `key: value` syntax in its text format. The specific content within a response varies based on the API endpoint you've used. For example, a response from a [model prediction](https://docs.clarifai.com/api-guide/predict/images#predict-via-url) request will differ significantly from one obtained when [creating a concept](https://docs.clarifai.com/api-guide/concepts/create-get-update/#add-concepts). Example response from REST API request {Output2} Example response from gRPC API request {Output1} ## Response Sections Typically, an API response consists of several key sections, including: - **Id** — The `id` is a unique identifier assigned to an API operation or resource. It serves as a reference that allows tracking, retrieval, and management of the request or resource in future interactions. Example: `id: "a6fbf0c4f4bf4bb09f2158ad3f8dfc4e"`. - **Status** — Indicates the overall success or failure of the request. - **Outputs** — Contains details about the data processed by the API (such as images, videos, or text) and the results of the analysis, such as timestamps, resource information, and other relevant details. - **Results** — This section, which is mostly found within the outputs section, contains the core results of the API call, such as predictions or analysis data. ### Status Section The top-level `status` section provides information about the outcome of the API request. It includes: - `code` — A numeric or string code indicating the status (such as 10000 or SUCCESS for a successful response). See a list of all the Clarifai’s status codes [here](https://docs.clarifai.com/api-guide/api-overview/status-codes). - `description` — A human-readable message describing the status (such as "Ok", "Download pending"). - `req_id` — A unique identifier for the request, useful for debugging and tracking. Below is an example of a successful response from a gRPC API request: ```text status { code: SUCCESS description: "Ok" req_id: "a6073f6b81df496faa084355a05f81f0" } ``` Below is an example of a successful response from a REST API request: ```text "status": { "code": 10000, "description": "Ok", "req_id": "20ab5bcf0566429f9b350e50372456ce" }, ``` Below is an example of a failed response from a gRPC API request: ```text code: MODEL_DOES_NOT_EXIST description: "Model does not exist" details: "Model does not exist" req_id: "4d6cf25f4e0c48b19ff3ec1755ac1b29" ``` Below is an example of a failed response from a REST API request: ```text "status": { "code": 21200, "description": "Model does not exist", "details": "Model does not exist", "req_id": "09715165a0d24a109f3951b61d73dfe8" } ``` ### Outputs Section Beyond the `status` output, the content of the response varies depending on the type of operation you performed. Below are some of the most common details found in API responses. #### Resource Details The response can include details about the resource involved in the request, such as an input, model, workflow, or application. For example, when analyzing an image input, you can find the `id`, `name`, and other relevant details of the model used for processing. ```text "model": { "id": "general-image-recognition", "name": "Image Recognition", "model_version": { "id": "aa7f35c01e0642fda5cf400f543e7c40", "created_at": "2018-03-06T21:10:24.454834Z", "status": { "code": 21100, "description": "Model is trained and ready for deployment" }, } } ``` #### Timestamps Timestamps provide important information about when a resource was created and last modified. These timestamps are crucial for tracking changes, auditing, and understanding the lifecycle of resources in your Clarifai projects. These are the timestamps outputted in a successful response: - `created_at` — Indicates the exact date and time when a resource or an operation was first created or first performed, respectively. - `modified_at` — Indicates the exact date and time when a resource was last modified. Below is an example response from a gRPC API request: :::info The gRPC response includes timestamps in Unix epoch time, which represents the number of seconds since January 1, 1970 (UTC). For example, a `created_at` field of `1457543499` seconds and `608845000` nanoseconds converts to March 9, 2016, at 10:11:39.608845 AM UTC. The `nanos` field adds precision by representing fractional seconds. ::: ```text model { id: "general-image-recognition", name: "Image Recognition", created_at { seconds: 1457543499, nanos: 608845000 }, modified_at { seconds: 1739900944, nanos: 809517000 }, } ``` Below is an example response from a REST API request: ```text "model": { "id": "general-image-recognition", "name": "Image Recognition", "created_at": "2016-03-09T17:11:39.608845Z", "modified_at": "2025-02-18T17:49:04.809517Z", } ``` :::info In REST API responses, [timestamps](https://github.com/protocolbuffers/protobuf/blob/0bfe41b27e3dd8a30ae383210d7af10c28a642ea/src/google/protobuf/timestamp.proto#L108-L144) are converted to the [RFC 3339](https://www.ietf.org/rfc/rfc3339.txt) standard. The standard represents date and time as a string in this format: `{year}-{month}-{day}T{hour}:{minute}:{second}[.{fractional_seconds}]Z`. These are the key formatting rules: - Year (`{year}`) is always four digits (such as `2025`). - Month, day, hour, minute, and second are zero-padded to two digits each (such as `02`, `09`, `17`). - Fractional seconds (optional) can go up to 9 digits, supporting up to nanosecond resolution (such as `.608845`). - The "Z" suffix indicates the UTC time zone (such as `2025-02-18T17:49:04.809517Z`). ::: ### Results Section When you make a successful API call to Clarifai, the "results" section is where the real action happens. It's the payload containing the information you requested — the insights, predictions, or data derived from your input. The exact content of the "results" section varies significantly depending on the API endpoint you're using. Below are some common elements you might encounter across different Clarifai API endpoints (_the examples are for REST API requests_): - **Image recognition requests** — The `outputs` field is a common container for storing the core results of prediction or analysis tasks. Inside `outputs`, the `data` field includes the actual results, such as recognized `concepts` ([labels](https://docs.clarifai.com/portal-guide/inputs-manager/concepts)) and their confidence scores. ```text { "outputs": [ { "data": { "concepts": [ { "id": "dog", "name": "dog", "value": 0.98 }, { "id": "pet", "name": "pet", "value": 0.95 } ] } } ] } ``` - **Concepts creation requests** — The `concepts` field contains the details of the newly created concept, including its ID, name, and other relevant attributes. ```text { "concepts": [ { "id": "charlie", "name": "Charlie", "app_id": "test-app" } ] } ``` - **Object detection requests** — The `regions` field contains information about the [detected objects](https://docs.clarifai.com/api-guide/predict/images#visual-detection) within an image or video. This includes bounding box coordinates, associated `concepts`, and their corresponding confidence scores. ```text { "outputs": [ { "data": { "regions": [ { "region_info": { "bounding_box": { "top_row": 0.1, "left_col": 0.2, "bottom_row": 0.5, "right_col": 0.6 } }, "data": { "concepts": [{ "id": "cat", "name": "cat", "value": 0.97 }] } } ] } } ] } ``` - **Workflow execution requests** — The results contain the outputs of each step in the [workflow](https://docs.clarifai.com/api-guide/workflows/workflow-predict#images) execution. ```text { "results": [ { "outputs": [ { "model": { "id": "general-image-recognition" }, "data": { "concepts": [{ "id": "tree", "name": "tree", "value": 0.92 }] } } ] } ] } ``` ## Response Size Limit API responses are limited to a maximum size of 128 MB. For models deployed using our [Compute Orchestration capabilities](https://docs.clarifai.com/compute/overview), response handling depends on the output type: * **Text outputs** — Text responses are automatically truncated to fit within the 128 MB limit and are returned to the client. * **Binary outputs** (image, audio, video) — Binary data cannot be safely truncated. If a binary response exceeds the size limit, the output is cleared and the request returns a `FAILURE` status. Here is an example of a failure response: ```text output.status.code = status_code_pb2.FAILURE output.status.description = "Binary data cleared due to size limit" output.status.details = ( "Image/audio/video data was removed because response exceeded size limit" ) ``` This behavior ensures API stability and prevents request timeouts caused by oversized responses. --- ## Clarifai CLI # Clarifai CLI **Learn how to use the Clarifai Command Line Interface (CLI)** Clarifai’s Command Line Interface (CLI) is a powerful, user-friendly tool designed to simplify and enhance your experience with our AI platform. By offering a streamlined way to execute tasks directly from the terminal, the CLI eliminates the need for extensive coding or constant reliance on the web interface. Bundled within our [Python SDK package](https://docs.clarifai.com/resources/api-overview/python-sdk), the CLI empowers both technical and non-technical users to efficiently execute a wide range of tasks and boost productivity on the Clarifai platform. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## Installation To begin, install the latest version of the `clarifai` Python SDK package. ```text pip install --upgrade clarifai ``` Once installed, you can confirm the version by running the following command: ```text clarifai --version ``` ## Basics The CLI tool supports a broad range of use cases with intuitive commands and convenient aliases. ```text Usage: clarifai [OPTIONS] COMMAND [ARGS]... Clarifai CLI Options: --version Show the version and exit. --config TEXT Path to config file --context TEXT Context to use for this command --help Show this message and exit. Commands: app (a) Manage Apps: create, list, delete artifact (af) Manage Artifacts: create, upload, download, list, get, delete computecluster (cc) Manage Compute Clusters: create, delete, list config Manage multiple configuration profiles (contexts) deployment (dp) Manage Deployments: create, delete, list, get (status), logs list-instances (li) List available compute instances login Login command to set PAT and other configurations logout Log out by clearing saved credentials model Manage & Develop Models: init, upload, deploy, serve, predict, status, logs, undeploy, list nodepool (np) Manage Nodepools: create, delete, list pipeline (pl) Manage pipelines: upload, init, list, etc pipeline-step (pipelinestep, ps) Manage pipeline steps: upload, test, list, etc pipelinerun (pr) Manage Pipeline Version Runs: pause, cancel, resume, monitor pipelinetemplate (pt) Manage pipeline templates: list, discover, etc run Execute a script with the current context's environment shell-completion Shell completion script whoami Display current user and context info ``` The `--help` option is particularly useful to quickly understand the available functionalities and how to use them. clarifai COMMAND --help For example: clarifai login --help Produces this output: ```text Usage: clarifai login [OPTIONS] [API_URL] Login command to set PAT and other configurations. Options: --pat TEXT Personal Access Token --user-id TEXT User ID (or org ID) --name TEXT Name for the context --help Show this message and exit. ``` ## Clarifai Login clarifai login [OPTIONS] [API_URL] The `clarifai login` command is used to authenticate and configure your connection to the Clarifai platform. This involves setting up a [Personal Access Token](https://docs.clarifai.com/control/authentication/pat) (PAT) and user ID for making API requests. > **Note:** To find your Clarifai user ID, open the collapsible left sidebar and click **Settings**, then select **Account** from the dropdown — your user ID is listed there. > From the same **Settings** menu, select **Secrets** to generate a new PAT or copy an existing one. :::note Authentication Precedence Authentication follows a defined order of precedence. If you set values (such as `CLARIFAI_PAT`) as environment variables in your terminal, these take priority and are used first. If no relevant environment variables are found, the CLI falls back to the credentials and settings defined in the [active context](#clarifai-config). **For security best practices, we recommend using environment variables instead of hardcoding credentials directly in your code.** Here is an example of setting PAT as an environment variable: export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ::: ### Log in clarifai login The `clarifai login` command will prompt you to enter your Clarifai user ID and PAT. The PAT input field is masked for security purposes — simply paste the value and press Enter. > **Note:** The masked field supports **Backspace** (delete one character), **Ctrl+U** (clear the entire line), **Ctrl+W** (delete the last word), and **Ctrl+C** (cancel input). After you provide your credentials, they’re validated automatically, and you’ll be notified immediately if there are any errors. A default context (`default`) is also created for you automatically. Example ```text clarifai login Enter your Clarifai user ID: alfrick > To authenticate, you'll need a Personal Access Token (PAT). > Create one at: https://clarifai.com/alfrick/settings/secrets > Tip: Set CLARIFAI_PAT environment variable to skip this prompt. Enter your Personal Access Token (PAT): ******************************** > Verifying token... ✅ Success! You're logged in as alfrick 💡 Tip: Use `clarifai config` to manage multiple accounts or environments [INFO] 10:39:07.660424 Login successful for user 'alfrick' in context 'default' | thread=8480497856 ``` :::tip What is a Context? A context refers to the active environment settings that determine how your commands interact with the Clarifai platform. Think of a context as a saved set of credentials (such as a PAT key or a specific user ID) you want to work with. ::: ### Non-Interactive Login You can log in non-interactively by providing your PAT directly. This is useful for CI/CD pipelines and scripting. clarifai login --pat YOUR_PAT_HERE The CLI will validate your PAT, auto-detect your user ID, and list your organizations for selection. If you want to skip org selection, provide a user ID: clarifai login --pat YOUR_PAT_HERE --user-id your_user_id ### Log in With a User ID You can log in using your user ID. clarifai login --user_id your_user_id_here Example ```text clarifai login --user_id alfrick > To authenticate, you'll need a Personal Access Token (PAT). > Create one at: https://clarifai.com/alfrick/settings/secrets > Tip: Set CLARIFAI_PAT environment variable to skip this prompt. Enter your Personal Access Token (PAT): ******************************** > Verifying token... ✅ Success! You're logged in as alfrick 💡 Tip: Use `clarifai config` to manage multiple accounts or environments [INFO] 11:59:31.377843 Login successful for user 'alfrick' in context 'default' | thread=8480497856 ``` ### Log in With a Custom API URL You can optionally specify a custom API URL if you are connecting to a Clarifai instance other than the default. clarifai login your_custom_url_here Example ```text clarifai login https://api-dev.clarifai.com/ Enter your Clarifai user ID: alfrick > To authenticate, you'll need a Personal Access Token (PAT). > Create one at: https://clarifai.com/alfrick/settings/secrets > Tip: Set CLARIFAI_PAT environment variable to skip this prompt. Enter your Personal Access Token (PAT): ******************************** > Verifying token... ``` ## Clarifai Logout clarifai logout [OPTIONS] The `clarifai logout` command is used to securely remove saved PATs and credentials from your local environment. By default, the command launches an interactive menu. You can also use flags for non-interactive or scripted workflows. This is useful when rotating credentials, switching accounts, or cleaning up unused contexts. ### Interactive Logout When run without any flags, the command shows the currently active context and launches an interactive, numbered menu for you to select an action. clarifai logout These are the menu menu options: **1. Switch to another context** — Lists all other available contexts along with their associated user IDs and allows you to select one. No credentials are cleared. If only a single context exists, the CLI informs you that there are no other contexts to switch to. **2. Log out of current context (clear credentials)** — Clears the PAT from the active context’s configuration. The context itself (including `user_id` and `api_base`) is preserved, allowing you to re-authenticate later using `clarifai login` without re-entering all details. **3. Log out and delete current context** — Removes the entire context entry, including credentials and metadata. If the current context is the only one available, the CLI clears the PAT but retains the context (at least one context must always exist). If other contexts are available, the CLI automatically switches the active context to the next available one. **4. Log out of all contexts** — Clears the PAT from every configured context in the local config file. This option is useful on shared machines or in security-sensitive environments. **5. Cancel** — Exits the menu without making any changes. Example ```text clarifai logout Current context is configured for user 'alfrick' (context: 'default', api: https://api.clarifai.com) 1. Switch to another context 2. Log out of current context (clear credentials) 3. Log out and delete current context 4. Log out of all contexts 5. Cancel Enter choice: ``` ### Clear Current Session You can log out of the context you are currently using without deleting the context settings. clarifai logout --current Example ```text clarifai logout --current Logged out of context 'default'. ``` To also remove the context entry itself, use the `--delete` flag. clarifai logout --current --delete > **Caution:** Use the `--delete` flag with caution. Clearing a PAT only requires you to log in again, but deleting a context removes the named reference entirely, requiring you to recreate it manually if needed later. ### Targeted Context Logout You can log out of a specific named context without deleting the context settings. clarifai logout --context context-name Example ```text clarifai logout --context default Logged out of context 'default'. ``` To also remove the context entry itself, use the `--delete` flag. clarifai logout --context context-name --delete ### Total Reset You can clear all saved credentials across every configured context at once. This is useful for security auditing or starting fresh. clarifai logout --all :::note Environment Variable Warning After any logout action, if the `CLARIFAI_PAT` environment variable is still set, the CLI displays a warning: ``` Warning: CLARIFAI_PAT environment variable is still set. Run `unset CLARIFAI_PAT` (Linux/macOS) or `$env:CLARIFAI_PAT = ''` (PowerShell) to fully log out. ``` This warning is important because environment variables take precedence over values stored in the config file. As a result, clearing credentials from the config alone does not fully log you out if `CLARIFAI_PAT` is still defined. ::: ## Clarifai Config The `clarifai config` command lets you create and manage various aspects of your Clarifai configuration profiles — known as _contexts_ — which represent different interaction environments or setups, as explained earlier. Each context can store specific authentication details (such as PATs), API URLs, and other configuration settings relevant to a particular Clarifai project or environment. This command includes subcommands for creating, deleting, and switching between named Clarifai contexts, among other actions. ```text Usage: clarifai config [OPTIONS] COMMAND [ARGS]... Manage multiple configuration profiles (contexts). Authentication Precedence: 1. Environment variables (e.g., `CLARIFAI_PAT`) are used first if set. 2. The settings from the active context are used if no environment variables are provided. Options: --help Show this message and exit. Commands: create-context (create) Create a new context. current-context (current) Show the current context's details. delete-context (delete) Delete a context. edit (e) Open the configuration file for editing. env (get-env) Print env vars for the active context. get-contexts (list-contexts, ls) List all available contexts. use-context (use) Set the current context. view (show) Display the current configuration. ``` ### Create Context The `create-context` (or `create`) subcommand creates a new Clarifai context. Note that you'll be prompted to configure the context by providing the necessary details for that context — such as user ID, API URL, and PAT. :::note If you'd like to use the default values (such as the `base-url`), simply press Enter. ::: Usage: clarifai config create-context [OPTIONS] NAME Create a new context. Options: --user-id TEXT User ID --base-url TEXT Base URL --pat TEXT Personal access token Or: Usage: clarifai config create [OPTIONS] NAME Create a new context. Options: --user-id TEXT User ID --base-url TEXT Base URL --pat TEXT Personal access token Here is how you can create a new context by providing a unique name for it: clarifai config create-context your_context_name_here Example ```text clarifai config create-context my_new_context user id: XXXX base url (default: https://api.clarifai.com): personal access token value (default: "ENVVAR" to get our of env var rather than config): XXXX [INFO] 20:05:52.602933 Validating the Context Credentials... | thread=21200 [INFO] 20:05:59.957903 ✅ Context is valid | thread=21200 [INFO] 20:05:59.970741 Context 'my_new_context' created successfully | thread=21200 ``` Here is how you can create a new context with all values specified inline: ```text clarifai config create-context your_context_name_here --user-id your_user_id_here --pat your_pat_here --base-url https://api.clarifai.com ``` Example ```text [INFO] 21:00:58.015800 Validating the Context Credentials... | thread=20976 [INFO] 21:00:59.414716 ✅ Context is valid | thread=20976 [INFO] 21:00:59.424841 Context 'my_new_context' created successfully | thread=20976 ``` ### Display Current Context The `current-context` (or `current`) subcommand displays the details of the currently active Clarifai context. It helps you verify which context is in use and view its associated configuration, such as the user ID, PAT, and base URL. ```text Usage: clarifai config current-context [OPTIONS] Show the current context's details. Options: -o, --output-format [name|json|yaml] ``` > **Note:** The `-o` flag is the short form of the `--output-format` flag. Or: ```text Usage: clarifai config current [OPTIONS] Show the current context's details. Options: -o, --output-format [name|json|yaml] ``` Here is how you can show the currently active context: clarifai config current-context Example ```text clarifai config current-context my_new_context ``` Here is how you can show only the name of the current context: clarifai config current-context --output-format name Example ```text clarifai config current-context --output-format name my_new_context ``` Here is how you can show context details in JSON format: clarifai config current-context -o json Example ```text clarifai config current-context -o json {"CLARIFAI_API_BASE": "https://api.clarifai.com", "CLARIFAI_PAT": "XXXX", "CLARIFAI_USER_ID": "XXXX"} ``` Here is how you can show context details in YAML format: clarifai config current-context -o yaml Example ```text clarifai config current-context -o yaml CLARIFAI_API_BASE: https://api.clarifai.com CLARIFAI_PAT: XXXX CLARIFAI_USER_ID: XXXX ``` ### Delete Context The `delete-context` or (`delete`) subcommand deletes an existing Clarifai context. ```text Usage: clarifai config delete-context [OPTIONS] NAME Delete a context. ``` Or: ```text Usage: clarifai config delete [OPTIONS] NAME Delete a context. ``` Here is how you can delete a context by specifying its name: clarifai config delete-context your_context_name_here Example ```text clarifai config delete-context my_new_context my_new_context deleted ``` ### Edit Configuration File The `edit` (alias `e`) subcommand opens the CLI configuration file for the current context in your default text editor. This allows you to manually add, modify, or remove contexts and their settings — such as PATs, application IDs, or base URLs. This is a useful way to set up or manage multiple contexts directly. > **Note:** Remember to save the file after making changes. ```text Usage: clarifai config edit [OPTIONS] Open the configuration file for editing. ``` Or: ```text Usage: clarifai config e [OPTIONS] Open the configuration file for editing. ``` Here is how you can open the configuration file of your current context for editing: clarifai config edit ### Display Environment Variables The `env` (or `get-env`) subcommand prints the environment variables that correspond to your active Clarifai context. It’s useful if you want to export these variables for use in other tools, scripts, or terminals. The context stores the following variables: | Variable | Description | |----------|-------------| | `CLARIFAI_API_BASE` | Base API URL (default: `https://api.clarifai.com`) | | `CLARIFAI_PAT` | Personal Access Token for authentication | | `CLARIFAI_USER_ID` | Your Clarifai user ID | | `CLARIFAI_HF_TOKEN` | Hugging Face access token for downloading gated or private model checkpoints | ```text Usage: clarifai config env [OPTIONS] Print env vars for the active context. ``` Or: ```text Usage: clarifai config get-env [OPTIONS] Print env vars for the active context. ``` Here is how you can display the environment variables of your current context: clarifai config env Example ```text clarifai config env export CLARIFAI_API_BASE="https://api.clarifai.com" export CLARIFAI_PAT="XXXX" export CLARIFAI_USER_ID="XXXX" export CLARIFAI_HF_TOKEN="XXXX" ``` > **Note:** For Unix-like systems, running `eval "$(clarifai config env)"` configures your current shell session by loading Clarifai-related environment variables directly from your active CLI configuration. This automatically sets up authentication for subsequent Clarifai commands or scripts run within that same session, eliminating the need to manually enter credentials. It's particularly useful for scripting where environment variables are preferred for authentication and for quickly setting up a terminal session with the correct Clarifai credentials. For Windows, you'd typically need to run `clarifai config env` first, copy the output, and then manually run the `set` commands it generates. ### Get All Contexts The `get-contexts` (or `list-contexts`, or `ls`) subcommand lists all Clarifai contexts defined in your configuration file, displayed in a table format. The currently active context is marked with an asterisk (`*`). This provides an overview of the different Clarifai setups you can switch between. ```text Usage: clarifai config get-contexts [OPTIONS] List all available contexts. Options: -o, --output-format [wide|name|json|yaml] ``` Or: ```text Usage: clarifai config list-contexts [OPTIONS] List all available contexts. Options: -o, --output-format [wide|name|json|yaml] ``` > **Note:** The `-o` flag is the short form of the `--output-format` flag.   Here’s how you can list all contexts in a table. By default, the `wide` output format is used, which displays a detailed table with information for each context. clarifai config get-contexts Example ```text clarifai config get-contexts NAME USER_ID API_BASE PAT default XXXX https://api.clarifai.com c1eaa**** * my_new_context XXXX https://api.clarifai.com c1eaa**** test_context XXXX https://api.clarifai.com c1eaa**** ``` Here is how you can list only the names of the contexts: clarifai config get-contexts -o name Example ```text clarifai config get-contexts -o name default my_new_context test_context ``` Here is how you can output context data as JSON: clarifai config get-contexts --output-format json Here is how you can output context data as YAML: clarifai config get-contexts -o yaml ### Switch Contexts The `use-context` (or `use`) subcommand sets a different Clarifai context as the active one. This is useful when you need to switch between environments or credentials without manually updating your configuration each time. ```text Usage: clarifai config use-context [OPTIONS] NAME Set the current context. ``` Or: ```text Usage: clarifai config use [OPTIONS] NAME Set the current context. ``` Here’s how you can switch to a different context by specifying its name: clarifai config use-context your_context_name_here The Clarifai CLI will now use the new specified context for all subsequent operations. Example ```text clarifai config use-context test_context Set test_context as the current context ``` ### Display Current Configuration The `view` (or `show`) subcommand displays the full configuration file, including all defined contexts and their associated details. The currently active context is also indicated in the output. ```text Usage: clarifai config view [OPTIONS] Display the current configuration. Options: -o, --output-format [json|yaml] ``` Or: ```text Usage: clarifai config show [OPTIONS] Display the current configuration. Options: -o, --output-format [json|yaml] ``` > **Note:** The `-o` flag is the short form of the `--output-format` flag.   Here’s how to display the current configuration. By default, the output is shown in YAML format. clarifai config show Example ```text clarifai config show contexts: default: CLARIFAI_USER_ID: alfrick my_new_context: CLARIFAI_API_BASE: https://api.clarifai.com CLARIFAI_PAT: XXXX CLARIFAI_USER_ID: XXXX test_context: CLARIFAI_API_BASE: https://api.clarifai.com CLARIFAI_PAT: XXXX CLARIFAI_USER_ID: XXXX current-context: test_context ``` Here is how you can display the current configuration in JSON format: clarifai config show --output-format json Example ```text clarifai config show --output-format json { "current-context": "test_context", "contexts": { "default": { "CLARIFAI_USER_ID": "alfrick" }, "my_new_context": { "CLARIFAI_API_BASE": "https://api.clarifai.com", "CLARIFAI_PAT": "XXXX", "CLARIFAI_USER_ID": "XXXX" }, "test_context": { "CLARIFAI_API_BASE": "https://api.clarifai.com", "CLARIFAI_PAT": "XXXX", "CLARIFAI_USER_ID": "XXXX" } } } ``` ## Context Override (`--context`) Any command can override the active context for a single invocation using the global `--context` flag. This lets you target different environments or accounts without switching your default context. clarifai --context staging model list clarifai --context prod-openai model predict openai/chat-completion/models/GPT-4 "Hello" This is useful when you need to work with multiple accounts or environments in a single terminal session. ## Clarifai Whoami clarifai whoami [OPTIONS] The `clarifai whoami` command displays information about the currently authenticated user and active context. It reads from the local config file without making an API call by default. ### Basic Usage clarifai whoami ### Include Organizations clarifai whoami --orgs ### Full Profile clarifai whoami --all ### JSON Output (for Scripting) clarifai whoami -o json ## Clarifai App clarifai app [COMMAND] [OPTIONS] The `clarifai app` command provides CRUD operations for Clarifai apps. Alias: `a`. > **Note:** See the [applications documentation](/create/applications/create) for a full guide on creating and managing apps. ### Create an App clarifai app create my-app You can specify a base workflow: clarifai app create my-app --base-workflow Universal ### List Apps clarifai app list ### Delete an App clarifai app delete my-app Aliases: `c` (create), `ls` (list), `rm` (delete). ## Clarifai Model Init The `clarifai model init` command scaffolds a new Clarifai model project. The recommended workflow uses `--toolkit` and `--model-name` to automatically generate all required files with the correct configuration for your model. > **Note:** See the [model upload documentation](/compute/upload) for a full guide on building and uploading models. clarifai model init [OPTIONS] [MODEL_PATH] If `MODEL_PATH` is not provided, the current directory is used. When `--model-name` is provided, a directory is automatically created using the model name. > **Note:** Running `clarifai model init` inside an existing model directory updates the files in place rather than creating a new subdirectory. This means you can safely re-run the command to refresh generated files without duplicating your project structure. ### Initialize With Toolkit (Recommended) The `--toolkit` flag selects a pre-configured inference framework. Combined with `--model-name`, this is the fastest way to get a model ready for local serving or cloud deployment. clarifai model init --toolkit vllm --model-name Qwen/Qwen3-0.6B This creates a `Qwen3-0.6B/` directory with all required files: ``` Qwen3-0.6B/ ├── 1/ │ └── model.py # Model implementation ├── requirements.txt # Dependencies └── config.yaml # Simplified config (auto-filled) ``` The generated `config.yaml` is minimal — no TODOs or placeholders: ```yaml model: id: "Qwen3-0.6B" build_info: python_version: "3.11" compute: instance: g5.xlarge checkpoints: repo_id: Qwen/Qwen3-0.6B type: huggingface when: runtime ``` Fields like `user_id`, `app_id`, and `model_type_id` are resolved automatically from your CLI context at serve/deploy time. **Available toolkits:** | Toolkit | Category | Description | |---------|----------|-------------| | `vllm` | GPU | High-throughput LLM serving with [vLLM](https://docs.clarifai.com/compute/toolkits/vllm) | | `sglang` | GPU | Fast LLM serving with [SGLang](https://docs.clarifai.com/compute/toolkits/sglang) | | `huggingface` | GPU | [HuggingFace](https://docs.clarifai.com/compute/toolkits/hf) Transformers (direct inference) | | `ollama` | Local | [Ollama](https://docs.clarifai.com/compute/toolkits/ollama) local LLM server | | `lmstudio` | Local | [LM Studio](https://docs.clarifai.com/compute/toolkits/lmstudio) local LLM server | | `python` | Other | Blank Python model (default if `--toolkit` omitted) | | `mcp` | Other | [MCP](https://docs.clarifai.com/compute/agents/mcp) tool server (FastMCP) | | `openai` | Other | [OpenAI-compatible](https://docs.clarifai.com/compute/inference/open-ai) API wrapper | More toolkit examples: ```bash # GPU toolkits with HuggingFace models clarifai model init --toolkit vllm --model-name Qwen/Qwen3-0.6B clarifai model init --toolkit sglang --model-name Qwen/Qwen2-7B clarifai model init --toolkit huggingface --model-name google/gemma-2b # Local server toolkits clarifai model init --toolkit ollama --model-name llama3.1 clarifai model init --toolkit lmstudio --model-name qwen/qwen3-4b # Other toolkits clarifai model init --toolkit mcp my-mcp-server clarifai model init --toolkit openai my-wrapper ``` #### Smart Instance Selection When you use `--toolkit` with a GPU toolkit (vLLM, SGLang, HuggingFace) and `--model-name`, the CLI automatically estimates VRAM requirements and selects the optimal GPU instance. For **vLLM and SGLang** models, the CLI fetches the model's `config.json` from HuggingFace to calculate the **exact KV cache** based on the model's architecture (layer count, KV heads, head dimension, context length). This prevents out-of-memory errors that occur when a model's context window requires more KV cache than a simple heuristic predicts. The estimation includes: - **Model weights** — dtype-aware and quantization-aware sizing - **KV cache** — exact calculation from model architecture for the full context window - **Framework overhead** — 2 GiB fixed + 10% of model weights (CUDA context, PyTorch runtime, activations) - **GPU utilization headroom** — 90% utilization factor (matching vLLM/SGLang default `gpu_memory_utilization=0.9`) ```text $ clarifai model init --toolkit vllm --model-name Qwen/Qwen3-4B Instance: g5.xlarge (Estimated 15.9 GiB VRAM (7.5 GiB weights + 5.6 GiB KV cache for 40960 ctx), fits g5.xlarge (22 GiB)) ``` For SGLang models, pre-Ampere GPUs (T4, V100) are automatically excluded since SGLang requires compute capability >= 8.0. Instance recommendations are restricted to **AWS, GCP, and Vultr** cloud providers. > **Note:** See [available cloud instances](/compute/cloud-instances) for the full list. ### Basic Initialization If no `--toolkit` is provided, the command initializes a blank Python model using `ModelClass`: clarifai model init Initialize in a new directory: clarifai model init my_custom_model ### Initialize With GitHub Template You can initialize your model using a template from a GitHub repository: ```bash # Clone a public GitHub repository clarifai model init --github-url https://github.com/Clarifai/runners-examples/tree/main/local-runners/ollama-model-upload # Clone a specific branch clarifai model init --github-url https://github.com/Clarifai/runners-examples/tree/vllm-tool-calling # Clone a private repository with a GitHub PAT clarifai model init --github-url https://github.com/your-username/my-private-model --github-pat YOUR_GITHUB_PAT_TOKEN ``` ### Initialize for Streaming Video Use the `--streaming-video` flag to scaffold a model project pre-configured for video livestreaming workflows, including polygon filtering support: clarifai model init --streaming-video This flag enables a streaming video consumer and adds `ffmpeg`/`av` to the generated Dockerfile. ## Clarifai Model Serve clarifai model serve [OPTIONS] [MODEL_PATH] The `clarifai model serve` command runs a model locally for development and testing. It replaces the older `local-runner` command (which still works as an alias). > **Note:** See the [local runners documentation](/compute/local-runners) for a full guide. ### Basic Usage ```bash # Run in current Python environment (fastest) clarifai model serve ./my-model # Auto-create virtualenv and install dependencies clarifai model serve ./my-model --mode env # Build and run inside Docker clarifai model serve ./my-model --mode container ``` ### Standalone gRPC Mode For offline development without a Clarifai login: ```bash # Start a standalone gRPC server (no login required) clarifai model serve ./my-model --grpc # Custom port clarifai model serve ./my-model --grpc --port 9000 ``` ### Options | Option | Default | Description | |--------|---------|-------------| | `MODEL_PATH` | `.` | Model directory containing `config.yaml` | | `--mode` | `none` | `none` (current env), `env` (auto-venv), `container` (Docker) | | `--grpc` | off | Standalone gRPC server — no API connection needed | | `--port` | 8000 | Server port (with `--grpc`) | | `--concurrency` | 32 | Max concurrent requests | | `--keep-image` | off | Keep Docker image after exit (container mode) | | `--health-check-port` | auto | Port for the health check HTTP server | | `--disable-health-check` | off | Disable the health check server entirely | | `--auto-find-health-check-port` | off | Automatically find an available port if the default is in use | | `-v, --verbose` | off | Show detailed logs | ### How It Works **API-connected mode (default):** 1. Validates `config.yaml` and resolves credentials from your CLI context 2. Auto-creates infrastructure ([compute cluster, nodepool](/compute/deployments/clusters-nodepools), app, model, version, runner, deployment) 3. Starts the model server 4. Displays a code snippet, Playground URL, and predict command 5. Cleans up ephemeral resources (version, runner, deployment) on Ctrl+C **Standalone gRPC mode (`--grpc`):** Starts a standalone gRPC server with no Clarifai API connection. No login required. Useful for offline development. ## Clarifai Model Upload clarifai model upload [OPTIONS] [MODEL_PATH] Upload a model to Clarifai without deploying it. Useful when you want to separate the upload and deploy steps. > **Note:** See the [model upload documentation](/compute/upload) for a full guide. ```bash clarifai model upload ./my-model clarifai model upload ./my-model --platform linux/amd64 clarifai model upload ./my-model -v # verbose ``` After upload, the output includes a clickable model URL, a code snippet for predictions, and suggested next steps (deploy, predict). ## Clarifai Model Deploy clarifai model deploy [OPTIONS] [MODEL_PATH] The `clarifai model deploy` command uploads, builds, and deploys a model to Clarifai cloud compute in one step. All infrastructure ([compute cluster, nodepool](/compute/deployments/clusters-nodepools), [deployment](/compute/deployments/deploy-model)) is **auto-created** — no manual setup required. ### Basic Examples ```bash # Deploy from a local directory (uses instance from config.yaml) clarifai model deploy ./my-model # Explicit instance selection clarifai model deploy ./my-model --instance g5.xlarge # GPU shorthands also work clarifai model deploy ./my-model --instance a10g # Deploy an already-uploaded model by URL clarifai model deploy --model-url https://clarifai.com/user/app/models/id --instance g5.xlarge ``` ### Browse Available Instances Use `--instance-info` or the standalone [`clarifai list-instances`](#clarifai-list-instances) command to see all available hardware across cloud providers: ```bash clarifai model deploy --instance-info clarifai model deploy --instance-info --cloud gcp clarifai model deploy --instance-info --cloud aws --region us-east-1 ``` Example Output ```text Available instance types (use the ID with --instance flag): --instance value Cloud Region GPUs Accelerator GPU Memory CPU CPU Memory ------------------ ------- --------- ------ ------------- ------------ ------- ------------ g4dn.xlarge aws us-east-1 1 NVIDIA-T4 15360Mi 3535m 14197Mi g5.xlarge aws us-east-1 1 NVIDIA-A10G 23028Mi 3535m 13878Mi g6e.xlarge aws us-east-1 1 NVIDIA-L40S 46068Mi 3535m 29033Mi g6e.2xlarge aws us-east-1 1 NVIDIA-L40S 46068Mi 7525m 59343Mi t3a.2xlarge aws us-east-1 0 - - 7525m 29033Mi ... ``` ### Advanced Options ```bash # Specific cloud/region clarifai model deploy ./my-model --instance g5.xlarge --cloud aws --region us-west-2 # Autoscaling clarifai model deploy ./my-model --instance g5.xlarge --min-replicas 2 --max-replicas 10 # Use existing infrastructure clarifai model deploy ./my-model --instance g5.xlarge --compute-cluster-id my-cc --nodepool-id my-np ``` ### Options Reference | Option | Default | Description | |--------|---------|-------------| | `MODEL_PATH` | `.` | Local model directory to upload and deploy | | `--instance` | auto | Hardware instance type (e.g., `g5.xlarge`, `a10g`) | | `--instance-info` | — | Show available instances table, then exit | | `--model-url` | — | Deploy an already-uploaded model (skip upload) | | `--model-version-id` | latest | Specific version to deploy | | `--min-replicas` | 1 | Minimum running replicas | | `--max-replicas` | 5 | Maximum replicas for autoscaling | | `--cloud` | auto | Cloud provider (`aws`, `gcp`, `vultr`) | | `--region` | auto | Cloud region (e.g., `us-east-1`) | | `--compute-cluster-id` | auto | Use existing compute cluster | | `--nodepool-id` | auto | Use existing nodepool | | `-v, --verbose` | off | Show detailed logs | ### Instance Name Resolution The `--instance` flag accepts multiple naming formats, all resolved against the live API catalog: | Input Format | Example | Resolved To | |-------------|---------|-------------| | API instance ID | `g5.xlarge` | `g5.xlarge` | | GPU shorthand | `a10g` | `g5.xlarge` (via accelerator match) | | Legacy nodepool name | `gpu-nvidia-a10g` | `g5.xlarge` (strips `gpu-nvidia-` prefix) | ### Deploy Phases When you run `clarifai model deploy`, it progresses through these phases: ```text ── Validate ── Config validation, HuggingFace repo access check ── Upload ──── Build and push model image ── Deploy ──── Create compute cluster, nodepool, deployment ── Monitor ─── Stream pod events until ready ── Ready ───── Show model URL, predict command, log/status commands ``` Example Output ```text ── Ready ────────────────────────────────────────────── Model deployed successfully! Model: https://clarifai.com/user/app/models/my-model Version: abc12345 Deployment: deploy-my-model-abc123 Instance: g5.xlarge Cloud: AWS / us-east-1 ── Next Steps ───────────────────────────────────────── Predict: clarifai model predict user/app/models/my-model "Hello" Logs: clarifai model logs --deployment "deploy-my-model-abc123" Status: clarifai model status --deployment "deploy-my-model-abc123" Undeploy: clarifai model undeploy --deployment "deploy-my-model-abc123" ``` ## Clarifai List Instances clarifai list-instances [OPTIONS] Browse all available compute instances across cloud providers. Alias: `li`. > **Note:** See the [cloud instances reference](/compute/cloud-instances) for the full list of available hardware. ### Basic Usage ```bash # List all available instances clarifai list-instances # Filter by cloud provider clarifai li --cloud aws # Filter by GPU type clarifai li --gpu H100 # Multi-GPU instances only clarifai li --min-gpus 2 # Minimum GPU memory clarifai li --min-gpu-mem 48Gi # Combined filters clarifai li --cloud aws --gpu L40S ``` ### Options Reference | Option | Description | |--------|-------------| | `--cloud` | Filter by cloud provider (`aws`, `gcp`, `vultr`, `azure`) | | `--region` | Filter by region (e.g., `us-east-1`, `us-central1`) | | `--gpu` | Filter by GPU name (e.g., `A10G`, `H100`, `L40S`) | | `--min-gpus` | Minimum GPU count | | `--min-gpu-mem` | Minimum GPU memory (e.g., `48Gi`, `80Gi`) | ## Clarifai Model Predict clarifai model predict [OPTIONS] [MODEL] [INPUT] Run predictions against any Clarifai model directly from the CLI. > **Note:** See the [inference documentation](/compute/inference/clarifai/api) for a full guide, or the [OpenAI-compatible API](/compute/inference/open-ai) for chat-style usage. ### Input Modes ```bash # Text input clarifai model predict openai/chat-completion/models/GPT-4 "Hello world" # Pipe from stdin echo "Explain quantum computing" | clarifai model predict openai/chat-completion/models/GPT-4 # Image/video/audio file clarifai model predict my/app/models/detector --file photo.jpg # Media URL clarifai model predict my/app/models/detector --url https://example.com/img.jpg # Named parameters clarifai model predict my/app/models/llm -i prompt="Hello" -i max_tokens=200 # OpenAI chat mode clarifai model predict openai/chat-completion/models/GPT-4 --chat "What is AI?" # JSON output clarifai model predict openai/chat-completion/models/GPT-4 "Hello" -o json # Inspect available methods clarifai model predict openai/chat-completion/models/GPT-4 --info ``` ### Options | Option | Description | |--------|-------------| | `MODEL` | Model as `user/app/models/id` or full URL (positional) | | `INPUT` | Text input (positional) | | `--file PATH` | Local file (image, audio, video) | | `--url URL` | Remote media URL | | `-i KEY=VALUE` | Named parameter (repeatable) | | `--inputs JSON` | All parameters as JSON string | | `--chat TEXT` | OpenAI chat mode | | `--method NAME` | Explicit method override | | `--info` | Show available methods, then exit | | `-o text\|json` | Output format (default: `text`) | | `--deployment ID` | Route to specific deployment | | `--model-url URL` | Full model URL (alternative to positional) | ### Auto-Detection - Streaming methods are detected automatically and yield chunks in real-time - Method selection: `--chat` uses OpenAI chat, text input prefers streaming/generate, media uses predict - `--info` shows all available methods with their parameter signatures ## Clarifai Model Status clarifai model status [OPTIONS] [MODEL_REF] Check deployment status for a model or a specific deployment. > **Note:** See [managing deployments](/compute/deployments/manage-compute) for a full guide. ```bash # By deployment ID clarifai model status --deployment deploy-abc123 # By model reference (lists all deployments for the model) clarifai model status user/app/models/my-model # By model URL clarifai model status --model-url https://clarifai.com/user/app/models/my-model ``` Example Output ```text ── Deployment: deploy-my-model-dd8481 ────────────────── Model: user/app/models/my-model Version: 720aa2682d9f Min replicas: 1 Max replicas: 5 Nodepool: deploy-np-g5-xlarge Compute cluster: deploy-cc-aws-us-east-1 Created: 2026-03-01 11:47:45 ``` | Option | Description | |--------|-------------| | `MODEL_REF` | Model as `user/app/models/id` (positional) | | `--model-url URL` | Full model URL | | `--deployment ID` | Show a specific deployment | ## Clarifai Model Logs clarifai model logs [OPTIONS] Stream logs from a deployed model's runner. > **Note:** See [managing deployments](/compute/deployments/manage-compute) for more on monitoring and observability. ```bash # By deployment ID (recommended) clarifai model logs --deployment deploy-abc123 # By model URL clarifai model logs --model-url https://clarifai.com/user/app/models/id # Kubernetes scheduling/scaling events clarifai model logs --deployment deploy-abc123 --log-type events # Print current logs and exit clarifai model logs --deployment deploy-abc123 --no-follow # Stop after 60 seconds clarifai model logs --deployment deploy-abc123 --duration 60 ``` | Option | Default | Description | |--------|---------|-------------| | `--deployment ID` | — | Deployment ID (auto-resolves model/nodepool) | | `--model-url URL` | — | Clarifai model URL | | `--log-type` | `model` | `model` (stdout/stderr) or `events` (k8s scheduling/scaling) | | `--follow/--no-follow` | follow | Continuously tail or print and exit | | `--duration N` | unlimited | Stop after N seconds | --- ## Clarifai Model Undeploy clarifai model undeploy [OPTIONS] [MODEL_REF] Permanently remove a deployment. > **Note:** See [managing deployments](/compute/deployments/manage-compute) for a full guide. ```bash # By deployment ID clarifai model undeploy --deployment deploy-abc123 # By model reference (auto-selects if exactly 1 deployment) clarifai model undeploy user/app/models/my-model # By model URL clarifai model undeploy --model-url https://clarifai.com/user/app/models/my-model ``` | Option | Description | |--------|-------------| | `MODEL_REF` | Model as `user/app/models/id` (positional) | | `--model-url URL` | Full model URL | | `--deployment ID` | Target a specific deployment | --- ## Clarifai Model List clarifai model list [OPTIONS] [USER_ID] List models for a user or across the platform. ```bash clarifai model list # List your models clarifai model list # List another user's models clarifai model list -a # Filter by app ``` ## Clarifai Deployment The `clarifai deployment` command provides direct management of deployments by ID. Alias: `dp`. > **Note:** See [managing deployments](/compute/deployments/manage-compute) and [clusters and nodepools](/compute/deployments/clusters-nodepools) for full guides. ### Get Deployment Details Show details for a single deployment: ```bash clarifai deployment get deploy-abc123 clarifai deployment status deploy-abc123 # alias ``` ### List Deployments ```bash clarifai deployment ls # All deployments clarifai deployment ls # Filter by nodepool ``` ### Stream Deployment Logs ```bash clarifai deployment logs deploy-abc123 clarifai deployment logs deploy-abc123 --log-type events clarifai deployment logs deploy-abc123 --no-follow clarifai deployment logs deploy-abc123 --duration 60 ``` | Option | Default | Description | |--------|---------|-------------| | `DEPLOYMENT_ID` | — | Deployment ID (positional, required) | | `--log-type` | `model` | `model` (stdout/stderr) or `events` (k8s scheduling/scaling) | | `--follow/--no-follow` | follow | Continuously tail or print and exit | | `--duration N` | unlimited | Stop after N seconds | ### Delete a Deployment Delete a deployment by ID (no nodepool argument required): clarifai deployment rm deploy-abc123 ### Create a Deployment clarifai deployment create <nodepool_id> <deployment_id> --config deployment.yaml ## Clarifai Pipeline clarifai pipeline [COMMAND] [OPTIONS] The `clarifai pipeline` command lets you create and manage [pipelines](/compute/pipelines) on the Clarifai platform. Alias: `pl`. > **Note:** See the [Create and Run Pipelines](/compute/pipelines/create-api) guide for a full end-to-end walkthrough. ### Compile clarifai pipeline compile [OPTIONS] PATH Compile YAML/config-based pipeline assets from a Python pipeline definition. The `PATH` argument is the path to the Python file containing your pipeline definition. | Option | Description | |--------|-------------| | `PATH` | Path to the Python pipeline definition file (positional, required) | | `--output-dir PATH` | Directory to write the compiled `config.yaml` and step folders (required) | ```bash clarifai pipeline compile pipeline_definition.py --output-dir ./my-pipeline ``` ### Init clarifai pipeline init [OPTIONS] [PIPELINE_PATH] Initialize a new pipeline project structure, either from a predefined template or interactively. When `PIPELINE_PATH` is not specified, the current directory is used. | Option | Description | |--------|-------------| | `PIPELINE_PATH` | Path where the pipeline project structure will be created (positional, optional) | | `--template TEXT` | Initialize from a named template. Run `clarifai pipelinetemplate ls` to list available templates (e.g., `classifier-pipeline-resnet-quick-start`) | | `--set TEXT` | Override template parameters inline. Format: `--set key=value`, repeatable. Use `--set id=` to rename the pipeline or `--set num_epochs=20` to override a model parameter default | | `--user_id TEXT` | User ID for generated configs. Overrides the user ID from your login context | | `--app_id TEXT` | App ID for generated configs. Overrides the app ID from your login context | ```bash # Interactive initialization in the current directory clarifai pipeline init # Initialize in a named directory clarifai pipeline init my-pipeline # Initialize from a template clarifai pipeline init --template classifier-pipeline-resnet-quick-start # Initialize from a template with parameter overrides clarifai pipeline init --template classifier-pipeline-resnet-quick-start --set id=my-pipeline --set num_epochs=20 ``` ### List clarifai pipeline list [OPTIONS] List all pipelines for the user. Alias: `ls`. | Option | Description | |--------|-------------| | `--app_id TEXT` | App ID to list pipelines from (required) | | `--user_id TEXT` | User ID to list pipelines from. Defaults to the current user if not provided | | `--page_no INTEGER` | Page number to list | | `--per_page INTEGER` | Number of items per page | ```bash clarifai pipeline list --app_id my-app clarifai pipeline ls --app_id my-app --user_id my-user-id ``` ### Run clarifai pipeline run [OPTIONS] [PATH] Run a pipeline and monitor its progress. `PATH` is an optional path to a pipeline directory or config file (defaults to `.`). When provided, config precedence is `config-lock.yaml` > `config.yaml`. | Option | Type | Description | |--------|------|-------------| | `PATH` | `PATH` | Path to a pipeline directory or config file. Defaults to current directory | | `--pipeline_id` | `TEXT` | Pipeline ID to run | | `--pipeline_version_id` | `TEXT` | Pipeline version ID to run | | `--pipeline_version_run_id` | `TEXT` | Pipeline version run ID. A UUID is generated if not provided | | `--user_id` | `TEXT` | User ID of the pipeline | | `--app_id` | `TEXT` | App ID that contains the pipeline | | `--pipeline_url` | `TEXT` | Full pipeline URL to run | | `--instance` | `TEXT` | Hardware instance type (e.g., `g5.xlarge`, `A10G`). Auto-creates compute cluster and nodepool | | `--cloud` | `TEXT` | Cloud provider (e.g., `aws`, `gcp`). Auto-detected from `--instance` if omitted | | `--region` | `TEXT` | Cloud region (e.g., `us-east-1`). Auto-detected from `--instance` if omitted | | `--nodepool_id` | `TEXT` | **[Advanced]** Existing nodepool ID (skips auto-creation) | | `--compute_cluster_id` | `TEXT` | **[Advanced]** Existing compute cluster ID (skips auto-creation) | | `--timeout` | `INTEGER` | Maximum time to wait for completion in seconds. Default: `3600` (1 hour) | | `--monitor_interval` | `INTEGER` | Interval between status checks in seconds. Default: `10` | | `--log_file` | `PATH` | File path to write logs to. Logs are displayed on the console if omitted | | `--monitor` | `FLAG` | Monitor an existing pipeline run instead of starting a new one. Requires `--pipeline_version_run_id` | | `--set` | `TEXT` | Override parameter values inline. Format: `--set key=value`. Repeatable | | `--overrides-file` | `PATH` | Path to a JSON/YAML file containing parameter overrides | | `--dev` | `FLAG` | Upload local code to an ephemeral dev pipeline before running. Only changed steps are re-uploaded | ```bash # Run on on-demand instance compute clarifai pipeline run --instance g5.xlarge # Run on an existing cluster and nodepool clarifai pipeline run --compute_cluster_id my-cluster --nodepool_id my-nodepool # Override parameters at runtime clarifai pipeline run --instance g5.xlarge --set num_epochs=20 --set batch_size=32 # Monitor an existing run clarifai pipeline run --monitor --pipeline_version_run_id # Run with dev mode (re-upload only changed steps) clarifai pipeline run --instance g5.xlarge --dev ``` ### Upload clarifai pipeline upload [OPTIONS] [PATH] Upload a pipeline and its associated pipeline steps to Clarifai. `PATH` is the path to the pipeline configuration file or directory containing `config.yaml`. Defaults to the current directory if not specified. | Option | Description | |--------|-------------| | `PATH` | Path to the pipeline config file or directory (positional, optional) | | `--no-lockfile` | Skip generating a `config-lock.yaml` file | ```bash # Upload from current directory clarifai pipeline upload # Upload from a specific path clarifai pipeline upload ./my-pipeline # Upload without generating a lock file clarifai pipeline upload --no-lockfile ``` ### Validate Lock clarifai pipeline validate-lock [OPTIONS] [LOCKFILE_PATH] Validate a `config-lock.yaml` file for schema correctness and reference consistency. If `LOCKFILE_PATH` is not provided, the command looks for `config-lock.yaml` in the current directory. ```bash # Validate in current directory clarifai pipeline validate-lock # Validate a specific lock file clarifai pipeline validate-lock ./my-pipeline/config-lock.yaml ``` ## Clarifai Pipeline Step clarifai pipelinestep [COMMAND] [OPTIONS] The `clarifai pipelinestep` command lets you manage individual pipeline steps independently of a full pipeline. Aliases: `pipeline-step`, `ps`. > **Note:** See [pipeline steps](/compute/pipelines/create-api#pipeline-steps) for a full guide. ### Init clarifai pipelinestep init [OPTIONS] [PIPELINE_STEP_PATH] Initialize a new pipeline step directory structure. If `PIPELINE_STEP_PATH` is not specified, the current directory is used. The command creates the following structure: ```text ├── 1/ │ └── pipeline_step.py # Step implementation logic ├── requirements.txt # Step dependencies └── config.yaml # Step configuration ``` ```bash # Initialize in the current directory clarifai pipelinestep init # Initialize in a named directory clarifai pipelinestep init my-step ``` ### List clarifai pipelinestep list [OPTIONS] List all pipeline steps for the user. Alias: `ls`. | Option | Description | |--------|-------------| | `--app_id TEXT` | App ID to list pipeline steps from (required) | | `--user_id TEXT` | User ID to list pipeline steps from. Defaults to the current user if not provided | | `--pipeline_id TEXT` | Filter pipeline steps by pipeline ID | | `--page_no INTEGER` | Page number to list | | `--per_page INTEGER` | Number of items per page | ```bash clarifai pipelinestep list --app_id my-app clarifai pipelinestep ls --app_id my-app --pipeline_id my-pipeline ``` ### Local Run clarifai pipelinestep local-run [OPTIONS] [PIPELINE_STEP_PATH] Run a pipeline step locally in a Docker container, without uploading it to Clarifai. This is useful for validating your step's logic and dependencies during development before deploying. `PIPELINE_STEP_PATH` is the path to the pipeline step directory (containing `config.yaml`, `requirements.txt`, and `1/pipeline_step.py`). Defaults to the current directory. The command reuses the same Docker build infrastructure as `clarifai model serve --mode container`, but executes `pipeline_step.py` once and exits. | Option | Description | |--------|-------------| | `PIPELINE_STEP_PATH` | Path to the pipeline step directory (positional, optional) | | `--mode [container]` | Execution mode. Currently only `container` is supported | | `--step-args TEXT` | Arguments to pass to `pipeline_step.py` (e.g., `"--param_a hello --param_b world"`) | | `--keep-image` | Keep the Docker image after the step finishes | ```bash # Run in the current directory clarifai pipelinestep local-run # Run a step in a specific directory clarifai pipelinestep local-run ./my-step # Pass arguments to the step script clarifai pipelinestep local-run ./my-step --step-args "--input_text hello" # Keep the Docker image after execution clarifai pipelinestep local-run ./my-step --keep-image ``` ### Upload clarifai pipelinestep upload [OPTIONS] [PIPELINE_STEP_PATH] Upload a pipeline step to Clarifai. `PIPELINE_STEP_PATH` is the path to the pipeline step directory. Defaults to the current directory if not specified. By default, a `Dockerfile` is automatically generated during upload. Use `--skip_dockerfile` if you want to use an existing one you've already edited. | Option | Description | |--------|-------------| | `PIPELINE_STEP_PATH` | Path to the pipeline step directory (positional, optional) | | `--skip_dockerfile` | Skip auto-generating a Dockerfile, so an existing one is used instead | ```bash # Upload from current directory clarifai pipelinestep upload # Upload from a specific path clarifai pipelinestep upload ./my-step # Upload using an existing Dockerfile clarifai pipelinestep upload ./my-step --skip_dockerfile ``` --- ## gRPC API Clients # gRPC API Clients **Clarifai API provides gRPC clients in the most popular languages** You can access the Clarifai API through clients in many of the most popular programming languages. Our clients are built on [gRPC](https://grpc.io/) and are accessible through HTTP+JSON channels as well as gRPC channels. ## Official Clients | Available Clients | | :--- | | [Clarifai Python](https://github.com/Clarifai/clarifai-python-grpc/) | | [Clarifai Java](https://github.com/Clarifai/clarifai-java-grpc/) | | [Clarifai NodeJS](https://github.com/Clarifai/clarifai-nodejs-grpc) | | [Clarifai C\#](https://github.com/Clarifai/clarifai-csharp-grpc/) | | [Clarifai PHP](https://github.com/Clarifai/clarifai-php-grpc/) | | [Clarifai Swift](https://github.com/Clarifai/clarifai-swift-grpc) | | [Clarifai Rust](https://github.com/Clarifai/clarifai-rust-grpc) | | [Clarifai Go](https://github.com/Clarifai/clarifai-go-grpc) | | [Clarifai C++](https://github.com/Clarifai/clarifai-cpp-grpc) | ## Manually-built Clients \(deprecated\) | Deprecated Clients | | :--- | | [C\#](https://github.com/Clarifai/clarifai-csharp) | | [Java](https://github.com/Clarifai/clarifai-java) | | [JavaScript](https://github.com/Clarifai/clarifai-javascript) \([Reference Docs](https://sdk.clarifai.com/js/latest/index.html)\) | | [PHP](https://github.com/Clarifai/clarifai-php) | ## Client Installation Instructions Here are the installation instructions and the initialization code for some of our most commonly used clients. For information on installing our other clients, please follow the links above. :::tip authorization Learn how to set up authorization with the various API clients [here](https://docs.clarifai.com/clarifai-basics/authentication/authorize). ::: :::warning CORS Policy Error Learn how to resolve CORS errors in your JavaScript application(s) [here](https://docs.clarifai.com/clarifai-basics/authentication/authorize/#cors-policy). ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ```python ############################################################################################## # Installation ############################################################################################## python -m pip install clarifai-grpc ############################################################################################## # Initialize the gRPC-based client to communicate with the Clarifai platform. ############################################################################################## # Import the Clarifai gRPC-based objects needed from clarifai_grpc.channel.clarifai_channel import ClarifaiChannel from clarifai_grpc.grpc.api import resources_pb2, service_pb2, service_pb2_grpc from clarifai_grpc.grpc.api.status import status_pb2, status_code_pb2 # Construct the communications channel channel = ClarifaiChannel.get_grpc_channel() # Construct the V2Stub object for accessing all the Clarifai API functionality stub = service_pb2_grpc.V2Stub(channel) ############################################################################################## # This is where you set up the metadata object that's used to authenticate. # This authorization will be used by every Clarifai API call. # Change the following authorization key to your own credentials # Example: metadata = (('authorization', 'Key ' + 'a123457612345678'),) ############################################################################################## metadata = (('authorization', 'Key ' + 'YOUR_CLARIFAI_PAT_HERE'),) # Or, if you were to use an API Key: # metadata = (('authorization', 'Key ' + 'YOUR_CLARIFAI_API_KEY_HERE'),) # Yes, the word 'Key' appears in addition to the alphanumeric PAT or API Key ############################################################################################## # A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: # user_id (your user id) and app_id (app id that contains the model of interest). # Both of them are specified as string values. ############################################################################################## userDataObject = resources_pb2.UserAppIDSet(user_id='YOUR_USER_ID_HERE', app_id='YOUR_APPLICATION_ID_HERE') ``` ```javascript /////////////////////////////////////////////////////////////////////////////// // Installation /////////////////////////////////////////////////////////////////////////////// npm install clarifai-nodejs-grpc ///////////////////////////////////////////////////////////////////////////////////////////// // Initialize the gRPC-based client to communicate with the Clarifai platform. //////////////////////////////////////////////////////////////////////////////////////////// // Import the Clarifai gRPC-based client const {ClarifaiStub, grpc} = require("clarifai-nodejs-grpc"); // Construct the stub object for accessing all the Clarifai API functionality const stub = ClarifaiStub.grpc(); //////////////////////////////////////////////////////////////////////////////////////////// // This is where you set up the Metadata object that's used to authenticate. // This authorization will be used by every Clarifai API call. // Change the following authorization key to your own credentials // Example: metadata.set("authorization", "Key " + "a123457612345678"); //////////////////////////////////////////////////////////////////////////////////////////// const metadata = new grpc.Metadata(); metadata.set("authorization", "Key " + "YOUR_CLARIFAI_PAT_HERE"); // Or, if you were to use an API Key: // metadata.set("authorization", "Key " + "YOUR_CLARIFAI_API_KEY_HERE"); // Yes, the word 'Key' appears in addition to the alphanumeric PAT or API Key ///////////////////////////////////////////////////////////////////////////////////////////// // A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: // user_id (your user id) and app_id (app id that contains the model of interest). // Both of them are specified as string values. ///////////////////////////////////////////////////////////////////////////////////////////// user_app_id: { "user_id": "YOUR_USER_ID_HERE", "app_id": "YOUR_APPLICATION_ID_HERE" } ``` ```java ////////////////////////////////////////////////////////////////////////////////////////////// // Installation ///////////////////////////////////////////////////////////////////////////////////////////// Via Gradle: repositories { mavenCentral() } dependencies { implementation 'com.clarifai:clarifai-grpc:LATEST_VERSION_HERE' } Via Maven: mavenCentral mavenCentral https://repo1.maven.org/maven2 com.clarifai clarifai-grpc LATEST_VERSION_HERE ///////////////////////////////////////////////////////////////////////////////////////////// // Initialize the gRPC-based client to communicate with the Clarifai platform. //////////////////////////////////////////////////////////////////////////////////////////// // Import the Clarifai gRPC-based objects needed import com.clarifai.grpc.api.*; import com.clarifai.channel.ClarifaiChannel; import com.clarifai.credentials.ClarifaiCallCredentials; ///////////////////////////////////////////////////////////////////////////////////////////// // Construct the communications channel. // Construct the stub object for accessing all the Clarifai API functionality. // Set up the authorization that will be used by every Clarifai API call. ///////////////////////////////////////////////////////////////////////////////////////////// Channel channel = ClarifaiChannel.INSTANCE.getGrpcChannel(); V2Grpc.V2BlockingStub stub = V2Grpc.newBlockingStub(channel) .withCallCredentials(new ClarifaiCallCredentials("YOUR_CLARIFAI_PAT_HERE")); // Or, if you were to use an API Key: //.withCallCredentials(new ClarifaiCallCredentials("YOUR_CLARIFAI_API_KEY_HERE")); ///////////////////////////////////////////////////////////////////////////////////////////// // A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: // user_id (your user id) and app_id (app id that contains the model of interest). // Both of them are specified as string values. ///////////////////////////////////////////////////////////////////////////////////////////// .setUserAppId(UserAppIDSet.newBuilder().setUserId("YOUR_USER_ID_HERE").setAppId("YOUR_APPLICATION_ID_HERE")) ``` ```php ////////////////////////////////////////////////////////////////////////////// // Installation ////////////////////////////////////////////////////////////////////////////// composer require clarifai/clarifai-php-grpc // gRPC for PHP extension is required to use the Clarifai gRPC PHP client // Check how to add gRPC to your PHP installation here: https://github.com/grpc/grpc/tree/master/src/php //////////////////////////////////////////////////////////////////////////////// // Set Clarifai Namespace // - A variety of standard objects are provided in the Clarifai namespace // from the client library. The ones that are necessary depend on the // specific RPC call being made. The namespaces used in the example // code throughout this documentation are included below for reference, // although you likely won't need all of them in your application. //////////////////////////////////////////////////////////////////////////////// // Various data structures that are used in the RPC calls to the Clarifai Platform // These operate as standardization wrappers for various types of data. // Data Types use Clarifai\Api\Image; use Clarifai\Api\Text; use Clarifai\Api\Video; // ML Structures use Clarifai\Api\Concept; use Clarifai\Api\Model; // Wrapper Types use Clarifai\Api\Data; use Clarifai\Api\Input; // Various Request objects. These specify the structure of the actual RPC request between // the client and the platform. use Clarifai\Api\PostModelOutputsRequest; use Clarifai\Api\PostConceptsSearchesRequest; use Clarifai\Api\ConceptQuery; // Output configuration objects use Clarifai\Api\OutputInfo; use Clarifai\Api\OutputConfig; // The request status code object. This contains information on the success or failure of // the API operation. use Clarifai\Api\Status\StatusCode; ///////////////////////////////////////////////////////////////////////////////////////////// // Initialize the gRPC-based client to communicate with the Clarifai platform. //////////////////////////////////////////////////////////////////////////////////////////// // The Clarifai PHP Client repository includes an autoload.php helper file that needs to be included require __DIR__ . '/vendor/autoload.php'; // Enable use of the ClarifaiClient object from the Clarifai namespace use Clarifai\ClarifaiClient; // Construct the initialization object for accessing all the Clarifai API functionality $client = ClarifaiClient::grpc(); //////////////////////////////////////////////////////////////////////////////////////////// // This is where you set up the Metadata object that's used to authenticate. // This authorization will be used by every Clarifai API call. // Change the following authorization key to your own credentials // Example: $metadata = ['Authorization' => ['Key a123457612345678']]; //////////////////////////////////////////////////////////////////////////////////////////// $metadata = ['Authorization' => ['Key YOUR_CLARIFAI_PAT_HERE']]; // Or, if you were to use an API Key: // $metadata = ['Authorization' => ['Key YOUR_CLARIFAI_API_KEY_HERE']]; // Yes, the word 'Key' appears in addition to the alphanumeric PAT or API Key ///////////////////////////////////////////////////////////////////////////////////////////// // A UserAppIDSet object is needed when using a PAT. It contains two pieces of information: // user_id (your user id) and app_id (app id that contains the model of interest). // Both of them are specified as string values. ///////////////////////////////////////////////////////////////////////////////////////////// use Clarifai\Api\UserAppIDSet; $userDataObject = new UserAppIDSet([ 'user_id' => 'YOUR_USER_ID_HERE', 'app_id' => 'YOUR_APPLICATION_ID_HERE' ]); ``` ```csharp /////////////////////////////////////////////////////////////////////////////// // Installation /////////////////////////////////////////////////////////////////////////////// dotnet add package ClarifaiGrpc /////////////////////////////////////////////////////////////////////////////// // Initialize client /////////////////////////////////////////////////////////////////////////////// using System; using System.Collections.Generic; using Clarifai.Api; using Clarifai.Channels; using Grpc.Core; using StatusCode = Clarifai.Api.Status.StatusCode; var client = new V2.V2Client(ClarifaiChannel.Grpc()); var metadata = new Metadata { {"Authorization", "Key {YOUR_PERSONAL_TOKEN}"} }; ``` ## gRPC vs HTTP Channels The Clarifai API offers you an encrypted gRPC channel, as well as an HTTPS+JSON channel for making requests. Why the different options? Learn about the many convenient benefits of using the Clarifai API built on gRPC. ### Why did we build our API on gRPC in the first place? ![grpc vs http clarifai](/img/grpc-vs-http.png) #### Cutting edge performance Clarifai gRPC is built to deliver lightweight microservices. This is one of the keys to Clarifai Amada’s ability to load and balance thousands of different instances of machine learning models, and deliver MLOps at scale. With low latency and high throughput communication, we can deliver high performance machine learning services anywhere you are. Clarifai also uses gRPC to reduce network constraints. Our API messages are serialized using Protobuf, a compact, binary \(though non human-readable\) message format that is always smaller than the equivalent JSON. gRPC Protobuf serializes very quickly on the server and client. gRPC is specifically designed for HTTP/2, a major revision of HTTP that provides significant performance benefits over HTTP. The HTTP/2 protocol is efficient both when sending and receiving messages. HTTP/2 also eliminates head-of-line blocking by allowing multiplexing of multiple HTTP/2 calls over a single TCP connection. #### Clean code and resource management There simply is no formal definition of how to build an HTTP API with JSON, and there is ongoing debate about the best format for URLs, HTTP verbs, and response codes. The gRPC specification is prescriptive about the format a gRPC service must follow, which means that behavior is consistent across platforms and implementations. Resource management is also made easier by virtue of the fact that gRPC also allows clients to specify how long they are willing to wait for an RPC to complete. The deadline is sent to the server, and the server has the ability to “timeout” in-progress gRPC/HTTP/database requests. #### A gift for languages Clarifai is a global, multi-lingual organization. We offer multi-lingual support for our models, and our API is a “polyglot” as well. Clarifai takes advantage of the fact that gRPC tooling supports all popular development languages, and we offer clients in many of the most popular programming languages. Code generation of the client creates a strongly-typed client, and eliminates duplication of messages on the client and server. Clarifai automatically creates RESTful JSON Web APIs from gRPC services. This allows us to support both gRPC and JSON web APIs, without duplicating effort of building separate services for both. #### Streaming capabilities Two-way real-time communication is needed for many promising ML use cases. gRPC offers support for bi-directional streaming, allowing gRPC services to push messages in real-time without polling. All streaming combinations are natively supported when using our gRPC clients: unary \(no streaming\), server to client streaming, client to server streaming and bi-directional streaming. ### When to use the HTTP Channel We recommend using the encrypted gRPC channel for most of our customers in most use cases, but the HTTP+JSON channel does have its advantages: #### Familiarity Working with a RESTful JSON Web API will be familiar to many developers. In these cases, you may want to evaluate the tradeoff between development time and the additional performance offered by the gRPC channel. #### Browser support It's impossible to directly call a gRPC service from a browser today. gRPC uses HTTP/2 features which major browsers currently do not support. #### Human readability HTTP API requests are sent as text and can be read and created by humans. gRPC messages are encoded with Protobuf by default. While serializing Protobuf is more efficient and the payloads are smaller, its binary format isn't human readable. Additional tooling is required to analyze Protobuf payloads and to compose requests by hand. --- ## Node.js SDK # Node.js SDK **Discover the power of the Node.js SDK** [The Clarifai Node.js SDK](https://github.com/Clarifai/clarifai-nodejs) is your gateway for effortlessly integrating AI capabilities into your JavaScript and Node.js applications. Built using TypeScript, this SDK empowers you to easily interact with the Clarifai API using an intuitive and object-oriented approach. With TypeScript's strong typing and modern features, you can automate your AI workflows with minimal code. It’s the ideal tool to bring AI-powered features to your Node.js projects. ## Installation You can install the latest stable version of the Clarifai Node.js SDK by running: ```sh npm install clarifai-nodejs ``` ## Authentication Clarifai uses [Personal Access Tokens](https://docs.clarifai.com/control/authentication/pat) (PATs) to authenticate API requests. You can obtain one from your personal settings page by navigating to the **Security** section. You can export your PAT as an environment variable. Then, import and initialize the API client. Here is how to set your PAT as an environment variable through the terminal: ```cmd export CLARIFAI_PAT=YOUR_PAT_HERE ``` Alternatively, use [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from a `.env` file. This is useful for managing secrets in Node.js applications. ## How to Use Node.js SDK in Next.js Server Components To integrate the Clarifai Node.js SDK in a Next.js App Directory project with server components, you need to explicitly add the [clarifai-nodejs-grpc](https://github.com/Clarifai/clarifai-nodejs-grpc) package to the [serverComponentsExternalPackages](https://nextjs.org/docs/app/api-reference/next-config-js/serverComponentsExternalPackages) configuration in your `next.config.js` file. This ensures that Next.js properly handles the gRPC dependencies required by the Clarifai SDK, allowing you to use AI-powered inference in your server-side components. ```js /** @type {import('next').NextConfig} */ const nextConfig = { experimental: { serverComponentsExternalPackages: ['clarifai-nodejs-grpc'], }, } module.exports = nextConfig ``` ## Minimum System Requirements The Clarifai Node.js package is lightweight and designed to run on any modern system. You’ll need Node.js version 18 or higher to run it. > **Note**: The package supports all major operating systems (Windows, macOS — including Intel and Apple Silicon — and Linux) as long as Node.js 18+ is installed. { const result = response?.[0].data?.conceptsList[0].name ?? "unrecognized"; console.log(result); }) .catch(console.error); ``` ### Using Workflows Using a custom workflow built on [clarifai.com](https://docs.clarifai.com/portal-guide/workflows/) to analyze sentiment of a given image. For list of all available workflows visit [clarifai workflows page](https://clarifai.com/explore/workflows) ```ts import { Input, Workflow } from "clarifai-nodejs"; const input = Input.getInputFromUrl({ inputId: "test-image", imageUrl: "https://samples.clarifai.com/celebrity.jpeg", }); const workflow = new Workflow({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID! }, workflowId: "workflow-238a93", }); workflow .predict({ inputs: [input], }) .then((response) => { const result = response.resultsList[0].outputsList[0].data?.regionsList[0].data ?.conceptsList[0].name ?? "unrecognized"; console.log(result); }) .catch(console.error); ``` ### Listing available apps in an user account On Clarifai, apps act as a central repository for models, datasets, inputs and other resources and information. Checkout how to create apps on [clarifai portal](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/). ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const list = await user .listApps({ pageNo: 1, perPage: 20, params: { sortAscending: true, }, }) .next(); const apps = list.value; console.log(apps); ``` For full usage details, checkout our [API Reference docs](https://docs.clarifai.com/nodejs-sdk/installation-guide/modules) --> --- ## Pagination # Pagination **Learn how to paginate your GET or POST requests** Many of our API endpoints support pagination, a crucial feature when handling large outputs. It helps manage and display results efficiently by breaking them into smaller, manageable batches. You can provide `page` and `per_page` params to the API request, and your results will be split into pages. - `page` — Indicates the page number⁠; defaults to 1. - `per_page` — Indicates the number of results that will be contained in each page; defaults to 128. You can get up to 1,000 results per page. Creating a pagination request may vary depending on whether you're working with a GET or a POST endpoint. :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination.py"; import NodePaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination.js"; import JavaPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination.java"; import CurlPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination.sh"; import JavaScriptPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination.html"; import PHPPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination.php"; import PythonPaginationPost from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/post_pagination.py"; import NodePaginationPost from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/post_pagination.js"; import JavaPaginationPost from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/post_pagination.java"; import CurlPaginationPost from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/post_pagination.sh"; import JavaScriptPaginationPost from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/post_pagination.html"; import PHPPaginationPost from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/post_pagination.php"; import PythonSDKPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination-py-sdk.py"; import NodeSDKPaginationGet from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/pagination-node-sdk.js"; ## GET Endpoints For GET requests, the pagination parameters are included in the query string of the URL. {PythonSDKPaginationGet} {NodeSDKPaginationGet} {PythonPaginationGet} {JavaScriptPaginationGet} {NodePaginationGet} {JavaPaginationGet} {PHPPaginationGet} {CurlPaginationGet} ## POST Endpoints For POST requests, the pagination parameters are included in the request body. {PythonPaginationPost} {JavaScriptPaginationPost} {NodePaginationPost} {JavaPaginationPost} {PHPPaginationPost} {CurlPaginationPost} --- ## Python SDK # Python SDK **Discover the power of the Python SDK** [The Python SDK](https://github.com/Clarifai/clarifai-python/) is designed to streamline your AI workflows, making it easier than ever to interact with the Clarifai API. Whether you're building AI applications or integrating Clarifai into your existing systems, the Python SDK is your go-to tool for effortless API interaction. With an intuitive, object-oriented approach, you can accelerate your AI development with just a few lines of code. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; ## Installation ### Install via PyPI You can install the latest stable version of the Clarifai [Python SDK](https://pypi.org/project/clarifai/) using `pip`: ```python pip install -U clarifai ``` For additional functionality, you can install all the optional libraries mentioned [here](https://github.com/Clarifai/clarifai-python/blob/491d5444f5ae5da234012022e1ba4e83739242a4/setup.py) by running: ```python pip install "clarifai[all]" ``` Once installed, you can confirm the version by running the following command: ```text clarifai --version ``` ### Install from Source (for Development) If you’re contributing to development or need the latest updates, install from source: ```python git clone https://github.com/Clarifai/clarifai-python.git cd clarifai-python python3 -m venv .venv source .venv/bin/activate pip install -e . ``` ## Authentication Clarifai uses [Personal Access Tokens](https://docs.clarifai.com/control/authentication/pat) (PATs) to authenticate API requests. You can obtain one from your personal settings page by navigating to the **Security** section. To set your PAT as an environment variable and use it with the Python SDK, run: export CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE set CLARIFAI_PAT=YOUR_PERSONAL_ACCESS_TOKEN_HERE ## Minimum System Requirements The Clarifai Python package is lightweight and designed to run on most modern systems. No special hardware is required. It will run on standard consumer-grade laptops and desktops. Below are the minimum system requirements for different operating systems. ### Supported Operating Systems - **Windows**: Windows 10 (64-bit) or later - **macOS**: macOS 11.0 (Big Sur) or later, Apple Silicon (M1, M2, M3, …) only > **Note**: Intel-based Macs are not supported - **Linux**: Ubuntu 18.04 LTS or later, or equivalent distributions ### Python Requirements - Python 3.11 – 3.12 (3.12 is supported by default) - Pip 21.0 or higher ### Hardware Requirements - **Processor**: - Windows/Linux: x86_64 or ARM64 (Intel, AMD, or ARM) - macOS: Apple Silicon only (M1, M2, M3, …) - **Memory**: Minimum 2 GB RAM (4 GB or more is recommended) - **Disk Space**: At least 200 MB free storage for installation and dependencies - **GPU**: Not required. However, for more intensive tasks, you may need access to a GPU. We provide various [GPU options](https://docs.clarifai.com/compute/cloud-instances) (e.g., NVIDIA A100, L40S) through our cloud compute services. These resources are managed on our platform, not on your local machine. --- ## Rate Limits # Rate Limits **Learn about our API rate limits** Our API has restrictions that limit the rate at which requests are made. These measures help us maintain the health, fairness, and security of our API. :::info Our default rate limit is **15 requests per second** for all users, regardless of the pricing plan. You need to contact us if you need a customized API rate limit. ::: We control the number of requests that can be made to our API within a given timeframe. This helps to: - **Maintain the stability and reliability of our API** — With rate limits, we ensure our service is not overwhelmed with excessive requests from a single entity, providing smooth and fair access for all our users. - **Better handle increased demand on our infrastructure** — With rate limits, we can handle a sudden surge in API requests and distribute the increased load on our infrastructure more evenly. - **Enhance the security of our API** — With rate limits, we can mitigate the risk of abuse, denial-of-service attacks, and other malicious activities. ## Mitigating Rate Limits You may bump into the API rate limit if you make requests exceeding your quota within the specified time frame. When the limit is reached, the API temporarily halts processing additional requests until a designated time interval has elapsed. The error response could look like this: ```text { "status": { "code": "CONN_THROTTLED", "description": "Making too many requests", "details": "exceeded limit of 15 requests per second", "req_id": "..." }, ... } ``` As you can see above, the response includes the `CONN_THROTTLED` error, whose corresponding [status code](https://docs.clarifai.com/api-guide/advanced-topics/status-codes) is 11005. It’s a best practice to mitigate the rate limit errors in your code to ensure smooth access to the Clarifai API. For example, you can implement a back-off strategy, where you wait for 15 seconds before retrying the requests. Here are HTTP-based code samples on how you could list Clarifai's [model types](https://docs.clarifai.com/api-guide/model/create-get-update-and-delete#list-model-types) and also handle rate limit errors. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonRateLimits from "!!raw-loader!../../../code_snippets/api-guide/others/api-rate-limits.py"; import NodeRateLimits from "!!raw-loader!../../../code_snippets/api-guide/others/api-rate-limits.js"; {PythonRateLimits} {NodeRateLimits} --- ## Status Codes # Status Codes **Understand Clarifai's error codes** All our API endpoints return a status code and a description that gives details of the status. A full list of those status codes is shown below. If a status comes back that you do not see below, please reach out to support@clarifai.com. ## Generic | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 10000 | SUCCESS | Ok | | 10001 | SUCCESS_WARNING_API_DEPRECATED | | | 10002 | SUCCESS_WARNING_CLIENT_DEPRECATED | | | 10010 | MIXED_STATUS | Mixed Success | | 10020 | FAILURE | Failure | | 10030 | TRY_AGAIN | Try the request again | | 10040 | NOT_IMPLEMENTED | | | 10050 | MOVED | Resource moved | ## Clarifai Connection Codes: 11xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 11000 | CONN_ACCOUNT_ISSUES | Account or plan issue, such as expired credentials, account still in trial, or feature not supported in your tier | | 11001 | CONN_TOKEN_INVALID | Invalid authentication token used | | 11002 | CONN_CREDENTIALS_INVALID | Invalid authentication credentials | | 11003 | CONN_EXCEED_HOURLY_LIMIT | Throttle hourly request limit exceeded | | 11004 | CONN_EXCEED_MONTHLY_LIMIT | Throttle monthly request limit exceeded | | 11005 | CONN_THROTTLED | Throttler and billing stuff; making too many requests | | 11006 | CONN_EXCEEDS_LIMITS | Throttler and billing stuff; account limits exceeded | | 11007 | CONN_INSUFFICIENT_SCOPES | API key has insufficient permissions | | 11008 | CONN_KEY_INVALID | Invalid API key or Invalid API key/application pair | | 11009 | CONN_KEY_NOT_FOUND | API key not found | | 11100 | CONN_BAD_REQUEST_FORMAT | Bad request format; multipart form parsing, broken JSON, etc | | 11101 | CONN_DOES_NOT_EXIST | Resource does not exist; when path is bad | | 11102 | CONN_INVALID_REQUEST | Invalid request; something wrong with a header | | 11103 | CONN_METHOD_NOT_ALLOWED | Request method not allowed | | 11104 | CONN_NO_GDPR_CONSENT | No GDPR consent | | 11200 | CONN_AUTH_METHOD_DISABLED | Authentication method is disabled | ## Model/Custom Training Related Codes: 21xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 21100 | MODEL_TRAINED | Custom model has been already trained | | 21101 | MODEL_TRAINING | Custom model is currently training | | 21102 | MODEL_UNTRAINED | Custom model has not yet been trained | | 21103 | MODEL_QUEUED_FOR_TRAINING | Custom model is currently in queue for training, waiting on inputs to process. | | 21104 | MODEL_UPLOADING | Model uploading inputs | | 21105 | MODEL_UPLOADING_FAILED | Model uploading inputs failed | | 21106 | MODEL_TRAINING_FAILED | Generic error message for any type of model training error | | 21110 | MODEL_TRAINING_NO_DATA | Custom model training had no data | | 21111 | MODEL_TRAINING_NO_POSITIVES | Custom model training had no positive examples| | 21112 | MODEL_TRAINING_ONE_VS_N_SINGLE_CLASS | Custom model training was ONE_VS_N but with a single class | | 21113 | MODEL_TRAINING_TIMED_OUT | Training took longer than hard coded timeouts; contact support@clarifai.com if this continues to happen when creating new versions of your model | | 21114 | MODEL_TRAINING_WAITING_ERROR | Training got error waiting on your inputs to process, please contact support@clarifai.com | | 21115 | MODEL_TRAINING_UNKNOWN_ERROR | Training threw an unknown exception, please contact support@clarifai.com. | | 21116 | MODEL_TRAINING_MSG_REDELIVER | Training request was unexpectedly redelivered, contact support@clarifai.com if this continues to happen. | | 21117 | MODEL_TRAINING_INSUFFICIENT_DATA | Training got error due to insufficient labelled data | | 21118 | MODEL_TRAINING_INVALID_PARAMS | Invalid parameters during model training | | 21119 | MODEL_TRAINING_INVALID_DATA_TOLERANCE_EXCEEDED | Training is stopped because too much data was dropped | | 21150 | MODEL_MODIFY_SUCCESS | Model modification success | | 21151 | MODEL_MODIFY_PENDING | Model modification pending | | 21152 | MODEL_MODIFY_FAILED | Model modification failed | | 21200 | MODEL_DOES_NOT_EXIST | Model does not exist | | 21201 | MODEL_PERMISSION_DENIED | Model permission denied | | 21202 | MODEL_INVALID_ARGUMENT | Invalid model argument | | 21203 | MODEL_INVALID_REQUEST | Invalid model request | ## Model Evaluation Codes: 213xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 21300 | MODEL_EVALUATED | Model was successfully evaluated | | 21301 |MODEL_EVALUATING | Model is evaluating | | 21302 |MODEL_NOT_EVALUATED | Model is not yet evaluated | | 21303 | MODEL_QUEUED_FOR_EVALUATION | Model is queued for evaluation | | 21310 | MODEL_EVALUATION_TIMED_OUT | Model evaluation timed out; evaluation took longer than hard coded timeouts | | 21311 | MODEL_EVALUATION_WAITING_ERROR | Model evaluation timed out waiting on inputs to process | | 21312 | MODEL_EVALUATION_UNKNOWN_ERROR | Model evaluation unknown internal error | | 21313 | MODEL_PREDICTION_FAILED | Model prediction failed | | 21314 | MODEL_EVALUATION_MSG_REDELIVER | Evaluation message was redelivered | | 21315 | MODEL_EVALUATION_NEED_LABELS | Model evaluation failed because there are not enough annotated inputs. Please ensure there are at least 2 concepts in your model before evaluating | | 21316 | MODEL_EVALUATION_NEED_INPUTS | Model evaluation failed because there are not enough labeled inputs. Please ensure there are at least 5 labeled inputs per concept before evaluating | | 21317 | MODEL_EVALUATION_FAILED | Generic error code for evaluation failure | | - | -| Status codes through 21319 (inclusive) reserved for model evaluation *errors* (per logic in clients)| ## Model Deployment Codes: 2135x | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 21350 | MODEL_DEPLOYMENT_FAILED | Used when inference coordinator failed to deploy spire and throws an error | | 21351 | MODEL_DEPLOYING | Used when calling the inference coordinator to deploy a spire | | 21352 | MODEL_QUEUED_FOR_DEPLOYMENT | Used when training is completed | | 21353 | MODEL_NOT_DEPLOYED | Used when model spire deployment is manually taken down or due to inactivity | |21354 | MODEL_BUSY_PLEASE_RETRY | The model is currently overloaded with active requests. Retry your request after a short delay| ## Model Reference Codes: 214xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 21400 | MODEL_REFERENCE_INVALID_ARGUMENT | Used when a model reference field is not set properly | | 21420 | MODEL_EXAMPLE_INPUT_INVALID_ARGUMENT | Used when a model example input field is not set properly | ## Workflow Related Codes: 22xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 22001 | WORKFLOW_NO_MATCHING_INPUT | Workflow does not have specified input model | | 22002 | WORKFLOW_REQUIRE_TRAINED_MODEL | New model in workflow needs to be trained | | 22100 | WORKFLOW_DUPLICATE | Duplicate URL in your application. Check the documentation to allow duplications | | 22101 | WORKFLOW_UNSUPPORTED_FORMAT | Workflow format unsupported | | 22102 | WORKFLOW_DOES_NOT_EXIST | Workflow does not exist | | 22103 | WORKFLOW_PERMISSION_DENIED | Workflow permission denied | | 22104 | WORKFLOW_INVALID_ARGUMENT | Workflow invalid argument; error in the request somewhere | | 22105 | WORKFLOW_INVALID_RECIPE | | | 22106 | WORKFLOW_INVALID_TEMPLATE | Template workflow is invalid | | 22107 | WORKFLOW_INVALID_GRAPH | Workflow graph is invalid | | 22108 | WORKFLOW_INTERNAL_FAILURE | | | 22150 | WORKFLOW_MODIFY_SUCCESS | Workflow modification success | | 22151 | WORKFLOW_MODIFY_PENDING | Workflow modification pending | | 22152 | WORKFLOW_MODIFY_FAILED | Workflow modification failed | | 22153 | WORKFLOW_REINDEX_FAILED | Workflow reindexing failed | | 22999 | WORKFLOW_INVALID_REQUEST | Invalid request; error in the request somewhere | ## Concept Related Codes: 23xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 23150 | CONCEPT_MODIFY_SUCCESS | Concept modification success | | 23151 | CONCEPT_MODIFY_PENDING | Concept modification pending | | 23152 | CONCEPT_MODIFY_FAILED | Concept modification failed | ## Annotation Related Codes: 24xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 24150 | ANNOTATION_SUCCESS | Annotation success | | 24151 | ANNOTATION_PENDING | Annotation pending | | 24152 | ANNOTATION_FAILED | Annotation failed; check URL | | 24153 | | Annotation in progress | | 24154 |ANNOTATION_UNKNOWN_STATUS | Annotation unknown status | | 24155 | ANNOTATION_INVALID_ARGUMENT | Annotation invalid argument | | 24156 | ANNOTATION_PERMISSION_DENIED | Permission to annotation denied | | 24157 | ANNOTATION_AWAITING_REVIEW | Annotation still waiting for review | | 24158 | ANNOTATION_REVIEW_DENIED | Annotation review denied | | 24159 | ANNOTATION_AWAITING_CONSENSUS_REVIEW | Annotation still waiting for consensus review | | 24250 | ANNOTATION_MODIFY_SUCCESS | Annotation modification success | | 24251 | ANNOTATION_MODIFY_PENDING | Annotation modification pending | | 24252 | ANNOTATION_MODIFY_FAILED | Annotation modification failed | ## Metadata Related Codes: 249xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 24900 | METADATA_INVALID_PATCH_ARGUMENTS | | | 24901 | METADATA_PARSING_ISSUE | | | 24902 | METADATA_MANIPULATION_ISSUE | | ## Training Service Related Codes: 25xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 25000 | TRAINER_JOB_STATE_NONE | Custom Trainer unknown internal error | | 25001 | TRAINER_JOB_STATE_QUEUED | | | 25002 | TRAINER_JOB_STATE_RUNNING | | | 25003 | TRAINER_JOB_STATE_COMPLETE | | | 25004 | TRAINER_JOB_STATE_ERROR | Custom Trainer failed to retrieve data or train | ## Data Dump Related Codes: 251xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 25150 | DATA_DUMP_SUCCESS | | | 25151 | DATA_DUMP_PENDING | | | 25152 | DATA_DUMP_FAILED | | | 25153 | DATA_DUMP_IN_PROGRESS | | | 25154 | DATA_DUMP_NO_DATA | | | 25155 | DATA_DUMP_UNEXPECTED_ERROR | | | 25170 | DATA_DUMP_EXPORT_SUCCESS | | | 25171 | DATA_DUMP_EXPORT_PENDING | | | 25172 | DATA_DUMP_EXPORT_FAILED | | | 25173 | DATA_DUMP_EXPORT_IN_PROGRESS | | | 25174 | DATA_DUMP_EXPORT_UNEXPECTED_ERROR | | ## Duplicate Related Codes: 252xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 25200 | APP_DUPLICATION_SUCCESS | | | 25201 | APP_DUPLICATION_FAILED | | | 25202 | APP_DUPLICATION_PENDING | | | 25203 | APP_DUPLICATION_IN_PROGRESS | | | 25204 | APP_DUPLICATION_INVALID_REQUEST | | ## Module Related Codes: 253xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 25300 | MODULE_DOES_NOT_EXIST | | | 25301 | MODULE_PERMISSION_DENIED | | | 25302 | MODULE_INVALID_ARGUMENT | | | 25303 | MODULE_INVALID_REQUEST | | ## Bulk Operation Related Codes: 254xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 25400 | BULK_OPERATION_SUCCESS | | | 25401 | BULK_OPERATION_FAILED | | | 25402 | BULK_OPERATION_PENDING | | | 25403 | BULK_OPERATION_IN_PROGRESS | | | 25404 | BULK_OPERATION_INVALID_REQUEST | | | 25405 | BULK_OPERATION_CANCELLED | | | 25406 | BULK_OPERATION_UNEXPECTED_ERROR | | | 25407 | BULK_OPERATION_DELETED | | ## Input: Image Related Codes: 30xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 30000 | INPUT_SUCCESS (_Previously: INPUT_DOWNLOAD_SUCCESS_) | Download complete | | 30001 | INPUT_PENDING (_Previously: INPUT_DOWNLOAD_PENDING_) | Download pending; when things are async, this is the default status | | 30002 | INPUT_FAILED (_Previously: INPUT_DOWNLOAD_FAILED_) | Any type of error downloading and processing; download failed or we could not process it. Check URL or bytes you send in the request | | 30003 | INPUT_IN_PROGRESS (_Previously: INPUT_DOWNLOAD_IN_PROGRESS_) | Download in progress | | 30004 | INPUT_STATUS_UPDATE_FAILED | | | 30005 | INPUT_DELETE_FAILED | | | 30100 | INPUT_DUPLICATE | Duplicate URL in your application. Check the documentation to allow duplications. | | 30101 | INPUT_UNSUPPORTED_FORMAT | Input image format unsupported | | 30102 | INPUT_DOES_NOT_EXIST | Input does not exist | | 30103 | INPUT_PERMISSION_DENIED | Input permission denied | | 30104 | INPUT_INVALID_ARGUMENT | Input invalid argument | | 30105 | INPUT_OVER_LIMIT | Input image is larger than the allowed limit | | 30106 | INPUT_INVALID_URL | Input image URL invalid | | 30200 | INPUT_MODIFY_SUCCESS | Input image modification success | | 30201 | INPUT_MODIFY_PENDING | Input image modification pending | | 30203 | INPUT_MODIFY_FAILED | Input image modification failed | | 30210 | INPUT_STORAGE_HOST_FAILED | | | 30300 | ALL_INPUT_INVALID_BYTES | Input image decoding failed. Check URLs and bytes sent | | 30400 | INPUT_CLUSTER_SUCCESS | | | 30401 | INPUT_CLUSTER_PENDING | | | 30402 | INPUT_CLUSTER_FAILED | | | 30403 | INPUT_CLUSTER_IN_PROGRESS | | | 30500 | INPUT_REINDEX_SUCCESS | | | 30501 | INPUT_REINDEX_PENDING | | | 30502 | INPUT_REINDEX_FAILED | | | 30503 | INPUT_REINDEX_IN_PROGRESS | | ## Input: Video Related Codes: 31xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 31000 | INPUT_VIDEO_DOWNLOAD_SUCCESS | Download complete | | 31001 | INPUT_VIDEO_DOWNLOAD_PENDING | Download pending | | 31002 | INPUT_VIDEO_DOWNLOAD_FAILED | Download failed or we could not process it. Check URL or bytes you sent in the request. | | 31100 | INPUT_VIDEO_DUPLICATE | Duplicate URL in your application. Check the documentation to allow duplications. | | 31101 | INPUT_VIDEO_UNSUPPORTED_FORMAT | Input video format unsupported | | 31102 | INPUT_VIDEO_DOES_NOT_EXIST | Input does not exist | | 31103 | INPUT_VIDEO_PERMISSION_DENIED | Input permission denied | | 31104 | INPUT_VIDEO_INVALID_ARGUMENT | Input invalid argument | | 31105 | INPUT_VIDEO_OVER_LIMIT | Input video is larger the allowed limit | | 31106 | INPUT_VIDEO_INVALID_URL | Input video URL invalid | | 31200 | INPUT_VIDEO_MODIFY_SUCCESS | Input video modification success | | 31201 | INPUT_VIDEO_MODIFY_PENDING | Input video modification pending | | 31203 | INPUT_VIDEO_MODIFY_FAILED | Input video modification failed | | 31210 | INPUT_VIDEO_STORAGE_HOST_FAILED | | | 31300 | ALL_INPUT_VIDEOS_INVALID_BYTES | Input video decoding failed. Check URLs and bytes sent | ## Input Request Codes: 39xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 39996 | INPUT_CONNECTION_FAILED | Connection attempts to the input URL failed | | 39997 | REQUEST_DISABLED_FOR_MAINTENANCE | Sorry, this type of request has been disabled for maintenance. Please try again in a few hours. | | 39998 | INPUT_WRITES_DISABLED_FOR_MAINTENANCE | Input writes are disabled for maintenance. Please try again in a few hours. | | 39999 | INPUT_INVALID_REQUEST | Invalid input request | ## API Formatting Codes: 4000x | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 40001 | PREDICT_INVALID_REQUEST | Invalid request | | 40002 | SEARCH_INVALID_REQUEST | Invalid search request | | 40003 | CONCEPTS_INVALID_REQUEST | Invalid request | | 40004 | STATS_INVALID_REQUEST | | ## Other Related: 400xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 40010 | DATABASE_DUPLICATE_KEY | Object has a duplicate ID; another object with same ID already exist | | 40011 | DATABASE_STATEMENT_TIMEOUT | | | 40012 | DATABASE_INVALID_ROWS_AFFECTED | | | 40013 | DATABASE_DEADLOCK_DETECTED | | | 40014 | DATABASE_FAIL_TASK | | | 40015 | DATABASE_FAIL_TO_GET_CONNECTIONS | | | 40016 | DATABASE_TOO_MANY_CLIENTS | | | 40017 | DATABASE_CONSTRAINT_VIOLATED | Object violates a constraint. Try again with different values for the fields | | 40019 | | The requested operation is currently processing for this app | | 40020 | ASYNC_WORKER_MULTI_ERRORS | | | 40030 | RPC_REQUEST_QUEUE_FULL | Sorry, the server is too busy at the moment. Please try again later | | 40031 | RPC_SERVER_UNAVAILABLE | Sorry, the server is unavailable at the moment. Please try again later | | 40032 | RPC_REQUEST_TIMEOUT | Sorry, your request has timed out. Please try your request again | | 40033 | RPC_MAX_MESSAGE_SIZE_EXCEEDED | Sorry, the request sent is larger than the allowed limit. Please contact support@clarifai.com | | 40034 | EXTERNAL_CONNECTION_ERROR | Could not connect to external services | | 40035 | RPC_CANCELED | | | 40036 | RPC_UNKNOWN_METHOD | | | 40037 | REQUEST_CANCELED_BY_USER | | ## Queue Related Error Codes: 41xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 41000 | QUEUE_CONN_ERROR | Servers are busy. Please try again later | | 41002 | QUEUE_CLOSE_REQUEST_TIMEOUT | | | 41003 | QUEUE_CONN_CLOSED | | | 41004 | QUEUE_PUBLISH_ACK_TIMEOUT | | | 41005 | QUEUE_PUBLISH_ERROR | | | 41006 | QUEUE_SUBSCRIPTION_TIMEOUT | | | 41007 | QUEUE_SUBSCRIPTION_ERROR | | | 41008 | QUEUE_MARSHALLING_FAILED | | | 41009 | QUEUE_UNMARSHALLING_FAILED | | | 41010 | QUEUE_MAX_MSG_REDELIVERY_EXCEEDED | | | 41011 | QUEUE_ACK_FAILURE | | ## SQS Related Error Codes: 411xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 41100 | SQS_OVERLIMIT | | | 41101 | SQS_INVALID_RECEIPT_HANDLE | | | 41102 | SQS_UNKNOWN | | ## Visualization Error Codes: 42xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 42000 | | Visualization succeeded | | 42001 | | Visualization is pending | | 42002 | | Visualization failed | | 42003 | | Visualization invalid request | | 42004 | | Missing application visualization | | 42005 | | Too many URLs to visualize | | 42006 | | There is not inputs in app | ## Search Related Error Codes: 430xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 43001 | SEARCH_INTERNAL_FAILURE | Search internal issue | | 43002 | SEARCH_PROJECTION_FAILURE | Search projection failure | | 43003 | SEARCH_PREDICTION_FAILURE | Search prediction failure | | 43004 | SEARCH_BY_NOT_FULLY_INDEXED_INPUT | Can only search by a fully indexed input | | 43005 | SAVED_SEARCH_MODIFY_FAILED | | | 43040 | CLUSTER_INTERNAL_FAILURE | | ## Workflow Evaluation Error Codes: 431xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 43100 | EVALUATION_QUEUED | | | 43101 | EVALUATION_IN_PROGRESS | | | 43102 | EVALUATION_SUCCESS | | | 43103 | EVALUATION_FAILED_TO_RETRIEVE_DATA | | | 43104 | EVALUATION_INVALID_ARGUMENT | | | 43105 | EVALUATION_FAILED | | | 43106 | EVALUATION_PENDING | | | 43107 | EVALUATION_TIMED_OUT | | | 43108 | EVALUATION_UNEXPECTED_ERROR | | | 43109 | EVALUATION_MIXED | | ## Stripe Error Code: 44xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 44001 | STRIPE_EVENT_ERROR | | ## Redis/Cache Error Codes: 45xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 45001 | CACHE_MISS | | | 45002 | REDIS_SCRIPT_EXITED_WITH_FAILURE | | | 45003 | REDIS_STREAM_ERR | | | 45004 | REDIS_NO_CONSUMERS | | | 45005 | REDIS_STREAM_BACKOFF | | ## Sift Science Error Codes: 46xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 46001 | SIGNUP_EVENT_ERROR | | | 46002 | SIGNUP_FLAGGED | Signup not permitted | | 46003 | FILETYPE_UNSUPPORTED | Filetype not supported | ## Application Counts Related Error Codes: 470xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 47001 | APP_COUNT_INVALID_MESSAGE | | | 47002 | APP_COUNT_UPDATE_INCREMENT_FAILED | | | 47003 | APP_COUNT_REBUILD_FAILED | | | 47004 | APP_COUNT_INTERNAL_FAILURE | | ## Media Processor Related Error Codes: 471xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 47101 | MP_DOWNLOAD_ERROR | | | 47102 | MP_RESOLVE_DNS_ERROR | | | 47103 | MP_DOWNLOAD_MAX_SIZE_EXCEEDED_ERROR | | | 47104 | MP_IMAGE_DECODE_ERROR | | | 47105 | MP_INVALID_ARGUMENT | | | 47106 | MP_IMAGE_PROCESSING_ERROR | | ## DataTier Related Error Code: 472xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 47201 | DATATIER_CONN_ERROR | | ## User Legal Consent Status Related Code: 50xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 50001 | USER_CONSENT_FACE | | ## Workers Codes: 51xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 51000 | WORKER_MISSING | | | 51001 | WORKER_ACTIVE | | | 51002 | WORKER_INACTIVE | | ## Collectors Codes: 52xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 52000 | COLLECTOR_MISSING | | | 52001 | COLLECTOR_ACTIVE | | | 52002 | COLLECTOR_INACTIVE | | | 52003 | COLLECTOR_POST_INPUT_FAILED | | ## SSO Code: 53xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 53001 | SSO_IDENTITY_PROVIDER_DOES_NOT_EXIST | | ## Tasks Codes: 54xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 54001 | TASK_IN_PROGRESS | Auto-annotation or manual annotation task was created and is in progress | | 54002 | TASK_DONE | Manual annotation task is completed (note that auto-annotation tasks are never completed) | | 54003 | TASK_WONT_DO | Manual annotation task is marked as abandoned | | 54005 | TASK_FAILED | Auto-annotation or manual annotation has failed to run | |54006 | TASK_IDLE | Auto-annotation task job has finished processing its last batch and is waiting for more dataset assets | | 54100 | TASK_CONFLICT | The task operation is in conflict with the current state of the server | | 54101 | TASK_NOT_IMPLEMENTED | Certain task-related scenarios are not implemented | | 54102 | TASK_MISSING | Task was not found | ## Label Order Related Status Code: 55xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 55001 | LABEL_ORDER_PENDING | | | 55002 | LABEL_ORDER_IN_PROGRESS | | | 55003 | LABEL_ORDER_SUCCESS | | | 55004 | LABEL_ORDER_CANCELED | | ## License Related Status Codes: 600xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 60000 | LICENSE_ACTIVE | License is active | | 60001 | LICENSE_DOES_NOT_EXIST | License does not exist | | 60002 | LICENSE_NEED_UPDATE | License needs update | | 60003 | LICENSE_EXPIRED | License has expired | | 60004 | LICENSE_REVOKED | License has been revoked | | 60005 | LICENSE_DELETED | Hidden state not reflected to users | | 60006 | LICENSE_VOLUME_EXCEEDED | Exceeded volume limit on license | ## Password Related Status Codes: 610xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 61000 | PASSWORD_VALIDATION_SUCCESS | | | 61001 | PASSWORD_VALIDATION_FAILED | | | 61002 | PASSWORDPOLICY_INVALID_ARGUMENT | | ## Feature Flags Status Codes: 620xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 62000 | FEATUREFLAG_CONFIG_NOT_FOUND | | | 62001 | FEATUREFLAG_INVALID_ARGUMENT | | | 62002 | FEATUREFLAG_BLOCKED | | ## Maintenance Status Codes: 630xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 63000 | MAINTENANCE_SUCCESS | | | 63001 | MAINTENANCE_FAILED | | ## Datasets Status Codes: 64xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 64005 | DATASET_VERSION_PENDING | The dataset version is pending to be processed | | 64010 | DATASET_VERSION_IN_PROGRESS | The dataset version is currently being processed | | 64015 | DATASET_VERSION_READY | The dataset version is ready to be used | | 64020 | DATASET_VERSION_FAILURE | An error occurred during the dataset version processing | | 64025 | DATASET_VERSION_UNEXPECTED_ERROR | An unexpected error occurred during the dataset version processing | | 64030 | DATASET_VERSION_CONFLICT | An alteration to dataset version would create a conflict | | 64100 | DATASET_INPUT_SUCCESS | The dataset input was successfully added | | 64101 | DATASET_INPUT_DUPLICATE | The dataset input is a duplicate | ## Generic Job Status Codes: 640xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 64000 | JOB_QUEUED | | | 64001 | JOB_RUNNING | | | 64002 | JOB_COMPLETED | | | 64003 | JOB_FAILED | | | 64004 | JOB_CANCELLED | | | 64006 | JOB_UNEXPECTED_ERROR | | ## Auth Issues Codes | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 65000 | AUTH_MISSING_IDP_ASSOC | | | 66000 | LIST_OBJECTS_FAILED | | | 67000 | ARCHIVE_EXTRACT_FAILED | | ## Multipart Uploading Status Codes: 680xx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 68000 | UPLOAD_IN_PROGRESS | | | 68001 | UPLOAD_DONE | | | 68002 | UPLOAD_FAILED | | | 68003 | UPLOAD_UNEXPECTED_ERROR | | | 68004 | UPLOAD_EXPIRED | | ## Billing Related Issues: 69xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 69000 | BILLING_INVALID_INFO | | | | | | ## Internal Issues Codes: 98xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 98004 | INTERNAL_SERVER_ISSUE | | | 98005 | INTERNAL_FETCHING_ISSUE | | | 98006 | INTERNAL_DATABASE_ISSUE | | | 98009 | INTERNAL_UNEXPECTED_TIMEOUT | | | 98010 | INTERNAL_UNEXPECTED_V1 | | | 98011 | INTERNAL_UNEXPECTED_PANIC | | | 98012 | INTERNAL_UNEXPECTED_SPIRE | | | 98013 | INTERNAL_REDIS_UNAVAILABLE | | | 98014 | INTERNAL_RESOURCE_EXHAUSTED | | | 98015 | INTERNAL_REDIS_UNCATEGORIZED | | | 98016 | INTERNAL_AWS_UNCATEGORIZED | | | 98017 | INTERNAL_AZURE_UNCATEGORIZED | | ## Uncategorized Codes: 99xxx | CODE | ERROR | DESCRIPTION | | :--- | :--- |:--- | | 99001 | CONN_UNCATEGORIZED | | | 99002 | MODEL_UNCATEGORIZED | | | 99003 | INPUT_UNCATEGORIZED | | | 99004 | ANNOTATION_UNCATEGORIZED | | | 99005 | BILLING_UNCATEGORIZED | | | 99009 | INTERNAL_UNCATEGORIZED | | --- ## API References # API References **Learn about our API references** import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Node.js API Reference # Node.js API Reference **Clarifai Node.js API Reference** This is the API Reference documentation extracted from the [source code](https://github.com/Clarifai/clarifai-nodejs/tree/main/src/client). ## App App is a class that provides access to Clarifai API endpoints related to App information. This is its hierarchy: - `Lister` ↳ **`App`** ### Constructor • **new App**(`config`): `App` Initializes an App object. **Example** ```ts import { App } from "clarifai-nodejs"; export const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `config` | [`AppConfig`](#appconfig) | The configuration object for the App. | **Returns** `App` **Overrides** Lister.constructor **Defined in** [src/client/app.ts:102](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L102) ### Properties #### appInfo • `Private` **appInfo**: `App` **Defined in** [src/client/app.ts:86](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L86) #### info • **info**: `AsObject` **Defined in** [src/client/app.ts:87](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L87) ### Methods #### createDataset ▸ **createDataset**(`«destructured»`): `Promise`\ Creates a dataset for the app. **Example** ```ts import { Input, App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const dataset = await app.createDataset({ datasetId: "dog-image-collection", }); // Dataset is created, now let's build an image input that uses the new dataset id const inputProto = Input.getInputFromUrl({ datasetId: dataset.id, inputId: "dog-tiff", imageUrl: "https://samples.clarifai.com/dog.tiff", labels: ["dog"], geoInfo: { latitude: 40, longitude: -30, }, metadata: { Breed: "Saint Bernard" }, }); const input = new Input({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); // upload the input by using instance of the Input class // this input will be stored under the newly created dataset const inputJobId = await input.uploadInputs({ inputs: [inputProto], }); console.log(inputJobId); // job id of the input upload ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `datasetId` | `string` | | › `params?` | [`CreateDatasetParam`](#createdatasetparam) | **Returns** `Promise`\ A Dataset object for the specified dataset ID. **Defined in** [src/client/app.ts:429](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L429) #### createModel ▸ **createModel**(`«destructured»`): `Promise`\ Creates a model for the app. **Example** ```ts import { Model, App } from "clarifai-nodejs"; import { ModelVersion, OutputInfo, } from "clarifai-nodejs-grpc/proto/clarifai/api/resources_pb"; import { Struct } from "google-protobuf/google/protobuf/struct_pb.js"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); // Creating a new image crop model const newModelObject = await app.createModel({ modelId: "margin-100-image-cropper", params: { modelTypeId: "image-crop", description: "Custom crop model with 100px margin", }, }); // Initializing the newly created model const model = new Model({ modelId: newModelObject.id, authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); // Creating a GRPC compatible outputInfo object with custom margin parameters const outputInfo = new OutputInfo().setParams( Struct.fromJavaScript({ margin: 1.5 }), ); // GRPC compatible ModelVersion object with previously created output info config const modelVersion = new ModelVersion() .setDescription("Setting output info margin parameters to 1.5") .setOutputInfo(outputInfo); // Creating a new version of the model with previously created output info config const modelObjectWithVersion = await model.createVersion(modelVersion); console.log(modelObjectWithVersion); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `modelId` | `string` | | › `params?` | [`CreateModelParam`](#createmodelparam) | **Returns** `Promise`\ A Model object for the specified model ID. **Defined in** [src/client/app.ts:473](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L473) #### createModule ▸ **createModule**(`«destructured»`): `Promise`\ Creates a module for the app. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const module = await app.createModule({ moduleId: "new-module", description: "New module", }); console.log(module); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `description` | `string` | | › `moduleId` | `string` | **Returns** `Promise`\ A Module object for the specified module ID. **Defined in** [src/client/app.ts:514](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L514) #### createWorkflow ▸ **createWorkflow**(`«destructured»`): `Promise`\ Creates a workflow for the app. **Example** ```ts import { App } from "clarifai-nodejs"; import path from "path"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const workflowFile = path.resolve(__dirname, "workflow/moderation.yml"); const workflow = await app.createWorkflow({ configFilePath: workflowFile }); console.log(workflow); /** * Workflow config file in the path `workflow/moderation.yml`: */ /* workflow: id: test-mbmn nodes: - id: detector model: modelId: face-detection modelVersionId: 45fb9a671625463fa646c3523a3087d5 - id: cropper model: modelId: margin-110-image-crop modelVersionId: b9987421b40a46649566826ef9325303 nodeInputs: - nodeId: detector - id: face-sentiment model: modelId: face-sentiment-recognition modelVersionId: a5d7776f0c064a41b48c3ce039049f65 nodeInputs: - nodeId: cropper - id: moderation model: modelId: moderation-recognition modelVersionId: 7cde8e92896340b0998b8260d47f1502 */ ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `configFilePath` | `string` | `undefined` | | › `display?` | `boolean` | `true` | | › `generateNewId?` | `boolean` | `false` | **Returns** `Promise`\ A Workflow object for the specified workflow config. **Defined in** [src/client/app.ts:552](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L552) #### dataset ▸ **dataset**(`dataset_id`): `Promise`\ Returns a Dataset object for the existing dataset ID. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const dataset = await app.dataset({ datasetId: "dataset-id", }); console.log(dataset); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `dataset_id` | `Object` | The dataset ID for the dataset to interact with. | | `dataset_id.datasetId` | `string` | - | **Returns** `Promise`\ A Dataset object for the existing dataset ID. **Defined in** [src/client/app.ts:771](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L771) #### deleteDataset ▸ **deleteDataset**(`datasetId`): `Promise`\ Deletes a dataset for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); await app.deleteDataset({ datasetId: "dataset-id" }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `datasetId` | `Object` | The dataset ID for the app to delete. | | `datasetId.datasetId` | `string` | - | **Returns** `Promise`\ **Defined in** [src/client/app.ts:800](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L800) #### deleteModel ▸ **deleteModel**(`modelId`): `Promise`\ Deletes a model for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); await app.deleteModel({ modelId: "modelId" }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `modelId` | `Object` | The model ID for the model to delete. | | `modelId.modelId` | `string` | - | **Returns** `Promise`\ **Defined in** [src/client/app.ts:825](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L825) #### deleteModule ▸ **deleteModule**(`moduleId`): `Promise`\ Deletes a module for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); await app.deleteModule({ moduleId: "moduleId" }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `moduleId` | `Object` | The module ID for the module to delete. | | `moduleId.moduleId` | `string` | - | **Returns** `Promise`\ **Defined in** [src/client/app.ts:875](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L875) #### deleteWorkflow ▸ **deleteWorkflow**(`workflowId`): `Promise`\ Deletes a workflow for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); await app.deleteWorkflow({ workflowId: "workflowId" }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `workflowId` | `Object` | The workflow ID for the workflow to delete. | | `workflowId.workflowId` | `string` | - | **Returns** `Promise`\ **Defined in** [src/client/app.ts:850](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L850) #### listConcepts ▸ **listConcepts**(`«destructured»?`): `AsyncGenerator`\ Lists all the concepts for the app. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const list = await app.listConcepts().next(); const concepts = list.value; console.log(concepts); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Yields** Concepts in the app. **Defined in** [src/client/app.ts:391](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L391) #### listDataSets ▸ **listDataSets**(`«destructured»?`): `AsyncGenerator`\ Lists all the datasets for the app. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const list = await app.listDataSets().next(); const datasets = list.value; console.log(datasets); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `params?` | [`ListDatasetsParam`](#listdatasetsparam) | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Yields** Dataset - Dataset objects for the datasets in the app. **Notes** Defaults to 16 per page **Defined in** [src/client/app.ts:136](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L136) #### listInstalledModuleVersions ▸ **listInstalledModuleVersions**(`«destructured»?`): `AsyncGenerator`\ Lists all installed module versions in the app. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const list = await app.listInstalledModuleVersions().next(); const moduleVersions = list.value; console.log(moduleVersions); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `params?` | [`ListInstalledModuleVersionsParam`](#listinstalledmoduleversionsparam) | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Yields** Module - Module objects for the installed module versions in the app. **Notes** Defaults to 16 per page **Defined in** [src/client/app.ts:342](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L342) #### listModels ▸ **listModels**(`«destructured»?`): `AsyncGenerator`\ Lists all the available models for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const list = await app.listModels().next(); const models = list.value; console.log(models); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `{}` | | › `onlyInApp?` | `boolean` | `true` | | › `pageNo?` | `number` | `undefined` | | › `params?` | [`ListModelsParam`](#listmodelsparam) | `{}` | | › `perPage?` | `number` | `undefined` | **Returns** `AsyncGenerator`\ **Notes** Defaults to 16 per page **Defined in** [src/client/app.ts:180](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L180) #### listModules ▸ **listModules**(`«destructured»?`): `AsyncGenerator`\ Lists all the available modules for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const list = await app.listModules().next(); const modules = list.value; console.log(modules); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `onlyInApp?` | `boolean` | | › `pageNo?` | `number` | | › `params?` | [`ListModulesParam`](#listmodulesparam) | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Yields** Module - Module objects for the modules in the app. **Notes** Defaults to 16 per page **Defined in** [src/client/app.ts:291](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L291) #### listTrainableModelTypes ▸ **listTrainableModelTypes**(): `string`[] **Returns** `string`[] **Defined in** [src/client/app.ts:415](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L415) #### listWorkflows ▸ **listWorkflows**(`«destructured»?`): `AsyncGenerator`\ Lists all the available workflows for the user. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const list = await app.listWorkflows().next(); const workflows = list.value; console.log(workflows); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `{}` | | › `onlyInApp?` | `boolean` | `true` | | › `pageNo?` | `number` | `undefined` | | › `params?` | [`ListWorkflowsParam`](#listworkflowsparam) | `{}` | | › `perPage?` | `number` | `undefined` | **Returns** `AsyncGenerator`\ **Yields** Workflow - Workflow objects for the workflows in the app. **Notes** Defaults to 16 per page **Defined in** [src/client/app.ts:238](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L238) #### model ▸ **model**(`«destructured»`): `Promise`\ Returns a Model object for the existing model ID. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const model = await app.model({ modelId: "custom-crop-model", modelVersionId: "0.0.1", }); console.log(model); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `modelId` | `string` | | › `modelUserAppId?` | `Object` | | › `modelUserAppId.appId` | `string` | | › `modelUserAppId.userId` | `string` | | › `modelVersionId?` | `string` | **Returns** `Promise`\ A model object for the specified model ID. **Defined in** [src/client/app.ts:693](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L693) #### workflow ▸ **workflow**(`workflowId`): `Promise`\ Returns a Workflow object for the existing workflow ID. **Example** ```ts import { App } from "clarifai-nodejs"; const app = new App({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const workflow = await app.workflow({ workflowId: "workflowId" }); console.log(workflow); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `workflowId` | `Object` | The workflow ID for a existing workflow. | | `workflowId.workflowId` | `string` | - | **Returns** `Promise`\ A workflow object for the specified workflow ID. **Defined in** [src/client/app.ts:741](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L741) ## Dataset Dataset is a class that provides access to Clarifai API endpoints related to Dataset information. This is its hierarchy: - `Lister` ↳ **`Dataset`** ### Constructor • **new Dataset**(`«destructured»`): `Dataset` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `DatasetConfig` | **Returns** `Dataset` **Overrides** Lister.constructor **Defined in** [src/client/dataset.ts:39](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L39) ### Properties #### STUB • `Protected` **STUB**: `V2Stub` **Inherited from** Lister.STUB **Defined in** [src/client/base.ts:27](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L27) #### authHelper • `Protected` **authHelper**: `ClarifaiAuthHelper` **Inherited from** Lister.authHelper **Defined in** [src/client/base.ts:26](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L26) #### base • `Protected` **base**: `string` **Inherited from** Lister.base **Defined in** [src/client/base.ts:31](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L31) #### batchSize • `Private` **batchSize**: `number` = `128` **Defined in** [src/client/dataset.ts:36](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L36) #### defaultPageSize • **defaultPageSize**: `number` **Inherited from** Lister.defaultPageSize **Defined in** [src/client/lister.ts:10](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/lister.ts#L10) #### info • `Private` **info**: `Dataset` **Defined in** [src/client/dataset.ts:35](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L35) #### input • `Private` **input**: `Input` **Defined in** [src/client/dataset.ts:37](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L37) #### metadata • `Protected` **metadata**: [`string`, `string`][] **Inherited from** Lister.metadata **Defined in** [src/client/base.ts:28](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L28) #### pat • `Protected` **pat**: `string` **Inherited from** Lister.pat **Defined in** [src/client/base.ts:29](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L29) #### rootCertificatesPath • `Protected` **rootCertificatesPath**: `string` **Inherited from** Lister.rootCertificatesPath **Defined in** [src/client/base.ts:32](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L32) #### userAppId • `Protected` **userAppId**: `UserAppIDSet` **Inherited from** Lister.userAppId **Defined in** [src/client/base.ts:30](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L30) ### Methods #### convertStringToTimestamp ▸ **convertStringToTimestamp**(`dateStr`): `Timestamp` Converts a string to a Timestamp object. **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `dateStr` | `string` | The string to convert. | **Returns** `Timestamp` A Timestamp object representing the given date string. **Inherited from** Lister.convertStringToTimestamp **Defined in** [src/client/base.ts:100](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L100) #### createVersion ▸ **createVersion**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `description` | `string` | | › `id` | `string` | | › `metadata?` | `Record`\ | **Returns** `Promise`\ **Defined in** [src/client/dataset.ts:58](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L58) #### deleteVersion ▸ **deleteVersion**(`versionId`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `versionId` | `string` | **Returns** `Promise`\ **Defined in** [src/client/dataset.ts:93](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L93) #### grpcRequest ▸ **grpcRequest**\(`endpoint`, `requestData`): `Promise`\ Makes a gRPC request to the API. **Type parameters** | Name | Type | | :------ | :------ | | `TRequest` | extends `Message` | | `TResponseObject` | extends `Object` | | `TResponse` | extends `Object` | **Parameters** | Name | Type | | :------ | :------ | | `endpoint` | (`request`: `TRequest`, `metadata`: `Metadata`, `options`: `Partial`\) => `Promise`\ | | `requestData` | `TRequest` | **Returns** `Promise`\ A Promise resolving to the result of the gRPC method call. **Inherited from** Lister.grpcRequest **Defined in** [src/client/base.ts:77](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/base.ts#L77) #### listPagesData ▸ **listPagesData**\(`endpoint`, `requestData`, `pageNo?`, `perPage?`): `Promise`\ **Type parameters** | Name | Type | | :------ | :------ | | `TRequest` | extends `Message` | | `TResponseObject` | extends `Object` | | `TResponse` | extends `Object` | **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `endpoint` | (`request`: `TRequest`, `metadata`: `Metadata`, `options`: `Partial`\) => `Promise`\ | `undefined` | | `requestData` | `TRequest` | `undefined` | | `pageNo` | `number` | `1` | | `perPage` | `number` | `undefined` | **Returns** `Promise`\ **Inherited from** Lister.listPagesData **Defined in** [src/client/lister.ts:92](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/lister.ts#L92) #### listPagesGenerator ▸ **listPagesGenerator**\(`endpoint`, `requestData`, `pageNo?`, `perPage?`): `AsyncGenerator`\ **Type parameters** | Name | Type | | :------ | :------ | | `TRequest` | extends `Message` | | `TResponseObject` | extends `Object` | | `TResponse` | extends `Object` | **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `endpoint` | (`request`: `TRequest`, `metadata`: `Metadata`, `options`: `Partial`\) => `Promise`\ | `undefined` | | `requestData` | `TRequest` | `undefined` | | `pageNo` | `number` | `1` | | `perPage` | `number` | `undefined` | **Returns** `AsyncGenerator`\ **Inherited from** Lister.listPagesGenerator **Defined in** [src/client/lister.ts:23](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/lister.ts#L23) #### listVersions ▸ **listVersions**(`pageNo?`, `perPage?`): `AsyncGenerator`\ **Parameters** | Name | Type | | :------ | :------ | | `pageNo?` | `number` | | `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Defined in** [src/client/dataset.ts:113](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L113) #### uploadFromCSV ▸ **uploadFromCSV**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `batchSize?` | `number` | `128` | | › `csvPath` | `string` | `undefined` | | › `csvType` | ``"url"`` \| ``"raw"`` \| ``"file"`` | `undefined` | | › `inputType?` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | `"text"` | | › `labels?` | `boolean` | `true` | | › `uploadProgressEmitter?` | [`InputBulkUpload`](#inputbulkupload) | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/dataset.ts:176](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L176) #### uploadFromFolder ▸ **uploadFromFolder**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `batchSize?` | `number` | `undefined` | | › `folderPath` | `string` | `undefined` | | › `inputType` | ``"image"`` \| ``"text"`` | `undefined` | | › `labels?` | `boolean` | `false` | | › `uploadProgressEmitter?` | [`InputBulkUpload`](#inputbulkupload) | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/dataset.ts:138](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/dataset.ts#L138) ## Input Input is a class that provides access to Clarifai API endpoints related to Input information. This is its hierarchy: - `Lister` ↳ **`Input`** ### Constructor • **new Input**(`params`): `Input` Initializes an input object. **Example** ```ts import { Input } from "clarifai-nodejs"; export const input = new Input({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `params` | `Object` | The parameters for the Input object. | | `params.authConfig?` | `AuthConfig` | - | **Returns** `Input` **Overrides** Lister.constructor **Defined in** [src/client/input.ts:101](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L101) ### Properties #### numOfWorkers • `Private` **numOfWorkers**: `number` **Defined in** [src/client/input.ts:87](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L87) ### Methods #### bulkUpload ▸ **bulkUpload**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `batchSize?` | `number` | `128` | | › `inputs` | `Input`[] | `undefined` | | › `uploadProgressEmitter?` | [`InputBulkUpload`](#inputbulkupload) | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:1038](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L1038) #### deleteFailedInputs ▸ **deleteFailedInputs**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inputs` | `Input`[] | **Returns** `Promise`\ **Defined in** [src/client/input.ts:1165](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L1165) #### patchInputs ▸ **patchInputs**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `action?` | `string` | `"merge"` | | › `inputs` | `Input`[] | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:970](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L970) #### retryUploads ▸ **retryUploads**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `failedInputs` | `Input`[] | **Returns** `Promise`\ **Defined in** [src/client/input.ts:1209](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L1209) #### uploadAnnotations ▸ **uploadAnnotations**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `batchAnnot` | `Annotation`[] | `undefined` | | › `showLog?` | `boolean` | `true` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:1006](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L1006) #### uploadBatch ▸ **uploadBatch**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inputs` | `Input`[] | **Returns** `Promise`\ **Defined in** [src/client/input.ts:1090](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L1090) #### uploadFromBytes ▸ **uploadFromBytes**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioBytes?` | ``null`` \| `Uint8Array` | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imageBytes?` | ``null`` \| `Uint8Array` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textBytes?` | ``null`` \| `Uint8Array` | `null` | | › `videoBytes?` | ``null`` \| `Uint8Array` | `null` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:918](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L918) #### uploadFromFile ▸ **uploadFromFile**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioFile?` | ``null`` \| `string` | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imageFile?` | ``null`` \| `string` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textFile?` | ``null`` \| `string` | `null` | | › `videoFile?` | ``null`` \| `string` | `null` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:883](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L883) #### uploadFromUrl ▸ **uploadFromUrl**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioUrl?` | ``null`` \| `string` | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imageUrl?` | ``null`` \| `string` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textUrl?` | ``null`` \| `string` | `null` | | › `videoUrl?` | ``null`` \| `string` | `null` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:848](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L848) #### uploadInputs ▸ **uploadInputs**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `inputs` | `Input`[] | `undefined` | | › `showLog?` | `boolean` | `true` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:804](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L804) #### uploadText ▸ **uploadText**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `inputId` | `string` | `undefined` | | › `rawText` | `string` | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:953](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L953) #### waitForInputs ▸ **waitForInputs**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inputJobId` | `string` | **Returns** `Promise`\ **Defined in** [src/client/input.ts:1101](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L1101) #### getBboxProto ▸ **getBboxProto**(`«destructured»`): `Annotation` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `bbox` | `number`[] | | › `inputId` | `string` | | › `label` | `string` | **Returns** `Annotation` **Defined in** [src/client/input.ts:719](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L719) #### getImageInputsFromFolder ▸ **getImageInputsFromFolder**(`«destructured»`): `Input`[] Upload image inputs from folder. **Example** ```ts import { Input, Model } from "clarifai-nodejs"; import path from "path"; // Generate a new GRPC compatible Input object from buffer const imageInputs = Input.getImageInputsFromFolder({ // Ensure the directory contains a list of images folderPath: path.resolve(__dirname, "path/to/imageFolder"), }); // The input can now be used as an input for a model prediction methods const model = new Model({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, modelId: "multimodal-clip-embed", }); const prediction = await model.predict({ inputs: imageInputs, }); console.log(prediction); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `folderPath` | `string` | `undefined` | | › `labels?` | `boolean` | `false` | **Returns** `Input`[] **Defined in** [src/client/input.ts:433](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L433) #### getInputFromBytes ▸ **getInputFromBytes**(`«destructured»`): `Input` Creates an input proto from bytes. **Example** ```ts import { Input, Model } from "clarifai-nodejs"; import * as fs from "fs"; const imageBuffer = fs.readFileSync("path/to/image.jpg"); // Generate a new GRPC compatible Input object from buffer const imageInput = Input.getInputFromBytes({ inputId: "demo", imageBytes: imageBuffer, }); // The input can now be used as an input for a model prediction methods const model = new Model({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, modelId: "multimodal-clip-embed", }); const prediction = await model.predict({ inputs: [imageInput], }); console.log(prediction); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioBytes?` | ``null`` \| `Uint8Array` | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imageBytes?` | ``null`` \| `Uint8Array` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textBytes?` | ``null`` \| `Uint8Array` | `null` | | › `videoBytes?` | ``null`` \| `Uint8Array` | `null` | **Returns** `Input` An `Input` object for the specified input ID. **Defined in** [src/client/input.ts:231](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L231) #### getInputFromFile ▸ **getInputFromFile**(`«destructured»`): `Input` Create input proto from files. **Example** ```ts import { Input, Model } from "clarifai-nodejs"; import path from "path"; // Generate a new GRPC compatible Input object from buffer const imageInput = Input.getInputFromFile({ inputId: "demo", imageFile: path.resolve(__dirname, "path/to/image.jpg"), }); // The input can now be used as an input for a model prediction methods const model = new Model({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, modelId: "multimodal-clip-embed", }); const prediction = await model.predict({ inputs: [imageInput], }); console.log(prediction); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioFile?` | ``null`` \| `string` | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imageFile?` | ``null`` \| `string` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textFile?` | ``null`` \| `string` | `null` | | › `videoFile?` | ``null`` \| `string` | `null` | **Returns** `Input` - An Input object for the specified input ID. **Defined in** [src/client/input.ts:295](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L295) #### getInputFromUrl ▸ **getInputFromUrl**(`«destructured»`): `Input` Upload input from URL. **Example** ```ts import { Input, Model } from "clarifai-nodejs"; // Generate a new GRPC compatible Input object from buffer const imageInput = Input.getInputFromUrl({ inputId: "demo", imageUrl: "https://samples.clarifai.com/dog2.jpeg", }); // The input can now be used as an input for a model prediction methods const model = new Model({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, modelId: "multimodal-clip-embed", }); const prediction = await model.predict({ inputs: [imageInput], }); console.log(prediction); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioUrl?` | ``null`` \| `string` | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imageUrl?` | ``null`` \| `string` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textUrl?` | ``null`` \| `string` | `null` | | › `videoUrl?` | ``null`` \| `string` | `null` | **Returns** `Input` - Job ID for the upload request. **Defined in** [src/client/input.ts:359](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L359) #### getInputsFromCsv ▸ **getInputsFromCsv**(`«destructured»`): `Promise`\ Create Input proto from CSV File. Supported columns are: 'inputid', 'input', 'concepts', 'metadata', 'geopoints' **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `csvPath` | `string` | `undefined` | | › `csvType` | ``"url"`` \| ``"raw"`` \| ``"file"`` | `"raw"` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `inputType` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | `"text"` | | › `labels` | `boolean` | `true` | **Returns** `Promise`\ - An array of Input objects for the specified input ID. **Defined in** [src/client/input.ts:573](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L573) #### getMaskProto ▸ **getMaskProto**(`«destructured»`): `Annotation` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inputId` | `string` | | › `label` | `string` | | › `polygons` | `Polygon`[] | **Returns** `Annotation` **Defined in** [src/client/input.ts:760](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L760) #### getMultimodalInput ▸ **getMultimodalInput**(`«destructured»`): `Input` Create input proto for text and image from bytes or url **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `imageBytes?` | ``null`` \| `Uint8Array` | `null` | | › `imageUrl?` | ``null`` \| `string` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `rawText?` | ``null`` \| `string` | `null` | | › `textBytes?` | ``null`` \| `Uint8Array` | `null` | **Returns** `Input` - An Input object for the specified input ID. **Defined in** [src/client/input.ts:513](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L513) #### getProto ▸ **getProto**(`«destructured»`): `Input` Create input proto for image data type. **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `audioPb?` | ``null`` \| \{ `base64`: `string` } | `null` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `imagePb?` | ``null`` \| \{ `base64`: `string` ; `url?`: `undefined` } \| \{ `base64?`: `undefined` ; `url`: `string` } | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `textPb?` | ``null`` \| \{ `raw`: `string` } | `null` | | › `videoPb?` | ``null`` \| \{ `base64`: `string` } | `null` | **Returns** `Input` - An Input object for the specified input ID. **Defined in** [src/client/input.ts:119](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L119) #### getTextInput ▸ **getTextInput**(`«destructured»`): `Input` Create input proto for text data type from raw text. **Example** ```ts import { Input, Model } from "clarifai-nodejs"; // Generate a new GRPC compatible Input object from buffer const textInput = Input.getTextInput({ inputId: "demo", rawText: "Sample text for input generation", }); // The input can now be used as an input for a model prediction methods const model = new Model({ authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, modelId: "multimodal-clip-embed", }); const prediction = await model.predict({ inputs: [textInput], }); console.log(prediction); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `datasetId?` | ``null`` \| `string` | `null` | | › `geoInfo?` | ``null`` \| `AsObject` | `null` | | › `inputId` | `string` | `undefined` | | › `labels?` | ``null`` \| `string`[] | `null` | | › `metadata?` | ``null`` \| `Record`\ | `null` | | › `rawText` | `string` | `undefined` | **Returns** `Input` - An Input object for the specified input ID. **Defined in** [src/client/input.ts:476](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L476) #### getTextInputsFromFolder ▸ **getTextInputsFromFolder**(`«destructured»`): `Input`[] **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `datasetId` | ``null`` \| `string` | `null` | | › `folderPath` | `string` | `undefined` | | › `labels` | `boolean` | `false` | **Returns** `Input`[] **Defined in** [src/client/input.ts:688](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L688) ## Model Model is a class that provides access to Clarifai API endpoints related to Model information. This is its hierarchy: - `Lister` ↳ **`Model`** ### Constructor • **new Model**(`config`): `Model` Initializes a Model object. **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "face-detection", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); ``` **Parameters** | Name | Type | | :------ | :------ | | `config` | `ModelConfig` | **Returns** `Model` **Overrides** Lister.constructor **Defined in** [src/client/model.ts:132](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L132) ### Properties #### appId • `Private` **appId**: `string` **Defined in** [src/client/model.ts:111](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L111) #### id • `Private` **id**: `string` **Defined in** [src/client/model.ts:112](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L112) #### modelInfo • **modelInfo**: `Model` **Defined in** [src/client/model.ts:115](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L115) #### modelUserAppId • `Private` **modelUserAppId**: `undefined` \| `UserAppIDSet` **Defined in** [src/client/model.ts:113](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L113) #### modelVersion • `Private` **modelVersion**: `undefined` \| \{ `id`: `string` } **Defined in** [src/client/model.ts:114](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L114) #### runner • `Private` **runner**: `undefined` \| `RunnerSelector` **Defined in** [src/client/model.ts:117](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L117) #### trainingParams • `Private` **trainingParams**: `Record`\ **Defined in** [src/client/model.ts:116](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L116) ### Methods #### availableMethods ▸ **availableMethods**(): `Promise`\ **Returns** `Promise`\ **Defined in** [src/client/model.ts:647](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L647) #### constructRequestWithMethodSignature ▸ **constructRequestWithMethodSignature**(`request`, `config`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `request` | `PostModelOutputsRequest` | | `config` | `TextModelPredictConfig` | **Returns** `Promise`\ **Defined in** [src/client/model.ts:663](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L663) #### createVersion ▸ **createVersion**(`modelVersion`): `Promise`\ Creates a model version for the Model. **Example** ```ts import { Model } from "clarifai-nodejs"; import { ModelVersion, OutputInfo, } from "clarifai-nodejs-grpc/proto/clarifai/api/resources_pb"; import { Struct } from "google-protobuf/google/protobuf/struct_pb.js"; export const model = new Model({ modelId: "margin-100-image-cropper", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); // Creating a GRPC compatible outputInfo object with custom margin parameters const outputInfo = new OutputInfo().setParams( Struct.fromJavaScript({ margin: 1.5 }), ); // GRPC compatible ModelVersion object with previously created output info config const modelVersion = new ModelVersion() .setDescription("Setting output info margin parameters to 1.5") .setOutputInfo(outputInfo); // Creating a new version of the model with previously created output info config const modelObjectWithVersion = await model.createVersion(modelVersion); console.log(modelObjectWithVersion); ``` **Parameters** | Name | Type | | :------ | :------ | | `modelVersion` | `ModelVersion` | **Returns** `Promise`\ **Defined in** [src/client/model.ts:545](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L545) #### deleteVersion ▸ **deleteVersion**(`versionId`): `Promise`\ Deletes a model version for the Model. **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "face-detection", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); model.deleteVersion("version_id"); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `versionId` | `string` | The version ID to delete. | **Returns** `Promise`\ **Defined in** [src/client/model.ts:514](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L514) #### generate ▸ **generate**(`«destructured»`): `AsyncGenerator`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `TextModelPredictConfig` | **Returns** `AsyncGenerator`\ **Defined in** [src/client/model.ts:1120](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L1120) #### generateGrpc ▸ **generateGrpc**(`«destructured»`): `AsyncGenerator`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `TextModelPredictConfig` | **Returns** `AsyncGenerator`\ **Defined in** [src/client/model.ts:848](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L848) #### getParamInfo ▸ **getParamInfo**(`param`): `Promise`\\> Returns the param info for the model. **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "face-detection", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); model.getParamInfo("template"); ``` **Parameters** | Name | Type | | :------ | :------ | | `param` | `string` | **Returns** `Promise`\\> **Defined in** [src/client/model.ts:446](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L446) #### getParams ▸ **getParams**(`template?`, `saveTo?`): `Promise`\\> Returns the model params for the model type as object & also writes to a yaml file **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "face-detection", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const modelParams = await model.getParams("face-detection", "params.yml"); console.log(modelParams); ``` **Parameters** | Name | Type | Default value | Description | | :------ | :------ | :------ | :------ | | `template` | ``null`` \| `string` | `null` | The training template to use for the model type. | | `saveTo` | `string` | `"params.yaml"` | The file path to save the yaml file. | **Returns** `Promise`\\> - A promise that resolves to the model params for the model type. **Defined in** [src/client/model.ts:336](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L336) #### getRunner ▸ **getRunner**(): `undefined` \| `AsObject` Returns the runner for the model. **Returns** `undefined` \| `AsObject` - The runner for the model. **Defined in** [src/client/model.ts:230](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L230) #### listTrainingTemplates ▸ **listTrainingTemplates**(): `Promise`\ Lists all the training templates for the model type. **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "face-detection", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const trainingTemplates = await model.listTrainingTemplates(); console.log(trainingTemplates); ``` **Returns** `Promise`\ - A promise that resolves to a list of training templates for the model type. **Defined in** [src/client/model.ts:289](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L289) #### listVersions ▸ **listVersions**(`«destructured»?`): `AsyncGenerator`\ Lists all the versions for the model. **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "lvm-dummy-test", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const versions = await model.listVersions().next(); console.log(versions); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Notes** Defaults to 16 per page if pageNo is not specified **Defined in** [src/client/model.ts:589](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L589) #### loadInfo ▸ **loadInfo**(): `Promise`\ Loads the current model info. Usually called internally by other methods, to ensure the model info is loaded with latest data. **Returns** `Promise`\ **Defined in** [src/client/model.ts:238](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L238) #### methodSignatures ▸ **methodSignatures**(): `Promise`\ **Returns** `Promise`\ **Defined in** [src/client/model.ts:625](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L625) #### overrideModelVersion ▸ **overrideModelVersion**(`«destructured»`): `void` Overrides the model version. **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inferenceParams?` | `Record`\ | | › `outputConfig?` | `OutputConfig` | **Returns** `void` **Defined in** [src/client/model.ts:1314](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L1314) #### predict ▸ **predict**(`predictArgs`): `Promise`\ Predicts the model based on the given inputs. Useful for chat / text based llms **Parameters** | Name | Type | | :------ | :------ | | `predictArgs` | `TextModelPredictConfig` | **Returns** `Promise`\ **Defined in** [src/client/model.ts:740](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L740) ▸ **predict**(`«destructured»`): `Promise`\ Predicts the model based on the given inputs. Use the `Input` module to create the input objects. **Example** ```ts import { Model, Input } from "clarifai-nodejs"; export const model = new Model({ modelId: "multimodal-clip-embed", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); const input = Input.getInputFromBytes({ inputId: "intro-text", textBytes: Buffer.from("Hi my name is Jim."), }); const textPrediction = await model.predict({ inputs: [input], }); console.log(textPrediction); const imageInput = Input.getInputFromUrl({ inputId: "test-image", imageUrl: "https://goldenglobes.com/wp-content/uploads/2023/10/17-tomcruiseag.jpg", }); const imagePrediction = await model.predict({ inputs: [imageInput], }); console.log(imagePrediction); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `GeneralModelPredictConfig` | **Returns** `Promise`\ - A promise that resolves to the model prediction. **Defined in** [src/client/model.ts:758](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L758) #### predictByBytes ▸ **predictByBytes**(`«destructured»`): `Promise`\ Predicts the model based on the given inputs. Inputs can be provided as a Buffer. **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inferenceParams?` | `Record`\ | | › `inputBytes` | `Buffer` | | › `inputType` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | | › `outputConfig?` | `OutputConfig` | **Returns** `Promise`\ - A promise that resolves to the model prediction. **Defined in** [src/client/model.ts:1255](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L1255) #### predictByFilepath ▸ **predictByFilepath**(`«destructured»`): `Promise`\ Predicts the model based on the given inputs. Inputs can be provided as a filepath which can be read. **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `filepath` | `string` | | › `inferenceParams?` | `Record`\ | | › `inputType` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | | › `outputConfig?` | `OutputConfig` | **Returns** `Promise`\ - A promise that resolves to the model prediction. **Defined in** [src/client/model.ts:1221](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L1221) #### predictByUrl ▸ **predictByUrl**(`«destructured»`): `Promise`\ Predicts the model based on the given inputs. Inputs can be provided as a URL. **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inferenceParams?` | `Record`\ | | › `inputType` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | | › `outputConfig?` | `OutputConfig` | | › `url` | `string` | **Returns** `Promise`\ - A promise that resolves to the model prediction. **Defined in** [src/client/model.ts:1179](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L1179) #### setRunner ▸ **setRunner**(`runner`): `void` Sets the runner for the model. **Parameters** | Name | Type | | :------ | :------ | | `runner` | `Subset`\ | **Returns** `void` **Defined in** [src/client/model.ts:202](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L202) #### stream ▸ **stream**(`config`): `Promise`\ `void` ; `iterator`: `AsyncGenerator`\ ; `send`: (`request`: `PostModelOutputsRequest`) => `void` }\> **Parameters** | Name | Type | | :------ | :------ | | `config` | `TextModelPredictConfig` | **Returns** `Promise`\ `void` ; `iterator`: `AsyncGenerator`\ ; `send`: (`request`: `PostModelOutputsRequest`) => `void` }\> **Defined in** [src/client/model.ts:1034](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L1034) #### streamWithControl ▸ **streamWithControl**(`«destructured»`): `Object` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `TextModelPredictConfig` | **Returns** `Object` | Name | Type | | :------ | :------ | | `end` | () => `void` | | `iterator` | `AsyncGenerator`\ | | `send` | (`request`: `PostModelOutputsRequest`) => `void` | **Defined in** [src/client/model.ts:931](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L931) #### updateParams ▸ **updateParams**(`modelParams`): `void` Updates the model params for the model. **Example** ```ts import { Model } from "clarifai-nodejs"; export const model = new Model({ modelId: "face-detection", authConfig: { pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }, }); model.updateParams({ batchSize: 8, datasetVersion: "version_id", }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `modelParams` | `Record`\ | The model params to update. | **Returns** `void` **Defined in** [src/client/model.ts:415](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L415) #### getOutputDataFromModelResponse ▸ **getOutputDataFromModelResponse**(`outputs`): `undefined` \| `AsObject` **Parameters** | Name | Type | | :------ | :------ | | `outputs` | `AsObject`[] | **Returns** `undefined` \| `AsObject` **Defined in** [src/client/model.ts:641](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/model.ts#L641) ## RAG RAG is a class that provides access to Clarifai API endpoints related to RAG information. ### Constructor • **new RAG**(`«destructured»`): `RAG` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `RAGConfig` | **Returns** `RAG` **Defined in** [src/client/rag.ts:67](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L67) ### Properties #### app • **app**: `App` **Defined in** [src/client/rag.ts:65](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L65) #### authConfig • `Private` **authConfig**: `AuthConfig` **Defined in** [src/client/rag.ts:61](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L61) #### promptWorkflow • **promptWorkflow**: `Workflow` **Defined in** [src/client/rag.ts:63](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L63) ### Methods #### chat ▸ **chat**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `clientManageState?` | `boolean` | `true` | | › `messages` | `Message`[] | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/rag.ts:405](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L405) #### upload ▸ **upload**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `batchSize?` | `number` | `128` | | › `chunkOverlap?` | `number` | `200` | | › `chunkSize?` | `number` | `1024` | | › `datasetId?` | `string` | `undefined` | | › `filePath?` | `string` | `undefined` | | › `folderPath?` | `string` | `undefined` | | › `metadata?` | `Record`\ | `undefined` | | › `url?` | `string` | `undefined` | **Returns** `Promise`\ **Defined in** [src/client/rag.ts:283](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L283) #### validateInputs ▸ **validateInputs**(`workflowUrl?`, `workflow?`, `authConfig?`): `void` **Parameters** | Name | Type | | :------ | :------ | | `workflowUrl?` | `string` | | `workflow?` | [`Workflow`](#workflow) | | `authConfig?` | `AuthConfig` \| `UrlAuthConfig` | **Returns** `void` **Defined in** [src/client/rag.ts:89](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L89) **setup** ▸ **setup**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `appUrl?` | `$\{string}://$\{string}/$\{string}/$\{string}\` | `undefined` | | › `authConfig?` | [`AuthAppConfig`](#authappconfig) \| `Omit`\ & \{ `appId?`: `undefined` } | `undefined` | | › `baseWorkflow?` | `string` | `"Text"` | | › `llmUrl?` | `ClarifaiUrl` | [`mistral-7B-Instruct`](https://clarifai.com/mistralai/completion/models/mistral-7B-Instruct) | | › `maxResults?` | `number` | `5` | | › `minScore?` | `number` | `0.95` | | › `promptTemplate?` | `string` | `DEFAULT_RAG_PROMPT_TEMPLATE` | | › `workflowId?` | `string` | `undefined` | | › `workflowYamlFilename?` | `string` | `"prompter_wf.yaml"` | **Returns** `Promise`\ **Defined in** [src/client/rag.ts:109](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/rag.ts#L109) ## Search Search is a class that provides access to Clarifai API endpoints related to Search information. This is its hierarchy: - `Lister` ↳ **`Search`** ### Constructor • **new Search**(`«destructured»`): `Search` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `algorithm?` | `SupportedAlgorithm` | `DEFAULT_SEARCH_ALGORITHM` | | › `authConfig?` | `AuthConfig` | `undefined` | | › `metric?` | `SupportedMetric` | `DEFAULT_SEARCH_METRIC` | | › `topK?` | `number` | `DEFAULT_TOP_K` | **Returns** `Search` **Overrides** Lister.constructor **Defined in** [src/client/search.ts:59](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L59) ### Properties #### algorithm • `Private` **algorithm**: `SupportedAlgorithm` **Defined in** [src/client/search.ts:57](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L57) #### dataProto • `Private` **dataProto**: `Data` **Defined in** [src/client/search.ts:55](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L55) #### inputProto • `Private` **inputProto**: [`Input`](#input) **Defined in** [src/client/search.ts:56](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L56) #### metricDistance • `Private` **metricDistance**: ``"COSINE_DISTANCE"`` \| ``"EUCLIDEAN_DISTANCE"`` **Defined in** [src/client/search.ts:54](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L54) #### topK • `Private` **topK**: `number` **Defined in** [src/client/search.ts:53](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L53) ### Methods #### getAnnotProto ▸ **getAnnotProto**(`args`): `Annotation` **Parameters** | Name | Type | | :------ | :------ | | `args` | `Object` | | `args.concepts?` | \{ `id?`: `string` ; `language?`: `string` ; `name?`: `string` ; `value?`: `number` }[] | | `args.geoPoint?` | `Object` | | `args.geoPoint.geoLimit` | `number` | | `args.geoPoint.latitude` | `number` | | `args.geoPoint.longitude` | `number` | | `args.imageBytes?` | `unknown` | | `args.imageUrl?` | `string` | | `args.inputDatasetIds?` | `string`[] | | `args.inputStatusCode?` | `number` | | `args.inputTypes?` | (``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"``)[] | | `args.metadata?` | `Record`\ | | `args.textRaw?` | `string` | **Returns** `Annotation` **Defined in** [src/client/search.ts:94](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L94) #### getGeoPointProto ▸ **getGeoPointProto**(`longitude`, `latitude`, `geoLimit`): `Geo` **Parameters** | Name | Type | | :------ | :------ | | `longitude` | `number` | | `latitude` | `number` | | `geoLimit` | `number` | **Returns** `Geo` **Defined in** [src/client/search.ts:204](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L204) #### getInputProto ▸ **getInputProto**(`args`): `Input` **Parameters** | Name | Type | | :------ | :------ | | `args` | `Object` | | `args.concepts?` | \{ `id?`: `string` ; `language?`: `string` ; `name?`: `string` ; `value?`: `number` }[] | | `args.geoPoint?` | `Object` | | `args.geoPoint.geoLimit` | `number` | | `args.geoPoint.latitude` | `number` | | `args.geoPoint.longitude` | `number` | | `args.imageBytes?` | `unknown` | | `args.imageUrl?` | `string` | | `args.inputDatasetIds?` | `string`[] | | `args.inputStatusCode?` | `number` | | `args.inputTypes?` | (``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"``)[] | | `args.metadata?` | `Record`\ | | `args.textRaw?` | `string` | **Returns** `Input` **Defined in** [src/client/search.ts:168](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L168) #### listAllPagesGenerator ▸ **listAllPagesGenerator**\(`«destructured»`): `AsyncGenerator`\ **Type parameters** | Name | Type | | :------ | :------ | | `T` | extends `PostAnnotationsSearchesRequest` \| `PostInputsSearchesRequest` | **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `endpoint` | (`request`: `T`, `metadata`: `Metadata`, `options`: `Partial`\) => `Promise`\ | `undefined` | | › `page?` | `number` | `1` | | › `perPage?` | `number` | `undefined` | | › `requestData` | `T` | `undefined` | **Returns** `AsyncGenerator`\ **Defined in** [src/client/search.ts:221](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L221) #### query ▸ **query**(`«destructured»`): `AsyncGenerator`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `filters?` | \{ `concepts?`: \{ `id?`: `string` ; `language?`: `string` ; `name?`: `string` ; `value?`: `number` }[] ; `geoPoint?`: \{ `geoLimit`: `number` ; `latitude`: `number` ; `longitude`: `number` } ; `imageBytes?`: `unknown` ; `imageUrl?`: `string` ; `inputDatasetIds?`: `string`[] ; `inputStatusCode?`: `number` ; `inputTypes?`: (``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"``)[] ; `metadata?`: `Record`\ ; `textRaw?`: `string` }[] | | › `page?` | `number` | | › `perPage?` | `number` | | › `ranks?` | \{ `concepts?`: \{ `id?`: `string` ; `language?`: `string` ; `name?`: `string` ; `value?`: `number` }[] ; `geoPoint?`: \{ `geoLimit`: `number` ; `latitude`: `number` ; `longitude`: `number` } ; `imageBytes?`: `unknown` ; `imageUrl?`: `string` ; `inputDatasetIds?`: `string`[] ; `inputStatusCode?`: `number` ; `inputTypes?`: (``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"``)[] ; `metadata?`: `Record`\ ; `textRaw?`: `string` }[] | **Returns** `AsyncGenerator`\ **Defined in** [src/client/search.ts:290](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/search.ts#L290) ## User User is a class that provides access to Clarifai API endpoints related to user information. This is its hierarchy: - `Lister` ↳ **`User`** ### Constructor • **new User**(`authConfig?`): `User` Initializes a User object with the specified authentication configuration. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `authConfig` | `AuthConfig` | An object containing the authentication configuration. Defaults to an empty object. | **Returns** `User` **Overrides** Lister.constructor **Defined in** [src/client/user.ts:45](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L45) ### Methods #### app ▸ **app**(`appId`): `Promise`\ Returns an App object for the specified app ID. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const app = await user.app({ appId: "app_id", }); console.log(app); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `appId` | `Object` | The app ID for the app to interact with. | | `appId.appId` | `string` | - | **Returns** `Promise`\ An App object for the specified app ID. **Defined in** [src/client/user.ts:240](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L240) #### createApp ▸ **createApp**(`«destructured»`): `Promise`\ Creates an app for the user. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const app = await user.createApp({ appId: "app_id", baseWorkflow: "Universal", }); console.log(app); ``` **Parameters** | Name | Type | Default value | | :------ | :------ | :------ | | `«destructured»` | `Object` | `undefined` | | › `appId` | `string` | `undefined` | | › `baseWorkflow?` | `string` | `"Empty"` | **Returns** `Promise`\ An App object for the specified app ID. **Defined in** [src/client/user.ts:141](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L141) #### createRunner ▸ **createRunner**(`«destructured»`): `Promise`\ Creates a runner for the user. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const runner = await user.createRunner({ runnerId: "runner_id", labels: ["label to link runner"], description: "laptop runner", }); console.log(runner); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `description` | `string` | | › `labels` | `string`[] | | › `runnerId` | `string` | **Returns** `Promise`\ A runner object for the specified Runner ID. **Defined in** [src/client/user.ts:192](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L192) #### deleteApp ▸ **deleteApp**(`appId`): `Promise`\ Deletes an app for the user. **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `appId` | `Object` | The app ID for the app to delete. | | `appId.appId` | `string` | - | **Returns** `Promise`\ **Example** ```ts examples/user/deleteApp.ts ``` **Defined in** [src/client/user.ts:304](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L304) #### deleteRunner ▸ **deleteRunner**(`runnerId`): `Promise`\ Deletes a runner for the user. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); await user.deleteRunner({ runnerId: "runner_id" }); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `runnerId` | `Object` | The runner ID to delete. | | `runnerId.runnerId` | `string` | - | **Returns** `Promise`\ **Defined in** [src/client/user.ts:331](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L331) #### listApps ▸ **listApps**(`«destructured»?`): `AsyncGenerator`\ Lists all the apps for the user. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const list = await user .listApps({ pageNo: 1, perPage: 20, params: { sortAscending: true, }, }) .next(); const apps = list.value; console.log(apps); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `params?` | [`ListAppsRequestParams`](#listappsrequestparams) | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Yields** App objects for the user. **Note** Defaults to 16 per page if pageNo is specified and perPage is not specified. If both pageNo and perPage are None, then lists all the resources. **Defined in** [src/client/user.ts:62](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L62) #### listRunners ▸ **listRunners**(`«destructured»?`): `AsyncGenerator`\ Lists all the runners for the user. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const list = await user.listRunners().next(); const runners = list.value; console.log(runners); ``` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `params?` | [`ListRunnersRequestParams`](#listrunnersrequestparams) | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Yields** Runner objects for the user. **Note** Defaults to 16 per page if perPage is not specified. **Defined in** [src/client/user.ts:103](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L103) #### runner ▸ **runner**(`runnerId`): `Promise`\ Returns a Runner object if exists. **Example** ```ts import { User } from "clarifai-nodejs"; export const user = new User({ pat: process.env.CLARIFAI_PAT!, userId: process.env.CLARIFAI_USER_ID!, appId: process.env.CLARIFAI_APP_ID!, }); const runner = await user.runner({ runnerId: "runner_id" }); console.log(runner); ``` **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `runnerId` | `Object` | The runner ID to interact with. | | `runnerId.runnerId` | `string` | - | **Returns** `Promise`\ A Runner object for the existing runner ID. **Defined in** [src/client/user.ts:272](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L272) ## Workflow Workflow is a class that provides access to Clarifai API endpoints related to workflow information. This is its hierarchy: - `Lister` ↳ **`Workflow`** ### Constructor • **new Workflow**(`«destructured»`): `Workflow` **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `WorkflowConfig` | **Returns** `Workflow` **Overrides** Lister.constructor **Defined in** [src/client/workflow.ts:55](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L55) ### Properties #### appId • **appId**: `string` **Defined in** [src/client/workflow.ts:52](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L52) #### id • **id**: `string` **Defined in** [src/client/workflow.ts:51](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L51) #### outputConfig • `Private` **outputConfig**: `OutputConfig` **Defined in** [src/client/workflow.ts:53](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L53) #### versionId • `Private` **versionId**: `string` **Defined in** [src/client/workflow.ts:50](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L50) ### Methods #### exportWorkflow ▸ **exportWorkflow**(`outPath`): `Promise`\ Exports the workflow to a yaml file. **Parameters** | Name | Type | Description | | :------ | :------ | :------ | | `outPath` | `string` | The path to save the yaml file to. | **Returns** `Promise`\ **Example** ```typescript import { Workflow } from "./workflow"; const workflow = new Workflow("https://clarifai.com/clarifai/main/workflows/Demographics"); await workflow.export("out_path.yml"); ``` **Defined in** [src/client/workflow.ts:267](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L267) #### listVersions ▸ **listVersions**(`«destructured»`): `AsyncGenerator`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `pageNo?` | `number` | | › `perPage?` | `number` | **Returns** `AsyncGenerator`\ **Defined in** [src/client/workflow.ts:221](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L221) #### predict ▸ **predict**(`«destructured»`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `«destructured»` | `Object` | | › `inputs` | `Input`[] | | › `workflowStateId?` | `string` | **Returns** `Promise`\ **Defined in** [src/client/workflow.ts:84](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L84) #### predictByBytes ▸ **predictByBytes**(`inputBytes`, `inputType`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `inputBytes` | `Buffer` | | `inputType` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | **Returns** `Promise`\ **Defined in** [src/client/workflow.ts:160](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L160) #### predictByUrl ▸ **predictByUrl**(`url`, `inputType`): `Promise`\ **Parameters** | Name | Type | | :------ | :------ | | `url` | `string` | | `inputType` | ``"image"`` \| ``"text"`` \| ``"video"`` \| ``"audio"`` | **Returns** `Promise`\ **Defined in** [src/client/workflow.ts:199](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/workflow.ts#L199) ## Type Aliases ### AppConfig Ƭ **AppConfig**: \{ `authConfig`: [`AuthAppConfig`](#authappconfig) ; `url`: `ClarifaiAppUrl` } \| \{ `authConfig`: `AuthConfig` ; `url?`: `undefined` } **Defined in** [src/client/app.ts:55](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L55) ### AuthAppConfig Ƭ **AuthAppConfig**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `appId?` | `undefined` | | `base` | `undefined` \| `string` | | `pat` | `string` | | `rootCertificatesPath` | `undefined` \| `string` | | `token` | `undefined` \| `string` | | `ui` | `undefined` \| `string` | | `userId?` | `undefined` | **Defined in** [src/client/app.ts:50](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L50) ### CreateDatasetParam Ƭ **CreateDatasetParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `appId` | `undefined` \| `string` | | `bookmarkOrigin` | `undefined` \| `AsObject` | | `createdAt` | `undefined` \| `AsObject` | | `defaultAnnotationFilter` | `undefined` \| `AsObject` | | `defaultProcessingInfo` | `undefined` \| `AsObject` | | `description` | `undefined` \| `string` | | `image` | `undefined` \| `AsObject` | | `isStarred` | `undefined` \| `boolean` | | `metadata` | `undefined` \| `AsObject` | | `modifiedAt` | `undefined` \| `AsObject` | | `notes` | `undefined` \| `string` | | `starCount` | `undefined` \| `number` | | `userId` | `undefined` \| `string` | | `version` | `undefined` \| `AsObject` | | `visibility` | `undefined` \| `AsObject` | **Defined in** [src/client/app.ts:75](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L75) ### CreateModelParam Ƭ **CreateModelParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `appId` | `undefined` \| `string` | | `billingType` | `undefined` \| `BillingType` | | `bookmarkOrigin` | `undefined` \| `AsObject` | | `checkConsentsList` | `undefined` \| `string`[] | | `createdAt` | `undefined` \| `AsObject` | | `creator` | `undefined` \| `string` | | `defaultEvalInfo` | `undefined` \| `AsObject` | | `deployRestriction` | `undefined` \| `DeployRestriction` | | `description` | `undefined` \| `string` | | `displayName` | `undefined` \| `string` | | `featuredOrder` | `undefined` \| `AsObject` | | `image` | `undefined` \| `AsObject` | | `isStarred` | `undefined` \| `boolean` | | `languagesFullList` | `undefined` \| `AsObject`[] | | `languagesList` | `undefined` \| `string`[] | | `licenseType` | `undefined` \| `LicenseType` | | `metadata` | `undefined` \| `AsObject` | | `modelTypeId` | `undefined` \| `string` | | `modelVersion` | `undefined` \| `AsObject` | | `modifiedAt` | `undefined` \| `AsObject` | | `name` | `undefined` \| `string` | | `notes` | `undefined` \| `string` | | `outputInfo` | `undefined` \| `AsObject` | | `presets` | `undefined` \| `AsObject` | | `replicaCount` | `undefined` \| `number` | | `source` | `undefined` \| `Source` | | `starCount` | `undefined` \| `number` | | `task` | `undefined` \| `string` | | `toolkitsList` | `undefined` \| `string`[] | | `useCasesList` | `undefined` \| `string`[] | | `userId` | `undefined` \| `string` | | `versionCount` | `undefined` \| `number` | | `visibility` | `undefined` \| `AsObject` | | `workflowRecommended` | `undefined` \| `AsObject` | **Defined in** [src/client/app.ts:79](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L79) ### InputBulkUpload Ƭ **InputBulkUpload**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `[captureRejectionSymbol]` | `undefined` \| \(`error`: `Error`, `event`: `string` \| `symbol`, ...`args`: `AnyRest`) => `void` | | `addListener` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` | | `emit` | \(`eventName`: `string` \| `symbol`, ...`args`: `AnyRest`) => `boolean` & \(`event`: `K`, `payload`: `UploadEvents`[`K`]) => `boolean` | | `eventNames` | () => (`string` \| `symbol`)[] | | `getMaxListeners` | () => `number` | | `listenerCount` | \(`eventName`: `string` \| `symbol`, `listener?`: `Function`) => `number` | | `listeners` | \(`eventName`: `string` \| `symbol`) => `Function`[] | | `off` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` | | `on` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` & \(`event`: `K`, `listener`: (`payload`: `UploadEvents`[`K`]) => `void`) => `void` | | `once` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` | | `prependListener` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` | | `prependOnceListener` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` | | `rawListeners` | \(`eventName`: `string` \| `symbol`) => `Function`[] | | `removeAllListeners` | (`eventName?`: `string` \| `symbol`) => `this` | | `removeListener` | \(`eventName`: `string` \| `symbol`, `listener`: (...`args`: `any`[]) => `void`) => `this` | | `setMaxListeners` | (`n`: `number`) => `this` | **Defined in** [src/client/input.ts:80](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/input.ts#L80) ### ListAppsRequestParams Ƭ **ListAppsRequestParams**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `additionalFieldsList` | `undefined` \| `string`[] | | `featuredOnly` | `undefined` \| `boolean` | | `id` | `undefined` \| `string` | | `name` | `undefined` \| `string` | | `page` | `undefined` \| `number` | | `query` | `undefined` \| `string` | | `search` | `undefined` \| `string` | | `sortAscending` | `undefined` \| `boolean` | | `sortByCreatedAt` | `undefined` \| `boolean` | | `sortById` | `undefined` \| `boolean` | | `sortByModifiedAt` | `undefined` \| `boolean` | | `sortByName` | `undefined` \| `boolean` | | `sortByStarCount` | `undefined` \| `boolean` | | `starredOnly` | `undefined` \| `boolean` | | `templateOnly` | `undefined` \| `boolean` | **Defined in** [src/client/user.ts:22](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L22) ### ListDatasetsParam Ƭ **ListDatasetsParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `additionalFieldsList` | `undefined` \| `string`[] | | `bookmark` | `undefined` \| `boolean` | | `id` | `undefined` \| `string` | | `page` | `undefined` \| `number` | | `search` | `undefined` \| `string` | | `sortAscending` | `undefined` \| `boolean` | | `sortByCreatedAt` | `undefined` \| `boolean` | | `sortById` | `undefined` \| `boolean` | | `sortByModifiedAt` | `undefined` \| `boolean` | | `sortByStarCount` | `undefined` \| `boolean` | | `starredOnly` | `undefined` \| `boolean` | **Defined in** [src/client/app.ts:65](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L65) ### ListInstalledModuleVersionsParam Ƭ **ListInstalledModuleVersionsParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `page` | `undefined` \| `number` | **Defined in** [src/client/app.ts:73](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L73) ### ListModelsParam Ƭ **ListModelsParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `additionalFieldsList` | `undefined` \| `string`[] | | `bookmark` | `undefined` \| `boolean` | | `creator` | `undefined` \| `string` | | `dontFetchFromMain` | `undefined` \| `boolean` | | `featuredOnly` | `undefined` \| `boolean` | | `filterByUserId` | `undefined` \| `boolean` | | `inputFieldsList` | `undefined` \| `string`[] | | `languagesList` | `undefined` \| `string`[] | | `license` | `undefined` \| `string` | | `licenseType` | `undefined` \| `LicenseType` | | `minReplicas` | `undefined` \| `number` | | `modelTypeId` | `undefined` \| `string` | | `modelVersionIdsList` | `undefined` \| `string`[] | | `name` | `undefined` \| `string` | | `outputFieldsList` | `undefined` \| `string`[] | | `page` | `undefined` \| `number` | | `query` | `undefined` \| `string` | | `search` | `undefined` \| `string` | | `showReplicas` | `undefined` \| `boolean` | | `sortAscending` | `undefined` \| `boolean` | | `sortByCreatedAt` | `undefined` \| `boolean` | | `sortByModifiedAt` | `undefined` \| `boolean` | | `sortByName` | `undefined` \| `boolean` | | `sortByNumInputs` | `undefined` \| `boolean` | | `sortByStarCount` | `undefined` \| `boolean` | | `source` | `undefined` \| `number` | | `starredOnly` | `undefined` \| `boolean` | | `toolkitsList` | `undefined` \| `string`[] | | `trainedOnly` | `undefined` \| `boolean` | | `useCasesList` | `undefined` \| `string`[] | **Defined in** [src/client/app.ts:67](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L67) ### ListModulesParam Ƭ **ListModulesParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `additionalFieldsList` | `undefined` \| `string`[] | | `bookmark` | `undefined` \| `boolean` | | `filterByUserId` | `undefined` \| `boolean` | | `name` | `undefined` \| `string` | | `page` | `undefined` \| `number` | | `search` | `undefined` \| `string` | | `sortAscending` | `undefined` \| `boolean` | | `sortByCreatedAt` | `undefined` \| `boolean` | | `sortById` | `undefined` \| `boolean` | | `sortByModifiedAt` | `undefined` \| `boolean` | | `sortByStarCount` | `undefined` \| `boolean` | | `starredOnly` | `undefined` \| `boolean` | **Defined in** [src/client/app.ts:71](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L71) ### ListRunnersRequestParams Ƭ **ListRunnersRequestParams**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `computeClusterId` | `undefined` \| `string` | | `minReplicas` | `undefined` \| `number` | | `modelVersionIdsList` | `undefined` \| `string`[] | | `nodepoolId` | `undefined` \| `string` | | `page` | `undefined` \| `number` | **Defined in** [src/client/user.ts:24](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L24) ### ListWorkflowsParam Ƭ **ListWorkflowsParam**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `additionalFieldsList` | `undefined` \| `string`[] | | `bookmark` | `undefined` \| `boolean` | | `featuredOnly` | `undefined` \| `boolean` | | `id` | `undefined` \| `string` | | `page` | `undefined` \| `number` | | `query` | `undefined` \| `string` | | `search` | `undefined` \| `string` | | `searchTerm` | `undefined` \| `string` | | `sortAscending` | `undefined` \| `boolean` | | `sortByCreatedAt` | `undefined` \| `boolean` | | `sortById` | `undefined` \| `boolean` | | `sortByModifiedAt` | `undefined` \| `boolean` | | `sortByStarCount` | `undefined` \| `boolean` | | `starredOnly` | `undefined` \| `boolean` | **Defined in** [src/client/app.ts:69](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/app.ts#L69) ### UserConfig Ƭ **UserConfig**: `Object` **Type declaration** | Name | Type | | :------ | :------ | | `appId` | `string` | | `base` | `undefined` \| `string` | | `pat` | `string` | | `rootCertificatesPath` | `undefined` \| `string` | | `token` | `undefined` \| `string` | | `ui` | `undefined` \| `string` | | `userId` | `string` | **Defined in** [src/client/user.ts:21](https://github.com/Clarifai/clarifai-nodejs/blob/435d969/src/client/user.ts#L21) --- ## Postman # Postman **Learn how to use Postman with Clarifai APIs** [Postman](https://www.postman.com/) is a popular API client that lets you explore, test, and interact with REST APIs without writing any code. The Clarifai Postman collection gives you instant access to various endpoints across different resource areas — applications, inputs, models, workflows, datasets, search, compute orchestration, and more — with pre-configured authentication and ready-to-use request bodies. :::tip Access Postman collection You can access the Clarifai Postman collection [here](https://documenter.getpostman.com/view/24309294/2sBXijHWuM). ::: ## Prerequisites - An active [Clarifai account](https://clarifai.com/signup) - [Postman](https://www.postman.com/downloads/) desktop app or access to [Postman Web](https://web.postman.com/) - A Personal Access Token (PAT) — see [Step 2](#step-2-get-your-pat) below ## What's in the Collection The collection is organized into the following folders: | Folder | Description | | --- | --- | | **Applications** | Create and manage apps | | **Annotations** | Label inputs and manage annotation filters | | **Collectors** | Auto-capture inputs from external model calls | | **Concepts** | Manage labels, vocabularies, and concept graphs | | **Datasets** | Create datasets, add inputs, and version snapshots | | **Inputs** | Upload images, video, text, and audio | | **Models** | Create, train, evaluate, predict, and export models | | **Search** | Rank by visual similarity and filter by concept or region | | **Workflows** | Chain models into multi-step inference pipelines | | **Pipelines** | Orchestrate multi-step data processing runs | | **Artifacts** | Store and version model weights and config files | | **Secrets** | Manage credentials for compute infrastructure | | **Runners** | Register agents for on-premises model inference | | **Compute Orchestration** | Provision clusters, nodepools, and model deployments | | **Walkthroughs** | End-to-end guides (RAG, visual classifier, and more) | ## Getting Started ### Step 1: Import the Collection Click the button below to fork the Clarifai Postman collection directly into your workspace. [![Run in Postman](https://run.pstmn.io/button.svg)](https://god.gw.postman.com/run-collection/24309294-83bab89c-a68a-4fb1-83b7-0e3c20e2d96a?action=collection%2Ffork&source=rip_markdown&collection-url=entityId%3D24309294-83bab89c-a68a-4fb1-83b7-0e3c20e2d96a%26entityType%3Dcollection%26workspaceId%3D00399af6-b92f-47d8-938f-0cacf755c972) Alternatively, download the collection JSON and import it via **File → Import** in Postman. ### Step 2: Get Your PAT A Personal Access Token (PAT) is required to authenticate all API requests. See the [Personal Access Tokens documentation](https://docs.clarifai.com/control/authentication/pat) to learn how to get your PAT. ### Step 3: Set Your Collection Variables The collection uses **collection-level variables** — authentication and common IDs are defined once and applied automatically to every request. To set your variables: 1. Click the collection name in the Postman sidebar 2. Go to the **Variables** tab 3. Fill in the **Current Value** column for each variable The key variables to set before making any requests: | Variable | Description | | --- | --- | | `key` | Your Personal Access Token | | `user_id` | Your Clarifai username | | `app_id` | The ID of the app you want to work with | | `base_url` | API base URL — pre-set to `https://api.clarifai.com` | Additional variables (`model_id`, `workflow_id`, `input_id`, `dataset_id`, etc.) are used by their respective endpoint groups. Set them as you work through each section. ![Postman Variables tab showing key, user_id, and app_id fields](/img/postman/image-5.png) ### Step 4: Verify Authentication The collection is pre-configured with **collection-level auth** using the `Authorization: Key {{key}}` header. This means your PAT is applied automatically to every request — you don't need to set authentication per-request. To verify: click the collection name → **Authorization** tab. You should see the auth type set to `API Key` with the header value `Key {{key}}`. ![Postman Authorization tab showing Key auth configured at collection level](/img/postman/image-1.png) ### Step 5: Make Your First Request To confirm everything is working: 1. Open the **Applications** folder in the collection 2. Select **List Applications** 3. Click **Send** A successful response returns HTTP `200` with a JSON body containing your applications: ```json { "status": { "code": 10000, "description": "Ok" }, "apps": [...] } ``` ![Postman showing a 200 OK response from the List Applications endpoint](/img/postman/image-2.png) ## Using Environments (Optional) Instead of editing collection variables directly, you can use a **Postman Environment** to manage values separately. This is useful if you work with multiple Clarifai accounts or apps. To create an environment: 1. Click **Environments** in the left sidebar → **+** (New Environment) 2. Add the same variable names (`key`, `user_id`, `app_id`, etc.) with your values 3. Select the environment from the dropdown in the top-right corner of Postman Environment values take precedence over collection variable values when an environment is active. ## Generating Code Snippets Postman can generate equivalent code in Python, Node.js, cURL, Java, and other languages for any request in the collection. To generate a snippet: 1. Open any request 2. Click the **``** (Code) icon on the right panel 3. Select your preferred language You can use the generated snippet to make the same request programmatically via the Clarifai REST API. --- ## Python API Reference # Python API Reference **Clarifai Python SDK API Reference** This is the API Reference documentation extracted from the [source code](https://github.com/Clarifai/clarifai-python/tree/master/clarifai/client). ## [App](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/app.py) ```python class App(url=None, app_id=None, user_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` App is a class that provides access to Clarifai API endpoints related to App information. **Parameters:** - `url` (*str*) - The URL to initialize the app object - `app_id` (*str*) - The App ID for the App to interact with - `user_id` (*str*) - The user ID of the owner - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file - `**kwargs` - Additional keyword arguments: - `name` (*str*) - The name of the app - `description` (*str*) - The description of the app ### App.create_concepts ```python App.create_concepts(concept_ids, concepts=[]) ``` Add concepts to the app. **Parameters:** - `concept_ids` (*List[str]*) - List of concept IDs to add - `concepts` (*List[str]*) - Optional list of concept names **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") app.create_concepts(concept_ids=["concept_id_1", "concept_id_2"]) ``` ### App.create_concept_relations ```python App.create_concept_relations(subject_concept_id, object_concept_ids, predicates) ``` Creates concept relations between concepts of the app. **Parameters:** - `subject_concept_id` (*str*) - The concept ID of the subject concept - `object_concept_ids` (*List[str]*) - List of concept IDs of object concepts - `predicates` (*List[str]*) - List of predicates corresponding to each relation **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") app.create_concept_relations( subject_concept_id="subject_concept_id", object_concept_ids=["object_concept_id_1", "object_concept_id_2"], predicates=["hypernym", "synonym"] ) ``` ### App.create_dataset ```python App.create_dataset(dataset_id, **kwargs) ``` Creates a dataset for the app. **Parameters:** - `dataset_id` (*str*) - The dataset ID for the dataset to create - `**kwargs` - Additional keyword arguments passed to the Dataset **Returns:** `Dataset` - A Dataset object for the specified dataset ID **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") dataset = app.create_dataset(dataset_id="dataset_id") ``` ### App.create_model ```python App.create_model(model_id, **kwargs) ``` Creates a model for the app. **Parameters:** - `model_id` (*str*) - The model ID for the model to create - `**kwargs` - Additional keyword arguments passed to the Model **Returns:** `Model` - A Model object for the specified model ID **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") model = app.create_model(model_id="model_id") ``` ### App.create_workflow ```python App.create_workflow(config_filepath, generate_new_id=False, display=True) ``` Creates a workflow for the app from a YAML config file. **Parameters:** - `config_filepath` (*str*) - The path to the yaml workflow config file - `generate_new_id` (*bool*) - If True, generate a new workflow ID. Default False - `display` (*bool*) - If True, display the workflow nodes tree. Default True **Returns:** `Workflow` - A Workflow object for the specified workflow config **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") workflow = app.create_workflow(config_filepath="config.yml") ``` ### App.dataset ```python App.dataset(dataset_id, dataset_version_id=None, **kwargs) ``` Returns a Dataset object for the existing dataset ID. **Parameters:** - `dataset_id` (*str*) - The dataset ID for the dataset to interact with - `dataset_version_id` (*str*) - The version ID for the dataset version to interact with **Returns:** `Dataset` - A Dataset object for the existing dataset ID **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") dataset = app.dataset(dataset_id="dataset_id") ``` ### App.delete_concept_relations ```python App.delete_concept_relations(concept_id, concept_relation_ids=[]) ``` Delete concept relations of a concept of the app. **Parameters:** - `concept_id` (*str*) - The concept ID of the concept to delete relations for - `concept_relation_ids` (*List[str]*) - Optional list of concept relation IDs to delete **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") app.delete_concept_relations(concept_id="concept_id", concept_relation_ids=["id_1", "id_2"]) ``` ### App.delete_dataset ```python App.delete_dataset(dataset_id) ``` Deletes a dataset for the user. **Parameters:** - `dataset_id` (*str*) - The dataset ID for the app to delete **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") app.delete_dataset(dataset_id="dataset_id") ``` ### App.delete_model ```python App.delete_model(model_id) ``` Deletes a model for the user. **Parameters:** - `model_id` (*str*) - The model ID for the app to delete **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") app.delete_model(model_id="model_id") ``` ### App.delete_workflow ```python App.delete_workflow(workflow_id) ``` Deletes a workflow for the user. **Parameters:** - `workflow_id` (*str*) - The workflow ID for the app to delete **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") app.delete_workflow(workflow_id="workflow_id") ``` ### App.get_input_count ```python App.get_input_count() ``` Get count of all the inputs in the app. **Returns:** `int` - Count of inputs in the app **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") input_count = app.get_input_count() ``` ### App.inputs ```python App.inputs() ``` Returns an Inputs object for the app. **Returns:** `Inputs` - An Inputs object ### App.list_concepts ```python App.list_concepts(page_no=None, per_page=None) ``` Lists all the concepts for the app. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Concept` - Concept objects in the app **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") all_concepts = list(app.list_concepts()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### App.list_datasets ```python App.list_datasets(page_no=None, per_page=None) ``` Lists all the datasets for the app. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Dataset` - Dataset objects for the datasets in the app **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") all_datasets = list(app.list_datasets()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### App.list_models ```python App.list_models(filter_by={}, only_in_app=True, page_no=None, per_page=None) ``` Lists all the available models for the user. **Parameters:** - `filter_by` (*dict*) - A dictionary of filters to apply to the list of models - `only_in_app` (*bool*) - If True, only return models that are in the app. Default True - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Model` - Model objects for the models in the app **Example:** ```python from clarifai.client.user import User app = User(user_id="user_id").app(app_id="app_id") all_models = list(app.list_models()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### App.list_pipeline_steps ```python App.list_pipeline_steps(pipeline_id=None, filter_by={}, only_in_app=True, page_no=None, per_page=None) ``` Lists all the pipeline steps for the user. **Parameters:** - `pipeline_id` (*str*) - If provided, only list pipeline steps from this pipeline - `filter_by` (*dict*) - A dictionary of filters to apply to the list - `only_in_app` (*bool*) - If True, only return pipeline steps that are in the app. Default True - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `PipelineStep` - PipelineStep objects **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") all_pipeline_steps = list(app.list_pipeline_steps()) ``` ### App.list_pipelines ```python App.list_pipelines(filter_by={}, only_in_app=True, page_no=None, per_page=None) ``` Lists all the pipelines for the user. **Parameters:** - `filter_by` (*dict*) - A dictionary of filters to apply to the list of pipelines - `only_in_app` (*bool*) - If True, only return pipelines that are in the app. Default True - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Pipeline` - Pipeline objects **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") all_pipelines = list(app.list_pipelines()) ``` ### App.list_trainable_model_types ```python App.list_trainable_model_types() ``` Lists all the trainable model types. **Returns:** `List[str]` - List of all trainable model types **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") print(app.list_trainable_model_types()) ``` ### App.list_workflows ```python App.list_workflows(filter_by={}, only_in_app=True, page_no=None, per_page=None) ``` Lists all the available workflows for the user. **Parameters:** - `filter_by` (*dict*) - A dictionary of filters to apply to the list of workflows - `only_in_app` (*bool*) - If True, only return workflows that are in the app. Default True - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Workflow` - Workflow objects for the workflows in the app **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") all_workflows = list(app.list_workflows()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### App.model ```python App.model(model_id, model_version={'id': ""}, **kwargs) ``` Returns a Model object for the existing model ID. **Parameters:** - `model_id` (*str*) - The model ID for the model to interact with - `model_version` (*Dict*) - The model version ID for the model version to interact with **Returns:** `Model` - A Model object for the existing model ID **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") model_v1 = app.model(model_id="model_id", model_version={"id": "model_version_id"}) ``` ### App.patch_dataset ```python App.patch_dataset(dataset_id, action='merge', **kwargs) ``` Patches a dataset for the app. **Parameters:** - `dataset_id` (*str*) - The dataset ID for the dataset to patch - `action` (*str*) - The action to perform: merge/overwrite/remove. Default 'merge' - `**kwargs` - Additional keyword arguments to patch the Dataset **Returns:** `Dataset` - A Dataset object for the specified dataset ID ### App.patch_model ```python App.patch_model(model_id, action='merge', **kwargs) ``` Patches a model of the app. **Parameters:** - `model_id` (*str*) - The model ID of the model to patch - `action` (*str*) - The action to perform: merge/overwrite/remove. Default 'merge' - `**kwargs` - Additional keyword arguments to patch the Model **Returns:** `Model` - A Model object of the specified model ID ### App.patch_workflow ```python App.patch_workflow(workflow_id, action='merge', config_filepath=None, **kwargs) ``` Patches a workflow of the app. **Parameters:** - `workflow_id` (*str*) - The Workflow ID of the workflow to patch - `action` (*str*) - The action to perform: merge/overwrite/remove. Default 'merge' - `config_filepath` (*str*) - The path to the yaml workflow config file - `**kwargs` - Additional keyword arguments to patch the Workflow **Returns:** `Workflow` - A Workflow object of the specified workflow ID ### App.search ```python App.search(**kwargs) ``` Returns a Search object for the user and app ID. **Parameters:** - `**kwargs` - Arguments passed to the Search class (see Search class) **Returns:** `Search` - A Search object for the user and app ID **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") search_client = app.search(top_k=12, metric="euclidean") ``` ### App.search_concept_relations ```python App.search_concept_relations(concept_id=None, predicate=None, page_no=None, per_page=None, show_tree=False) ``` List all the concept relations of the app. **Parameters:** - `concept_id` (*str*) - The concept ID to filter the concept relations - `predicate` (*str*) - Type of relation to filter: 'hypernym', 'hyponym', or 'synonym' - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page - `show_tree` (*bool*) - If True, prints a rich tree representation of concept relations **Yields:** `ConceptRelation` - ConceptRelation objects in the app **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") all_concept_relations = list(app.search_concept_relations()) ``` ### App.workflow ```python App.workflow(workflow_id, **kwargs) ``` Returns a Workflow object for the existing workflow ID. **Parameters:** - `workflow_id` (*str*) - The workflow ID for the workflow to interact with **Returns:** `Workflow` - A Workflow object for the existing workflow ID **Example:** ```python from clarifai.client.app import App app = App(app_id="app_id", user_id="user_id") workflow = app.workflow(workflow_id="workflow_id") ``` --- ## [Artifact](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/artifact.py) ```python class Artifact(artifact_id="", user_id="", app_id="", **kwargs) ``` Artifact client for managing artifacts in Clarifai. **Parameters:** - `artifact_id` (*str*) - The artifact ID - `user_id` (*str*) - The user ID - `app_id` (*str*) - The app ID - `**kwargs` - Additional keyword arguments passed to the BaseClient ### Artifact.create ```python Artifact.create(artifact_id="", user_id="", app_id="") ``` Create a new artifact. **Parameters:** - `artifact_id` (*str*) - The artifact ID to create. Defaults to the ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization **Returns:** `Artifact` - An Artifact object for the created artifact **Raises:** `Exception` if artifact creation fails ### Artifact.delete ```python Artifact.delete(artifact_id="", user_id="", app_id="", **kwargs) ``` Delete this artifact. **Parameters:** - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization **Returns:** `bool` - True if deletion was successful **Raises:** `Exception` if artifact deletion fails ### Artifact.get ```python Artifact.get(artifact_id="", user_id="", app_id="", **kwargs) ``` Get information about this artifact. **Parameters:** - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization **Returns:** `resources_pb2.Artifact` - The artifact protobuf object **Raises:** `Exception` if artifact retrieval fails ### Artifact.list ```python Artifact.list(user_id="", app_id="", page=1, per_page=20, **kwargs) ``` List artifacts in an app. **Parameters:** - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization - `page` (*int*) - The page number for pagination. Default 1 - `per_page` (*int*) - The number of results per page. Default 20 **Yields:** `resources_pb2.Artifact` - Artifact protobuf objects **Raises:** `Exception` if listing fails --- ## [ArtifactVersion](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/artifact_version.py) ```python class ArtifactVersion(artifact_id="", version_id="", user_id="", app_id="", **kwargs) ``` ArtifactVersion client for managing artifact versions in Clarifai. **Parameters:** - `artifact_id` (*str*) - The artifact ID - `version_id` (*str*) - The artifact version ID - `user_id` (*str*) - The user ID - `app_id` (*str*) - The app ID - `**kwargs` - Additional keyword arguments passed to the BaseClient ### ArtifactVersion.delete ```python ArtifactVersion.delete(artifact_id="", version_id="", user_id="", app_id="", **kwargs) ``` Delete this artifact version. **Parameters:** - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `version_id` (*str*) - The artifact version ID. Defaults to the version ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization **Returns:** `bool` - True if deletion was successful **Raises:** `Exception` if deletion fails ### ArtifactVersion.download ```python ArtifactVersion.download(output_path="", artifact_id="", version_id="", user_id="", app_id="", force=False) ``` Download an artifact version with automatic progress tracking and retry logic. **Parameters:** - `output_path` (*str*) - The local path to save the file. If not provided, uses the original filename - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `version_id` (*str*) - The artifact version ID. Defaults to the version ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization - `force` (*bool*) - Whether to overwrite existing files without prompting. Default False **Returns:** `str` - The path where the file was saved **Raises:** `Exception` if download fails ### ArtifactVersion.get ```python ArtifactVersion.get(artifact_id="", version_id="", user_id="", app_id="", **kwargs) ``` Get information about this artifact version. **Parameters:** - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `version_id` (*str*) - The artifact version ID. Defaults to the version ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization **Returns:** `resources_pb2.ArtifactVersion` - The artifact version protobuf object **Raises:** `Exception` if retrieval fails ### ArtifactVersion.list ```python ArtifactVersion.list(artifact_id="", user_id="", app_id="", page=1, per_page=20, **kwargs) ``` List artifact versions for this artifact. **Parameters:** - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization - `page` (*int*) - The page number for pagination. Default 1 - `per_page` (*int*) - The number of results per page. Default 20 **Yields:** `resources_pb2.ArtifactVersion` - ArtifactVersion protobuf objects **Raises:** `Exception` if listing fails ### ArtifactVersion.upload ```python ArtifactVersion.upload(file_path, artifact_id="", description="", visibility=None, expires_at=None, version_id="", user_id="", app_id="") ``` Upload a file as a new artifact version with streaming support. **Parameters:** - `file_path` (*str*) - Path to the file to upload - `artifact_id` (*str*) - The artifact ID. Defaults to the artifact ID from initialization - `description` (*str*) - Description for the artifact version - `visibility` (*str*) - Visibility setting: "private", "public", or "org". Defaults to "private" - `expires_at` (*Timestamp*) - Optional expiration timestamp - `version_id` (*str*) - Optional version ID to assign - `user_id` (*str*) - The user ID. Defaults to the user ID from initialization - `app_id` (*str*) - The app ID. Defaults to the app ID from initialization **Returns:** `ArtifactVersion` - An ArtifactVersion object for the uploaded version **Raises:** `Exception` if upload fails --- ## [ComputeCluster](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/compute_cluster.py) ```python class ComputeCluster(compute_cluster_id=None, user_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` ComputeCluster is a class that provides access to Clarifai API endpoints related to Compute Cluster information. **Parameters:** - `compute_cluster_id` (*str*) - The ComputeCluster ID for the ComputeCluster to interact with - `user_id` (*str*) - The user ID of the user - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### ComputeCluster.create_nodepool ```python ComputeCluster.create_nodepool(config_filepath=None, nodepool_id=None, nodepool_config=None) ``` Creates a nodepool for the compute cluster. **Parameters:** - `config_filepath` (*str*) - The path to the nodepool config file - `nodepool_id` (*str*) - New nodepool ID for the nodepool to create - `nodepool_config` (*Dict[str, Any]*) - Optional dictionary containing nodepool configuration **Returns:** `Nodepool` - A Nodepool object for the specified nodepool ID **Example:** ```python from clarifai.client.compute_cluster import ComputeCluster compute_cluster = ComputeCluster(compute_cluster_id="compute_cluster_id", user_id="user_id") nodepool = compute_cluster.create_nodepool(config_filepath="config.yml") ``` ### ComputeCluster.delete_nodepools ```python ComputeCluster.delete_nodepools(nodepool_ids) ``` Deletes list of nodepools for the compute cluster. **Parameters:** - `nodepool_ids` (*List[str]*) - The nodepool IDs of the compute cluster to delete **Example:** ```python from clarifai.client.compute_cluster import ComputeCluster compute_cluster = ComputeCluster(compute_cluster_id="compute_cluster_id", user_id="user_id") compute_cluster.delete_nodepools(nodepool_ids=["nodepool_id1", "nodepool_id2"]) ``` ### ComputeCluster.list_nodepools ```python ComputeCluster.list_nodepools(page_no=None, per_page=None) ``` Lists all the available nodepools of the compute cluster. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Nodepool` - Nodepool objects for the nodepools in the compute cluster **Example:** ```python from clarifai.client.compute_cluster import ComputeCluster compute_cluster = ComputeCluster(compute_cluster_id="compute_cluster_id", user_id="user_id") all_nodepools = list(compute_cluster.list_nodepools()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### ComputeCluster.nodepool ```python ComputeCluster.nodepool(nodepool_id) ``` Returns a Nodepool object for the existing nodepool ID. **Parameters:** - `nodepool_id` (*str*) - The nodepool ID for the nodepool to interact with **Returns:** `Nodepool` - A Nodepool object for the existing nodepool ID **Example:** ```python from clarifai.client.compute_cluster import ComputeCluster compute_cluster = ComputeCluster(compute_cluster_id="compute_cluster_id", user_id="user_id") nodepool = compute_cluster.nodepool(nodepool_id="nodepool_id") ``` --- ## [Dataset](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/dataset.py) ```python class Dataset(url=None, dataset_id=None, dataset_version_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Dataset is a class that provides access to Clarifai API endpoints related to Dataset information. **Parameters:** - `url` (*str*) - The URL to initialize the dataset object - `dataset_id` (*str*) - The Dataset ID within the App to interact with - `dataset_version_id` (*str*) - The Dataset Version ID within the Dataset to interact with - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### Dataset.create_version ```python Dataset.create_version(**kwargs) ``` Creates a dataset version for the Dataset. **Parameters:** - `**kwargs` - Additional keyword arguments passed to Dataset Version: - `description` (*str*) - The description of the dataset version - `metadata` (*Dict[str, Any]*) - The metadata dictionary for the dataset version **Returns:** `Dataset` - A Dataset object for the newly created dataset version **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(dataset_id='dataset_id', user_id='user_id', app_id='app_id') dataset_version = dataset.create_version(description='dataset_version_description') ``` ### Dataset.delete_version ```python Dataset.delete_version(version_id) ``` Deletes a dataset version for the Dataset. **Parameters:** - `version_id` (*str*) - The version ID to delete **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(dataset_id='dataset_id', user_id='user_id', app_id='app_id') dataset.delete_version(version_id='version_id') ``` ### Dataset.export ```python Dataset.export(save_path, archive_url=None, local_archive_path=None, split='all', num_workers=4) ``` Exports the Clarifai protobuf dataset to a local archive. **Parameters:** - `save_path` (*str*) - The path to save the archive to - `archive_url` (*str*) - The URL to the Clarifai protobuf archive - `local_archive_path` (*str*) - The path to the local Clarifai protobuf archive - `split` (*str*) - Export dataset inputs in the directory format `{split}/{input_type}`. Default 'all' - `num_workers` (*int*) - Number of workers to use for downloading the archive. Default 4 **Example:** ```python from clarifai.client.dataset import Dataset Dataset().export(save_path='output.zip') ``` ### Dataset.get_upload_status ```python Dataset.get_upload_status(dataloader=None, delete_version=False, timeout=600, pre_upload_stats=None, pre_upload=False) ``` Creates a new dataset version and displays the upload status of the dataset. **Parameters:** - `dataloader` (*Type[ClarifaiDataLoader]*) - ClarifaiDataLoader object - `delete_version` (*bool*) - True if you want to delete the version after getting the upload status - `timeout` (*int*) - Timeout in seconds. Default 600 - `pre_upload_stats` (*Tuple*) - The pre upload stats for the dataset - `pre_upload` (*bool*) - True if you want to get the pre upload stats for the dataset **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(dataset_id='dataset_id', user_id='user_id', app_id='app_id') dataset.get_upload_status(dataloader) ``` ### Dataset.list_inputs ```python Dataset.list_inputs(page_no=None, per_page=None, input_type=None) ``` Lists all the inputs for the dataset. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page - `input_type` (*str*) - The type of input to list: 'image', 'video', 'audio', 'text' **Yields:** `Input` - Input objects in the dataset **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(dataset_id='dataset_id', user_id='user_id', app_id='app_id') all_dataset_inputs = list(dataset.list_inputs()) ``` ### Dataset.list_versions ```python Dataset.list_versions(page_no=None, per_page=None) ``` Lists all the versions for the dataset. **Parameters:** - `page_no` (*int*) - The page number to list. If None, lists all pages - `per_page` (*int*) - The number of items per page. If None, uses default **Yields:** `Dataset` - Dataset objects for the versions of the dataset **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(dataset_id='dataset_id', user_id='user_id', app_id='app_id') all_dataset_versions = list(dataset.list_versions()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### Dataset.merge_dataset ```python Dataset.merge_dataset(merge_dataset_id) ``` Merges another dataset into the current dataset. **Parameters:** - `merge_dataset_id` (*str*) - The dataset ID of the dataset to merge **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(dataset_id='dataset_id', user_id='user_id', app_id='app_id') dataset.merge_dataset(merge_dataset_id='merge_dataset_id') ``` ### Dataset.retry_upload_from_logs ```python Dataset.retry_upload_from_logs(log_file_path, dataloader, retry_duplicates=False, log_warnings=False, **kwargs) ``` Retries failed uploads from the log file. **Parameters:** - `log_file_path` (*str*) - Path to the log file - `dataloader` (*Type[ClarifaiDataLoader]*) - ClarifaiDataLoader object - `retry_duplicates` (*bool*) - True if you want to retry duplicate inputs. Default False - `log_warnings` (*bool*) - True if you want to save log warnings in a file. Default False ### Dataset.upload_dataset ```python Dataset.upload_dataset(dataloader, batch_size=32, get_upload_status=False, log_warnings=False, **kwargs) ``` Uploads a dataset to the app. **Parameters:** - `dataloader` (*Type[ClarifaiDataLoader]*) - ClarifaiDataLoader object - `batch_size` (*int*) - Batch size for concurrent upload of inputs and annotations (max: 128). Default 32 - `get_upload_status` (*bool*) - True if you want to get the upload status of the dataset. Default False - `log_warnings` (*bool*) - True if you want to save log warnings in a file. Default False ### Dataset.upload_from_csv ```python Dataset.upload_from_csv(csv_path, input_type='text', csv_type=None, labels=True, batch_size=128) ``` Uploads dataset from a csv file. **Parameters:** - `csv_path` (*str*) - Path to the csv file - `input_type` (*str*) - Type of the dataset: text, image. Default 'text' - `csv_type` (*str*) - Type of the csv file: raw, url, file_path - `labels` (*bool*) - True if csv file has labels column. Default True - `batch_size` (*int*) - Batch size for concurrent upload. Default 128 **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(user_id='user_id', app_id='demo_app', dataset_id='demo_dataset') dataset.upload_from_csv(csv_path='csv_path', input_type='text', csv_type='raw', labels=True) ``` **Note:** CSV file supports 'inputid', 'input', 'concepts', 'metadata', 'geopoints' columns. All data in the CSV should be in double quotes. Metadata should be in single quotes format. Geopoints should be in "long,lat" format. ### Dataset.upload_from_folder ```python Dataset.upload_from_folder(folder_path, input_type, labels=False, batch_size=128) ``` Upload dataset from folder. **Parameters:** - `folder_path` (*str*) - Path to the folder containing images or text files - `input_type` (*str*) - Type of the dataset: text, image - `labels` (*bool*) - True if folder name is the label for the inputs. Default False - `batch_size` (*int*) - Batch size for concurrent upload. Default 128 **Example:** ```python from clarifai.client.dataset import Dataset dataset = Dataset(user_id='user_id', app_id='demo_app', dataset_id='demo_dataset') dataset.upload_from_folder(folder_path='folder_path', input_type='text', labels=True) ``` **Note:** The filename is used as the input_id. --- ## [Deployment](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/deployment.py) ```python class Deployment(deployment_id=None, user_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Deployment is a class that provides access to Clarifai API endpoints related to Deployment information. **Parameters:** - `deployment_id` (*str*) - The Deployment ID for the Deployment to interact with - `user_id` (*str*) - The user ID of the user - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### Deployment.get_runner_selector ```python Deployment.get_runner_selector(user_id, deployment_id) # static method ``` Returns a RunnerSelector object for the given deployment_id. **Parameters:** - `user_id` (*str*) - The user ID for the deployment - `deployment_id` (*str*) - The deployment ID for the deployment **Returns:** `resources_pb2.RunnerSelector` - A RunnerSelector object for the given deployment_id --- ## [Inputs](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/input.py) ```python class Inputs(user_id=None, app_id=None, logger_level="INFO", base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Inputs is a class that provides access to Clarifai API endpoints related to Input information. **Parameters:** - `user_id` (*str*) - A user ID for authentication - `app_id` (*str*) - An app ID for the application to interact with - `logger_level` (*str*) - The logger level. Default "INFO" - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### Inputs.delete_annotations ```python Inputs.delete_annotations(input_ids, annotation_ids=[]) ``` Delete list of annotations of input objects from the app. **Parameters:** - `input_ids` (*List[str]*) - List of input IDs for which annotations to delete - `annotation_ids` (*List[str]*) - Optional list of annotation IDs to delete **Example:** ```python from clarifai.client.user import User input_obj = User(user_id="user_id").app(app_id="app_id").inputs() input_obj.delete_annotations(input_ids=['input_id_1', 'input_id_2']) ``` **Note:** 'annotation_ids' are optional but if provided, the number and order must match 'input_ids'. ### Inputs.delete_inputs ```python Inputs.delete_inputs(inputs) ``` Delete list of input objects from the app. **Parameters:** - `inputs` (*List[Input]*) - List of input objects to delete **Example:** ```python from clarifai.client.user import User input_obj = User(user_id="user_id").app(app_id="app_id").inputs() input_obj.delete_inputs(list(input_obj.list_inputs())) ``` ### Inputs.download_inputs ```python Inputs.download_inputs(inputs) ``` Download list of input objects from the app. **Parameters:** - `inputs` (*List[Input]*) - List of input objects to download **Returns:** `List[bytes]` - List of bytes for each input **Example:** ```python from clarifai.client.user import User input_obj = User(user_id="user_id").app(app_id="app_id").inputs() input_obj.download_inputs(list(input_obj.list_inputs())) ``` ### Inputs.get_bbox_proto ```python Inputs.get_bbox_proto(input_id, label, bbox, label_id=None, annot_id=None) # static method ``` Create an annotation proto for each bounding box, label input pair. **Parameters:** - `input_id` (*str*) - The input ID for the annotation to create - `label` (*str*) - Annotation label name - `bbox` (*List*) - A list of a single bbox's coordinates: [xmin, ymin, xmax, ymax] - `label_id` (*str*) - Annotation label ID - `annot_id` (*str*) - Annotation ID **Returns:** `Annotation` - An annotation object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs Inputs.get_bbox_proto(input_id='demo', label='demo', bbox=[x_min, y_min, x_max, y_max]) ``` ### Inputs.get_image_inputs_from_folder ```python Inputs.get_image_inputs_from_folder(folder_path, dataset_id=None, labels=False) # static method ``` Create input protos for image data type from folder. **Parameters:** - `folder_path` (*str*) - Path to the folder containing images - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to - `labels` (*bool*) - If True, use folder name as label. Default False **Returns:** `List[Input]` - A list of Input objects for the specified folder **Example:** ```python from clarifai.client.input import Inputs input_protos = Inputs.get_image_inputs_from_folder(folder_path='demo_folder') ``` ### Inputs.get_input ```python Inputs.get_input(input_id) ``` Get Input object of input with input_id provided from the app. **Parameters:** - `input_id` (*str*) - The input ID for the annotation to get **Returns:** `Input` - An Input object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs input_obj = Inputs(user_id='user_id', app_id='demo_app') input_obj.get_input(input_id='demo') ``` ### Inputs.get_input_from_bytes ```python Inputs.get_input_from_bytes(input_id, image_bytes=None, video_bytes=None, audio_bytes=None, text_bytes=None, dataset_id=None, **kwargs) # static method ``` Create input proto from bytes. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `image_bytes` (*bytes*) - The bytes for the image - `video_bytes` (*bytes*) - The bytes for the video - `audio_bytes` (*bytes*) - The bytes for the audio - `text_bytes` (*bytes*) - The bytes for the text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `Input` - An Input object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs image = open('demo.jpg', 'rb').read() video = open('demo.mp4', 'rb').read() input_proto = Inputs.get_input_from_bytes(input_id='demo', image_bytes=image, video_bytes=video) ``` ### Inputs.get_input_from_file ```python Inputs.get_input_from_file(input_id, image_file=None, video_file=None, audio_file=None, text_file=None, dataset_id=None, **kwargs) # static method ``` Create input proto from files. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `image_file` (*str*) - The file path for the image - `video_file` (*str*) - The file path for the video - `audio_file` (*str*) - The file path for the audio - `text_file` (*str*) - The file path for the text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `Input` - An Input object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs input_proto = Inputs.get_input_from_file(input_id='demo', video_file='file_path') ``` ### Inputs.get_input_from_url ```python Inputs.get_input_from_url(input_id, image_url=None, video_url=None, audio_url=None, text_url=None, dataset_id=None, **kwargs) # static method ``` Create input proto from url. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `image_url` (*str*) - The url for the image - `video_url` (*str*) - The url for the video - `audio_url` (*str*) - The url for the audio - `text_url` (*str*) - The url for the text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `Input` - An Input object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs input_proto = Inputs.get_input_from_url(input_id='demo', image_url='https://samples.clarifai.com/metro-north.jpg') ``` ### Inputs.get_inputs_from_csv ```python Inputs.get_inputs_from_csv(csv_path, input_type='text', csv_type='raw', dataset_id=None, labels=True) # static method ``` Create input protos from csv. **Parameters:** - `csv_path` (*str*) - Path to the csv file - `input_type` (*str*) - Type of input: 'text', 'image', 'video', 'audio'. Default 'text' - `csv_type` (*str*) - Type of csv file: 'raw', 'url', 'file_path'. Default 'raw' - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to - `labels` (*bool*) - True if csv file has labels column. Default True **Returns:** `List[Input]` - List of input protos **Example:** ```python from clarifai.client.input import Inputs input_protos = Inputs.get_inputs_from_csv(csv_path='filepath', input_type='text', csv_type='raw') ``` ### Inputs.get_mask_proto ```python Inputs.get_mask_proto(input_id, label, polygons, label_id=None, annot_id=None) # static method ``` Create an annotation proto for each polygon box, label input pair. **Parameters:** - `input_id` (*str*) - The input ID for the annotation to create - `label` (*str*) - Annotation label name - `polygons` (*List[List[float]]*) - Polygon x,y points iterable - `label_id` (*str*) - Annotation label ID - `annot_id` (*str*) - Annotation ID **Returns:** `Annotation` - An annotation object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs Inputs.get_mask_proto(input_id='demo', label='demo', polygons=[[[x,y],...,[x,y]],...]) ``` ### Inputs.get_multimodal_input ```python Inputs.get_multimodal_input(input_id, raw_text=None, text_bytes=None, image_url=None, image_bytes=None, dataset_id=None, **kwargs) # static method ``` Create input proto for text and image from bytes or url. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `raw_text` (*str*) - The raw text input - `text_bytes` (*bytes*) - The bytes for the text - `image_url` (*str*) - The url for the image - `image_bytes` (*bytes*) - The bytes for the image - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `Input` - An Input object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs input_proto = Inputs.get_multimodal_input( input_id='demo', raw_text='What time of day is it?', image_url='https://samples.clarifai.com/metro-north.jpg' ) ``` ### Inputs.get_text_input ```python Inputs.get_text_input(input_id, raw_text, dataset_id=None, **kwargs) # static method ``` Create input proto for text data type from raw text. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `raw_text` (*str*) - The raw text input - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `Input` - An Input object for the specified input ID **Example:** ```python from clarifai.client.input import Inputs input_proto = Inputs.get_text_input(input_id='demo', raw_text='This is a test') ``` ### Inputs.get_text_inputs_from_folder ```python Inputs.get_text_inputs_from_folder(folder_path, dataset_id=None, labels=False) # static method ``` Create input protos for text data type from folder. **Parameters:** - `folder_path` (*str*) - Path to the folder containing text files - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to - `labels` (*bool*) - If True, use folder name as label. Default False **Returns:** `List[Input]` - A list of Input objects for the specified folder **Example:** ```python from clarifai.client.input import Inputs input_protos = Inputs.get_text_inputs_from_folder(folder_path='demo_folder') ``` ### Inputs.list_annotations ```python Inputs.list_annotations(batch_input=None, page_no=None, per_page=None) ``` Lists all the annotations for the app. **Parameters:** - `batch_input` (*List[Input]*) - The input objects to list annotations from - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Annotation` - Annotation objects for the app **Example:** ```python from clarifai.client.user import User input_obj = User(user_id="user_id").app(app_id="app_id").inputs() all_inputs = list(input_obj.list_inputs(input_type='image')) all_annotations = list(input_obj.list_annotations(batch_input=all_inputs)) ``` **Note:** If `batch_input` is not given, lists all the annotations for the app. ### Inputs.list_inputs ```python Inputs.list_inputs(dataset_id=None, page_no=None, per_page=None, input_type=None) ``` Lists all the inputs for the app. **Parameters:** - `dataset_id` (*str*) - The dataset ID for the dataset to list inputs from - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page - `input_type` (*str*) - The type of input to list: 'image', 'video', 'audio', 'text' **Yields:** `Input` - Input objects for the app **Example:** ```python from clarifai.client.user import User input_obj = User(user_id="user_id").app(app_id="app_id").inputs() all_inputs = list(input_obj.list_inputs(input_type='image')) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### Inputs.patch_annotations ```python Inputs.patch_annotations(batch_annot, action='merge') ``` Patch image annotations to app. **Parameters:** - `batch_annot` (*List[Annotation]*) - Annotation batch protos - `action` (*str*) - Action to perform: 'merge', 'overwrite', 'remove'. Default 'merge' ### Inputs.patch_concepts ```python Inputs.patch_concepts(concept_ids, labels=[], values=[], action='overwrite') ``` Patch concepts to app. **Parameters:** - `concept_ids` (*List[str]*) - A list of concept IDs - `labels` (*List[str]*) - A list of label names - `values` (*List[float]*) - Concept values - `action` (*str*) - Action to perform. Default 'overwrite' ### Inputs.patch_inputs ```python Inputs.patch_inputs(inputs, action='merge') ``` Patch list of input objects to the app. **Parameters:** - `inputs` (*List[Input]*) - List of input objects to patch - `action` (*str*) - Action to perform: 'merge', 'overwrite', 'remove'. Default 'merge' ### Inputs.upload_annotations ```python Inputs.upload_annotations(batch_annot, show_log=True) ``` Upload image annotations to app. **Parameters:** - `batch_annot` (*List[Annotation]*) - Annotation batch protos - `show_log` (*bool*) - Show upload status log. Default True **Returns:** `List[Annotation]` - Failed annotation uploads (for retry) ### Inputs.upload_from_bytes ```python Inputs.upload_from_bytes(input_id, image_bytes=None, video_bytes=None, audio_bytes=None, text_bytes=None, dataset_id=None, **kwargs) ``` Upload input from bytes. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `image_bytes` (*bytes*) - The bytes for the image - `video_bytes` (*bytes*) - The bytes for the video - `audio_bytes` (*bytes*) - The bytes for the audio - `text_bytes` (*bytes*) - The bytes for the text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `str` - Job ID for the upload request **Example:** ```python from clarifai.client.input import Inputs input_obj = Inputs(user_id='user_id', app_id='demo_app') image = open('demo.jpg', 'rb').read() input_obj.upload_from_bytes(input_id='demo', image_bytes=image) ``` ### Inputs.upload_from_file ```python Inputs.upload_from_file(input_id, image_file=None, video_file=None, audio_file=None, text_file=None, dataset_id=None, **kwargs) ``` Upload input from file. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `image_file` (*str*) - The file for the image - `video_file` (*str*) - The file for the video - `audio_file` (*str*) - The file for the audio - `text_file` (*str*) - The file for the text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `str` - Job ID for the upload request **Example:** ```python from clarifai.client.input import Inputs input_obj = Inputs(user_id='user_id', app_id='demo_app') input_obj.upload_from_file(input_id='demo', audio_file='demo.mp3') ``` ### Inputs.upload_from_url ```python Inputs.upload_from_url(input_id, image_url=None, video_url=None, audio_url=None, text_url=None, dataset_id=None, **kwargs) ``` Upload input from url. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `image_url` (*str*) - The url for the image - `video_url` (*str*) - The url for the video - `audio_url` (*str*) - The url for the audio - `text_url` (*str*) - The url for the text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `str` - Job ID for the upload request **Example:** ```python from clarifai.client.input import Inputs input_obj = Inputs(user_id='user_id', app_id='demo_app') input_obj.upload_from_url(input_id='demo', image_url='https://samples.clarifai.com/metro-north.jpg') ``` ### Inputs.upload_inputs ```python Inputs.upload_inputs(inputs, show_log=True) ``` Upload list of input objects to the app. **Parameters:** - `inputs` (*List[Input]*) - List of input objects to upload - `show_log` (*bool*) - Show upload status log. Default True **Returns:** `str` - Job ID for the upload request ### Inputs.upload_text ```python Inputs.upload_text(input_id, raw_text, dataset_id=None, **kwargs) ``` Upload text from raw text. **Parameters:** - `input_id` (*str*) - The input ID for the input to create - `raw_text` (*str*) - The raw text - `dataset_id` (*str*) - The dataset ID for the dataset to add the input to **Returns:** `str` - Job ID for the upload request **Example:** ```python from clarifai.client.input import Inputs input_obj = Inputs(user_id='user_id', app_id='demo_app') input_obj.upload_text(input_id='demo', raw_text='This is a test') ``` --- ## [Lister](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/lister.py) ```python class Lister(page_size=16) ``` Lister class for obtaining paginated results from the Clarifai API. **Parameters:** - `page_size` (*int*) - The default page size for listing resources. Default 16 ### Lister.list_pages_generator ```python Lister.list_pages_generator(endpoint, proto_message, request_data, page_no=None, per_page=None) ``` Lists pages of a resource. **Parameters:** - `endpoint` (*Callable*) - The endpoint to call - `proto_message` (*Any*) - The proto message to use - `request_data` (*dict*) - The request data to use - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `dict` - The next item in the listing --- ## [Model](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/model.py) ```python class Model(url=None, model_id=None, model_version={'id': ""}, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, compute_cluster_id=None, nodepool_id=None, deployment_id=None, deployment_user_id=None, **kwargs) ``` Model is a class that provides access to Clarifai API endpoints related to Model information. **Parameters:** - `url` (*str*) - The URL to initialize the model object - `model_id` (*str*) - The Model ID to interact with - `model_version` (*dict*) - The Model Version to interact with. Default `{'id': ""}` - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file - `compute_cluster_id` (*str*) - Compute cluster ID for runner selector - `nodepool_id` (*str*) - Nodepool ID for runner selector - `deployment_id` (*str*) - Deployment ID for runner selector - `deployment_user_id` (*str*) - User ID for runner selector (organization or user) ### Model.async_generate ```python async Model.async_generate(*args, **kwargs) ``` Calls the model's async generate() method (UNARY_STREAMING). Supports raw proto inputs for backward compatibility. ### Model.async_predict ```python async Model.async_predict(*args, **kwargs) ``` Calls the model's async predict() method (UNARY_UNARY). Supports raw proto inputs for backward compatibility. ### Model.async_stream ```python async Model.async_stream(*args, **kwargs) ``` Calls the model's async stream() method (STREAMING_STREAMING). Supports raw proto inputs for backward compatibility. ### Model.create_version ```python Model.create_version(**kwargs) ``` Creates a model version for the Model. **Parameters:** - `**kwargs` - Additional keyword arguments passed to Model Version: - `description` (*str*) - The description of the model version - `concepts` (*list[Concept]*) - The concepts to associate with the model version - `output_info` (*resources_pb2.OutputInfo*) - The output info to associate with the model version **Returns:** `Model` - A Model object for the specified model ID **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_version = model.create_version(description='model_version_description') ``` ### Model.delete_version ```python Model.delete_version(version_id) ``` Deletes a model version for the Model. **Parameters:** - `version_id` (*str*) - The version ID to delete **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model.delete_version(version_id='version_id') ``` ### Model.evaluate ```python Model.evaluate(dataset=None, dataset_id=None, dataset_app_id=None, dataset_user_id=None, dataset_version_id=None, eval_id=None, extended_metrics=None, eval_info=None) ``` Run evaluation on the model. **Parameters:** - `dataset` (*Dataset*) - If Clarifai Dataset is set, ignores other dataset_ arguments - `dataset_id` (*str*) - Dataset ID. Default None - `dataset_app_id` (*str*) - App ID for cross app evaluation. Default None - `dataset_user_id` (*str*) - User ID for cross app evaluation. Default None - `dataset_version_id` (*str*) - Dataset version ID. Default None - `eval_id` (*str*) - Specific ID for the evaluation. Default None - `extended_metrics` (*dict*) - User custom metrics result. Default None - `eval_info` (*dict*) - Custom eval info. Default None **Returns:** `resources_pb2.EvalMetrics` - Evaluation metrics ### Model.generate ```python Model.generate(*args, **kwargs) ``` Calls the model's generate() method (UNARY_STREAMING). Supports raw proto inputs for backward compatibility. ### Model.generate_by_bytes ```python Model.generate_by_bytes(input_bytes, input_type=None, inference_params={}, output_config={}) ``` Generate the stream output on model based on the given bytes. **Parameters:** - `input_bytes` (*bytes*) - File bytes to predict on - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override: - `min_value` (*float*) - Minimum prediction confidence - `max_concepts` (*int*) - Maximum number of concepts to return - `select_concepts` (*list[Concept]*) - The concepts to select **Example:** ```python from clarifai.client.model import Model model = Model("https://clarifai.com/openai/chat-completion/models/GPT-4") stream_response = model.generate_by_bytes(b'Write a tweet on future of AI', inference_params=dict(temperature=str(0.7), max_tokens=30)) list_stream_response = [response for response in stream_response] ``` ### Model.generate_by_filepath ```python Model.generate_by_filepath(filepath, input_type=None, inference_params={}, output_config={}) ``` Generate the stream output on model based on the given filepath. **Parameters:** - `filepath` (*str*) - The filepath to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') stream_response = model.generate_by_filepath('/path/to/image.jpg', 'image') list_stream_response = [response for response in stream_response] ``` ### Model.generate_by_url ```python Model.generate_by_url(url, input_type=None, inference_params={}, output_config={}) ``` Generate the stream output on model based on the given URL. **Parameters:** - `url` (*str*) - The URL to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') stream_response = model.generate_by_url('url') list_stream_response = [response for response in stream_response] ``` ### Model.get_eval_by_id ```python Model.get_eval_by_id(eval_id, label_counts=False, test_set=False, binary_metrics=False, confusion_matrix=False, metrics_by_class=False, metrics_by_area=False) ``` Get detailed eval_metrics by eval_id with extra metric fields. **Parameters:** - `eval_id` (*str*) - Evaluation ID - `label_counts` (*bool*) - Set True to get label counts. Default False - `test_set` (*bool*) - Set True to get test set. Default False - `binary_metrics` (*bool*) - Set True to get binary metric. Default False - `confusion_matrix` (*bool*) - Set True to get confusion matrix. Default False - `metrics_by_class` (*bool*) - Set True to get metrics by class. Default False - `metrics_by_area` (*bool*) - Set True to get metrics by area. Default False **Returns:** `resources_pb2.EvalMetrics` - Evaluation metrics ### Model.get_latest_eval ```python Model.get_latest_eval(label_counts=False, test_set=False, binary_metrics=False, confusion_matrix=False, metrics_by_class=False, metrics_by_area=False) ``` Run `get_eval_by_id` method with the latest eval_id. **Parameters:** - `label_counts` (*bool*) - Set True to get label counts. Default False - `test_set` (*bool*) - Set True to get test set. Default False - `binary_metrics` (*bool*) - Set True to get binary metric. Default False - `confusion_matrix` (*bool*) - Set True to get confusion matrix. Default False - `metrics_by_class` (*bool*) - Set True to get metrics by class. Default False - `metrics_by_area` (*bool*) - Set True to get metrics by area. Default False **Returns:** `resources_pb2.EvalMetrics` if model is evaluated, otherwise None ### Model.get_param_info ```python Model.get_param_info(param) ``` Returns the parameter info for the specified parameter. **Parameters:** - `param` (*str*) - The parameter name to get information for **Returns:** `Dict[str, Any]` - Dictionary containing model parameter info for the specified param **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_params = model.get_params(template='template', save_to='model_params.yaml') param_info = model.get_param_info('learning_rate') ``` ### Model.get_params ```python Model.get_params(template=None, save_to='params.yaml') ``` Returns the model params for the model type and saves them to a yaml file. **Parameters:** - `template` (*str*) - The template to use for the model type. Required for most model types except 'clusterer' and 'embedding-classifier' - `save_to` (*str*) - The yaml file path to save the model params. Default 'params.yaml' **Returns:** `Dict[str, Any]` - Dictionary of model params for the model type **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_params = model.get_params(template='template', save_to='model_params.yaml') ``` ### Model.list_evaluations ```python Model.list_evaluations() ``` List all eval_metrics of current model version. **Returns:** `resources_pb2.EvalMetrics` - Evaluation metrics **Raises:** `Exception` if API call fails ### Model.list_training_templates ```python Model.list_training_templates() ``` Lists all the training templates for the model type. **Returns:** `List[str]` - List of training templates for the model type **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') print(model.list_training_templates()) ``` ### Model.list_versions ```python Model.list_versions(page_no=None, per_page=None) ``` Lists all the versions for the model. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Model` - Model objects for the versions of the model **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') all_model_versions = list(model.list_versions()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### Model.load_info ```python Model.load_info(validate=False) ``` Loads the model information from the Clarifai API. **Parameters:** - `validate` (*bool*) - If True, only validates the existence of the model without updating attributes. Default False **Raises:** `Exception` if the gRPC API call fails ### Model.load_input_types ```python Model.load_input_types() ``` Loads the input types for the model. **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model.load_input_types() ``` ### Model.predict ```python Model.predict(*args, **kwargs) ``` Calls the model's predict() method (UNARY_UNARY). Supports raw proto inputs for backward compatibility. **Parameters:** - `inputs` (*List[Input]*) - List of input protos (backward compat) or method arguments - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override ### Model.predict_by_bytes ```python Model.predict_by_bytes(input_bytes, input_type=None, inference_params={}, output_config={}) ``` Predicts the model based on the given bytes. **Parameters:** - `input_bytes` (*bytes*) - File bytes to predict on - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override: - `min_value` (*float*) - Minimum prediction confidence - `max_concepts` (*int*) - Maximum number of concepts to return - `select_concepts` (*list[Concept]*) - The concepts to select **Example:** ```python from clarifai.client.model import Model model = Model("https://clarifai.com/openai/chat-completion/models/GPT-4") model_prediction = model.predict_by_bytes(b'Write a tweet on future of AI', inference_params=dict(temperature=str(0.7), max_tokens=30)) ``` ### Model.predict_by_filepath ```python Model.predict_by_filepath(filepath, input_type=None, inference_params={}, output_config={}) ``` Predicts the model based on the given filepath. **Parameters:** - `filepath` (*str*) - The filepath to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_prediction = model.predict_by_filepath('/path/to/image.jpg') model_prediction = model.predict_by_filepath('/path/to/text.txt') ``` ### Model.predict_by_url ```python Model.predict_by_url(url, input_type=None, inference_params={}, output_config={}) ``` Predicts the model based on the given URL. **Parameters:** - `url` (*str*) - The URL to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_prediction = model.predict_by_url('url') ``` ### Model.stream ```python Model.stream(*args, **kwargs) ``` Calls the model's stream() method (STREAMING_STREAMING). Supports raw proto inputs for backward compatibility. ### Model.stream_by_bytes ```python Model.stream_by_bytes(input_bytes_iterator, input_type=None, inference_params={}, output_config={}) ``` Stream the model output based on the given bytes iterator. **Parameters:** - `input_bytes_iterator` (*Iterator[bytes]*) - Iterator of file bytes to predict on - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model("https://clarifai.com/openai/chat-completion/models/GPT-4") stream_response = model.stream_by_bytes(iter([b'Write a tweet on future of AI']), inference_params=dict(temperature=str(0.7), max_tokens=30)) list_stream_response = [response for response in stream_response] ``` ### Model.stream_by_filepath ```python Model.stream_by_filepath(filepath, input_type=None, inference_params={}, output_config={}) ``` Stream the model output based on the given filepath. **Parameters:** - `filepath` (*str*) - The filepath to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model("url") stream_response = model.stream_by_filepath('/path/to/image.jpg') list_stream_response = [response for response in stream_response] ``` ### Model.stream_by_url ```python Model.stream_by_url(url_iterator, input_type=None, inference_params={}, output_config={}) ``` Stream the model output based on the given URL iterator. **Parameters:** - `url_iterator` (*Iterator[str]*) - Iterator of URLs to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' - `inference_params` (*dict*) - The inference params to override - `output_config` (*dict*) - The output config to override **Example:** ```python from clarifai.client.model import Model model = Model("url") stream_response = model.stream_by_url(iter(['url'])) list_stream_response = [response for response in stream_response] ``` ### Model.train ```python Model.train(yaml_file=None) ``` Trains the model based on the given yaml file or model params. **Parameters:** - `yaml_file` (*str*) - The yaml file for the model params **Returns:** `str` - The model version ID for the model **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_params = model.get_params(template='template', yaml_file='model_params.yaml') model.train('model_params.yaml') ``` ### Model.training_status ```python Model.training_status(version_id=None, training_logs=False) ``` Get the training status for the model version. **Parameters:** - `version_id` (*str*) - The version ID to get the training status for - `training_logs` (*bool*) - Whether to save the training logs in a file. Default False **Returns:** `Dict[str, str]` - Dictionary of training status for the model version **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model.training_status(version_id='version_id', training_logs=True) ``` ### Model.update_params ```python Model.update_params(**kwargs) ``` Updates the model params for the model. **Parameters:** - `**kwargs` - Model params to update (e.g. `batch_size=8`, `dataset_version='dataset_version_id'`) **Example:** ```python from clarifai.client.model import Model model = Model(model_id='model_id', user_id='user_id', app_id='app_id') model_params = model.get_params(template='template', yaml_file='model_params.yaml') model.update_params(batch_size=8, dataset_version='dataset_version_id') ``` --- ## [ModelClient](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/model_client.py) ```python class ModelClient(stub, async_stub=None, request_template=None) ``` Client for calling model predict, generate, and stream methods. **Parameters:** - `stub` - The gRPC stub for the model - `async_stub` (*V2Stub*) - Optional async gRPC stub - `request_template` (*PostModelOutputsRequest*) - Template for the request with common fields like model_id, model_version, cluster ### ModelClient.available_methods ```python ModelClient.available_methods() ``` Get the available methods for this model. **Returns:** `List[str]` - The available methods ### ModelClient.fetch ```python ModelClient.fetch() ``` Fetch function signature definitions from the model and define the functions in the client. ### ModelClient.from_local_grpc ```python ModelClient.from_local_grpc(port=8000) # classmethod ``` Create a ModelClient connected to a local gRPC model server. Connects to a local gRPC server started with `clarifai model serve --grpc`. Method signatures are auto-discovered from the running model. **Parameters:** - `port` (*int*) - Port of the local gRPC server. Default 8000 **Returns:** `ModelClient` with predict/generate/stream methods ready to use **Example:** ```python from clarifai.client.model_client import ModelClient client = ModelClient.from_local_grpc(port=8000) response = client.predict(text="What is the future of AI?") print(response) ``` ### ModelClient.generate_client_script ```python ModelClient.generate_client_script(base_url=None, use_ctx=False, colorize=False) ``` Generate a client script for this model. **Returns:** `str` - The client script ### ModelClient.method_signature ```python ModelClient.method_signature(method_name) ``` Get the method signature for a method. **Parameters:** - `method_name` (*str*) - The name of the method **Returns:** `str` - The method signature --- ## [Nodepool](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/nodepool.py) ```python class Nodepool(nodepool_id=None, user_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Nodepool is a class that provides access to Clarifai API endpoints related to Nodepool information. **Parameters:** - `nodepool_id` (*str*) - The Nodepool ID for the Nodepool to interact with - `user_id` (*str*) - The user ID of the user - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### Nodepool.create_deployment ```python Nodepool.create_deployment(config_filepath=None, deployment_id=None, deployment_config=None) ``` Creates a deployment for the nodepool. **Parameters:** - `config_filepath` (*str*) - The path to the deployment config file - `deployment_id` (*str*) - New deployment ID for the deployment to create - `deployment_config` (*Dict[str, Any]*) - Optional dictionary containing deployment configuration **Returns:** `Deployment` - A Deployment object for the specified deployment ID **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") deployment = nodepool.create_deployment(config_filepath="config.yml") ``` ### Nodepool.create_runner ```python Nodepool.create_runner(config_filepath=None, runner_config=None) ``` Creates a runner for the nodepool. Only needed for local runners. **Parameters:** - `config_filepath` (*str*) - The path to the runner config file - `runner_config` (*Dict[str, Any]*) - Optional dictionary containing runner configuration **Returns:** `Runner` - A Runner object for the specified runner ID **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") runner = nodepool.create_runner(config_filepath="runner.yml") ``` ### Nodepool.delete_deployments ```python Nodepool.delete_deployments(deployment_ids) ``` Deletes list of deployments for the nodepool. **Parameters:** - `deployment_ids` (*List[str]*) - The list of deployment IDs to delete **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") nodepool.delete_deployments(deployment_ids=["deployment_id1", "deployment_id2"]) ``` ### Nodepool.delete_runners ```python Nodepool.delete_runners(runner_ids) ``` Deletes list of runners for the nodepool. **Parameters:** - `runner_ids` (*List[str]*) - The list of runner IDs to delete **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") nodepool.delete_runners(runner_ids=["runner_id1", "runner_id2"]) ``` ### Nodepool.deployment ```python Nodepool.deployment(deployment_id) ``` Returns a Deployment object for the existing deployment ID. **Parameters:** - `deployment_id` (*str*) - The deployment ID for the deployment to interact with **Returns:** `Deployment` - A Deployment object for the existing deployment ID **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") deployment = nodepool.deployment(deployment_id="deployment_id") ``` ### Nodepool.get_runner_selector ```python Nodepool.get_runner_selector(user_id, compute_cluster_id, nodepool_id) # static method ``` Returns a RunnerSelector object for the specified compute cluster and nodepool. **Parameters:** - `user_id` (*str*) - The user ID of the user - `compute_cluster_id` (*str*) - The compute cluster ID - `nodepool_id` (*str*) - The nodepool ID **Returns:** `resources_pb2.RunnerSelector` - A RunnerSelector object **Example:** ```python from clarifai.client.nodepool import Nodepool runner_selector = Nodepool.get_runner_selector(user_id="user_id", compute_cluster_id="compute_cluster_id", nodepool_id="nodepool_id") ``` ### Nodepool.list_deployments ```python Nodepool.list_deployments(filter_by={}, page_no=None, per_page=None) ``` Lists all the available deployments of the compute cluster. **Parameters:** - `filter_by` (*Dict[str, Any]*) - The filter to apply to the list of deployments - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Deployment` - Deployment objects for the nodepool **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") all_deployments = list(nodepool.list_deployments()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### Nodepool.list_runners ```python Nodepool.list_runners(model_version_ids=None, page_no=None, per_page=None) ``` Lists all the available runners for the model version in the nodepool. **Parameters:** - `model_version_ids` (*List[str]*) - Optional model version IDs to filter runners - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Runner` - Runner objects for the nodepool **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") all_runners = list(nodepool.list_runners()) ``` ### Nodepool.runner ```python Nodepool.runner(runner_id) ``` Returns a Runner object for the existing runner ID. **Parameters:** - `runner_id` (*str*) - The runner ID for the runner to interact with **Returns:** `Runner` - A Runner object for the existing runner ID **Example:** ```python from clarifai.client.nodepool import Nodepool nodepool = Nodepool(nodepool_id="nodepool_id", user_id="user_id") runner = nodepool.runner(runner_id="runner_id") ``` --- ## [Pipeline](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/pipeline.py) ```python class Pipeline(url=None, pipeline_id=None, pipeline_version_id=None, pipeline_version_run_id=None, user_id=None, app_id=None, nodepool_id=None, compute_cluster_id=None, log_file=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Pipeline is a class that provides access to Clarifai API endpoints related to Pipeline information. **Parameters:** - `url` (*str*) - The URL to initialize the pipeline object - `pipeline_id` (*str*) - The Pipeline ID to interact with - `pipeline_version_id` (*str*) - The Pipeline Version ID to interact with - `pipeline_version_run_id` (*str*) - The Pipeline Version Run ID. If not provided, a UUID is generated - `user_id` (*str*) - The User ID that owns the pipeline - `app_id` (*str*) - The App ID that contains the pipeline - `nodepool_id` (*str*) - The Nodepool ID to run the pipeline on - `compute_cluster_id` (*str*) - The Compute Cluster ID to run the pipeline on - `log_file` (*str*) - Path to file where logs should be written. If not provided, logs to console - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### Pipeline.monitor_only ```python Pipeline.monitor_only(timeout=3600, monitor_interval=10) ``` Monitor an existing pipeline run without starting a new one. **Parameters:** - `timeout` (*int*) - Maximum time to wait for completion in seconds. Default 3600 (1 hour) - `monitor_interval` (*int*) - Interval between status checks in seconds. Default 10 **Returns:** `Dict` - The pipeline run result ### Pipeline.patch_pipeline_version_run ```python Pipeline.patch_pipeline_version_run(pipeline_version_run_id, orchestration_status_code) ``` Patch a pipeline version run's orchestration status. This method can be used to pause, cancel, or resume a pipeline run. **Parameters:** - `pipeline_version_run_id` (*str*) - The pipeline version run ID to patch - `orchestration_status_code` (*int*) - The status code to set (e.g., JOB_PAUSED, JOB_CANCELLED, JOB_RUNNING) **Returns:** `Dict` - The response as a dictionary **Raises:** `UserError` if the patch request fails ### Pipeline.run ```python Pipeline.run(inputs=None, timeout=3600, monitor_interval=10, input_args_override=None) ``` Run the pipeline and monitor its progress. **Parameters:** - `inputs` (*List*) - List of inputs to run the pipeline with. If None, runs without inputs - `timeout` (*int*) - Maximum time to wait for completion in seconds. Default 3600 (1 hour) - `monitor_interval` (*int*) - Interval between status checks in seconds. Default 10 - `input_args_override` (*OrchestrationArgsOverride*) - Override arguments for the pipeline run **Returns:** `Dict` - The pipeline run result --- ## [PipelineStep](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/pipeline_step.py) ```python class PipelineStep(url=None, pipeline_step_id=None, pipeline_step_version_id=None, user_id=None, app_id=None, pipeline_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` PipelineStep is a class that provides access to Clarifai API endpoints related to PipelineStep information. **Parameters:** - `url` (*str*) - The URL to initialize the pipeline step object - `pipeline_step_id` (*str*) - The PipelineStep ID for the PipelineStep to interact with - `pipeline_step_version_id` (*str*) - The PipelineStep version ID - `user_id` (*str*) - The User ID for the PipelineStep to interact with - `app_id` (*str*) - The App ID for the PipelineStep to interact with - `pipeline_id` (*str*) - The Pipeline ID for the PipelineStep to interact with - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication - `token` (*str*) - A session token for authentication - `root_certificates_path` (*str*) - Path to the SSL root certificates file --- ## [Runner](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/runner.py) ```python class Runner(runner_id=None, user_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Runner is a class that provides access to Clarifai API endpoints related to Runner information. **Parameters:** - `runner_id` (*str*) - The Runner ID for the Runner to interact with - `user_id` (*str*) - The user ID of the user - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file --- ## [Search](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/search.py) ```python class Search(user_id, app_id, top_k=None, metric="cosine", algorithm="nearest_neighbor", pagination=False, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None) ``` Search class for performing vector search and filtering over Clarifai inputs. **Parameters:** - `user_id` (*str*) - User ID - `app_id` (*str*) - App ID - `top_k` (*int*) - Top K results to retrieve. Defaults to 10 - `metric` (*str*) - Similarity metric: 'cosine' or 'euclidean'. Default 'cosine' - `algorithm` (*str*) - Search algorithm: 'nearest_neighbor' or 'brute_force'. Default 'nearest_neighbor' - `pagination` (*bool*) - Enable pagination. Default False - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file **Raises:** - `UserError` if the metric is not 'cosine' or 'euclidean' - `UserError` if the algorithm is not 'nearest_neighbor' or 'brute_force' ### Search.query ```python Search.query(ranks=[{}], filters=[{}], page_no=None, per_page=None) ``` Perform a query with rank and filters. **Parameters:** - `ranks` (*List[Dict]*) - List of rank parameters. Default `[{}]` - `filters` (*List[Dict]*) - List of filter parameters. Default `[{}]` - `page_no` (*int*) - The page number to list (only available when `pagination=True`) - `per_page` (*int*) - The number of items per page (only available when `pagination=True`) **Returns:** Generator of query results **Example:** ```python from clarifai.client.search import Search # Get successful inputs of type image or text search = Search(user_id='user_id', app_id='app_id', top_k=10, metric='cosine') res = search.query(filters=[{'input_types': ['image', 'text'], 'input_status_code': 30000}]) # Vector search over inputs search = Search(user_id='user_id', app_id='app_id', metric='cosine', pagination=True) res = search.query(ranks=[{'image_url': 'https://samples.clarifai.com/dog.tiff'}], page_no=2, per_page=5) ``` **Note:** For rank and filter schema, refer to [schema](https://github.com/Clarifai/clarifai-python/tree/master/clarifai/schema/search.py). --- ## [User](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/user.py) ```python class User(user_id=None, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` User is a class that provides access to Clarifai API endpoints related to user information. **Parameters:** - `user_id` (*str*) - The user ID for the user to interact with - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### User.app ```python User.app(app_id, **kwargs) ``` Returns an App object for the specified app ID. **Parameters:** - `app_id` (*str*) - The app ID for the app to interact with **Returns:** `App` - An App object for the specified app ID **Example:** ```python from clarifai.client.user import User app = User("user_id").app("app_id") ``` ### User.compute_cluster ```python User.compute_cluster(compute_cluster_id) ``` Returns a ComputeCluster object for the specified compute cluster ID. **Parameters:** - `compute_cluster_id` (*str*) - The compute cluster ID for the compute cluster to interact with **Returns:** `ComputeCluster` - A ComputeCluster object for the specified compute cluster ID **Example:** ```python from clarifai.client.user import User compute_cluster = User("user_id").compute_cluster("compute_cluster_id") ``` ### User.create_app ```python User.create_app(app_id, base_workflow='Empty', **kwargs) ``` Creates an app for the user. **Parameters:** - `app_id` (*str*) - The app ID for the app to create - `base_workflow` (*str*) - The base workflow to use for the app. Examples: 'Universal', 'Language-Understanding', 'General'. Default 'Empty' - `**kwargs` - Additional keyword arguments passed to the App **Returns:** `App` - An App object for the specified app ID **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") app = client.create_app(app_id="app_id", base_workflow="Universal") ``` ### User.create_compute_cluster ```python User.create_compute_cluster(config_filepath=None, compute_cluster_id=None, compute_cluster_config=None) ``` Creates a compute cluster for the user. **Parameters:** - `config_filepath` (*str*) - The path to the compute cluster config file - `compute_cluster_id` (*str*) - New compute cluster ID for the compute cluster to create - `compute_cluster_config` (*Dict[str, Any]*) - Optional dictionary containing compute cluster configuration **Returns:** `ComputeCluster` - A Compute Cluster object for the specified compute cluster ID **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") compute_cluster = client.create_compute_cluster(config_filepath="config.yml") ``` ### User.create_secrets ```python User.create_secrets(secrets) ``` Creates secrets for the user. **Parameters:** - `secrets` (*List[Dict[str, Any]]*) - List of secret configurations. Each secret dict can contain: - `id` (*str*) - The name/ID of the secret (required) - `value` (*str*) - The secret value (required) - `description` (*str*) - Optional description of the secret - `expires_at` (*str*) - Optional expiration timestamp **Returns:** `List[Dict]` - List of dictionaries containing information about the created secrets **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") secrets = [{"id": "secret1", "value": "secret_value", "description": "My Secret"}] created_secrets = client.create_secrets(secrets) ``` ### User.delete_app ```python User.delete_app(app_id) ``` Deletes an app for the user. **Parameters:** - `app_id` (*str*) - The app ID for the app to delete **Example:** ```python from clarifai.client.user import User User("user_id").delete_app("app_id") ``` ### User.delete_compute_clusters ```python User.delete_compute_clusters(compute_cluster_ids) ``` Deletes a list of compute clusters for the user. **Parameters:** - `compute_cluster_ids` (*List[str]*) - The compute cluster IDs of the user to delete **Example:** ```python from clarifai.client.user import User User("user_id").delete_compute_clusters(compute_cluster_ids=["compute_cluster_id1", "compute_cluster_id2"]) ``` ### User.delete_runner ```python User.delete_runner(runner_id) ``` Deletes the specified runner. **Parameters:** - `runner_id` (*str*) - Runner ID to delete **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") client.delete_runner(runner_id="runner_id") ``` ### User.delete_secrets ```python User.delete_secrets(secret_ids) ``` Deletes a list of secrets for the user. **Parameters:** - `secret_ids` (*List[str]*) - The secret IDs of the user to delete **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") client.delete_secrets(secret_ids=["secret_id1", "secret_id2"]) ``` ### User.get_secret ```python User.get_secret(secret_id) ``` Returns a secret object if it exists. **Parameters:** - `secret_id` (*str*) - The secret ID to interact with **Returns:** `Dict` - A dictionary containing information about the existing secret ID **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") secret_info = client.get_secret(secret_id="secret_id") ``` ### User.get_user_info ```python User.get_user_info(user_id=None) ``` Returns the user information for the specified user ID. **Parameters:** - `user_id` (*str*) - The user ID for the user to interact with **Returns:** Response containing user information ### User.list_apps ```python User.list_apps(filter_by={}, page_no=None, per_page=None) ``` Lists all the apps for the user. **Parameters:** - `filter_by` (*Dict[str, Any]*) - A dictionary of filters to be applied to the list of apps - `page_no` (*int*) - The page number to list. If None, lists all pages - `per_page` (*int*) - The number of items per page. If None, uses default **Yields:** `App` - App objects for the user **Example:** ```python from clarifai.client.user import User apps = list(User("user_id").list_apps()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### User.list_cloud_providers ```python User.list_cloud_providers() ``` List available cloud providers (e.g. aws, gcp, azure, vultr). **Returns:** `list` - List of CloudProvider protobuf objects with id, name, special_handling fields **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") providers = client.list_cloud_providers() ``` ### User.list_cloud_regions ```python User.list_cloud_regions(cloud_provider_id) ``` List available regions for a cloud provider. **Parameters:** - `cloud_provider_id` (*str*) - The cloud provider ID (e.g. 'aws', 'gcp') **Returns:** `list` - List of CloudRegion protobuf objects **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") regions = client.list_cloud_regions("aws") ``` ### User.list_compute_clusters ```python User.list_compute_clusters(page_no=None, per_page=None) ``` List all compute clusters for the user. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `ComputeCluster` - ComputeCluster objects **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") all_compute_clusters = list(client.list_compute_clusters()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### User.list_instance_types ```python User.list_instance_types(cloud_provider_id, region) ``` List available GPU/instance types for a cloud provider and region. **Parameters:** - `cloud_provider_id` (*str*) - The cloud provider ID (e.g. 'aws', 'gcp') - `region` (*str*) - The region ID (e.g. 'us-east-1') **Returns:** `list` - List of InstanceType protobuf objects with id, description, compute_info, price, cloud_provider, region fields **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") instance_types = client.list_instance_types("aws", "us-east-1") ``` ### User.list_models ```python User.list_models(user_id=None, app_id=None, show=True, return_clarifai_model=False, **kwargs) ``` List models with optional filtering by user and app. **Parameters:** - `user_id` (*str*) - User ID to filter by. Use "all" to list all users' models - `app_id` (*str*) - App ID to filter by - `show` (*bool*) - If True, print a tabulated display of models. Default True - `return_clarifai_model` (*bool*) - If True, return Model objects instead of dicts. Default False **Returns:** `List[Dict]` or `List[Model]` - List of model info dicts, or Model objects if `return_clarifai_model=True` ### User.list_organizations ```python User.list_organizations() ``` List organizations the user belongs to via REST API. **Returns:** `List[Dict]` - List of dicts with 'id' and 'name' keys for each organization ### User.list_pipeline_steps ```python User.list_pipeline_steps(page_no=None, per_page=None) ``` List all pipeline steps for the user across all apps. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `PipelineStep` - PipelineStep objects **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") all_pipeline_steps = list(client.list_pipeline_steps()) ``` ### User.list_pipelines ```python User.list_pipelines(page_no=None, per_page=None) ``` List all pipelines for the user across all apps. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Pipeline` - Pipeline objects **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") all_pipelines = list(client.list_pipelines()) ``` ### User.list_runners ```python User.list_runners(filter_by={}, page_no=None, per_page=None) ``` List all runners for the user. **Parameters:** - `filter_by` (*dict*) - A dictionary of filters to apply to the list of runners - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Dict` - Dictionaries containing information about the runners **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") all_runners = list(client.list_runners()) ``` ### User.list_secrets ```python User.list_secrets(page_no=None, per_page=None) ``` List all secrets for the user. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Dict` - Dictionaries containing information about the secrets **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") all_secrets = list(client.list_secrets()) ``` ### User.patch_app ```python User.patch_app(app_id, action='overwrite', **kwargs) ``` Patch an app for the user. **Parameters:** - `app_id` (*str*) - The app ID for the app to patch - `action` (*str*) - The action to perform: overwrite/remove. Default 'overwrite' - `**kwargs` - Additional keyword arguments to patch the App **Returns:** `App` - Patched App object for the specified app ID ### User.patch_secrets ```python User.patch_secrets(secrets, action='overwrite') ``` Patches secrets for the user. **Parameters:** - `secrets` (*List[Dict[str, Any]]*) - List of secret configurations to patch. Each secret dict should contain: - `id` (*str*) - The name/ID of the secret to patch (required) - `value` (*str*) - Optional new secret value - `description` (*str*) - Optional new description - `expires_at` (*str*) - Optional new expiration timestamp - `action` (*str*) - The action to perform: overwrite/remove. Default 'overwrite' **Returns:** `List[Dict]` - List of dictionaries containing information about the patched secrets **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") secrets = [{"id": "secret1", "description": "Updated Secret Description"}] patched_secrets = client.patch_secrets(secrets, action="overwrite") ``` ### User.runner ```python User.runner(runner_id) ``` Returns a Runner object if it exists. **Parameters:** - `runner_id` (*str*) - The runner ID to interact with **Returns:** `Dict` - A dictionary containing information about the existing runner ID **Example:** ```python from clarifai.client.user import User client = User(user_id="user_id") runner_info = client.runner(runner_id="runner_id") ``` --- ## [Workflow](https://github.com/Clarifai/clarifai-python/blob/master/clarifai/client/workflow.py) ```python class Workflow(url=None, workflow_id=None, workflow_version={'id': ""}, output_config={'min_value': 0}, base_url="https://api.clarifai.com", pat=None, token=None, root_certificates_path=None, **kwargs) ``` Workflow is a class that provides access to Clarifai API endpoints related to Workflow information. **Parameters:** - `url` (*str*) - The URL to initialize the workflow object - `workflow_id` (*str*) - The Workflow ID to interact with - `workflow_version` (*Dict[str, str]*) - The Workflow Version to interact with. Default `{'id': ""}` for latest version - `output_config` (*Dict[str, Any]*) - The output config to interact with: - `min_value` (*float*) - The minimum value of the prediction confidence to filter - `max_concepts` (*int*) - The maximum number of concepts to return - `select_concepts` (*List[Concept]*) - The concepts to select - `sample_ms` (*int*) - The number of milliseconds to sample - `base_url` (*str*) - Base API url. Default "https://api.clarifai.com" - `pat` (*str*) - A personal access token for authentication. Can be set as env var CLARIFAI_PAT - `token` (*str*) - A session token for authentication. Can be set as env var CLARIFAI_SESSION_TOKEN - `root_certificates_path` (*str*) - Path to the SSL root certificates file ### Workflow.export ```python Workflow.export(out_path) ``` Exports the workflow to a yaml file. **Parameters:** - `out_path` (*str*) - The path to save the yaml file to **Example:** ```python from clarifai.client.workflow import Workflow workflow = Workflow("https://clarifai.com/clarifai/main/workflows/Demographics") workflow.export('out_path.yml') ``` ### Workflow.list_versions ```python Workflow.list_versions(page_no=None, per_page=None) ``` Lists all the versions of the workflow. **Parameters:** - `page_no` (*int*) - The page number to list - `per_page` (*int*) - The number of items per page **Yields:** `Workflow` - Workflow objects for versions of the workflow **Example:** ```python from clarifai.client.workflow import Workflow workflow = Workflow(user_id='user_id', app_id='app_id', workflow_id='workflow_id') workflow_versions = list(workflow.list_versions()) ``` **Note:** Defaults to 16 per page if `page_no` is specified and `per_page` is not. If both are None, lists all resources. ### Workflow.load_info ```python Workflow.load_info() ``` Loads the workflow info, including input types for the first node's model. ### Workflow.predict ```python Workflow.predict(inputs, workflow_state_id=None) ``` Predicts the workflow based on the given inputs. **Parameters:** - `inputs` (*List[Input]*) - The inputs to predict (max 32) - `workflow_state_id` (*str*) - Key for the workflow state cache that stores the workflow node predictions **Returns:** Response from the workflow prediction ### Workflow.predict_by_bytes ```python Workflow.predict_by_bytes(input_bytes, input_type=None) ``` Predicts the workflow based on the given bytes. **Parameters:** - `input_bytes` (*bytes*) - Bytes to predict on - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' **Returns:** Response from the workflow prediction ### Workflow.predict_by_filepath ```python Workflow.predict_by_filepath(filepath, input_type=None) ``` Predicts the workflow based on the given filepath. **Parameters:** - `filepath` (*str*) - The filepath to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' **Returns:** Response from the workflow prediction **Example:** ```python from clarifai.client.workflow import Workflow workflow = Workflow("https://clarifai.com/clarifai/main/workflows/Face-Sentiment") workflow_prediction = workflow.predict_by_filepath('filepath') ``` ### Workflow.predict_by_url ```python Workflow.predict_by_url(url, input_type=None) ``` Predicts the workflow based on the given URL. **Parameters:** - `url` (*str*) - The URL to predict - `input_type` (*str*) - The type of input: 'image', 'text', 'video', or 'audio' **Returns:** Response from the workflow prediction **Example:** ```python from clarifai.client.workflow import Workflow workflow = Workflow("https://clarifai.com/clarifai/main/workflows/Face-Sentiment") workflow_prediction = workflow.predict_by_url('url') ``` --- ## Complementary Topics # Complementary Topics **Learn about the advanced capabilities of our platform** Unlock the power of the Clarifai API. import DocCardList from '@theme/DocCardList'; import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; --- ## Batch Predict CSV on Custom Text Model # Batch Predict CSV on Custom Text Model **Enjoy the convenience of working with CSV files and text** Below is a script that can be used to run predictions on a batch of text/sentences stored in a CSV file, using your custom text model. To start, you'll need to create your own Custom Text Model, either via [our Portal](https://docs.clarifai.com/portal-guide/model/pcustom-model-walkthrough/) or [using the API](https://docs.clarifai.com/api-guide/model/custom-model-walkthrough/). Make sure to record the model ID and version ID that you want to use \(each model gets a version ID after being successfully trained\), as well as the API key of the application in which the model exists. This script assumes that you have a CSV file which has one column named "text" where the text you want to run predictions on is. It'll output another CSV file containing the predicted concepts for each text, together with confidence values. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonBatchPredict from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/batch_predict_csv.py"; {PythonBatchPredict} ## Example Usage Let's say you have the following CSV file, and you want to predict whether the sentence is grammatically positive or negative for each text in a row. You first build a custom text model that was created to map text into two concepts: "positive" and "negative. See our [Custom Text Model walkthrough](https://docs.clarifai.com/api-guide/model/custom-text-model-walkthrough/) on how to do that via our API. ```bash number,text 1,"We have never been to Asia, nor have we visited Africa." 2,"I am never at home on Sundays." 3,"One small action would change her life, but whether it would be for better or for worse was yet to be determined." 4,"The waitress was not amused when he ordered green eggs and ham." 5,"In that instant, everything changed." ``` With that, you can run the script on the CSV file in the following manner, which will produce a new CSV file. ```bash python nlp_model_predicts.py --api_key YOUR_API_KEY --model_id YOUR_MODEL_ID --model_version YOUR_MODEL_VERSION_ID --csv_file my_data.csv --top_n 2 ``` ```bash text,predict_1_concept,predict_1_value,predict_2_concept,predict_2_value "We have never been to Asia, nor have we visited Africa.",negative,1.000,positive,0.000 I am never at home on Sundays.,negative,1.000,positive,0.000 "One small action would change her life, but whether it would be for better or for worse was yet to be determined.",positive,1.000,negative,0.000 The waitress was not amused when he ordered green eggs and ham.,negative,1.000,positive,0.000 "In that instant, everything changed.",positive,0.998,negative,0.002 ``` --- ## Import Data with CSV and TSV Files # Import Data with CSV and TSV Files **Enjoy the convenience of working with spreadsheet data** You can upload your text directly from a `.csv` \([comma separated values](https://en.wikipedia.org/wiki/Comma-separated_values)\) or `.tsv` \([tab separated values](https://en.wikipedia.org/wiki/Tab-separated_values)\) file. This means you can work with your favorite spreadsheet software or text editor when preparing your data for upload. This can be especially useful when you want to upload inputs and concepts at the same time. Just use the provided "CSV template" to get started. ## CSV templates To help you get started, you can download `.csv` templates for images and text uploads here: - [Image Upload Template](/templates/ClarifaiImageUploadTemplate.csv) - [Text Upload Template](/templates/ClarifaiTextUploadTemplate.csv) - [Text Generation Upload Template (for LLaMA 3.1 8B Instruct model)](/templates/finetuning_dataset.csv) ### Working With Your CSV File At a minimum, you should add an image URL or text to the `input.data.image.url` or `input.data.text.raw` field respectively. You can add one concept per column to the `input.data.concepts[*].id` fields. For the `input.data.concepts[*].value` column, there are two options: enter the number `1` if the concept _is_ present in the input, enter the value `0` if the concept is _not_ present in the input \(a negative example\). If no value is entered, a default value of `1` will be assigned to your input. You can add columns for as many concepts as you like, and you can add new columns to add values for any other values supported by the API: | Field | Description | | :--- | :--- | | input.id | A unique identifier for your input | | input.data.image.url | The URL \(web address\) for your input | | input.data.text.raw | The "text" for your input | | input.data.concepts\[i\].id | Your custom concept | | input.data.concepts\[i\].value | The value for your custom concept \(`1` for true, `0` for false\) | | input.data.metadata | Any additional metadata in valid [JSON](https://www.json.org/json-en.html) format | | input.data.geo.geo\_point.latitude | Latitude for geodata | | input.data.geo.geo\_point.longitude | Longitude for geodata | Finally, you will need to save your work as a `.csv` file. If you are editing in Google Sheets, go to File >>> Download >>> Comma-separated values \(.csv, current sheet\). If you are using Excel, go to File >>> Save As >>> Browse >>> Save as Type >>> CSV. Once you have downloaded the `.csv` file, you can then upload it to the Clarifai platform. --- ## Intelligent Document Processing # Intelligent Document Processing **Capture information from an organization's form that are stored as PDF documents** ## Introduction In this series of posts, advanced users at Clarifai will present working solutions to help you kick-start your own AI solutions. ### The Use Case There is a problem facing many organizations as they attempt to modernize: digitizing documents. In order to effectively gain insights from their old paper records, organizations must transform them into digital versions. Now, simply making a digital copy of a document is actually rather easy⁠—simply scan it or take its photo. The problem, though, is that while this changes how the document is stored, it doesn't give you any real improvements to accessing the data therein. Previously, this required a laborious, manual, data entry process. Someone would have to transcribe the documents one-by-one, and enter each field into the books. This presents a problem to organizations that potentially have thousands and thousands of documents in their records, which can be intractable when it comes to the time and cost of the effort. Luckily, there's a middle-ground. Using Clarifai's publicly available Optical Character Recognition (OCR) models, organizations can leverage Artifical Intelligence to both do this in a quick and cost-effective manner, and without sacrificing the insights they would have from recording every single value. ### Assumptions Before we begin, let us make some assumptions: 1. The forms are standardized with static regions for fixed values; for example, the "name" field will always appear in the same location across all forms; 1. All the entries will be in English, using the Roman alphabet; 1. The organizations have a simple means of converting their paper documents to pdf documents, and storing them to a local file system, which is a common feature on most commercial print stations; 1. All the forms will be type-filled, not handwritten, so as to make generating examples easier. These assumptions were largely made to make this example succinct and easily digestable. ## Setup Before we get to the implementation, let's take a moment to provide an overview thereof. First off, the broad strokes have already been laid out: convert pdf to image, use Clarifai for OCR, and from that you'll have the text, which you then store in order to access later. Clearly, there are some gaps that need to be filled in though⁠—the largest of which is just _how_ the document will be processed. Working backwards a bit, the way in which the information will be recorded will be highly dependent on the organization's data policies. So to simplify things, we will utilize Clarifai's platform to store the annotated documents. Given assumption 1 above, we know that the fields will be in fixed locations. This means we can define those ahead of time, and here we've chosen to do so using a JSON file, in which we define the document's structure in a manner similar to: ```json { "field_1": [0.25, 0.25, 0.50, 0.50], "field_2": [0.50, 0.25, 0.75, 0.50], . . . "field_n": [0.25, 0.75, 0.50, 1.00] } ``` Each key-value pair in the JSON file corresponds to the field name, the key (`"field_n"`), and the region coordinates in the form of $[x_0, y_0, x_1, y_1]$. :::important note All the region coordinates on Clarifai are relative, not pixel values. This is important, as other image processing libraries might use the pixel values instead. We will address converting between these values below. ::: Given that we know the name of the field, and where it is on the image, we can easily iterate through all of these field values, and annotate the corresponding region on the image. Having the coordinate values will also let us take sub-crops of the document to use the OCR model to predict on; isolating the text associated with a given field. With this, we have a more fleshed out plan: > We assume that the user is already familiar with basic platform usage, and has an account. > If more information is needed here, please find the appropriate section of the document in order to access more in-depth information. 1. Convert PDF to Image, and upload it on the Clarifai platform for storage. 1. Read values from the JSON where the form's fields and their locations are defined. 1. For each field and region: - Extract a sub-crop for the field; - Use Clarifai's OCR model to predict the text associated with the field; - Write the predicted text back to the input as an annotation. Now let's dive into the implementation: Starting with the conversion of a PDF document to an image, we can handle this with the open-source library `pdf2image`, which does exactly what the name suggests. In order to be a bit more defensive with our programming, we will wrap the call to the `pdf2image.convert_from_path` method in a separate function, and do some quick sanity checking to make sure the PDF file exists. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import PythonSetup from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/intelligent_document_processing_setup.py"; import PythonFullImplementation from "!!raw-loader!../../../code_snippets/api-guide/advanced_topics/intelligent_document_processing_full_implementation.py"; {PythonSetup} This will return iterable images that correspond to the individual pages of the document. :::important note For simplicity's sake, our form only has one page. ::: ## Full implementation {PythonFullImplementation} --- ## Patching # Patching **Patch data in your applications** We designed PATCH to work over multiple resources at the same time \(bulk\) and be flexible enough for all your needs in order to minimize round trips to the server. Therefore, it might seem a little different to any PATCH you've seen before, but it's not complicated. All three actions that are supported do overwrite by default, but have special behavior for lists of objects \(for example, lists of concepts\). ## Merge The `merge` action will overwrite a `key:value` with `key:new_value` or append to an existing list of values, merging dictionaries that match by a corresponding `id` field. In the following examples, A is being patched into B to create the Result: ```text *Merges different key:values* A = `{"a":[1,2,3]}` B = `{"blah":true}` Result = `{"blah":true, "a":[1,2,3]}` *For id lists, merge will append* A = `{"a":[{"id": 1}]}` B = `{"a":[{"id": 2}]}` Result = `{"a":[{"id": 2}, {"id":1}]}` *Simple merge of key:values and within a list* A = `{"a":[{"id": "1", "other":true}], "blah":1}` B = `{"a":[{"id": "2"},{"id":"1", "other":false}]}` Result = `{"a":[{"id": "2"},{"id": "1"}], "blah":1}` *Different types should overwrite fine* A = `{"a":[{"id": "1"}], "blah":1}` B = `{"a":[{"id": "2"}], "blah":"string"}` Result = `{"a":[{"id": "2"},{"id": "1"}], "blah":1}` *Deep merge, notice the "id":"1" matches, so those dicts are merged in the list* A = `{"a":[{"id": "1","hey":true}], "blah":1}` B = `{"a":[{"id": "1","foo":"bar","hey":false},{"id":"2"}], "blah":"string"}` Result = `{"a":[{"hey":true,"id": "1","foo":"bar"},{"id":"2"}], "blah":1}` *For non-id lists, merge will append* A = `{"a":[{"blah": "1"}], "blah":1}` B = `{"a":[{"blah": "2"}], "blah":"string"}` Result = `{"a":[{"blah": "2"}, {"blah":"1"}], "blah":1}` *For non-id lists, merge will append* A = `{"a":[{"blah": "1"}], "blah":1, "dict":{"a":1,"b":2}}` B = `{"a":[{"blah": "2"}], "blah":"string"}` Result = `{"a":[{"blah": "2"}, {"blah":"1"}], "blah":1, "dict":{"a":1,"b":2}}` *Simple overwrite root element* A = `{"key1":true}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":true}` *Overwrite a sub element* A = `{"key1":{"key2":true}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":true, "key3":"value3"}}` *Merge a sub element* A = `{"key1":{"key2":{"key4":"value4"}}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":{"key4":"value4"}, "key3":"value3"}}` *Merge multiple trees* A = `{"key1":{"key2":{"key9":"value9"}, "key3":{"key4":"value4", "key10":[1,2,3]}}, "key6":{"key11":"value11"}}` B = `{"key1":{"key2":"value2", "key3":{"key4":{"key5":"value5"}}}, "key6":{"key7":{"key8":"value8"}}}` Result = `{"key1":{"key2":{"key9":"value9"}, "key3":{"key4":"value4", "key10":[1,2,3]}}, "key6":{"key7":{"key8":"value8"}, "key11":"value11"}}` *Merge {} element will replace* A = `{"key1":{"key2":{}}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":{}, "key3":"value3"}}` *Merge a null element does nothing* A = `{"key1":{"key2":null}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":"value2", "key3":"value3"}}` *Merge a blank list [] will replace root element* A = `{"key1":[]}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":[]}` *Merge a blank list [] will replace single element* A = `{"key1":{"key2":[]}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":[], "key3":"value3"}}` *Merge a blank list [] will remove nested objects* A = `{"key1":{"key2":[{"key3":"value3"}]}}` B = `{"key1":{"key2":{"key3":"value3"}}}` Result = `{"key1":{"key2":[{"key3":"value3"}]}}` *Merge an existing list with some other struct* A = `{"key1":{"key2":{"key3":[{"key4":"value4"}]}}}` B = `{"key1":{"key2":[]}}` Result = `{"key1":{"key2":{"key3":[{"key4":"value4"}]}}}` ``` ## Remove The `remove` action will overwrite a `key:value` with `key:new_value` or delete anything in a list that matches the provided values' ids. In the following examples, A is being patched into B to create the Result: ```text *Remove from list* A = `{"a":[{"id": "1"}], "blah":1}` B = `{"a":[{"id": "2"},{"id": "3"}, {"id":"1"}], "blah":"string"}` Result = `{"a":[{"id": "2"},{"id":"3"}], "blah":1}` *For non-id lists, remove will append* A = `{"a":[{"blah": "1"}], "blah":1}` B = `{"a":[{"blah": "2"}], "blah":"string"}` Result = `{"a":[{"blah": "2"}, {"blah":"1"}], "blah":1}` *Empty out a nested dictionary* A = `{"key1":{"key2":true}}` B = `{"key1":{"key2":"value2"}}` Result = `{"key1":{}}` *Remove the root element, should be empty* A = `{"key1":true}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{}` *Remove a sub element* A = `{"key1":{"key2":true}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key3":"value3"}}` *Remove a multiple sub elements* A = `{"key1":{"key2":{"key3":true}, "key4":true}}` B = `{"key1":{"key2":{"key3":{"key5":"value5"}}, "key4":{"key6":{"key7":"value7"}}}}` Result = `{"key1":{"key2":{}}}` *Remove one of the root elements if there are more than one* A = `{"key1":true}` B = `{"key1":{"key2":"value2", "key3":"value3"}, "key4":["a", "b", "c"]}` Result = `{"key4":["a", "b", "c"]}` *Remove with false should over write* A = `{"key1":{"key2":false, "key3":true}, "key4":false}` B = `{"key1":{"key2":"value2", "key3":"value3"}, "key4":[{"key5":"value5", "key6":"value6"}, {"key7": "value7"}]}` Result = `{"key1":{"key2":false}, "key4":false}` *Only objects with id's can be put into lists* A = `{"key1":[{"key2":true}]}` B = `{"key1":[{"key2":"value2"}, {"key3":"value3"}]}` Result = `{}` *Elements with {} should do nothing* A = `{"key1":{}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":"value2", "key3":"value3"}}` *Elements with nil should do nothing* A = `{"key1":{"key2":null}}` B = `{"key1":{"key2":"value2", "key3":"value3"}}` Result = `{"key1":{"key2":"value2", "key3":"value3"}}` ``` ## Overwrite The `overwrite` action will overwrite the old object with the new object. If you want to change a field or sub-object with the `overwrite` action, it is suggested to first call `Get` to obtain the original object. Then you can change the field or sub-object you would like to overwrite, followed by assembling the `overwrite` request with the entire object. In the following examples, A is being patched into B to create the Result: ```text A = `{"a":[{"id": "1"}], "blah":1}` B = `{"a":[{"id": "2"}], "blah":"string", "foo": "bar}` Result = `{"a":[{"id": "1"}], "blah":1}` A = `{}` B = `{"a":[{"blah": "2"}], "blah":"string"}` Result = `{}` ``` _Patch inputs with overwrite action will overwrite data object_ ```text *Before Patch* "input": { "id": "68be8de5a7de42c4873bf63fb6b8683d", "data": { "image": { "url": "https://samples.clarifai.com/your-image.jpg", }, "concepts": [ { "id": "car", "name": "car", "value": 1, "app_id": "your-application-id" } ], "geo": { "geo_point": { "longitude": 40.7129, "latitude": 74.0058 } } }, } *Patch Request* { "inputs": [ { "id": "68be8de5a7de42c4873bf63fb6b8683d", "data": { "concepts": [ { "id": "ferrari", "value": 1.0 } ], "metadata": { "foo": "bar" } } } ], "action": "overwrite" } *Result* "input": { "id": "68be8de5a7de42c4873bf63fb6b8683d", "data": { "image": { "url": "https://samples.clarifai.com/your-image.jpg", }, "concepts": [ { "id": "ferrari", "name": "ferrari", "value": , "app_id": "your-application-id" } ], "metadata": { "foo": "bar" } }, } ``` _Patch models with overwrite action will overwrite output\_info and name_ ```text *Before Patch* "model": { "id": "test-model-1580486147", "name": "test-model-1580486147", "app_id": "test-app-1580486122", "output_info": { "data": { "concepts": [ { "id": "car", "name": "car", "value": 1, "language": "en", "app_id": "test-app-1580486122" }, { "id": "ferrari", "name": "ferrari", "value": 1, "language": "en", "app_id": "test-app-1580486122" } ] }, "output_config": { "concepts_mutually_exclusive": false, "closed_environment": false, "max_concepts": 0, "min_value": 0, "test_split_percent": 10, "embed_model_version_id": "bb186755eda04f9cbb6fe32e816be104", "invalid_data_tolerance_percent": 5 }, "type": "concept", "type_ext": "concept" } } *Patch Request* { "models": [ { "id": "test-model-1580486147", "name": "my-new-model", "output_info": { "data": { "concepts": [ { "id": "animal" }, { "id": "dog" }, { "id": "cat" } ] }, "output_config": { "concepts_mutually_exclusive": true } } } ], "action": "overwrite" } *Result* "model": { "id": "test-model-1580486147", "name": "my-new-model", "app_id": "test-app-1580486122", "output_info": { "data": { "concepts": [ { "id": "animal", "name": "animal", "value": 1, "language": "en", "app_id": "test-app-1580486122" }, { "id": "cat", "name": "cat", "value": 1, "language": "en", "app_id": "test-app-1580486122" }, { "id": "dog", "name": "dog", "value": 1, "language": "en", "app_id": "test-app-1580486122" } ] }, "output_config": { "concepts_mutually_exclusive": true, "closed_environment": false, "max_concepts": 0, "min_value": 0, "test_split_percent": 10, "embed_model_version_id": "bb186755eda04f9cbb6fe32e816be104", "invalid_data_tolerance_percent": 5 }, "type": "concept", "type_ext": "concept" } } ``` --- ## Build RAG Apps # Build RAG Apps **Use the Clarifai SDKs to build RAG Applications** In the field of text generation, Retrieval-Augmented Generation (RAG) enhances the capabilities of Large Language Models (LLMs) by combining information retrieval with natural language generation. This approach addresses key limitations of LLMs by retrieving relevant data from external knowledge bases in real time, enriching responses with increased precision and contextual relevance. Clarifai SDKs streamline the creation of RAG-based applications by minimizing the complexity of integrating retrieval and generation steps. This makes it easier for developers to build powerful, context-aware AI solutions with improved accuracy and reliability. Click [here](https://www.clarifai.com/blog/what-is-rag-retrieval-augmented-generation) to learn more about RAG. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import CodeInit from "!!raw-loader!../../../code_snippets/python-sdk/rag/ini.py"; import CodeInitTS from "!!raw-loader!../../../code_snippets/python-sdk/rag/initialization.ts"; import CodeDU from "!!raw-loader!../../../code_snippets/python-sdk/rag/du.py"; import CodeDUTS from "!!raw-loader!../../../code_snippets/python-sdk/rag/datasetUpload.ts"; import CodeChat from "!!raw-loader!../../../code_snippets/python-sdk/rag/chat.py"; import CodeChatTS from "!!raw-loader!../../../code_snippets/python-sdk/rag/chat.ts"; import CodeChat2 from "!!raw-loader!../../../code_snippets/python-sdk/rag/chat2.py"; import CodeWF1 from "!!raw-loader!../../../code_snippets/python-sdk/rag/workflow1.py"; import CodeWF2 from "!!raw-loader!../../../code_snippets/python-sdk/rag/workflow2.py"; import CodeOutputDU from "!!raw-loader!../../../code_snippets/python-sdk/rag/outputs/du.txt"; import CodeOutputChat from "!!raw-loader!../../../code_snippets/python-sdk/rag/outputs/chat.txt"; import CodeOutputChat2 from "!!raw-loader!../../../code_snippets/python-sdk/rag/outputs/chat2.txt"; import CodeOutputWF1 from "!!raw-loader!../../../code_snippets/python-sdk/rag/outputs/wf1.txt"; import CodeOutputWF2 from "!!raw-loader!../../../code_snippets/python-sdk/rag/outputs/wf2.txt"; :::info Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them. ::: ## Prerequisites * You need a PAT (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can generate the PAT key in your personal settings page by navigating to the [Security section](https://clarifai.com/settings/security). * Clone the Clarifai Examples repository to get the data files required for the building RAG. ``` !git clone https://github.com/Clarifai/examples.git %cd /content/examples/ ``` :::info To run on a local system use: cd examples/ ::: :::note Before you proceed install ```llama_index``` using ```pip install llama-index-core==0.10.24``` ::: ## Initializing RAG The first step in building a RAG-based application is initializing the RAG object. You can do this in three ways: - **Using User ID** – Automatically creates a new app. - **Using App URL** – Initializes RAG with an existing app and its data. - **Using Workflow URL or ID** – Uses a specific workflow that includes the RAG Prompter and LLM model for seamless integration. :::tip You can set a specific version of LLM like this: `https://clarifai.com/mistralai/completion/models/mistral-7B-Instruct/model_version/version_id`. ::: {CodeInit} Image Output {CodeInitTS} We're using Mistral-7B-Instruct as the LLM for this RAG setup, but you can choose from various LLMs available in the Clarifai Community models [platform](https://clarifai.com/explore/models?filterData=%5B%7B%22field%22%3A%22use_cases%22%2C%22value%22%3A%5B%22llm%22%5D%7D%5D&page=1&perPage=24). The Clarifai SDKs let you configure parameters like `min_score`, `max_results`, and `prompt_template` to fine-tune data retrieval. You can also initialize RAG using a workflow created in the Clarifai Portal that includes a RAG Prompter. There are two ways to set this up — one is by providing the workflow URL as a parameter. :::info You should only use `RAG(workflow_url)` or `RAG(workflow)` when a rag workflow already exists in your app. ::: {CodeWF1} Output {CodeOutputWF1} The next option is to pass `workflow_id` parameter in `RAG.setup()`. This will create a new workflow in your app with the defined parameters. {CodeWF2} Output {CodeOutputWF2} ## Dataset Upload Next, upload your dataset—here, we're using a Vehicle Repair Manual as the RAG source. You can use the previously created RAG object to handle the upload. One key advantage of the Clarifai SDKs is that embeddings are automatically generated and stored in the vector database during upload, making the data instantly ready for retrieval. :::info Supported formats for upload include: DOC, PDF, plain text files, folders containing PDFs or DOCs, and URLs pointing to PDF, DOC, or text files. ::: {CodeDU} Output {CodeOutputDU} {CodeDUTS} ## Chat In the final step, we are going to perform information retrieval using RAG based on the data we provided. {CodeChat} Output {CodeOutputChat} {CodeChatTS} Now let's ask questions that are related to the answer we received before so that we can be sure the RAG has understood the context properly. {CodeChat2} Output {CodeOutputChat2} --- ## General AI Glossary # General AI Glossary **A Glossary of General AI Terms for Using the Clarifai Platform Effectively** ## A ### A/B Testing A statistical way of comparing two (or more) techniques, typically an incumbent against a new rival. It aims to determine which technique performs better, and whether the difference is statistically significant. ### Accuracy The fraction of correct predictions a model got right. The goal of any model is to get it to see the world as you see it. - In Multi-class classification, accuracy is determined by the number of correct predictions divided by the total number of examples. - In Binary classification, or for two mutually exclusive classes, accuracy is determined by the number of true positives added to the number of true negatives, divided by the total number of examples. ### Activation Function In artificial neural networks, the activation function of a node defines the output of that node given an input or set of inputs. ### Active Learning A machine learning term that refers to various methods for actively improving the performance of trained models. ### Adversarial Example Adversarial examples are specialized inputs created with the purpose of confusing a neural network, resulting in the misclassification of a given input. These notorious inputs are indistinguishable to the human eye, but cause the network to fail to identify the contents of the image. ### Adversarial Machine Learning A research field that lies at the intersection of machine learning (ML) and computer security. It enables the safe adoption of ML techniques in adversarial settings like spam filtering, malware detection, etc. ### Agents In the context of AI, agents are software that can independently perform specific tasks without human intervention. They often employ various tools, like calculators or web browsing, to process data and develop solutions. ### Agent System Operators [Agent system operators](https://docs.clarifai.com/portal-guide/agent-system-operators/) are "non-trainable," or "fixed function," models that help you connect, route, and control the inputs and outputs that you send through your workflows. Operator models are critical building blocks for creating more advanced workflows. :::note Operators Examples - **Concept Thresholder** allows you to threshold input concepts according to both a threshold and an operator (>, >=, =, \ " threshold type and set the threshold value to 0.9, only concepts that have been predicted with a confidence score greater than 0.9 will be sent as outputs from the concept thresholder, and other concepts will be ignored. - **Region Thresholder** allows you to threshold regions based on the concepts that they contain using a threshold per concept and an overall operator (>, >=, =, \ ![Convolution](/img/glossary_convolution.gif) Source: Deep Learning Methods for Vision [[2](https://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/)] ### Convolutional Neural Network Convolutional neural networks are deep artificial neural networks that are used primarily to classify images (e.g. name what they see), cluster them by similarity (photo search), and perform object recognition within scenes. ### CoreML A proprietary format used to encode weights for Apple devices that takes advantage of the hardware-accelerated neural engine present on iPhone and iPad devices. ### CreateML A no-code training tool created by Apple that will train machine learning models and export them to CoreML. It supports classification and object detection along with several types of non computer-vision models (such as sound, activity, and text classification). ### Curse of Dimensionality The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings, such as the three-dimensional physical space of everyday experience. ### Custom Dataset A set of images and annotations pertaining to a domain-specific problem. In contrast to a research benchmark dataset like Coco or Pascal Voe. ### Custom Training This refers to the process of training a machine learning model on your own specific dataset to perform a particular task. This is in contrast to using pre-trained models, which are already trained on generic datasets and can be fine-tuned for specific tasks. ## D ### Data In the data science and AI world, data is any collection of information that is converted into a digital form. Data is plural, with the singular being “datum.” :::tip It’s important to distinguish between structured and unstructured data: - Structured data is highly specific and is stored in a predefined format, such as a spreadsheet table; - Unstructured data is a conglomeration of many varied types of data that are stored in their native formats, such as images, video, audio, and text. ::: ### Data Annotation The process of labeling datasets to be used as inputs for machine learning models. ### Data Curation The process of collecting, organizing, cleaning, labeling, and maintaining data for use in training and testing models. ### Data Mining The process by which patterns are discovered within large sets of data with the goal of extracting useful information from it. ### Dataset A collection of data and ground truth of outputs that you use to train machine learning models by example. ### De-Duplication The removal of identical data, or data that is so similar that for all intents and purposes it can be considered duplicate data. Using visual search, a similarity threshold can be set to decide what should be removed. ### Deep Learning The general term for machine learning using layered (or deep) algorithms to learn patterns in data. It is most often used for supervised learning problems. ### Deep Neural Network An artificial neural network (ANN) with multiple layers between the input and output layers. It uses sophisticated mathematical modeling to process data in complex ways. ### Deploy Taking the results of a trained model and using them to make inferences on real-world data. This could mean hosting a model on a server or installing it on an edge device. ### Detection Also known as object detection. It involves identifying the presence, location and type of objects within images or video frames. :::warning Detection Models Detection comprises two tasks; listing “what” things appear in an image, and “where” they appear. Results are returned as bounding boxes along with the names of the detected items. ::: ### Diversity, Equity, and Inclusion (DEI) A term used to describe policies and programs that promote the representation and participation of different groups of individuals, including people of different ages, races and ethnicities, abilities and disabilities, genders, religions, cultures, and sexual orientations. ### Domain Adaptation A type of transfer learning, domain adaptation is a technique to improve the performance of a model where there is little data in the target domain by using knowledge learned by another model in a related domain. An example could be training a model to recognize taxis using a model that recognizes cars. ### Domain Model Focuses on understanding a single domain, such as travel, weddings, food, not-safe-for-work (NSFW), etc. ## E ### Edge AI Data is processed on the same device that produces it, or at most, on a nearby computer. Edge AI means there’s no reliance on distant cloud servers or other remote computing nodes, allowing the AI to work faster, and respond more accurately to time-sensitive events. ### Edge Computing A distributed computing framework that brings enterprise applications closer to data sources such as loT devices or local edge servers. ### Embeddings A low-dimensional representation of a model’s input that has rich semantic information. It involves conversions of data to a feature representation where certain properties can be represented by notions of distance in a neural network. In other words, the translation of data to a continuous, fixed-length representation of something that is otherwise difficult to represent. :::warning Embedding models Computers and models can’t understand images and text like humans do. Embedding models take unstructured input like images, audio, text, and video and transform them into a series of numbers called vectors which can then be input into the prediction models. ::: ### Embedding Space The d-dimensional vector space that features from a higher-dimensional vector space are mapped to. Ideally, the embedding space contains a structure that yields meaningful mathematical results. ### Emotional AI Emotional AI refers to technologies that use affective computing and artificial intelligence techniques to sense, learn about and interact with human emotional life. ### Endpoint A task or end goal for a machine learning model. For example, we might get this question: **Question**: “Is X Endpoint doable in your models?” **Answer**: Reference Clarifai’s API documentation to review endpoints and determine if we can do something. Our explorer tool essentially translates our API prediction scripts. ### Ensemble Models Ensemble models are a machine learning approach to combine multiple other models in the prediction process. While the individual models may not perform very well, when combined they can be very powerful indeed. ### Epoch It refers to one complete pass through the entire training dataset. It's a fundamental unit of measurement that signifies how many times the training data has been exposed to the learning algorithm. ### Evaluation The process of assessing a model's performance on a specific task. It's essentially how you check how well your model learned from the training data and how well it can generalize to unseen data. :::warning Evaluation Leaderboard [Evaluation Leaderboard](https://docs.clarifai.com/portal-guide/evaluate/leaderboard) is a ranking system that compares the performance of your models based on their evaluation results. It’s a scoreboard that provides useful insights for the model versions in your apps and ranks them according to selected benchmark metrics. ::: ### Extensible Markup Language (XML) A markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. ## F ### F Score A weighted average of the true positive rate of recall and precision. ### Facial Recognition An application capability of identifying or verifying a person from an image or a video frame by comparing selected facial features from the image and a face database. ### False Negatives An error where a model falsely predicts an input as not having a desired outcome, when one is actually present. (Actual Yes, Predicted No). ### False Positives An error where a model falsely predicts the presence of the desired outcome in an input, when in reality it is not present (Actual No, Predicted Yes). ### Feature Extraction The process of transforming raw data into a more usable format for machine learning algorithms. It involves identifying and extracting the most relevant pieces of information (features such as texture, shape, lines, and edges) from the data, while discarding irrelevant details. ### Fine-Tuning [Fine-tuning](https://docs.clarifai.com/portal-guide/model/deep-training/) is a deep learning technique that refers to taking a pre-trained model and further training it on a new dataset or task. The term "fine-tuning" implies making small adjustments or refinements to the already learned representations in the pre-trained model rather than training from scratch. It leverages the power of pre-trained models to improve their performance on a new, related task. It involves taking a pre-trained model, which was previously trained on a vast dataset for a general-purpose task, and tailoring it to a more specific task. :::warning template types You can take advantage of a variety of our pre-configured [templates](https://docs.clarifai.com/portal-guide/model/deep-training/#template-types) when developing your deep fine-tuned models. Templates give you the control to choose the specific architecture used by your neural network, and also define a set of hyperparameters that you can use to fine-tune the way your model learns. Examples include MMClassification_ResNet_50_RSB_A1 and Clarifai_InceptionBatchNorm for visual classification tasks, MMDetection_YoloF and MMDetection_SSD for visual detection tasks, and MMSegmentation_SegFormer for visual segmentation tasks. ::: ### Folksonomy User-generated system of classifying and organizing online content into different categories by the use of metadata such as electronic tags. ### Framework Deep learning frameworks implement neural network concepts. Some are designed for training and inference —TensorFlow, PyTorch, FastAI, etc. And others are designed particularly for speedy inference — OpenVino, TensorRT, etc. ## G ### Generalization Refers to a model's ability to make correct predictions on new, previously unseen data as opposed to the data used to train the model. ### Generative Adversarial Networks (GANs) A class of artificial intelligence algorithms used in unsupervised machine learning, implemented by a system of two neural networks contesting with each other in a zero-sum game framework. This technique can generate photographs that look at least superficially authentic to human observers, having many realistic characteristics (though in tests, people can tell the real ones from those generated in many cases). ### Generative AI Models that can be trained using existing content like text, audio files, or images to create new original content. ### Graphics Processing Unit (GPU) A specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mobile phones, personal computers, and more. ### Graphics Processing Unit (GPU) Memory The amount of information your GPU can fit on it. A bigger GPU will be able to process more information in parallel which means it can support bigger models (or bigger batch sizes) without running out of memory. If you run out of GPU memory, it will crash your program. ### Green AI Efforts to make AI more energy-efficient and environmentally friendly are gaining momentum, with a focus on reducing the carbon footprint of training and running AI models. ### Grid Search Grid search is a tuning technique that attempts to compute the optimal values of hyperparameters for training models by performing an exhaustive search through a subset of hyperparameters. ### Ground Truth The answer key for your dataset. This is how you judge how well your model is doing and calculate the loss function we use for gradient descent. It's also what we use to calculate our metrics. :::warning important Having a good ground truth is extremely important. Your model will learn to predict based on the ground truth you give it to replicate. ::: ### gRPC [gRPC](https://docs.clarifai.com/api-guide/api-overview/api-clients/grpc-vs-http-channels/) (gRPC Remote Procedure Calls) is an open-source framework developed by Google that facilitates efficient and robust communication between services, typically in microservices architectures. It enables the definition and implementation of remote procedure calls (RPCs) with a focus on performance, scalability, and flexibility. gRPC leverages HTTP/2 for transport, Protocol Buffers (protobuf) for serialization, and supports multiple programming languages. We initially built our API on gRPC. ## H ### Hashing In machine learning, a mechanism for bucketing categorical data, particularly when the number of categories is large, but the number of categories actually appearing in the dataset is comparatively small. ### Hidden Layer A synthetic layer in a neural network between the input layer (that is, the features) and the output layer (the prediction). Hidden layers typically contain an activation function (such as ReLU) for training. A deep neural network contains more than one hidden layer. ### Holdout Data Examples intentionally not used during training. The validation dataset and test dataset are examples of holdout data. It helps evaluate your model's ability to generalize to data other than the data on which it was trained. ### Hosted Model A set of trained weights located in the cloud that you can receive predictions from via an API. ### Hugging Face Hugging Face is a leading platform in the field of natural language processing (NLP) and machine learning, providing tools and resources that simplify the process of building, training, and deploying machine learning models. Its extensive libraries, pre-trained models, and collaborative ecosystem empower developers and researchers to advance their NLP projects efficiently and effectively. ### Human Workforce Workers who can help to complete work on an as-needed basis, which for purposes usually means labeling data (images). ### Hyperparameter The levers by which you can tune your model during training. These include things like learning rate and batch size. You can experiment with changing hyperparameters to see which ones perform best with a given model for your dataset. ## I ### Inference Making predictions using the weights you save after training your model. ### lmageNet A large visual database designed for use in visual object recognition software research. ### Image Recognition The ability of software to identify objects, places, people, writing and actions in images. ### Image Segmentation The process of dividing a digital image into multiple segments with the goal of simplifying the representation of an image into something that is easier to analyze. Segmentation divides whole images into pixel groupings, which can then be labeled and classified. ### Image-to-Text Image-to-text generation, also known as image captioning, refers to the process of generating textual descriptions or captions for images. It involves using a model to analyze the content of an image and then generate a coherent and relevant textual description that describes what is happening in the image — similar to how humans would describe it. ### Implicit Bias Automatically making an association or assumption based on one's mental models and memories. Implicit bias can affect how data is collected and classified, and how machine learning systems are designed and developed. ### Indexing Indexing collects, parses, and stores your inputs to facilitate fast and accurate information retrieval. Indexing happens automatically every time you add new inputs to your app. Indexing enables responsive visual search, data clustering, concept search and model training. ### Information Retrieval The area of Computer Science studying the process of searching for information in a document, searching for documents themselves, and also searching for metadata that describes data and for databases of texts, images, or sounds. ### Input An input is the data you're providing into the Clarifai platform. Inputs and their predictions are indexed so that they can be used for search. You can also add your own concepts to inputs to use when training your own model. When you upload data to an app, that is known as "POSTing" inputs. ### Input Layer The first layer (the one that receives the input data) in a neural network. ### Inputs-Manager The Inputs-Manager page is the main page showcasing all the inputs in your app. ### Input-Viewer The Input-Viewer page is the main page that showcases the details of a single input available in your app. If you click an input listed on the Input-Manager page, you'll be redirected to the viewer page for that input, where you can see its details. ### Intelligent Character Recognition (ICR) Related technology to OCR designed to recognize handwritten characters. ### loT The interconnection via the internet of computing devices embedded in everyday objects, enabling them to send and receive data. ## J ### Jetson An edge computing device created by NVIDIA that includes an onboard GPU. ### JSON A freeform data serialization format originally created as part of JavaScript but now used much more broadly. Many annotation formats use JSON to encode their bounding boxes. ### Jupyter Notebook A common data science tool that enables you to execute Python code visually. Each "cell" in the notebook is a block of code that you can execute by hitting "Ctrl+Enter." The results of the execution are displayed below the cell. ## K ### Knowledge Graph [A knowledge graph](https://docs.clarifai.com/api-guide/concepts/knowledge_graph) is not the kind of graph you may be used to showing the relationship between two variables. Rather, it’s a collection of nodes and edges where the nodes represent concepts, entities, relationships, and events, and the edges represent the connections between them. ## L ### Label Assigning a class, category, or concept to a specific object in your dataset. ### Labeler [Labeler](https://docs.clarifai.com/portal-guide/annotate/labeling-tools) tasks platform provides special tools for working with images, videos, and texts for classification, detection, and segmentation labeling tasks. With the tools, you can annotate your inputs faster and more conveniently. ### Labeling Labeling (also known as annotating) refers to the process of adding one or more relevant tags, or keywords — usually referred to as concepts — that best describe the state of your inputs. :::warning AI-Assisted Labeling [AI-assisted](https://docs.clarifai.com/portal-guide/annotate/ai-assist/) labeling is an innovative Clarifai feature that leverages artificial intelligence technology to assist and optimize the process of annotating data. ::: ### Labeling Criteria An essential guide to the labeling requirements for a project. It should include instructions for the labeling process itself as well as written definitions and a multitude of visual examples for each concept. These examples help the labelers anticipate the visual differences and edge cases that can occur within one single concept. ### Layers The parts that make up the composition of a neural network. ### Learning Rate Learning rate is a crucial hyperparameter that governs the step size the model takes when updating its internal parameters during training. It essentially controls how much the model adjusts its weights in response to the errors it makes on the training data. :::warning Step Size Step size refers to the amount by which the model parameters are adjusted during each iteration of the training process. A higher learning rate signifies a larger step size, and a lower learning rate corresponds to a smaller step size. ::: ### Logistic Regression A type of regression model that outputs a continuous value from a linear combination of input features. Unlike classification models, regression provides an answer to a question of *quantity*. :::info - **Classification** is the task of predicting a discrete class label. - **Regression** is the task of predicting a continuous quantity. ::: ### LOPQ **L**ocally **O**ptimized **P**roduct **Q**uantization is used to find the different embeddings for search, which we implement instead of the General model. This is a more bespoke approach to building Visual Search Models. ### Loss Function Loss function (also known as a cost function or objective function) is a mathematical function that measures the difference between the predicted output of a model and the actual output (target value). The purpose of a loss function is to quantify how well or poorly a model's predictions match the true data, providing a way to guide the model's training process by giving feedback on its performance. ## M ### Machine Intelligence An umbrella term that encompasses machine learning, deep learning and classical learning algorithms. ### Machine Learning A general term for algorithms that can learn patterns from existing data and use these patterns to make predictions or decisions with new data. ### Masks [Mask](https://docs.clarifai.com/portal-guide/annotate/label-types#masks) is a type of image segmentation that defines the exact boundaries of an object at a pixel level. ### Masked Language Model A language model that predicts the probability of candidate tokens to fill in blanks in a sequence. :::warning Masked Language Models Language models that predict the probability of which words make the most sense to fill in blanks in a sequence. A simple example could be “Good _____, how are you?” where probable candidate words could be “morning,” “day,” or “evening.” ::: ### Metadata Metadata are additional pieces of information you attach to your inputs when uploading them to the Clarifai platform. This can include product IDs, user IDs, or any other relevant details necessary for achieving specific business outcomes. Note that the metadata should be a valid JSON object. For example, you can use metadata to quickly filter your data and combine it with other ranking criteria. ### Misclassification Rate Rate used to gauge how often a model’s predictions are wrong. ### MLOps Also known as Machine Learning Operations. Best practices for organizations to operationalize machine learning. Often involves collaboration between data scientists and DevOps professionals to manage production ML. ### Modality A high-level data category. For example, numbers, text, images, video and audio are five different modalities. ### Model Models convert inputs to outputs. They generate predictions based on the patterns extracted from the input data in concepts. Clarifai provides many different models that each ‘see’ the world differently — with a unique group of concepts. :::warning models Clarifai has built some great models for you to use, but there are times when you wish you had a model that sees the world the way you see it — with your own concepts. You can use your own model by adding images with concepts, training it, and then specifying it when making predictions. ::: ### ModelOps ModelOps is focused primarily on the governance and life cycle management of a wide range of operationalized artificial intelligence (AI) and decision models, including machine learning, knowledge graphs, rules, optimization, linguistic and agent-based models. ### Model Size The number of parameters (or neurons) a model has and can be measured in terms of the size of the weights file on disk. ### Model Training Model training is the heart of machine learning. It's where a machine learning model learns to perform a specific task by being exposed to a large amount of data. ### Modules [Modules](https://docs.clarifai.com/portal-guide/modules/) are custom plugins that provide both UI elements and the ability to perform some computation around our API. Modules enable you to extend the Clarifai UIs, augment backend processing capabilities, and build customized solutions quickly and easily. ### Multi-Class Classification Classification problems that distinguish between more than two classes. For example, there are approximately 53 species of maple trees, so a model that categorized maple tree species would be multi-class. ### Multimodal Model A model whose inputs and/or outputs include more than one modality. For example, consider a model that takes both an image and a text caption (two modalities) as features, and outputs a score indicating how appropriate the text caption is for the image. ## N ### Named Entity Recognition (NER) A sub-task of information extraction that seeks to identify and classify named entities in text into predetermined categories, such as the names, locations, parts of speech, etc. ### Natural Language Processing (NLP) A branch of Al that helps computers understand, interpret, and manipulate human language. This field of study focuses on helping machines understand human language in order to improve human-computer interfaces. ### Natural Language Understanding Determining a user's intentions based on what the user typed or said. For example, a search engine uses natural language understanding to determine what the user is searching for based on what the user typed or said. ### Neural Architecture Search Automatically trying many variations of model layouts and hyperparameters to find the optimal configuration. ### Neural Network Series of algorithms that endeavor to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates. ### Neuro-Symbolic The combining of neural and symbolic AI architectures to address complementary strengths and weaknesses of each, providing a robust AI capable of reasoning, learning, and cognitive modeling. ### Neuron A unit in an Artificial Neural Network processing multiple input values to generate a single output value. :::tip In machine learning, the “neurons” are instead mathematical models that simulate transmitting information the way that neurons do in the human brain — instead of electrical and chemical signals, like we have in human brains, these artificial points take numbers in and spit numbers out. ::: ### Noise Signals with no causal relation to the target function. ### Normalization The process of converting an actual range of values into a standard range of values, typically -1 to +1 or O to 1. ## O ### Object Detection A computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. This technique also involves localizing the object in question, which differentiates it from classification, which only tells the type of object. ### Object Recognition Also known as object classification. A computer vision technique for identifying objects in images or videos. ### Object Tracking The process of following a specific object of interest, or multiple objects, in a given scene. It traditionally has applications in video and real-world interactions where observations are made following an initial object detection. ### On-Premise Software Software that is installed and runs on computers located on the premises of the organization using that software versus at a remote facility such as a server farm or on the cloud. ### One-Shot Classification A model that only requires that you have one training example of each class you want to predict on. The model is still trained on several instances, but they only have to be in a domain similar to your training example. ### Open Neural Network Exchange (ONNX) ONNX is an open format that represents machine learning models. ### OpenAI A U.S. AI research organization whose mission is to ensure that artificial general intelligence benefits all humanity. ### Operation An operation is an action that is performed via our API or User Interface. It can include actions such as predictions, searches, input uploads, training custom models, model evaluations and more. ### Optical Character Recognition (OCR) A computer system that takes images of typed, handwritten, or printed text and converts them into machine-readable text. ### Optimization The selection of the best element (with regard to some criterion) from some set of available alternatives. ### Organization Clarifai Organizations is a feature within the Community platform that lets you consolidate multiple Clarifai accounts into an organization. It allows you to manage your company’s projects centrally, enhance collaboration within your team, and drive better results with your AI-powered software solutions. ### Output An output, usually in the form of a prediction or predictions, is the data returned to you when you send an input into a model. Because of their close relationship, the terms "outputs" and "predictions" are sometimes used interchangeably. ![Inputs to outputs](/img/glossary_inputs_outputs.png) ### Outsourced Labeling Paying people to annotate, or label, your data. Its effectiveness can depend on the domain expertise of annotators. Providing comprehensive labeling criteria is crucial for training annotators before beginning a project. ### Overfitting A machine learning problem where an algorithm is unable to discern information that is relevant to its assigned task from information which is irrelevant within training data. Overfitting inhibits the algorithm's predictive performance when dealing with new data, as it could recognize an unrelated feature as a positive aspect for a concept. ## P ### Parameter Any characteristic that can be used to help define or classify a system. In Al, they are used to clarify exactly what an algorithm should be seeking to identify as important data when performing its target function. ### Pattern Recognition A branch of machine learning that focuses on the recognition of patterns and regularities in data, although it is in some cases considered to be nearly synonymous with machine learning. ### Personal Access Tokens [A Personal Access Token](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens) (usually shortened as PAT) is a kind of key that authenticates a user across all applications they have access to. It's not linked to a specific application. ### Pipeline The process of going from raw images to prediction. Usually this encompasses collecting images, annotation, data inspection and quality assurance, transformation, preprocessing and augmentation, training, evaluation, deployment, inference (and then repeating the cycle to improve the predictions). ### Polygon A (usually non-rectangular) region defining an object with more detail than a rectangular bounding box. Polygon annotations can be used to train segmentation models or to enhance the performance of object-detection models by enabling a more accurate bounding box to be maintained after augmentation. ### Pooling It's also known as downsampling. It reduces the dimensionality of each feature map, but retains the most important information for that image. In the end, pooling makes it possible to recognize elements no matter where they are located. The goal of this step is to progressively reduce the spatial size of the input representation, making it more manageable to process. ### Portal Portal is a web application that allows you to interact with the Clarifai platform. You can view all the inputs you have added, perform searches, train new models, and more. ### Positive Predictive Value (PPV) Very similar to precision, except that it takes prevalence into account. In the case where the classes are perfectly balanced (meaning the prevalence is 50%), the positive predictive value is equivalent to precision. ### Postman [Postman](https://docs.clarifai.com/api-guide/api-overview/helpful-api-resources/using-postman-with-clarifai-apis) is a software tool that acts as an API platform for building and using APIs. With Postman, you can use, hit, or test the Clarifai API without the need to use the Portal or call the endpoints programmatically. Postman also allows you to make API calls and generate code snippets in your favorite programming language. ### Precision Indicator of a machine learning model's performance – the quality of a positive prediction made by the model. Refers to the number of true positives divided by the total number of positive predictions. ### Prediction An attempt by a model to replicate the ground truth. It’s an answer to the question: “What can you tell me about this input?” When you call predict on an input, you will receive one or more predictions of different concepts that apply to it. Predictions vary based on the concepts included in a given model. A prediction usually contains a confidence value for each class. :::warning Predictive Model A model that uses observations measured in a sample to gauge the probability that a different sample or remainder of the population will exhibit the same behavior or have the same outcome. ::: ### Pre-Trained Model A model, or the component of a model, that has been preliminary trained, generally using another data set; for example, finding lines, corners, and patterns of colors. Pre-training on a large dataset like the huge Common Objects in Context (COCO), which has 330,000 images with 1.5 million objects to detect, can reduce the number of custom images you need to obtain satisfactory results. ### Prevalence The rate of how often the “yes” condition actually occurs in a sample. ### Private The term “private” refers to resources within the Clarifai platform that you own and are not available for everyone to access. They can be apps, models, workflows, or other resources that are only accessible to you or anyone within your team. ### Production The deployment environment where the model will run in the wild on real-world images (as opposed to the testing environment where the model is developed). ### Prompt Engineering The practice of crafting effective prompts to elicit better outputs from language models has become an area of interest, especially with models like ChatGPT and DALL-E. ### Proximal Policy Optimization (PPO) PPO is a simplified and efficient reinforcement learning algorithm that refines policy functions to optimize performance in a given environment. It builds on TRPO's approach to ensuring stable policy updates but simplifies the process, making it more practical for implementation. ### Pruning The use of a search algorithm to cut off undesirable solutions to a problem in an Al system. It reduces the number of decisions that can be made by the Al system. ### Public The term “public” refers to resources within the Clarifai platform that are generally accessible to all users. They can be apps, models, workflows, or other resources that any user can interact with and integrate into their own use case. ### PyTorch A popular open-source deep learning framework developed by Facebook. It focuses on accelerating the path from research prototyping to production deployment. ## Q ### Quick Training A process that accelerates the training of machine learning models by utilizing advanced techniques and optimized architectures. It often involves transfer learning, where a pre-trained model is fine-tuned with a specific, smaller dataset, making the training more efficient and less resource-intensive. ## R ### RAG (Retrieval Augmented Generation) This combines a retrieval system with a generator to improve the factual accuracy of generated text. It retrieves relevant documents and uses them to inform the generation process. ### Rate Limit [Rate limit](https://docs.clarifai.com/api-guide/api-overview/rate-limits) is a control mechanism we implement to manage the number of requests an individual user or application can make to our API within a specific period. ### Recall (Sensitivity) The fraction of relevant instances that have been retrieved over the total amount of relevant instances. ### Receiver Operating Characteristic (ROC) Curve This is a commonly used graph that summarizes the performance of classifiers over all possible thresholds. It is generated by plotting the True Positive Rate (y-axis) against the False Positive Rate (x-axis) as you vary the threshold for assigning observations to a given class. ### Recurrent Neural Network A type of artificial network with loops in them, allowing recorded information, like data and outcomes, to persist by being passed from one step of the network to the next. They can be thought of as multiple copies of the same network with each passing information on to its successor. ### Regression A statistical measure used to determine the strength of the relationships between dependent and independent variables. ### Reinforcement Learning A type of machine learning in which machines are "taught" to achieve their target function through a process of experimentation and reward receiving positive reinforcement when its processes produce the desired result and negative reinforcement when they do not. This is differentiated from supervised learning, which would require an annotation for every individual action the algorithm would take. ### ReLU In the context of artificial neural networks, the ReLU (rectified linear unit) activation function is an activation function that outputs the same as its input if the input is positive, and zero if the input is negative. A related function is the leaky rectified linear unit (leaky rectified linear unit), which assigns a small positive slope for x < 0. ( ͡° ͜ʖ ͡°) ### Resource A machine learning resource is any reusable component that provides functionality, data, or structure to facilitate the development, training, and deployment of machine learning models and applications. Examples include apps, models, workflows, datasets, and modules. ### Responsible AI An umbrella term for aspects of making appropriate business and ethical choices when adopting AI, including business and societal value, risk, trust, transparency, fairness, bias mitigation, explainability, accountability, safety, privacy, and regulatory compliance. ### Review The process of [reviewing](https://docs.clarifai.com/portal-guide/annotate/review) the work performed by your team of labelers for quality control purposes. ## S ### Scopes [Scopes](https://docs.clarifai.com/clarifai-basics/authentication/scopes) provide control over the set of functionality and features available when using our API. Modifying scopes allows for fine-grained control over the data that users have access to, and can help keep your app secure from unauthorized intrusions. ### Scripts An aggregation of several commands that when executed together, perform specific actions. ### Search All the inputs in your app are indexed by both the concepts applied by the app’s default base workflow, and by their own properties. Search, in the context of Clarifai, refers to finding relevant inputs in your app by text \(match concepts\), reverse image search \(similar-looking images\), or any other criteria. :::tip Rank and Filter - The rank feature in Clarifai's search functionality allows users to specify criteria for prioritizing search results based on relevance or similarity to a reference. - The filter feature in Clarifai's search functionality allows users to narrow down search results based on specific criteria or conditions. Filters act as constraints that refine the set of results returned by a search query. ::: :::warning Search Operations - In Clarifai, the OR search operation provides users with a flexible means to retrieve search results that satisfy one or more specified criteria. By employing the OR filter, users can broaden their search scope to include items that meet any of the provided conditions. - In Clarifai, the AND operation allows users to refine search results by specifying multiple criteria that must all be met simultaneously. For example, if a user searches for images containing both "dog" and "deer" concepts, only images that are labeled as both dogs and deers will be returned. ::: ### Search Query A query that a user feeds into a search engine to satisfy his or her information needs. If the query itself is a piece of visual content then that is what is known as a "visual search query." ### Secure Data Hosting [Secure Data Hosting](https://docs.clarifai.com/product-updates/upcoming-api-changes/secure-data-hosting) (SDH) is an extra layer of security and access control to the data we store on Amazon S3. It acts as an intermediary or middleman, allowing authorized users to access the content stored on Amazon S3. This lets us enhance the security of users’ data while leveraging the robust storage and infrastructure capabilities provided by Amazon S3. ### Segmentation Segmentation is a type of labeling where each pixel in an input is labeled with given concepts, providing pixel-by-pixel details for a given object. For example, image segmentation is considered more precise than other object detection methods because it labels individual pixels within an image. ### Segmentation Model Instead of bounding boxes returned for each concept, this model indicates via a heat map and trace (think coloring book) of regions for each concept. ### Selective Filtering When a model ignores "noise" to focus on valuable information. ### Siamese Networks A different way of classifying image where instead of training one model to learn to classify image inputs it trains two neural networks that learn simultaneously to find similarity between images. ### Signal Inputs, information, data. ### Software Development Kit (SDK) A set of software development tools that allows for the creation of applications on a specific platform. Example is the Clarifai's Python SDK. ### Specificity The rate of how often a model predicts "no," when it's actually "no." ### Standard Classification The process by which an input is assigned to one of a fixed set of categories. In machine learning, this is often achieved by learning a function that maps an input to a score for each potential category. ### Starring This refers to marking any resource — be it your own or publicly accessible in the Community — as a favorite. Simply click the star icon to designate any resource, including apps, models, datasets, workflows, and modules, as your favorite. ### Strong AI A theoretical form of AI that replicates human functions, such as reasoning, planning, and problem-solving. ### Structured Data Data that resides in a fixed field within a file or record. Structured data is typically stored in a relational database. It can consist of numbers and text, and sourcing can happen automatically or manually, as long as it's within an RDBMS structure. ### Supervised Learning A machine learning approach that's defined by its use of labeled datasets. These datasets are designed to train or "supervise" algorithms into classifying data or predicting outcomes accurately. Using labeled inputs and outputs, the model can measure its accuracy and learn over time. ### Symbiotic Intelligence A combination of human and artificial intelligence. Instead of relying on memory, or having to open a book, or visit a website, an enhanced human could have access to all of the information that is stored on the internet, and an advanced AI could feed the relevant data points to the human brain, enabling the human to be fully in control. ### Synthetic Data Training data that is artificially generated rather than produced by real-world events. ### Synthetic Intelligence Synthetic intelligence (SI) is an alternative term for artificial intelligence emphasizing that the intelligence of machines need not be an imitation or in any way artificial; it can be a genuine form of intelligence. An analogy can be made with simulated diamonds (such as cubic zirconia) versus synthetic diamonds (real diamonds made of carbon created by humans). ## T ### Target Function The end goal of an algorithm. ### Task [Task](https://docs.clarifai.com/portal-guide/annotate/create-a-task) is the work that needs to be done for labeling the inputs in an app. It's a powerful way to help your team annotate inputs fast and efficiently. ### Taxonomy In essence, a taxonomy is a models worldview, or the framework for how your model sees its training data. In practice, it’s a list of visually-distinct model concepts and the definitions of those concepts. ### Temporal Data Data recorded at different points in time. ### TensorFlow A large-scale, distributed, machine learning platform that serves as a virtual “sandbox” for testing machine learning prediction parameters. It’s an open-source software library used for machine learning applications, such as neural networks. ### Test Dataset The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset. ### Torch A scientific computing framework with wide support for machine learning algorithms, written in C and Lua. ### Train Training a model is the process of iteratively adjusting its parameters to converge on the weights that optimally mimic the training data. It’s when you teach a model to “learn” from all the annotated concepts on your inputs. ### Training Dataset Training data is the initial dataset used to train machine learning algorithms. Models create and refine their rules using this data. It's a set of data samples used to fit the parameters of a machine learning model to train it by example. ### Transfer Learning [Transferring information](https://docs.clarifai.com/portal-guide/model/model-types/transfer-learning) from one machine learning task to another. It might involve transferring knowledge from the solution of a simpler task to a more complex one, or involve transferring knowledge from a task where there is more data to one where there is less data. ### Transformer A neural network that transforms a sequence of elements (like words in a sentence) into another sequence to solve sequence-to-sequence tasks. ### True Negatives Actual negatives that are correctly identified as such (Actual No, Predicted No). ### True Positives Actual positives that are correctly identified as such (Actual Yes, Predicted Yes). ### Trust Region Policy Optimization (TRPO) TRPO is a reinforcement learning technique that iteratively updates policy functions to enhance performance. It guarantees that each new policy is at least as good as the previous one, avoiding large updates that could degrade performance. ### Turing Test A test developed by Alan Turing in 1950, used to identify true artificial intelligence. It tested a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. ## U ### Unstructured Data Unstructured data is information that either does not have a predefined data model or is not organized in a predefined manner. Unstructured data may include documents, images, video and audio. ### Unsupervised Learning Uses machine learning algorithms to analyze and cluster unlabeled datasets. These algorithms discover hidden patterns or data groupings without the need for human intervention. Its ability to discover similarities and differences in information makes it the ideal solution for exploratory data analysis, cross-selling strategies, customer segmentation, and image recognition. ` ### Usage Dashboard [Usage dashboard](https://docs.clarifai.com/portal-guide/usage-dashboard) provides measurable insights into your utilization of the Clarifai Community platform during the selected period. With the dashboard, you can monitor your consumption of Clarifai’s resources, better understand how you interact with the Clarifai platform, and optimize your configurations accordingly. ## V ### Validate The model is given new, previously unseen data, and then metrics are collected on how well it performs predictions on them. This is analogous to a human learning math problems using one set of questions, then tested to see if they learned properly with a different set of questions. ### Validation Data Set The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration. ### Variance The error due to sensitivity to fluctuations in the training set computed as the expectation of the squared deviation of a random variable from its mean. ### Vector Database A vector database or a vector store is a specialized type of database designed to store, manage, and search high-dimensional vectors efficiently. These vectors typically represent data points such as images, text, audio, or other forms of data that have been transformed into numerical arrays by machine learning models. The primary purpose of a vector database is to facilitate rapid and accurate similarity searches, which are crucial for various machine learning tasks. ### Verify/Verification The process of verifying that labeled data has been labeled correctly in adherence to the ground truth. ### Video Frame Interpolation The process of synthesizing several frames of video, between two originally adjacent frames of video. ### Visual Anomaly [Visual anomaly](https://docs.clarifai.com/portal-guide/model/model-types/visual-anomaly) is a type of deep fine-tuned model that can be used to identify unusual or anomalous patterns in images and videos that differ from the expected norm. ### Visual Dictionary A document that defines every model concept with a written definition and also a wide array of visual examples. This helps establish ground truth by providing confirmation that each involved party understands the model’s worldview, or taxonomy. ### Visual Match Instead of doing a search, which returns the items in the database in sorted order, a visual match couple be considered returning a yes/no answer of whether the query is close enough to any item in the database to be considered a “match.” ### Visual Recognition The ability of software to identify objects, places, people, writing, and actions in images and videos. ### Visual Search The ability of software to find visually similar content based on an image or video query. ## W ### WeakAI Also known as narrow Al, weak Al refers to a non-sentient computer system that operates within a predetermined range of skills and usually focuses on a singular task or small set of tasks. Most Al in use today is weak Al. ### Weight A coefficient for a feature in a linear model, or an edge in a deep network. The goal of training a linear model is to determine the ideal weight for each feature. If a weight is 0, then its corresponding feature does not contribute to the model. ### Width The number of neurons in a particular layer of a neural network. ### Workflows [Workflows](https://docs.clarifai.com/portal-guide/workflows/) is a useful Clarifai's feature that allows you to combine multiple models and carry out different operations. With workflows, you can create a powerful multimodal system that meets various use cases in a single API call — instead of relying only on one model. ### Wrapped Models Wrapped models are those hosted externally, but we deploy them on our platform using their third-party API keys. :::warning Clarifai-hosted models Clarifai-hosted models are the ones we host within our Clarifai Cloud. ::: ## Y ### YAML A markup language originally invented by Yahoo that is now commonly used as a format for configuration files. --- ## Generative AI Glossary # Generative AI Glossary **A Glossary of Generative AI Terms for Using the Clarifai Platform Effectively** ## A ### Adversarial Autoencoder (AAE) A type of autoencoder which combines the principles of adversarial loss, integral to GANs, and the architecture of an autoencoder. This combination empowers the model to learn complex distributions of data effectively. ### Audio Synthesis This involves using AI to create new, artificial sounds or voice outputs. Such sounds can be as simple as a specific tone or as complex as a mimicked form of speech. ### Autoregressive Models These are generative models that produce data by conditioning each element's probability on previous elements in a sequence. For example, WaveNet and PixelCNN are autoregressive models for creating music and images, respectively. ### Autoencoder An autoencoder is an artificial neural network utilized for learning efficient encodings of input data. It has two crucial components: an encoder that compresses the input data and a decoder that reconstructs the data from its reduced form. ### Autoregressive Generative Models These models predict the distribution of subsequent sequence elements using prior sequence elements to implicitly establish a distribution across sequences using Conditional Probability's Chain Rule. The main architectures for autoregressive models are causal convolutional networks and recurrent neural networks. ## B ### BERT (Bidirectional Encoder Representations from Transformers) BERT, developed by Google, is a pre-trained transformer-based language model. It stands out for its bidirectional training approach, which allows it to understand the context of a word based on all of its surroundings (left and right of the word). ### BLOOM Developed by The BLOOM project, Bloom is a large-scale language model that can execute a vast array of natural language understanding and generation tasks accurately. ## C ### ChatGPT Developed by OpenAI, ChatGPT is a specialized large-scale language model that generates human-like text. It's a popular choice for developing AI powered chatbots due to its convincing conversation-generation capabilities. ### CLIP (Contrastive Language—Imagen Pretraining) This involves using AI to create new, artificial sounds or voice outputs. Such sounds can be as simple as a specific tone or as complex as a mimicked form of speech. ### Close-Book QA Close-book QA, also known as zero-shot QA, refers to the ability of an LLM to answer questions without access to any additional information or context beyond its internal knowledge base. :::tip open-book QA Close-book QA stands in contrast to open-book QA, where the LLM can access and process external sources of information, such as documents, web pages, or knowledge bases. ::: ### Conditional GANs (cGANs) These are a type of GAN where a conditional variable is introduced to the input layer, allowing the model to generate data conditioned on certain factors. This augmentation provides the model with the capability to generate data with desired characteristics. ### Cross-modal Cross-modal learning refers to using information from one modality to understand or make predictions in another modality. This could involve translating or transforming the data in some way. For example, a cross-modal learning system might be designed to accept text input and output a related image or vice versa. ### CycleGAN A type of GAN that can translate an image from a source domain to a target domain without paired examples. It's particularly useful in tasks like photo enhancement, image colorization, and style transfer for unpaired photo-to-photo translation. ## D ### DALL-E 2 This is an updated version of DALL-E, an AI model developed by OpenAI to generate images from textual descriptions. It's an excellent example of a multi-modal AI system. ### Data Distribution In machine learning, data distribution refers to the overall layout or spread of data points within a dataset. In the case of generative models such as GANs, the generator seeks to mimic the actual data distribution. ### Deepfake Synthetic media in which a person in an existing image or video is replaced with someone else's likeness using machine learning techniques. While they could serve interactive entertainment purposes, deepfakes may mislead viewers, often with unintended consequences. ### Diffusion In AI, 'diffusion' refers to a technique used for generating new data by starting with a portion of actual data, then gradually adding random noise. This process is generally reversed, with a neural network trained to predict the reverse process of noise addition to the data. ### Discriminator In a GAN, the discriminator is the component that tries to differentiate real data instances from the fictitious ones fabricated by the generator. It helps refine the generator's ability to create realistic data. ## E ### Embedding An embedding represents data in a new form, often a vector space, facilitating comparisons and calculations with other data points. Similar items should have similar embeddings, making it an essential feature for many AI tasks, like recommendation systems and natural language processing. ### Emergence/Emergent Behavior ("sharp left turns," intelligence explosions). In artificial intelligence, emergence refers to complex phenomena that arise from simple rules or processes. Radical concepts like "sharp left turns" and "intelligence explosions" denote sudden, dramatic developments in AI, often related to AGI's emergence. ## F ### Few-Shot Learning A machine learning method where the model learns to perform a task from a few examples per class. For instance, it can correctly categorize new data after being shown only a few samples from each category. ### Fine-Tuning A form of transfer learning wherein a pre-trained model is slightly modified or adjusted to perform a new task. This process allows for more efficient use of the pre-trained models by adjusting them to solve tasks similar to the ones they were originally trained on. ### Foundation Model In AI, foundation models are large-scale AI models trained on diverse and extensive data meant to be fine-tuned or adapted for more specific tasks. These are called foundation models, as they offer a robust and broad foundation that can be built upon for various AI tasks. ## G ### Generative Models for Images These are generative models like GANs, VAEs, and DALL-E, trained on image data and capable of generating new images that reflect the patterns found in the training data. ### Generative Pre-Trained Transformer (GPT) GPT is a family of neural network models trained to generate content. These models are pre-trained on vast amounts of text data, allowing them to generate coherent and relevant text based on user prompts. GPT models can automate content creation and analyze customer feedback for insights, fostering personalized interactions. ### Generator In a Generative Adversarial Network, the generator is the component that creates new instances of data by learning to mimic the real data distribution. ### GPT-1, GPT-2, GPT-3, and GPT-4 Progressive versions of the generative pre-trained transformers developed by OpenAI. Each model sees improvements and expansions on its predecessors, offering advanced text generation capabilities and greater application versatility. GPT-3, for instance, is an extremely sophisticated model known for its wide-ranging applicability, including translation, question-answering, and text completion tasks. ### GPT-J GPT-J is an open-source large language model developed by EleutherAI in 2021. It is a generative pre-trained transformer model with 6 billion parameters, similar to GPT-3, but with some architectural differences. GPT-J was trained on a large-scale dataset called The Pile, a mixture of sources from different domains. ### GPT-Neo GPT-Neo is a family of transformer-based language models from EleutherAI based on the GPT architecture. It is an open-source alternative to GPT-3 that can generate natural language texts using deep learning. The GPT-Neo model comes in 125M, 1.3B, and 2.7B parameter variants. This allows users to choose the model size that best fits their specific use case and computational constraints. ### Grounding It is the process of linking a model's output to factual and verifiable information sources. This technique enhances the accuracy and reliability of the model, especially in applications where factual correctness is critical. Grounding reduces the risk of the model generating unfounded or incorrect content. ## H ### Hallucination In AI, a hallucination occurs when a model makes erroneous conclusions and generates content that doesn't correspond to reality. These erroneous outputs indicate problems in the workings of the AI model. Team vigilance is necessary to maintain the accuracy and reliability of AI systems in identifying and mitigating hallucinations. ## I ### Image Translation A task in computer vision where the goal is to map or translate one image into another, often using a model known as GANs. For example, translating a daytime scene into a nighttime scene. ### Inpainting A generative task where the AI is meant to fill in missing or corrupted parts of an image. Typical applications include photo restoration and the completion of unfinished art. ## L ### Langchain [Langchain](https://docs.clarifai.com/integrations/langchain/) is a concept in AI and machine learning that affects reasoning capability. When prompting an LLM, the "chain-of-thought" technique improves the model's reasoning by breaking tasks into smaller, discrete steps. A more complex approach, "tree-of-thought," allows logical steps to branch and backtrack. ### Large Language Models (LLMs) Large-scale AI models trained on extensive text data, such as GPT 3 and BERT. They can respond to prompts, generate text, answer questions, create poetry, and even generate code. This ability can enable personalized and authentic customer interactions and assist in automating customer-facing content. ### Latent Space In generative models, latent space refers to a compressed input data representation. It is the transition medium between the noise injected into the GAN’s generator and its output. ### Llama 2 Llama 2 is a collection of pre-trained and fine-tuned large language models (LLMs) created and publicly released by Meta AI. It is available in three model sizes: 7, 13, and 70 billion parameters. Llama 2-Chat is a fine-tuned version of Llama 2, specifically optimized for dialogue-based scenarios. ## M ### Machine Learning Bias Bias in machine learning can occur from intentionally or unintentionally biased data or algorithms making incorrect assumptions, leading to skewed decisions. Understanding and addressing this bias ensures fair and accurate treatment for all customers. ### Midjourney Midjourney is a text-to-image AI service developed by an independent research lab. It allows users to generate images based on textual descriptions, creating a wide range of art forms, from realistic to abstract styles and is especially known for its high-quality, well-structured, and detailed images. ### Mistral 7B Mistral 7B, introduced by Mistral AI, is an LLM that has gathered attention due to its efficiency and strong performance. It is a 7.3 billion-parameter model, making it smaller than other models like GPT-3 (175 billion) but still powerful for various tasks. Despite its size, Mistral 7B has shown impressive performance on various benchmarks, even surpassing some larger models in specific areas. ### Mixture of Experts It is a machine learning method where specialized models, or “experts”, handle different parts of data distribution. The final prediction is a blend of these expert outputs, adjusted by a “gating” system that determines each expert’s relevance. This leverages individual strengths to form a more robust model. ### Modalities Refer to the various types of data that a model can process and interpret. These include text, images, audio, video, and other forms of sensory data. Each modality represents a unique form of information, offering distinct insights and characteristics that can be utilized in AI applications. ### Mode Collapse This phrase refers to a situation when the Generator in a Generative Adversarial Network begins to produce the same output (or a narrow set of outputs) repetitively rather than generating diverse outputs. It destabilizes the learning process and poses a challenge in GAN training. ### Multi-Modal AI This type of AI has the capability to process and understand inputs from different data types, like text, speech, images, and videos. Thus, these AI models can deal with diverse data inputs, enhancing their applicability in various contexts. ### Multimodal A multimodal learning model makes predictions by accepting and analyzing various types of input, such as audio and video data, improving its understanding of scenarios like movie scenes. ## N ### NeRF (Neural Radiance Fields) A method for creating a three-dimensional scene from two-dimensional images using a neural network. NeRF can create a photorealistic rendering, synthesize views, and offer more capabilities in understanding and reconstructing scenes from 2D images. ## O ### Outpainting A generative task where the AI is asked to extend the existing content of an image. It fills the areas beyond the image boundaries with plausible content that seamlessly connects with the original image context. ## P ### Parameter Efficient Fine-Tuning (PEFT) Full parameter fine-tuning traditionally involves adjusting all parameters across all layers of a pre-trained model. While it typically yields optimal performance, it is resource-intensive and time-consuming, demanding significant GPU resources and time. On the other hand, PEFT offers a way to fine-tune models with minimal resources and costs. One notable PEFT method is Low-Rank Adaptation (LoRA). :::warning Lora LoRA is a game-changer for fine-tuning LLMs on resource-constrained devices or environments. It achieves this by exploiting inherent low-rank structures within the model's parameters. These structures capture essential patterns and relationships in the data, allowing LoRA to focus on these during fine-tuning, rather than modifying the entire parameter space. ::: ### Parameters Parameters are the fundamental elements that define the behavior and output of a model. They are akin to settings or dials that can be adjusted to control various aspects of the model's performance, such as its responsiveness, creativity, and accuracy. Parameters play a crucial role in fine-tuning the model to achieve desired results, whether it's generating text, images, or other forms of content. They are essential for optimizing the model to suit specific tasks or applications. ### Plugins / tools AI agents such as LLMs may have the ability to use 'tools' via APIs that give them new capabilities. For example, LLMs equipped with web search capabilities can access data not present in their training dataset, which can significantly reduce the risk of hallucinations. ### Pre-Encoded Knowledge QA This involves utilizing the model's built-in pre-encoded knowledge base to respond to questions. The model is provided with a large collection of facts and relationships, which it uses to generate answers when given prompts or questions. The pre-existing knowledge base equips the model with the ability to answer questions that demand a good understanding of the world. ### Prompt A prompt is the initial input or direction given to an AI model to execute a task or answer a query. It sets the starting context for the model's generation process. ### Prompt Tokens / Sampled Tokens / Completion These terms relate to how the AI uses tokens or units of data as input or output. A prompt token starts the model's data generation process, the model chooses sampled tokens during this process, and completion signifies the model's output following the prompt. ## Q ### Quantization Quantization is a model compression method that involves converting the weights and activations within an LLM from a high-precision data representation to a lower-precision one – without sacrificing significant accuracy. This means transitioning from a data type capable of holding more information, such as a 32-bit floating-point number (FP32), to one with less capacity, such as an 8-bit or 4-bit integer (INT8 or INT4). ## R ### RLHF (Reinforcement Learning from Human Feedback) This technique incorporates human feedback into the learning process of an AI model. Evaluators provide feedback on the model's outputs, which can help improve the model's performance over time. ## S ### Self-Supervised Learning This is a type of machine learning that trains an algorithm to analyze and infer from test data that hasn't been classified, labeled or categorized, as opposed to supervised learning that usually involves training data which has been labeled, classified and categorized. ### Sequence Generation This is a task in natural language processing where the model generates a sequence of words or symbols, such as in text generation. It's one of the capabilities of auto-regressive language models, including GPT and BERT. ### Style Transfer This generative model application involves capturing the artistic style of one image (the style source) and transferring it onto another image (the content source). ### StyleGAN Developed by NVIDIA, StyleGAN is a GANbased model known for its high-quality and consistent outputs. Particularly, it gained attention for its capability to generate hyperrealistic images of human faces. ### Super Resolution An application of generative models that involves increasing the resolution of an image. Using these models, lower-quality images can be enhanced successfully. ### Symbolic Artificial Intelligence A type of AI that leverages symbolic reasoning mechanisms to solve problems and represent knowledge. It typically involves the usage of symbols to represent concepts and the relationships between them. ### System Prompt This refers to the predefined instructions that set the general behavior of an AI system, like a chatbot. It begins every interaction and influences how the AI responds to user inputs. ## T ### Temperature It is a parameter that controls the randomness and creativity of a model's output. A higher temperature setting results in more varied and unpredictable responses, fostering creativity. Conversely, a lower temperature yields more deterministic and predictable outputs, enhancing coherence and reliability. This parameter is essential for fine-tuning the balance between novelty and accuracy in generated content. ### Token In the context of neural networks, tokenization is the process of encoding text into numerical values. Tokens may represent letters, groups of letters, or whole words. ### Translation Translation refers to the process of automatically converting text or speech from one language (source language) to another language (target language), preserving the original meaning as closely as possible. ## V ### Variational Autoencoder (VAE) Unlike traditional autoencoders, which can learn any function to reconstruct data, a VAE places additional constraints on encoded representations, so they learn parameters of a probability distribution representing the data. ## W ### Weakly Supervised Learning This is an approach to supervised learning in which the training data is noisy, limited, or, imprecise; however, these weakly labeled samples are often easier and cheaper to obtain, resulting in larger effective training sets. ## Z ### Zero-Shot Learning A type of machine learning where the model can make predictions about data it has never encountered during its training. It leverages similarities between what it has seen and the novel data to make predictions. --- ## Clarifai Platform Overview # Clarifai Platform Overview **Your full-stack platform for the entire AI lifecycle** The Clarifai's full-stack portal is the fastest and easiest AI workspace for any skill level. It has everything you need to build reliable AI-powered apps. With the low-code, no-code solution, you can discover, build, and share futuristic AI models, workflows, and app components. It gives you the power to search, sort, and organize your AI projects right out of the box. Its intuitive, feature-rich, drag-and-drop user interface makes the world’s most advanced computer vision AI technology accessible to anyone. :::info Anything that you can do on the platform, you can also do directly with our API. If you want to build powerful UIs, you can do so with our [API](https://docs.clarifai.com/api-guide/api-overview/). ::: Our platform offers powerful tools for the full AI lifecycle. It provides a seamless, end-to-end AI workspace that you can use to manage your data, annotate your inputs, and build custom AI models. ![](/img/community_2/general_image_recognition_model.png) The Clarifai Community platform provides the following: - Access to a massive collection of user-contributed state-of-the-art models, workflows, datasets, and custom UI for a variety of tasks. This lets you avoid reinventing the wheel by ensuring you have world-class AI in one place. You can easily customize your needs on the same platform. - Ability to manage your AI resources within your organization and the general public audience with notes, metadata, and model cards. - Ability to view predictions of any model or workflow as you browse to test your data. - Ability to easily filter and sort resources by name, type, license, data, and more as an extensible model, data, workflow, and UI catalog. - Ability to collaborate with AI experts from all over the world. You'll get the right tools to collaborate in the public community and within your organization. This lets you accelerate your AI adoption efforts. ## Search ![](/img/community_2/search_community_resources.png) You can effortlessly find what you need by searching through your own and Community resources — apps, models, workflows, and modules. This is how we treat searches: - Multiple-word queries function as an "OR" (e.g., "face detection" yields results with both "face" and "detection"). - Dashes (-) are treated as spaces since spaces aren't allowed in resource IDs. - Dashes are added to search queries between texts and numbers. For instance, if the search query is "llama70b" or "gpt4," we also consider "llama-70-b" or "gpt-4" in the search results. - Partial-word searches, such as "fac," return results containing relevant terms like "face-detection." All resources are searchable by: - **Resource ID**—you can provide a unique identifier of a particular resource. - **User ID**—you can provide a unique identifier assigned to an individual user. - **Short description**—you can provide a brief summary that provides key information about what you're looking for. - **Markdown notes**—you can perform text-based searches within the markdown notes of any resource. If you input any text in the search field, the results will include any relevant information where the search phrase is found within the markdown notes. This feature enhances search functionality and allows for more precise and context-aware discovery of resources based on the content within their markdown notes. ## Starring You can mark any resource — be it your own or publicly accessible in the Community — as a favorite. Simply click the star icon to designate any resource, including apps, models, datasets, workflows, and modules, as your favorite. You can access your favorite resources by navigating to the top-right menu bar and selecting the "Starred" option. If you choose the "All" option, you'll get a list encompassing both starred and unstarred resources. You can also organize your starred resources by sorting them in either descending order (highest number of stars first) or ascending order (lowest number of stars first) according to your preference. ![](/img/community_2/starring_resources.png) ## Markdown Notes You can add notes in Markdown format that explain what a resource is about, such as its purpose, usage instructions, required configurations, or any other relevant details to provide clarity and guidance for users interacting with it. For example, to add descriptive notes for an app, navigate to the app's **Overview** page. From there, click the pencil icon that appears within the **Notes** section to begin adding your notes. ![](/img/community_2/markdown_notes.png) You can then add your notes in Markdown format. Note that you can insert custom variables such as `{{user_id}}` and `{{app_id}}` into the Markdown template. These variables are dynamically replaced with the corresponding `user_id` and `app_id` extracted from the URL, allowing you to personalize content within your templates. Here is an example: ![](/img/community_2/markdown_notes_1.png) If you click the **Save** button, the corresponding `user_id` and `app_id` will be dynamically displayed. ![](/img/community_2/markdown_notes_3.png) ## Use Model / Workflow If you want to use a model or a workflow for making API calls, you need to click the **Use Model / Workflow** button at the upper right corner of the individual page of a model or workflow. The modal that pops up has snippets in various programming languages, which you can copy and use in your development environment. ![](/img/community_2/use_model_workflow.png) --- ## Data Privacy and Security # Data Privacy and Security **Learn how we secure and handle your data** At Clarifai, we uphold the highest standards of data privacy and security, backed by our SOC 2 Type 2 compliance. Your data is always treated as private and confidential by default. :::tip Trust Center Visit our [Trust Center](https://trust.clarifai.com/) to view our compliance documentation, including certifications, attestations, and audit reports aligned with global standards. ::: ## Default Data Handling and User Control Clarifai is designed to provide you with a rich, interactive platform where you can access your inputs and view prediction history. To enable this, we securely store the data you send to our platform. However, you remain in complete control of its retention. Here’s how we handle your data: - **Secure Storage by Default:** When you send inputs (such as images, text, or video) to your application, the data and its resulting predictions are stored securely. This allows you to review, search, and manage your data directly within the Clarifai Portal. - **Your Data is Not Used for Training:** We do not use your private data to train our models or any other models on the platform unless you explicitly choose to share your inputs and annotations with the "Community." ## Metadata Usage and Analytics Practices To deliver and improve our service, we log non-sensitive metadata for operational purposes. We handle the metadata for: - **Operational Logging:** We log metadata required for billing, platform monitoring, and providing the service, such as the number of requests or compute usage. - **Platform Improvement:** For our proprietary models, we may analyze usage patterns and performance in aggregate to enhance model accuracy and efficiency. This analysis does not use your private input data. Any features that might involve more detailed logging will require your explicit opt-in. --- ## Python SDK Notebook Examples # Python SDK Notebook Examples **Learn how to use the Clarifai Python SDK** Here are comprehensive step-by-step walkthroughs within Jupyter or Colab notebooks that showcase how to harness the power of the Clarifai SDKs. |Notebook | Description | Open in Colab | |----------|---------------------------|-------------------------------| | [Basics](https://github.com/Clarifai/examples/blob/main/basics/basics.ipynb) | Create, manage, update, and delete Clarifai resources, including apps, datasets, inputs, and models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/basics/basics.ipynb) | | [CLI](https://github.com/Clarifai/examples/blob/main/CLI/compute_orchestration.ipynb) | Clarifai provides a user-friendly command line interface (CLI) that simplifies various tasks |[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)]( https://colab.research.google.com/github/Clarifai/examples/blob/main/CLI/compute_orchestration.ipynb) | | [Compute Orchestration](https://github.com/Clarifai/examples/blob/main/ComputeOrchestration/crud_operations.ipynb) |Use our Compute Orchestration system to create, get, list, and delete compute clusters, nodepools, and model deployments.|[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/ComputeOrchestration/crud_operations.ipynb) | | [Data Utils](https://github.com/Clarifai/examples/tree/main/Data_Utils) | Get a range of multimedia data utilities designed to streamline your data management and processing operations. [Image annotation loader](https://github.com/Clarifai/examples/blob/main/Data_Utils/Image%20Annotation/image_annotation_loader.ipynb) — A comprehensive framework for loading, exporting, and analyzing different annotated datasets. [Data ingestion pipelines](https://github.com/Clarifai/examples/blob/main/Data_Utils/Ingestion%20pipelines/README.md) — Load text files (PDF, DOC, etc.), transform and chunk the content, and upload it to the Clarifai platform. [Multimodal dataloader](https://github.com/Clarifai/examples/blob/main/Data_Utils/Ingestion%20pipelines/Multimodal_dataloader.ipynb) [Ready-to-use foundational pipelines](https://github.com/Clarifai/examples/blob/main/Data_Utils/Ingestion%20pipelines/Ready_to_use_foundational_pipelines.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Data_Utils/Image%20Annotation/image_annotation_loader.ipynb) | |[RAG](https://github.com/Clarifai/examples/blob/main/RAG/RAG.ipynb)| Use Retrieval Augmented Generation (RAG) to improve Large Language Models (LLMs) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/RAG/RAG.ipynb) | | [Concept management](https://github.com/Clarifai/examples/blob/main/concepts/concept_management.ipynb) | Establish a hierarchical relationship between concepts using concept relations | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/concepts/concept_management.ipynb) | | [Datasets basics](https://github.com/Clarifai/examples/blob/main/datasets/basics.ipynb) | Merge datasets and list inputs of a dataset |[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/datasets/basics.ipynb) | | [Dataset export](https://github.com/Clarifai/examples/blob/main/datasets/export/dataset_export.ipynb) | Export datasets from a Clarifai app | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)]( https://colab.research.google.com/github/Clarifai/examples/blob/main/datasets/export/dataset_export.ipynb) | | [Dataset upload](https://github.com/Clarifai/examples/blob/main/datasets/upload/dataset_upload.ipynb) | Upload datasets into a Clarifai app | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/datasets/upload/dataset_upload.ipynb) | | [Inputs upload](https://github.com/Clarifai/examples/blob/main/datasets/upload/input_upload.ipynb) | Upload inputs with various types of data, such as metadata, geo info, or bounding box annotations, into a Clarifai app |[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/datasets/upload/input_upload.ipynb) | | [Models predict (inference)](https://github.com/Clarifai/examples/tree/main/models/model_predict) | Get predictions with text, image, video, and audio inputs with different types of models | | | [Evaluation for embedding classification](https://github.com/Clarifai/examples/blob/main/models/model_eval/embedding_classifier_eval.ipynb) | Evaluate the performance of embedding classifier models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_eval/embedding_classifier_eval.ipynb) | |[Evaluation for text classification](https://github.com/Clarifai/examples/blob/main/models/model_eval/text_classification_eval.ipynb)| Evaluate the performance of text classifier models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_eval/text_classification_eval.ipynb) | |[Evaluation for visual classification](https://github.com/Clarifai/examples/blob/main/models/model_eval/visual_classifier_eval.ipynb) | Evaluate the performance of visual classifier models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_eval/visual_classifier_eval.ipynb) | | [Evaluation for visual detection](https://github.com/Clarifai/examples/blob/main/models/model_eval/visual_detector_eval.ipynb) | Evaluate the performance of visual detector models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_eval/visual_detector_eval.ipynb) | | [Training for image classification ](https://github.com/Clarifai/examples/blob/main/models/model_train/image-classification_training.ipynb) | Train image classifier models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_train/image-classification_training.ipynb) | | [Training for image detection](https://github.com/Clarifai/examples/blob/main/models/model_train/image-detection_training.ipynb) | Train image detector models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_train/image-detection_training.ipynb) | | [Training for image segmentation](https://github.com/Clarifai/examples/blob/main/models/model_train/image-segmentation_training.ipynb) | Train image segmentation models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_train/image-segmentation_training.ipynb) | | [Training for text classification](https://github.com/Clarifai/examples/blob/main/models/model_train/text-classification_training.ipynb) | Train text classifier models |[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_train/text-classification_training.ipynb) | | [Training for transfer learn](https://github.com/Clarifai/examples/blob/main/models/model_train/transfer_learn.ipynb) | Train transfer learn models | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_train/transfer_learn.ipynb) | | [Model upload](https://github.com/Clarifai/examples/tree/main/models/model_upload) | [Upload image classifiers](https://github.com/Clarifai/examples/tree/main/models/model_upload/image-classifier/nsfw-image-classifier) [Upload image detectors](https://github.com/Clarifai/examples/tree/main/models/model_upload/image-detector/detr-resnet-image-detection) [Upload LLMs](https://github.com/Clarifai/examples/tree/main/models/model_upload/llms) [Upload speech recognition models](https://github.com/Clarifai/examples/tree/main/models/model_upload/speech-recognition/openai-whisper) | | | [Cross-modal search](https://github.com/Clarifai/examples/blob/main/search/cross_modal_search.ipynb) | Perform vector search over your own data | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/search/cross_modal_search.ipynb) | | [Create workflows](https://github.com/Clarifai/examples/blob/main/workflows/create_workflow.ipynb) | Create various types of workflows | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/workflows/create_workflow.ipynb) | | [Export workflows](https://github.com/Clarifai/examples/blob/main/workflows/export_workflow.ipynb) | Download a YAML file representing your workflow | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/workflows/export_workflow.ipynb) | |[Patch workflows](https://github.com/Clarifai/examples/blob/main/workflows/patch_workflow.ipynb) | Perform patch operations by merging, removing, or overwriting data | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)]( https://colab.research.google.com/github/Clarifai/examples/blob/main/workflows/patch_workflow.ipynb) | ## Integration Examples |Notebook | Description | Open in Colab | |----------|---------------------------|-------------------------------| |[LangChain](https://github.com/Clarifai/examples/tree/main/Integrations/Langchain)| | | | | [Agent: Doc-retrieval using ReAct Docstore](https://github.com/Clarifai/examples/blob/main/Integrations/Langchain/Agents/Doc-retrieve_using_Langchain-ReAct_Agent.ipynb) |[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Langchain/Agents/Doc-retrieve_using_Langchain-ReAct_Agent.ipynb) | | | [Agent: Retrieval QA using Clarifai vector store with conversation memory](https://github.com/Clarifai/examples/blob/main/Integrations/Langchain/Agents/Retrieval_QA_with_Conversation_memory.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Langchain/Agents/Retrieval_QA_with_Conversation_memory.ipynb) | | | [Chains: Using PostgreSQL database with LangChain](https://github.com/Clarifai/examples/blob/main/Integrations/Langchain/Chains/PostgreSQL_LLM.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Langchain/Chains/PostgreSQL_LLM.ipynb) | | | [Chains: Prompt templates and chains](https://github.com/Clarifai/examples/blob/main/Integrations/Langchain/Chains/Prompt-templates_and_chains.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Langchain/Chains/Prompt-templates_and_chains.ipynb) | | | [Chains: Retrieval QA Chain with Clarifai vector store](https://github.com/Clarifai/examples/blob/main/Integrations/Langchain/Chains/Retrieval_QA_chain_with_Clarifai_Vectorstore.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Langchain/Chains/Retrieval_QA_chain_with_Clarifai_Vectorstore.ipynb) | | | [Chains: Router Chain with Clarifai SDK prompt templates](https://github.com/Clarifai/examples/blob/main/Integrations/Langchain/Chains/Router_chain_examples_with_Clarifai_SDK.ipynb) |[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Langchain/Chains/Router_chain_examples_with_Clarifai_SDK.ipynb) | | | | | | [Unstructured.io](https://github.com/Clarifai/examples/blob/main/Integrations/Unstructured/readme.md) | | | | | [S3 data ingestion](https://github.com/Clarifai/examples/blob/main/Integrations/Unstructured/Clarifai_Unstructured_integration_demo.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Unstructured/Clarifai_Unstructured_integration_demo.ipynb) | | | [GitHub data ingestion](https://github.com/Clarifai/examples/blob/main/Integrations/Unstructured/Clarifai_github_using_unstructured_io_integration_example.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Unstructured/Clarifai_github_using_unstructured_io_integration_example.ipynb) | | | [DropBox data ingestion](https://github.com/Clarifai/examples/blob/main/Integrations/Unstructured/Dropbox_Clarifai_Unstructured_integration_example.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Clarifai/examples/blob/main/Integrations/Unstructured/Dropbox_Clarifai_Unstructured_integration_example.ipynb) | --- ## Additional Tips # Additional Tips **Enhance your implementation with Clarifai** This page provides extra guidance to help you get the most out of your development experience with Clarifai. Whether you're optimizing performance or just looking for best practices, these tips offer practical advice beyond the basics. import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from "@theme/CodeBlock"; import WebhookIntegration from "!!raw-loader!../../code_snippets/api-guide/others/webhook_integration.py"; import CachingPredictions from "!!raw-loader!../../code_snippets/api-guide/others/caching_predictions.py"; ## Webhook Integration You can enable real-time automation by integrating Clarifai webhooks into your backend system. For example, you can set up a secure Flask endpoint to receive and validate webhook notifications for events like model training completion or new input uploads. {WebhookIntegration} ## Caching Predictions You can avoid repeated computation for identical inputs by caching model predictions locally. For example, you can introduce a simple decorator-based caching mechanism to store and retrieve results from disk, which optimizes performance for resource-intensive prediction tasks. {CachingPredictions} --- ## Troubleshooting # Troubleshooting **Solve common issues quickly and effectively** Troubleshooting helps you identify and fix the most common issues encountered when working with Clarifai-based scripts and applications. Whether you're facing installation errors, API authentication problems, or image processing glitches, this guide offers step-by-step solutions to get you back on track. ## Common Installation Issues Setups often break due to missing or incorrectly installed packages. These errors are usually easy to resolve once you've verified your environment and dependencies. ### Unsupported Environment **Error**: You may encounter install-time issues (pip refusal, syntax errors, or compilation failures), run-time issues (syntax errors, missing modules, or incompatible APIs), or architecture incompatibilities. These errors indicate that the Clarifai Python SDK is running in an unsupported environment. Refer to the [system requirements guide](https://docs.clarifai.com/resources/api-overview/python-sdk#minimum-system-requirements) to ensure your setup meets the minimum requirements for successful installation and use of the Clarifai package. #### Missing Dependencies **Error**: `Import could not be resolved` or `ModuleNotFoundError` This error means that Python can't find the required module — likely because it wasn't installed or was installed in the wrong environment. Try the following solutions: ```bash # 1. Make sure you installed the required package pip install # 2. Check if packages are installed pip list | grep -E "(clarifai|openai|opencv|numpy)" # 3. If using conda, activate your environment conda activate your_environment_name pip install # 4. Upgrade pip, then try again pip install --upgrade pip pip install ``` #### Package Not Installed Properly **Error**: `ModuleNotFoundError: No module named 'cv2'` This usually means OpenCV isn't installed or is incompatible with your current setup. Try the following in order: ```bash pip install opencv-python # or pip install opencv-contrib-python # or (for conda users) conda install -c conda-forge opencv ``` #### Dependency Conflicts **Error**: Version conflicts occur when different projects require different versions of the same library. This often results in import errors, broken functionality, or one project overwriting another project’s dependencies. You can solve it by using a virtual environment. A virtual environment isolates your project’s dependencies, protecting you from dependency conflicts, version mismatches, and system corruption. Here is how to create a virtual environment: ```bash # Create a new virtual environment python3 -m venv venv # Activate the environment (Linux/macOS) source venv/bin/activate # Activate the environment (Windows PowerShell) venv\Scripts\activate ``` Once activated, any packages you install with `pip` will be isolated to that environment. ## API and Authentication Issues API errors often stem from incorrect or missing credentials. Make sure your [Personal Access Token](https://docs.clarifai.com/control/authentication/pat) (PAT) is properly set and valid. ### Unconfigured PAT Key **Error**: `Please set the CLARIFAI_PAT environment variable` This happens when the Clarifai PAT is not found in your environment. Set it based on your platform: ```bash # Linux / macOS export CLARIFAI_PAT="your_actual_api_key_here" # Windows Command Prompt set CLARIFAI_PAT=your_actual_api_key_here # Windows PowerShell $env:CLARIFAI_PAT="your_actual_api_key_here" # (Optional) Make it permanent echo 'export CLARIFAI_PAT="your_key_here"' >> ~/.bashrc source ~/.bashrc ``` ### Incorrect PAT **Error**: `Authentication failed` or `Invalid token` These indicate that the PAT was not set correctly or has expired. To fix: 1. Generate a new PAT from your personal settings page by navigating to the **Security** section 2. Ensure the key is copied properly 3. Check for extra spaces or characters when setting the environment variable 4. Confirm your account is active and has API access ## Runtime Errors Runtime errors occur while executing your code and often result from unexpected data or an unhandled response. ### Index Error **Error**: `IndexError: list index out of range` This typically means the code tried to access an element in a list that doesn't exist — often due to an empty or malformed API response. Try the following: 1. Ensure the input (e.g., an image URL) is valid and accessible 2. Try a different input or a simpler model 3. Confirm your internet connection is stable 4. Log the full API response to investigate ### Network Errors **Error**: `Network error` or `Connection timeout` These errors suggest that your script couldn't reach the Clarifai servers. Possible solutions: 1. Check your internet connection 2. Retry after a few minutes — there might be temporary downtime 3. Ensure your firewall or proxy isn’t blocking traffic 4. Check [Clarifai’s status page](https://status.clarifai.com) ## Image Processing Issues Working with images can introduce additional complications — from data corruption to file permission errors. ### Corrupted or Unreadable Images **Error**: Generated images are corrupted or won't open This usually happens if the image data wasn’t fully received or saved correctly. Use error checking like this: ```python if img_np is not None and img_np.size > 0: cv2.imwrite("output.jpg", img_np) print("✅ Image saved successfully") else: print("❌ Failed to process image data") ``` ### Permission Denied **Error**: Permission denied when saving files Your script may not have write permissions in the current directory. Steps to resolve: ```bash # Check directory permissions ls -la # Navigate to a writable directory cd ~/Documents # Run the script again python /path/to/your/script.py ``` ## Performance Issues Slow performance or memory-related crashes can often be addressed through optimization and lighter alternatives. ### Scripts Running Slowly Common reasons and optimizations: 1. **Large input files:** Resize images before processing 2. **Slow internet:** Use local files when possible 3. **Complex models:** Choose simpler models when speed is a priority 4. **Server load:** Run scripts during off-peak hours ### Out of Memory Errors These usually occur when handling high-resolution images or large batches. Tips to reduce memory usage: ```bash # Use a lightweight version of OpenCV pip install opencv-python-headless # Or resize images in your code before processing ``` ## Additional Debugging Tips Debugging effectively helps you isolate and resolve issues faster. Use these tips to gain visibility into what your code is doing. ### Enable Verbose Output Add print statements to verify assumptions at runtime: ```python print(f"PAT loaded: {'Yes' if pat else 'No'}") print(f"Image URL: {image_url}") print(f"Model prediction type: {type(model_prediction)}") ``` ### Check API Responses Always validate the API response before processing it: ```python if model_prediction.outputs and len(model_prediction.outputs) > 0: # Process results pass else: print("No outputs received from the model") ``` ## Getting Help If you're still facing issues: 1. **Read the error message carefully** – it often points to the exact problem 2. **Review the full traceback** – it tells you where the failure occurred 3. **Test with minimal examples** – start simple and build up 4. **Check Clarifai’s status page** – [status.clarifai.com](https://status.clarifai.com) 5. **Consult the documentation, especially on interpreting error codes** – [docs.clarifai.com](https://docs.clarifai.com/resources/api-overview/status-codes)