Skip to main content

Inference from Workflows

Learn how to perform predictions with workflows using Clarifai SDKs


Experience streamlined and powerful inference capabilities through Clarifai SDKs when working with workflows. Unleash the full potential of your models by seamlessly integrating them into your workflows to obtain insightful predictions and enhance your data processing pipelines.By effortlessly integrating your models into workflows, you gain access to a streamlined process that delivers insightful predictions with minimal friction. Clarifai SDKs ensures a smooth experience, allowing you to effortlessly embed models into your applications. This not only enhances the predictive capabilities of your models but also optimizes the overall efficiency of your data processing pipelines.

The below inference examples are reference implementation that can be extended or reused for respective task based on model types.

Text Classifier

The Predict API in the Clarifai SDKs empowers users to seamlessly perform text classification through a pre-built workflow tailored specifically for classification tasks. Use the capabilities of the Predict API to effortlessly categorize and analyze text, leveraging a robust and efficient classification model for enhanced accuracy and ease of use.

from clarifai.client.workflow import Workflow

# Your PAT (Personal Access Token) can be found in the Account's Security section

# Initialize the workflow_url
workflow_url = "https://clarifai.com/clarifai/main/workflows/Text-Moderation"
text_classfication_workflow = Workflow(
url= workflow_url , pat="YOUR_PAT"
)
raw_text = b"I love your product very much"
result = text_classfication_workflow.predict_by_bytes(raw_text, input_type="text")
print(result.results[0].outputs[0].data)
Output
concepts {

id: "ai_pkltpBZ7"

name: "toxic"

value: 0.996599138

app_id: "main"

}

concepts {

id: "ai_PlX1h51x"

name: "insult"

value: 0.515220046

app_id: "main"

}

concepts {

id: "ai_jf9vs404"

name: "obscene"

value: 0.0849464387

app_id: "main"

}

concepts {

id: "ai_2Dv0D4Nm"

name: "identity_hate"

value: 0.0308352802

app_id: "main"

}

concepts {

...

value: 0.00677669281

app_id: "main"

}

Text to Text - Summarization

Through the Predict API, you can seamlessly integrate a text summarization model into your workflow, enabling efficient extraction of key insights from extensive textual content. Streamline the summarization process and enhance the effectiveness of your applications with the powerful capabilities of Clarifai's Text Summarization model.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/Text/workflows/summary"
# creating the workflow
text_summarization_workflow = Workflow(
url= workflow_url, pat="YOUR_PAT"
)
# initialize the input text (minimum 200 characters)
text = b"""The word pollution was derived from the Latin word pollution, which means to make dirty. Pollution is the process of making the environment pollute the water and the air by adding harmful substances. Pollution causes an imbalance in the environment. This imbalance threatened the very survival of all forms of life. It's a threat to the whole world. India ranks at the bottom of 125 of the 132 countries in the 2012 Environmental Performance Index. This report is produced by researchers at Yale and Columbia University in association with the World Economic Forum.

Pollution of the environment is a serious problem of industrialized societies. Industrial development and the green revolution have had a negative impact on the environment. People have converted the life support system of all living people into their own resources and have greatly disrupted the natural ecological balance. Serious degradation and depletion have been caused due to overuse, abuse and mismanagement of resources to meet human greed."""


result = text_summarization_workflow.predict_by_bytes(text, input_type="text")
print(result.results[0].outputs[0].data.text.raw)
Output
- Pollution, taken from the Latin word meaning "to make dirty", is the act of using harmful substances to dirty the air and water around us, to the point where it threatens all life forms

- India ranks terribly low on the Environmental Performance Index, an evaluation of countries' environmental statuses, produced by researchers at Yale and Columbia University in collaboration with the World Economic Forum

- While pollution is a problem for all societies, industrialized societies are particularly afflicted due to the further damage industrial development has caused to the environment

- However, the worst impact of pollution may be yet to come, as warns the author, and the best course of action going forward needs to be evaluated.

Would you like help with anything else regarding this summary?

Text to Text - Generation

With our Predict API, you can seamlessly generate textual content by providing a prompt as input to a workflow that incorporates a powerful text generation model. This feature allows you to use the capabilities of state-of-the-art language models, enabling you to dynamically generate text for a variety of applications, from creative writing to automated content creation.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/Text/workflows/text-generation"
# Creating the workflow
text_generation_workflow = Workflow(
url=workflow_url,
pat="YOUR_PAT",
)

# getting the prediction
result = text_generation_workflow.predict_by_bytes(
b"Good Morning! I think it's going to be a great day!", input_type="text"
)
print(result.results[0].outputs[0].data.text.raw)
Output
Good morning! I'm glad to hear you're starting the day with such a positive attitude. How can I assist you today to make it even better?

Visual Classifier

Effortlessly conduct image classification through the Predict API by seamlessly integrating with a workflow designed around powerful classification models. This streamlined process empowers users to harness the capabilities of classification models, enabling accurate and efficient categorization of images.

Visit this link to view the available models for this use case.

from clarifai.client.workflow import Workflow

image_url = "https://cdn-bfgco.nitrocdn.com/PLGJnButkKeCQigeKyiwHwnQLZJDOQZI/assets/images/optimized/rev-96fa12a/delamere.com/wp-content/uploads/2022/02/Picture1-1024x683.jpg"
workflow_url = "https://clarifai.com/clarifai/main/workflows/Moderation"
# Creating the workflow
image_classifcation_workflow = Workflow(
url=workflow_url, pat="YOUR_PAT"
)

# Getting the predictions
result = image_classifcation_workflow.predict_by_url(image_url ,
input_type="image",
)
print(result.results[0].outputs[0].data)
Output
concepts {

id: "ai_8QQwMjQR"

name: "drug"

value: 0.99999994

app_id: "main"

}

concepts {

id: "ai_QD1zClSd"

name: "safe"

value: 4.93858607e-08

app_id: "main"

}

concepts {

id: "ai_V76bvrtj"

name: "explicit"

value: 3.21394289e-09

app_id: "main"

}

concepts {

id: "ai_kBBGf7r8"

name: "gore"

value: 1.82284843e-09

app_id: "main"

}

concepts {

...

value: 8.18517032e-10

app_id: "main"

}

Visual Segmenter

By leveraging the Predict API, you can seamlessly integrate image segmentation models into your workflow. This powerful functionality allows you to precisely categorize different regions within an image, unlocking enhanced capabilities for object identification, analysis, and understanding.

from clarifai.client.workflow import Workflow
import cv2


image_url = "https://s3.amazonaws.com/samples.clarifai.com/featured-models/general-elephants.jpg"
workflow_url = "https://clarifai.com/clarifai/main/workflows/Visual-Segmenter"
# Creating the workflow
image_segmentation_workflow = Workflow(
url=workflow_url, pat="YOUR_PAT"
)
# Getting the predictions
result = image_segmentation_workflow.predict_by_url(
image_url ,
input_type="image",
)
im_b = result.results[0].outputs[0].data.regions[0].region_info.mask.image.base64
image_np = np.frombuffer(im_b, np.uint8)
img_np = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
# Display the image
plt.axis("off")
plt.imshow(img_np[..., ::-1])
Image Output

Image to Text

By leveraging the Predict API, you can effortlessly integrate OCR capabilities into your applications through a streamlined workflow that involves an OCR model. This functionality allows you to extract and interpret text from images, empowering your applications to recognize and process textual information efficiently.

from clarifai.client.workflow import Workflow
from clarifai.client.user import User

# Your PAT (Personal Access Token) can be found in the Account's Security section
# Specify the correct user_id/app_id pairings
# Since you're making inferences outside your app's scope
USER_ID = "user_id"
APP_ID = "app_id"

app = User(user_id=USER_ID, pat="YOUR_PAT").create_app(
app_id=APP_ID, base_workflow="Empty"
)
image_url = "https://s3.amazonaws.com/samples.clarifai.com/ocr_img_00417302.jpg"
# create a yaml file specifying the workflow structure
# eg:
"""language_aware_ocr.yml
workflow:
id: wf-ocr
nodes:
- id: ocr-workflow
model:
model_id: language-aware-multilingual-ocr-multiplex

- id: text-aggregator
model:
model_id: text-aggregation
model_type_id: text-aggregation-operator
output_info:
params:
avg_word_width_window_factor: 2.0
avg_word_height_window_factor: 1.0

node_inputs:
- node_id: ocr-workflow

- id: language-id-operator
model:
model_id: language-id
model_type_id: language-id-operator
output_info:
params:
library: "fasttext"
topk: 1
threshold: 0.1
lowercase: true

node_inputs:
- node_id: text-aggregator


"""


ocr_workflow = app.create_workflow(config_filepath="configs/language_aware_ocr.yml")
prediction = ocr_workflow.predict_by_url(
image_url ,
input_type="image",
)

# Get workflow results
print(prediction.results[0].outputs[-1].data)
Output
2024-01-11 09:09:54 INFO     clarifai:  wf-ocr Workflow is still deploying, please wait...           workflow.py:89

INFO:clarifai:wf-ocr Workflow is still deploying, please wait...

2024-01-11 09:10:57 INFO clarifai: wf-ocr Workflow is still deploying, please wait... workflow.py:89

INFO:clarifai:wf-ocr Workflow is still deploying, please wait...

text {

raw: "H WerP\nRADOAS TESTXIT\nTRRTESTINCEERINCMODEB\nEPAGSTED\nONETIMEIISE\nDITED1YEARWPRRNTY"

text_info {

encoding: "UnknownTextEnc"

}

}

Text to Image

Create captivating visuals effortlessly with the help of Predict API. The Image Generation feature empowers developers to seamlessly generate images based on textual prompts through a streamlined workflow utilizing the powerful text-to-image model.

from clarifai.client.workflow import Workflow
import cv2


workflow_url ="https://clarifai.com/clarifai/Visual/workflows/image-generation"
# Creating the workflow
image_generation_workflow = Workflow(
url=workflow_url,
pat="YOUR_PAT",
)

# Getting the predictions
result = image_generation_workflow.predict_by_bytes(
b"cat with a hat", input_type="text"
)
# Extract the base64-encoded image data from the result object
im_b = result.results[0].outputs[0].data.image.base64
# Convert the base64-encoded image data to a NumPy array of uint8 values
image_np = np.frombuffer(im_b, np.uint8)
# Decode the NumPy array to obtain the image in OpenCV format (BGR color)
img_np = cv2.imdecode(image_np, cv2.IMREAD_COLOR)

# Display the image
plt.axis("off")
plt.imshow(img_np[..., ::-1])
Image Output

Image to Image

By leveraging our Predict API, you can seamlessly enhance the quality of any image through a streamlined workflow featuring our state-of-the-art upscaling model. Empower your applications with higher-resolution visuals, delivering a superior visual experience to your users.

from clarifai.client.workflow import Workflow
import cv2



workflow_url = "https://clarifai.com/clarifai/Visual/workflows/upscale-workflow"
image_url = "https://s3.amazonaws.com/samples.clarifai.com/featured-models/image-captioning-statue-of-liberty.jpeg"

# Creating the workflow
image_upscale_workflow = Workflow(
url=workflow_url,
pat="YOUR_PAT",
)

# Getting the predictions
result = image_upscale_workflow.predict_by_url(
image_url ,
input_type="image",
)

im_b = result.results[0].outputs[0].data.image.base64
image_np = np.frombuffer(im_b, np.uint8)
img_np = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
# Display the image
plt.axis("off")
plt.imshow(img_np[..., ::-1])
Image Output

Text to Audio

Leverage the power of the Predict API to seamlessly transform text into audio. This process involves submitting the text as input to a dedicated workflow that incorporates a state-of-the-art text-to-speech model. By integrating this feature, you can effortlessly generate high-quality audio outputs, expanding the scope of your applications and enhancing the overall user experience.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/Text/workflows/text-to-audio"
# Creating thr workflow
text_to_audio_workflow = Workflow(
url=workflow_url,
pat="YOUR_PAT",
)

# Getting the predictions
result = text_to_audio_workflow.predict_by_bytes(
b"I love your product very much", input_type="text"
)

# Save the audio file
with open("output_audio.wav", mode="bx") as f:
f.write(result.results[0].outputs[0].data.audio.base64)

Audio to Text

With the Clarifai Predict API, seamlessly convert audio to text by incorporating it as input to a dedicated speech-to-text model within a workflow. Effortlessly use the capabilities of Clarifai's advanced technology to transcribe audio content, enabling a wide range of applications such as transcription services, voice command recognition, and more.

from clarifai.client.user import User


# Create a new app (replace USER_ID and APP_ID with your own)
app = User(user_id="user_id", pat="YOUR_PAT").create_app(
app_id="APP_ID", base_workflow="Empty"
)
# Create ASR-Sentiment workflow
asr_sentiment_workflow = app.create_workflow(
config_filepath="configs/asr_sentiment.yml"
)

audio_url = "https://s3.amazonaws.com/samples.clarifai.com/GreatPresentation2.wav"

# Infer workflow on an audio input
prediction = asr_sentiment_workflow.predict_by_url(
audio_url,
input_type="audio",
)

# Get workflow results
print(prediction.results[0].outputs[-1].data)
Output
concepts {

id: "LABEL_2"

name: "LABEL_2"

value: 0.9802844524383545

app_id: "text-classification"

}

concepts {

id: "LABEL_1"

name: "LABEL_1"

value: 0.017947228625416756

app_id: "text-classification"

}

concepts {

id: "LABEL_0"

name: "LABEL_0"

value: 0.0017683181213214993

app_id: "text-classification"

}

This workflow seamlessly integrates detection, recognition, and embedding processes to yield precise face landmarks. This comprehensive approach not only identifies faces but also extracts meaningful embeddings, empowering users to conduct efficient visual searches based on the unique characteristics captured within the detected faces.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/main/workflows/Face"
image_url = "https://samples.clarifai.com/face-det.jpg"
# creating the workflow
face_search_workflow = Workflow(
url=workflow_url, pat="YOUR_PAT"
)

# Getting the predictions
result = face_search_workflow.predict_by_url(
url=image_url, input_type="image"
)
Output
regions {

id: "e2cb37cbc6a0e59f7afc4d74b55ecb91"

region_info {

bounding_box {

top_row: 0.265738964

left_col: 0.189666823

bottom_row: 0.519693851

right_col: 0.326894283

}

}

data {

clusters {

id: "428_280"

projection: -0.032409668

projection: 0.20954895

}

}

}

regions {

id: "abf63d045dae71ef5c6532e8edfda65f"

region_info {

bounding_box {

top_row: 0.174794614

left_col: 0.665749371

bottom_row: 0.395655245

...

projection: -0.0704193115

projection: -0.0369262695

}

}

}

Leverage the Predict API to detect a wide array of common objects within images, and use the potential of general embeddings to enable efficient and accurate visual search on identified regions.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/main/workflows/General-Detection"
video_url = "https://samples.clarifai.com/beer.mp4"
# Creating the workflows
object_search_workflow = Workflow(
url=workflow_url, pat="YOUR_PAT"
)
# getting the predictions
result = object_search_workflow.predict_by_url(
url=video_url, input_type="video"
)
print(result.results[0].outputs[-1].data)
Output
frames {

frame_info {

time: 500

}

data {

}

id: "faa9f3d5c8569123d8bea365ea478031"

}

frames {

frame_info {

index: 1

time: 1500

}

data {

regions {

id: "c58812cf4c3066e55fb16532dfde6986"

region_info {

bounding_box {

top_row: 0.0844013914

bottom_row: 1

right_col: 0.858514369

}

}
}


}
}

Text to Embedding

Incorporate the power of embeddings and clustering for comprehensive text processing by seamlessly integrating the respective nodes within your workflow. This dynamic combination allows you to enhance the understanding and organization of diverse textual data, unlocking a lot of possibilities for advanced language analysis and categorization.

from clarifai.client.workflow import Workflow


# input text
text = b"""The word pollution was derived from the Latin word pollution, which means to make dirty. Pollution is the process of making the environment pollute the water and the air by adding harmful substances. Pollution causes an imbalance in the environment. This imbalance threatened the very survival of all forms of life. It's a threat to the whole world. India ranks at the bottom of 125 of the 132 countries in the 2012 Environmental Performance Index. This report is produced by researchers at Yale and Columbia University in association with the World Economic Forum.
Pollution of the environment is a serious problem of industrialized societies. Industrial development and the green revolution have had a negative impact on the environment. People have converted the life support system of all living people into their own resources and have greatly disrupted the natural ecological balance. Serious degradation and depletion have been caused due to overuse, abuse and mismanagement of resources to meet human greed."""

workflow_url = "https://clarifai.com/clarifai/main/workflows/Language-Understanding"
# Creating the workflow
text_embedding_workflow = Workflow(
url= workflow_url,
pat="YOUR_PAT",
)



# getting the predictions
result = text_embedding_workflow.predict_by_bytes(text, input_type="text")
print(result.results[0].outputs[-1].data)
Output
clusters {

id: "24_21"

projection: 0.203612357

projection: -0.0755017698

}

Multimodal

Seamlessly leverage the power of multimodal inputs within your workflows using the Clarifai Predict API. With this capability, you can effortlessly transmit data in various modalities, such as text, images, or a combination of both, to a workflow equipped with a compatible model.

from clarifai.client.input import Inputs
from clarifai.client.workflow import Workflow

# setting up input image and prompt
prompt = "What time of day is it?"
image_url = "https://samples.clarifai.com/metro-north.jpg"

workflow_url = "https://clarifai.com/clarifai/Visual/workflows/multimodal-to-text"

multi_input = Inputs.get_multimodal_input(input_id="", image_url=image_url, raw_text=prompt)

# Creating the workflow
text_embedding_workflow = Workflow(
url=workflow_url,
pat="YOUR_PAT",
)

# Getting the predictions
result = text_embedding_workflow.predict(
inputs=[
multi_input
]
)
print(result.results[0].outputs[-1].data)
Output

text {

raw: "Based on the lighting and the sky\'s color in the image, it appears to be either dawn or dusk. The sky has a mix of light and dark hues, which is typical during twilight hours when the sun is below the horizon and the light is diffused. However, without additional context, it\'s not possible to determine with certainty whether it\'s morning or evening. The presence of snow on the ground suggests that it might be during the winter months when the days are shorter."

text_info {

encoding: "UnknownTextEnc"

}

}

Visual Detector

This workflow enables you to seamlessly detect a diverse range of common objects within a video, providing precise information about their location. The Predict API processes the video frames and generates regions of interest, outlining areas within each frame that potentially contain the identified objects.

# Create or load an app


from clarifai.client.user import User
from clarifai.client.app import App


# Create or load app (replace USER_ID and APP_ID with your own)
APP_ID = "APP_ID"
USER_ID="USER_ID"
try:
app = User(user_id=USER_ID).create_app(app_id=APP_ID, base_workflow="Empty")
except Exception as e:
app = App(app_id=APP_ID, user_id=USER_ID)


# # Create workflow from yaml file


"""bytetrack.yaml
workflow:
id: demo-bytetrack
nodes:
- id: object-detection
model:
user_id: clarifai
app_id: main
model_id: person-vehicle-detection-yolov5
- id: track
model:
model_id: "byte-tracker"
model_type_id: "byte-tracker"
output_info:
params:
min_visible_frames: 1
max_disappeared: 30
confidence_thresh: 0.1
node_inputs:
- node_id: object-detection
"""


workflow = app.create_workflow(config_filepath="./bytetrack.yaml")


result = workflow.predict_by_url("https://samples.clarifai.com/beer.mp4", input_type="video")
print(result.results[0].outputs[-1].data)
Output
frames {

frame_info {

time: 500

}

data {

regions {

id: "172c80386cbad83149135e715200f5a8"

region_info {

bounding_box {

top_row: 0.6157978773117065

left_col: 0.45200759172439575

bottom_row: 0.6784101724624634

right_col: 0.49515417218208313

}

}

data {

concepts {

id: "ai_BPXQSP1m"

name: "vehicle"

value: 0.2520264685153961

app_id: "main"

}

}

value: 0.2520264685153961

}

}

id: "faa9f3d5c8569123d8bea365ea478031"

}

frames {

frame_info {

index: 1

time: 1500

}

data {

}

id: "8e5f8672bdda2f2682d59ccc019d48c0"

}

frames {

frame_info {

index: 2

time: 2500

}

data {

}

id: "3f4cd8b6cbe2361de2d3a3f84906723c"

}

frames {

frame_info {

index: 3

time: 3500

}

data {

}

id: "049f7331f17764126fa433ccc7eb27a6"

}

frames {

frame_info {

index: 4

time: 4500

}

data {

}

id: "a815862119825cfb037834ec5dc24619"

}

frames {

frame_info {

index: 5

time: 5500

}

data {

}

id: "14572f138018d23fcd39a87f0c51880e"

}

frames {

frame_info {

index: 6

time: 6500

}

data {

}

id: "7790d9923639183be7213e8330736ea5"

}

frames {

frame_info {

index: 7

time: 7500

}

data {

}

id: "32224efe9c43139c9f3070930bae4e6c"

}

frames {

frame_info {

index: 8

time: 8500

}

data {

}

id: "0c4dd5d602afa6754bcfd441998412af"

}

tracks {

id: "1"

time_info {

}

}

Batch Predict - Workflows

Efficiently process multiple inputs in a single request by leveraging the Predict API's batch prediction feature in workflows. This allows you to streamline the prediction process, saving time and resources. Simply submit a batch of inputs to the model, and receive comprehensive predictions in return.

info

The batch size should not exceed 32.

from clarifai.client.input import Inputs
from clarifai.client.model import Model
from clarifai.client.workflow import Workflow

model_url = "https://clarifai.com/clarifai/main/models/general-image-recognition"
image_url = "https://samples.clarifai.com/metro-north.jpg"

# here is an example of creating an input proto list of size 32
proto_list=[]
for i in range(0,32):
proto_list.append(Inputs.get_input_from_url(input_id = f'demo_{i}', image_url=image_url))


workflow_url = "https://clarifai.com/clarifai/main/workflows/Moderation"
# Creating the workflow
image_classifcation_workflow = Workflow(
url=workflow_url, pat="YOUR_PAT"
)

# Getting the predictions
result = image_classifcation_workflow.predict(proto_list)
print(len(result.results))
Output
32