Skip to main content

Inference from Workflows

Learn how to perform predictions with workflows using Clarifai SDKs

Experience streamlined and powerful inference capabilities through Clarifai SDKs when working with workflows. Unleash the full potential of your models by seamlessly integrating them into your workflows to obtain insightful predictions and enhance your data processing pipelines.By effortlessly integrating your models into workflows, you gain access to a streamlined process that delivers insightful predictions with minimal friction. Clarifai SDKs ensures a smooth experience, allowing you to effortlessly embed models into your applications. This not only enhances the predictive capabilities of your models but also optimizes the overall efficiency of your data processing pipelines.

The below inference examples are reference implementation that can be extended or reused for respective task based on model types.

Text Classifier​

The Predict API in the Clarifai SDKs empowers users to seamlessly perform text classification through a pre-built workflow tailored specifically for classification tasks. Use the capabilities of the Predict API to effortlessly categorize and analyze text, leveraging a robust and efficient classification model for enhanced accuracy and ease of use.

from clarifai.client.workflow import Workflow

# Your PAT (Personal Access Token) can be found in the Account's Security section

# Initialize the workflow_url
workflow_url = "https://clarifai.com/clarifai/main/workflows/Text-Moderation"
text_classfication_workflow = Workflow(
    url= workflow_url , pat="YOUR_PAT"
)
raw_text = b"I love your product very much"
result = text_classfication_workflow.predict_by_bytes(raw_text, input_type="text")
print(result.results[0].outputs[0].data)
Output
concepts {

  id: "ai_pkltpBZ7"

  name: "toxic"

  value: 0.996599138

  app_id: "main"

}

concepts {

  id: "ai_PlX1h51x"

  name: "insult"

  value: 0.515220046

  app_id: "main"

}

concepts {

  id: "ai_jf9vs404"

  name: "obscene"

  value: 0.0849464387

  app_id: "main"

}

concepts {

  id: "ai_2Dv0D4Nm"

  name: "identity_hate"

  value: 0.0308352802

  app_id: "main"

}

concepts {

...

  value: 0.00677669281

  app_id: "main"

}

Text to Text - Summarization​

Through the Predict API, you can seamlessly integrate a text summarization model into your workflow, enabling efficient extraction of key insights from extensive textual content. Streamline the summarization process and enhance the effectiveness of your applications with the powerful capabilities of Clarifai's Text Summarization model.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/Text/workflows/summary"
# creating the workflow
text_summarization_workflow = Workflow(
    url= workflow_url, pat="YOUR_PAT"
)
# initialize the input text (minimum 200 characters)
text = b"""The word pollution was derived from the Latin word pollution, which means to make dirty. Pollution is the process of making the environment pollute the water and the air by adding harmful substances. Pollution causes an imbalance in the environment. This imbalance threatened the very survival of all forms of life. It's a threat to the whole world. India ranks at the bottom of 125 of the 132 countries in the 2012 Environmental Performance Index. This report is produced by researchers at Yale and Columbia University in association with the World Economic Forum.

Pollution of the environment is a serious problem of industrialized societies. Industrial development and the green revolution have had a negative impact on the environment. People have converted the life support system of all living people into their own resources and have greatly disrupted the natural ecological balance. Serious degradation and depletion have been caused due to overuse, abuse and mismanagement of resources to meet human greed."""


result = text_summarization_workflow.predict_by_bytes(text, input_type="text")
print(result.results[0].outputs[0].data.text.raw)
Output
- Pollution, taken from the Latin word meaning "to make dirty", is the act of using harmful substances to dirty the air and water around us, to the point where it threatens all life forms

- India ranks terribly low on the Environmental Performance Index, an evaluation of countries' environmental statuses, produced by researchers at Yale and Columbia University in collaboration with the World Economic Forum

- While pollution is a problem for all societies, industrialized societies are particularly afflicted due to the further damage industrial development has caused to the environment

- However, the worst impact of pollution may be yet to come, as warns the author, and the best course of action going forward needs to be evaluated. 

Would you like help with anything else regarding this summary?

Text to Text - Generation​

With our Predict API, you can seamlessly generate textual content by providing a prompt as input to a workflow that incorporates a powerful text generation model. This feature allows you to use the capabilities of state-of-the-art language models, enabling you to dynamically generate text for a variety of applications, from creative writing to automated content creation.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/Text/workflows/text-generation"
# Creating the workflow
text_generation_workflow = Workflow(
    url=workflow_url,
    pat="YOUR_PAT",
)

# getting the prediction
result = text_generation_workflow.predict_by_bytes(
    b"Good Morning! I think it's going to be a great day!", input_type="text"
)
print(result.results[0].outputs[0].data.text.raw)
Output
Good morning! I'm glad to hear you're starting the day with such a positive attitude. How can I assist you today to make it even better?

Visual Classifier​

Effortlessly conduct image classification through the Predict API by seamlessly integrating with a workflow designed around powerful classification models. This streamlined process empowers users to harness the capabilities of classification models, enabling accurate and efficient categorization of images.

Visit this link to view the available models for this use case.

from clarifai.client.workflow import Workflow

image_url =     "https://cdn-bfgco.nitrocdn.com/PLGJnButkKeCQigeKyiwHwnQLZJDOQZI/assets/images/optimized/rev-96fa12a/delamere.com/wp-content/uploads/2022/02/Picture1-1024x683.jpg"
workflow_url = "https://clarifai.com/clarifai/main/workflows/Moderation"
# Creating the workflow
image_classifcation_workflow = Workflow(
    url=workflow_url, pat="YOUR_PAT"
)

# Getting the predictions
result = image_classifcation_workflow.predict_by_url(image_url ,
    input_type="image",
)
print(result.results[0].outputs[0].data)
Output
concepts {

  id: "ai_8QQwMjQR"

  name: "drug"

  value: 0.99999994

  app_id: "main"

}

concepts {

  id: "ai_QD1zClSd"

  name: "safe"

  value: 4.93858607e-08

  app_id: "main"

}

concepts {

  id: "ai_V76bvrtj"

  name: "explicit"

  value: 3.21394289e-09

  app_id: "main"

}

concepts {

  id: "ai_kBBGf7r8"

  name: "gore"

  value: 1.82284843e-09

  app_id: "main"

}

concepts {

...

  value: 8.18517032e-10

  app_id: "main"

}

Visual Segmenter​

By leveraging the Predict API, you can seamlessly integrate image segmentation models into your workflow. This powerful functionality allows you to precisely categorize different regions within an image, unlocking enhanced capabilities for object identification, analysis, and understanding.

from clarifai.client.workflow import Workflow
import cv2


image_url = "https://s3.amazonaws.com/samples.clarifai.com/featured-models/general-elephants.jpg"
workflow_url = "https://clarifai.com/clarifai/main/workflows/Visual-Segmenter"
# Creating the workflow
image_segmentation_workflow = Workflow(
    url=workflow_url, pat="YOUR_PAT"
)
# Getting the predictions
result = image_segmentation_workflow.predict_by_url(
image_url ,
    input_type="image",
)
im_b = result.results[0].outputs[0].data.regions[0].region_info.mask.image.base64
image_np = np.frombuffer(im_b, np.uint8)
img_np = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
# Display the image
plt.axis("off")
plt.imshow(img_np[..., ::-1])
Image Output

Image to Text​

By leveraging the Predict API, you can effortlessly integrate OCR capabilities into your applications through a streamlined workflow that involves an OCR model. This functionality allows you to extract and interpret text from images, empowering your applications to recognize and process textual information efficiently.

from clarifai.client.workflow import Workflow
from clarifai.client.user import User

# Your PAT (Personal Access Token) can be found in the Account's Security section
# Specify the correct user_id/app_id pairings
# Since you're making inferences outside your app's scope
USER_ID = "user_id"
APP_ID = "app_id"

app = User(user_id=USER_ID, pat="YOUR_PAT").create_app(
    app_id=APP_ID, base_workflow="Empty"
)
image_url =  "https://s3.amazonaws.com/samples.clarifai.com/ocr_img_00417302.jpg"
# create a yaml file specifying the workflow structure
# eg:
"""language_aware_ocr.yml
workflow:
  id: wf-ocr
  nodes:
    - id: ocr-workflow
      model:
          model_id: language-aware-multilingual-ocr-multiplex

    - id: text-aggregator
      model:
          model_id: text-aggregation
          model_type_id: text-aggregation-operator
          output_info:
            params:
              avg_word_width_window_factor: 2.0
              avg_word_height_window_factor: 1.0

      node_inputs:
        - node_id: ocr-workflow

    - id: language-id-operator
      model:
          model_id: language-id
          model_type_id: language-id-operator
          output_info:
            params:
              library: "fasttext"
              topk: 1
              threshold:  0.1
              lowercase: true

      node_inputs:
        - node_id: text-aggregator


"""


ocr_workflow = app.create_workflow(config_filepath="configs/language_aware_ocr.yml")
prediction = ocr_workflow.predict_by_url(
   image_url ,
    input_type="image",
)

# Get workflow results
print(prediction.results[0].outputs[-1].data)
Output
2024-01-11 09:09:54 INFO     clarifai:  wf-ocr Workflow is still deploying, please wait...           workflow.py:89

INFO:clarifai:wf-ocr Workflow is still deploying, please wait...

2024-01-11 09:10:57 INFO     clarifai:  wf-ocr Workflow is still deploying, please wait...           workflow.py:89

INFO:clarifai:wf-ocr Workflow is still deploying, please wait...

text {

  raw: "H WerP\nRADOAS TESTXIT\nTRRTESTINCEERINCMODEB\nEPAGSTED\nONETIMEIISE\nDITED1YEARWPRRNTY"

  text_info {

    encoding: "UnknownTextEnc"

  }

}

Text to Image​

Create captivating visuals effortlessly with the help of Predict API. The Image Generation feature empowers developers to seamlessly generate images based on textual prompts through a streamlined workflow utilizing the powerful text-to-image model.

from clarifai.client.workflow import Workflow
import cv2


workflow_url ="https://clarifai.com/clarifai/Visual/workflows/image-generation"
# Creating the workflow
image_generation_workflow = Workflow(
    url=workflow_url,
    pat="YOUR_PAT",
)

# Getting the predictions
result = image_generation_workflow.predict_by_bytes(
    b"cat with a hat", input_type="text"
)
# Extract the base64-encoded image data from the result object
im_b = result.results[0].outputs[0].data.image.base64
# Convert the base64-encoded image data to a NumPy array of uint8 values
image_np = np.frombuffer(im_b, np.uint8)
# Decode the NumPy array to obtain the image in OpenCV format (BGR color)
img_np = cv2.imdecode(image_np, cv2.IMREAD_COLOR)

# Display the image
plt.axis("off")
plt.imshow(img_np[..., ::-1])
Image Output

Image to Image​

By leveraging our Predict API, you can seamlessly enhance the quality of any image through a streamlined workflow featuring our state-of-the-art upscaling model. Empower your applications with higher-resolution visuals, delivering a superior visual experience to your users.

from clarifai.client.workflow import Workflow
import cv2



workflow_url = "https://clarifai.com/clarifai/Visual/workflows/upscale-workflow"
image_url = "https://s3.amazonaws.com/samples.clarifai.com/featured-models/image-captioning-statue-of-liberty.jpeg"

# Creating the workflow
image_upscale_workflow = Workflow(
    url=workflow_url,
    pat="YOUR_PAT",
)

# Getting the predictions
result = image_upscale_workflow.predict_by_url(
    image_url ,
    input_type="image",
)

im_b = result.results[0].outputs[0].data.image.base64
image_np = np.frombuffer(im_b, np.uint8)
img_np = cv2.imdecode(image_np, cv2.IMREAD_COLOR)
# Display the image
plt.axis("off")
plt.imshow(img_np[..., ::-1])
Image Output

Text to Audio​

Leverage the power of the Predict API to seamlessly transform text into audio. This process involves submitting the text as input to a dedicated workflow that incorporates a state-of-the-art text-to-speech model. By integrating this feature, you can effortlessly generate high-quality audio outputs, expanding the scope of your applications and enhancing the overall user experience.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/Text/workflows/text-to-audio"
# Creating thr workflow
text_to_audio_workflow = Workflow(
    url=workflow_url,
    pat="YOUR_PAT",
)

# Getting the predictions
result = text_to_audio_workflow.predict_by_bytes(
    b"I love your product very much", input_type="text"
)

# Save the audio file
with open("output_audio.wav", mode="bx") as f:
    f.write(result.results[0].outputs[0].data.audio.base64)

Audio to Text​

With the Clarifai Predict API, seamlessly convert audio to text by incorporating it as input to a dedicated speech-to-text model within a workflow. Effortlessly use the capabilities of Clarifai's advanced technology to transcribe audio content, enabling a wide range of applications such as transcription services, voice command recognition, and more.

from clarifai.client.user import User


# Create a new app (replace USER_ID and APP_ID with your own)
app = User(user_id="user_id", pat="YOUR_PAT").create_app(
    app_id="APP_ID", base_workflow="Empty"
)
# Create ASR-Sentiment workflow
asr_sentiment_workflow = app.create_workflow(
    config_filepath="configs/asr_sentiment.yml"
)

audio_url = "https://s3.amazonaws.com/samples.clarifai.com/GreatPresentation2.wav"

# Infer workflow on an audio input
prediction = asr_sentiment_workflow.predict_by_url(
    audio_url,
    input_type="audio",
)

# Get workflow results
print(prediction.results[0].outputs[-1].data)
Output
concepts {

  id: "LABEL_2"

  name: "LABEL_2"

  value: 0.9802844524383545

  app_id: "text-classification"

}

concepts {

  id: "LABEL_1"

  name: "LABEL_1"

  value: 0.017947228625416756

  app_id: "text-classification"

}

concepts {

  id: "LABEL_0"

  name: "LABEL_0"

  value: 0.0017683181213214993

  app_id: "text-classification"

}

Visual Detector - Face search​

This workflow seamlessly integrates detection, recognition, and embedding processes to yield precise face landmarks. This comprehensive approach not only identifies faces but also extracts meaningful embeddings, empowering users to conduct efficient visual searches based on the unique characteristics captured within the detected faces.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/main/workflows/Face"
image_url = "https://samples.clarifai.com/face-det.jpg"
# creating the workflow
face_search_workflow = Workflow(
    url=workflow_url, pat="YOUR_PAT"
)

# Getting the predictions
result = face_search_workflow.predict_by_url(
    url=image_url, input_type="image"
)
Output
regions {

  id: "e2cb37cbc6a0e59f7afc4d74b55ecb91"

  region_info {

    bounding_box {

      top_row: 0.265738964

      left_col: 0.189666823

      bottom_row: 0.519693851

      right_col: 0.326894283

    }

  }

  data {

    clusters {

      id: "428_280"

      projection: -0.032409668

      projection: 0.20954895

    }

  }

}

regions {

  id: "abf63d045dae71ef5c6532e8edfda65f"

  region_info {

    bounding_box {

      top_row: 0.174794614

      left_col: 0.665749371

      bottom_row: 0.395655245

...

      projection: -0.0704193115

      projection: -0.0369262695

    }

  }

}

Visual Detector - Object Search​

Leverage the Predict API to detect a wide array of common objects within images, and use the potential of general embeddings to enable efficient and accurate visual search on identified regions.

from clarifai.client.workflow import Workflow


workflow_url = "https://clarifai.com/clarifai/main/workflows/General-Detection"
video_url = "https://samples.clarifai.com/beer.mp4"
# Creating the workflows
object_search_workflow = Workflow(
    url=workflow_url, pat="YOUR_PAT"
)
# getting the predictions
result = object_search_workflow.predict_by_url(
    url=video_url, input_type="video"
)
print(result.results[0].outputs[-1].data)
Output
frames {

  frame_info {

    time: 500

  }

  data {

  }

  id: "faa9f3d5c8569123d8bea365ea478031"

}

frames {

  frame_info {

    index: 1

    time: 1500

  }

  data {

    regions {

      id: "c58812cf4c3066e55fb16532dfde6986"

      region_info {

        bounding_box {

          top_row: 0.0844013914

          bottom_row: 1

          right_col: 0.858514369

        }

      }
    }
    
    
    }
}

Text to Embedding​

Incorporate the power of embeddings and clustering for comprehensive text processing by seamlessly integrating the respective nodes within your workflow. This dynamic combination allows you to enhance the understanding and organization of diverse textual data, unlocking a lot of possibilities for advanced language analysis and categorization.

from clarifai.client.workflow import Workflow


# input text
text = b"""The word pollution was derived from the Latin word pollution, which means to make dirty. Pollution is the process of making the environment pollute the water and the air by adding harmful substances. Pollution causes an imbalance in the environment. This imbalance threatened the very survival of all forms of life. It's a threat to the whole world. India ranks at the bottom of 125 of the 132 countries in the 2012 Environmental Performance Index. This report is produced by researchers at Yale and Columbia University in association with the World Economic Forum.
Pollution of the environment is a serious problem of industrialized societies. Industrial development and the green revolution have had a negative impact on the environment. People have converted the life support system of all living people into their own resources and have greatly disrupted the natural ecological balance. Serious degradation and depletion have been caused due to overuse, abuse and mismanagement of resources to meet human greed."""

workflow_url = "https://clarifai.com/clarifai/main/workflows/Language-Understanding"
# Creating the workflow
text_embedding_workflow = Workflow(
    url= workflow_url,
    pat="YOUR_PAT",
)



# getting the predictions
result = text_embedding_workflow.predict_by_bytes(text, input_type="text")
print(result.results[0].outputs[-1].data)
Output
clusters {

  id: "24_21"

  projection: 0.203612357

  projection: -0.0755017698

}

Multimodal​

Seamlessly leverage the power of multimodal inputs within your workflows using the Clarifai Predict API. With this capability, you can effortlessly transmit data in various modalities, such as text, images, or a combination of both, to a workflow equipped with a compatible model.

from clarifai.client.input import Inputs
from clarifai.client.workflow import Workflow

# setting up input image and prompt
prompt = "What time of day is it?"
image_url = "https://samples.clarifai.com/metro-north.jpg"

workflow_url = "https://clarifai.com/clarifai/Visual/workflows/multimodal-to-text"

multi_input = Inputs.get_multimodal_input(input_id="", image_url=image_url, raw_text=prompt)

# Creating the workflow
text_embedding_workflow = Workflow(
    url=workflow_url,
    pat="YOUR_PAT",
)

# Getting the predictions
result = text_embedding_workflow.predict(
    inputs=[
        multi_input
    ]
)
print(result.results[0].outputs[-1].data)
Output

text {

  raw: "Based on the lighting and the sky\'s color in the image, it appears to be either dawn or dusk. The sky has a mix of light and dark hues, which is typical during twilight hours when the sun is below the horizon and the light is diffused. However, without additional context, it\'s not possible to determine with certainty whether it\'s morning or evening. The presence of snow on the ground suggests that it might be during the winter months when the days are shorter."

  text_info {

    encoding: "UnknownTextEnc"

  }

}

Visual Detector​

This workflow enables you to seamlessly detect a diverse range of common objects within a video, providing precise information about their location. The Predict API processes the video frames and generates regions of interest, outlining areas within each frame that potentially contain the identified objects.

# Create or load an app


from clarifai.client.user import User
from clarifai.client.app import App


# Create or load app (replace USER_ID and APP_ID with your own)
APP_ID = "APP_ID"
USER_ID="USER_ID"
try:
 app = User(user_id=USER_ID).create_app(app_id=APP_ID, base_workflow="Empty")
except Exception as e:
 app = App(app_id=APP_ID, user_id=USER_ID)


# # Create workflow from yaml file


"""bytetrack.yaml
workflow:
 id: demo-bytetrack
 nodes:
   - id: object-detection
     model:
       user_id: clarifai
       app_id: main
       model_id: person-vehicle-detection-yolov5
   - id: track
     model:
       model_id: "byte-tracker"
       model_type_id: "byte-tracker"
       output_info:
         params:
           min_visible_frames: 1
           max_disappeared: 30
           confidence_thresh:  0.1
     node_inputs:
       - node_id: object-detection
"""


workflow = app.create_workflow(config_filepath="./bytetrack.yaml")


result = workflow.predict_by_url("https://samples.clarifai.com/beer.mp4", input_type="video")
print(result.results[0].outputs[-1].data)
Output
frames {

  frame_info {

    time: 500

  }

  data {

    regions {

      id: "172c80386cbad83149135e715200f5a8"

      region_info {

        bounding_box {

          top_row: 0.6157978773117065

          left_col: 0.45200759172439575

          bottom_row: 0.6784101724624634

          right_col: 0.49515417218208313

        }

      }

      data {

        concepts {

          id: "ai_BPXQSP1m"

          name: "vehicle"

          value: 0.2520264685153961

          app_id: "main"

        }

      }

      value: 0.2520264685153961

    }

  }

  id: "faa9f3d5c8569123d8bea365ea478031"

}

frames {

  frame_info {

    index: 1

    time: 1500

  }

  data {

  }

  id: "8e5f8672bdda2f2682d59ccc019d48c0"

}

frames {

  frame_info {

    index: 2

    time: 2500

  }

  data {

  }

  id: "3f4cd8b6cbe2361de2d3a3f84906723c"

}

frames {

  frame_info {

    index: 3

    time: 3500

  }

  data {

  }

  id: "049f7331f17764126fa433ccc7eb27a6"

}

frames {

  frame_info {

    index: 4

    time: 4500

  }

  data {

  }

  id: "a815862119825cfb037834ec5dc24619"

}

frames {

  frame_info {

    index: 5

    time: 5500

  }

  data {

  }

  id: "14572f138018d23fcd39a87f0c51880e"

}

frames {

  frame_info {

    index: 6

    time: 6500

  }

  data {

  }

  id: "7790d9923639183be7213e8330736ea5"

}

frames {

  frame_info {

    index: 7

    time: 7500

  }

  data {

  }

  id: "32224efe9c43139c9f3070930bae4e6c"

}

frames {

  frame_info {

    index: 8

    time: 8500

  }

  data {

  }

  id: "0c4dd5d602afa6754bcfd441998412af"

}

tracks {

  id: "1"

  time_info {

  }

}

Batch Predict - Workflows​

Efficiently process multiple inputs in a single request by leveraging the Predict API's batch prediction feature in workflows. This allows you to streamline the prediction process, saving time and resources. Simply submit a batch of inputs to the model, and receive comprehensive predictions in return.

info

The batch size should not exceed 32.

from clarifai.client.input import Inputs
from clarifai.client.model import Model
from clarifai.client.workflow import Workflow

model_url = "https://clarifai.com/clarifai/main/models/general-image-recognition"
image_url = "https://samples.clarifai.com/metro-north.jpg"

# here is an example of creating an input proto list of size 32
proto_list=[]
for i in range(0,32):
    proto_list.append(Inputs.get_input_from_url(input_id = f'demo_{i}', image_url=image_url))


workflow_url = "https://clarifai.com/clarifai/main/workflows/Moderation"
# Creating the workflow
image_classifcation_workflow = Workflow(
    url=workflow_url, pat="YOUR_PAT"
)

# Getting the predictions
result = image_classifcation_workflow.predict(proto_list)
print(len(result.results))
Output
32
© 2024 Clarifai, Inc. All rights reserved