Advanced Inference Options

Learn how to perform advance inference operations using Clarifai SDKs

The inference API contains certain features that provides more flexibility while running predictions on inputs. This can help the end users to manipulate the outputs required for their tasks.

Batch Predict

Efficiently process multiple inputs in a single request by leveraging the Predict API's batch prediction feature. This allows you to streamline the prediction process, saving time and resources. Simply submit a batch of inputs to the model, and receive comprehensive predictions in return.

info

The batch size should not exceed 128.

Python
Typescript

from clarifai.client.input import Inputs
from clarifai.client.model import Model

model_url = "https://clarifai.com/openai/chat-completion/models/gpt-4o-mini"
prompt = "What's the future of AI?"

# here is an example of creating an input proto list of size 16
proto_list=[]
for i in range(16):
    proto_list.append(Inputs.get_input_from_bytes(input_id = f'demo_{i}', text_bytes=prompt.encode()))

# pass the input proto as paramater to the predict function
model_prediction = Model(url=model_url, pat="YOUR_PAT").predict(
    proto_list
)

# Check the length of predictions to see if all inputs were passed successfully

print(len(model_prediction.outputs))

Output

import { Inputs } from 'clarifai-client';
import { Model } from 'clarifai-client';

const modelUrl: string = "https://clarifai.com/openai/chat-completion/models/gpt-4o-mini";
const prompt: string = "What's the future of AI?";

// here is an example of creating an input proto list of size 16
const protoList: any[] = [];
for (let i = 0; i < 16; i++) {
    protoList.push(Inputs.getInputFromBytes({ inputId: `demo_${i}`, textBytes: Buffer.from(prompt) }));
}

// passthe input proto as parameter to the predict function
const modelPrediction = new Model({ url: modelUrl }).predict(protoList);

// Check the length of predictions to see if all inputs were passed successfully
console.log(modelPrediction.outputs.length);

Different Base_URL

Use the flexibility of the Predict API to obtain model predictions by tailoring the base_url. Customize your endpoint to seamlessly integrate with different environments, ensuring a versatile and adaptable approach to accessing model predictions.

info

This feature is for Enteprise that use on-prem deployments. So the base_url can be used to point to the respective servers.

Python

from clarifai.client.model import Model

# Your PAT (Personal Access Token) can be found in the Account's Security section
# Specify the correct user_id/app_id pairings
# Since you're making inferences outside your app's scope
#USER_ID = "cohere"
#APP_ID = "embed"

# You can set the model using model URL or model ID.
# Change these to whatever model you want to use
# eg : MODEL_ID = 'cohere-embed-english-v3_0'
# You can also set a particular model version by specifying the  version ID
# eg: MODEL_VERSION_ID = 'model_version'
#  Model class objects can be inititalised by providing its URL or also by defining respective user_id, app_id and model_id

# eg : model = Model(user_id="clarifai", app_id="main", model_id=MODEL_ID)

input_text = """In India Green Revolution commenced in the early 1960s that led to an increase in food grain production, especially in Punjab, Haryana, and Uttar Pradesh. Major milestones in this undertaking were the development of high-yielding varieties of wheat. The Green revolution is revolutionary in character due to the introduction of new technology, new ideas, the new application of inputs like HYV seeds, fertilizers, irrigation water, pesticides, etc. As all these were brought suddenly and spread quickly to attain dramatic results thus it is termed as a revolution in green agriculture.
"""
# The predict API gives the flexibility to generate predictions for data provided through URL, Filepath and bytes format.

# Example for prediction through URL:
# model_prediction = Model(model_url).predict_by_url(URL ,input_type="text")

# Example for prediction through Filepath:
# model_prediction = Model(model_url).predict_by_filepath(image_filepath, input_type="text")

model_url = "https://clarifai.com/cohere/embed/models/cohere-embed-english-v3_0"

# You can pass the new base url as paramater while initializing the Model object
model_prediction = Model(url=model_url, pat="YOUR_PAT",base_url="New Base URL").predict_by_bytes(
    input_text, "text"
)

embeddings = model_prediction.outputs[0].data.embeddings[0].vector

num_dimensions = model_prediction.outputs[0].data.embeddings[0].num_dimensions

print(embeddings[:10])

Output

[-0.02596100978553295,

 0.023946398869156837,

 -0.07173235714435577,

 0.032294824719429016,

 0.020313993096351624,

 -0.026998838409781456,

 0.008684193715453148,

 -0.016651064157485962,

 -0.012316598556935787,

 0.00042328768176957965]

Adding Root Certificate

A root certificate is considered as an extra level of security when communicating through API's. It's a self-signed certificate that verifies the legitimacy of other certificates. This establishes a chain of trust, ensuring you connect to real API's and your data is encrypted. The Clarifai SDKs allows users to add their own root certificates to enhance data security and privacy.

Visit this page to learn more about root certificates.

Python

from clarifai.client.model import Model

# Your PAT (Personal Access Token) can be found in the Account's Security section
# Specify the correct user_id/app_id pairings
# Since you're making inferences outside your app's scope
#USER_ID = "clarifai"
#APP_ID = "main"

# You can set the model using model URL or model ID.
# Change these to whatever model you want to use
# eg : MODEL_ID = "general-image-recognition"
# You can also set a particular model version by specifying the  version ID
# eg: MODEL_VERSION_ID = "aa7f35c01e0642fda5cf400f543e7c40"
#  Model class objects can be inititalised by providing its URL or also by defining respective user_id, app_id and model_id

# eg : model = Model(user_id="clarifai", app_id="main", model_id=MODEL_ID)

model_url = "https://clarifai.com/clarifai/main/models/general-image-recognition"
image_url = "https://samples.clarifai.com/metro-north.jpg"

# The predict API gives flexibility to generate predictions for data provided through URL,Filepath and bytes format.

# Example for prediction through Bytes:
# model_prediction = model.predict_by_bytes(input_bytes, input_type="image")

# Example for prediction through Filepath:
# model_prediction = Model(model_url).predict_by_filepath(filepath, input_type="image")

model_prediction = Model(url=model_url, pat="YOUR_PAT",root_certificates_path="PATH_TO_ROOT_CERTIFICATE").predict_by_url(
    image_url, input_type="image"
)

# Get the output
print(model_prediction.outputs[0].data)

Output

concepts {

  id: "ai_HLmqFqBf"

  name: "train"

  value: 0.999604881

  app_id: "main"

}

concepts {

  id: "ai_fvlBqXZR"

  name: "railway"

  value: 0.999297619

  app_id: "main"

}

concepts {

  id: "ai_SHNDcmJ3"

  name: "subway system"

  value: 0.99825567

  app_id: "main"

}

concepts {

  id: "ai_6kTjGfF6"

  name: "station"

  value: 0.998010933

  app_id: "main"

}

concepts {

  id: "ai_RRXLczch"

  name: "locomotive"

  value: 0.997254908

  app_id: "main"

}

concepts {

  id: "ai_Xxjc3MhT"

  name: "transportation system"

  value: 0.996976852

  app_id: "main"

}

concepts {

  id: "ai_VRmbGVWh"

  name: "travel"

  value: 0.988967717

  app_id: "main"

}

concepts {

  id: "ai_jlb9q33b"

  name: "commuter"

  value: 0.98089534

  app_id: "main"

}

concepts {

  id: "ai_2gkfMDsM"

  name: "platform"

  value: 0.980635285

  app_id: "main"

}

concepts {

  id: "ai_n9vjC1jB"

  name: "light"

  value: 0.974186838

  app_id: "main"

}

concepts {

  id: "ai_sQQj52KZ"

  name: "train station"

  value: 0.96878773

  app_id: "main"

}

concepts {

  id: "ai_l4WckcJN"

  name: "blur"

  value: 0.967302203

  app_id: "main"

}

concepts {

  id: "ai_WBQfVV0p"

  name: "city"

  value: 0.96151042

  app_id: "main"

}

concepts {

  id: "ai_TZ3C79C6"

  name: "road"

  value: 0.961382031

  app_id: "main"

}

concepts {

  id: "ai_CpFBRWzD"

  name: "urban"

  value: 0.960375667

  app_id: "main"

}

concepts {

  id: "ai_tr0MBp64"

  name: "traffic"

  value: 0.959969819

  app_id: "main"

}

concepts {

  id: "ai_GjVpxXrs"

  name: "street"

  value: 0.947492182

  app_id: "main"

}

concepts {

  id: "ai_mcSHVRfS"

  name: "public"

  value: 0.934322

  app_id: "main"

}

concepts {

  id: "ai_J6d1kV8t"

  name: "tramway"

  value: 0.931958199

  app_id: "main"

}

concepts {

  id: "ai_6lhccv44"

  name: "business"

  value: 0.929547548

  app_id: "main"

}

Prediction Paramaters

These parameters play a crucial role in configuring and customizing your prediction requests, ensuring accurate and tailored results based on your specific use case. Understanding and appropriately setting these prediction parameters will enhance your experience and enable you to extract meaningful insights from the Clarifai platform. The below parameters allows users to modify the predictions received from the model according to their needs.

Param Name	Param Description	Usage example
temperature	Temperature is a parameter of OpenAI ChatGPT, GPT-3 and GPT-4 models that governs the randomness and thus the creativity of the responses. It is always a number between 0 and 1.	`inference_params = dict(temperature=0.2) Model(model_url).predict(inputs,inference_params=inference_params)`
max_tokens	Max_tokens is a parameter for GPT models. This parameter shows the maximum number of tokens that can be processed inorder to get the response to your needs.	`inference_params = dict(max_tokens=100) Model(model_url).predict(inputs,inference_params=inference_params)`
min_value	The minimum value of the prediction confidence to filter.	`output_config = dict(min_value=0.6) Model(model_url).predict(inputs,output_config=output_config)`
max_concepts	The maximum number of concepts to return.	`output_config = dict(max_concepts=3) Model(model_url).predict(inputs,output_config=output_config)`
select_concepts	The concepts to select.	`output_config = dict(select_concepts=["concept_name"]) Model(model_url).predict(inputs,output_config=output_config)`

Advanced Inference Options

Batch Predict​

Different Base_URL​

Adding Root Certificate​

Prediction Paramaters​

What did you think of this doc?

Batch Predict

Different Base_URL

Adding Root Certificate

Prediction Paramaters