Skip to main content

Deep Fine-Tuning

Learn how to fine-tune pre-trained models

Fine-tuning is a deep learning technique that refers to taking a pre-trained model and further training it on a new dataset or task. The term "fine-tuning" implies making small adjustments or refinements to the already learned representations in the pre-trained model rather than training from scratch.

Fine-tuning leverages the power of pre-trained models to improve their performance on a new, related task. It involves taking a pre-trained model, which was previously trained on a vast dataset for a general-purpose task, and tailoring it to a more specific task.

Why Choose Deep Fine-Tuning?​

Clarifai offers a variety of pre-built models that are designed to help you create AI solutions quickly and efficiently. Clarifai models are the recommended starting point for many users because they offer incredibly fast training times, especially when you customize them using the transfer learning model type.

But there are some cases where accuracy and the ability to carefully target solutions takes priority over speed and ease of use. Additionally, you may need a model to learn new features not recognized by existing Clarifai models.

For such cases, it is possible to "deep fine-tune" your custom models and integrate them directly within your workflows.

You might consider deep fine-tuning if:

  • You have a custom tailored dataset. This will help you to tailor the model to a specific application or domain, such as customizing it with proprietary data from a private company.
  • You have accurate labels. This provides a strong foundation for training your models, resulting in improved performance, reduced errors, and better alignment with the desired task or domain.
  • You have the expertise and time to fine-tune models. So, you can modify the model's behavior to eliminate unwanted traits and instill desired ones.
  • You want to reduce hallucinations, especially when presenting the model with questions or prompts it hasn't encountered during its initial training.

Template Types​

You can take advantage of a variety of pre-configured templates when building your deep fine-tuned models. Templates give you the control to choose the specific architecture used by your neural network, and also define a set of hyperparameters that you can use to fine-tune the way that your model learns.

Here are the template types we offer:

How to Fine-Tune a Model​

Let's demonstrate how you can create and fine-tune a deep learning model.

Step 1: Choose a Model Type​

Before diving into fine-tuning, it's crucial to decide on the appropriate deep learning model architecture. Different model types excel at various tasks and generate distinct outputs based on your inputs and desired AI functionality.

Click here to learn more about the model types available on the Clarifai platform.

note

To illustrate how fine-tuning works, let's opt for a visual classifier model. This type of model excels at categorizing images into predefined classes or concepts. Our goal here is to create a fine-tuned visual classifier that can distinguish between images of cats and dogs.

Step 2: Prepare Training Data​

Preparing data for fine-tuning ensures that the custom model learns effectively from the new task or domain, generalizes well to unseen data, and produces reliable predictions.

Ensure that you collect high-quality, well-prepared data that will help achieve optimized performance in your model.

You can prepare your data using your preferred spreadsheet software. Click here to download a CSV template you can use to prepare your data.

To illustrate how fine-tuning works, we'll prepare the following simple dataset consisting of images of dogs and cats. Note that for your model to perform well, you need to provide it with enough diverse data to learn meaningful patterns. Click here to learn how you can get images to enrich your dataset.

Sample Dataset
https://samples.clarifai.com/dog1.jpeg
https://samples.clarifai.com/dog2.jpeg
https://samples.clarifai.com/dog3.jpeg
https://samples.clarifai.com/dog2_tiny.jpeg
https://samples.clarifai.com/dog.tiff
https://samples.clarifai.com/cat1.jpeg
https://samples.clarifai.com/cat2.jpeg
https://samples.clarifai.com/cat3.jpeg
https://samples.clarifai.com/featured-models/blip-flying-cat.jpg
https://samples.clarifai.com/featured-models/social-media-cat-laying-down.jpg
Number of Inputs

In general, deep fine-tuned models need more data than those trained using the transfer learning technique. For most use cases, you’ll need at least 1000 training inputs, but it could be much more than this depending on your specific scenario.

Step 3: Create an App​

Click here to learn how to create an application on the Clarifai portal.

note

When creating the application, go with the default Image/Video option as the primary input type. And in the collapsible Advanced Settings field, also go with the default Universal as the base workflow.

Step 4: Add and Annotate Inputs​

Select the Inputs option on your app’s collapsible left sidebar. Next, use the inputs uploader pop-up window to upload the images of dogs you prepared to a dataset within your application.

Also, label the images with the "dog" concept.

Click the Upload inputs button to add the annotated images of dogs to your app.

Similarly, use the inputs uploader pop-up window to upload the images of cats you prepared to the same dataset.

Also, label the images with the "cat" concept.

Click the Upload inputs button to add the annotated images of cats to your app.

Step 5: Choose a Model Type​

Select the Models option on your app’s collapsible left sidebar.

Click the Add Model button on the upper-right corner of the page. On the window that pops up, select the Build a Custom Model option and click the Continue button.

You’ll be redirected to a page where you can choose the type of model you want to create and fine-tune.

Select the Visual Classifier option.

Step 6: Create the Model​

The ensuing page allows you to create and fine-tune a visual classifier model that categorizes images into a set of predefined concepts.

Provide a unique ID and click the Continue to Configure model button to create your model.

After creating the model, set it up for training.

  • Dataset — Select the dataset you previously created that has the images of the dogs and cats. You can also select a specific version for it.
  • Concepts — Select the list of concepts you want the model to predict from the existing concepts in your app. For this example, let's select "cat" and "dog."
  • Invalid Data Tolerance Percent — Optionally, you can set a tolerance threshold (0 to 100) for the percentage of invalid inputs during training, and if this threshold is exceeded, training is stopped with an error.
  • Template — Select a pre-configured model template you want to use to fine-tune your model. When you choose a deep training template, you will see the hyperparameters that are available within that template populated with default values. You can adjust these values as desired. For this example, we’ll go with the recommended template: MMClassification_ResNet_50_RSB_A1. Click here to learn more about the visual classification templates.
note

Notice that the estimated duration for the training process is displayed for you. This Training Time Estimator feature offers transparency in expected training costs.

Step 7: Train the Model​​

After configuring the training settings, click the Train Model button to initiate training your model.

You'll be redirected to the individual page of your model.

If you check the model's versions table, you’ll notice that the model is still being trained.

Many hours may be required to deep train models with large numbers of inputs and complex taxonomies. You can cancel a deep fine-tuning job at any time by clicking the Cancel training button.

caution

Deep fine-tuning is billed at an hourly rate, and for canceled jobs, you will be charged for the time you've used to train your model. You can learn more about deep fine-tuning pricing here.

You can check the training progress by clicking the View Training Log button. If you click the button, a small sidebar will appear with details of the training exercise. You can also download the training log data by clicking the download button.

Step 8: Use the Model​​

After the model has been trained, the status will change to Model Trained.

In the Actions column, you can carry out the following:

Once you've created and trained your new model, you can put it to work. It will be ready to be evaluated, combined with other models and agent operators in a workflow, or used to serve inference requests as it is.

That's it!

© 2025 Clarifai, Inc. All rights reserved