Table of Contents

Search Images by Visual Similarity

Image similarity search converts images into numerical vectors (embeddings) and finds the closest matches in vector space. Instead of relying on filenames, tags, or manual labels, it compares the visual content of images directly. Two photos of a sunset will match each other even if they have completely different filenames and no metadata.

This tutorial builds a working image similarity search system: loading a multimodal embedding model, indexing a folder of images, and finding visually similar matches. It also covers text-to-image search, where a natural language description finds matching images, with the same model.


Why Visual Similarity Search Matters

Two enterprise problems that on-device image similarity search solves:

  1. E-commerce visual search. Customers upload a photo of a product they want to find. A visual search system matches it against the product catalog instantly, without requiring text descriptions. Running this locally keeps proprietary catalog data and customer images on-premises.
  2. Duplicate and near-duplicate photo detection. Media libraries, digital asset management systems, and forensic workflows need to find duplicate or near-duplicate images across large collections. Embedding-based comparison catches rotated, cropped, and color-adjusted variants that hash-based methods miss.

Prerequisites

Requirement Minimum
.NET SDK 8.0+
RAM 8+ GB
VRAM 2+ GB (optional: the model also runs on CPU)
Disk ~500 MB free for model download
Image files A folder of .jpg, .png, or .jpeg images to index

Step 1: Create the Project

dotnet new console -n ImageSimilaritySearch
cd ImageSimilaritySearch
dotnet add package LM-Kit.NET

Step 2: Understand Image Embeddings

An image embedding model converts an image into a fixed-size vector of floating-point numbers. Images with similar visual content produce vectors that are close together in this high-dimensional space. Cosine similarity measures how close two vectors are (1.0 = identical, 0.0 = unrelated).

embeddinggemma-2 (Google EmbeddingGemma 2) embeds text, code and images into one 768-dimensional vector space. A single model therefore serves both image-to-image search and text-to-image search: the vector of the sentence "a sandy beach" is directly comparable with the vector of a beach photo.

                   ┌────────────────────┐
  beach.jpg   ───► │                    │──► [0.42, 0.18, -0.61, ...]  ─┐
                   │                    │                                ├─ close
  coast.jpg   ───► │  embeddinggemma-2  │──► [0.40, 0.20, -0.58, ...]  ─┘
                   │                    │
  circuit.jpg ───► │  (one vector space │──► [-0.33, 0.71, 0.12, ...]  ── far
                   │   for text and     │
  "a sandy    ───► │   images)          │──► [0.39, 0.21, -0.55, ...]  ── cross-modal match
   beach"          └────────────────────┘                                  with beach.jpg

Images are sized by the model's own processor before encoding (280 vision tokens per image by default). Configuration.DefaultImageDetail raises or lowers that budget for every pipeline at once.


Step 3: Write the Program

This program loads the model, indexes all images in a folder, and supports both image-to-image and text-to-image search.

using System.Text;
using LMKit.Model;
using LMKit.Embeddings;
using LMKit.Media.Image;

Console.InputEncoding = Encoding.UTF8;
Console.OutputEncoding = Encoding.UTF8;

// ──────────────────────────────────────
// 1. Load the multimodal embedding model
// ──────────────────────────────────────
Console.WriteLine("Loading embedding model...");
using LM model = LM.LoadFromModelID("embeddinggemma-2",
    downloadingProgress: (_, len, read) =>
    {
        if (len.HasValue) Console.Write($"\r  Downloading: {(double)read / len.Value * 100:F1}%   ");
        return true;
    },
    loadingProgress: p => { Console.Write($"\r  Loading: {p * 100:F0}%   "); return true; });
Console.WriteLine(" Done.\n");

// One embedder serves images and text alike.
var embedder = new Embedder(model);

// ──────────────────────────────────────
// 2. Index images as embedding vectors
// ──────────────────────────────────────
string imageFolder = args.Length > 0 ? args[0] : "images";

if (!Directory.Exists(imageFolder))
{
    Console.WriteLine($"Folder not found: {imageFolder}");
    Console.WriteLine("Usage: dotnet run -- <path-to-image-folder>");
    return;
}

string[] imageFiles = Directory.GetFiles(imageFolder, "*.*")
    .Where(f => f.EndsWith(".jpg", StringComparison.OrdinalIgnoreCase) ||
                f.EndsWith(".png", StringComparison.OrdinalIgnoreCase) ||
                f.EndsWith(".jpeg", StringComparison.OrdinalIgnoreCase))
    .ToArray();

Console.WriteLine($"Indexing {imageFiles.Length} images...\n");

var imageIndex = new List<(string Path, float[] Embedding)>();

foreach (string imagePath in imageFiles)
{
    using ImageBuffer image = ImageBuffer.LoadAsRGB(imagePath);
    float[] embedding = embedder.GetEmbeddings(image);
    imageIndex.Add((imagePath, embedding));
    Console.WriteLine($"  Indexed: {Path.GetFileName(imagePath)}");
}

Console.WriteLine($"\nIndex complete: {imageIndex.Count} images ({embedder.EmbeddingSize} dimensions)\n");

// ──────────────────────────────────────
// 3. Search by image or by text
// ──────────────────────────────────────
Console.WriteLine("Enter an image filename to find similar images, or a text description for cross-modal search.");
Console.WriteLine("Type 'quit' to exit.\n");

while (true)
{
    Console.ForegroundColor = ConsoleColor.Green;
    Console.Write("Search: ");
    Console.ResetColor();

    string? input = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(input) || input.Equals("quit", StringComparison.OrdinalIgnoreCase))
        break;

    float[] queryEmbedding;

    // An existing file is a query image; anything else is a text query.
    if (File.Exists(input))
    {
        Console.WriteLine($"  Searching by image: {Path.GetFileName(input)}\n");
        using ImageBuffer queryImage = ImageBuffer.LoadAsRGB(input);
        queryEmbedding = embedder.GetEmbeddings(queryImage);
    }
    else
    {
        Console.WriteLine($"  Searching by text: \"{input}\"\n");
        queryEmbedding = embedder.GetEmbeddings(input);
    }

    var results = imageIndex
        .Select(item => new
        {
            item.Path,
            Similarity = Embedder.GetCosineSimilarity(queryEmbedding, item.Embedding)
        })
        .OrderByDescending(x => x.Similarity)
        .Take(5)
        .ToList();

    Console.ForegroundColor = ConsoleColor.Cyan;
    Console.WriteLine("  Top matches:");
    Console.ResetColor();

    foreach (var match in results)
    {
        Console.ForegroundColor = ConsoleColor.DarkGray;
        Console.Write($"  {match.Similarity:F4}  ");
        Console.ResetColor();
        Console.WriteLine(Path.GetFileName(match.Path));
    }

    Console.WriteLine();
}

Run it:

dotnet run -- "path/to/image/folder"

Step 4: Example Output

Scores below are illustrative; the ranking is what matters.

Loading embedding model...
  Loading: 100%  Done.

Indexing 12 images...

  Indexed: beach-sunset.jpg
  Indexed: mountain-lake.jpg
  Indexed: city-skyline.jpg
  Indexed: golden-retriever.jpg
  Indexed: tabby-cat.jpg
  Indexed: red-sports-car.jpg
  Indexed: ocean-waves.jpg
  Indexed: forest-trail.jpg
  Indexed: office-desk.jpg
  Indexed: pizza-closeup.jpg
  Indexed: laptop-keyboard.jpg
  Indexed: flower-garden.jpg

Index complete: 12 images (768 dimensions)

Enter an image filename to find similar images, or a text description for cross-modal search.
Type 'quit' to exit.

Search: images/beach-sunset.jpg
  Searching by image: beach-sunset.jpg

  Top matches:
  1.0000  beach-sunset.jpg
  0.8734  ocean-waves.jpg
  0.6219  mountain-lake.jpg
  0.4102  forest-trail.jpg
  0.3011  flower-garden.jpg

Search: a cute pet animal
  Searching by text: "a cute pet animal"

  Top matches:
  0.4821  golden-retriever.jpg
  0.4544  tabby-cat.jpg
  0.2102  flower-garden.jpg
  0.1541  forest-trail.jpg
  0.1293  beach-sunset.jpg

Tuning Similarity Thresholds

Absolute scores depend on the model and on the collection, so calibrate a threshold on a few known pairs from your own data. As a starting point:

Use Case Starting Threshold Rationale
Duplicate detection 0.95+ Only near-identical images
Visual search (similar items) 0.70+ Visually related but not identical
Broad category matching 0.40+ Same general category (e.g., "outdoor scenes")
Text-to-image search Rank, then cut Cross-modal scores sit lower than image-to-image scores; keep the top results rather than a fixed cutoff

Apply a threshold to the ranked results of Step 3:

float threshold = 0.70f;

var filtered = results.Where(r => r.Similarity >= threshold).ToList();

if (filtered.Count == 0)
{
    Console.WriteLine("  No matches above the similarity threshold.");
}
else
{
    foreach (var match in filtered)
        Console.WriteLine($"  {match.Similarity:F4}  {Path.GetFileName(match.Path)}");
}

Batch Indexing for Large Collections

When indexing hundreds or thousands of images, embed them in batches for better throughput. Each batch is disposed once its vectors are computed, so memory stays bounded:

const int BatchSize = 32;
var imageIndex = new List<(string Path, float[] Embedding)>();

foreach (string[] batch in imageFiles.Chunk(BatchSize))
{
    var images = batch.Select(ImageBuffer.LoadAsRGB).ToList();
    try
    {
        float[][] embeddings = embedder.GetEmbeddings(images);
        for (int i = 0; i < batch.Length; i++)
            imageIndex.Add((batch[i], embeddings[i]));
    }
    finally
    {
        foreach (ImageBuffer image in images)
            image.Dispose();
    }
}

For very large collections, persist the vectors so they are computed only once:

const string IndexFile = "image_index.bin";

// Save
using (var writer = new BinaryWriter(File.Create(IndexFile)))
{
    writer.Write(imageIndex.Count);
    foreach (var (path, embedding) in imageIndex)
    {
        writer.Write(path);
        writer.Write(embedding.Length);
        foreach (float value in embedding)
            writer.Write(value);
    }
}

// Load
var loaded = new List<(string Path, float[] Embedding)>();
using (var reader = new BinaryReader(File.OpenRead(IndexFile)))
{
    int count = reader.ReadInt32();
    for (int i = 0; i < count; i++)
    {
        string path = reader.ReadString();
        var embedding = new float[reader.ReadInt32()];
        for (int j = 0; j < embedding.Length; j++)
            embedding[j] = reader.ReadSingle();
        loaded.Add((path, embedding));
    }
}

Vectors are only comparable with vectors from the same model: re-index when you change it. For a managed store with search built in, see Build Semantic Search with Embeddings.


Smaller Vectors with Matryoshka Truncation

EmbeddingGemma 2 is trained so that the first 512, 256 or 128 dimensions of a vector are a valid embedding on their own. Truncate and re-normalize to shrink an index, and compare truncated vectors only with vectors truncated to the same size:

static float[] Truncate(float[] vector, int dimensions)
{
    float[] head = vector[..dimensions];
    float norm = MathF.Sqrt(head.Sum(x => x * x));
    return head.Select(x => x / norm).ToArray();
}

float[] compact = Truncate(embedding, 256); // a third of the storage

Choosing an Embedding Model

Model ID Inputs Dimensions Best For
embeddinggemma-2 Text, code and images 768 (truncatable to 512, 256, 128) Image-to-image and text-to-image search with one model (recommended)
nomic-embed-vision Images only 768 Lightweight image encoder (about 97 MB) for tight memory budgets
nomic-embed-text Text only 768 Text queries against nomic-embed-vision image vectors (the two share a space)
harrier-oss:0.6b Text only 1024 Text-only multilingual search (not for images)

With embeddinggemma-2, text and image vectors come from the same model and are directly comparable. With the Nomic pair, image vectors come from nomic-embed-vision and text queries from nomic-embed-text; never mix vectors across the two families.


Common Issues

Problem Cause Fix
InvalidModelException: does not support image embeddings Using a text-only model for image embedding Use embeddinggemma-2 (or nomic-embed-vision) for image embeddings
Low similarity between obviously similar images Fine detail lost at the default image budget Raise Configuration.DefaultImageDetail before creating the embedder
Text-to-image scores are low Cross-modal scores sit lower than image-to-image scores Rank results instead of applying an image-to-image threshold; use descriptive text
Slow indexing on large folders Processing hundreds of high-resolution images Use batch embedding; run on a GPU
All scores cluster close together Collection contains very similar images Normal for homogeneous collections; use a higher threshold

Next Steps

Share