Vision Embedding & Tagging API (sb-vision)

The sb-vision crate provides local vision embedding and image tagging for StudioBrain. It wraps BLIP-2, CLIP, and ViT models via the candle ML framework (preferred) with a tract-onnx fallback for mobile CPU paths.

Inputs are pre-thumbnailed to 512×512 by sb-thumbnails; this crate re-samples internally to the model-specific size.

Public API

Three functions form the public API, consumed by desktop, mobile (Tauri), cloud AI services, ComfyUI integration, and the manifest indexer:

use image::DynamicImage;
 
/// Compute a vision embedding for an image.
pub fn embed(image: &DynamicImage) -> anyhow::Result<Vec<f32>>;
 
/// Produce string tags for an image (descending confidence order).
pub fn tag(image: &DynamicImage) -> anyhow::Result<Vec<String>>;
 
/// Compute embedding and tags in a single call (more efficient).
pub fn embed_and_tag(image: &DynamicImage) -> anyhow::Result<(Vec<f32>, Vec<String>)>;

Return Types

Models

Florence-2 Status

Florence-2 support is stubbed in this release. candle-transformers 0.9 does not yet ship Florence-2’s custom attention implementation. Two paths forward:

1. candle path — unblocked when candle-transformers ships Florence-2.
2. ONNX path — florence2-tract feature, requires exporting the model to ONNX and pushing to GHCR artifact registry.

Feature Flags

The crate ships with zero native dependencies by default. Enable a backend feature for real inference.

Quick Start

use image::DynamicImage;
 
// Open an image (sb-thumbnails handles the 512x512 pre-thumbnail).
let img = image::open("asset.jpg").unwrap();
 
// Get a 768-dimensional BLIP embedding.
let embedding = sb_vision::embed(&img).unwrap();
 
// Get ImageNet-1K tags (e.g., ["mountain", "landscape", "outdoor"]).
let tags = sb_vision::tag(&img).unwrap();
 
// Or get both in one call (shares preprocessing).
let (embedding, tags) = sb_vision::embed_and_tag(&img).unwrap();

Model Cache

Models are downloaded from HuggingFace Hub and cached on disk:

Cache layout:

~/.cache/sb-vision/
  Salesforce_blip-image-captioning-large/
    model.safetensors
    tokenizer.json
    config.json

Environment Variables

Python Bindings (PyO3)

Enable the pyo3 feature for Python consumption by the cloud AI service:

from sb_vision import embed_from_path, tag_from_path, embed_and_tag_from_path
 
embedding = embed_from_path("asset.jpg")
tags = tag_from_path("asset.jpg")
embedding, tags = embed_and_tag_from_path("asset.jpg")

Backfill Helpers

For assets ingested before real vision embeddings were available (SBAI-2199), the crate ships helpers to identify stale rows:

use sb_vision::embed_tag::{is_zero_embedding, needs_backfill, CLIP_EMBED_DIM, BLIP_EMBED_DIM};
 
// Detect all-zero placeholder vectors.
assert!(is_zero_embedding(&[0.0; 512]));
 
// Check if an embedding needs regeneration.
assert!(needs_backfill(None, CLIP_EMBED_DIM));         // missing
assert!(needs_backfill(Some(&zeros), CLIP_EMBED_DIM)); // all-zero
assert!(needs_backfill(Some(&[0.0; 1]), CLIP_EMBED_DIM)); // wrong dim

Error Handling

All API functions return anyhow::Result. The structured VisionError enum is available for callers that need to branch on failure mode:

Backend Selection Order

When multiple backend features are enabled, the active backend is determined at compile time:

1. candle-cpu / candle-cuda / candle-metal — BLIP/ViT via candle (preferred)
2. onnx-clip — CLIP ViT-B/32 via ONNX Runtime (embedding-only)
3. onnx — tract-onnx CPU fallback (mobile stub)
4. No feature — VisionError::FeatureDisabled

Consumers