---
id: clip-benchmark
version: "1.6.2"
license: MIT license
license_treatment: permissive
maintenance: aging
---

# clip-benchmark — CLIP-like models benchmarks on various datasets

License: permissive · Maintenance: aging · Downloads: 449.0K/mo

## What it is and what it does

CLIP Benchmark is a standardized evaluation framework for CLIP-like vision-language models. It provides a command-line interface to benchmark models like OpenCLIP, Japanese CLIP, and NLLB CLIP across multiple datasets and tasks. The package handles zero-shot image classification and retrieval, linear probing, and image captioning, supporting datasets from torchvision, TensorFlow Datasets, and VTAB, as well as multilingual and compositional tasks.

You specify a model, dataset, and task; the framework loads the model, runs inference, and outputs evaluation metrics to JSON. Results can be aggregated into CSV tables for comparison. The package depends on torch, torchvision, transformers, open-clip-torch, scikit-learn, pycocoevalcap, webdataset, and tqdm—a substantial set of deep-learning dependencies that must be installed alongside it.

Use it for:

- Compare zero-shot classification accuracy of different CLIP variants on standard datasets like CIFAR-10 or ImageNet.
- Evaluate multilingual CLIP models (Japanese CLIP, NLLB) on language-specific or cross-lingual retrieval tasks.
- Run linear probing experiments to measure how well frozen CLIP embeddings transfer to downstream tasks.
- Benchmark image captioning models on COCO captions with standard metrics via pycocoevalcap.
- Generate reproducible evaluation tables across multiple models and datasets for research papers or model selection.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Evaluates CLIP-like vision-language models on standard datasets using zero-shot classification, zero-shot retrieval, linear probing, and captioning tasks.

Yes, if you need to systematically evaluate CLIP-like models on standard benchmarks. The low install friction and permissive license make it accessible. However, the aging maintenance status (385 days since release) and Pre-Alpha development status mean you should verify compatibility with your PyTorch and transformers versions before relying on it for production comparisons. No known security vulnerabilities.

## Install

pip install clip-benchmark
uv add clip-benchmark
poetry add clip-benchmark

## Installing clip-benchmark

Before you install: Low install friction with a pure-Python wheel. Maintenance status is aging (385 days since last release), though the package remains functional and receives occasional updates.

License in practice: MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.

Quickstart:

pip install clip-benchmark

clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification \
  --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu \
  --output=result.json --batch_size=64

Requires torch, torchvision, and transformers; evaluation tasks require sufficient GPU memory depending on model size and batch size.

Verify before relying:

- Whether aging maintenance status (385 days since release) affects compatibility with recent PyTorch or transformers versions.
- Support status for Python versions beyond 3.8 (classifiers list only up to 3.8).

## Package facts

- License: MIT license (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: aging
- Downloads: 449.0K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

CLIP model evaluation, vision language benchmark, zero-shot classification, multimodal model testing, image-text retrieval benchmark, CLIP performance comparison, model evaluation framework, vision-language-models, benchmarking, evaluation-framework

[View on SkillFed](https://skillfed.io/packages/clip-benchmark) · [View on PyPI](https://pypi.org/project/clip-benchmark/)