clip-benchmark

What it is and what it does

CLIP Benchmark is a standardized evaluation framework for CLIP-like vision-language models. It provides a command-line interface to benchmark models like OpenCLIP, Japanese CLIP, and NLLB CLIP across multiple datasets and tasks. The package handles zero-shot image classification and retrieval, linear probing, and image captioning, supporting datasets from torchvision, TensorFlow Datasets, and VTAB, as well as multilingual and compositional tasks.

You specify a model, dataset, and task; the framework loads the model, runs inference, and outputs evaluation metrics to JSON. Results can be aggregated into CSV tables for comparison. The package depends on torch, torchvision, transformers, open-clip-torch, scikit-learn, pycocoevalcap, webdataset, and tqdm—a substantial set of deep-learning dependencies that must be installed alongside it.

Use it for:

• Compare zero-shot classification accuracy of different CLIP variants on standard datasets like CIFAR-10 or ImageNet.
• Evaluate multilingual CLIP models (Japanese CLIP, NLLB) on language-specific or cross-lingual retrieval tasks.
• Run linear probing experiments to measure how well frozen CLIP embeddings transfer to downstream tasks.
• Benchmark image captioning models on COCO captions with standard metrics via pycocoevalcap.
• Generate reproducible evaluation tables across multiple models and datasets for research papers or model selection.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Evaluates CLIP-like vision-language models on standard datasets using zero-shot classification, zero-shot retrieval, linear probing, and captioning tasks.

Yes, if you need to systematically evaluate CLIP-like models on standard benchmarks. The low install friction and permissive license make it accessible. However, the aging maintenance status (385 days since release) and Pre-Alpha development status mean you should verify compatibility with your PyTorch and transformers versions before relying on it for production comparisons. No known security vulnerabilities.

Install

clip-benchmark on PyPI

pip install clip-benchmark

uv add clip-benchmark

poetry add clip-benchmark

Installing clip-benchmark

Before you install

Low install friction with a pure-Python wheel. Maintenance status is aging (385 days since last release), though the package remains functional and receives occasional updates.

License in practice

MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.

Quickstart

pip install clip-benchmark

clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification \
  --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu \
  --output=result.json --batch_size=64

Requires torch, torchvision, and transformers; evaluation tasks require sufficient GPU memory depending on model size and batch size.

Verify before relying

• Whether aging maintenance status (385 days since release) affects compatibility with recent PyTorch or transformers versions.
• Support status for Python versions beyond 3.8 (classifiers list only up to 3.8).

Package facts

Evidence: clip_benchmark-1.6.2-py2.py3-none-any.whl

Keywords: clip_benchmark

Tags

More Artificial Intelligence packages

Further reading