{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/5"}],"enrichment":{"capability":"Evaluates CLIP-like vision-language models on standard datasets using zero-shot classification, zero-shot retrieval, linear probing, and captioning tasks.","skillfed_tags":["vision-language-models","benchmarking","evaluation-framework"],"use_cases":["Compare zero-shot classification accuracy of different CLIP variants on standard datasets like CIFAR-10 or ImageNet.","Evaluate multilingual CLIP models (Japanese CLIP, NLLB) on language-specific or cross-lingual retrieval tasks.","Run linear probing experiments to measure how well frozen CLIP embeddings transfer to downstream tasks.","Benchmark image captioning models on COCO captions with standard metrics via pycocoevalcap.","Generate reproducible evaluation tables across multiple models and datasets for research papers or model selection."],"what_it_does":"CLIP Benchmark is a standardized evaluation framework for CLIP-like vision-language models. It provides a command-line interface to benchmark models like OpenCLIP, Japanese CLIP, and NLLB CLIP across multiple datasets and tasks. The package handles zero-shot image classification and retrieval, linear probing, and image captioning, supporting datasets from torchvision, TensorFlow Datasets, and VTAB, as well as multilingual and compositional tasks.\n\nYou specify a model, dataset, and task; the framework loads the model, runs inference, and outputs evaluation metrics to JSON. Results can be aggregated into CSV tables for comparison. The package depends on torch, torchvision, transformers, open-clip-torch, scikit-learn, pycocoevalcap, webdataset, and tqdm\u2014a substantial set of deep-learning dependencies that must be installed alongside it.","worth_installing":"Yes, if you need to systematically evaluate CLIP-like models on standard benchmarks. The low install friction and permissive license make it accessible. However, the aging maintenance status (385 days since release) and Pre-Alpha development status mean you should verify compatibility with your PyTorch and transformers versions before relying on it for production comparisons. No known security vulnerabilities."},"id":"clip-benchmark","links":{"html":"https://skillfed.io/packages/clip-benchmark","md":"https://skillfed.io/packages/clip-benchmark.md","pypi":"https://pypi.org/project/clip-benchmark/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-07-25","license_spdx":null,"license_treatment":"permissive","name":"clip-benchmark","python_support":"supports_current","summary":"CLIP-like models benchmarks on various datasets"},"popularity":{"monthly_downloads":449044,"position":6598,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.6.2"}