dtlpymetrics

What it is and what it does

dtlpymetrics is a scoring and metrics application for the Dataloop platform that automates quality assessment of annotations. It calculates multiple layers of scores—raw annotation scores (geometry overlap via IOU or distance, label matching, and attribute matching), per-annotation overall scores, per-user confusion scores, per-item label confusion counts, and per-item overall scores—across image and video datasets. It supports classification, bounding box, polygon, segmentation, and point annotations.

The package integrates with Dataloop's quality workflows (qualification tasks, honeypot tasks, and consensus tasks) and can be added as custom nodes to pipelines to compute scores automatically when quality items are completed. For videos, scores are calculated frame-by-frame and then aggregated per annotation; confusion scores are omitted for videos due to their multi-frame nature. Any calculated scores replace previous scores for all items in a task.

Use it for:

• Measure inter-annotator agreement in consensus tasks by comparing each annotator's annotations against all others and generating confusion matrices.
• Validate annotator quality in honeypot and qualification tasks by scoring their annotations against ground truth and tracking user confusion scores over time.
• Automate quality scoring in image annotation pipelines by adding scoring nodes that trigger when quality task assignments are completed.
• Assess label confusion for specific classes to identify which labels annotators confuse most often, guiding retraining or clarification.
• Evaluate geometry accuracy (bounding box overlap, polygon IOU, point distance) alongside label correctness to identify systematic annotation errors.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Calculates and generates quality scores for annotations in image and video datasets, measuring agreement between annotators and comparing against reference annotations using geometry, label, and attribute matching.

Yes, if you use Dataloop for annotation workflows and need automated quality scoring. The package is actively maintained, has low install friction, and fills a specific role in Dataloop's quality assurance ecosystem. However, verify the license before use in proprietary contexts, and confirm that dtlpy is available and configured in your environment.

Install

dtlpymetrics on PyPI

pip install dtlpymetrics

uv add dtlpymetrics

poetry add dtlpymetrics

Installing dtlpymetrics

Before you install

Low install friction with a pure-Python wheel. Actively maintained as of February 2026. Depends on dtlpy, shapely, matplotlib, and seaborn—all stable, widely-used libraries.

License in practice

License status is unclear; no SPDX identifier or raw license text is available in the package metadata. Verify the actual license before using in proprietary or copyleft-sensitive contexts.

Quickstart

pip install dtlpymetrics

import dtlpymetrics
# Use functions to calculate scores for quality tasks and model predictions
# See docs/dtlpymetrics_fxns.md for available functions

Requires dtlpy, which is Dataloop's Python SDK; you must have a Dataloop account and environment to use this package meaningfully.

Verify before relying

• Whether the package requires specific versions of dtlpy, shapely, matplotlib, or seaborn beyond what pip resolves.
• Whether confusion scores are actually calculated for video items, despite the note that they are not due to multi-frame nature.
• Performance characteristics when scoring large datasets or high-resolution videos.

Package facts

Evidence: dtlpymetrics-1.2.32-py3-none-any.whl

Tags

More Quality Assurance packages

Further reading