{"categories":[{"label":"Quality Assurance","url":"https://skillfed.io/packages/category/software-development-quality-assurance/2"}],"enrichment":{"capability":"Calculates and generates quality scores for annotations in image and video datasets, measuring agreement between annotators and comparing against reference annotations using geometry, label, and attribute matching.","skillfed_tags":["annotation-quality","dataloop-integration","metrics"],"use_cases":["Measure inter-annotator agreement in consensus tasks by comparing each annotator's annotations against all others and generating confusion matrices.","Validate annotator quality in honeypot and qualification tasks by scoring their annotations against ground truth and tracking user confusion scores over time.","Automate quality scoring in image annotation pipelines by adding scoring nodes that trigger when quality task assignments are completed.","Assess label confusion for specific classes to identify which labels annotators confuse most often, guiding retraining or clarification.","Evaluate geometry accuracy (bounding box overlap, polygon IOU, point distance) alongside label correctness to identify systematic annotation errors."],"what_it_does":"dtlpymetrics is a scoring and metrics application for the Dataloop platform that automates quality assessment of annotations. It calculates multiple layers of scores\u2014raw annotation scores (geometry overlap via IOU or distance, label matching, and attribute matching), per-annotation overall scores, per-user confusion scores, per-item label confusion counts, and per-item overall scores\u2014across image and video datasets. It supports classification, bounding box, polygon, segmentation, and point annotations.\n\nThe package integrates with Dataloop's quality workflows (qualification tasks, honeypot tasks, and consensus tasks) and can be added as custom nodes to pipelines to compute scores automatically when quality items are completed. For videos, scores are calculated frame-by-frame and then aggregated per annotation; confusion scores are omitted for videos due to their multi-frame nature. Any calculated scores replace previous scores for all items in a task.","worth_installing":"Yes, if you use Dataloop for annotation workflows and need automated quality scoring. The package is actively maintained, has low install friction, and fills a specific role in Dataloop's quality assurance ecosystem. However, verify the license before use in proprietary contexts, and confirm that dtlpy is available and configured in your environment."},"id":"dtlpymetrics","links":{"html":"https://skillfed.io/packages/dtlpymetrics","md":"https://skillfed.io/packages/dtlpymetrics.md","pypi":"https://pypi.org/project/dtlpymetrics/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-02-26","license_spdx":null,"license_treatment":"unclear","name":"dtlpymetrics","python_support":"unspecified","summary":"Scoring and metrics app"},"popularity":{"monthly_downloads":506019,"position":6290,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.2.32"}