math-verify

What it is and what it does

Math-Verify is a mathematical expression evaluator built to assess LLM outputs on math tasks. It solves the problem that existing evaluators often fail due to strict format requirements, limited parsing, or inflexible comparison logic—sometimes underestimating model performance significantly. The package works in three stages: extracting answers from model output using regex patterns with configurable priority, parsing extracted text (including LaTeX and plain expressions) into a common SymPy representation while normalizing malformations and handling special cases like percentages and complex numbers, and finally comparing the parsed answer to a gold standard using both symbolic and numerical methods.

The parser supports three extraction targets (LaTeX, plain expressions, and literal strings) and handles a wide range of mathematical constructs: set theory, intervals, equations, inequalities, matrices, complex numbers, and relations. Comparison is intelligent—it recognizes equivalent forms (e.g., a + b equals b + a), handles precision-based numeric equality, and includes special logic for sets, intervals, and flipped inequalities. The package is designed for researchers and developers evaluating LLM math reasoning, with optional command-line tools for batch evaluation on datasets like MATH-Hard, MATH-500, GSM8K, AMC23, and AIME24.

Use it for:

• Evaluate LLM outputs on standardized math benchmarks (MATH, GSM8K, AMC, AIME) to measure model reasoning accuracy.
• Grade student or model answers in educational platforms where answers may be written in multiple equivalent forms.
• Batch-process CSV files of model predictions and gold answers to identify systematic evaluation errors in existing pipelines.
• Extract and normalize mathematical answers from unstructured LLM outputs before feeding them to downstream grading systems.
• Compare symbolic expressions for equivalence in automated homework or quiz systems that accept multiple valid forms.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Evaluates mathematical expressions from LLM outputs by parsing LaTeX and plain expressions, converting them to a common symbolic form, and comparing them against gold-standard answers for equivalence.

Yes, with conditions. Install if you need to evaluate LLM math outputs and can work with Python >= 3.10; the package has low install friction, no known vulnerabilities, and outperforms existing evaluators on MATH dataset. Do not install if you require ongoing active development—the package is aging (last commit 2026-01-10) and may not receive rapid updates for new edge cases or datasets.

Install

math-verify on PyPI

pip install math-verify

uv add math-verify

poetry add math-verify

Installing math-verify

Before you install

Low friction: pure Python wheel with a single runtime dependency (latex2sympy2_extended). Maintenance status is aging—last commit 2026-01-10—but the package is actively used (top 5000 on PyPI) and carries no known vulnerabilities.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects that can accommodate attribution.

Quickstart

pip install 'math-verify[antlr4_13_2]'

from math_verify import parse, verify

gold = parse("${1,3} \\cup {2,4}$")
answer = parse("${1,2,3,4}$")
verify(gold, answer)  # Returns True or False

Requires Python >= 3.10. Must specify an antlr4 runtime extra (antlr4_13_2, antlr4_11_0, or antlr4_9_3) during installation to avoid potential runtime issues.

Verify before relying

• Whether the 0.1328 MATH dataset score remains current or if newer evaluations exist.
• Performance characteristics (speed, memory) when processing large batches of answers or complex symbolic expressions.
• Stability and compatibility of the three supported antlr4 runtime versions in production environments.

Package facts

Evidence: math_verify-0.9.0-py3-none-any.whl

Keywords: verification, math, evaluation

Tags

More Artificial Intelligence packages

Further reading