bigquery-schema-generator

What it is and what it does

BigQuery Schema Generator is a command-line tool and Python library that infers BigQuery table schemas by scanning complete datasets. Unlike BigQuery's built-in auto-detection (which examines only the first 500 records), this tool processes all input records to produce a more accurate schema, especially useful when data fields are added organically over time and may not appear in early records.

The tool accepts newline-delimited JSON or CSV format on standard input and outputs a BigQuery-compatible schema in JSON format. It can be used as a standalone command-line utility or imported as a library for programmatic schema generation. It supports various options for handling nulls, quoted values, type inference modes, and schema merging with existing schemas.

Use it for:

• Generate accurate BigQuery schemas for datasets where fields appear sporadically across records, ensuring no data is lost during table creation.
• Pre-validate and generate schemas for CSV or JSON data before bulk loading into BigQuery using the bq load tool.
• Merge new schema information from incremental data exports with existing table schemas to keep them up-to-date.
• Debug and inspect inferred field types and structure from raw data files before committing to a table design.
• Automate schema generation in data pipelines where data format or structure may vary across sources.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Generates BigQuery table schemas from complete JSON or CSV datasets by analyzing all records, rather than relying on BigQuery's auto-detection which examines only the first 500 records.

Yes, with conditions. The tool fills a genuine gap in BigQuery's schema detection workflow. However, dormant maintenance (944 days since last release) and high install friction are concerns. Install if you regularly work with BigQuery data ingestion and need reliable schema inference across complete datasets; avoid if you need active maintenance or have strict dependency constraints.

Install

bigquery-schema-generator on PyPI

pip install bigquery-schema-generator

uv add bigquery-schema-generator

poetry add bigquery-schema-generator

Installing bigquery-schema-generator

Before you install

High install friction: the package is distributed as a source tarball with no runtime dependencies listed, suggesting potential build or environment setup requirements. Maintenance is dormant—last release was 944 days ago, though the repository remains active with a recent commit.

License in practice

Licensed under Apache 2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions; suitable for both open-source and proprietary projects.

Quickstart

pip3 install bigquery-schema-generator
generate-schema < file.data.json > file.schema.json
# Or for CSV:
generate-schema --input_format csv < file.data.csv > file.schema.json

Requires Python 3.6 or higher; the generate-schema command may be installed in different locations depending on your Python environment (e.g., /usr/bin, /usr/local/bin, or ~/.local/bin).

Verify before relying

• Whether the high install friction is due to compiled dependencies, system libraries, or simply source-only distribution.
• Current compatibility with modern BigQuery API versions and schema features.
• Performance characteristics when processing very large datasets.

Package facts

Evidence: bigquery-schema-generator-1.6.1.tar.gz

Tags

More Database packages