pytorch-pretrained-bert

What it is and what it does

This package provides op-for-op PyTorch reimplementations of four major pre-trained transformer models: BERT, OpenAI GPT, Transformer-XL, and GPT-2. It includes pre-trained weights converted from the original TensorFlow checkpoints and a suite of task-specific model heads for common NLP workflows like sequence classification, token classification, question answering, and masked language modeling.

The package is built on torch, numpy, boto3, requests, tqdm, and regex. It is designed for developers who want to fine-tune or adapt these models for downstream NLP tasks without reimplementing the transformer architecture. The repository includes examples, notebooks, and a command-line interface for checkpoint conversion, though the package itself has not seen a new release since April 2019.

Use it for:

• Fine-tune BERT for text classification tasks like sentiment analysis or intent detection.
• Use pre-trained GPT or GPT-2 for language generation and completion tasks.
• Adapt Transformer-XL for long-sequence modeling and perplexity evaluation on language corpora.
• Perform question answering with BertForQuestionAnswering on SQuAD-like datasets.
• Extract contextual embeddings from BERT for downstream machine learning pipelines.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides PyTorch implementations of BERT, GPT, Transformer-XL, and GPT-2 models with pre-trained weights and fine-tuning examples for natural language processing tasks.

Yes, if you need to work with BERT, GPT, or Transformer-XL in PyTorch and are comfortable with a package that has not received updates since April 2019. The implementation is well-tested, has low install friction, and carries permissive licensing. However, consider whether a more actively maintained alternative (such as HuggingFace Transformers, which this package predates) better suits your project's long-term maintenance needs.

Install

pytorch-pretrained-bert on PyPI

pip install pytorch-pretrained-bert

uv add pytorch-pretrained-bert

poetry add pytorch-pretrained-bert

Installing pytorch-pretrained-bert

Before you install

Low install friction with pure Python wheels for both Python 2 and 3. Active maintenance with recent commits; however, the package has not received a release since April 2019 despite ongoing repository activity, suggesting it may be in maintenance mode rather than active development.

License in practice

Licensed under Apache (permissive), allowing commercial and private use with minimal restrictions. You may use, modify, and distribute the code freely provided you include a copy of the license and notice of changes.

Quickstart

pip install pytorch-pretrained-bert

from pytorch_pretrained_bert import BertModel, BertTokenizer

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encoded = tokenizer.encode('Hello world')
output = model(encoded)

Requires PyTorch 0.4.1 or 1.0.0+; torch must be installed separately. Optional: ftfy and SpaCy for original OpenAI GPT tokenization (ftfy pinned to 4.4.3 for Python 2).

Verify before relying

• Whether pre-trained model weights are still downloadable from the original sources (Google, OpenAI) given the package's age.
• Compatibility with modern PyTorch versions beyond 1.0.0.
• Performance parity claims (e.g., ~91 F1 on SQuAD) on current datasets and hardware.

Package facts

Evidence: pytorch_pretrained_bert-0.6.2-py2-none-any.whl; pytorch_pretrained_bert-0.6.2-py3-none-any.whl

Keywords: BERT, NLP, deep, learning, google

Tags

More Artificial Intelligence packages

Further reading