tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models

tiktoken Permissive license MIT License Copyright (c) 2022 OpenAI, Shantanu Jain Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"),… (full text in the JSON record) Active ★ 18,986 v0.13.0 released 2026-05-15

Install

tiktoken on PyPI

pip

pip install tiktoken

uv

uv add tiktoken

poetry

poetry add tiktoken

Package facts

License	MIT License Copyright (c) 2022 OpenAI, Shantanu Jain Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"),… (full text in the JSON record) (permissive)
Python support	supports the current Python release (>=3.9)
Install friction	medium — platform-specific wheel
Runtime dependencies	2 — regex, requests
Maintenance	actively maintained — 90 days since the last release
Last repo commit	2026-05-24
First released	2022-12-15
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: tiktoken-0.13.0-cp310-cp310-macosx_10_12_x86_64.whl; tiktoken-0.13.0-cp310-cp310-macosx_11_0_arm64.whl; tiktoken-0.13.0-cp310-cp310-manylinux_2_28_aarch64.whl; tiktoken-0.13.0-cp310-cp310-manylinux_2_28_x86_64.whl; tiktoken-0.13.0-cp310-cp310-musllinux_1_2_aarch64.whl; tiktoken-0.13.0-cp310-cp310-musllinux_1_2_x86_64.whl; tiktoken-0.13.0-cp310-cp310-win_amd64.whl; tiktoken-0.13.0-cp311-cp311-macosx_10_12_x86_64.whl; tiktoken-0.13.0-cp311-cp311-macosx_11_0_arm64.whl; tiktoken-0.13.0-cp311-cp311-manylinux_2_28_aarch64.whl; tiktoken-0.13.0-cp311-cp311-manylinux_2_28_x86_64.whl; tiktoken-0.13.0-cp311-cp311-musllinux_1_2_aarch64.whl; tiktoken-0.13.0-cp311-cp311-musllinux_1_2_x86_64.whl; tiktoken-0.13.0-cp311-cp311-win_amd64.whl; tiktoken-0.13.0-cp312-cp312-macosx_10_13_x86_64.whl; tiktoken-0.13.0-cp312-cp312-macosx_11_0_arm64.whl; tiktoken-0.13.0-cp312-cp312-manylinux_2_28_aarch64.whl; tiktoken-0.13.0-cp312-cp312-manylinux_2_28_x86_64.whl; tiktoken-0.13.0-cp312-cp312-musllinux_1_2_aarch64.whl; tiktoken-0.13.0-cp312-cp312-musllinux_1_2_x86_64.whl

About tiktoken

from the package's own PyPI description — quoted content, verbatim

⏳ tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")

The open source version of tiktoken can be installed from PyPI:

pip install tiktoken

The tokeniser API is documented in tiktoken/core.py.

Example code using tiktoken can be found in the OpenAI Cookbook.

Performance

tiktoken is between 3-6x faster than a comparable open source tokeniser:

image (image)

Performance measured on 1GB of text using the GPT-2 tokeniser, using GPT2TokenizerFast from tokenizers==0.13.2, transformers==4.24.0 and tiktoken==0.2.0.

Getting help

Please post questions in the issue tracker.

If you...

Read as markdown · JSON record · Source repository

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

tiktoken is a fast byte-pair encoding tokenizer that converts text into token sequences for use with OpenAI's language models, supporting multiple encoding schemes like o200k_base and gpt-4o.

Medium install friction due to platform-specific compiled wheels across CPython 3.10–3.12 on macOS, Linux, and Windows. Active maintenance with release 90 days ago and strong GitHub presence signal reliable upkeep. Runtime dependencies on regex and requests are lightweight.

MIT License permits unrestricted use, modification, and distribution with minimal restrictions, making it suitable for both open-source and commercial projects.

Usage

pip install tiktoken

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("hello world")
text = enc.decode(tokens)

Requires Python >=3.9; platform-specific wheels are pre-built but installation requires a compatible Python version and architecture.

Verdict: tiktoken is a well-maintained, permissively licensed tokenizer from OpenAI with no known vulnerabilities and top-1000 PyPI popularity. Medium install friction is offset by pre-built wheels and active development. Suitable for production use in token counting and text-to-token conversion workflows.

Needs verification

Whether the educational submodule (tiktoken._educational) is documented or supported for production use
Performance characteristics and memory overhead when tokenizing very large texts or concurrent requests
Specific performance advantage claims (3-6x faster) and their applicability to current versions

openai tokenizerbpe tokenizationtoken countingtext to tokensgpt tokenizerencode decode textlanguage model tokens