---
id: tiktoken
version: "0.13.0"
license: MIT License Copyright (c) 2022 OpenAI, Shantanu Jain Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"),… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# tiktoken — tiktoken is a fast BPE tokeniser for use with OpenAI's models

License: permissive · Maintenance: active · Popularity: top 1,000 on PyPI

## Install

pip install tiktoken
uv add tiktoken
poetry add tiktoken

## Description

# ⏳ tiktoken

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
OpenAI's models.

```python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
```

The open source version of `tiktoken` can be installed from [PyPI](https://pypi.org/project/tiktoken):
```
pip install tiktoken
```

The tokeniser API is documented in `tiktoken/core.py`.

Example code using `tiktoken` can be found in the
[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).


## Performance

`tiktoken` is between 3-6x faster than a comparable open source tokeniser:

![image](https://raw.githubusercontent.com/openai/tiktoken/main/perf.svg)

Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.


## Getting help

Please post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).

If you...

## AI interpretation — verify before relying

tiktoken is a fast byte-pair encoding tokenizer that converts text into token sequences for use with OpenAI's language models, supporting multiple encoding schemes like o200k_base and gpt-4o.

Verdict: tiktoken is a well-maintained, permissively licensed tokenizer from OpenAI with no known vulnerabilities and top-1000 PyPI popularity. Medium install friction is offset by pre-built wheels and active development. Suitable for production use in token counting and text-to-token conversion workflows.

[View on SkillFed](https://skillfed.io/packages/tiktoken) · [View on PyPI](https://pypi.org/project/tiktoken/)