pinecone-text

What it is and what it does

Pinecone Text is a utility library that bridges text data and Pinecone's vector search engine by providing encoders that convert documents and queries into sparse or dense vectors. It wraps multiple encoding strategies—BM25 for traditional sparse vectors, SPLADE for learned sparse representations, and integrations with Sentence Transformers and OpenAI's embedding models for dense vectors—allowing developers to prepare text for hybrid search without writing encoding logic themselves.

The package is designed for use with Pinecone's hybrid search, which combines sparse and dense vectors for improved retrieval. It handles tokenization, model loading, and vector formatting, but requires explicit installation of optional dependencies for SPLADE, Sentence Transformers, or OpenAI support. BM25 is available by default and can be initialized with precomputed parameters or fitted to a custom corpus; SPLADE uses a fixed HuggingFace model; dense encoders delegate to external services or local models.

Use it for

• Prepare a corpus of documents for BM25-based sparse vector indexing in Pinecone without implementing tokenization and IDF calculation yourself.
• Encode queries and documents using SPLADE for learned sparse retrieval when BM25 alone is insufficient.
• Generate dense embeddings via OpenAI's API and store them in Pinecone for semantic search without managing API calls directly.
• Combine BM25 sparse vectors with Sentence Transformer dense vectors for hybrid search in a single pipeline.
• Load and reuse precomputed BM25 parameters (fitted on MS MARCO) to encode new documents without retraining.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

pinecone-text on PyPI

Before you install

Low install friction with six runtime dependencies. Maintenance status is aging—last release was 368 days ago—but the package remains functional for current Python versions (3.9–3.11); note that SPLADE and Sentence Transformers encoders have known incompatibilities with Python 3.12 due to PyTorch constraints.

SPLADE and SentenceTransformerEncoder are incompatible with Python 3.12 due to PyTorch compatibility issues; optional extras (splade, dense, openai) require separate installation and their own dependencies.

License in practice

License treatment is unclear; no SPDX or raw license metadata is available in the package metadata. Verify the actual license terms before use in proprietary or restricted contexts.

Quickstart

pip install pinecone-text

from pinecone_text.sparse import BM25Encoder

corpus = ["The quick brown fox", "The lazy dog"]
bm25 = BM25Encoder()
bm25.fit(corpus)
vector = bm25.encode_documents("brown fox")

Package facts

Evidence: pinecone_text-0.11.0-py3-none-any.whl

Tags

More Text Processing packages

See also langchain-pinecone · sentence-transformers · model2vec · pinecone-plugin-interface · llama-index-vector-stores-pinecone · pinecone-plugin-inference · bm25s · floret · voyageai · meilisearch

Further reading