pinecone-text

What it is and what it does

Pinecone Text is a utility library that bridges text data and Pinecone's vector search engine by providing encoders that convert documents and queries into sparse or dense vectors. It wraps multiple encoding strategies—BM25 for traditional sparse vectors, SPLADE for learned sparse representations, and integrations with Sentence Transformers and OpenAI's embedding models for dense vectors—allowing developers to prepare text for hybrid search without writing encoding logic themselves.

The package is designed for use with Pinecone's hybrid search, which combines sparse and dense vectors for improved retrieval. It handles tokenization, model loading, and vector formatting, but requires explicit installation of optional dependencies for SPLADE, Sentence Transformers, or OpenAI support. BM25 is available by default and can be initialized with precomputed parameters or fitted to a custom corpus; SPLADE uses a fixed HuggingFace model; dense encoders delegate to external services or local models.

Use it for:

• Prepare a corpus of documents for BM25-based sparse vector indexing in Pinecone without implementing tokenization and IDF calculation yourself.
• Encode queries and documents using SPLADE for learned sparse retrieval when BM25 alone is insufficient.
• Generate dense embeddings via OpenAI's API and store them in Pinecone for semantic search without managing API calls directly.
• Combine BM25 sparse vectors with Sentence Transformer dense vectors for hybrid search in a single pipeline.
• Load and reuse precomputed BM25 parameters (fitted on MS MARCO) to encode new documents without retraining.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides sparse and dense text encoders for converting documents and queries into vectors compatible with Pinecone's hybrid search, supporting BM25, SPLADE, Sentence Transformers, and OpenAI embeddings.

Yes, with conditions. The package is useful for developers building hybrid search on Pinecone and want to avoid writing encoding logic, but maintenance is aging (368 days since last release) and license terms are unclear. Install if you are committed to Pinecone's platform and can work within Python 3.9–3.11 (avoid 3.12 for SPLADE and Sentence Transformers). Verify the license before use in proprietary contexts. No known security vulnerabilities.

Install

pinecone-text on PyPI

pip install pinecone-text

uv add pinecone-text

poetry add pinecone-text

Installing pinecone-text

Before you install

Low install friction with six runtime dependencies. Maintenance status is aging—last release was 368 days ago—but the package remains functional for current Python versions (3.9–3.11); note that SPLADE and Sentence Transformers encoders have known incompatibilities with Python 3.12 due to PyTorch constraints.

License in practice

License treatment is unclear; no SPDX or raw license metadata is available in the package metadata. Verify the actual license terms before use in proprietary or restricted contexts.

Quickstart

pip install pinecone-text

from pinecone_text.sparse import BM25Encoder

corpus = ["The quick brown fox", "The lazy dog"]
bm25 = BM25Encoder()
bm25.fit(corpus)
vector = bm25.encode_documents("brown fox")

SPLADE and SentenceTransformerEncoder are incompatible with Python 3.12 due to PyTorch compatibility issues; optional extras (splade, dense, openai) require separate installation and their own dependencies.

Verify before relying

• Whether the package is still actively maintained or in maintenance-only mode given the 368-day gap since last release.
• Specific performance characteristics or throughput limits for encoding large document batches.
• Whether BM25's static document frequency model is suitable for your use case or if dynamic retraining is needed.

Package facts

Evidence: pinecone_text-0.11.0-py3-none-any.whl

Tags

More Text Processing packages

Further reading