spark-nlp

What it is and what it does

Spark NLP is a natural language processing library built on top of Apache Spark that brings NLP and machine learning to distributed environments. It provides access to pretrained pipelines and models across multiple languages, supporting tasks like tokenization, part-of-speech tagging, named entity recognition, sentiment analysis, machine translation, question answering, and text generation. The library integrates state-of-the-art transformer models and can import models from TensorFlow, ONNX, and OpenVINO frameworks.

The package is designed for production use and scales across distributed Spark clusters. It supports multiple programming languages through the JVM ecosystem and offers specialized variants for GPU acceleration, Apple Silicon, and AArch64 architectures. The library requires Java 8 or 11 and Apache Spark 3.0 or later.

Use it for:

• Build production NLP pipelines that scale across Spark clusters for large-scale text processing
• Extract named entities, perform sentiment analysis, or classify documents using pretrained models
• Translate text between languages or generate summaries and answers from documents
• Integrate transformer models into Spark ML workflows for feature engineering
• Process multilingual text data in a single distributed pipeline

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Spark NLP provides distributed natural language processing on Apache Spark, offering pretrained pipelines and models for tokenization, named entity recognition, sentiment analysis, machine translation, and embeddings across multiple languages.

Yes, if you need production-grade NLP at scale on Spark. The library is actively maintained with no security vulnerabilities and offers a comprehensive suite of pretrained models and tasks. Install only if you already have Apache Spark 3.0+ and Java 8 or 11 in your environment; it is not suitable for lightweight single-machine NLP work.

Install

spark-nlp on PyPI

pip install spark-nlp

uv add spark-nlp

poetry add spark-nlp

Installing spark-nlp

Before you install

Installation is straightforward (low friction) with no runtime dependencies to manage. The package is actively maintained with a recent release 51 days ago. Requires Java 8 or 11 and Apache Spark 3.0+ to be present in your environment.

License in practice

Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions.

Quickstart

pip install spark-nlp==6.4.2

from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline('explain_document_dl', lang='en')
result = pipeline.annotate("The Mona Lisa is a 16th century oil painting created by Leonardo.")

Requires Java 8 or 11 (Oracle or OpenJDK) and Apache Spark 3.0+ to be installed and configured in your environment.

Verify before relying

• Whether pretrained model downloads are automatic or require separate setup steps
• Memory and disk requirements for downloading and caching pretrained models
• Performance characteristics on single-machine vs. true distributed clusters
• Exact Python version support (classifiers list 3.6–3.9 but requires_python is unspecified)

Package facts

Evidence: spark_nlp-6.4.2-py2.py3-none-any.whl

Keywords: NLP, spark, vision, speech, deep, learning, transformer, tensorflow, BERT, GPT-2, Wav2Vec2, ViT

Tags

More Scientific/Engineering packages