---
id: spark-nlp
version: "6.4.2"
license: unclear
license_treatment: permissive
maintenance: active
---

# spark-nlp — John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.

License: permissive · Maintenance: active · Downloads: 1.2M/mo

## What it is and what it does

Spark NLP is a natural language processing library built on top of Apache Spark that brings NLP and machine learning to distributed environments. It provides access to pretrained pipelines and models across multiple languages, supporting tasks like tokenization, part-of-speech tagging, named entity recognition, sentiment analysis, machine translation, question answering, and text generation. The library integrates state-of-the-art transformer models and can import models from TensorFlow, ONNX, and OpenVINO frameworks.

The package is designed for production use and scales across distributed Spark clusters. It supports multiple programming languages through the JVM ecosystem and offers specialized variants for GPU acceleration, Apple Silicon, and AArch64 architectures. The library requires Java 8 or 11 and Apache Spark 3.0 or later.

Use it for:

- Build production NLP pipelines that scale across Spark clusters for large-scale text processing
- Extract named entities, perform sentiment analysis, or classify documents using pretrained models
- Translate text between languages or generate summaries and answers from documents
- Integrate transformer models into Spark ML workflows for feature engineering
- Process multilingual text data in a single distributed pipeline

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Spark NLP provides distributed natural language processing on Apache Spark, offering pretrained pipelines and models for tokenization, named entity recognition, sentiment analysis, machine translation, and embeddings across multiple languages.

Yes, if you need production-grade NLP at scale on Spark. The library is actively maintained with no security vulnerabilities and offers a comprehensive suite of pretrained models and tasks. Install only if you already have Apache Spark 3.0+ and Java 8 or 11 in your environment; it is not suitable for lightweight single-machine NLP work.

## Install

pip install spark-nlp
uv add spark-nlp
poetry add spark-nlp

## Installing spark-nlp

Before you install: Installation is straightforward (low friction) with no runtime dependencies to manage. The package is actively maintained with a recent release 51 days ago. Requires Java 8 or 11 and Apache Spark 3.0+ to be present in your environment.

License in practice: Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions.

Quickstart:

pip install spark-nlp==6.4.2

from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline('explain_document_dl', lang='en')
result = pipeline.annotate("The Mona Lisa is a 16th century oil painting created by Leonardo.")

Requires Java 8 or 11 (Oracle or OpenJDK) and Apache Spark 3.0+ to be installed and configured in your environment.

Verify before relying:

- Whether pretrained model downloads are automatic or require separate setup steps
- Memory and disk requirements for downloading and caching pretrained models
- Performance characteristics on single-machine vs. true distributed clusters
- Exact Python version support (classifiers list 3.6–3.9 but requires_python is unspecified)

## Package facts

- License: not declared (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: active
- Downloads: 1.2M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

distributed NLP on Spark, pretrained NLP pipelines, named entity recognition, sentiment analysis at scale, machine translation, text embeddings, transformer models for NLP, distributed-nlp, spark-ml, transformers

[View on SkillFed](https://skillfed.io/packages/spark-nlp) · [View on PyPI](https://pypi.org/project/spark-nlp/)