NLP Pipelines: From Embeddings to Entity Extraction

You run the notebook. The NER model correctly extracts all 12 entities from your test paragraph. Sentiment analysis returns sensible scores. The embedding search finds the right document on the first try. You push the code to a feature branch and open the PR with a comment: “NLP pipeline ready for integration.”

Then real data arrives. A customer support ticket contains smart quotes that your tokenizer maps to unknown tokens. A product description in Vietnamese slips through your English-only language detector and produces garbage embeddings. A batch job that processed 500 documents in the notebook now needs to process 2 million, and your embedding generation code calls the model one document at a time. The named entity recognizer that hit 91% F1 on your test set misses half the company names in financial filings because they contain abbreviations it never trained on. Your “ready for integration” PR sits there, mocking you.

The gap between notebook NLP and production NLP is not model quality. It is every piece of engineering around the model that the notebook never forced you to build. And that gap is enormous.

Text Preprocessing Is Where Production Pipelines Diverge

The model gets all the attention. The preprocessing gets none. This is the wrong priority. In production, preprocessing failures cause more silent accuracy degradation than model choice.

Start with encoding normalization. Unicode has multiple representations of the same character. The string “cafe” with a combining acute accent and “caf\u00e9” with a precomposed character look identical to humans but produce different token sequences. Run unicodedata.normalize('NFC', text) on every input before tokenization. Skip this step and your embedding space treats visually identical strings as different documents. At corpus scale, this creates retrieval failures that are nearly impossible to debug because the search query and the matching document look the same in every log. You will lose hours to this.

Language detection is the next silent killer. If your corpus is 95% English, the 5% of documents in other languages will produce embeddings that cluster unpredictably. fastText’s lid.176.bin model identifies 176 languages in under 1ms per document. Set a confidence threshold at 0.7. Route documents below threshold to a review queue rather than embedding them with the wrong language model. The alternative is discovering six months later that your search results are polluted by garbage embeddings from misclassified documents. This pattern breaks regularly in production.

Tokenization consistency matters more than tokenizer choice. If your training data was tokenized with the model’s default tokenizer but your inference pipeline applies custom preprocessing that changes whitespace handling, you have introduced a training-serving skew that degrades accuracy without any visible error. No exceptions to this rule: the tokenizer in production must be byte-for-byte identical to the tokenizer used during training or fine-tuning. Serialize it. Version it. Test it.

With clean inputs secured, the next question is how you generate embeddings without blowing your latency budget.

Embedding Generation Architecture

The architectural decision that shapes everything downstream is whether embeddings are generated in the request path or in a batch pipeline. This is not a theoretical trade-off. It determines your latency budget, infrastructure cost, and failure modes.

Batch embedding is for your corpus. Process documents in batches of 256-512 on GPU. At this batch size, a single A10G processes roughly 1,000 documents per second with a 768-dimensional model. The critical detail: batch processing must use the exact same preprocessing pipeline as real-time query embedding. Any divergence means your query vectors live in a subtly different space than your corpus vectors, and cosine similarity comparisons silently degrade.

Real-time embedding is for queries. A user’s search query or input text needs to be embedded at request time. With a distilled model like all-MiniLM-L6-v2, this takes 8-12ms on GPU. The latency budget for the entire NLP stack in a synchronous request path is typically 50-150ms. That includes preprocessing, embedding, vector search, and any post-processing. Exceed 200ms and you’re better off pre-computing results asynchronously.

Dimensionality trade-offs are real and underappreciated. OpenAI’s text-embedding-3-large produces 3072-dimensional vectors. Each vector consumes 12KB at float32. A million-document corpus at 3072 dimensions requires roughly 12GB of vector storage before indexing overhead. The same corpus at 768 dimensions needs 3GB. Query latency scales with dimensionality because distance computation is O(d). Matryoshka embeddings let you truncate dimensions at query time, losing under 2% recall going from 3072 to 768 on most benchmarks. Test on your actual data before committing to maximum dimensionality. The default is almost certainly overkill for your use case.

Named Entity Recognition in Production

NER in the notebook is a solved problem. NER in production is a systems engineering problem, and model selection is the least interesting variable.

spaCy is the pragmatic default for high-throughput pipelines. The en_core_web_lg model runs on CPU at sub-5ms per document and handles the standard entity types (PERSON, ORG, GPE, DATE) at 86-89% F1. It ships as a pip-installable package with no external dependencies. For pipelines processing thousands of documents per second, spaCy’s speed advantage is decisive. Start here unless you have a strong reason not to.

Hugging Face transformers win when accuracy on specific entity types justifies the infrastructure cost. A fine-tuned roberta-base model for domain-specific NER (extracting drug names from clinical text, ticker symbols from financial filings) hits 92-95% F1 on those entity types. But it requires GPU, adds 20-60ms per document, and introduces the complexity of model serving infrastructure (Triton, TorchServe, or a custom FastAPI wrapper with batched inference). That 5% F1 improvement costs you 10x in compute. Make sure it is worth it.

Cloud APIs (Google Cloud NLP, AWS Comprehend, Azure Text Analytics) are the right choice when you need entity extraction but do not have the team to run model infrastructure. The trade-off is latency (50-200ms per API call including network), cost at scale, and the inability to fine-tune for domain-specific entity types. For prototyping and low-throughput use cases, they are the fastest path to production.

But here is the production concern that none of these model comparisons address: entity consistency. When the same entity appears in different forms across documents (“JP Morgan”, “JPMorgan Chase”, “J.P. Morgan & Co.”), the NER model will correctly extract each occurrence. But downstream systems need to know these are the same entity. Entity linking or normalization (mapping extracted entities to a canonical knowledge base) is where most production NER pipelines underinvest. Without it, your entity extraction produces accurate noise. Every entity found. None of them connected.

Document Classification and Sentiment at Scale

Document classification in production is rarely a single-label problem. A customer support ticket is simultaneously about “billing,” “account access,” and “service cancellation.” Multi-label classification with calibrated confidence scores gives downstream systems something they can actually route on.

The practical architecture: use a fine-tuned distilbert-base-uncased as the classifier. It is 60% smaller than BERT with 97% of the accuracy on most classification benchmarks. Set confidence thresholds per label based on the cost of misclassification. A “fraud” label with 0.6 confidence should route to human review. A “general inquiry” label with 0.6 confidence can auto-route without review. These thresholds are business decisions, not model parameters. Get your product team to own them.

Sentiment analysis deserves a harder look than most teams give it. Binary positive/negative sentiment is rarely useful. Nobody makes real decisions on it. The version that actually drives action is aspect-based sentiment: “The product quality is excellent but the shipping was terrible.” Extracting sentiment per aspect requires either a fine-tuned model trained on aspect-annotated data or a two-stage pipeline (extract aspects first, classify sentiment per aspect second). The two-stage approach is more debuggable in production because you can inspect which aspects were detected before seeing how they were scored.

Text Chunking for LLM Consumption

Every RAG architecture depends on chunking, and most chunking strategies are wrong in ways that only surface at scale.

Fixed-size character splitting is the default in every tutorial. It is also the worst strategy for anything beyond a demo. Do not use it. It cuts sentences mid-word, splits paragraphs across chunks, and creates chunks where the first half is about one topic and the second half is about another. The embedding for that chunk represents the average of two unrelated topics, which means it is a poor match for queries about either one.

Semantic chunking splits at meaning boundaries. The simplest version uses sentence embeddings: embed each sentence, compute cosine similarity between consecutive sentences, and split where similarity drops below a threshold (0.3-0.5 works for most corpora). This produces chunks that are topically coherent. The cost is an additional embedding pass during ingestion. Worth it.

Document-aware chunking uses the document’s own structure. Split at heading boundaries, paragraph breaks, or section markers. For HTML content, the DOM structure provides natural chunk boundaries. For PDFs, layout analysis tools like unstructured.io or pymupdf4llm extract structural elements. This approach produces chunks that respect the author’s organizational intent. Building robust data engineering pipelines for text processing means investing in these structural parsing capabilities early.

Either approach dramatically outperforms naive splitting. But your embeddings will change over time, and that creates its own problem.

The Embedding Refresh Problem

Your embedding model will change. Your chunking strategy will improve. Your preprocessing pipeline will get a bug fix that subtly changes tokenization output. Every one of these changes invalidates your existing embeddings. This is not a hypothetical. It will happen.

The naive approach is to re-embed everything whenever anything changes. For a million-document corpus, this takes 4-8 hours of GPU time and requires a strategy for serving queries during the re-indexing window. The practical approach is blue-green indexing: build the new index in parallel, swap the alias once the new index is complete and validated, and keep the old index available for rollback.

Version your embeddings. Store the model identifier, preprocessing pipeline version, and chunking parameters as metadata on every vector. When you query, filter by the current version. This lets you incrementally migrate to a new embedding generation without a hard cutover and without mixing vectors from incompatible models in the same search results.

The hardest part of the embedding refresh problem is knowing when you need to do it. A model change is obvious. A subtle bug fix in your Unicode normalization that affects 0.3% of documents is not. The answer is a continuous evaluation pipeline: a golden set of query-document pairs where you know the correct retrieval result. Run this evaluation after every pipeline deployment. If recall drops, investigate before the degradation compounds. Do not wait for user complaints. By then, the damage is done.

Evaluation That Goes Beyond F1

F1 is the metric everyone reports and the metric that hides the most production-relevant failures.

A model with 90% F1 on NER might achieve that by correctly extracting PERSON and DATE entities at 95%+ while completely failing on ORG entities that happen to be rare in the test set. Per-type F1 breakdown is the minimum. Report F1 separately for every entity type, weighted by the downstream importance of that type to your application. Aggregate F1 is a vanity metric. Per-type F1 is actionable.

Partial match rate is the metric F1 misses entirely. The model extracts “Morgan” instead of “JPMorgan Chase.” That is a partial match. F1 counts it as a complete miss. But in production, a partial extraction is often worse than no extraction because downstream systems receive confident but incomplete data. They act on it.

Latency P99 matters as much as accuracy for synchronous NLP. A model that is 3% more accurate but has 5x higher P99 latency will degrade user experience more than the accuracy improvement helps. Track accuracy and latency together. Plot them on the same dashboard. Make trade-off decisions explicit.

For embedding quality, measure retrieval recall at k (R@5, R@10) on your golden dataset, not on public benchmarks. Public benchmarks test general-purpose text similarity. Your production queries have domain-specific vocabulary, abbreviations, and implicit context that general benchmarks do not capture. The model that ranks first on MTEB might rank fifth on your actual retrieval workload. Investing in natural language processing as a production discipline means building these domain-specific evaluation suites from the start.

Building AI-powered systems that hold up under real traffic requires treating every component of the NLP pipeline, from Unicode normalization to embedding refresh, as production infrastructure worthy of monitoring, versioning, and evaluation. The model is the smallest part of a production NLP system. Everything around it determines whether it works or just looks like it works.