The Vocabulary Problem › Technical and Domain-Specific Text
Language AI HandbookAll chapters The Vocabulary Problem › Technical and Domain-Specific TextOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Beyond everyday language variation, NLP systems increasingly face the challenge of processing technical text: source code, scientific papers, medical records, legal documents. Each of these domains introduces vocabulary that general-purpose models have never encountered, and the structure of the unknown tokens differs substantially from everyday OOV words.
Unlike new slang or neologisms, which at least follow the phonological patterns of the language, technical vocabulary often violates standard word formation rules entirely. Gene names like BRCA1 and TP53 are alphanumeric. Chemical formulas like C6H12O6 mix letters, numbers, and subscripts. Programming identifiers like XMLHttpRequest combine acronyms and camelCase. These tokens don't just miss the vocabulary: they wouldn't benefit much from the [UNK] treatment because their structure contains meaningful information that a word-level model cannot exploit.
The Challenge of Specialized Vocabulary
Consider what happens when a general NLP model encounters source code. Variable names, function names, and identifiers are created by developers using essentially arbitrary character combinations, and they follow conventions that are entirely distinct from natural language word formation. Different developers and codebases use different naming conventions, all of which appear as distinct surface forms to a word-level tokenizer:
- CamelCase:
getUserById,XMLHttpRequest,processPayment - Snake case:
get_user_by_id,calculate_total_price - Screaming snake case:
MAX_RETRY_COUNT,API_BASE_URL - Special tokens:
__init__,self.model,@property
The same underlying concept, a function that retrieves a user by their identifier, is expressed as entirely different surface forms depending on the developer's convention. getUserById and get_user_by_id are synonymous, but a word-level model sees them as two completely different, probably unknown tokens. This is a representation problem as well as an OOV problem. Even if both tokens were in the vocabulary, the model would have no way to know they refer to the same concept.
Scientific text uses morphologically complex terms built from Greek and Latin roots that follow systematic patterns. "Phosphofructokinase" is a long word, but it decomposes cleanly: "phospho-" refers to a phosphate group, "fructo-" refers to fructose, and "-kinase" identifies the protein class. A biochemist encountering the term "phosphoglycerokinase" for the first time immediately understands the general category: a kinase acting on a phosphoglycerol substrate. A word-level model sees it as an unknown token with no relationship to any other term.
Medical text combines both patterns: complex Latinate terms ("thrombocytopenia," meaning a deficiency of blood platelets; "atherosclerosis," meaning hardening of arteries due to plaque buildup) alongside cryptic abbreviations ("prn" for "as needed," "bid" for "twice daily," "qid" for "four times daily"). Neither category is well-handled by word-level models.
import re
def split_camel_case(identifier):
"""Split camelCase and PascalCase identifiers into component words."""
# Insert space before uppercase letters that follow lowercase
result = re.sub(r"([a-z])([A-Z])", r"\1 \2", identifier)
# Handle consecutive uppercase (acronyms like XML, HTTP)
result = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1 \2", result)
return result.lower().split()
def split_snake_case(identifier):
"""Split snake_case identifiers into component words."""
return identifier.lower().split("_")
camel_examples = [
"getUserById",
"XMLHttpRequest",
"processHTMLDocument",
"calculateTotalPrice",
]
snake_examples = [
"get_user_by_id",
"xml_http_request",
"process_html_document",
"calculate_total_price",
]Splitting Programming Identifiers into Components ================================================================= CamelCase: getUserById -> ['get', 'user', 'by', 'id'] XMLHttpRequest -> ['xml', 'http', 'request'] processHTMLDocument -> ['process', 'html', 'document'] calculateTotalPrice -> ['calculate', 'total', 'price'] snake_case: get_user_by_id -> ['get', 'user', 'by', 'id'] xml_http_request -> ['xml', 'http', 'request'] process_html_document -> ['process', 'html', 'document'] calculate_total_price -> ['calculate', 'total', 'price']
The decompositions reveal an important insight. Both getUserById and get_user_by_id contain the same four common words ("get", "user", "by", "id"), each of which almost certainly appears in any general vocabulary. But the combined identifier is almost certainly OOV in its raw form. The camelCase splitter also handles acronyms correctly, keeping "XML" and "HTML" together rather than splitting each uppercase letter individually, which preserves semantically meaningful units.
This suggests a general and powerful principle: complex tokens often contain recognizable sub-parts, and a tokenization scheme that can identify and decompose those parts intelligently would dramatically reduce OOV rates without requiring a massive vocabulary. The model could learn that "get" followed by a noun-like element describes a retrieval operation, regardless of whether that noun is "user," "order," "product," or any other entity type. The insight generalizes far beyond code: medical terms built from Greek roots, chemical names, and morphologically complex words in any language all exhibit the same structure. That intuition is precisely what motivates subword tokenization, and we'll see how BPE and WordPiece turn this intuition into an algorithm in the following chapters.