Alternative Data and NLP in Quant Strategies › Natural Language Processing for Finance

Text represents the largest and most underexploited category of alternative data. Earnings call transcripts, news articles, analyst reports, social media posts, regulatory filings, and patent documents collectively contain enormous information that humans cannot process at scale. Natural language processing provides the tools to extract structured insights from this unstructured text.

The fundamental challenge in text analytics is representation: how do we convert words and sentences into numerical features that machine learning models can process? This problem is more complex than it might initially appear because language is inherently high-dimensional, context-dependent, and ambiguous. Every sentence carries meaning through the individual words it contains, their arrangement, their relationships to one another, and the broader context in which they appear. Our goal is to capture as much of this meaning as possible in a numerical form that algorithms can manipulate.

The bag-of-words model is the simplest approach. It treats a document as an unordered collection of words, ignoring grammar and word order entirely. Under this representation, each unique word in the vocabulary becomes a separate feature dimension, and documents are represented as vectors of word counts or frequencies. The resulting vector space has as many dimensions as there are unique words in the corpus, which can easily reach tens of thousands of dimensions for realistic text collections.

Consider two short sentences about a stock:

• "Revenue growth exceeded expectations"
• "Expectations exceeded by revenue growth"

A bag-of-words representation would treat these identically: both contain the same words with the same frequencies. The model captures what topics are discussed but not how they relate. This limitation reveals a fundamental tradeoff. Simpler models like bag-of-words are computationally efficient and interpretable. However, they discard the relationships conveyed by word order. More sophisticated approaches attempt to recover this lost information, but at the cost of greater complexity.

Raw word counts suffer from a significant limitation: common words like "the," "and," and "company" dominate the representation despite carrying little discriminative information. These high-frequency words appear in nearly every document, so their presence tells us almost nothing about what makes any particular document unique or meaningful. If we want our numerical representation to capture the distinctive content of each document, we need a weighting scheme that acknowledges that not all words contribute equally to meaning.

The TF-IDF (Term Frequency-Inverse Document Frequency) weighting scheme addresses this limitation through a simple insight. Words that appear frequently in a specific document but rarely across the broader corpus are likely the most informative features. Words that appear everywhere carry little discriminative power regardless of how often they appear in any single document. This intuition leads to a weighting formula that combines two complementary measurements.

The TF-IDF score for a term $t$ in document $d$ within corpus $D$ combines two components to emphasize distinctive terms while downweighting common ones. It measures both how often a term appears in a specific document (term frequency) and how rare that term is across all documents (inverse document frequency):

$$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \text{IDF}(t, D)$$

where:

• $t$: the term (word) being scored
• $d$: the specific document containing the term
• $D$: the entire corpus (collection of all documents)
• $\text{TF}(t, d)$: term frequency, measuring how often term $t$ appears in document $d$ (typically normalized by document length)
• $\text{IDF}(t, D)$: inverse document frequency, downweighting common terms by measuring how rare $t$ is across the corpus
• $\text{TF-IDF}(t, d, D)$: the combined score, higher values indicate terms that are both frequent in the document and rare in the corpus

The multiplication of these two components creates a weighting scheme. Terms that appear frequently in a specific document but rarely across the corpus receive the highest scores, making them the most distinctive features of that document. This multiplicative structure means that both conditions must be satisfied for a term to receive a high weight. A word appearing frequently in one document but also appearing in every other document will have its high term frequency nullified by a low inverse document frequency. Similarly, an extremely rare word that appears only once in a single document receives a limited boost because its term frequency is low despite its high corpus-level rarity.

The inverse document frequency component quantifies how rare or common a term is across the entire corpus. This measurement provides the mechanism for penalizing ubiquitous terms while rewarding distinctive vocabulary. The formula takes a ratio of the total number of documents to the number of documents containing the term, then applies a logarithmic transformation to moderate the scaling behavior.

The inverse document frequency is computed as:

$$\text{IDF}(t, D) = \log \frac{|D|}{|\{d \in D : t \in d\}|}$$

where:

• $t$: the term being evaluated
• $D$: the entire corpus (collection of documents)
• $|D|$: total number of documents in the corpus (cardinality of set $D$)
• $|\{d \in D : t \in d\}|$: number of documents containing term $t$ (size of the subset where $t$ appears)
• $\log$: natural logarithm function, so the penalty grows gradually rather than explosively

The IDF formula penalizes common words through a logarithmic ratio that captures distinctiveness. To understand why this formula works, consider the extreme cases at both ends of the term frequency spectrum. If a term appears in all documents, the ratio $\frac{|D|}{|D|} = 1$ yields $\log(1) = 0$, assigning zero weight because ubiquitous words provide no discriminative power. This makes intuitive sense: if every document contains the word "the," knowing that a particular document contains "the" tells us nothing useful for distinguishing it from other documents. Conversely, if a term appears in only one document, the ratio $\frac{|D|}{1} = |D|$ maximizes $\log(|D|)$, giving high weight to this uniquely identifying term. Such rare terms are precisely the vocabulary that makes a document distinctive.

The logarithm serves an important mathematical purpose beyond simply computing a ratio. Without the logarithmic transformation, the IDF values would grow linearly with corpus size for rare terms, potentially creating extreme weights that dominate the feature representation. The logarithm ensures the penalty grows gradually, not explosively, creating smooth weighting transitions as term frequency increases. This gradient allows IDF to distinguish between moderately common and very common terms without overweighting rare terms. A term appearing in half the documents receives a moderate penalty, while a term appearing in 90% of documents receives a stronger penalty, but the difference remains manageable and interpretable.