Vector Space Model & TF-IDF (1968) › The Problem: When Keywords Aren't Enough
History of Language AIAll chapters Vector Space Model & TF-IDF (1968) › The Problem: When Keywords Aren't EnoughOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Boolean systems let users combine terms with operators such as AND, OR, and NOT. A query might read computer AND programming NOT FORTRAN. A document either satisfied the expression or did not; the result set itself contained no graded relevance ranking.
A narrow Boolean query could miss documents that used different terminology: "electronic computers" would not match "digital computers" unless the query anticipated both phrases. A broad query could return too many matches without indicating which ones best addressed the information need.
Librarians and information scientists already treated relevance as graded rather than binary. Manual subject headings provided some structure, but they were expensive to assign and could vary across indexers. A numerical representation offered a way to compute degrees of match automatically as collections grew.
The deeper challenge was one of representation. Boolean systems treated words as atomic symbols with no relationship to each other. The word "computer" and the word "machine" were as different as "computer" and "banana." But humans know that in many contexts, "computer" and "machine" are related concepts, while "computer" and "banana" are not. How could this kind of semantic relationship be captured mathematically? How could a system understand that a document heavy with certain technical terms was likely about a particular topic, even if it never used the exact query terms?