Vector Space Model & TF-IDF (1968) › Implementation and Refinement

Large vocabularies make document vectors high-dimensional. Storing every coordinate would waste memory on zeros because each document contains only a small fraction of the collection vocabulary.

SMART used sparse vectors that stored only nonzero (term, weight) pairs. Cosine similarity then needed to examine only dimensions present in both vectors, reducing storage and computation.

The system also introduced several refinements to the basic TF-IDF formula. Different variants of term frequency normalization were explored: should TF grow linearly with term count, or should it saturate (using logarithms or other sublinear functions) to prevent documents that happened to mention a term many times from dominating? Should document length be explicitly normalized to prevent long documents from having artificially high similarity scores simply because they contained more words? These questions led to a family of TF-IDF variants, each optimizing for different characteristics of document collections.

Salton and his team evaluated weighting variants on collections of abstracts and scientific papers. Collections such as Cranfield aeronautics abstracts and CACM computer-science papers became information-retrieval benchmarks. These experiments allowed TF-IDF and cosine retrieval to be compared with Boolean systems under shared conditions.

SMART also supported query expansion and relevance feedback. After an initial retrieval, users could mark documents as relevant or non-relevant. The system adjusted the query vector with terms from relevant documents, moving it toward their region of the vector space without requiring a fully manual reformulation.