Freebase (2007) › The Problem: Unstructured Knowledge and the Question-Answering Gap

The fundamental challenge that Freebase addressed was the gap between how humans store information and how computers could process it. Throughout the early 2000s, the web had grown into an enormous repository of human knowledge, containing information about virtually every topic imaginable. However, this information existed primarily as unstructured text: paragraphs in Wikipedia articles, descriptions on company websites, reviews in online databases. While humans could read these pages and extract the information they needed, computers struggled to perform the same kind of synthesis and reasoning.

Consider a simple question: "What awards did Meryl Streep win?" A human researcher could find this information by reading Wikipedia articles, scanning through award ceremony records, and piecing together information from multiple sources. But traditional search engines and databases could not answer this directly. They could find documents that mentioned both "Meryl Streep" and "awards," but they could not extract and synthesize the specific information needed. The answers existed on the web, but they were buried in unstructured text that required human interpretation to understand.

This unstructured nature of web information created problems for applications that needed to reason about knowledge. Recommendation systems could not easily determine that two movies shared the same director if that information appeared only in narrative text. Question-answering systems could not extract precise facts like "the population of Tokyo" when that number might be embedded in paragraphs about urban development. Information extraction systems struggled to maintain consistency when the same entity appeared under different names or when relationships were expressed through varied linguistic constructions.

The problem extended beyond mere information retrieval. Even when information could be found, there was no standardized way to represent relationships between entities. One website might describe a person as "born in New York City," another as "from Manhattan," and a third as "originating in NYC." All three statements referred to the same fact, but expressed differently, making it difficult for systems to recognize equivalence or reason about the underlying relationships.

Traditional databases offered structured representations but required predefined schemas that could not adapt to the diverse and evolving nature of real-world knowledge. They could store information about movies and actors if you designed tables for films and the people who performed in them, along with the relevant relationships. But what if you wanted to add information about filming locations, box office performance, or critical reception? Each new type of information required schema modifications, making traditional databases too rigid for the fluid, interconnected nature of human knowledge.

The semantic web vision of the late 1990s and early 2000s had proposed using formal ontologies and RDF triples to structure knowledge, but these approaches proved too complex and cumbersome for widespread adoption. They required extensive manual curation, complex query languages, and substantial technical expertise to maintain. The gap between the vision of structured, queryable knowledge and practical implementation remained wide.

Freebase's founders recognized that what was needed was a knowledge representation system that combined the structured nature of databases with the flexibility and collaborative nature of wikis. The system would need to handle the scale of web information while providing enough structure to support computational reasoning. It would need to be flexible enough to accommodate new types of entities and relationships without requiring schema changes, yet structured enough that applications could reliably query and reason about the knowledge it contained.