IBM Watson on Jeopardy! (2011) › Limitations and Challenges

Watson's errors also exposed the limits of its evidence-driven approach. It could fail when a clue depended on irony, implicit cultural context, subjective judgment, or evidence not represented well in its indexed sources and features.

Watson sometimes failed on questions that required understanding subtle cultural references, recognizing irony or sarcasm, or making connections that relied on common sense reasoning rather than explicit factual knowledge. The system excelled when questions could be answered through retrieval and pattern matching, but struggled when questions required deeper understanding, creative thinking, or the kind of intuitive reasoning that humans perform effortlessly.

The system's knowledge was also frozen in time, based on training data from before the competition. It couldn't access real-time information or learn from new experiences during the competition. This limitation meant that Watson couldn't answer questions about very recent events or update its knowledge based on information that became available after its training period.

Watson was highly optimized for Jeopardy!-style question answering. Its performance did not imply that the same system could transfer to a new domain without rebuilding sources, features, training data, and evaluation procedures.