MMLU › Worked Example

Let us walk through a concrete MMLU question to understand the reasoning required. Consider this example from the College Medicine subject:

Question: A 45-year-old man presents with progressive dyspnea on exertion and orthopnea. Physical examination reveals elevated jugular venous pressure, hepatomegaly, and peripheral edema. Echocardiography shows normal left ventricular function but a restrictive filling pattern. Which of the following is the most likely diagnosis?

A. Dilated cardiomyopathy B. Hypertrophic cardiomyopathy C. Restrictive cardiomyopathy D. Arrhythmogenic right ventricular cardiomyopathy

Analysis: The symptoms presented here form a classic clinical picture that requires careful parsing of physiological relationships. Dyspnea on exertion, which means shortness of breath during physical activity, combined with orthopnea, difficulty breathing when lying flat, suggests cardiac dysfunction affecting the lungs. The physical examination findings give important localization clues. Elevated jugular venous pressure indicates increased pressure in the right side of the heart or superior vena cava. Hepatomegaly, or liver enlargement, suggests congestion of the portal venous system, while peripheral edema indicates fluid accumulation in the extremities due to venous congestion. Together, these three findings constitute the classic triad of right-sided heart failure.

However, the differentiator that distinguishes between the four cardiac conditions offered lies in the echocardiography findings. The test reveals normal left ventricular function but a restrictive filling pattern. This is the key diagnostic clue. Normal left ventricular function rules out conditions primarily characterized by systolic dysfunction, where the pumping action of the heart is compromised. The restrictive filling pattern indicates that the ventricles are stiff and cannot relax properly during diastole, the phase of the cardiac cycle when the heart fills with blood.

Examining each option in turn:

• Dilated cardiomyopathy (A) typically shows enlarged ventricles with systolic dysfunction. The heart becomes baggy and dilated, leading to poor contraction. This contradicts the finding of normal left ventricular function.

• Hypertrophic cardiomyopathy (B) shows ventricular thickening and often outflow obstruction. While it can affect diastolic function, the primary characteristic is myocardial thickening rather than restriction, and it often presents with dynamic obstruction of the left ventricular outflow tract, which is not mentioned here.

• Restrictive cardiomyopathy (C) is characterized by stiff ventricles that impair filling, known as diastolic dysfunction, while maintaining normal systolic function and chamber size. This matches the description perfectly: the heart can pump normally (normal function) but cannot fill properly (restrictive pattern).

• Arrhythmogenic right ventricular cardiomyopathy (D) primarily affects the right ventricle with fibrofatty replacement of the myocardium. While it can cause right-sided failure, it would typically show structural abnormalities of the right ventricle on echocardiography rather than a restrictive filling pattern affecting the left ventricle.

The correct answer is C. Restrictive cardiomyopathy.

This example illustrates that MMLU questions often require multiple layers of reasoning:

1. Domain-specific terminology recognition (dyspnea, orthopnea, jugular venous pressure): The model must understand medical vocabulary to parse the clinical presentation.

2. Differential diagnosis reasoning (comparing multiple pathological conditions): The model must consider several diseases that affect the same organ system and present with overlapping symptoms.

3. Key feature identification (the restrictive filling pattern is the diagnostic clue): Among many clinical findings, the model must identify which one discriminates between the options.

4. Elimination of distractors (each wrong answer describes a real condition with overlapping symptoms): The incorrect options are real diseases that could reasonably present with similar symptoms, requiring fine-grained understanding to eliminate them based on the specific findings provided.

Let us also consider an example from a non-medical domain to illustrate how the reasoning demands change across subjects. Consider a question from Formal Logic:

Question: Which of the following is a valid argument form?

A. If P then Q; P is false; therefore Q is false B. If P then Q; Q is true; therefore P is true C. If P then Q; P is true; therefore Q is true D. If P then Q; Q is false; therefore P is true

Here, the required reasoning is purely formal rather than empirical. The task is to identify which of these four inference patterns is logically valid, meaning the conclusion must be true whenever the premises are true.

Option A commits the fallacy of denying the antecedent: knowing that P implies Q does not tell us anything about what happens when P is false. Q could still be true for other reasons.

Option B commits the fallacy of affirming the consequent: knowing that Q is true does not mean P must be true, since Q might be true for reasons unrelated to P.

Option C is modus ponens, the most basic valid argument form in propositional logic. If "if P then Q" is true and P is true, then Q must be true. This is valid.

Option D inverts the relationship incorrectly: if Q is false in "if P then Q," then by modus tollens we should conclude that P is false, not true. Option D gets this backwards.

The correct answer is C.

Comparing these two examples reveals how different MMLU's demands are across subjects. The medicine question requires building up a clinical picture from multiple factual pieces of knowledge and connecting them to disease taxonomy. The logic question requires applying formal rules independently of any factual knowledge. A model that excels at one type may struggle with the other, which is precisely what cross-subject analysis of MMLU scores reveals about model strengths and weaknesses.