Conversational AI › Challenges and Limitations

LLMs can contradict themselves across turns. In a long conversation, the model may forget facts established earlier or take positions inconsistent with earlier statements. For instance, if the user mentions they're vegetarian in turn 3, the model might recommend a meat-heavy dish in turn 15, having effectively "forgotten" the earlier constraint. This is a fundamental consequence of how attention works: the model attends to its full context window, but distant tokens have lower influence on generation. When the conversation history is long, early facts are diluted by subsequent tokens.

The problem is compounded by the fact that LLMs don't maintain explicit state. A rule-based system would have stored "user is vegetarian" as a database entry that persists reliably. An LLM relies on the text of the conversation to encode all such facts, which makes them susceptible to the same attention dynamics that affect any other token in the sequence.

Mitigations include: explicit state tracking injected into the prompt at each turn, having the system summarize confirmed facts and prepend them to each request, and periodic coherence checking where a secondary model reviews the conversation for contradictions. For applications where consistency is critical, such as medical information systems, explicit fact storage is safer than relying on the model's conversational memory.

Conversational systems are prone to hallucinating facts, especially when users ask about specific details (statistics, dates, names) that the model doesn't have reliable training signal for. Hallucination is particularly insidious in dialogue because the conversational context makes responses feel credible. A confident-sounding response in a natural-feeling conversation is more likely to be trusted than a confident-sounding response in an isolated text box.

The social dynamics of conversation amplify this risk. Users are accustomed to expecting that conversational partners know what they're talking about. When someone confidently states a fact in conversation, we typically accept it unless we have specific reason to doubt it. This conversational trust calibration is appropriate for most human interactions but creates real risk when talking to a system that can hallucinate fluently.

Retrieval-augmented approaches help significantly: ground responses in retrieved documents and instruct the model to cite or qualify claims that aren't in the retrieved context. But they don't eliminate the problem entirely, especially for edge cases outside the retrieval corpus. For high-stakes applications like medical or legal information, the system should be designed to consistently express appropriate uncertainty and recommend consulting authoritative sources.

Conversational systems encounter sensitive topics: politics, religion, health, mental health, legal questions, and requests that sit in ethical gray areas. Appropriate handling requires careful calibration:

Too restrictive systems refuse to answer benign questions, frustrating users and failing to deliver value. A cooking assistant that refuses to discuss alcohol because alcohol is "sensitive" is badly miscalibrated. Too permissive systems give advice they shouldn't (medical, legal, financial) or produce content that causes harm.

System prompts can establish scope constraints, but they don't provide perfect protection. Models can be prompted in ways that circumvent system instructions, and edge cases always exist where the appropriate response is ambiguous. A question about medication dosages might be asked by a nurse who needs accurate information or by someone in crisis who should not receive that information. The system often cannot distinguish between these cases, which creates a fundamental tension between usefulness and safety.

Calibrating this tension requires ongoing monitoring and iteration. Deploy with conservative defaults, monitor edge cases, and adjust as you understand your real user population better. A system calibrated for a medical professional audience and one calibrated for a general consumer audience should have different defaults for the same underlying model.

Most LLM-based systems don't have persistent memory across sessions by default. Each new conversation starts fresh. For applications like a personal assistant where continuity matters, this requires explicit memory infrastructure: storing facts about the user and retrieving them at the start of each session.

Implementing this well is non-trivial. You need to decide what to remember, how long to retain it, how to update beliefs when they change ("I used to prefer window seats but now I prefer aisle"), and how to handle user requests to forget stored information. The right-to-be-forgotten has legal implications in many jurisdictions under privacy regulations, and any system that stores personal information about users must have a clear data retention and deletion policy.

There's also a design question about transparency: should users be aware of what the system remembers about them? Users who know the system has memory may share more useful information, but they may also be uncomfortable with what they've implicitly consented to. Transparent memory management, where users can see and edit stored facts about themselves, is both better UX and more ethical.

Conversational systems can be misused in ways that cause real harm: generating misinformation at scale, assisting with harmful activities, psychological manipulation through simulated relationships. The conversational format creates risks that don't exist for single-shot generation: users can use multi-turn interactions to gradually steer models toward producing content they'd refuse to generate directly. A series of seemingly innocuous questions can establish context that makes a harmful request seem acceptable.

This is sometimes called "jailbreaking through context building." A user establishes a fictional scenario, then gradually pushes the assistant to produce content within that scenario that would be declined in a direct request. The model's tendency to stay in context, one of its most useful properties for legitimate conversations, becomes a liability when the context is being manipulated.

Alignment training (RLHF and related techniques, covered in the alignment chapter) addresses some of this, but the attack surface for conversational misuse is large and evolves as researchers and adversaries find new prompting strategies. Production systems typically layer multiple defenses: alignment-trained base models, safety classifiers on input and output, system prompt instructions, and human review of flagged conversations.