Conversational AI › Anatomy of a Modern Conversational System

Even with a single LLM as the main engine, production conversational systems have multiple layers. Understanding these layers helps you make principled engineering decisions rather than treating the LLM as a magic box.

In an LLM-based chatbot, conversation history is passed directly in the context window. Each turn is represented as a structured message with a role and content:

The model attends over all previous messages to generate the next response. This means it naturally carries context forward: it won't ask again what city you want to visit. But it also means the conversation length is bounded by the context window, and costs grow linearly with conversation length. For a system with a 4,096-token context window and conversations that run 50 turns, you'll hit the limit in a single session. Even with large-context models (128K tokens or more), the cost of attending over a long context on every generation step is real.

This architecture collapses what the statistical pipeline handled with separate modules into a single representation. The "dialogue state" is implicit in the sequence of tokens in the context window. This is simultaneously elegant and fragile: elegant because no explicit state management is needed, fragile because the model's ability to recall early facts degrades as the context grows, and because there's no mechanism to guarantee that specific facts are reliably tracked.

In a two-party conversation, turns alternate between user and assistant. In multi-party conversations (group chats, voice assistants in shared spaces), turn management becomes more complex. Even in simple two-party systems, you need to handle several non-trivial cases:

• Interruptions and clarifications: The user changes direction mid-conversation, asking a new question before the previous one was answered.
• Ambiguous inputs: The user says something that could mean multiple things, and the system must either ask for clarification or make a reasonable default assumption.
• Empty or malformed inputs: The user submits a blank message, a typo-heavy fragment, or a voice-to-text transcription that garbled the input.
• Topic shifts: The user pivots from one topic to another, expecting the system to follow.
• Grounding: Confirming that both parties share the same understanding before taking an action. ("So I'm booking a table for four on Saturday the 25th at 7pm. Does that look right?")

Modern LLMs handle much of this implicitly by having seen diverse dialogue patterns during training. But for high-stakes applications, explicit logic to handle these edge cases provides more reliability than trusting the model's training distribution.

The grounding step deserves special emphasis. In task-oriented applications, confirming before taking irreversible actions is critical. A system that sends an email, places an order, or deletes a file without explicit confirmation from the user creates a category of errors that cannot be undone. The confirmation step is a simple safety mechanism that experienced practitioners rarely omit.

A fundamental challenge in conversational AI is memory: what information persists across turns, and for how long? The challenge has two dimensions: the technical constraint of context window limits, and the semantic question of what information is worth retaining.

There are three main memory patterns that practitioners combine in different configurations:

In-context memory is the simplest: include the full conversation history in every request. This works well for short conversations but becomes expensive and eventually impossible as conversations grow longer. Its advantage is perfect recall within the window: the model can reference any turn from the current session without any special retrieval mechanism.

Summarization-based memory compresses older parts of the conversation into a summary that takes less space. The model periodically summarizes what has been discussed, and that summary replaces the raw history. A well-constructed summary captures the key facts, decisions, and established preferences without preserving the exact wording of each turn. This loses detail but keeps the context window manageable. The compression step requires calling the model (or a separate summarization model), which adds latency and cost at the compression trigger point.

External memory stores facts and preferences in a database outside the model. Relevant memories are retrieved and injected into the context as needed, similar to the retrieval-augmented generation pattern. This can support persistent user profiles across sessions: the system remembers that you prefer window seats on flights, don't eat shellfish, and have a standing Tuesday meeting. At the start of each new conversation, relevant preferences are retrieved and included in the system prompt. This decouples memory from context length, allowing effectively unlimited long-term retention.

In practice, many production systems combine these approaches: raw history for the current session, summarization across a session when the history grows long, and a persistent user profile retrieved from a database at session start. The right combination depends on the application: a short customer service interaction needs none of this complexity, while a personal productivity assistant benefits from all three layers.

Even without a separate NLU module, it helps to think in terms of dialogue acts: the communicative purpose of an utterance. The concept originates in speech act theory from linguistics, where philosophers like Austin and Searle argued that utterances don't just convey information but perform social actions. Asking "can you pass the salt?" is grammatically a question, but pragmatically a request. In dialogue systems, we classify utterances by their communicative function because that determines what an appropriate response looks like.

Common dialogue acts include:

• Inform: Providing information ("The flight departs at 10am").
• Request: Asking for information or action ("What's your budget?").
• Confirm: Verifying understanding ("So you want business class?").
• Acknowledge: Showing attention ("Got it!").
• Clarify: Resolving ambiguity ("Did you mean Paris, France or Paris, Texas?").
• Reject: Declining a request ("I can't book third-party hotels through this system").
• Commit: Making a promise about future action ("I'll look that up for you").
• Greet and Close: Conversation management acts that open and terminate exchanges.

The key insight is that the appropriate system response depends more on the dialogue act than the specific words. A user who says "tell me the price" and one who says "how much does it cost?" are making the same request act and deserve the same response. Older NLU systems had to classify the act explicitly before deciding what to do. LLMs produce responses that fulfill these acts without explicit labeling, but understanding the space of dialogue acts helps when debugging failure modes or designing evaluation criteria. If a system consistently fails to confirm before taking irreversible actions, for instance, you can diagnose that as a confirm-act deficiency and address it in the system prompt or through post-processing logic.

The dialogue act framework also helps when designing evaluation rubrics. Rather than asking "is this a good response?" you can ask more specific questions: did the system use an inform act when a request act was needed? Did it confirm when confirmation was appropriate? Breaking quality down by dialogue act type makes evaluation more systematic and reveals patterns that aggregate metrics hide.