Semantic Parsing and Meaning Representations › Text-to-SQL

Text-to-SQL is semantic parsing where the target representation is a SQL query. It lets people query relational databases through natural-language questions. Enterprises store data in tables, and people who need answers may not know SQL. Reliability matters: a wrong query can silently return incorrect rows, and a user who cannot read SQL may not notice.

A SQL query has a precise syntax and an executable semantics. Consider:

SELECT name
FROM students
WHERE age > 20;

The query denotes a table. Its rows are the names of students whose age is greater than 20. This is a denotational view of SQL: the meaning of a query is the table it returns.

SQL semantics is not exactly first-order logic. It differs in several ways:

• It uses bag semantics by default, so duplicate rows can appear unless DISTINCT is used.
• It has three-valued logic because of NULL.
• It has aggregation, grouping, sorting, and nested queries.

These features make SQL expressive, but they also make the mapping from natural language challenging. A parser that assumes set semantics or two-valued logic can make mistakes when duplicates or NULL-sensitive predicates affect the result.

SQL queries can use constructs such as these:

• SELECT list: which columns or expressions to return.
• FROM clause: which tables to read.
• WHERE clause: row-level filters.
• GROUP BY: partitions rows into groups.
• HAVING: filters groups after aggregation.
• ORDER BY: sorts the result.
• LIMIT: restricts the number of rows.
• Set operations: UNION, INTERSECT, EXCEPT.
• Subqueries: nested queries in the FROM, WHERE, or SELECT clause.

A natural-language question need not spell out SQL clauses. Under the schema used below, one valid solution to "Which departments have more than two enrolled students?" joins the three tables, groups by department, counts distinct students, and applies a HAVING condition. Equivalent queries can express the same answer condition with other SQL structures. The parser must choose a structure that captures the intended relationships and count.

The count must refer to students rather than enrollment records. A student can have several enrollment rows, so the query needs to avoid counting that person repeatedly. The question supplies the intended unit; the schema determines how to recover it from the stored rows.

Schema linking is the process of mapping mentions in the question to tables, columns, and values in the database schema. A linking error can make a syntactically valid query answer the wrong question.

Schema names and storage conventions need not match how users phrase questions. Column names are abbreviated, tables are normalized, and values are stored in formats that the question does not mention. A question about "departments" may need to join a dept table through a dept_id column that the question never names.

Consider the question "Which departments have more than two enrolled students?" The word "departments" likely maps to the departments table or the department_id column. "Students" maps to the students table. "Enrolled" maps to the enrollments table. "More than two" maps to a comparison with the number 2. None of these mappings is guaranteed. A database might call the table dept, division, or org_unit. A column might be named student_count rather than student_id. The question might use a synonym that does not appear in the schema.

Real databases can contain hundreds of tables and thousands of columns. A parser can narrow the candidate set using signals such as string similarity, type compatibility, schema relationships, and learned representations.

Schema linking must handle:

• Synonyms and abbreviations: "employees" versus emp, "departments" versus dept.
• Ambiguous column names: name might exist in several tables.
• Values: "the Computer Science department" might require matching a string value.
• Joins: the question may require joining tables through foreign keys.
• Types: comparing a date column to a string, or an integer column to a word number.
• Dialect differences: SQLite, MySQL, PostgreSQL, and BigQuery differ in functions, quoting, and date handling.

A query can be syntactically valid and still be wrong because it linked "students" to the wrong table or joined on the wrong key.

Schema linking is therefore not a preprocessing detail. It is a core part of the semantic parsing problem, and schema-linking errors can propagate into the generated query and change its result.

Text-to-SQL systems expected to handle new combinations of familiar operations need compositional generalization. A model that has seen many simple SELECT ... WHERE ... queries may fail on nested queries, set operations, or aggregation with HAVING. The displayed solution to "Which departments have more than two enrolled students?" uses grouping and counting.

Generating the displayed plan combines joins, grouping, a distinct count, and a HAVING condition in a configuration the model may not have seen during training. Other valid SQL formulations can express the same answer condition.

A training set may cover individual SQL operations without covering all their combinations. Models can memorize patterns and still fail on new combinations of known components. Benchmarks such as Spider were designed to test cross-domain generalization with held-out databases and complex query structures.

The problem is compounded by the fact that the training set is finite and the space of schemas is open-ended. A model that learns to copy column names can do well on one database and fail on the next.

The main benchmarks tell the story of the field:

• WikiSQL (Zhong et al., 2017): single-table questions over Wikipedia tables. Its single-table query structures are narrower than the multi-table tasks in later benchmarks.
• Spider (Yu et al., 2018): cross-domain, multi-table, complex SQL. It became the standard benchmark for years.
• BIRD (Li et al., 2023): large databases, external knowledge, and more realistic value grounding.
• Spider 2.0 (Lei et al., 2024 preprint): enterprise workflows, real schemas, and agentic settings where the system may need to inspect data and run multiple queries.

The progression from WikiSQL to Spider to BIRD to Spider 2.0 tracks the field's growing ambition. WikiSQL provided a large single-table benchmark with learned sequence-to-sequence baselines. Spider added cross-domain generalization and complex queries. BIRD added large databases and external knowledge. Spider 2.0 added enterprise workflows and agentic settings.

Evaluation has two common modes:

• Exact match: compare a prediction with a gold query under the benchmark's normalization and structural-matching rules. For example, Spider compares parsed clause components as sets. Literal string comparison, illustrated below, is stricter.
• Execution accuracy: execute the generated query and the gold query on a database and compare the result tables. This is closer to the task, but it can reward a wrong query that happens to produce the right answer on a particular database.

Execution accuracy can accept different SQL strings that return the same result. But it has its own failure modes. A query can return the right answer for the wrong reason. A query can be correct on one database instance and wrong on another. Test-suite accuracy addresses this by executing queries against multiple database instances and comparing results. Agreement on a finite test suite for general SQL does not by itself establish equivalence on every relevant database instance.

In practice, a good evaluation combines execution accuracy with manual inspection and with robustness checks on paraphrases and schema variations.

Evaluation metrics can disagree even when two queries mean the same thing to the database. A small execution check separates literal text matching from denotation matching. This demonstration is not an implementation of Spider's structural metric.