upstream/upstream-sqlc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/sqlc-dev/sqlc.git (main, 2026-09-16; blobs > 131072B omitted)
Some checks failed
kotlin / test (push) Has been skipped
Details
python / test (push) Has been skipped
Details
typescript / test (push) Has been skipped
Details
go / build darwin/amd64 (push) Has been cancelled
Details
go / build linux/amd64 (push) Has been cancelled
Details
go / build windows/amd64 (push) Has been cancelled
Details
go / build darwin/arm64 (push) Has been cancelled
Details
go / build linux/arm64 (push) Has been cancelled
Details
go / build windows/arm64 (push) Has been cancelled
Details
go / test (push) Has been cancelled
Details
go / vuln_check (push) Has been cancelled
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-16 02:17:34 +05:00
commit 0f4df23290
7340 changed files with 238743 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

342
internal/goldeneye/README.md Normal file
View File

@@ -0,0 +1,342 @@
# goldeneye
`goldeneye` generates the dialect seeds under `internal/engine/<engine>/dialect`
from a live database, and verifies the committed ones against it. A dialect
is the JSONL that gives an engine its type system and standard library —
`types.jsonl`, `functions.jsonl`, `operators.jsonl`, `relations.jsonl` and
the `extensions/` bundles — read by `internal/core/seed`. Each engine package
here asks the database what it knows, writes the answer in that shape, and
the tests compare it with what is committed, byte for byte. A difference
means the committed dialect has drifted from the database.
It is a nested Go module, so its only dependencies beyond the standard
library are the database clients — PostgreSQL's, MySQL's and SQL Server's
drivers, and Spanner's gRPC client with what it stands on — and it never
shares code with the analysis that reads the files: the files are the
contract. Run it from this directory:
```bash
go run ./cmd/goldeneye install clickhouse # download the pinned clickhouse binary once
go run ./cmd/goldeneye install duckdb # download the current DuckDB 2.0 preview build once
go run ./cmd/goldeneye install sqlite # build the pinned sqlite3 shells once; needs a C compiler
go run ./cmd/goldeneye check # check every engine whose database is available
go run ./cmd/goldeneye check postgresql # check one engine
go run ./cmd/goldeneye check spanner # SPANNER_SERVER_URI=localhost:15000, a Spanner Omni container
go run ./cmd/goldeneye generate [engine] # rewrite the generated files from the database
go test ./... # the same checks as tests; engines without a database skip
```
`generate` and `check` say which engines they skipped for lack of a
database; naming an engine makes its database required.
## What is generated, and what is not
A generator owns only the files it produces; `dialect.json` is always written
by hand, and so are the lists an engine cannot describe. Both commands leave
the hand-written files alone, and the checks do not look at them.
- **`postgresql`** reads a live server named by `POSTGRESQL_SERVER_URI`:
`functions.jsonl` is `pg_catalog`'s functions, `relations.jsonl` the tables
and views of `pg_catalog` and `information_schema`, and each contrib
extension gets a directory under `extensions/` holding the types and
functions `CREATE EXTENSION` adds, so the server needs contrib installed.
A function that one of those extensions puts in `pg_catalog`, as
`adminpack` does, belongs to the extension's directory rather than the
catalog's list. The server has to be the major release pinned in
`postgresql.Major`, since every release adds to the catalogs; the top-level
`types.jsonl` and `operators.jsonl` are hand-written.
- **`mysql`** reads a live server named by `MYSQL_SERVER_URI`, a go-sql-driver
DSN such as `root:mysecretpassword@tcp(127.0.0.1:3306)/mysql`, and writes
into `internal/engine/dolphin/dialect`, since sqlc's MySQL engine is named
after its parser. MySQL keeps no catalog of its types, functions or
operators — the help tables describe a built-in function no further than
its name — so `types.jsonl`, `functions.jsonl` and `operators.jsonl` are
hand-written. What it does describe is its data dictionary:
`relations.jsonl` is every view of `information_schema`, read from
`information_schema` itself, with their names in lower case, since MySQL
matches them in any case and sqlc's parser lowercases every identifier. A
column's type is spelled the way a declaration does, `bigint unsigned`
included, which `types.jsonl` lists as an alias of `bigint`. The other
schemas `mysqld --initialize` creates — `mysql`, `performance_schema`, `sys`
— are tables rather than views, and every table a dialect seeds is one the
analysis core hands codegen as a model, so they are left out until codegen
knows a system schema when it sees one. The server has to be the major
release pinned in `mysql.Major`, since every release adds to
`information_schema`.
- **`duckdb`** reads the DuckDB CLI named by `DUCKDB`, or the one `install`
put in the user cache directory, or `duckdb` on `PATH`: `types.jsonl`,
`functions.jsonl` and `operators.jsonl` come from `duckdb_types()` and
`duckdb_functions()`. The CLI has to be a DuckDB 2.0 build, the release
darkwing is pinned against, which has no release to download yet: until
2.0 is out, `install` downloads the current build of DuckDB's v2.0
preview channel, `duckdb.DefaultVersion`, a rolling tarball per platform
under `artifacts.duckdb.org` with no per-build download and no checksum
to pin, so what a run logs is the version the CLI reports, and
`duckdb.GeneratedFrom` records the build the committed dialect came
from. A check against a later build reports what the later build added;
regenerate, and update `GeneratedFrom`, to move the dialect along. Once
2.0 is released, the installer should pin the release and its checksums
the way the clickhouse one does.
- **`clickhouse`** needs no server: `types.jsonl` comes from
`system.data_type_families` of an ephemeral `clickhouse local` process,
every family that is not an alias becoming a type carrying the spellings
that alias it, with a category decided by its name. The binary is
downloaded once per pinned release by `install` into the user cache
directory, or supplied through the `CLICKHOUSE` environment variable; the
pinned release and the SHA-512 of each platform's download live in
`clickhouse/install.go`, and a download that does not match is discarded.
ClickHouse describes its functions no further than their names, so
`functions.jsonl` is hand-written.
- **`mssql`** reads a live server named by `MSSQL_SERVER_URI`, in any form
the go-mssqldb driver accepts, such as
`sqlserver://sa:password@127.0.0.1:1433?encrypt=disable`. SQL Server keeps
no catalog of its intrinsic functions or its operators — GETDATE and LEN
are not objects — so `types.jsonl` and `functions.jsonl` are hand-written.
What it does describe is its catalog: `relations.jsonl` is every view of
the `sys` and `INFORMATION_SCHEMA` schemas, listed from a database of its
own, since the views a query sees are the ones a user database has and
`master` lists internal views no query can name; each view's columns are
what `sys.dm_exec_describe_first_result_set` says a `SELECT *` from it
returns, the type spelled the way a declaration spells it —
`nvarchar(128)`, `decimal(10,2)`, `varbinary(max)` — and the nullability
the server computes. Names are written in lower case, since SQL Server
matches them in any case under its default collations and sqlc's parser
lowercases every identifier. Both schemas hold nothing but views, so no
table is seeded that codegen would take for a model. The server has to
be the major release pinned in `mssql.Major`, since every release adds
to the catalog views.
- **`spanner`** reads a live Spanner Omni server — the downloadable
Spanner, run from its container image — named by `SPANNER_SERVER_URI`,
the gRPC endpoint such as `localhost:15000`, reached without TLS or
credentials as Omni is, and writes into `internal/engine/googlesql/dialect`,
since sqlc's engine is named after the language Spanner speaks. Spanner
keeps no catalog of its types, functions or operators, so `types.jsonl`,
`functions.jsonl` and `operators.jsonl` are hand-written. What it does
describe is its information schema: `relations.jsonl` is every view of
`INFORMATION_SCHEMA` and `SPANNER_SYS`, read from `INFORMATION_SCHEMA`
itself in a database created for the purpose in the instance Omni's
single server provides, `projects/default/instances/default`. Names are
kept as the catalog spells them, in upper case: Spanner matches a name
in any case, but sqlc's GoogleSQL engine matches one as it is spelled,
so a query reaches these views by their upper-case names until the
engine folds case; a column's type is spelled in lower case the way the seed
spells one, an `ARRAY<T>` as `T` with the array flag, a `STRUCT<a T>` as
`struct(a: t)` and a `PROTO<p.M>` as `proto('p.M')`, since a seed writes
a type's arguments in parentheses. The container image is pinned in the
gen workflow and `docker-compose.yml`.
- **`sqlite`** needs no server either: `functions.jsonl` comes from
`pragma_function_list` of a `sqlite3` shell run against an in-memory
database. Which functions a SQLite has is decided when it is compiled, so
`install` downloads the pinned release's amalgamation, checked against the
SHA3-256 the download page lists, and compiles the shell from it with the
compiler `CC` names, or `cc` — once with the options sqlite.org's own
configure turns on by default, which gives `functions.jsonl`, and once
more per option in `sqlite/install.go`'s extension list, each of which
gets a directory under `extensions/` holding the functions its build adds
over the default one, the way each PostgreSQL contrib extension holds what
`CREATE EXTENSION` adds; a schema that says `CREATE VIRTUAL TABLE ... USING
fts5` loads the option's directory, through the `modules` map in the
hand-written `dialect.json`. SQLite describes its functions as far as their
names, their kinds and the number of arguments each overload takes, and no
further — it types values, not functions — so what each returns and what
its arguments hold is read from the amalgamation: every function is
registered with the C functions that implement it, and those set their
result through `sqlite3_result_*` and read their arguments through
`sqlite3_value_*`. A function a shell reports that the source does not
register fails the run rather than being guessed at. Whether an aggregate
returns NULL over no rows is found by running it over none; the scalar
functions that return NULL for arguments that are not are a short list in
`sqlite/signatures.go`, since a SQLite function returns NULL as often by
setting no result as by saying so. The pinned release is the one the main module's
driver embeds. SQLite has no catalog of types or operators, so
`types.jsonl` and `operators.jsonl` are hand-written. A function that
returns an integer for an integer and a real for a real — `abs`, `ceil`,
`floor`, `trunc`, `sum` — is written once over `any`, returning the real
that a text or blob argument gets, and once more per spelling
`types.jsonl` gives integer and real, returning that type, the way
PostgreSQL's catalog has a `sum` per numeric type; the overload over `any`
comes first, since the legacy compiler resolves by arity alone and takes
it. `install` builds one more shell, for the analysis check below;
nothing is generated from it.
## Layout
- `dialect/` — the record types the files are made of, mirrored from
`internal/core/seed`, and the helpers that write a generated set of files
into an engine directory or diff it against what is committed.
- `endtoend/` — finds the analyze cases, splits their query files, and
compares an engine's answer with a case's committed output.
- `analysis/` — the shape of that answer: the JSON `sqlc analyze` prints.
- `postgresql/`, `mysql/`, `mssql/`, `spanner/`, `duckdb/`, `clickhouse/`,
`sqlite/` — one package per engine, each exposing `Locate`, `Version` and
`Generate`, `Analyze` where the engine has an analysis check, and tests
that run the checks.
- `cmd/goldeneye/` — the command.
## Analysis checks
`check` also verifies the `analyze_*` cases under `internal/endtoend/testdata`
against what the database itself reports. A case is an
`analyze_<name>/<engine>` directory whose `exec.json` runs the analyze
command; `endtoend/` finds them. The engine package loads the case's
`schema.sql` and optional `fixture.sql` into the database, runs `query.sql`
there, prints what the database reports in the JSON shape `sqlc analyze`
prints, and compares it with the committed `stdout.json` byte for byte. A
difference means sqlc's analysis disagrees with the database. A case that
asks for `--ast` is skipped, since only sqlc can print that.
- **`clickhouse`** runs each case in an ephemeral `clickhouse local` process.
Column types come from the executed query's result header, provenance from
`EXPLAIN QUERY TREE`, and parameters from sentinel constants substituted for
`?`, `sqlc.arg()` and `sqlc.narg()`, since ClickHouse itself never sees a
placeholder; `INSERT ... VALUES` parameters map onto `DESCRIBE TABLE`.
- **`sqlite`** runs each case through one more shell `install` builds, under
`analysis/`: every extension option at once, so that any case's schema
loads, and `SQLITE_ENABLE_COLUMN_METADATA`, which lets the shell's `.stats
stmt` say which table column each result column of a statement is read
from. That column's declared type is the result column's, and its NOT
NULL decides nullability, the rowid counting as NOT NULL. SQLite types
values rather than expressions, so a column the library has no origin for
— an aggregate, an arithmetic result — is typed by the storage class of
the value it returns, which is why a case wants a `fixture.sql`: the query
is run over the fixture, with each parameter bound to a value of the
column it stands in for, and again over no rows, and a column is nullable
when either run returns a NULL for it — an aggregate over nothing, the far
side of an outer join. The library reports nothing about a parameter but
its number, so parameters are found in the bytecode `EXPLAIN` prints, the
way ClickHouse's are found in its query tree: each is followed from the
register its `Variable` loads, through copies and the expressions it is an
argument of, to the first opcode that uses it against something the
catalog can name — the other operand of a comparison, the row a seek lands
on, the position in the record an `Insert` writes, the column of an IN
list's ephemeral table it comes back out of. One that reaches nothing
nameable is described by what the program requires of it, when it
requires anything: `MustBeInt` makes LIMIT's an integer. `sqlc.arg(name)`
becomes `?N`, numbered as sqlc numbers them, so a repeated name is one
parameter. Two things the check reports that sqlc does not: a bare column
selected alongside an aggregate is NULL over no rows, and so nullable, and
a comparison such as `x IS NULL` is an integer, since that is what SQLite
returns.
- **`mysql`** runs each case in a database of its own on the server named by
`MYSQL_SERVER_URI`, and asks the server three things about a query. What a
driver sees: the query is run, with every parameter a user variable set to
NULL, and each result column's name, type and nullability are read from
the result set's metadata as go-sql-driver reports them. What the resolver
made of it: the optimizer trace prints each query block back after name
resolution and before optimisation, with every column qualified, every
alias kept and every `SELECT *` expanded, which says which table a result
column is read from and what a parameter is compared with or assigned to.
And for a statement the trace does not expand — an `INSERT ... VALUES`, a
single-table `UPDATE` or `DELETE` — the note `EXPLAIN` leaves, which prints
the statement the same way; a `SELECT` cannot be read from the note, since
it is printed after optimisation, and a lookup on a unique key against an
empty table has folded to `NULL = (@x)` there. Views and derived tables are
kept as the query wrote them rather than merged, so that a column read
through one is reported as its column and an `information_schema` view is
not resolved away into the dictionary tables behind it. MySQL itself
reports nothing about a parameter but its position, so a parameter is
described by its partner: a column's type and nullability come from
`information_schema`, a column of a derived table or CTE from what its
block projects, and an expression's from running it, over the tables it
reads, as a query of its own; a `LIMIT` or `OFFSET` count is a `bigint
unsigned`. Two things the driver keeps to itself: the table a result
column comes from, which is why the trace is read for it, and the length
beside the one wire type every size of `TEXT` and `BLOB` is sent as, which
is why a column read from a table, directly or through a derived table,
is spelled the way the table declares it.
- **`duckdb`** runs each case through the CLI, which loads the schema and
fixture into an in-memory database of their own, one process per
question, and is asked four things about each query. What its
parameters are: the query is prepared and explained as JSON with a
string sentinel bound to each parameter, `EXECUTE q('goldeneye_1',
...)`, and the unoptimized logical plan the CLI prints under
`explain_output = 'all'` shows each as `CAST('goldeneye_k' AS T)`, `T`
being the type the binder gave the parameter, or bare when that type is
`VARCHAR`; a sentinel the binder converts on the spot, as an `INSERT`'s
`VALUES` are, is bound to NULL instead. What its result columns are: `DESCRIBE`, with each parameter
replaced by a NULL of its type, names and types them; DuckDB describes
no DML, so a `RETURNING` column is the target table's column it names.
Which table a result column is read from and which column a parameter
stands in for: DuckDB prints a plan with every column by its bare name
and every aliased expression by its alias, so these are read from the
query text, the select list's items, a star expanded to its table's
columns, and the operand beside each parameter, resolved against the
`FROM` clause and the catalog, `duckdb_columns()`, from which a column
read from a table takes its declared type and nullability; a parameter
the query casts takes the cast's type as DuckDB spells it, and one
inside a subquery, whose tables the statement's scope does not name,
takes the binder's. And whether
an expression can be NULL, which DuckDB does not track: the query is
run, with each parameter bound to a value of its type, over the fixture
and over no rows, and a column is nullable when either run returns a
NULL for it. DuckDB spells an enum column by its labels whether the
schema named the type or not, so labels that are those of an enum the
schema created name that type, and a spelling `types.jsonl` lists as an
alias — `json`, which DuckDB's own catalog lists as a spelling of
`varchar` — is reported by the dialect's name for it.
- **`mssql`** describes each case in a database of its own on the server
named by `MSSQL_SERVER_URI`, without running anything: the schema is
loaded one statement at a time, since a `CREATE TYPE` has to be its own
batch before a table can use the type, and the server is asked three
things about each query. What a driver would see:
`sys.dm_exec_describe_first_result_set` describes each result column —
its name, its type spelled the way a declaration spells it, whether it
can be NULL, and which table column it is read from. What each parameter
would be: `sp_describe_undeclared_parameters` says what type the server
would give each parameter the query leaves undeclared, which is the type
of a `CAST(@p AS T)`; it describes a parameter only when it is used
once, so each appearance of a repeated one becomes a variable of its
own. And what each parameter stands in for: the estimated showplan,
compiled with `SET SHOWPLAN_XML ON` and the parameters declared as those
types, prints every column as a reference naming its table and every
variable as `@variable`, and a parameter's partner is the column on the
other side of the `Compare` it is an operand of, the column an `Assign`
sets to it, or the column of a seek whose range expression it is, with a
named expression such as `Expr1002` followed to its definition; a
parameter under a `CONVERT` the query wrote has no partner, since the
cast says what it is. A parameter with a partner is described as that
column, from the catalog of the case's database. Three things the
describing function keeps to itself: a `json` or `vector` column is
described by the `nvarchar(max)` it is sent to a driver as, so a column
read from a table is typed from `sys.columns` instead; a type the
schema created is reported beside the system type it stands on, and the
dialect reports it by its own name; and a spelling `types.jsonl` lists
as an alias — `numeric`, `timestamp` — is reported by the dialect's name
for it, `decimal`, `rowversion`. One thing it says that sqlc does not: a
computed column — a cast, an arithmetic — is nullable whatever its
arguments, since a conversion that fails under `ANSI_WARNINGS OFF`
yields NULL, where sqlc follows the arguments, and a computed column
names the column it is computed from, as the one an update through the
result set would write, where sqlc gives an expression no table. The
cases compute over nullable columns, and a computed column is reported
without a table.
- **`spanner`** creates a database of each case's own from its schema in
the instance named by `SPANNER_SERVER_URI`, writes its fixture there in a
read-write transaction, and compiles each query in `PLAN` mode, which
runs nothing: a DML statement is compiled in a read-write transaction
that is rolled back. The server reports the name and type of each
result column and the type of each parameter the query leaves
undeclared, the way the wire spells them — `STRING`, `ARRAY<INT64>` —
without the length a declaration gives a `STRING(10)` or whether a
column can be NULL, and the query plan, which says where each comes
from: the children of `Serialize Result` after the relation it
serializes are the result columns, a `Scan` of a table defines a
variable per column it reads, and a comparison is a `Function` whose
description reads `($col = @param)`. A result column the plan reads
from a table column, directly or through the variables a join or a
batch passes it through, is described from the case's
`INFORMATION_SCHEMA`, with its declared length and nullability; one the
plan reads as a parameter is the column the parameter is compared with,
which is why the optimizer substituted it. A parameter compared with a
table column is described as that column. A DML plan lists the values
it writes before the columns it returns — the table's key columns, then
the columns an `UPDATE` sets or an `INSERT` inserts, read from the
statement — and a parameter written to a column is described as it,
while a `THEN RETURN` column is the table's column of that name. Two
things Spanner does not say: whether an expression can be NULL, which
is reported as it is not, and anything about a `STRUCT` returned as a
column, which Spanner rejects.
The other engines have no analysis check yet.

64
internal/goldeneye/analysis/analysis.go Normal file
View File

@@ -0,0 +1,64 @@
// Package analysis is the shape of an engine's answer about a query: the
// JSON `sqlc analyze` prints, so that a case's committed stdout.json can be
// compared with what the database itself reports, byte for byte.
package analysis
import (
"bytes"
"encoding/json"
)
// Query is what was found out about one query.
type Query struct {
Name string `json:"name"`
Cmd string `json:"cmd"`
Columns []Column `json:"columns"`
Params []Param `json:"params"`
}
// Column describes a result column, or the column a parameter stands in
// for.
type Column struct {
Name string `json:"name"`
Type *TypeExpr `json:"type,omitempty"`
Table string `json:"table,omitempty"`
}
// Param is one parameter and what it is compared with or assigned to.
type Param struct {
Number int `json:"number"`
Column Column `json:"column"`
}
// TypeExpr is a type as a call expression: a lowercased name applied to an
// ordered argument list, each argument another type, an integer, a boolean
// or a quoted string, optionally labelled. Nullability is an attribute of
// the type rather than a wrapper.
type TypeExpr struct {
Name string `json:"name"`
Nullable bool `json:"nullable,omitempty"`
Args []TypeArg `json:"args,omitempty"`
}
// TypeArg is one argument of a TypeExpr: a type, an integer, a boolean, a
// quoted string, or an identifier — a bare word that is not a type, such as
// the function an AggregateFunction names.
type TypeArg struct {
Label string `json:"label,omitempty"`
Type *TypeExpr `json:"type,omitempty"`
Int *int64 `json:"int,omitempty"`
Bool *bool `json:"bool,omitempty"`
String *string `json:"string,omitempty"`
Ident *string `json:"ident,omitempty"`
}
// Encode prints the answer the way sqlc analyze does.
func Encode(queries []Query) ([]byte, error) {
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetIndent("", " ")
if err := enc.Encode(queries); err != nil {
return nil, err
}
return buf.Bytes(), nil
}

272
internal/goldeneye/clickhouse/analyze.go Normal file
View File

@@ -0,0 +1,272 @@
package clickhouse
import (
"context"
"encoding/json"
"fmt"
"regexp"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// analyze runs every query against the schema and fixture and records what
// ClickHouse reports about each.
func analyze(ctx context.Context, l local, schema, fixture string, queries []endtoend.Query) ([]analysis.Query, error) {
out := make([]analysis.Query, 0, len(queries))
for _, q := range queries {
aq, err := analyzeQuery(ctx, l, schema, fixture, q)
if err != nil {
return nil, fmt.Errorf("%s: %w", q.Name, err)
}
out = append(out, aq)
}
return out, nil
}
func analyzeQuery(ctx context.Context, l local, schema, fixture string, q endtoend.Query) (analysis.Query, error) {
sql, phs := bindPlaceholders(q.SQL)
explain := returnsRows(sql)
var script strings.Builder
for _, stmt := range []string{schema, fixture} {
if s := strings.TrimRight(strings.TrimSpace(stmt), ";"); s != "" {
script.WriteString(s)
script.WriteString(";\n")
}
}
if explain {
script.WriteString("EXPLAIN QUERY TREE " + sql + ";\n")
}
script.WriteString(sql + ";\n")
results, err := l.run(ctx, script.String())
if err != nil {
return analysis.Query{}, err
}
aq := analysis.Query{
Name: q.Name,
Cmd: q.Cmd,
Columns: []analysis.Column{},
Params: []analysis.Param{},
}
if !explain {
return analyzeExec(ctx, l, script.String(), sql, phs, aq)
}
if len(results) != 2 {
return analysis.Query{}, fmt.Errorf("expected the query tree and one result set, got %d results", len(results))
}
var lines []string
for _, row := range results[0].Data {
var line string
if err := json.Unmarshal(row["explain"], &line); err != nil {
return analysis.Query{}, fmt.Errorf("reading query tree: %w", err)
}
lines = append(lines, line)
}
tree, err := parseQueryTree(lines)
if err != nil {
return analysis.Query{}, err
}
// Names and types come from the block header of the executed query, the
// same header a driver sees. The tree only adds where each came from.
_, nodes := projection(firstNode(tree.root.children))
for i, col := range results[1].Meta {
ac := column(col.Name, col.Type)
if i < len(nodes) && len(nodes) == len(results[1].Meta) {
ac.Table = tree.sourceTable(nodes[i])
}
aq.Columns = append(aq.Columns, ac)
}
sentinels := tree.sentinels()
for i, ph := range phs {
ac := analysis.Column{}
if sentinel := sentinels[i+1]; sentinel != nil {
ac = tree.paramColumn(sentinel)
}
if ph.Name != "" {
ac.Name = ph.Name
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}
func column(name, typ string) analysis.Column {
if typ == "" {
return analysis.Column{Name: name}
}
return analysis.Column{Name: name, Type: parseType(typ)}
}
// returnsRows reports whether a statement produces a result set and so can
// be explained as a query tree.
func returnsRows(sql string) bool {
head := strings.ToLower(strings.TrimSpace(sql))
if strings.HasPrefix(head, "(") {
return true
}
for _, kw := range []string{"select", "with", "show", "describe", "desc", "exists"} {
if strings.HasPrefix(head, kw) && (len(head) == len(kw) || !isWordByte(head[len(kw)])) {
return true
}
}
return false
}
// sentinels finds the constants the placeholders were substituted with,
// keyed by placeholder ordinal.
func (t *queryTree) sentinels() map[int]*treeNode {
found := map[int]*treeNode{}
var walk func(n *treeNode)
walk = func(n *treeNode) {
if n.kind == "CONSTANT" {
if k, ok := sentinelOrdinal(n); ok {
found[k] = n
return
}
}
for _, c := range n.children {
walk(c)
}
}
walk(t.root)
return found
}
// sentinelOrdinal recognises a constant folded directly from `NULL + k` or
// `4294967295 + k`, the shapes sentinelFor produces, and returns k. A
// constant folded from a larger expression that merely contains a sentinel
// does not match; the sentinel is found nested inside it instead.
func sentinelOrdinal(c *treeNode) (int, bool) {
fn := firstNode(c.section("EXPRESSION").childrenOrNil())
if fn == nil || fn.kind != "FUNCTION" || fn.attrs["function_name"] != "plus" {
return 0, false
}
args := fn.section("ARGUMENTS").list()
if len(args) != 2 || args[0].kind != "CONSTANT" || args[1].kind != "CONSTANT" {
return 0, false
}
base := args[0].attrs["constant_value"]
if base != "NULL" && base != "UInt64_"+limitBase {
return 0, false
}
k, err := strconv.Atoi(strings.TrimPrefix(args[1].attrs["constant_value"], "UInt64_"))
if err != nil {
return 0, false
}
return k, true
}
// paramColumn describes what a placeholder is compared with or assigned to:
// the other operand of the function it is an argument of, preferring a
// column over an expression, or the projected column it stands for.
func (t *queryTree) paramColumn(sentinel *treeNode) analysis.Column {
list := sentinel.parent
if list != nil && list.kind == "LIST" && list.parent != nil {
switch owner := list.parent; {
case owner.kind == "" && owner.text == "ARGUMENTS":
// Prefer a column operand, then an expression, then a constant.
rank := map[string]int{"COLUMN": 0, "FUNCTION": 1, "CONSTANT": 2}
var best *treeNode
for _, sib := range list.children {
if sib == sentinel {
continue
}
if r, ok := rank[sib.kind]; ok && (best == nil || r < rank[best.kind]) {
best = sib
}
}
if best != nil {
return t.describe(best)
}
case owner.kind == "" && owner.text == "PROJECTION":
names, nodes := projection(owner.parent)
for i, n := range nodes {
if n == sentinel && i < len(names) {
ac := column(names[i], sentinel.attrs["constant_value_type"])
return ac
}
}
}
}
return column("", sentinel.attrs["constant_value_type"])
}
// describe turns a tree expression into a column description.
func (t *queryTree) describe(n *treeNode) analysis.Column {
switch n.kind {
case "COLUMN":
ac := column(n.attrs["column_name"], n.attrs["result_type"])
ac.Table = t.sourceTable(n)
return ac
case "FUNCTION":
return column(n.attrs["function_name"], n.attrs["result_type"])
case "CONSTANT":
// A constant folded from a function call, such as toDate(now()),
// still names the function, which is what a placeholder compared
// with it is named after.
name := ""
if fn := firstNode(n.section("EXPRESSION").childrenOrNil()); fn != nil && fn.kind == "FUNCTION" {
name = fn.attrs["function_name"]
}
return column(name, n.attrs["constant_value_type"])
}
return analysis.Column{}
}
var insertValuesRe = regexp.MustCompile(`(?is)^insert\s+into\s+(?:table\s+)?([\w.` + "`" + `"]+)\s*(?:\(([^)]*)\))?\s*(?:format\s+)?values\b`)
// analyzeExec runs a statement that returns no rows. The only parameters it
// can describe are those of an INSERT ... VALUES, which map positionally
// onto the target columns reported by DESCRIBE TABLE.
func analyzeExec(ctx context.Context, l local, script, sql string, phs []placeholder, aq analysis.Query) (analysis.Query, error) {
m := insertValuesRe.FindStringSubmatch(sql)
if m != nil {
script += "DESCRIBE TABLE " + m[1] + ";\n"
}
results, err := l.run(ctx, script)
if err != nil {
return analysis.Query{}, err
}
var targets []analysis.Column
if m != nil && len(results) == 1 {
byName := map[string]analysis.Column{}
var all []analysis.Column
table := strings.Trim(m[1][strings.LastIndexByte(m[1], '.')+1:], "`\"")
for _, row := range results[0].Data {
var name, typ string
json.Unmarshal(row["name"], &name)
json.Unmarshal(row["type"], &typ)
ac := column(name, typ)
ac.Table = table
byName[name] = ac
all = append(all, ac)
}
if strings.TrimSpace(m[2]) == "" {
targets = all
} else {
for _, name := range strings.Split(m[2], ",") {
targets = append(targets, byName[strings.Trim(strings.TrimSpace(name), "`\"")])
}
}
}
for i, ph := range phs {
ac := analysis.Column{}
if len(targets) > 0 {
ac = targets[i%len(targets)]
}
if ph.Name != "" {
ac.Name = ph.Name
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}

43
internal/goldeneye/clickhouse/check.go Normal file
View File

@@ -0,0 +1,43 @@
package clickhouse
import (
"context"
"os"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// Analyze runs a case's queries through the clickhouse binary and returns
// the analysis in the JSON shape sqlc analyze prints.
func Analyze(ctx context.Context, binary string, c endtoend.Case) ([]byte, error) {
schema, err := os.ReadFile(c.Schema)
if err != nil {
return nil, err
}
var fixture []byte
if c.Fixture != "" {
if fixture, err = os.ReadFile(c.Fixture); err != nil {
return nil, err
}
}
queries, err := c.Queries()
if err != nil {
return nil, err
}
out, err := analyze(ctx, local{binary: binary}, string(schema), string(fixture), queries)
if err != nil {
return nil, err
}
return analysis.Encode(out)
}
// Check compares what ClickHouse reports for a case with the output the
// case committed, returning a diff when they differ.
func Check(ctx context.Context, binary string, c endtoend.Case) (string, error) {
got, err := Analyze(ctx, binary, c)
if err != nil {
return "", err
}
return c.Compare(got)
}

146
internal/goldeneye/clickhouse/clickhouse.go Normal file
View File

@@ -0,0 +1,146 @@
// Package clickhouse generates the ClickHouse dialect seed under
// internal/engine/clickhouse/dialect from a clickhouse binary, run as an
// ephemeral `clickhouse local` process that needs no server.
//
// types.jsonl comes from system.data_type_families: every family that is not
// an alias becomes a type, the families that alias it become its aliases,
// and its category is decided by its name, since ClickHouse records no such
// thing. ClickHouse describes its functions no further than their names —
// system.functions carries no signatures — so functions.jsonl is written by
// hand and is not this package's business.
//
// The package also verifies the ClickHouse analyze cases under
// internal/endtoend/testdata against the same binary: each case's schema and
// fixture are loaded into a `clickhouse local` process and its queries run
// there. Result column types come from the executed query's result header,
// provenance from EXPLAIN QUERY TREE, and parameters from sentinel constants
// substituted for the placeholders, since ClickHouse itself never sees a ?.
// The answer is printed in the JSON shape sqlc analyze prints and compared
// with the case's committed stdout.json byte for byte.
//
// The binary is downloaded once per pinned version by Install, or supplied
// through the CLICKHOUSE environment variable.
package clickhouse
import (
"context"
"encoding/json"
"fmt"
"os/exec"
"sort"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name of the engine directory the dialect lives under.
const Engine = "clickhouse"
// typeFamilies lists every data type ClickHouse knows and, for a spelling
// such as INT or VARCHAR, the type it stands for.
const typeFamilies = `
SELECT name, alias_to
FROM system.data_type_families
ORDER BY name`
// Version reports the release a binary is.
func Version(ctx context.Context, binary string) (string, error) {
out, err := exec.CommandContext(ctx, binary, "local", "--version").Output()
if err != nil {
return "", fmt.Errorf("clickhouse local --version: %w", err)
}
return strings.TrimSpace(string(out)), nil
}
// Generate reads the dialect from the binary.
func Generate(ctx context.Context, binary string) (dialect.Files, error) {
results, err := local{binary: binary}.run(ctx, typeFamilies)
if err != nil {
return nil, err
}
if len(results) != 1 {
return nil, fmt.Errorf("clickhouse: expected one result set, got %d", len(results))
}
var families []family
for _, row := range results[0].Data {
var f family
if err := json.Unmarshal(row["name"], &f.Name); err != nil {
return nil, fmt.Errorf("clickhouse: system.data_type_families: %w", err)
}
if err := json.Unmarshal(row["alias_to"], &f.AliasTo); err != nil {
return nil, fmt.Errorf("clickhouse: system.data_type_families: %w", err)
}
families = append(families, f)
}
types, err := dialect.JSONL(readTypes(families))
if err != nil {
return nil, err
}
return dialect.Files{dialect.TypesFile: types}, nil
}
type family struct {
Name string
AliasTo string
}
// readTypes turns the families into types: one per family that is not an
// alias, carrying the spellings that alias it. The seed reads names without
// regard to case, so an alias that only differs in case from its type —
// bool for Bool, ENUM for Enum — is dropped.
func readTypes(families []family) []dialect.Type {
byName := map[string]*dialect.Type{}
var names []string
for _, f := range families {
if f.AliasTo != "" {
continue
}
byName[f.Name] = &dialect.Type{Name: f.Name, Category: category(f.Name)}
names = append(names, f.Name)
}
for _, f := range families {
if f.AliasTo == "" || strings.EqualFold(f.Name, f.AliasTo) {
continue
}
if t, ok := byName[f.AliasTo]; ok {
t.Aliases = append(t.Aliases, f.Name)
}
}
sort.Strings(names)
types := make([]dialect.Type, 0, len(names))
for _, name := range names {
types = append(types, *byName[name])
}
return types
}
// category assigns a type the PostgreSQL category letter the seed package
// uses, by the family's name: N for numbers, B for booleans, S for strings
// and the identifiers stored as them, D for dates and times, T for time
// spans, A for arrays and U for everything else.
func category(name string) string {
switch {
case name == "Bool":
return "B"
case name == "Array":
return "A"
case strings.HasPrefix(name, "Interval"):
return "T"
case hasPrefix(name, "UInt", "Int", "Float", "BFloat", "Decimal"):
return "N"
case name == "String", name == "FixedString", name == "UUID", name == "IPv4", name == "IPv6":
return "S"
case hasPrefix(name, "Date", "Time"):
return "D"
}
return "U"
}
func hasPrefix(name string, prefixes ...string) bool {
for _, prefix := range prefixes {
if strings.HasPrefix(name, prefix) {
return true
}
}
return false
}

66
internal/goldeneye/clickhouse/clickhouse_test.go Normal file
View File

@@ -0,0 +1,66 @@
package clickhouse
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// TestDialect verifies the committed ClickHouse dialect against what the
// pinned clickhouse binary reports. It skips unless the binary is installed.
func TestDialect(t *testing.T) {
binary, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, binary)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, binary)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Engine)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}
// TestAnalyzeCases verifies every ClickHouse analyze case under
// internal/endtoend/testdata against what ClickHouse reports. It skips
// unless the binary is installed.
func TestAnalyzeCases(t *testing.T) {
binary, err := Locate()
if err != nil {
t.Skip(err)
}
cases, err := endtoend.Cases(Engine)
if err != nil {
t.Fatal(err)
}
if len(cases) == 0 {
t.Fatal("no clickhouse analyze cases found")
}
for _, c := range cases {
t.Run(c.Name, func(t *testing.T) {
diff, err := Check(context.Background(), binary, c)
if err != nil {
t.Fatal(err)
}
if diff != "" {
t.Errorf("%s does not match what ClickHouse reports (-committed +clickhouse):\n%s", c.Output, diff)
}
})
}
}

221
internal/goldeneye/clickhouse/install.go Normal file
View File

@@ -0,0 +1,221 @@
package clickhouse
import (
"archive/tar"
"compress/gzip"
"context"
"crypto/sha512"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
)
// DefaultVersion is the ClickHouse release the dialect is generated from.
// Bumping it is a deliberate change: the query tree format and type
// inference can shift between releases, so regenerate and review the dialect
// after changing it, and add the new release's assets to the table below.
const DefaultVersion = "25.8.2.29"
// asset is one downloadable build of ClickHouse. Linux builds are published
// as clickhouse-common-static tarballs holding the binary at
// usr/bin/clickhouse; macOS builds are published as bare binaries.
type asset struct {
Version string
OS string
Arch string
Name string
SHA512 string
}
// assets lists every build Install knows how to fetch, with the SHA-512 of
// the download. A version that is not in this table cannot be installed:
// verifying the download is the point of the table.
//
// The tarball checksums are the ones in the .sha512 files ClickHouse
// publishes next to them. ClickHouse publishes no checksum for the macOS
// binaries, so those were computed from the downloads.
var assets = []asset{
{"25.8.2.29", "linux", "amd64", "clickhouse-common-static-25.8.2.29-amd64.tgz", "6ff0aa1ffac6e564970174422ecde0d645cdb96812247a6e544d39cad6d78a514265f90a2bc7b4bad49903cea96eddd16a415a45b2aeaf9164461be76331bdee"},
{"25.8.2.29", "linux", "arm64", "clickhouse-common-static-25.8.2.29-arm64.tgz", "68204ca4d4e472790f808ee376251fae82e58066a31f35a40d15d442ce5988d697f18a1208d28b8bb8e2dfad4b20b7fcb5107e2178472abcd97251b8de7f058e"},
{"25.8.2.29", "darwin", "amd64", "clickhouse-macos", "2805805ad2506e37a3e71b4ae9e797bdc010a9368dc28e99bcaaa2c70a72cfdd031c0fce8fc304248fad73211d28d645f75c6432cfae1e1e54d72d04e8626cd4"},
{"25.8.2.29", "darwin", "arm64", "clickhouse-macos-aarch64", "4c9237e85c8d4e1aced2b339b32e086b4f23fa14f99754b056a7e33dda88c4fb5d52e09e29d20181ec5b580caadb00f36613802e71755ff34927535eb8babf79"},
}
// releaseTag returns the GitHub release tag for a version. ClickHouse tags
// its March and August releases as LTS and everything else as stable.
func releaseTag(version string) (string, error) {
parts := strings.Split(version, ".")
if len(parts) != 4 {
return "", fmt.Errorf("invalid ClickHouse version %q: want MAJOR.MINOR.PATCH.BUILD", version)
}
minor, err := strconv.Atoi(parts[1])
if err != nil {
return "", fmt.Errorf("invalid ClickHouse version %q: %w", version, err)
}
suffix := "-stable"
if minor == 3 || minor == 8 {
suffix = "-lts"
}
return "v" + version + suffix, nil
}
// releaseAsset finds the build for a platform in the table.
func releaseAsset(version, goos, goarch string) (asset, error) {
for _, a := range assets {
if a.Version == version && a.OS == goos && a.Arch == goarch {
return a, nil
}
}
for _, a := range assets {
if a.Version == version {
return asset{}, fmt.Errorf("no ClickHouse %s build is listed for %s/%s", version, goos, goarch)
}
}
return asset{}, fmt.Errorf("ClickHouse %s is not in the asset table; add its downloads and checksums to install.go", version)
}
// url is the asset's download address on GitHub.
func (a asset) url() (string, error) {
tag, err := releaseTag(a.Version)
if err != nil {
return "", err
}
return "https://github.com/ClickHouse/ClickHouse/releases/download/" + tag + "/" + a.Name, nil
}
// tarball reports whether the download is an archive rather than the binary.
func (a asset) tarball() bool {
return strings.HasSuffix(a.Name, ".tgz")
}
// cachedBinary is where Install puts the binary for a version.
func cachedBinary(version string) (string, error) {
dir, err := os.UserCacheDir()
if err != nil {
return "", err
}
return filepath.Join(dir, "sqlc-clickhouse", version, "clickhouse"), nil
}
// Locate finds a clickhouse binary: the CLICKHOUSE environment variable wins,
// then the cached copy of DefaultVersion.
func Locate() (string, error) {
if path := os.Getenv("CLICKHOUSE"); path != "" {
return path, nil
}
path, err := cachedBinary(DefaultVersion)
if err != nil {
return "", err
}
if _, err := os.Stat(path); err != nil {
return "", fmt.Errorf("clickhouse %s is not installed: run `go run ./cmd/goldeneye install clickhouse` in internal/goldeneye, or set CLICKHOUSE to a clickhouse binary", DefaultVersion)
}
return path, nil
}
// Install downloads the clickhouse binary for a version into the cache and
// returns its path. It is a no-op when the version is already cached. The
// download is checked against the table's SHA-512 before it is installed.
func Install(ctx context.Context, version, goos, goarch string, progress io.Writer) (string, error) {
dest, err := cachedBinary(version)
if err != nil {
return "", err
}
if _, err := os.Stat(dest); err == nil {
return dest, nil
}
a, err := releaseAsset(version, goos, goarch)
if err != nil {
return "", err
}
url, err := a.url()
if err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", err
}
fmt.Fprintf(progress, "downloading %s\n", url)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("downloading %s: %s", url, resp.Status)
}
// Write next to the destination and rename so a partial or corrupt
// download never masquerades as an installed binary.
tmp, err := os.CreateTemp(filepath.Dir(dest), "clickhouse-*.partial")
if err != nil {
return "", err
}
defer os.Remove(tmp.Name())
// Hash every byte that comes off the wire, including the parts of a
// tarball after the binary, which extraction would otherwise not read.
sum := sha512.New()
body := io.TeeReader(resp.Body, sum)
var src io.Reader = body
if a.tarball() {
src, err = binaryInTarball(body)
if err != nil {
return "", fmt.Errorf("downloading %s: %w", url, err)
}
}
if _, err := io.Copy(tmp, src); err != nil {
tmp.Close()
return "", err
}
if _, err := io.Copy(io.Discard, body); err != nil {
tmp.Close()
return "", err
}
if err := tmp.Close(); err != nil {
return "", err
}
if got := hex.EncodeToString(sum.Sum(nil)); got != a.SHA512 {
return "", fmt.Errorf("downloading %s: SHA-512 mismatch: got %s, want %s", url, got, a.SHA512)
}
if err := os.Chmod(tmp.Name(), 0o755); err != nil {
return "", err
}
if err := os.Rename(tmp.Name(), dest); err != nil {
return "", err
}
return dest, nil
}
// binaryInTarball positions a reader at the clickhouse binary inside a
// clickhouse-common-static tarball.
func binaryInTarball(r io.Reader) (io.Reader, error) {
gz, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
tr := tar.NewReader(gz)
for {
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
return nil, errors.New("tarball does not contain usr/bin/clickhouse")
}
if err != nil {
return nil, err
}
if hdr.Typeflag == tar.TypeReg && strings.HasSuffix(hdr.Name, "/usr/bin/clickhouse") {
return tr, nil
}
}
}

82
internal/goldeneye/clickhouse/local.go Normal file
View File

@@ -0,0 +1,82 @@
package clickhouse
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
)
// local runs SQL through an ephemeral `clickhouse local` process.
type local struct {
binary string
}
// resultSet is one JSON-format result printed by clickhouse local. Only
// statements that return rows print one; DDL and INSERT print nothing.
type resultSet struct {
Meta []resultColumn `json:"meta"`
Data []map[string]json.RawMessage `json:"data"`
Rows int `json:"rows"`
}
type resultColumn struct {
Name string `json:"name"`
Type string `json:"type"`
}
// run executes a multi-statement script in a fresh process with fresh
// storage and returns the result sets in statement order. Any statement
// failing fails the whole run with ClickHouse's own error message.
func (l local) run(ctx context.Context, script string) ([]resultSet, error) {
dir, err := os.MkdirTemp("", "sqlc-goldeneye-clickhouse-*")
if err != nil {
return nil, err
}
defer os.RemoveAll(dir)
queries := filepath.Join(dir, "queries.sql")
if err := os.WriteFile(queries, []byte(script), 0o600); err != nil {
return nil, err
}
// stdin must not be inherited: clickhouse local reads it as table data
// and blocks until it is closed.
cmd := exec.CommandContext(ctx, l.binary, "local",
"--multiquery",
"--queries-file", queries,
"--output-format", "JSON",
"--path", filepath.Join(dir, "data"),
)
cmd.Stdin = nil
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(stderr.String())
if msg == "" {
msg = err.Error()
}
return nil, errors.New(msg)
}
var results []resultSet
dec := json.NewDecoder(&stdout)
for {
var rs resultSet
err := dec.Decode(&rs)
if errors.Is(err, io.EOF) {
return results, nil
}
if err != nil {
return nil, fmt.Errorf("decoding clickhouse local output: %w", err)
}
results = append(results, rs)
}
}

68
internal/goldeneye/clickhouse/queries.go Normal file
View File

@@ -0,0 +1,68 @@
package clickhouse
import (
"fmt"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// placeholder is one parameter reference in a query, in order of appearance.
type placeholder struct {
Number int
Name string // sqlc.arg / sqlc.narg name, empty for ?
}
// Placeholders are substituted with constant expressions that carry their
// ordinal, so they can be told apart from each other and from literal NULLs
// once ClickHouse has folded them: the query tree prints the expression a
// folded constant came from. NULL coerces to any type, so a comparison
// against it analyzes with the other operand's type. LIMIT and OFFSET
// reject NULL and only accept unsigned integers, so those get a value no
// query would plausibly contain.
const limitBase = "4294967295"
func sentinelFor(lastWord string, ordinal int) string {
switch strings.ToLower(lastWord) {
case "limit", "offset":
return fmt.Sprintf("toUInt64(%s + %d)", limitBase, ordinal)
}
return fmt.Sprintf("(NULL + %d)", ordinal)
}
// bindPlaceholders rewrites sqlc's parameter syntax (?, sqlc.arg(name),
// sqlc.narg(name)) into constants ClickHouse can analyze. ClickHouse binds
// every ? positionally, so each placeholder is its own parameter even when
// a name repeats, which is how sqlc numbers them too.
func bindPlaceholders(sql string) (string, []placeholder) {
var phs []placeholder
out := endtoend.Rewrite(sql, func(name, lastWord string) string {
phs = append(phs, placeholder{Number: len(phs) + 1, Name: name})
return sentinelFor(lastWord, len(phs))
})
return out, phs
}
func isWordByte(c byte) bool {
return c == '_' || c == '.' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'
}
// skipQuoted returns the index just past the quoted token starting at i,
// honouring backslash escapes and doubled quotes.
func skipQuoted(s string, i int) int {
q := s[i]
j := i + 1
for j < len(s) {
switch {
case s[j] == '\\' && j+1 < len(s):
j += 2
case s[j] == q && j+1 < len(s) && s[j+1] == q:
j += 2
case s[j] == q:
return j + 1
default:
j++
}
}
return len(s)
}

236
internal/goldeneye/clickhouse/tree.go Normal file
View File

@@ -0,0 +1,236 @@
package clickhouse
import (
"fmt"
"regexp"
"strconv"
"strings"
)
// treeNode is one line of EXPLAIN QUERY TREE output. Lines of the form
// `KIND id: N, key: value, ...` are nodes with a kind and attributes; every
// other line (section headers such as PROJECTION or ARGUMENTS, and the
// `name Type` lines under PROJECTION COLUMNS) keeps only its text.
type treeNode struct {
kind string
id int
attrs map[string]string
text string
parent *treeNode
children []*treeNode
}
// queryTree is a parsed EXPLAIN QUERY TREE dump.
type queryTree struct {
root *treeNode
byID map[int]*treeNode
}
var nodeLineRe = regexp.MustCompile(`^([A-Z_]+) id: (\d+)(?:, (.*))?$`)
// parseQueryTree builds the tree from the dump's lines, using the two-space
// indentation to recover nesting.
func parseQueryTree(lines []string) (*queryTree, error) {
root := &treeNode{id: -1, text: "<root>"}
t := &queryTree{root: root, byID: map[int]*treeNode{}}
stack := []*treeNode{root}
for _, line := range lines {
if strings.TrimSpace(line) == "" {
continue
}
indent := len(line) - len(strings.TrimLeft(line, " "))
if indent%2 != 0 {
return nil, fmt.Errorf("query tree: unexpected indentation in %q", line)
}
depth := indent/2 + 1
if depth > len(stack) {
return nil, fmt.Errorf("query tree: line %q is nested too deeply", line)
}
stack = stack[:depth]
parent := stack[len(stack)-1]
n := &treeNode{id: -1, text: line[indent:], parent: parent}
if m := nodeLineRe.FindStringSubmatch(n.text); m != nil {
n.kind = m[1]
n.id, _ = strconv.Atoi(m[2])
n.attrs = parseAttrs(m[3])
t.byID[n.id] = n
}
parent.children = append(parent.children, n)
stack = append(stack, n)
}
return t, nil
}
var attrKeyRe = regexp.MustCompile(`^, ([a-z_]+): `)
// parseAttrs splits `key: value, key: value` where values may themselves
// contain commas inside parentheses or quotes, as types and constants do.
func parseAttrs(s string) map[string]string {
attrs := map[string]string{}
if s == "" {
return attrs
}
// Positions at which a new `, key: ` begins at top level.
var cuts []int
depth := 0
inQuote := false
for i := 0; i < len(s); i++ {
switch c := s[i]; {
case c == '\'':
inQuote = !inQuote
case inQuote:
case c == '(':
depth++
case c == ')':
depth--
case c == ',' && depth == 0 && attrKeyRe.MatchString(s[i:]):
cuts = append(cuts, i)
}
}
cuts = append(cuts, len(s))
start := 0
for _, cut := range cuts {
pair := s[start:cut]
if key, value, ok := strings.Cut(pair, ": "); ok {
attrs[key] = value
}
start = cut + 2
}
return attrs
}
// section returns the child section header of a node, such as PROJECTION
// or JOIN TREE, or nil.
func (n *treeNode) section(name string) *treeNode {
for _, c := range n.children {
if c.kind == "" && c.text == name {
return c
}
}
return nil
}
// list returns the nodes of the LIST under a section header.
func (n *treeNode) list() []*treeNode {
if n == nil {
return nil
}
for _, c := range n.children {
if c.kind == "LIST" {
return c.children
}
}
return nil
}
// firstQuery returns the QUERY node describing a result set: the node itself
// or, for a UNION, its first branch, whose projection names the union's
// columns.
func firstQuery(n *treeNode) *treeNode {
for depth := 0; n != nil && depth < 32; depth++ {
switch n.kind {
case "QUERY":
return n
case "UNION":
n = firstNode(n.section("QUERIES").list())
default:
return nil
}
}
return nil
}
func firstNode(nodes []*treeNode) *treeNode {
if len(nodes) == 0 {
return nil
}
return nodes[0]
}
// projection returns the output column names of a query and the expression
// node producing each, in order.
func projection(q *treeNode) (names []string, nodes []*treeNode) {
q = firstQuery(q)
if q == nil {
return nil, nil
}
if cols := q.section("PROJECTION COLUMNS"); cols != nil {
for _, c := range cols.children {
names = append(names, projectedName(c.text))
}
}
return names, q.section("PROJECTION").list()
}
// projectedName strips the trailing type from a `name Type` line. The type
// is a single token that may carry a parenthesised argument list; the name
// may contain spaces of its own, as `plus(id, 1)` does.
func projectedName(line string) string {
end := len(line)
if strings.HasSuffix(line, ")") {
depth := 0
for end > 0 {
end--
if line[end] == ')' {
depth++
} else if line[end] == '(' {
depth--
if depth == 0 {
break
}
}
}
}
for end > 0 && isWordByte(line[end-1]) {
end--
}
return strings.TrimSpace(line[:end])
}
// sourceTable resolves a COLUMN node to the table it reads from, following
// column references through subqueries and CTEs. Columns computed by an
// expression have no source and yield "".
func (t *queryTree) sourceTable(col *treeNode) string {
for depth := 0; col != nil && col.kind == "COLUMN" && depth < 32; depth++ {
id, err := strconv.Atoi(col.attrs["source_id"])
if err != nil {
return ""
}
src := t.byID[id]
if src == nil {
return ""
}
switch src.kind {
case "TABLE":
name := src.attrs["table_name"]
if i := strings.LastIndexByte(name, '.'); i >= 0 {
name = name[i+1:]
}
return name
case "QUERY", "UNION":
// The column is one of the subquery's output columns; keep
// following whatever expression produces it there.
want := col.attrs["column_name"]
names, nodes := projection(src)
col = nil
for i, name := range names {
if name == want && i < len(nodes) {
col = nodes[i]
break
}
}
default:
return ""
}
}
return ""
}
// childrenOrNil returns a node's children, tolerating a nil node.
func (n *treeNode) childrenOrNil() []*treeNode {
if n == nil {
return nil
}
return n.children
}

160
internal/goldeneye/clickhouse/types.go Normal file
View File

@@ -0,0 +1,160 @@
package clickhouse
import (
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
)
// A type is a call expression, the way ClickHouse itself models one: a
// lowercased name applied to an ordered argument list. Each argument is
// another type, an integer, a boolean or a quoted string, optionally
// labelled, so Array, Map and LowCardinality are ordinary names and nothing
// about a nested type is lost. Nullability is an attribute of a type rather
// than a wrapper, since every engine has it and only ClickHouse spells it as
// a type: Nullable(T) becomes T with nullable set, at whatever depth it
// appears. The shape maps one to one onto a protobuf message with a oneof
// for the argument value:
//
// Map(String, Nullable(UInt32))
// {"name": "map", "args": [
// {"type": {"name": "string"}},
// {"type": {"name": "uint32", "nullable": true}}]}
//
// Tuple(lat Float64, lon Float64)
// {"name": "tuple", "args": [
// {"label": "lat", "type": {"name": "float64"}},
// {"label": "lon", "type": {"name": "float64"}}]}
//
// Enum8('a' = 1, 'b' = 2)
// {"name": "enum8", "args": [{"label": "a", "int": 1}, {"label": "b", "int": 2}]}
//
// DateTime64(3, 'UTC')
// {"name": "datetime64", "args": [{"int": 3}, {"string": "UTC"}]}
//
// An identifier argument such as the function in AggregateFunction(uniq,
// String) is a word rather than a type. Resolving names against the
// catalog is the reader's job; the output only records what was said.
// parseType turns a ClickHouse type string into its expression.
func parseType(t string) *analysis.TypeExpr {
name, args := splitType(t)
name = strings.ToLower(strings.TrimSpace(name))
if name == "nullable" && len(args) == 1 {
expr := parseType(args[0])
expr.Nullable = true
return expr
}
if name == "" {
name = "nothing"
}
expr := &analysis.TypeExpr{Name: name}
for _, a := range args {
expr.Args = append(expr.Args, parseArg(a))
}
// LowCardinality(Nullable(T)) is the only order ClickHouse accepts for
// a nullable low-cardinality column, and the nullability is the
// column's: it reads as a nullable LowCardinality(T).
if name == "lowcardinality" && len(expr.Args) == 1 && expr.Args[0].Type != nil && expr.Args[0].Type.Nullable {
expr.Nullable = true
expr.Args[0].Type.Nullable = false
}
// The function an aggregate-function type names is a word, not a type.
if name == "aggregatefunction" || name == "simpleaggregatefunction" {
if len(expr.Args) > 0 && expr.Args[0].Type != nil && len(expr.Args[0].Type.Args) == 0 {
fn := expr.Args[0].Type.Name
expr.Args[0] = analysis.TypeArg{Ident: &fn}
}
}
return expr
}
// parseArg parses one argument: a quoted string, an integer, a boolean, a
// labelled argument (`lat Float64` in a Tuple, `'a' = 1` in an Enum), or a
// type.
func parseArg(a string) analysis.TypeArg {
a = strings.TrimSpace(a)
if strings.HasPrefix(a, "'") {
end := skipQuoted(a, 0)
lit := unquote(a[1 : end-1])
if rest := strings.TrimSpace(a[end:]); strings.HasPrefix(rest, "=") {
arg := parseArg(rest[1:])
arg.Label = lit
return arg
}
return analysis.TypeArg{String: &lit}
}
if n, err := strconv.ParseInt(a, 10, 64); err == nil {
return analysis.TypeArg{Int: &n}
}
switch strings.ToLower(a) {
case "true", "false":
b := strings.EqualFold(a, "true")
return analysis.TypeArg{Bool: &b}
}
if i := labelEnd(a); i > 0 {
arg := parseArg(a[i+1:])
arg.Label = a[:i]
return arg
}
return analysis.TypeArg{Type: parseType(a)}
}
// labelEnd returns the index of the space separating a label from the type
// it labels, or -1 when the argument has no label: a space that comes before
// any parenthesis, as in `lat Float64` or `tags Array(String)`.
func labelEnd(a string) int {
head := a
if p := strings.IndexByte(a, '('); p >= 0 {
head = a[:p]
}
return strings.IndexByte(head, ' ')
}
// unquote undoes the escaping inside a single-quoted ClickHouse literal.
func unquote(s string) string {
s = strings.ReplaceAll(s, `\'`, `'`)
s = strings.ReplaceAll(s, `''`, `'`)
return strings.ReplaceAll(s, `\\`, `\`)
}
// splitType splits `Base(arg, arg)` into its base name and top-level
// arguments, leaving nested parentheses and quoted strings intact.
func splitType(t string) (string, []string) {
t = strings.TrimSpace(t)
open := strings.IndexByte(t, '(')
if open < 0 || !strings.HasSuffix(t, ")") {
return t, nil
}
base := strings.TrimSpace(t[:open])
inner := t[open+1 : len(t)-1]
var (
args []string
depth int
quote byte
start int
)
for i := 0; i < len(inner); i++ {
c := inner[i]
switch {
case quote != 0:
if c == '\\' {
i++
} else if c == quote {
quote = 0
}
case c == '\'' || c == '"' || c == '`':
quote = c
case c == '(':
depth++
case c == ')':
depth--
case c == ',' && depth == 0:
args = append(args, strings.TrimSpace(inner[start:i]))
start = i + 1
}
}
args = append(args, strings.TrimSpace(inner[start:]))
return base, args
}

277
internal/goldeneye/cmd/goldeneye/main.go Normal file
View File

@@ -0,0 +1,277 @@
// Command goldeneye generates the dialect seeds under
// internal/engine/<engine>/dialect from a live database, checks the
// committed ones against it, and checks the analyze cases under
// internal/endtoend/testdata against what the database itself reports.
//
// Usage, from internal/goldeneye:
//
// go run ./cmd/goldeneye install clickhouse # download the pinned clickhouse binary
// go run ./cmd/goldeneye install duckdb # download the current DuckDB 2.0 preview build
// go run ./cmd/goldeneye install sqlite # build the pinned sqlite3 shells from source
// go run ./cmd/goldeneye generate [engine] # rewrite the generated files from the database
// go run ./cmd/goldeneye check [engine] # compare the committed files and analyze cases with the database
//
// Without an engine, generate and check cover every engine whose database
// is available and say which ones they skipped. `go test ./...` runs the
// same checks as tests.
package main
import (
"context"
"errors"
"flag"
"fmt"
"io"
"os"
"runtime"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/clickhouse"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/duckdb"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
"github.com/sqlc-dev/sqlc/internal/goldeneye/mssql"
"github.com/sqlc-dev/sqlc/internal/goldeneye/mysql"
"github.com/sqlc-dev/sqlc/internal/goldeneye/postgresql"
"github.com/sqlc-dev/sqlc/internal/goldeneye/spanner"
"github.com/sqlc-dev/sqlc/internal/goldeneye/sqlite"
)
func main() {
if err := run(context.Background(), os.Args[1:], os.Stdout, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "goldeneye:", err)
os.Exit(1)
}
}
const usage = `usage:
goldeneye install clickhouse|duckdb|sqlite [-version V]
put the pinned release of an engine into the user cache directory: clickhouse is
downloaded, duckdb is downloaded from its v2.0 preview channel, sqlite is built from
the downloaded amalgamation with cc or $CC
goldeneye generate [engine]
rewrite the generated dialect files from the database, for every available engine or one
goldeneye check [engine]
compare the committed dialect files and analyze cases with the database, for every available engine or one
postgresql, mysql, mssql and spanner read the server POSTGRESQL_SERVER_URI, MYSQL_SERVER_URI,
MSSQL_SERVER_URI and SPANNER_SERVER_URI name
engines: clickhouse, duckdb, mssql, mysql, postgresql, spanner, sqlite`
// engine is one database goldeneye knows how to read a dialect from.
type engine struct {
name string
// dir is the engine directory the dialect lives under, when it is not
// named after the engine: MySQL's is dolphin, after its parser, and
// Spanner's is googlesql, after the language.
dir string
// cases is the name of the analyze case directories under
// internal/endtoend/testdata, when it is not the engine's name:
// Spanner's are googlesql's.
cases string
// locate finds the database — a binary or a connection URL — or says
// why it is not available.
locate func() (string, error)
// version describes the release the database is, for the log.
version func(context.Context, string) (string, error)
// generate reads the dialect from the database.
generate func(context.Context, string) (dialect.Files, error)
// analyze asks the database what it reports for an analyze case, in
// the shape sqlc analyze prints. Nil for an engine without one yet.
analyze func(context.Context, string, endtoend.Case) ([]byte, error)
}
var engines = []engine{
{clickhouse.Engine, "", "", clickhouse.Locate, clickhouse.Version, clickhouse.Generate, clickhouse.Analyze},
{duckdb.Engine, "", "", duckdb.Locate, duckdb.Version, duckdb.Generate, duckdb.Analyze},
{mssql.Engine, "", "", mssql.Locate, mssql.Version, mssql.Generate, mssql.Analyze},
{mysql.Engine, mysql.Dir, "", mysql.Locate, mysql.Version, mysql.Generate, mysql.Analyze},
{postgresql.Engine, "", "", postgresql.Locate, postgresql.Version, postgresql.Generate, nil},
{spanner.Engine, spanner.Dir, spanner.Cases, spanner.Locate, spanner.Version, spanner.Generate, spanner.Analyze},
{sqlite.Engine, "", "", sqlite.Locate, sqlite.Version, sqlite.Generate, sqlite.Analyze},
}
// dialectDir returns the engine's dialect directory.
func (e engine) dialectDir() (string, error) {
if e.dir != "" {
return dialect.Dir(e.dir)
}
return dialect.Dir(e.name)
}
// installer puts the binary an engine is read through in place, for the
// engines that need no server and whose release is pinned in their package.
type installer struct {
defaultVersion string
install func(ctx context.Context, version, goos, goarch string, progress io.Writer) (string, error)
}
var installers = map[string]installer{
clickhouse.Engine: {clickhouse.DefaultVersion, clickhouse.Install},
duckdb.Engine: {duckdb.DefaultVersion, duckdb.Install},
sqlite.Engine: {sqlite.DefaultVersion, sqlite.Install},
}
func run(ctx context.Context, args []string, stdout, stderr io.Writer) error {
if len(args) == 0 {
fmt.Fprintln(stderr, usage)
return errors.New("a command is required")
}
switch args[0] {
case "install":
return install(ctx, args[1:], stdout, stderr)
case "generate":
return forEach(ctx, args[1:], stderr, generate)
case "check":
return forEach(ctx, args[1:], stderr, check)
case "help", "-h", "--help":
fmt.Fprintln(stdout, usage)
return nil
}
fmt.Fprintln(stderr, usage)
return fmt.Errorf("unknown command %q", args[0])
}
func install(ctx context.Context, args []string, stdout, stderr io.Writer) error {
if len(args) == 0 {
return errors.New("install takes the engine to install: clickhouse, duckdb or sqlite")
}
inst, ok := installers[args[0]]
if !ok {
return fmt.Errorf("install takes the engine to install, clickhouse, duckdb or sqlite, not %q", args[0])
}
fs := flag.NewFlagSet("install "+args[0], flag.ContinueOnError)
fs.SetOutput(stderr)
version := fs.String("version", inst.defaultVersion, args[0]+" release to install")
if err := fs.Parse(args[1:]); err != nil {
return err
}
path, err := inst.install(ctx, *version, runtime.GOOS, runtime.GOARCH, stderr)
if err != nil {
return err
}
fmt.Fprintln(stdout, path)
return nil
}
// forEach runs a command over the named engine, or over every engine that
// is available when none is named. An engine that is named has to be
// available; one that is not named is skipped with a note when it is not.
func forEach(ctx context.Context, args []string, stderr io.Writer, cmd func(context.Context, engine, string, io.Writer) error) error {
if len(args) > 1 {
return errors.New("at most one engine may be named")
}
name := ""
if len(args) == 1 {
name = args[0]
}
found := false
var failed []string
for _, e := range engines {
if name != "" && e.name != name {
continue
}
found = true
handle, err := e.locate()
if err != nil {
if name != "" {
return err
}
fmt.Fprintf(stderr, "skipping %s: %v\n", e.name, err)
continue
}
if err := cmd(ctx, e, handle, stderr); err != nil {
fmt.Fprintf(stderr, "%s: %v\n", e.name, err)
failed = append(failed, e.name)
}
}
if !found {
return fmt.Errorf("unknown engine %q", name)
}
if len(failed) > 0 {
return fmt.Errorf("%d engine(s) failed: %v", len(failed), failed)
}
return nil
}
func generate(ctx context.Context, e engine, handle string, stderr io.Writer) error {
version, err := e.version(ctx, handle)
if err != nil {
return err
}
fmt.Fprintf(stderr, "%s: generating from %s\n", e.name, version)
files, err := e.generate(ctx, handle)
if err != nil {
return err
}
dir, err := e.dialectDir()
if err != nil {
return err
}
if err := dialect.Write(dir, files); err != nil {
return err
}
fmt.Fprintf(stderr, "%s: wrote %d file(s) to %s\n", e.name, len(files), dir)
return nil
}
func check(ctx context.Context, e engine, handle string, stderr io.Writer) error {
version, err := e.version(ctx, handle)
if err != nil {
return err
}
fmt.Fprintf(stderr, "%s: checking against %s\n", e.name, version)
files, err := e.generate(ctx, handle)
if err != nil {
return err
}
dir, err := e.dialectDir()
if err != nil {
return err
}
report, err := dialect.Check(dir, files)
if err != nil {
return err
}
if report != "" {
return fmt.Errorf("%s does not match the database\n%s", dir, report)
}
fmt.Fprintf(stderr, "%s: ok, %d file(s) match\n", e.name, len(files))
return checkAnalyzeCases(ctx, e, handle, stderr)
}
// checkAnalyzeCases compares what the database reports for each of the
// engine's analyze cases with the case's committed output.
func checkAnalyzeCases(ctx context.Context, e engine, handle string, stderr io.Writer) error {
if e.analyze == nil {
return nil
}
name := e.cases
if name == "" {
name = e.name
}
cases, err := endtoend.Cases(name)
if err != nil {
return err
}
var report strings.Builder
for _, c := range cases {
got, err := e.analyze(ctx, handle, c)
if err != nil {
fmt.Fprintf(&report, "%s: %v\n", c.Name, err)
continue
}
diff, err := c.Compare(got)
if err != nil {
return err
}
if diff != "" {
fmt.Fprintf(&report, "%s (-committed +database)\n%s", c.Name, diff)
}
}
if report.Len() > 0 {
return fmt.Errorf("analyze cases do not match the database\n%s", report.String())
}
fmt.Fprintf(stderr, "%s: ok, %d analyze case(s) match\n", e.name, len(cases))
return nil
}

303
internal/goldeneye/dialect/dialect.go Normal file
View File

@@ -0,0 +1,303 @@
// Package dialect describes the files a dialect seed is made of — the JSONL
// records under internal/engine/<engine>/dialect that give an engine its
// type system and standard library — and provides what a generator needs to
// write them and a check needs to compare them with what is committed.
//
// The record types mirror the ones in internal/core/seed, which reads the
// files. They are repeated here rather than imported so that this module
// never shares code with the analysis it checks: the files are the contract,
// and a change to their shape has to be made on both sides.
package dialect
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"runtime"
"sort"
"strings"
)
// The files a dialect directory is made of.
const (
SettingsFile = "dialect.json"
TypesFile = "types.jsonl"
OperatorsFile = "operators.jsonl"
CastsFile = "casts.jsonl"
FunctionsFile = "functions.jsonl"
RelationsFile = "relations.jsonl"
)
// ExtensionsDir is the directory under a dialect holding one directory per
// extension the dialect knows, each a smaller bundle of the same files.
const ExtensionsDir = "extensions"
// Type is a type the dialect defines. Aliases are spellings of the same type
// that a schema may use in a column definition.
type Type struct {
Name string `json:"name"`
Category string `json:"category"`
Aliases []string `json:"aliases,omitempty"`
}
// Operator is a single operator overload.
type Operator struct {
Name string `json:"name"`
Left string `json:"left"`
Right string `json:"right"`
Result string `json:"result"`
}
// Cast is a single cast between two types. Context is 'i'mplicit,
// 'a'ssignment or 'e'xplicit.
type Cast struct {
Source string `json:"source"`
Target string `json:"target"`
Context string `json:"context,omitempty"`
}
// Function is a function the dialect ships with. Kind is 'f'unction,
// 'a'ggregate, 'w'indow or 'p'rocedure.
type Function struct {
Name string `json:"name"`
Kind string `json:"kind,omitempty"`
Args []Arg `json:"args,omitempty"`
Returns string `json:"returns"`
Nullable bool `json:"nullable,omitempty"`
}
// Arg is one of a function's parameters. Mode is 'i'n, 'o'ut, 'b'oth,
// 't'able or 'v'ariadic, and defaults to in.
type Arg struct {
Name string `json:"name,omitempty"`
Type string `json:"type"`
Mode string `json:"mode,omitempty"`
HasDefault bool `json:"has_default,omitempty"`
}
// Relation is a table or view the dialect ships with, such as one of
// PostgreSQL's system catalogs. Kind is 'r' for a table or 'v' for a view,
// and defaults to a table.
type Relation struct {
Catalog string `json:"catalog,omitempty"`
Schema string `json:"schema"`
Name string `json:"name"`
Kind string `json:"kind,omitempty"`
Columns []Column `json:"columns"`
}
// Column is one of a relation's columns.
type Column struct {
Name string `json:"name"`
Type string `json:"type"`
NotNull bool `json:"not_null,omitempty"`
Array bool `json:"array,omitempty"`
Length int `json:"length,omitempty"`
}
// Files is what a generator produces: the content of each file it writes,
// keyed by slash-separated path relative to the dialect directory, such as
// "functions.jsonl" or "extensions/hstore/types.jsonl". A generator owns
// only the files it produces; the rest of a dialect, dialect.json above all,
// is written by hand and is neither generated nor checked.
type Files map[string][]byte
// JSONL encodes records one per line, the way the committed files are
// written: encoding/json's compact form, with a newline after each record.
func JSONL[T any](records []T) ([]byte, error) {
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
for _, record := range records {
if err := enc.Encode(record); err != nil {
return nil, err
}
}
return buf.Bytes(), nil
}
// ReadTypes reads the hand-written types.jsonl of a dialect directory, for
// a generator that writes a function once per spelling of a type.
func ReadTypes(dir string) ([]Type, error) {
blob, err := os.ReadFile(filepath.Join(dir, TypesFile))
if err != nil {
return nil, err
}
var types []Type
dec := json.NewDecoder(bytes.NewReader(blob))
for dec.More() {
var t Type
if err := dec.Decode(&t); err != nil {
return nil, fmt.Errorf("%s: %w", filepath.Join(dir, TypesFile), err)
}
types = append(types, t)
}
return types, nil
}
// Dir returns the dialect directory of an engine,
// internal/engine/<engine>/dialect, found relative to this source file so
// the working directory does not matter.
func Dir(engine string) (string, error) {
_, file, _, ok := runtime.Caller(0)
if !ok {
return "", errors.New("cannot locate the goldeneye source directory")
}
dir := filepath.Join(filepath.Dir(file), "..", "..", "engine", engine, "dialect")
if _, err := os.Stat(dir); err != nil {
return "", err
}
return filepath.Clean(dir), nil
}
// Names lists the files in name order, so that a run is reported and
// written the same way every time.
func (f Files) Names() []string {
names := make([]string, 0, len(f))
for name := range f {
names = append(names, name)
}
sort.Strings(names)
return names
}
// Write writes the files into dir, creating extension directories as
// needed. Files the generator did not produce are left alone.
func Write(dir string, files Files) error {
for _, name := range files.Names() {
path := filepath.Join(dir, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
if err := os.WriteFile(path, files[name], 0o644); err != nil {
return err
}
}
return nil
}
// Check compares the files with the ones committed in dir and returns a
// report of every file that differs, or "" when each matches byte for byte.
func Check(dir string, files Files) (string, error) {
var report strings.Builder
for _, name := range files.Names() {
want, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(name)))
if errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(&report, "%s: not committed\n", name)
continue
}
if err != nil {
return "", err
}
if bytes.Equal(want, files[name]) {
continue
}
fmt.Fprintf(&report, "%s (-committed +database)\n%s", name, Diff(string(want), string(files[name])))
}
return report.String(), nil
}
// Diff is a line diff of two texts, marking lines only in want with "-" and
// lines only in got with "+", showing two lines of context around each
// change and eliding the rest. The committed files run to thousands of
// lines and usually differ in a handful, so the common head and tail are
// stripped before the longest common subsequence of the middle is found.
func Diff(want, got string) string {
a := lines(want)
b := lines(got)
head := 0
for head < len(a) && head < len(b) && a[head] == b[head] {
head++
}
tail := 0
for tail < len(a)-head && tail < len(b)-head && a[len(a)-1-tail] == b[len(b)-1-tail] {
tail++
}
var edits []edit
for _, line := range a[:head] {
edits = append(edits, edit{' ', line})
}
edits = append(edits, lcsEdits(a[head:len(a)-tail], b[head:len(b)-tail])...)
for _, line := range a[len(a)-tail:] {
edits = append(edits, edit{' ', line})
}
const context = 2
show := make([]bool, len(edits))
for i, e := range edits {
if e.kind == ' ' {
continue
}
for j := max(0, i-context); j <= min(len(edits)-1, i+context); j++ {
show[j] = true
}
}
var out strings.Builder
elided := false
for i, e := range edits {
if !show[i] {
elided = true
continue
}
if elided && out.Len() > 0 {
out.WriteString("...\n")
}
elided = false
fmt.Fprintf(&out, "%c %s\n", e.kind, e.line)
}
return out.String()
}
type edit struct {
kind byte // ' ', '-' or '+'
line string
}
func lines(s string) []string {
if s == "" {
return nil
}
return strings.Split(strings.TrimSuffix(s, "\n"), "\n")
}
// lcsEdits is the edit script between a and b by longest common subsequence.
func lcsEdits(a, b []string) []edit {
lcs := make([][]int, len(a)+1)
for i := range lcs {
lcs[i] = make([]int, len(b)+1)
}
for i := len(a) - 1; i >= 0; i-- {
for j := len(b) - 1; j >= 0; j-- {
if a[i] == b[j] {
lcs[i][j] = lcs[i+1][j+1] + 1
} else {
lcs[i][j] = max(lcs[i+1][j], lcs[i][j+1])
}
}
}
var edits []edit
i, j := 0, 0
for i < len(a) && j < len(b) {
switch {
case a[i] == b[j]:
edits = append(edits, edit{' ', a[i]})
i++
j++
case lcs[i+1][j] >= lcs[i][j+1]:
edits = append(edits, edit{'-', a[i]})
i++
default:
edits = append(edits, edit{'+', b[j]})
j++
}
}
for ; i < len(a); i++ {
edits = append(edits, edit{'-', a[i]})
}
for ; j < len(b); j++ {
edits = append(edits, edit{'+', b[j]})
}
return edits
}

846
internal/goldeneye/duckdb/analyze.go Normal file
View File

@@ -0,0 +1,846 @@
package duckdb
import (
"bytes"
"context"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"regexp"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// The analyze cases are checked against the DuckDB CLI, which runs each
// case's schema and fixture into an in-memory database of their own, one
// process per question, and is asked four things about each query. What
// its parameters are: the query is prepared and explained, as JSON, with
// a string sentinel bound to each parameter, and the unoptimized logical
// plan the CLI prints first shows each as CAST('goldeneye_k' AS T), T
// being the type the binder gave the parameter, or bare when that type
// is VARCHAR. What its result columns are:
// DESCRIBE, with each parameter replaced by a NULL of that type, names and
// types them. Which column each is read from and which column a parameter
// stands in for: DuckDB prints a plan with every column by its bare name,
// so these are read from the query text, the select list's items and the
// operand beside each parameter resolved against the FROM clause and the
// catalog, from which a column read from a table takes its declared type
// and nullability. And whether an expression can be NULL, which DuckDB
// does not track: the query is run, with each parameter bound to a value
// of its type, over the fixture and over no rows, and a column is nullable
// when either run returns a NULL for it.
// placeholder is one parameter of a query as sqlc numbers them: a $n by
// its number, and each $name, sqlc.arg name or ? in turn at its first
// appearance, taking the lowest number no $n took.
type placeholder struct {
Number int
Name string
}
var (
numberedRe = regexp.MustCompile(`^\$([0-9]+)`)
namedRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)`)
sqlcArgRe = regexp.MustCompile(`^sqlc\.(n?arg|slice)\(\s*'?([A-Za-z_][A-Za-z0-9_]*)'?\s*\)`)
)
// bind rewrites the query so that every parameter is a $k numbered as
// sqlc numbers it, and lists the parameters in that order.
func bind(query string) (string, []placeholder) {
type occurrence struct {
start, end int
number int // for $n
name string // for $name or sqlc.arg, "" for ?
}
var occ []occurrence
i := 0
for i < len(query) {
c := query[i]
switch {
case c == '\'' || c == '"':
i = quotedEnd(query, i)
case strings.HasPrefix(query[i:], "--"):
end := strings.IndexByte(query[i:], '\n')
if end < 0 {
i = len(query)
} else {
i += end
}
case strings.HasPrefix(query[i:], "/*"):
end := strings.Index(query[i:], "*/")
if end < 0 {
i = len(query)
} else {
i += end + 2
}
case c == '?':
occ = append(occ, occurrence{start: i, end: i + 1})
i++
case c == '$' && numberedRe.MatchString(query[i:]):
m := numberedRe.FindStringSubmatch(query[i:])
n, _ := strconv.Atoi(m[1])
occ = append(occ, occurrence{start: i, end: i + len(m[0]), number: n})
i += len(m[0])
case c == '$' && namedRe.MatchString(query[i:]):
m := namedRe.FindStringSubmatch(query[i:])
occ = append(occ, occurrence{start: i, end: i + len(m[0]), name: m[1]})
i += len(m[0])
case c == 's' && sqlcArgRe.MatchString(query[i:]):
m := sqlcArgRe.FindStringSubmatch(query[i:])
occ = append(occ, occurrence{start: i, end: i + len(m[0]), name: m[2]})
i += len(m[0])
default:
i++
}
}
taken := map[int]bool{}
for _, o := range occ {
if o.number > 0 {
taken[o.number] = true
}
}
byName := map[string]int{}
next := 1
assign := func(o occurrence) int {
if o.number > 0 {
return o.number
}
if n, ok := byName[o.name]; ok && o.name != "" {
return n
}
for taken[next] {
next++
}
n := next
taken[n] = true
if o.name != "" {
byName[o.name] = n
}
return n
}
var phs []placeholder
numbered := map[int]bool{}
var out strings.Builder
last := 0
for _, o := range occ {
n := assign(o)
if !numbered[n] {
numbered[n] = true
phs = append(phs, placeholder{Number: n, Name: o.name})
}
out.WriteString(query[last:o.start])
out.WriteString("$" + strconv.Itoa(n))
last = o.end
}
out.WriteString(query[last:])
for i := 1; i < len(phs); i++ {
for j := i; j > 0 && phs[j-1].Number > phs[j].Number; j-- {
phs[j-1], phs[j] = phs[j], phs[j-1]
}
}
return out.String(), phs
}
// nullMarker is what a NULL is printed as when the CLI prints CSV, unlike
// any value a case holds.
const nullMarker = "<goldeneye:null>"
// analyzer holds the CLI a case runs through, the case's schema and
// fixture, and the catalog of the schema.
type analyzer struct {
binary string
schema string // the schema's statements, each ending in ;
fixture string // the fixture's, or ""
tables map[string][]column
enums map[string][]string // the labels of each enum type the schema created
// canonical names the type the dialect reports a spelling as, for
// each alias types.jsonl lists: DuckDB spells a JSON column json,
// which its own catalog lists as a spelling of varchar.
canonical map[string]string
}
// readAliases reads the aliases the generated types.jsonl gives each
// type, keyed by alias.
func readAliases() (map[string]string, error) {
dir, err := dialect.Dir(Engine)
if err != nil {
return nil, err
}
types, err := dialect.ReadTypes(dir)
if err != nil {
return nil, err
}
canonical := map[string]string{}
for _, t := range types {
for _, alias := range t.Aliases {
canonical[strings.ToLower(alias)] = strings.ToLower(t.Name)
}
}
return canonical, nil
}
// run executes a script in a fresh in-memory database loaded with the
// schema and, unless told otherwise, the fixture, and returns what the
// CLI printed in the mode asked for: "json" prints one JSON array per
// statement that returns rows, "csv" prints rows with a header and NULL
// as nullMarker, and "" prints the CLI's own text, which is how EXPLAIN
// draws a plan. A statement that fails ends the script with DuckDB's own
// error message.
func (a *analyzer) run(ctx context.Context, script, mode string, withFixture bool) (string, error) {
args := []string{"-bail"}
switch mode {
case "json":
args = append(args, "-json")
case "csv":
args = append(args, "-csv", "-header", "-nullvalue", nullMarker)
}
args = append(args, ":memory:")
cmd := exec.CommandContext(ctx, a.binary, args...)
prelude := a.schema
if withFixture {
prelude += a.fixture
}
cmd.Stdin = strings.NewReader(prelude + script)
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(stderr.String())
if msg == "" {
msg = err.Error()
}
return "", errors.New(msg)
}
return stdout.String(), nil
}
// query runs one statement and decodes the rows it printed.
func (a *analyzer) query(ctx context.Context, statement string) ([]map[string]json.RawMessage, error) {
out, err := a.run(ctx, statement+";\n", "json", true)
if err != nil {
return nil, err
}
var rows []map[string]json.RawMessage
dec := json.NewDecoder(strings.NewReader(out))
for {
var rs []map[string]json.RawMessage
err := dec.Decode(&rs)
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, fmt.Errorf("decoding duckdb output: %w", err)
}
rows = append(rows, rs...)
}
return rows, nil
}
func str(v json.RawMessage) string {
var s string
json.Unmarshal(v, &s)
return s
}
// column is what the catalog says about a column of a table.
type column struct {
name string
spelling string // the type as DuckDB spells it
typ *analysis.TypeExpr
nullable bool
}
// statements joins a script's statements, each ending in a semicolon.
func statements(src string) string {
s := strings.TrimRight(strings.TrimSpace(src), ";")
if s == "" {
return ""
}
return s + ";\n"
}
// Analyze runs a case's queries through the CLI and returns what DuckDB
// reports in the JSON shape sqlc analyze prints.
func Analyze(ctx context.Context, binary string, c endtoend.Case) ([]byte, error) {
schema, err := os.ReadFile(c.Schema)
if err != nil {
return nil, err
}
var fixture []byte
if c.Fixture != "" {
if fixture, err = os.ReadFile(c.Fixture); err != nil {
return nil, err
}
}
queries, err := c.Queries()
if err != nil {
return nil, err
}
a := &analyzer{binary: binary, schema: statements(string(schema)), fixture: statements(string(fixture))}
if a.canonical, err = readAliases(); err != nil {
return nil, err
}
if err := a.readCatalog(ctx); err != nil {
return nil, err
}
out := make([]analysis.Query, 0, len(queries))
for _, q := range queries {
aq, err := a.analyzeQuery(ctx, q)
if err != nil {
return nil, fmt.Errorf("%s: %w", q.Name, err)
}
out = append(out, aq)
}
return analysis.Encode(out)
}
// Check compares what DuckDB reports for a case with the output the case
// committed, returning a diff when they differ.
func Check(ctx context.Context, binary string, c endtoend.Case) (string, error) {
got, err := Analyze(ctx, binary, c)
if err != nil {
return "", err
}
return c.Compare(got)
}
// readCatalog reads the tables and enum types the schema created.
func (a *analyzer) readCatalog(ctx context.Context) error {
rows, err := a.query(ctx, `SELECT type_name, labels FROM duckdb_types() WHERE database_name = 'memory' AND NOT internal AND logical_type = 'ENUM' ORDER BY type_name`)
if err != nil {
return err
}
a.enums = map[string][]string{}
for _, row := range rows {
var labels []string
json.Unmarshal(row["labels"], &labels)
a.enums[strings.ToLower(str(row["type_name"]))] = labels
}
rows, err = a.query(ctx, `SELECT table_name, column_name, data_type, is_nullable FROM duckdb_columns() WHERE database_name = 'memory' ORDER BY table_oid, column_index`)
if err != nil {
return err
}
a.tables = map[string][]column{}
for _, row := range rows {
table := strings.ToLower(str(row["table_name"]))
var nullable bool
json.Unmarshal(row["is_nullable"], &nullable)
spelling := str(row["data_type"])
a.tables[table] = append(a.tables[table], column{
name: str(row["column_name"]),
spelling: spelling,
typ: a.parseType(spelling),
nullable: nullable,
})
}
return nil
}
// lookup finds a column of a table, which DuckDB matches in any case.
func (a *analyzer) lookup(table, name string) (column, bool) {
for _, col := range a.tables[strings.ToLower(table)] {
if strings.EqualFold(col.name, name) {
return col, true
}
}
return column{}, false
}
// resolve finds the table a reference names in a scope: the table its
// qualifier aliases, or the one table of the scope that has the column.
func (a *analyzer) resolve(sc scope, r ref) (string, column, bool) {
tables := sc.tables
if sc.target.name != "" {
tables = append([]tableRef{sc.target}, tables...)
}
if r.qualifier != "" {
if sc.ctes[r.qualifier] {
return "", column{}, false
}
for _, t := range tables {
if t.alias == r.qualifier || (t.alias == "" && t.name == r.qualifier) {
col, ok := a.lookup(t.name, r.column)
return t.name, col, ok
}
}
return "", column{}, false
}
var found string
var fc column
for _, t := range tables {
if col, ok := a.lookup(t.name, r.column); ok {
if found != "" && found != t.name {
return "", column{}, false
}
found, fc = t.name, col
}
}
return found, fc, found != ""
}
// describe reads what the catalog says about a column read from a table.
func describe(table string, col column) analysis.Column {
return analysis.Column{Name: col.name, Type: withNullable(col.typ, col.nullable), Table: table}
}
func withNullable(t *analysis.TypeExpr, nullable bool) *analysis.TypeExpr {
if t == nil {
return nil
}
out := *t
out.Nullable = nullable
return &out
}
// binding is what is known about one parameter: the type the binder gave
// it, spelled as DuckDB spells it, and the column it stands in for.
type binding struct {
spelling string
typ *analysis.TypeExpr
column *analysis.Column
}
var (
sentinelCastRe = regexp.MustCompile(`CAST\('goldeneye_([0-9]+)' AS `)
sentinelRe = regexp.MustCompile(`'goldeneye_([0-9]+)'`)
sentinelErrRe = regexp.MustCompile(`goldeneye_([0-9]+)`)
)
// explain prepares the query, explains it with a sentinel bound to each
// parameter and returns the type the binder gave each, keyed by number.
// A parameter whose sentinel the binder converts on the spot, as an
// INSERT's VALUES are, is bound to NULL instead and reported by nothing:
// the error names the sentinel for most types, and for the rest the
// parameters not yet bound to NULL are tried in turn.
func (a *analyzer) explain(ctx context.Context, sql string, phs []placeholder) (map[int]string, error) {
null := map[int]bool{}
for attempt := 0; attempt <= len(phs); attempt++ {
args := make([]string, len(phs))
for i, ph := range phs {
if null[ph.Number] {
args[i] = "NULL"
} else {
args[i] = fmt.Sprintf("'goldeneye_%d'", ph.Number)
}
}
script := "PREPARE goldeneye AS " + sql + ";\nSET explain_output = 'all';\nEXPLAIN (FORMAT json) EXECUTE goldeneye(" + strings.Join(args, ", ") + ");\n"
out, err := a.run(ctx, script, "", true)
if err != nil {
if !strings.Contains(err.Error(), "Conversion Error") && !strings.Contains(err.Error(), "can't be cast") {
return nil, err
}
retry := false
if m := sentinelErrRe.FindStringSubmatch(err.Error()); m != nil {
n, _ := strconv.Atoi(m[1])
if !null[n] {
null[n] = true
retry = true
}
}
if !retry {
for _, ph := range phs {
if !null[ph.Number] {
null[ph.Number] = true
retry = true
break
}
}
}
if !retry {
return nil, err
}
continue
}
return sentinelTypes(out)
}
return nil, errors.New("could not bind the parameters")
}
// planNode is one operator of a plan the CLI prints as JSON. Its
// extra_info holds the operator's expressions, each a string or a list
// of strings.
type planNode struct {
Name string `json:"name"`
Children []planNode `json:"children"`
ExtraInfo map[string]json.RawMessage `json:"extra_info"`
}
// sentinelTypes reads the type each sentinel is cast to out of the plans
// the CLI printed, one JSON array per plan: the unoptimized logical plan
// comes first, and the first cast of a sentinel wins. A sentinel the
// plan prints bare is a VARCHAR, which the binder needs no cast for.
func sentinelTypes(out string) (map[int]string, error) {
var texts []string
var walk func(n planNode)
walk = func(n planNode) {
for _, raw := range n.ExtraInfo {
var one string
if json.Unmarshal(raw, &one) == nil {
texts = append(texts, one)
continue
}
var many []string
if json.Unmarshal(raw, &many) == nil {
texts = append(texts, many...)
}
}
for _, c := range n.Children {
walk(c)
}
}
dec := json.NewDecoder(strings.NewReader(out))
for {
var plan []planNode
err := dec.Decode(&plan)
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, fmt.Errorf("decoding the plan: %w", err)
}
for _, n := range plan {
walk(n)
}
}
types := map[int]string{}
for _, text := range texts {
for _, m := range sentinelCastRe.FindAllStringSubmatchIndex(text, -1) {
n, _ := strconv.Atoi(text[m[2]:m[3]])
if _, ok := types[n]; ok {
continue
}
// The type runs to the parenthesis closing the CAST.
depth := 1
i := m[1]
for ; i < len(text); i++ {
if text[i] == '(' {
depth++
} else if text[i] == ')' {
depth--
if depth == 0 {
break
}
}
}
types[n] = strings.Join(strings.Fields(text[m[1]:i]), " ")
}
}
for _, text := range texts {
for _, m := range sentinelRe.FindAllStringSubmatchIndex(text, -1) {
n, _ := strconv.Atoi(text[m[2]:m[3]])
if _, ok := types[n]; !ok && !strings.HasSuffix(text[:m[0]], "CAST(") {
types[n] = "VARCHAR"
}
}
}
return types, nil
}
// analyzeQuery describes one query.
func (a *analyzer) analyzeQuery(ctx context.Context, q endtoend.Query) (analysis.Query, error) {
sql, phs := bind(q.SQL)
aq := analysis.Query{
Name: q.Name,
Cmd: q.Cmd,
Columns: []analysis.Column{},
Params: []analysis.Param{},
}
t := text(tokenize(sql))
sc := t.readScope()
// Parameters: a partner column first, then the cast the query wraps
// the parameter in, then the type the binder gave it.
bindings := map[int]*binding{}
var casts []string
var castNumbers []int
for _, ph := range phs {
k := strconv.Itoa(ph.Number)
b := &binding{}
bindings[ph.Number] = b
if t.inSubquery(k) {
// A parameter inside a subquery is compared with a column of
// the subquery's own tables, which the statement's scope does
// not name.
if typ, ok := t.castOf(k); ok {
casts = append(casts, typ)
castNumbers = append(castNumbers, ph.Number)
}
continue
}
if sc.kind == "insert" {
if pos, ok := t.valuesPosition(k); ok {
cols := t.insertColumns()
if cols == nil {
for _, col := range a.tables[sc.target.name] {
cols = append(cols, col.name)
}
}
if pos < len(cols) {
if col, ok := a.lookup(sc.target.name, cols[pos]); ok {
c := describe(sc.target.name, col)
b.column, b.spelling, b.typ = &c, col.spelling, col.typ
continue
}
}
}
}
if r, ok := t.partner(k); ok {
if table, col, ok := a.resolve(sc, r); ok {
c := describe(table, col)
b.column, b.spelling, b.typ = &c, col.spelling, col.typ
continue
}
}
if typ, ok := t.castOf(k); ok {
casts = append(casts, typ)
castNumbers = append(castNumbers, ph.Number)
}
}
if len(casts) > 0 {
// DuckDB spells the cast's type: VARCHAR(5) is VARCHAR, mood is
// ENUM('sad', 'ok').
var items []string
for i, typ := range casts {
items = append(items, fmt.Sprintf("CAST(NULL AS %s) AS p%d", typ, i))
}
rows, err := a.query(ctx, "DESCRIBE SELECT "+strings.Join(items, ", "))
if err != nil {
return analysis.Query{}, err
}
for i, row := range rows {
if i < len(castNumbers) {
b := bindings[castNumbers[i]]
b.spelling = str(row["column_type"])
b.typ = a.parseType(b.spelling)
}
}
}
if len(phs) > 0 {
types, err := a.explain(ctx, sql, phs)
if err != nil {
return analysis.Query{}, err
}
for n, spelling := range types {
if b := bindings[n]; b != nil && b.typ == nil {
b.spelling = spelling
b.typ = a.parseType(spelling)
}
}
}
// Result columns.
var columns []analysis.Column
switch sc.kind {
case "select":
rows, err := a.query(ctx, "DESCRIBE "+a.substitute(sql, phs, bindings, false))
if err != nil {
return analysis.Query{}, err
}
for _, row := range rows {
columns = append(columns, analysis.Column{Name: str(row["column_name"]), Type: a.parseType(str(row["column_type"]))})
}
a.attribute(sc, t.selectItems(), columns)
default:
// A RETURNING column is the target table's column it names;
// DuckDB describes no DML, so an expression is typed by nothing.
for _, it := range t.returningItems() {
switch {
case it.star:
for _, col := range a.tables[sc.target.name] {
columns = append(columns, describe(sc.target.name, col))
}
case it.ref != nil:
if table, col, ok := a.resolve(sc, *it.ref); ok {
c := describe(table, col)
c.Name = it.name
columns = append(columns, c)
continue
}
columns = append(columns, analysis.Column{Name: it.name})
default:
columns = append(columns, analysis.Column{Name: it.name})
}
}
}
if len(columns) > 0 {
nullable := a.observe(ctx, sql, phs, bindings, len(columns))
for i := range columns {
if columns[i].Table == "" && columns[i].Type != nil && nullable[i] {
columns[i].Type = withNullable(columns[i].Type, true)
}
}
}
aq.Columns = append(aq.Columns, columns...)
for _, ph := range phs {
b := bindings[ph.Number]
ac := analysis.Column{Type: b.typ}
if b.column != nil {
ac = *b.column
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}
// attribute says which table each result column is read from, by
// matching the select list's items, a star expanded to its table's
// columns, against the columns DuckDB described. A list that expands to
// another number of columns than DuckDB describes attributes nothing.
func (a *analyzer) attribute(sc scope, items []item, columns []analysis.Column) {
var origins []*analysis.Column
for _, it := range items {
switch {
case it.star && it.ref == nil:
for _, t := range sc.tables {
for _, col := range a.tables[t.name] {
c := describe(t.name, col)
origins = append(origins, &c)
}
}
case it.star:
table, ok := a.aliased(sc, it.ref.qualifier)
if !ok {
return
}
for _, col := range a.tables[table] {
c := describe(table, col)
origins = append(origins, &c)
}
case it.ref != nil:
if table, col, ok := a.resolve(sc, *it.ref); ok {
c := describe(table, col)
origins = append(origins, &c)
} else {
origins = append(origins, nil)
}
default:
origins = append(origins, nil)
}
}
if len(origins) != len(columns) {
return
}
for i, o := range origins {
if o != nil {
columns[i].Type = o.Type
columns[i].Table = o.Table
}
}
}
// aliased finds the table a qualifier names in a scope.
func (a *analyzer) aliased(sc scope, qualifier string) (string, bool) {
for _, t := range sc.tables {
if t.alias == qualifier || (t.alias == "" && t.name == qualifier) {
if _, ok := a.tables[t.name]; ok {
return t.name, true
}
}
}
return "", false
}
// substitute replaces each parameter with a NULL of its type, or with a
// value of its type when values is set, for DuckDB to describe or run the
// query. A parameter of unknown type becomes a bare NULL. Strings,
// quoted identifiers and comments are left alone.
func (a *analyzer) substitute(sql string, phs []placeholder, bindings map[int]*binding, values bool) string {
repl := map[int]string{}
for _, ph := range phs {
r := "NULL"
if b := bindings[ph.Number]; b != nil && b.spelling != "" {
r = "CAST(NULL AS " + b.spelling + ")"
if values {
// The value is chosen by the type as spelled, since an
// alias such as JSON takes values its canonical type
// does not.
if v, ok := a.zero(a.parseTypeWith(b.spelling, false)); ok {
r = "CAST('" + strings.ReplaceAll(v, "'", "''") + "' AS " + b.spelling + ")"
}
}
}
repl[ph.Number] = r
}
var out strings.Builder
i := 0
for i < len(sql) {
c := sql[i]
switch {
case c == '\'' || c == '"':
end := quotedEnd(sql, i)
out.WriteString(sql[i:end])
i = end
case strings.HasPrefix(sql[i:], "--"):
end := strings.IndexByte(sql[i:], '\n')
if end < 0 {
end = len(sql)
} else {
end += i
}
out.WriteString(sql[i:end])
i = end
case strings.HasPrefix(sql[i:], "/*"):
end := strings.Index(sql[i:], "*/")
if end < 0 {
end = len(sql)
} else {
end += i + 2
}
out.WriteString(sql[i:end])
i = end
case c == '$' && numberedRe.MatchString(sql[i:]):
m := numberedRe.FindStringSubmatch(sql[i:])
n, _ := strconv.Atoi(m[1])
if r, ok := repl[n]; ok {
out.WriteString(r)
} else {
out.WriteString(m[0])
}
i += len(m[0])
default:
out.WriteByte(c)
i++
}
}
return out.String()
}
// observe runs the query with a value bound to each parameter, over the
// fixture and over no rows, and reports which of the first n result
// columns came back NULL in either run. A run the CLI rejects observes
// nothing.
func (a *analyzer) observe(ctx context.Context, sql string, phs []placeholder, bindings map[int]*binding, n int) []bool {
nullable := make([]bool, n)
statement := a.substitute(sql, phs, bindings, true) + ";\n"
runs := []bool{true}
if a.fixture != "" {
runs = append(runs, false)
}
for _, withFixture := range runs {
out, err := a.run(ctx, statement, "csv", withFixture)
if err != nil {
continue
}
records, err := csv.NewReader(strings.NewReader(out)).ReadAll()
if err != nil {
continue
}
for _, rec := range records[min(1, len(records)):] {
for j, v := range rec {
if j < n && v == nullMarker {
nullable[j] = true
}
}
}
}
return nullable
}

351
internal/goldeneye/duckdb/duckdb.go Normal file
View File

@@ -0,0 +1,351 @@
// Package duckdb generates the DuckDB dialect seed under
// internal/engine/duckdb/dialect — types.jsonl, functions.jsonl and
// operators.jsonl — from a live DuckDB CLI, the same way the postgresql
// package generates PostgreSQL's from a live server, and verifies the
// DuckDB analyze cases under internal/endtoend/testdata against the same
// CLI.
//
// The CLI is a DuckDB 2.0 build, the release darkwing is pinned against,
// which has no release to download yet: Install fetches the current build
// of DuckDB's v2.0 preview channel into the user cache directory. The CLI
// is located through the DUCKDB environment variable, then the cached
// build, then "duckdb" on PATH.
package duckdb
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"sort"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name of the engine directory the dialect lives under.
const Engine = "duckdb"
// Locate finds the DuckDB CLI: the DUCKDB environment variable wins, then
// the cached build of DefaultVersion, then "duckdb" on PATH.
func Locate() (string, error) {
if path := os.Getenv("DUCKDB"); path != "" {
return path, nil
}
if path, err := cachedBinary(DefaultVersion); err == nil {
if _, err := os.Stat(path); err == nil {
return path, nil
}
}
path, err := exec.LookPath("duckdb")
if err != nil {
return "", errors.New("no duckdb CLI found: run `go run ./cmd/goldeneye install duckdb` in internal/goldeneye, set DUCKDB to a DuckDB 2.0 binary, or put duckdb on PATH")
}
return path, nil
}
// Version reports the release a CLI is.
func Version(ctx context.Context, binary string) (string, error) {
out, err := exec.CommandContext(ctx, binary, "--version").Output()
if err != nil {
return "", fmt.Errorf("duckdb --version: %w", err)
}
return "DuckDB " + strings.TrimSpace(string(out)), nil
}
// query runs a SQL statement against an in-memory database and decodes the
// CLI's JSON output into rows.
func query(ctx context.Context, binary, sql string, rows any) error {
cmd := exec.CommandContext(ctx, binary, "-json", ":memory:", "-c", sql)
out, err := cmd.Output()
if err != nil {
if exit, ok := err.(*exec.ExitError); ok {
return fmt.Errorf("duckdb: %s: %s", err, exit.Stderr)
}
return fmt.Errorf("duckdb: %w", err)
}
// An empty result set prints nothing rather than [].
if len(out) == 0 {
return nil
}
return json.Unmarshal(out, rows)
}
type typeRow struct {
TypeName string `json:"type_name"`
LogicalType string `json:"logical_type"`
Category *string `json:"type_category"`
}
type functionRow struct {
Name string `json:"function_name"`
FunctionType string `json:"function_type"`
ParameterTypes []string `json:"parameter_types"`
Varargs *string `json:"varargs"`
ReturnType *string `json:"return_type"`
}
// metaTypes are type ids that never describe a column's value: sentinels and
// binder-internal types the dump lists alongside the real ones.
var metaTypes = map[string]bool{
"null": true,
"unknown": true,
"invalid": true,
"any": true,
"type": true,
"lambda": true,
"table": true,
"pointer": true,
}
// categoryLetter maps duckdb_types() categories onto the PostgreSQL category
// letters the seed package uses.
func categoryLetter(category *string) string {
if category == nil {
return "U"
}
switch *category {
case "NUMERIC":
return "N"
case "STRING":
return "S"
case "DATETIME":
return "D"
case "BOOLEAN":
return "B"
case "COMPOSITE":
return "C"
default:
return "U"
}
}
func readTypes(ctx context.Context, binary string) ([]dialect.Type, error) {
var rows []typeRow
err := query(ctx, binary, `
SELECT type_name, logical_type, type_category
FROM duckdb_types()
WHERE database_name = 'system'
ORDER BY type_name`, &rows)
if err != nil {
return nil, err
}
// Group the dump's one-row-per-spelling by logical type: the spelling
// matching the logical type id is the canonical name, the rest are
// aliases. A spelling is listed once per schema it is visible in, so
// an alias is kept once.
grouped := map[string]*dialect.Type{}
var order []string
seen := map[string]bool{}
for _, row := range rows {
logical := strings.ToLower(row.LogicalType)
if metaTypes[logical] {
continue
}
t, ok := grouped[logical]
if !ok {
t = &dialect.Type{Name: logical, Category: categoryLetter(row.Category)}
grouped[logical] = t
order = append(order, logical)
}
if t.Category == "U" {
t.Category = categoryLetter(row.Category)
}
if name := strings.ToLower(row.TypeName); name != logical && !seen[logical+"\x00"+name] {
seen[logical+"\x00"+name] = true
t.Aliases = append(t.Aliases, name)
}
}
sort.Strings(order)
types := make([]dialect.Type, 0, len(order)+1)
// "any" stands in for the generic parameters of DuckDB's polymorphic
// functions (ANY, T, K, V); the analyzer resolves a call returning it
// to the type of the call's first argument.
types = append(types, dialect.Type{Name: "any", Category: "U"})
for _, name := range order {
types = append(types, *grouped[name])
}
return types, nil
}
// typeNames is the set of names the seed declares, for filtering out function
// overloads over generic or binder-internal types.
func typeNames(types []dialect.Type) map[string]bool {
names := map[string]bool{}
for _, t := range types {
names[t.Name] = true
for _, alias := range t.Aliases {
names[alias] = true
}
}
return names
}
// seedTypeName maps a duckdb_functions() type spelling to a seeded type name:
// lowercased, with modifiers dropped (DECIMAL(18,3) is a decimal), array
// markers kept (fixed sizes and generic bounds included: DOUBLE[3] and
// ANY[] are arrays), and the generic parameters of polymorphic functions
// mapped to "any". It reports false for the internal types a seed cannot
// describe.
func seedTypeName(name string, known map[string]bool) (string, bool) {
name = strings.ToLower(strings.TrimSpace(name))
if close := strings.LastIndexByte(name, ']'); close == len(name)-1 {
if open := strings.LastIndexByte(name, '['); open != -1 {
elementName, ok := seedTypeName(name[:open], known)
return elementName + "[]", ok
}
}
if open := strings.IndexByte(name, '('); open != -1 {
name = name[:open]
}
switch name {
case "any", "t", "k", "v":
return "any", true
}
if !known[name] {
return "", false
}
return name, true
}
// isOperatorName reports a function named by symbols rather than an
// identifier — the spelling of a binary or prefix operator.
func isOperatorName(name string) bool {
return strings.IndexFunc(name, func(r rune) bool {
return (r < 'a' || r > 'z') && (r < '0' || r > '9') && r != '_' && r != '$'
}) != -1
}
func functionKind(functionType string) string {
switch functionType {
case "aggregate":
return "a"
case "window":
return "w"
default:
return ""
}
}
func readFunctions(ctx context.Context, binary string, known map[string]bool) ([]dialect.Function, []dialect.Operator, error) {
var rows []functionRow
err := query(ctx, binary, `
SELECT DISTINCT function_name, function_type, parameter_types, varargs, return_type
FROM duckdb_functions()
WHERE database_name = 'system'
AND schema_name = 'main'
AND function_type IN ('scalar', 'aggregate', 'window', 'macro')
ORDER BY function_name, parameter_types::VARCHAR, return_type`, &rows)
if err != nil {
return nil, nil, err
}
var funcs []dialect.Function
var operators []dialect.Operator
seenFunc := map[string]bool{}
seenOp := map[string]bool{}
for _, row := range rows {
if row.ReturnType == nil {
continue
}
returns, ok := seedTypeName(*row.ReturnType, known)
if !ok {
continue
}
args := make([]string, 0, len(row.ParameterTypes))
resolved := true
for _, param := range row.ParameterTypes {
arg, ok := seedTypeName(param, known)
if !ok {
resolved = false
break
}
args = append(args, arg)
}
if !resolved {
continue
}
if isOperatorName(row.Name) {
// Operator-named functions become operator entries; sqlc's
// engine reports them as operator applications. Only the
// binary form over concrete scalar types is representable —
// the seed's operator table, unlike its function table, does
// not register generic or array types on demand.
if row.FunctionType != "scalar" || len(args) != 2 || row.Varargs != nil {
continue
}
if concrete := !strings.Contains(returns, "any") && !strings.Contains(returns, "[]") &&
!strings.Contains(args[0], "any") && !strings.Contains(args[0], "[]") &&
!strings.Contains(args[1], "any") && !strings.Contains(args[1], "[]"); !concrete {
continue
}
key := row.Name + "\x00" + strings.Join(args, "\x00")
if seenOp[key] {
continue
}
seenOp[key] = true
operators = append(operators, dialect.Operator{
Name: row.Name,
Left: args[0],
Right: args[1],
Result: returns,
})
continue
}
fn := dialect.Function{
Name: row.Name,
Kind: functionKind(row.FunctionType),
Returns: returns,
// An aggregate over no rows returns NULL — except count,
// which returns 0.
Nullable: row.FunctionType == "aggregate" && !strings.HasPrefix(row.Name, "count"),
}
for _, arg := range args {
fn.Args = append(fn.Args, dialect.Arg{Type: arg})
}
if row.Varargs != nil {
vararg, ok := seedTypeName(*row.Varargs, known)
if !ok {
continue
}
fn.Args = append(fn.Args, dialect.Arg{Type: vararg, Mode: "v"})
}
key := fn.Name + "\x00" + fn.Kind + "\x00" + strings.Join(args, "\x00")
if seenFunc[key] {
continue
}
seenFunc[key] = true
funcs = append(funcs, fn)
}
return funcs, operators, nil
}
// Generate reads the dialect from the CLI.
func Generate(ctx context.Context, binary string) (dialect.Files, error) {
types, err := readTypes(ctx, binary)
if err != nil {
return nil, err
}
funcs, operators, err := readFunctions(ctx, binary, typeNames(types))
if err != nil {
return nil, err
}
files := dialect.Files{}
if files[dialect.TypesFile], err = dialect.JSONL(types); err != nil {
return nil, err
}
if files[dialect.FunctionsFile], err = dialect.JSONL(funcs); err != nil {
return nil, err
}
if files[dialect.OperatorsFile], err = dialect.JSONL(operators); err != nil {
return nil, err
}
return files, nil
}

66
internal/goldeneye/duckdb/duckdb_test.go Normal file
View File

@@ -0,0 +1,66 @@
package duckdb
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// TestDialect verifies the committed DuckDB dialect against what the DuckDB
// CLI reports. It skips unless a CLI is found.
func TestDialect(t *testing.T) {
binary, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, binary)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, binary)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Engine)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}
// TestAnalyzeCases verifies every DuckDB analyze case under
// internal/endtoend/testdata against what the CLI reports. It skips
// unless a CLI is found.
func TestAnalyzeCases(t *testing.T) {
binary, err := Locate()
if err != nil {
t.Skip(err)
}
cases, err := endtoend.Cases(Engine)
if err != nil {
t.Fatal(err)
}
if len(cases) == 0 {
t.Fatal("no duckdb analyze cases found")
}
for _, c := range cases {
t.Run(c.Name, func(t *testing.T) {
diff, err := Check(context.Background(), binary, c)
if err != nil {
t.Fatal(err)
}
if diff != "" {
t.Errorf("%s does not match what DuckDB reports (-committed +duckdb):\n%s", c.Output, diff)
}
})
}
}

136
internal/goldeneye/duckdb/install.go Normal file
View File

@@ -0,0 +1,136 @@
package duckdb
import (
"archive/tar"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"
)
// DefaultVersion is the DuckDB preview channel the CLI is downloaded from:
// the v2.0 development builds, which DuckDB publishes as a rolling tarball
// per platform under https://artifacts.duckdb.org/<channel>/ until 2.0 is
// released. A channel has no per-build download and no checksum to pin,
// so Install fetches whatever build the channel holds and the version the
// CLI reports is what a run logs; GeneratedFrom records the build the
// committed dialect came from, and a check against a later build reports
// what the later build added. Once 2.0 is released, pin the release and
// its checksums here the way the clickhouse package does.
const DefaultVersion = "v2.0-cyanoptera"
// GeneratedFrom is the build the committed dialect was generated from, as
// `duckdb --version` reports it. Update it when regenerating.
const GeneratedFrom = "v2.0.0-alpha41396 (Cyanoptera) d41e527b18"
// channelURL is the download address of a channel's CLI tarball for a
// platform. DuckDB publishes one macOS build for both architectures.
func channelURL(channel, goos, goarch string) (string, error) {
var name string
switch {
case goos == "linux" && (goarch == "amd64" || goarch == "arm64"):
name = "duckdb-cli-linux-" + goarch + ".tar.gz"
case goos == "darwin":
name = "duckdb-cli-osx-universal.tar.gz"
default:
return "", fmt.Errorf("no DuckDB %s build is published for %s/%s", channel, goos, goarch)
}
return "https://artifacts.duckdb.org/" + channel + "/" + name, nil
}
// cachedBinary is where Install puts the binary for a channel.
func cachedBinary(channel string) (string, error) {
dir, err := os.UserCacheDir()
if err != nil {
return "", err
}
return filepath.Join(dir, "sqlc-duckdb", channel, "duckdb"), nil
}
// Install downloads the channel's current CLI into the cache and returns
// its path. It is a no-op when the channel is already cached: remove the
// cached directory to fetch the channel's newer build.
func Install(ctx context.Context, channel, goos, goarch string, progress io.Writer) (string, error) {
dest, err := cachedBinary(channel)
if err != nil {
return "", err
}
if _, err := os.Stat(dest); err == nil {
return dest, nil
}
url, err := channelURL(channel, goos, goarch)
if err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", err
}
fmt.Fprintf(progress, "downloading %s\n", url)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("downloading %s: %s", url, resp.Status)
}
// Write next to the destination and rename so a partial download never
// masquerades as an installed binary.
tmp, err := os.CreateTemp(filepath.Dir(dest), "duckdb-*.partial")
if err != nil {
return "", err
}
defer os.Remove(tmp.Name())
src, err := binaryInTarball(resp.Body)
if err != nil {
tmp.Close()
return "", fmt.Errorf("downloading %s: %w", url, err)
}
if _, err := io.Copy(tmp, src); err != nil {
tmp.Close()
return "", err
}
if err := tmp.Close(); err != nil {
return "", err
}
if err := os.Chmod(tmp.Name(), 0o755); err != nil {
return "", err
}
if err := os.Rename(tmp.Name(), dest); err != nil {
return "", err
}
return dest, nil
}
// binaryInTarball positions a reader at the duckdb binary inside a CLI
// tarball, which holds it at the top level.
func binaryInTarball(r io.Reader) (io.Reader, error) {
gz, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
tr := tar.NewReader(gz)
for {
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
return nil, errors.New("tarball does not contain duckdb")
}
if err != nil {
return nil, err
}
if hdr.Typeflag == tar.TypeReg && strings.TrimPrefix(hdr.Name, "./") == "duckdb" {
return tr, nil
}
}
}

657
internal/goldeneye/duckdb/text.go Normal file
View File

@@ -0,0 +1,657 @@
package duckdb
import (
"strings"
)
// DuckDB prints a plan with every column by its bare name and every
// aliased expression by its alias, and describes a query no further than
// its result columns' names and types, so which table a result column is
// read from and which column a parameter is compared with are read from
// the query text: the select list's items, the FROM clause's tables and
// aliases, and the operand beside each parameter. The text is tokenized
// and read at the top level of the statement, outside the parentheses a
// subquery or a CTE body sits in.
// token is one lexical element of a query.
type token struct {
kind byte // 'w' word, 'q' quoted identifier, 's' string, 'n' number, 'p' parameter, 'o' operator or punctuation
text string
}
var operators = []string{"::", "<>", "!=", "<=", ">=", "!~~", "~~", "->>", "->", "||", "=", "<", ">", "(", ")", ",", ".", "*", "+", "-", "/", "%", "[", "]", ";", "^", "&", "|", "~", "!", ":"}
// tokenize splits a query into tokens, dropping comments. A parameter is
// $ followed by digits, which is how bind spells every parameter.
func tokenize(src string) []token {
var out []token
i := 0
for i < len(src) {
c := src[i]
switch {
case c == ' ' || c == '\t' || c == '\n' || c == '\r':
i++
case strings.HasPrefix(src[i:], "--"):
end := strings.IndexByte(src[i:], '\n')
if end < 0 {
i = len(src)
} else {
i += end
}
case strings.HasPrefix(src[i:], "/*"):
end := strings.Index(src[i:], "*/")
if end < 0 {
i = len(src)
} else {
i += end + 2
}
case c == '\'':
end := quotedEnd(src, i)
out = append(out, token{'s', src[i:end]})
i = end
case c == '"':
end := quotedEnd(src, i)
out = append(out, token{'q', strings.ReplaceAll(src[i+1:end-1], `""`, `"`)})
i = end
case c == '$' && i+1 < len(src) && isDigit(src[i+1]):
end := i + 1
for end < len(src) && isDigit(src[end]) {
end++
}
out = append(out, token{'p', src[i+1 : end]})
i = end
case isDigit(c) || (c == '.' && i+1 < len(src) && isDigit(src[i+1])):
end := i
for end < len(src) && (isDigit(src[end]) || src[end] == '.' || src[end] == 'e' || src[end] == 'E' || src[end] == '_') {
end++
}
out = append(out, token{'n', src[i:end]})
i = end
case isWordStart(c):
end := i
for end < len(src) && isWordByte(src[end]) {
end++
}
out = append(out, token{'w', src[i:end]})
i = end
default:
matched := false
for _, op := range operators {
if strings.HasPrefix(src[i:], op) {
out = append(out, token{'o', op})
i += len(op)
matched = true
break
}
}
if !matched {
out = append(out, token{'o', string(c)})
i++
}
}
}
return out
}
// quotedEnd returns the index just past the quoted token starting at i,
// whose delimiter is escaped by doubling it.
func quotedEnd(s string, i int) int {
q := s[i]
j := i + 1
for j < len(s) {
switch {
case s[j] == q && j+1 < len(s) && s[j+1] == q:
j += 2
case s[j] == q:
return j + 1
default:
j++
}
}
return len(s)
}
func isDigit(c byte) bool { return c >= '0' && c <= '9' }
func isWordStart(c byte) bool {
return c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= 0x80
}
func isWordByte(c byte) bool { return isWordStart(c) || isDigit(c) }
// text is a tokenized query.
type text []token
func (t text) at(i int) token {
if i < 0 || i >= len(t) {
return token{}
}
return t[i]
}
// isWord reports whether token i is the keyword, in any case.
func (t text) isWord(i int, word string) bool {
tok := t.at(i)
return tok.kind == 'w' && strings.EqualFold(tok.text, word)
}
// isOp reports whether token i is the operator.
func (t text) isOp(i int, op string) bool {
tok := t.at(i)
return tok.kind == 'o' && tok.text == op
}
// isName reports whether token i can name something: a word or a quoted
// identifier.
func (t text) isName(i int) bool {
k := t.at(i).kind
return k == 'w' || k == 'q'
}
// group returns the index of the parenthesis closing the one at i.
func (t text) group(i int) int {
depth := 0
for j := i; j < len(t); j++ {
switch {
case t.isOp(j, "("):
depth++
case t.isOp(j, ")"):
depth--
if depth == 0 {
return j
}
}
}
return len(t) - 1
}
// ref is a column reference: a qualifier, empty for none, and a column,
// or "*" for every column of the qualifier.
type ref struct {
qualifier, column string
}
// readRef reads a dotted name starting at i, returning the parts and the
// index after them.
func (t text) readRef(i int) ([]string, int) {
var parts []string
for t.isName(i) || t.isOp(i, "*") {
parts = append(parts, t.at(i).text)
if !t.isOp(i+1, ".") {
return parts, i + 1
}
i += 2
}
return parts, i
}
// keywords that end a list of tables or select items at the top level.
var clauseKeywords = map[string]bool{
"where": true, "group": true, "order": true, "limit": true, "offset": true, "having": true,
"qualify": true, "window": true, "returning": true, "set": true, "values": true, "select": true,
"union": true, "except": true, "intersect": true, "on": true, "from": true, "join": true,
"left": true, "right": true, "full": true, "inner": true, "cross": true, "outer": true,
"natural": true, "asof": true, "semi": true, "anti": true, "positional": true, "lateral": true,
"using": true, "with": true, "into": true, "default": true, "fetch": true, "for": true,
"tablesample": true, "sample": true, "by": true, "repeatable": true,
}
// typeWords are the words a multi-word type name continues with:
// TIMESTAMP WITH TIME ZONE, DOUBLE PRECISION, CHARACTER VARYING.
var typeWords = map[string]bool{"with": true, "without": true, "time": true, "zone": true, "precision": true, "varying": true}
// tableRef is one table of a statement's scope, by name and alias.
type tableRef struct {
name, alias string
}
// scope is what the statement reads and writes: the tables of its FROM,
// JOIN and USING clauses and the table a DML statement targets, with the
// names of its CTEs, which are not tables.
type scope struct {
kind string // select, insert, update, delete
target tableRef
tables []tableRef
ctes map[string]bool
}
// readScope reads a statement's scope from its top-level tokens.
func (t text) readScope() scope {
sc := scope{ctes: map[string]bool{}}
i := 0
// WITH name [(cols)] AS (...), ...
if t.isWord(0, "with") {
i = 1
if t.isWord(i, "recursive") {
i++
}
for t.isName(i) {
sc.ctes[strings.ToLower(t.at(i).text)] = true
i++
if t.isOp(i, "(") {
i = t.group(i) + 1
}
if t.isWord(i, "as") {
i++
}
if t.isWord(i, "not") || t.isWord(i, "materialized") {
for !t.isOp(i, "(") && i < len(t) {
i++
}
}
if t.isOp(i, "(") {
i = t.group(i) + 1
}
if t.isOp(i, ",") {
i++
continue
}
break
}
}
sc.kind = "select"
switch {
case t.isWord(i, "insert"):
sc.kind = "insert"
for i < len(t) && !t.isWord(i, "into") {
i++
}
sc.target, i = t.readTable(i + 1)
case t.isWord(i, "update"):
sc.kind = "update"
sc.target, i = t.readTable(i + 1)
case t.isWord(i, "delete"):
sc.kind = "delete"
if t.isWord(i+1, "from") {
i++
}
sc.target, i = t.readTable(i + 1)
}
for ; i < len(t); i++ {
switch {
case t.isOp(i, "("):
i = t.group(i)
case t.isWord(i, "from") || t.isWord(i, "join"):
i = t.readTables(i+1, &sc.tables) - 1
case t.isWord(i, "using") && !t.isOp(i+1, "("):
i = t.readTables(i+1, &sc.tables) - 1
}
}
// A CTE is not a table, whatever table shares its name.
tables := sc.tables[:0]
for _, ref := range sc.tables {
if !sc.ctes[ref.name] {
tables = append(tables, ref)
}
}
sc.tables = tables
return sc
}
// readTable reads a table name with an optional alias.
func (t text) readTable(i int) (tableRef, int) {
parts, j := t.readRef(i)
if len(parts) == 0 {
return tableRef{}, i
}
ref := tableRef{name: strings.ToLower(parts[len(parts)-1])}
if t.isWord(j, "as") {
j++
}
if t.isName(j) && !clauseKeywords[strings.ToLower(t.at(j).text)] {
ref.alias = strings.ToLower(t.at(j).text)
j++
}
return ref, j
}
// readTables reads a comma-separated list of table references, skipping
// subqueries and table functions, and returns the index after it.
func (t text) readTables(i int, into *[]tableRef) int {
for i < len(t) {
if t.isOp(i, "(") {
// A derived table or a table function's arguments.
i = t.group(i) + 1
if t.isWord(i, "as") {
i++
}
if t.isName(i) && !clauseKeywords[strings.ToLower(t.at(i).text)] {
i++
}
} else if t.isName(i) && !clauseKeywords[strings.ToLower(t.at(i).text)] {
ref, j := t.readTable(i)
if t.isOp(j, "(") {
// A table function: its result is not a table.
j = t.group(j) + 1
} else {
*into = append(*into, ref)
}
i = j
} else {
return i
}
if t.isOp(i, ",") {
i++
continue
}
return i
}
return i
}
// item is one entry of a select list or a RETURNING list.
type item struct {
star bool // every column of qualifier, or of every table
ref *ref // a column reference, when the item is one
name string // the alias, or the column name of a reference
}
// items splits a select or RETURNING list into its items, given the index
// of the first token after the keyword.
func (t text) items(start int) []item {
var out []item
i := start
for i < len(t) {
end := i
for end < len(t) && !t.isOp(end, ",") && !(t.at(end).kind == 'w' && clauseKeywords[strings.ToLower(t.at(end).text)] && !t.isWord(end, "on")) && !t.isOp(end, ";") {
if t.isOp(end, "(") {
end = t.group(end)
}
end++
}
if end > i {
out = append(out, t.item(i, end))
}
if !t.isOp(end, ",") {
break
}
i = end + 1
}
return out
}
// item classifies the tokens of one select item.
func (t text) item(start, end int) item {
parts, j := t.readRef(start)
if len(parts) > 0 && parts[len(parts)-1] == "*" {
it := item{star: true}
if len(parts) > 1 {
it.ref = &ref{qualifier: strings.ToLower(parts[len(parts)-2]), column: "*"}
}
return it
}
var it item
if len(parts) > 0 && j == end {
it.ref = &ref{column: parts[len(parts)-1]}
if len(parts) > 1 {
it.ref.qualifier = strings.ToLower(parts[len(parts)-2])
}
it.name = parts[len(parts)-1]
return it
}
if len(parts) > 0 && j == end-1 && t.isName(end-1) {
// A reference followed by a bare alias.
it.ref = &ref{column: parts[len(parts)-1]}
if len(parts) > 1 {
it.ref.qualifier = strings.ToLower(parts[len(parts)-2])
}
it.name = t.at(end - 1).text
return it
}
if len(parts) > 0 && j == end-2 && t.isWord(end-2, "as") && t.isName(end-1) {
it.ref = &ref{column: parts[len(parts)-1]}
if len(parts) > 1 {
it.ref.qualifier = strings.ToLower(parts[len(parts)-2])
}
it.name = t.at(end - 1).text
return it
}
if t.isWord(end-2, "as") && t.isName(end-1) {
it.name = t.at(end - 1).text
}
return it
}
// selectItems finds the statement's select list: the items after its
// top-level SELECT, or every column when a FROM-first query has none.
func (t text) selectItems() []item {
for i := 0; i < len(t); i++ {
switch {
case t.isOp(i, "("):
i = t.group(i)
case t.isWord(i, "select"):
j := i + 1
if t.isWord(j, "distinct") || t.isWord(j, "all") {
j++
if t.isWord(j, "on") && t.isOp(j+1, "(") {
j = t.group(j+1) + 1
}
}
return t.items(j)
}
}
if t.isWord(0, "from") {
return []item{{star: true}}
}
return nil
}
// returningItems finds the items of a top-level RETURNING clause.
func (t text) returningItems() []item {
for i := 0; i < len(t); i++ {
switch {
case t.isOp(i, "("):
i = t.group(i)
case t.isWord(i, "returning"):
return t.items(i + 1)
}
}
return nil
}
// inSubquery reports whether parameter k sits inside a subquery: a
// parenthesis that a SELECT, WITH, FROM or VALUES opens.
func (t text) inSubquery(k string) bool {
i := t.find(k)
depth := 0
for j := i - 1; j >= 0; j-- {
switch {
case t.isOp(j, ")"):
depth++
case t.isOp(j, "("):
if depth > 0 {
depth--
continue
}
if t.isWord(j+1, "select") || t.isWord(j+1, "with") || t.isWord(j+1, "from") || t.isWord(j+1, "values") {
return true
}
}
}
return false
}
// find returns the index of parameter k.
func (t text) find(k string) int {
for i, tok := range t {
if tok.kind == 'p' && tok.text == k {
return i
}
}
return -1
}
var comparisons = map[string]bool{"=": true, "<>": true, "!=": true, "<": true, ">": true, "<=": true, ">=": true, "~~": true, "!~~": true}
// partner finds the column a parameter is compared with or assigned to:
// the column reference on the other side of the comparison, LIKE or IN
// it is an operand of, or the column a SET assigns it to.
func (t text) partner(k string) (ref, bool) {
i := t.find(k)
if i < 0 {
return ref{}, false
}
// col op $k, col [NOT] LIKE $k, col IN ($k, ...)
j := i - 1
if t.isOp(j, "(") || t.isOp(j, ",") {
for j >= 0 && (t.isOp(j, ",") || t.at(j).kind == 'p' || t.at(j).kind == 's' || t.at(j).kind == 'n') {
j--
}
if t.isOp(j, "(") {
j--
}
}
if t.at(j).kind == 'o' && comparisons[t.at(j).text] || t.isWord(j, "like") || t.isWord(j, "ilike") || t.isWord(j, "glob") || t.isWord(j, "in") {
if t.isWord(j-1, "not") {
j--
}
if r, ok := t.refEnding(j - 1); ok {
return r, true
}
}
// $k op col
j = i + 1
if t.at(j).kind == 'o' && comparisons[t.at(j).text] {
if r, ok := t.refStarting(j + 1); ok {
return r, true
}
}
return ref{}, false
}
// refEnding reads the column reference whose last token is at i.
func (t text) refEnding(i int) (ref, bool) {
if !t.isName(i) {
return ref{}, false
}
r := ref{column: t.at(i).text}
if t.isOp(i-1, ".") && t.isName(i-2) {
r.qualifier = strings.ToLower(t.at(i - 2).text)
}
return r, true
}
// refStarting reads the column reference starting at i, which must not be
// a function call.
func (t text) refStarting(i int) (ref, bool) {
parts, j := t.readRef(i)
if len(parts) == 0 || t.isOp(j, "(") {
return ref{}, false
}
r := ref{column: parts[len(parts)-1]}
if len(parts) > 1 {
r.qualifier = strings.ToLower(parts[len(parts)-2])
}
return r, true
}
// castOf returns the type a parameter is cast to, as $k::T or CAST($k AS
// T), spelled as the query spells it.
func (t text) castOf(k string) (string, bool) {
i := t.find(k)
if i < 0 {
return "", false
}
if t.isOp(i+1, "::") {
return t.typeText(i + 2), true
}
if t.isWord(i-2, "cast") && t.isOp(i-1, "(") && t.isWord(i+1, "as") {
return t.typeText(i + 2), true
}
return "", false
}
// typeText reads a type starting at i: a dotted name, optional arguments
// in parentheses and any number of [] or [N] suffixes.
func (t text) typeText(i int) string {
start := i
_, i = t.readRef(i)
for t.at(i).kind == 'w' && typeWords[strings.ToLower(t.at(i).text)] {
// Multi-word names: TIMESTAMP WITH TIME ZONE.
i++
}
if t.isOp(i, "(") {
i = t.group(i) + 1
}
for t.isOp(i, "[") {
for i < len(t) && !t.isOp(i, "]") {
i++
}
i++
}
var b strings.Builder
for j := start; j < i && j < len(t); j++ {
tok := t.at(j)
if j > start && (tok.kind == 'w' || tok.kind == 'q' || tok.kind == 'n' || tok.kind == 's') && (t.at(j-1).kind == 'w' || t.at(j-1).kind == 'q' || t.isOp(j-1, ",")) {
b.WriteByte(' ')
}
b.WriteString(tok.text)
}
return b.String()
}
// valuesPosition reports, for a parameter inside an INSERT's VALUES, the
// index of the column it is a value for, counting positions within its
// row.
func (t text) valuesPosition(k string) (int, bool) {
i := t.find(k)
if i < 0 {
return 0, false
}
for j := 0; j < len(t); j++ {
if t.isOp(j, "(") {
j = t.group(j)
continue
}
if !t.isWord(j, "values") {
continue
}
// Rows follow, each a parenthesised list.
for r := j + 1; t.isOp(r, "("); r = t.group(r) + 2 {
end := t.group(r)
if i < r || i > end {
if !t.isOp(end+1, ",") {
break
}
continue
}
pos, depth := 0, 0
for q := r + 1; q < i; q++ {
switch {
case t.isOp(q, "("):
depth++
case t.isOp(q, ")"):
depth--
case t.isOp(q, ",") && depth == 0:
pos++
}
}
return pos, true
}
return 0, false
}
return 0, false
}
// insertColumns lists the columns an INSERT names, or nil for all.
func (t text) insertColumns() []string {
for i := 0; i < len(t); i++ {
if t.isWord(i, "into") {
_, j := t.readRef(i + 1)
if t.isWord(j, "as") {
j += 2
} else if t.isName(j) && !clauseKeywords[strings.ToLower(t.at(j).text)] {
j++
}
if !t.isOp(j, "(") {
return nil
}
var cols []string
for q := j + 1; q < t.group(j); q++ {
if t.isName(q) {
cols = append(cols, t.at(q).text)
}
}
return cols
}
}
return nil
}

220
internal/goldeneye/duckdb/types.go Normal file
View File

@@ -0,0 +1,220 @@
package duckdb
import (
"sort"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
)
// parseType reads a type the way DuckDB spells one — DECIMAL(10,2),
// INTEGER[], INTEGER[3], STRUCT(a INTEGER, b VARCHAR), MAP(VARCHAR,
// INTEGER), UNION(num INTEGER, str VARCHAR), ENUM('a', 'b'), TIMESTAMP
// WITH TIME ZONE — into an expression in lower case, named the way the
// dialect names it when the spelling is one of the aliases types.jsonl
// lists. DuckDB spells an enum column by its labels whether the schema
// named the type or not, so labels that are those of an enum the schema
// created name that type.
func (a *analyzer) parseType(s string) *analysis.TypeExpr {
return a.parseTypeWith(s, true)
}
// parseTypeWith reads a type, naming an alias as the dialect does when
// canonical is set and as DuckDB spells it otherwise.
func (a *analyzer) parseTypeWith(s string, canonical bool) *analysis.TypeExpr {
s = strings.TrimSpace(s)
if strings.HasSuffix(s, "]") {
open := strings.LastIndexByte(s, '[')
if open > 0 {
t := &analysis.TypeExpr{Name: "array", Args: []analysis.TypeArg{{Type: a.parseTypeWith(s[:open], canonical)}}}
if n, err := strconv.ParseInt(strings.TrimSpace(s[open+1:len(s)-1]), 10, 64); err == nil {
t.Args = append(t.Args, analysis.TypeArg{Int: &n})
}
return t
}
}
name, args := s, ""
if open := strings.IndexByte(s, '('); open >= 0 && strings.HasSuffix(s, ")") {
name, args = strings.TrimSpace(s[:open]), s[open+1:len(s)-1]
}
name = strings.ToLower(name)
if c, ok := a.canonical[name]; ok && canonical {
name = c
}
t := &analysis.TypeExpr{Name: name}
if args == "" {
return t
}
switch name {
case "struct", "union":
for _, f := range splitTop(args, ',') {
f = strings.TrimSpace(f)
label, typ := f, ""
if strings.HasPrefix(f, `"`) {
end := quotedEnd(f, 0)
label, typ = strings.ReplaceAll(f[1:end-1], `""`, `"`), f[end:]
} else if i := strings.IndexByte(f, ' '); i > 0 {
label, typ = f[:i], f[i+1:]
}
t.Args = append(t.Args, analysis.TypeArg{Label: label, Type: a.parseTypeWith(typ, canonical)})
}
case "enum":
var labels []string
for _, l := range splitTop(args, ',') {
l = strings.TrimSpace(l)
if strings.HasPrefix(l, "'") && strings.HasSuffix(l, "'") && len(l) >= 2 {
l = strings.ReplaceAll(l[1:len(l)-1], "''", "'")
}
labels = append(labels, l)
}
if named, ok := a.enumNamed(labels); ok {
return &analysis.TypeExpr{Name: named}
}
for _, l := range labels {
l := l
t.Args = append(t.Args, analysis.TypeArg{String: &l})
}
default:
for _, arg := range splitTop(args, ',') {
arg = strings.TrimSpace(arg)
if n, err := strconv.ParseInt(arg, 10, 64); err == nil {
t.Args = append(t.Args, analysis.TypeArg{Int: &n})
} else {
t.Args = append(t.Args, analysis.TypeArg{Type: a.parseTypeWith(arg, canonical)})
}
}
}
return t
}
// enumNamed finds the enum type the schema created with these labels,
// the first by name when two share them.
func (a *analyzer) enumNamed(labels []string) (string, bool) {
names := make([]string, 0, len(a.enums))
for name := range a.enums {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
have := a.enums[name]
if len(have) != len(labels) {
continue
}
same := true
for i := range have {
if have[i] != labels[i] {
same = false
break
}
}
if same {
return name, true
}
}
return "", false
}
// splitTop splits on a separator outside parentheses, brackets and
// quotes.
func splitTop(s string, sep byte) []string {
var out []string
depth, start := 0, 0
var quote byte
for i := 0; i < len(s); i++ {
c := s[i]
switch {
case quote != 0:
if c == quote {
quote = 0
}
case c == '\'' || c == '"':
quote = c
case c == '(' || c == '[':
depth++
case c == ')' || c == ']':
depth--
case c == sep && depth == 0:
out = append(out, s[start:i])
start = i + 1
}
}
return append(out, s[start:])
}
// zero is a value of a type, spelled as a string DuckDB casts to the type:
// what a parameter is bound to when the query is run to see which of its
// columns can be NULL. A type no value is known for reports false, and
// the parameter is bound to NULL.
func (a *analyzer) zero(t *analysis.TypeExpr) (string, bool) {
if t == nil {
return "", false
}
switch t.Name {
case "tinyint", "smallint", "integer", "bigint", "hugeint", "utinyint", "usmallint", "uinteger", "ubigint", "uhugeint",
"decimal", "float", "double", "bignum", "varint":
return "1", true
case "varchar", "text", "char", "bpchar":
return "x", true
case "json":
return "{}", true
case "blob":
return "", true
case "bit":
return "0", true
case "boolean":
return "true", true
case "uuid":
return "00000000-0000-0000-0000-000000000000", true
case "date":
return "2000-01-01", true
case "time", "time with time zone":
return "00:00:00", true
case "timestamp", "timestamp with time zone", "timestamp_s", "timestamp_ms", "timestamp_ns", "datetime":
return "2000-01-01 00:00:00", true
case "interval":
return "1 day", true
case "map":
return "{}", true
case "array":
if len(t.Args) == 0 || t.Args[0].Type == nil {
return "", false
}
elem, ok := a.zero(t.Args[0].Type)
if !ok {
return "", false
}
n := 1
if len(t.Args) > 1 && t.Args[1].Int != nil {
n = int(*t.Args[1].Int)
}
elems := make([]string, n)
for i := range elems {
elems[i] = elem
}
return "[" + strings.Join(elems, ", ") + "]", true
case "struct":
var fields []string
for _, f := range t.Args {
v, ok := a.zero(f.Type)
if !ok {
return "", false
}
fields = append(fields, f.Label+": "+v)
}
return "{" + strings.Join(fields, ", ") + "}", true
case "union":
if len(t.Args) == 0 {
return "", false
}
return a.zero(t.Args[0].Type)
case "enum":
if len(t.Args) > 0 && t.Args[0].String != nil {
return *t.Args[0].String, true
}
}
if labels, ok := a.enums[t.Name]; ok && len(labels) > 0 {
return labels[0], true
}
return "", false
}

140
internal/goldeneye/endtoend/endtoend.go Normal file
View File

@@ -0,0 +1,140 @@
// Package endtoend finds the analyze cases under internal/endtoend/testdata
// and compares an engine's own answer with the output a case committed,
// the way the dialect package compares a generated dialect with the
// committed one.
package endtoend
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"runtime"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Case is one analyze case: the files sqlc analyze ran with, the fixture
// loaded before the queries run against a real database, and the output
// sqlc committed.
type Case struct {
Name string // analyze_params/clickhouse
Dir string
Schema string
Query string
Fixture string // empty when the case has no fixture.sql
Output string
}
// Testdata returns the end-to-end testdata directory, found relative to this
// source file so the working directory does not matter.
func Testdata() (string, error) {
_, file, _, ok := runtime.Caller(0)
if !ok {
return "", errors.New("cannot locate the testcheck source directory")
}
dir := filepath.Join(filepath.Dir(file), "..", "..", "endtoend", "testdata")
if _, err := os.Stat(dir); err != nil {
return "", err
}
return filepath.Clean(dir), nil
}
// Cases lists the analyze cases for an engine: every analyze_*/<engine>
// directory whose exec.json runs the analyze command. A case that asks for
// the AST is skipped, since only sqlc can print that.
func Cases(engine string) ([]Case, error) {
root, err := Testdata()
if err != nil {
return nil, err
}
dirs, err := filepath.Glob(filepath.Join(root, "analyze_*", engine))
if err != nil {
return nil, err
}
var cases []Case
for _, dir := range dirs {
c, ok, err := load(dir)
if err != nil {
return nil, fmt.Errorf("%s: %w", dir, err)
}
if ok {
cases = append(cases, c)
}
}
return cases, nil
}
func load(dir string) (Case, bool, error) {
blob, err := os.ReadFile(filepath.Join(dir, "exec.json"))
if errors.Is(err, os.ErrNotExist) {
return Case{}, false, nil
}
if err != nil {
return Case{}, false, err
}
var exec struct {
Command string `json:"command"`
Args []string `json:"args"`
}
if err := json.Unmarshal(blob, &exec); err != nil {
return Case{}, false, fmt.Errorf("exec.json: %w", err)
}
if exec.Command != "analyze" {
return Case{}, false, nil
}
var schema, query string
for i := 0; i < len(exec.Args); i++ {
switch arg := exec.Args[i]; arg {
case "--schema", "-s", "--dialect", "-d":
i++
if arg == "--schema" || arg == "-s" {
if i < len(exec.Args) {
schema = exec.Args[i]
}
}
case "--ast":
return Case{}, false, nil
default:
if !strings.HasPrefix(arg, "-") {
query = arg
}
}
}
if schema == "" || query == "" {
return Case{}, false, errors.New("exec.json: analyze needs --schema and a query file")
}
c := Case{
Name: filepath.Join(filepath.Base(filepath.Dir(dir)), filepath.Base(dir)),
Dir: dir,
Schema: filepath.Join(dir, schema),
Query: filepath.Join(dir, query),
Output: filepath.Join(dir, "stdout.json"),
}
if fixture := filepath.Join(dir, "fixture.sql"); fileExists(fixture) {
c.Fixture = fixture
}
return c, true, nil
}
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}
// Compare checks an engine's answer against the case's committed output
// byte for byte and returns a line diff when they differ, or "" when they
// match.
func (c Case) Compare(got []byte) (string, error) {
want, err := os.ReadFile(c.Output)
if err != nil {
return "", err
}
if bytes.Equal(want, got) {
return "", nil
}
return dialect.Diff(string(want), string(got)), nil
}

183
internal/goldeneye/endtoend/query.go Normal file
View File

@@ -0,0 +1,183 @@
package endtoend
import (
"fmt"
"os"
"regexp"
"strings"
"unicode"
)
// Query is one entry of a sqlc query file.
type Query struct {
Name string
Cmd string
SQL string
}
// Queries reads the case's query file and splits it into its queries.
func (c Case) Queries() ([]Query, error) {
src, err := os.ReadFile(c.Query)
if err != nil {
return nil, err
}
queries, err := ParseQueries(string(src))
if err != nil {
return nil, fmt.Errorf("%s: %w", c.Query, err)
}
return queries, nil
}
var headerRe = regexp.MustCompile(`^\s*--\s*name:\s*(\S+)\s+(:\S+)\s*$`)
// ParseQueries splits a sqlc query file on its `-- name: X :cmd` headers.
func ParseQueries(src string) ([]Query, error) {
var queries []Query
var cur *Query
var body []string
flush := func() error {
if cur == nil {
return nil
}
sql := strings.TrimSpace(strings.Join(body, "\n"))
sql = strings.TrimRight(sql, "; \t\r\n")
if sql == "" {
return fmt.Errorf("query %s has no body", cur.Name)
}
cur.SQL = sql
queries = append(queries, *cur)
return nil
}
for _, line := range strings.Split(src, "\n") {
if m := headerRe.FindStringSubmatch(line); m != nil {
if err := flush(); err != nil {
return nil, err
}
cur = &Query{Name: m[1], Cmd: m[2]}
body = body[:0]
continue
}
if cur != nil {
body = append(body, line)
}
}
if err := flush(); err != nil {
return nil, err
}
if len(queries) == 0 {
return nil, fmt.Errorf("no queries found: expected `-- name: Name :cmd` headers")
}
return queries, nil
}
var namedArgRe = regexp.MustCompile(`^sqlc\.(n?arg|slice)\(\s*'?([A-Za-z_][A-Za-z0-9_]*)'?\s*\)`)
// typedParamRe matches ClickHouse's {name:Type} parameter, whose type is
// the query's own business: the engine binds it as it binds any other.
var typedParamRe = regexp.MustCompile(`^\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*:\s*[^}]+\}`)
// Rewrite replaces every parameter reference in a query — ?, sqlc.arg(name),
// sqlc.narg(name), sqlc.slice(name) and ClickHouse's {name:Type} — with
// what bind returns for it, in order of appearance, skipping string
// literals, quoted identifiers and comments. bind is handed the name, empty for a ?, and the word before the
// reference, so that a LIMIT or OFFSET can be bound differently from a
// value; the second count of a LIMIT ?, ? is handed LIMIT as well. Each
// engine decides what a reference becomes and how the references are
// numbered.
func Rewrite(sql string, bind func(name, lastWord string) string) string {
var (
out strings.Builder
lastWord string
i = 0
)
for i < len(sql) {
c := sql[i]
switch {
case c == '\'' || c == '"' || c == '`':
end := skipQuoted(sql, i)
out.WriteString(sql[i:end])
i = end
case strings.HasPrefix(sql[i:], "--"):
end := strings.IndexByte(sql[i:], '\n')
if end < 0 {
end = len(sql)
} else {
end += i
}
out.WriteString(sql[i:end])
i = end
case strings.HasPrefix(sql[i:], "/*"):
end := strings.Index(sql[i:], "*/")
if end < 0 {
end = len(sql)
} else {
end += i + 2
}
out.WriteString(sql[i:end])
i = end
case c == '?':
out.WriteString(bind("", lastWord))
lastWord = afterReference(lastWord)
i++
case c == 's' && namedArgRe.MatchString(sql[i:]):
m := namedArgRe.FindStringSubmatch(sql[i:])
out.WriteString(bind(m[2], lastWord))
lastWord = afterReference(lastWord)
i += len(m[0])
case c == '{' && typedParamRe.MatchString(sql[i:]):
m := typedParamRe.FindStringSubmatch(sql[i:])
out.WriteString(bind(m[1], lastWord))
lastWord = afterReference(lastWord)
i += len(m[0])
case isWordByte(c):
end := i
for end < len(sql) && isWordByte(sql[end]) {
end++
}
lastWord = sql[i:end]
out.WriteString(lastWord)
i = end
default:
if !unicode.IsSpace(rune(c)) && c != ',' && c != '(' {
lastWord = ""
}
out.WriteByte(c)
i++
}
}
return out.String()
}
// afterReference is the word the reference after one is preceded by: a
// LIMIT's, so that both counts of LIMIT ?, ? are bound as counts, and
// otherwise none.
func afterReference(lastWord string) string {
if strings.EqualFold(lastWord, "limit") {
return lastWord
}
return ""
}
func isWordByte(c byte) bool {
return c == '_' || c == '.' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'
}
// skipQuoted returns the index just past the quoted token starting at i,
// honouring backslash escapes and doubled quotes.
func skipQuoted(s string, i int) int {
q := s[i]
j := i + 1
for j < len(s) {
switch {
case s[j] == '\\' && j+1 < len(s):
j += 2
case s[j] == q && j+1 < len(s) && s[j+1] == q:
j += 2
case s[j] == q:
return j + 1
default:
j++
}
}
return len(s)
}

52
internal/goldeneye/go.mod Normal file
View File

@@ -0,0 +1,52 @@
module github.com/sqlc-dev/sqlc/internal/goldeneye
go 1.26.0
require (
cloud.google.com/go/spanner v1.95.1
github.com/go-sql-driver/mysql v1.10.0
github.com/jackc/pgx/v5 v5.10.0
github.com/microsoft/go-mssqldb v1.11.0
google.golang.org/api v0.297.0
google.golang.org/grpc v1.83.2
google.golang.org/protobuf v1.36.12
)
require (
cloud.google.com/go v0.123.0 // indirect
cloud.google.com/go/auth v0.23.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/iam v1.12.0 // indirect
cloud.google.com/go/longrunning v1.2.0 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.20 // indirect
github.com/googleapis/gax-go/v2 v2.24.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.67.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/time v0.15.0 // indirect
google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260715232425-e75dac1f907d // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
)

137
internal/goldeneye/go.sum Normal file
View File

@@ -0,0 +1,137 @@
cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE=
cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU=
cloud.google.com/go/auth v0.23.2 h1:pxSCpfiji41hpzpPdMCftEUCezpgpqmmDdYiAjCKXxo=
cloud.google.com/go/auth v0.23.2/go.mod h1:4DhBRcqvtljQN3dJ57qtqbib5ZGCYE5f2crfiiC2EM0=
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
cloud.google.com/go/iam v1.12.0 h1:Aki3bX9aHUDKPHfnRJfDcTdVedvy6quGBQcTqx3DRXk=
cloud.google.com/go/iam v1.12.0/go.mod h1:FEZ4lXpADAC2AIpQY7LANNjjwyQ2jK439CI2VaD+sLY=
cloud.google.com/go/longrunning v1.2.0 h1:WjYH3YHBGCxGJP9M4dWGHBfXr/cFIjMkNgWcJj7/iMM=
cloud.google.com/go/longrunning v1.2.0/go.mod h1:5KMQALFGOCtFoi2xSOA1u3H7WKlhmckgiyFw7+LGQp0=
cloud.google.com/go/spanner v1.95.1 h1:9HYr+AAeAOubn0NZAYv34dFHQ3NbIUcWHZmgJvufPzk=
cloud.google.com/go/spanner v1.95.1/go.mod h1:Z2+83J5oVDmd1n5ntVMmjEuiNoXOpAyNeG7y1tuEHk0=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.0 h1:4gRPBpN1f6xt88yi4WR26m7XaD9OlWtVT6bWPdGUIok=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.0/go.mod h1:G7QVLxw1j1JVyrO1MA95S8m8HStaaleDZYTcfGgjB2o=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0 h1:CU4+EJeJi3TKYWEcYuSdWsjzw0nVsK/H0MSQOiPcymU=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0/go.mod h1:q0+UTSRvShwUCrR/s5HtyInYphN7Wvxb7snFM3u+SLA=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 h1:fhqpLE3UEXi9lPaBRpQ6XuRW0nU7hgg4zlmZZa+a9q4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0/go.mod h1:7dCRMLwisfRH3dBupKeNCioWYUZ4SS09Z14H+7i8ZoY=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0 h1:MaKvxE6D0KkjOg6Wd9M00iqP5PR0kUxCfiezes4JweM=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0/go.mod h1:i2h9fsTFKZorh8RdV2IcSUf/Qj98GlTkrTvUbX/s8as=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0 h1:nCYfgcSyHZXJI8J0IWE5MsCGlb2xp9fJiXyxWgmOFg4=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0/go.mod h1:ucUjca2JtSZboY8IoUqyQyuuXvwbMBVwFOm0vdQPNhA=
github.com/AzureAD/microsoft-authentication-library-for-go v1.7.2 h1:RHK7bS+HQMslb1sZpAokUt+zTVmue0hKSs2C791hhzU=
github.com/AzureAD/microsoft-authentication-library-for-go v1.7.2/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik=
github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/envoyproxy/go-control-plane v0.14.0 h1:hbG2kr4RuFj222B6+7T83thSPqLjwBIfQawTkC++2HA=
github.com/envoyproxy/go-control-plane/envoy v1.37.0 h1:u3riX6BoYRfF4Dr7dwSOroNfdSbEPe9Yyl09/B6wBrQ=
github.com/envoyproxy/go-control-plane/envoy v1.37.0/go.mod h1:DReE9MMrmecPy+YvQOAOHNYMALuowAnbjjEMkkWOi6A=
github.com/envoyproxy/protoc-gen-validate v1.3.3 h1:MVQghNeW+LZcmXe7SY1V36Z+WFMDjpqGAGacLe2T0ds=
github.com/envoyproxy/protoc-gen-validate v1.3.3/go.mod h1:TsndJ/ngyIdQRhMcVVGDDHINPLWB7C82oDArY51KfB0=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw=
github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:au07oEsX2xN0ktxqI+Sida1w446QrXBRJ0nee3SNZlA=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A=
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.3.20 h1:t/xL64VUoN69MuMRQuJETqYGOw4Z9mSRJK9epIEtwFk=
github.com/googleapis/enterprise-certificate-proxy v0.3.20/go.mod h1:L3D/IQExI6LqEjBdXcZQ1WluSgigQmSwBboFstVPM4w=
github.com/googleapis/gax-go/v2 v2.24.0 h1:myMaPYyF9MecEmvQqMqomIwn9t/4KCZN9qnwsS76wlg=
github.com/googleapis/gax-go/v2 v2.24.0/go.mod h1:IaTHBDd7NHxSCiu0vEs8pQZu4dGZrWwuSoxCnk16OFM=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/microsoft/go-mssqldb v1.11.0 h1:YbDqolEjGH9hBfvKzONTf5/dbl9RKXmizMJE93lVxNs=
github.com/microsoft/go-mssqldb v1.11.0/go.mod h1:goQLDOPlMN/l1REhnNPElMoY/yX+fUWn1+7UoFJPH9Y=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.67.0 h1:yI1/OhfEPy7J9eoa6Sj051C7n5dvpj0QX8g4sRchg04=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.67.0/go.mod h1:NoUCKYWK+3ecatC4HjkRktREheMeEtrXoQxrqYFeHSc=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 h1:OyrsyzuttWTSur2qN/Lm0m2a8yqyIjUVBZcxFPuXq2o=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0/go.mod h1:C2NGBr+kAB4bk3xtMXfZ94gqFDtg/GkI7e9zqGh5Beg=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/api v0.297.0 h1:WktxTsnnx0yZNnsR6j0q6hR21RnnK81FHTOPy/ux4OE=
google.golang.org/api v0.297.0/go.mod h1:S4m8x0M6OkQpkOzGk1y9JG2sm4fFQrMh6dxzjCTszhE=
google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d h1:C9v1o0/4quuhOAfmRXA2j+we0PqZIp8traLdeogF3Ms=
google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d/go.mod h1:Wz2wFJntZFmLGo7pLDXZ3wYk5hyc0Mb+SkHhDDXT+lU=
google.golang.org/genproto/googleapis/api v0.0.0-20260715232425-e75dac1f907d h1:QwnJwPte4XXAkhPu26LTDIahnsMSUV0kK8HkxbC+Pc4=
google.golang.org/genproto/googleapis/api v0.0.0-20260715232425-e75dac1f907d/go.mod h1:WRrQ7/7N19PypuT0fxLOL5Lq0waoiRri4FbtHDEKrGE=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU=
google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

725
internal/goldeneye/mssql/analyze.go Normal file
View File

@@ -0,0 +1,725 @@
package mssql
import (
"context"
"database/sql"
"fmt"
"os"
"regexp"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// The analyze cases are checked against a live server, which is asked
// three things about each query, none of which runs it. What a driver
// would see: sys.dm_exec_describe_first_result_set describes each result
// column — its name, its type spelled the way a declaration spells it,
// whether it can be NULL, and which table column it is read from. What
// each parameter would be: sp_describe_undeclared_parameters says what
// type the server would give each parameter the query leaves undeclared.
// And what each parameter stands in for: the estimated showplan, compiled
// with the parameters declared as those types, says which column each is
// compared with or assigned to, and a parameter with such a partner is
// described as that column, from the catalog of the case's database.
//
// Two things the describing function keeps to itself: a json or vector
// column is described by the nvarchar(max) it is sent to a driver as, so
// a column read from a table is typed from the catalog instead, and a
// type SQL Server calls timestamp is written as the rowversion the
// dialect names it.
// placeholder is one parameter of a query as sqlc numbers them: each ?
// in turn, and each @name or sqlc.arg name once, at its first appearance.
// Each appearance becomes a variable of its own, since the server
// describes an undeclared parameter only when it is used once.
type placeholder struct {
Number int
Name string
Vars []string // the variables its appearances became, without the @
}
var identRe = regexp.MustCompile(`^@([A-Za-z_][A-Za-z0-9_]*)`)
// bind rewrites the query so that every parameter is a variable used once,
// and lists the parameters in sqlc's order. The query's own @name
// references are kept, since that is how sqlc's SQL Server queries name
// their parameters; a sqlc.arg becomes one, and each ? in turn becomes
// one named after its position.
func bind(query string) (string, []placeholder) {
positional := 0
sql := endtoend.Rewrite(query, func(name, _ string) string {
if name == "" {
positional++
name = "p" + strconv.Itoa(positional)
}
return "@" + name
})
var (
phs []placeholder
numbers = map[string]int{}
out strings.Builder
)
i := 0
for i < len(sql) {
c := sql[i]
switch {
case c == '\'' || c == '"' || c == '[':
end := skipQuoted(sql, i)
out.WriteString(sql[i:end])
i = end
case strings.HasPrefix(sql[i:], "--"):
end := strings.IndexByte(sql[i:], '\n')
if end < 0 {
end = len(sql)
} else {
end += i
}
out.WriteString(sql[i:end])
i = end
case strings.HasPrefix(sql[i:], "/*"):
end := strings.Index(sql[i:], "*/")
if end < 0 {
end = len(sql)
} else {
end += i + 2
}
out.WriteString(sql[i:end])
i = end
case c == '@' && (i == 0 || !isWordByte(sql[i-1])) && identRe.MatchString(sql[i:]):
m := identRe.FindStringSubmatch(sql[i:])
name := m[1]
n, ok := numbers[name]
if !ok {
n = len(phs) + 1
numbers[name] = n
phs = append(phs, placeholder{Number: n, Name: name})
}
ph := &phs[n-1]
v := name
if k := len(ph.Vars); k > 0 {
v = fmt.Sprintf("%s__%d", name, k+1)
}
ph.Vars = append(ph.Vars, v)
out.WriteString("@" + v)
i += len(m[0])
default:
out.WriteByte(c)
i++
}
}
return out.String(), phs
}
func isWordByte(c byte) bool {
return c == '_' || c == '@' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'
}
// skipQuoted returns the index just past the quoted token starting at i: a
// string, a double-quoted identifier or a bracketed one, whose closing
// delimiter is escaped by doubling it.
func skipQuoted(s string, i int) int {
open := s[i]
close := open
if open == '[' {
close = ']'
}
j := i + 1
for j < len(s) {
switch {
case s[j] == close && j+1 < len(s) && s[j+1] == close:
j += 2
case s[j] == close:
return j + 1
default:
j++
}
}
return len(s)
}
// splitStatements splits a script into the statements it is made of, on
// the semicolons outside strings, brackets, comments and BEGIN ... END
// blocks, and on the GO lines a T-SQL script separates batches with. A
// CREATE TYPE has to be its own batch before a table can use the type, so
// a schema is loaded one statement at a time; a trigger or procedure
// body keeps its semicolons, since a BEGIN, or a CASE, opens a block that
// its END closes.
func splitStatements(src string) []string {
var stmts []string
flush := func(s string) {
if s = strings.TrimSpace(s); s != "" && !strings.EqualFold(s, "go") {
stmts = append(stmts, s)
}
}
start := 0
i := 0
depth := 0
if isGoLine(src, 0) {
for i < len(src) && src[i] != '\n' {
i++
}
start = i
}
for i < len(src) {
c := src[i]
switch {
case c == '\'' || c == '"' || c == '[':
i = skipQuoted(src, i)
case strings.HasPrefix(src[i:], "--"):
end := strings.IndexByte(src[i:], '\n')
if end < 0 {
i = len(src)
} else {
i += end
}
case strings.HasPrefix(src[i:], "/*"):
end := strings.Index(src[i:], "*/")
if end < 0 {
i = len(src)
} else {
i += end + 2
}
case isWordByte(c):
end := i
for end < len(src) && isWordByte(src[end]) {
end++
}
switch word := strings.ToLower(src[i:end]); word {
case "begin":
// BEGIN TRAN[SACTION] and BEGIN DISTRIBUTED are statements,
// not blocks.
if !nextWordIn(src, end, "tran", "transaction", "distributed") {
depth++
}
case "case":
depth++
case "end":
if depth > 0 {
depth--
}
}
i = end
case c == ';' && depth == 0:
flush(src[start:i])
start = i + 1
i++
case c == '\n' && isGoLine(src, i+1):
flush(src[start:i])
i++
for i < len(src) && src[i] != '\n' {
i++
}
start = i
depth = 0
default:
i++
}
}
flush(src[start:])
return stmts
}
// nextWordIn reports whether the word after position i, past spaces and
// comments, is one of the words.
func nextWordIn(src string, i int, words ...string) bool {
for i < len(src) {
switch {
case src[i] == ' ' || src[i] == '\t' || src[i] == '\n' || src[i] == '\r':
i++
case strings.HasPrefix(src[i:], "--"):
end := strings.IndexByte(src[i:], '\n')
if end < 0 {
return false
}
i += end
case strings.HasPrefix(src[i:], "/*"):
end := strings.Index(src[i:], "*/")
if end < 0 {
return false
}
i += end + 2
default:
end := i
for end < len(src) && isWordByte(src[end]) {
end++
}
for _, w := range words {
if strings.EqualFold(src[i:end], w) {
return true
}
}
return false
}
}
return false
}
// isGoLine reports whether the line starting at i is a GO batch separator.
func isGoLine(src string, i int) bool {
end := strings.IndexByte(src[i:], '\n')
if end < 0 {
end = len(src)
} else {
end += i
}
return strings.EqualFold(strings.TrimSpace(src[i:end]), "go")
}
// column is what the catalog says about a column of the case's database.
type column struct {
name string
typ *analysis.TypeExpr
nullable bool
}
// relation is a table or view of the case's database, by schema and name,
// in lower case.
type relation struct {
schema, name string
}
// analyzer holds the session a case is described in and the catalog of
// the case's database.
type analyzer struct {
conn *sql.Conn
catalog map[relation][]column
// userTypes are the alias types the schema created, by name in lower
// case, which the server reports beside the system type they stand on
// and the dialect reports by their own name.
userTypes map[string]bool
// canonical names the type the dialect reports a spelling as, for each
// alias types.jsonl lists: SQL Server spells numeric and timestamp in
// its catalog, which the dialect names decimal and rowversion.
canonical map[string]string
}
var dbNameRe = regexp.MustCompile(`[^A-Za-z0-9_]+`)
// Analyze loads a case's schema and fixture into a database of their own
// on the server, describes its queries there and returns what SQL Server
// reports in the JSON shape sqlc analyze prints.
func Analyze(ctx context.Context, dsn string, c endtoend.Case) ([]byte, error) {
conn, err := open(ctx, dsn)
if err != nil {
return nil, err
}
defer conn.Close()
if err := checkVersion(ctx, conn); err != nil {
return nil, err
}
schema, err := os.ReadFile(c.Schema)
if err != nil {
return nil, err
}
var fixture []byte
if c.Fixture != "" {
if fixture, err = os.ReadFile(c.Fixture); err != nil {
return nil, err
}
}
queries, err := c.Queries()
if err != nil {
return nil, err
}
db := quote("goldeneye_" + dbNameRe.ReplaceAllString(c.Name, "_"))
for _, stmt := range []string{
"USE master",
"DROP DATABASE IF EXISTS " + db,
"CREATE DATABASE " + db,
"USE " + db,
} {
if _, err := conn.ExecContext(ctx, stmt); err != nil {
return nil, err
}
}
defer conn.ExecContext(context.WithoutCancel(ctx), "USE master; DROP DATABASE IF EXISTS "+db)
for _, stmt := range splitStatements(string(schema)) {
if _, err := conn.ExecContext(ctx, stmt); err != nil {
return nil, fmt.Errorf("loading %s: %w", c.Schema, err)
}
}
for _, stmt := range splitStatements(string(fixture)) {
if _, err := conn.ExecContext(ctx, stmt); err != nil {
return nil, fmt.Errorf("loading %s: %w", c.Fixture, err)
}
}
a := &analyzer{conn: conn}
if a.canonical, err = readAliases(); err != nil {
return nil, err
}
if err := a.readCatalog(ctx); err != nil {
return nil, err
}
out := make([]analysis.Query, 0, len(queries))
for _, q := range queries {
aq, err := a.analyzeQuery(ctx, q)
if err != nil {
return nil, fmt.Errorf("%s: %w", q.Name, err)
}
out = append(out, aq)
}
return analysis.Encode(out)
}
// readAliases reads the aliases the hand-written types.jsonl gives each
// type, keyed by alias.
func readAliases() (map[string]string, error) {
dir, err := dialect.Dir(Engine)
if err != nil {
return nil, err
}
types, err := dialect.ReadTypes(dir)
if err != nil {
return nil, err
}
canonical := map[string]string{}
for _, t := range types {
for _, alias := range t.Aliases {
canonical[strings.ToLower(alias)] = strings.ToLower(t.Name)
}
}
return canonical, nil
}
// Check compares what SQL Server reports for a case with the output the
// case committed, returning a diff when they differ.
func Check(ctx context.Context, dsn string, c endtoend.Case) (string, error) {
got, err := Analyze(ctx, dsn, c)
if err != nil {
return "", err
}
return c.Compare(got)
}
// tablesQuery lists the tables and views of the current database, and the
// columns of each: the type as the schema declared it, whether that is a
// type the schema created, and the dimensions of a vector, which is all
// the describing function does not say.
const tablesQuery = `
SELECT s.name, o.name, c.name, t.name, t.is_user_defined, c.vector_dimensions
FROM sys.columns c
JOIN sys.objects o ON o.object_id = c.object_id
JOIN sys.schemas s ON s.schema_id = o.schema_id
JOIN sys.types t ON t.user_type_id = c.user_type_id
WHERE o.type IN ('U', 'V')
ORDER BY s.name, o.name, c.column_id`
// readCatalog reads the tables the schema created. The server spells each
// column's type by describing a SELECT * from the table, and the catalog
// says which columns that spelling is not the declared type of.
func (a *analyzer) readCatalog(ctx context.Context) error {
a.catalog = map[relation][]column{}
a.userTypes = map[string]bool{}
rows, err := a.conn.QueryContext(ctx, tablesQuery)
if err != nil {
return err
}
type declared struct {
name string
typeName string
userDefined bool
dimensions sql.NullInt64
}
decls := map[relation][]declared{}
var order []relation
for rows.Next() {
var (
schema, table, col string
d declared
)
if err := rows.Scan(&schema, &table, &col, &d.typeName, &d.userDefined, &d.dimensions); err != nil {
rows.Close()
return err
}
d.name = col
rel := relation{strings.ToLower(schema), strings.ToLower(table)}
if _, ok := decls[rel]; !ok {
order = append(order, rel)
}
decls[rel] = append(decls[rel], d)
if d.userDefined {
a.userTypes[strings.ToLower(d.typeName)] = true
}
}
rows.Close()
if err := rows.Err(); err != nil {
return err
}
for _, rel := range order {
described, err := a.describe(ctx, "SELECT * FROM "+quote(rel.schema)+"."+quote(rel.name))
if err != nil {
return fmt.Errorf("%s.%s: %w", rel.schema, rel.name, err)
}
byName := map[string]declared{}
for _, d := range decls[rel] {
byName[strings.ToLower(d.name)] = d
}
var cols []column
for _, dc := range described {
d, ok := byName[strings.ToLower(dc.name)]
if !ok {
return fmt.Errorf("%s.%s: the server describes a column %q the catalog does not list", rel.schema, rel.name, dc.name)
}
t := dc.typ
switch {
case d.userDefined:
t = &analysis.TypeExpr{Name: strings.ToLower(d.typeName)}
case strings.EqualFold(d.typeName, "json"):
t = &analysis.TypeExpr{Name: "json"}
case strings.EqualFold(d.typeName, "vector") && d.dimensions.Valid:
n := d.dimensions.Int64
t = &analysis.TypeExpr{Name: "vector", Args: []analysis.TypeArg{{Int: &n}}}
}
cols = append(cols, column{name: strings.ToLower(dc.name), typ: t, nullable: dc.nullable})
}
a.catalog[rel] = cols
}
return nil
}
// described is one result column as sys.dm_exec_describe_first_result_set
// reports it.
type described struct {
name string
typ *analysis.TypeExpr
nullable bool
source *columnRef // the table column it is read from, or nil
}
const describeColumnsQuery = `
SELECT column_ordinal, name, system_type_name, user_type_name, is_nullable,
source_schema, source_table, source_column, is_computed_column, error_message
FROM sys.dm_exec_describe_first_result_set(@p1, NULL, 1)
WHERE is_hidden = 0
ORDER BY column_ordinal`
// describe asks the server what the first result set of a statement
// looks like. A statement that returns no rows describes as no columns; one
// the server cannot compile describes as its error.
func (a *analyzer) describe(ctx context.Context, query string) ([]described, error) {
rows, err := a.conn.QueryContext(ctx, describeColumnsQuery, query)
if err != nil {
return nil, err
}
defer rows.Close()
var out []described
for rows.Next() {
var (
ordinal int
name, sysType, userType sql.NullString
nullable sql.NullBool
srcSchema, srcTable, srcColumn sql.NullString
computed sql.NullBool
errMsg sql.NullString
)
if err := rows.Scan(&ordinal, &name, &sysType, &userType, &nullable, &srcSchema, &srcTable, &srcColumn, &computed, &errMsg); err != nil {
return nil, err
}
if errMsg.Valid {
return nil, fmt.Errorf("cannot be described: %s", errMsg.String)
}
d := described{
name: name.String,
typ: a.typeOf(sysType.String, userType),
nullable: nullable.Bool,
}
// A computed column names the column it is computed from, as the
// one an update through the result set would write; a column is
// read from a table only when it is that column.
if srcTable.Valid && !computed.Bool {
d.source = &columnRef{schema: srcSchema.String, table: srcTable.String, column: srcColumn.String}
}
out = append(out, d)
}
return out, rows.Err()
}
// typeOf reads a type the server spelled, which is the system type unless
// the server also names a type the schema created, which the dialect
// reports by that name and no argument.
func (a *analyzer) typeOf(systemType string, userType sql.NullString) *analysis.TypeExpr {
if userType.Valid && a.userTypes[strings.ToLower(userType.String)] {
return &analysis.TypeExpr{Name: strings.ToLower(userType.String)}
}
return parseType(systemType, a.canonical)
}
// parseType reads a type spelled the way a declaration spells it —
// nvarchar(max), decimal(10,2), datetime2(7) — into an expression, named
// the way the dialect names it when the spelling is one of the aliases
// the dialect lists: numeric is decimal, timestamp is rowversion.
func parseType(s string, canonical map[string]string) *analysis.TypeExpr {
s = strings.TrimSpace(strings.ToLower(s))
name, args := s, ""
if open := strings.IndexByte(s, '('); open >= 0 && strings.HasSuffix(s, ")") {
name, args = strings.TrimSpace(s[:open]), s[open+1:len(s)-1]
}
if c, ok := canonical[name]; ok {
name = c
}
t := &analysis.TypeExpr{Name: name}
if args == "" {
return t
}
for _, arg := range strings.Split(args, ",") {
arg = strings.TrimSpace(arg)
if n, err := strconv.ParseInt(arg, 10, 64); err == nil {
t.Args = append(t.Args, analysis.TypeArg{Int: &n})
} else {
ident := arg
t.Args = append(t.Args, analysis.TypeArg{Ident: &ident})
}
}
return t
}
// withNullable copies a type with its nullability set.
func withNullable(t *analysis.TypeExpr, nullable bool) *analysis.TypeExpr {
if t == nil {
return nil
}
out := *t
out.Nullable = nullable
return &out
}
// lookup finds a column of the case's database in the catalog.
func (a *analyzer) lookup(ref columnRef) (column, bool) {
cols, ok := a.catalog[relation{strings.ToLower(ref.schema), strings.ToLower(ref.table)}]
if !ok {
return column{}, false
}
for _, col := range cols {
if strings.EqualFold(col.name, ref.column) {
return col, true
}
}
return column{}, false
}
// parameter is what sp_describe_undeclared_parameters says about one
// variable the statement leaves undeclared.
type parameter struct {
systemType string
userType sql.NullString
}
// undeclaredParameters asks the server what type it would give each
// parameter, keyed by the variable's name without the @.
func (a *analyzer) undeclaredParameters(ctx context.Context, query string) (map[string]parameter, error) {
rows, err := a.conn.QueryContext(ctx, "EXEC sp_describe_undeclared_parameters @p1", query)
if err != nil {
return nil, err
}
defer rows.Close()
cols, err := rows.Columns()
if err != nil {
return nil, err
}
index := map[string]int{}
for i, c := range cols {
index[c] = i
}
params := map[string]parameter{}
for rows.Next() {
vals := make([]any, len(cols))
ptrs := make([]any, len(cols))
for i := range vals {
ptrs[i] = &vals[i]
}
if err := rows.Scan(ptrs...); err != nil {
return nil, err
}
str := func(col string) sql.NullString {
i, ok := index[col]
if !ok || vals[i] == nil {
return sql.NullString{}
}
return sql.NullString{String: fmt.Sprint(vals[i]), Valid: true}
}
name := strings.TrimPrefix(str("name").String, "@")
params[name] = parameter{
systemType: str("suggested_system_type_name").String,
userType: str("suggested_user_type_name"),
}
}
return params, rows.Err()
}
// analyzeQuery describes one query.
func (a *analyzer) analyzeQuery(ctx context.Context, q endtoend.Query) (analysis.Query, error) {
sql, phs := bind(q.SQL)
aq := analysis.Query{
Name: q.Name,
Cmd: q.Cmd,
Columns: []analysis.Column{},
Params: []analysis.Param{},
}
described, err := a.describe(ctx, sql)
if err != nil {
return analysis.Query{}, err
}
for _, d := range described {
ac := analysis.Column{Name: strings.ToLower(d.name), Type: withNullable(d.typ, d.nullable)}
if d.source != nil {
if col, ok := a.lookup(*d.source); ok {
ac.Type = withNullable(col.typ, d.nullable)
}
ac.Table = strings.ToLower(d.source.table)
}
aq.Columns = append(aq.Columns, ac)
}
if len(phs) == 0 {
return aq, nil
}
params, err := a.undeclaredParameters(ctx, sql)
if err != nil {
return analysis.Query{}, err
}
var decl []string
for _, ph := range phs {
for _, v := range ph.Vars {
p, ok := params[v]
if !ok {
return analysis.Query{}, fmt.Errorf("the server did not describe parameter @%s", v)
}
decl = append(decl, "@"+v+" "+p.systemType)
}
}
partners, err := a.partners(ctx, "DECLARE "+strings.Join(decl, ", ")+";\n"+sql)
if err != nil {
return analysis.Query{}, err
}
for _, ph := range phs {
ac := analysis.Column{}
found := false
for _, v := range ph.Vars {
ref, ok := partners[v]
if !ok {
continue
}
col, ok := a.lookup(ref)
if !ok {
continue
}
ac = analysis.Column{Name: col.name, Type: withNullable(col.typ, col.nullable), Table: strings.ToLower(ref.table)}
found = true
break
}
if !found {
p := params[ph.Vars[0]]
ac.Type = a.typeOf(p.systemType, p.userType)
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}

153
internal/goldeneye/mssql/mssql.go Normal file
View File

@@ -0,0 +1,153 @@
// Package mssql generates the SQL Server dialect seed under
// internal/engine/mssql/dialect from a live server, and verifies the SQL
// Server analyze cases under internal/endtoend/testdata against the same
// server.
//
// SQL Server keeps no catalog of its built-in functions or operators — the
// intrinsic functions such as GETDATE and LEN are not objects — so
// types.jsonl and functions.jsonl are written by hand and are not this
// package's business. What it does describe is its catalog: relations.jsonl
// is every view of the sys and INFORMATION_SCHEMA schemas, with each view's
// columns as the server itself describes a SELECT * from it.
//
// The package also verifies the SQL Server analyze cases under
// internal/endtoend/testdata against the same server: each case's schema
// and fixture are loaded into a database of their own and its queries
// described there, without being run. Result columns come from
// sys.dm_exec_describe_first_result_set, which says what each column is
// called, what type it has, whether it can be NULL and which table column
// it is read from; parameters from sp_describe_undeclared_parameters, which
// says what type the server would give each, and from the estimated
// showplan, which says which column each is compared with or assigned to.
// The answer is printed in the JSON shape sqlc analyze prints and compared
// with the case's committed stdout.json byte for byte.
//
// The server is named by MSSQL_SERVER_URI, in any form the go-mssqldb
// driver accepts, such as
// sqlserver://sa:password@127.0.0.1:1433?encrypt=disable, and has to be
// the major release in Major, since each release adds to the catalog views.
package mssql
import (
"context"
"database/sql"
"errors"
"fmt"
"os"
"strconv"
"strings"
_ "github.com/microsoft/go-mssqldb"
"github.com/microsoft/go-mssqldb/msdsn"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name sqlc knows the dialect by, and the name of the analyze
// case directories under internal/endtoend/testdata.
const Engine = "mssql"
// Major is the SQL Server major release the dialect is generated from, as
// SERVERPROPERTY('ProductMajorVersion') reports it: 17 is SQL Server 2025.
// Bumping it is a deliberate change: every release adds views and columns
// to the catalog, so regenerate and review the dialect after changing it.
const Major = 17
// Locate returns the server to generate from, named by MSSQL_SERVER_URI.
func Locate() (string, error) {
dsn := os.Getenv("MSSQL_SERVER_URI")
if dsn == "" {
return "", errors.New("MSSQL_SERVER_URI is not set")
}
if _, err := msdsn.Parse(dsn); err != nil {
return "", fmt.Errorf("MSSQL_SERVER_URI: %w", err)
}
return dsn, nil
}
// open connects to the server and holds one session, since a case's
// database is the session's current database and the showplan is a
// session setting.
func open(ctx context.Context, dsn string) (*sql.Conn, error) {
db, err := sql.Open("sqlserver", dsn)
if err != nil {
return nil, err
}
conn, err := db.Conn(ctx)
if err != nil {
db.Close()
return nil, err
}
return conn, nil
}
// Version reports the release a server is.
func Version(ctx context.Context, dsn string) (string, error) {
conn, err := open(ctx, dsn)
if err != nil {
return "", err
}
defer conn.Close()
var v string
if err := conn.QueryRowContext(ctx, "SELECT @@VERSION").Scan(&v); err != nil {
return "", err
}
// The first line names the product and build; the rest is the copyright
// and the platform.
v, _, _ = strings.Cut(v, "\n")
return strings.TrimSpace(v), nil
}
// checkVersion refuses a server of another major release than the dialect
// is generated from, whose catalog would differ from the committed one
// without anything being wrong.
func checkVersion(ctx context.Context, conn *sql.Conn) error {
var v string
if err := conn.QueryRowContext(ctx, "SELECT CAST(SERVERPROPERTY('ProductMajorVersion') AS nvarchar(10))").Scan(&v); err != nil {
return err
}
major, err := strconv.Atoi(v)
if err != nil {
return fmt.Errorf("ProductMajorVersion %q: %w", v, err)
}
if major != Major {
return fmt.Errorf("the dialect is generated from SQL Server major release %d, but the server is release %d", Major, major)
}
return nil
}
// Generate reads the dialect from the server: relations.jsonl, the views of
// sys and INFORMATION_SCHEMA. They are read from a database of their own,
// since the views a query sees are the ones a user database has, and
// master lists internal views of its own that no query can name.
func Generate(ctx context.Context, dsn string) (dialect.Files, error) {
conn, err := open(ctx, dsn)
if err != nil {
return nil, err
}
defer conn.Close()
if err := checkVersion(ctx, conn); err != nil {
return nil, err
}
db := quote("goldeneye_dialect")
for _, stmt := range []string{
"USE master",
"DROP DATABASE IF EXISTS " + db,
"CREATE DATABASE " + db,
"USE " + db,
} {
if _, err := conn.ExecContext(ctx, stmt); err != nil {
return nil, err
}
}
defer conn.ExecContext(context.WithoutCancel(ctx), "USE master; DROP DATABASE IF EXISTS "+db)
relations, err := readRelations(ctx, conn)
if err != nil {
return nil, err
}
blob, err := dialect.JSONL(relations)
if err != nil {
return nil, err
}
return dialect.Files{dialect.RelationsFile: blob}, nil
}

66
internal/goldeneye/mssql/mssql_test.go Normal file
View File

@@ -0,0 +1,66 @@
package mssql
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// TestDialect verifies the committed SQL Server dialect against what the
// server reports. It skips unless MSSQL_SERVER_URI names a server.
func TestDialect(t *testing.T) {
dsn, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, dsn)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, dsn)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Engine)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}
// TestAnalyzeCases verifies every SQL Server analyze case under
// internal/endtoend/testdata against what the server reports. It skips
// unless MSSQL_SERVER_URI names a server.
func TestAnalyzeCases(t *testing.T) {
dsn, err := Locate()
if err != nil {
t.Skip(err)
}
cases, err := endtoend.Cases(Engine)
if err != nil {
t.Fatal(err)
}
if len(cases) == 0 {
t.Fatal("no mssql analyze cases found")
}
for _, c := range cases {
t.Run(c.Name, func(t *testing.T) {
diff, err := Check(context.Background(), dsn, c)
if err != nil {
t.Fatal(err)
}
if diff != "" {
t.Errorf("%s does not match what SQL Server reports (-committed +mssql):\n%s", c.Output, diff)
}
})
}
}

243
internal/goldeneye/mssql/plan.go Normal file
View File

@@ -0,0 +1,243 @@
package mssql
import (
"context"
"encoding/xml"
"fmt"
"strings"
)
// The estimated showplan is the one thing that says what a parameter is
// compared with or assigned to. It is compiled, not run, with SET
// SHOWPLAN_XML ON, and prints the plan as XML in which every column is a
// ColumnReference naming its schema, table and column, and every variable
// one naming only @variable. A parameter's partner is the column on the
// other side of the Compare it is an operand of, the column an Assign
// sets to it, or the column of a seek whose range expression it is. An
// expression the plan computes once and refers to by name, Expr1002, is
// followed to its DefinedValue. A parameter the query casts, CAST(@p AS
// T), has no partner: the cast says what it is, as it does to sqlc, and
// the plan shows it as a Convert the query asked for rather than one the
// server added.
// columnRef names a table column the plan refers to.
type columnRef struct {
schema, table, column string
}
// node is one element of the showplan XML.
type node struct {
XMLName xml.Name
Attrs []xml.Attr `xml:",any,attr"`
Children []node `xml:",any"`
}
func (n *node) attr(name string) string {
for _, a := range n.Attrs {
if a.Name.Local == name {
return a.Value
}
}
return ""
}
// unbracket strips the [brackets] the plan quotes names with.
func unbracket(s string) string {
if strings.HasPrefix(s, "[") && strings.HasSuffix(s, "]") {
return strings.ReplaceAll(s[1:len(s)-1], "]]", "]")
}
return s
}
// plan is a parsed showplan with the expressions it defines by name.
type plan struct {
root node
defined map[string]*node // Expr1002 -> the ScalarOperator defining it
}
// partners compiles a batch and returns the column each variable is
// compared with or assigned to, keyed by the variable's name without the
// @. A variable with no such column is absent.
func (a *analyzer) partners(ctx context.Context, batch string) (map[string]columnRef, error) {
if _, err := a.conn.ExecContext(ctx, "SET SHOWPLAN_XML ON"); err != nil {
return nil, err
}
defer a.conn.ExecContext(context.WithoutCancel(ctx), "SET SHOWPLAN_XML OFF")
rows, err := a.conn.QueryContext(ctx, batch)
if err != nil {
return nil, fmt.Errorf("showplan: %w", err)
}
defer rows.Close()
var blob string
for rows.Next() {
var s string
if err := rows.Scan(&s); err != nil {
return nil, err
}
blob += s
}
if err := rows.Err(); err != nil {
return nil, err
}
p := &plan{defined: map[string]*node{}}
if err := xml.Unmarshal([]byte(blob), &p.root); err != nil {
return nil, fmt.Errorf("showplan: %w", err)
}
p.collectDefined(&p.root)
partners := map[string]columnRef{}
p.walk(&p.root, partners)
return partners, nil
}
// collectDefined records every DefinedValue that names an expression.
func (p *plan) collectDefined(n *node) {
if n.XMLName.Local == "DefinedValue" && len(n.Children) >= 2 {
ref := &n.Children[0]
if ref.XMLName.Local == "ColumnReference" && ref.attr("Table") == "" && !strings.HasPrefix(ref.attr("Column"), "@") {
p.defined[ref.attr("Column")] = &n.Children[1]
}
}
for i := range n.Children {
p.collectDefined(&n.Children[i])
}
}
// walk finds the partners in a subtree, keeping the first found for each
// variable.
func (p *plan) walk(n *node, partners map[string]columnRef) {
set := func(vars []string, ref columnRef) {
for _, v := range vars {
if _, ok := partners[v]; !ok {
partners[v] = ref
}
}
}
switch n.XMLName.Local {
case "Compare":
var sides []*node
for i := range n.Children {
if n.Children[i].XMLName.Local == "ScalarOperator" {
sides = append(sides, &n.Children[i])
}
}
if len(sides) == 2 {
for i, side := range sides {
vars := p.variables(side, nil)
if len(vars) == 0 {
continue
}
if cols := p.columns(sides[1-i], nil); len(cols) > 0 {
set(vars, cols[0])
}
}
}
case "Assign":
var target *columnRef
for i := range n.Children {
c := &n.Children[i]
switch c.XMLName.Local {
case "ColumnReference":
if ref, ok := refOf(c); ok && target == nil {
target = &ref
}
case "ScalarOperator":
if target != nil {
set(p.variables(c, nil), *target)
}
}
}
case "Prefix", "StartRange", "EndRange":
var cols []columnRef
var exprs []*node
for i := range n.Children {
c := &n.Children[i]
switch c.XMLName.Local {
case "RangeColumns":
for j := range c.Children {
if ref, ok := refOf(&c.Children[j]); ok {
cols = append(cols, ref)
}
}
case "RangeExpressions":
for j := range c.Children {
exprs = append(exprs, &c.Children[j])
}
}
}
for i, e := range exprs {
if i < len(cols) {
set(p.variables(e, nil), cols[i])
}
}
}
for i := range n.Children {
p.walk(&n.Children[i], partners)
}
}
// refOf reads a ColumnReference that names a table column.
func refOf(n *node) (columnRef, bool) {
if n.XMLName.Local != "ColumnReference" || n.attr("Table") == "" {
return columnRef{}, false
}
return columnRef{
schema: unbracket(n.attr("Schema")),
table: unbracket(n.attr("Table")),
column: n.attr("Column"),
}, true
}
// variables lists the variables a subtree refers to, following named
// expressions and leaving out the ones under a cast the query wrote.
func (p *plan) variables(n *node, seen map[string]bool) []string {
var out []string
if n.XMLName.Local == "Convert" && n.attr("Implicit") == "0" {
return nil
}
if n.XMLName.Local == "ColumnReference" {
col := n.attr("Column")
switch {
case strings.HasPrefix(col, "@"):
return []string{col[1:]}
case n.attr("Table") == "":
if def, ok := p.defined[col]; ok && !seen[col] {
if seen == nil {
seen = map[string]bool{}
}
seen[col] = true
vars := p.variables(def, seen)
delete(seen, col)
return vars
}
}
}
for i := range n.Children {
out = append(out, p.variables(&n.Children[i], seen)...)
}
return out
}
// columns lists the table columns a subtree refers to, following named
// expressions.
func (p *plan) columns(n *node, seen map[string]bool) []columnRef {
var out []columnRef
if n.XMLName.Local == "ColumnReference" {
if ref, ok := refOf(n); ok {
return []columnRef{ref}
}
col := n.attr("Column")
if def, ok := p.defined[col]; ok && !seen[col] {
if seen == nil {
seen = map[string]bool{}
}
seen[col] = true
cols := p.columns(def, seen)
delete(seen, col)
return cols
}
}
for i := range n.Children {
out = append(out, p.columns(&n.Children[i], seen)...)
}
return out
}

123
internal/goldeneye/mssql/relations.go Normal file
View File

@@ -0,0 +1,123 @@
package mssql
import (
"context"
"database/sql"
"fmt"
"sort"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// systemSchemas are the schemas whose relations are generated: the catalog
// views, which are what a query of the server's catalog reads. Both hold
// nothing but views, so every relation a dialect seeds from them is one
// the analysis core never hands codegen as a model.
var systemSchemas = []string{
"sys",
"INFORMATION_SCHEMA",
}
// viewQuery lists a schema's views. They are sorted in Go, by the bytes
// of their lower-cased names, so that the output does not depend on the
// server's collation.
const viewQuery = `
SELECT o.name
FROM sys.all_objects o
JOIN sys.schemas s ON s.schema_id = o.schema_id
WHERE o.type = 'V' AND s.name = @p1`
// describeQuery asks the server to describe a SELECT * from a view: each
// column's name, its type spelled the way a declaration spells it —
// nvarchar(128), decimal(10,2), varbinary(max) — and whether it can be
// NULL. A view the server cannot describe reports an error message on a
// row of its own rather than failing the query.
const describeQuery = `
SELECT column_ordinal, name, system_type_name, is_nullable, error_message
FROM sys.dm_exec_describe_first_result_set(@p1, NULL, 0)
ORDER BY column_ordinal`
// readRelations reads the views of the system schemas. Their names are
// written in lower case: SQL Server matches them in any case under its
// default collations, INFORMATION_SCHEMA spells its own in upper case, and
// sqlc's SQL Server parser lowercases every identifier, so lower case is
// how a query reaches them. The type of each column is spelled the way the
// server describes it to a driver, which is the way a declaration spells
// it, numeric and timestamp included, since the seed reads the aliases
// types.jsonl lists.
func readRelations(ctx context.Context, conn *sql.Conn) ([]dialect.Relation, error) {
var relations []dialect.Relation
for _, schema := range systemSchemas {
names, err := views(ctx, conn, schema)
if err != nil {
return nil, fmt.Errorf("%s: %w", schema, err)
}
for _, name := range names {
columns, err := describe(ctx, conn, schema, name)
if err != nil {
return nil, fmt.Errorf("%s.%s: %w", schema, name, err)
}
relations = append(relations, dialect.Relation{
Schema: strings.ToLower(schema),
Name: strings.ToLower(name),
Kind: "v",
Columns: columns,
})
}
}
return relations, nil
}
func views(ctx context.Context, conn *sql.Conn, schema string) ([]string, error) {
rows, err := conn.QueryContext(ctx, viewQuery, schema)
if err != nil {
return nil, err
}
defer rows.Close()
var names []string
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
return nil, err
}
names = append(names, name)
}
sort.Slice(names, func(i, j int) bool { return strings.ToLower(names[i]) < strings.ToLower(names[j]) })
return names, rows.Err()
}
func describe(ctx context.Context, conn *sql.Conn, schema, name string) ([]dialect.Column, error) {
rows, err := conn.QueryContext(ctx, describeQuery, "SELECT * FROM "+quote(schema)+"."+quote(name))
if err != nil {
return nil, err
}
defer rows.Close()
var columns []dialect.Column
for rows.Next() {
var (
ordinal int
colName sql.NullString
typeName sql.NullString
nullable sql.NullBool
errMsg sql.NullString
)
if err := rows.Scan(&ordinal, &colName, &typeName, &nullable, &errMsg); err != nil {
return nil, err
}
if errMsg.Valid {
return nil, fmt.Errorf("cannot be described: %s", errMsg.String)
}
columns = append(columns, dialect.Column{
Name: strings.ToLower(colName.String),
Type: typeName.String,
NotNull: !nullable.Bool,
})
}
return columns, rows.Err()
}
// quote brackets an identifier the way T-SQL does.
func quote(name string) string {
return "[" + strings.ReplaceAll(name, "]", "]]") + "]"
}

787
internal/goldeneye/mysql/analyze.go Normal file
View File

@@ -0,0 +1,787 @@
package mysql
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"fmt"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"os"
"regexp"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// The analyze cases are checked against a live server, which is asked
// three things about each query. What a driver sees: the query is run,
// with every parameter a user variable set to NULL, and each result
// column's name, type and nullability are read from the result set's
// metadata, the way go-sql-driver reports them. What the resolver made of
// it: the optimizer trace prints each query block back after name
// resolution, with every column qualified, every alias kept and every
// SELECT * expanded, which says which table each result column is read
// from and what each parameter is compared with or assigned to. And for a
// statement the trace does not expand — an INSERT ... VALUES, a
// single-table UPDATE or DELETE — the note EXPLAIN leaves, which prints
// the statement the same way. Views and derived tables are kept as the
// query wrote them rather than merged, so that a column read through one
// is reported as its column. MySQL itself reports nothing about a
// parameter but its position, so a parameter is described by its partner:
// a column's type and nullability come from information_schema, a column
// of a derived table from what its block projects, and an expression's
// from running it, over the tables it reads, as a query of its own.
// placeholder is one parameter of a query as sqlc numbers them: each ? in
// turn, and each sqlc.arg name once, at its first appearance. Each becomes
// a user variable, which MySQL prints back by name, except in a LIMIT or
// OFFSET, where MySQL allows nothing but a number.
type placeholder struct {
Number int
Name string
Limit bool
Sentinel string // the variable's name, or the number
}
// limitBase is added to a LIMIT parameter's number, so that the count is
// unlike any a query would write.
const limitBase = 4000000000
func bind(sql string) (string, []placeholder) {
var phs []placeholder
numbers := map[string]int{}
out := endtoend.Rewrite(sql, func(name, lastWord string) string {
n, ok := numbers[name]
if name == "" || !ok {
n = len(phs) + 1
ph := placeholder{Number: n, Name: name}
switch strings.ToLower(lastWord) {
case "limit", "offset":
ph.Limit = true
ph.Sentinel = strconv.Itoa(limitBase + n)
default:
ph.Sentinel = "goldeneye_" + strconv.Itoa(n)
}
phs = append(phs, ph)
if name != "" {
numbers[name] = n
}
}
ph := phs[n-1]
if ph.Limit {
return ph.Sentinel
}
return "@" + ph.Sentinel
})
return out, phs
}
// column is what information_schema says about a column: its type as
// COLUMN_TYPE spells it, read into an expression.
type column struct {
name string
typ *analysis.TypeExpr
nullable bool
}
// typeOfColumn is a column's type as it was declared, from COLUMN_TYPE, which
// carries the arguments and the unsigned that DATA_TYPE does not: "decimal(10,2)
// unsigned" is the family "decimal unsigned" applied to 10 and 2, and
// "enum('a','b')" is enum applied to its members. A trailing word such as
// zerofill is part of the family too.
func typeOfColumn(columnType string) *analysis.TypeExpr {
s := strings.TrimSpace(columnType)
open := strings.IndexByte(s, '(')
if open < 0 {
return &analysis.TypeExpr{Name: strings.ToLower(s)}
}
close := strings.LastIndexByte(s, ')')
if close < open {
return &analysis.TypeExpr{Name: strings.ToLower(s)}
}
// The family is spelled in lower case; an enum's members keep theirs,
// since they are values.
name := strings.ToLower(strings.TrimSpace(s[:open]))
if rest := strings.ToLower(strings.TrimSpace(s[close+1:])); rest != "" {
name += " " + rest
}
t := &analysis.TypeExpr{Name: name}
for _, a := range splitArgs(s[open+1 : close]) {
a = strings.TrimSpace(a)
if strings.HasPrefix(a, "'") && strings.HasSuffix(a, "'") && len(a) >= 2 {
v := unquoteMember(a[1 : len(a)-1])
t.Args = append(t.Args, analysis.TypeArg{String: &v})
continue
}
if n, err := strconv.ParseInt(a, 10, 64); err == nil {
t.Args = append(t.Args, analysis.TypeArg{Int: &n})
}
}
return t
}
// splitArgs splits a type's argument list on the commas outside quotes. A
// backslash inside a quoted member escapes the character after it.
func splitArgs(s string) []string {
var out []string
start, quoted := 0, false
for i := 0; i < len(s); i++ {
switch {
case quoted && s[i] == '\\':
i++
case s[i] == '\'':
quoted = !quoted
case s[i] == ',' && !quoted:
out = append(out, s[start:i])
start = i + 1
}
}
return append(out, s[start:])
}
// unquoteMember reads the body of a quoted enum or set member as
// information_schema spells it: a quote is doubled and a backslash escapes
// the character after it.
func unquoteMember(s string) string {
var out strings.Builder
for i := 0; i < len(s); i++ {
switch {
case s[i] == '\\' && i+1 < len(s):
i++
out.WriteByte(s[i])
case s[i] == '\'' && i+1 < len(s) && s[i+1] == '\'':
i++
out.WriteByte('\'')
default:
out.WriteByte(s[i])
}
}
return out.String()
}
// withNullable copies a type with its nullability set.
func withNullable(t *analysis.TypeExpr, nullable bool) *analysis.TypeExpr {
out := *t
out.Nullable = nullable
return &out
}
// relation is a table the catalog knows, by schema and name.
type relation struct {
schema, name string
}
// analyzer holds the session a case runs in and the catalog: the case's
// own schema, and information_schema, whose names are matched and
// reported in lower case, since MySQL matches them in any case and the
// dialect spells them that way.
type analyzer struct {
conn *sql.Conn
db string // the case's schema
catalog map[relation][]column // columns in order
}
// table returns the catalog's name for a table of a schema, or "" when
// the catalog has no such table.
func (a *analyzer) table(schema, name string) (relation, bool) {
if schema == "" {
schema = a.db
}
if strings.EqualFold(schema, "information_schema") {
schema, name = "information_schema", strings.ToLower(name)
}
rel := relation{schema, name}
_, ok := a.catalog[rel]
return rel, ok
}
var dbNameRe = regexp.MustCompile(`[^A-Za-z0-9_]+`)
// Analyze loads a case's schema and fixture into a database of their own
// on the server, runs its queries there and returns what MySQL reports in
// the JSON shape sqlc analyze prints.
func Analyze(ctx context.Context, dsn string, c endtoend.Case) ([]byte, error) {
conn, err := open(ctx, dsn)
if err != nil {
return nil, err
}
defer conn.Close()
schema, err := os.ReadFile(c.Schema)
if err != nil {
return nil, err
}
var fixture []byte
if c.Fixture != "" {
if fixture, err = os.ReadFile(c.Fixture); err != nil {
return nil, err
}
}
queries, err := c.Queries()
if err != nil {
return nil, err
}
name := "goldeneye_" + dbNameRe.ReplaceAllString(c.Name, "_")
db := quote(name)
for _, stmt := range []string{
"DROP DATABASE IF EXISTS " + db,
"CREATE DATABASE " + db,
"USE " + db,
"SET SESSION optimizer_trace = 'enabled=on'",
"SET SESSION optimizer_trace_max_mem_size = 67108864",
// A view or derived table is kept as the query wrote it rather
// than merged into the block that reads it, so that a column read
// from one is reported as its column, and an information_schema
// view is not resolved away into the data dictionary tables
// behind it.
"SET SESSION optimizer_switch = 'derived_merge=off,derived_condition_pushdown=off'",
} {
if _, err := conn.ExecContext(ctx, stmt); err != nil {
return nil, err
}
}
defer conn.ExecContext(context.WithoutCancel(ctx), "DROP DATABASE IF EXISTS "+db)
if _, err := conn.ExecContext(ctx, string(schema)); err != nil {
return nil, fmt.Errorf("loading %s: %w", c.Schema, err)
}
if strings.TrimSpace(string(fixture)) != "" {
if _, err := conn.ExecContext(ctx, string(fixture)); err != nil {
return nil, fmt.Errorf("loading %s: %w", c.Fixture, err)
}
}
a := &analyzer{conn: conn, db: name}
if a.catalog, err = readCatalog(ctx, conn, name); err != nil {
return nil, err
}
out := make([]analysis.Query, 0, len(queries))
for _, q := range queries {
aq, err := a.analyzeQuery(ctx, q)
if err != nil {
return nil, fmt.Errorf("%s: %w", q.Name, err)
}
out = append(out, aq)
}
return analysis.Encode(out)
}
// Check compares what MySQL reports for a case with the output the case
// committed, returning a diff when they differ.
// Check compares a case's committed output with what MySQL reports. A
// column read from a table is compared in full, since information_schema
// spells its whole type; an expression's type comes from the wire, which
// carries the family and nothing of its arguments, so an expression, and
// a parameter typed by one, is compared by family alone.
func Check(ctx context.Context, dsn string, c endtoend.Case) (string, error) {
got, err := Analyze(ctx, dsn, c)
if err != nil {
return "", err
}
want, err := os.ReadFile(c.Output)
if err != nil {
return "", err
}
var queries []analysis.Query
if err := json.Unmarshal(want, &queries); err != nil {
return "", fmt.Errorf("%s: %w", c.Output, err)
}
for i := range queries {
for j := range queries[i].Columns {
familyOnly(&queries[i].Columns[j])
}
for j := range queries[i].Params {
familyOnly(&queries[i].Params[j].Column)
}
}
want, err = analysis.Encode(queries)
if err != nil {
return "", err
}
if bytes.Equal(want, got) {
return "", nil
}
return dialect.Diff(string(want), string(got)), nil
}
// familyOnly drops the arguments of a column's type when the column is not
// read from a table, which is as much as the wire says about it.
func familyOnly(col *analysis.Column) {
if col.Table == "" && col.Type != nil {
col.Type.Args = nil
}
}
const catalogQuery = `
SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, DATA_TYPE, COLUMN_TYPE, IS_NULLABLE
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA IN (?, 'information_schema')
ORDER BY TABLE_SCHEMA, TABLE_NAME, ORDINAL_POSITION`
func readCatalog(ctx context.Context, conn *sql.Conn, db string) (map[relation][]column, error) {
rows, err := conn.QueryContext(ctx, catalogQuery, db)
if err != nil {
return nil, err
}
defer rows.Close()
catalog := map[relation][]column{}
for rows.Next() {
var schema, table, name, dataType, columnType, nullable string
if err := rows.Scan(&schema, &table, &name, &dataType, &columnType, &nullable); err != nil {
return nil, err
}
if schema == "information_schema" {
table, name = strings.ToLower(table), strings.ToLower(name)
}
rel := relation{schema, table}
catalog[rel] = append(catalog[rel], column{
name: name,
typ: typeOfColumn(columnType),
nullable: nullable == "YES",
})
}
return catalog, rows.Err()
}
// statement is everything the server said about one query.
type statement struct {
a *analyzer
trace trace
note *text // the statement as EXPLAIN printed it, nil without one
aliases map[string]tableRef
results []*sql.ColumnType // of the executed query, nil for one returning no rows
target *relation // the table an INSERT ... VALUES writes
targets []string // its columns, in row order
}
func (a *analyzer) analyzeQuery(ctx context.Context, q endtoend.Query) (analysis.Query, error) {
sql, phs := bind(q.SQL)
var sets []string
for _, ph := range phs {
if !ph.Limit {
sets = append(sets, "@"+ph.Sentinel+" = NULL")
}
}
if len(sets) > 0 {
if _, err := a.conn.ExecContext(ctx, "SET "+strings.Join(sets, ", ")); err != nil {
return analysis.Query{}, err
}
}
// EXPLAIN resolves and optimises the statement without running it,
// leaving its note and its trace behind. Only the traditional format
// leaves the note, and since MySQL 26 the default format is the tree.
if err := drain(a.conn.QueryContext(ctx, "EXPLAIN FORMAT=TRADITIONAL "+sql)); err != nil {
return analysis.Query{}, err
}
note, err := readNote(ctx, a.conn)
if err != nil {
return analysis.Query{}, err
}
tr, err := readTrace(ctx, a.conn)
if err != nil {
return analysis.Query{}, err
}
s := &statement{a: a, trace: tr, aliases: map[string]tableRef{}}
if note != "" {
t := tokenize(note)
s.note = &t
t.aliases(s.aliases)
s.target, s.targets = s.insertTargets(t)
}
for _, t := range tr.blocks {
t.aliases(s.aliases)
}
aq := analysis.Query{
Name: q.Name,
Cmd: q.Cmd,
Columns: []analysis.Column{},
Params: []analysis.Param{},
}
if returnsRows(sql) {
rows, err := a.conn.QueryContext(ctx, sql)
if err != nil {
return analysis.Query{}, err
}
s.results, err = rows.ColumnTypes()
rows.Close()
if err != nil {
return analysis.Query{}, err
}
// Names, types and nullability are the result set's. The top
// block's SELECT list adds which table each column is read from,
// when the two line up. MySQL sends every size of TEXT and BLOB
// as the one wire type and tells them apart by the length beside
// it, which the driver keeps to itself, so a column read from a
// table, directly or through a derived table, is spelled the way
// the table declares it.
var items []item
if top, ok := tr.blocks[1]; ok {
items = top.selectItems()
}
for i, ct := range s.results {
ac := analysis.Column{Name: ct.Name(), Type: typeOf(ct)}
if len(items) == len(s.results) {
top := tr.blocks[1]
if it := items[i]; it.end-it.start == 1 && top.at(it.start).kind == tkIdent {
tok := top.at(it.start)
if rel, ok := s.resolve(tok.parts); ok {
ac.Table = rel.name
}
if col, ok := s.origin(top, tok, 0); ok {
ac.Type = withNullable(col.typ, ac.Type.Nullable)
}
}
}
aq.Columns = append(aq.Columns, ac)
}
}
for _, ph := range phs {
ac := analysis.Column{}
if t, i, ok := s.locate(ph); ok {
ac = s.describe(ctx, t, t.partner(i), 0)
}
if ph.Name != "" {
ac.Name = ph.Name
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}
// drain reads a result set to the end and closes it.
func drain(rows *sql.Rows, err error) error {
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
}
return rows.Err()
}
// readNote returns the statement EXPLAIN printed back, from the note it
// leaves among the warnings, or "" when it left none.
func readNote(ctx context.Context, conn *sql.Conn) (string, error) {
rows, err := conn.QueryContext(ctx, "SHOW WARNINGS")
if err != nil {
return "", err
}
defer rows.Close()
note := ""
for rows.Next() {
var level, message string
var code int
if err := rows.Scan(&level, &code, &message); err != nil {
return "", err
}
if code == 1003 {
note = message
}
}
return note, rows.Err()
}
// readTrace returns the optimizer trace of the last statement.
func readTrace(ctx context.Context, conn *sql.Conn) (trace, error) {
rows, err := conn.QueryContext(ctx, "SELECT TRACE, MISSING_BYTES_BEYOND_MAX_MEM_SIZE FROM information_schema.OPTIMIZER_TRACE")
if err != nil {
return trace{}, err
}
defer rows.Close()
blob := ""
for rows.Next() {
var missing int64
if err := rows.Scan(&blob, &missing); err != nil {
return trace{}, err
}
if missing > 0 {
return trace{}, fmt.Errorf("the optimizer trace was truncated by %d bytes", missing)
}
}
if err := rows.Err(); err != nil {
return trace{}, err
}
return parseTrace(blob)
}
// returnsRows reports whether a statement produces a result set.
func returnsRows(sql string) bool {
head := strings.ToLower(strings.TrimSpace(sql))
if strings.HasPrefix(head, "(") {
return true
}
for _, kw := range []string{"select", "with", "show", "describe", "desc", "explain", "table", "values"} {
if strings.HasPrefix(head, kw) && (len(head) == len(kw) || !isWordByte(head[len(kw)])) {
return true
}
}
return false
}
// typeOf is a result column's type as the driver reports it, spelled the
// way a column declaration does: "bigint unsigned" rather than the
// driver's "UNSIGNED BIGINT".
func typeOf(ct *sql.ColumnType) *analysis.TypeExpr {
name := strings.ToLower(ct.DatabaseTypeName())
if rest, ok := strings.CutPrefix(name, "unsigned "); ok {
name = rest + " unsigned"
}
nullable, _ := ct.Nullable()
return &analysis.TypeExpr{Name: name, Nullable: nullable}
}
// resolve returns the catalog table a qualified column name is read from:
// the qualifier is an alias, or a table of the case's schema, or a table
// of the schema the name spells out.
func (s *statement) resolve(parts []string) (relation, bool) {
if len(parts) < 2 {
return relation{}, false
}
qualifier := parts[len(parts)-2]
if ref, ok := s.aliases[qualifier]; ok {
return s.a.table(ref.schema, ref.name)
}
schema := ""
if len(parts) >= 3 {
schema = parts[len(parts)-3]
}
return s.a.table(schema, qualifier)
}
// locate finds the text a placeholder was printed in and where: the
// innermost query block whose expanded_query has it, or else the note.
func (s *statement) locate(ph placeholder) (text, int, bool) {
find := func(t text) int {
if ph.Limit {
return t.findNumber(ph.Sentinel)
}
return t.findVar(ph.Sentinel)
}
for _, sel := range s.trace.order() {
t := s.trace.blocks[sel]
if i := find(t); i >= 0 {
return t, i, true
}
}
if s.note != nil {
if i := find(*s.note); i >= 0 {
return *s.note, i, true
}
}
return text{}, -1, false
}
// describe turns a parameter's partner into the column description sqlc
// prints for the parameter.
func (s *statement) describe(ctx context.Context, t text, p partner, depth int) analysis.Column {
switch p.kind {
case partnerOperand:
return s.operand(ctx, t, p.start, p.end, depth)
case partnerProjection:
if s.results != nil {
if top, ok := s.trace.blocks[1]; ok && top.src == t.src && p.index < len(s.results) {
ct := s.results[p.index]
return analysis.Column{Name: ct.Name(), Type: typeOf(ct)}
}
}
if items := t.selectItems(); p.index < len(items) {
return analysis.Column{Name: items[p.index].alias}
}
case partnerInsert:
if s.target != nil && len(s.targets) > 0 {
return s.column(*s.target, s.targets[p.index%len(s.targets)])
}
case partnerLimit:
// A LIMIT or OFFSET count is an unsigned 64-bit integer to MySQL.
return analysis.Column{Type: &analysis.TypeExpr{Name: "bigint unsigned"}}
}
return analysis.Column{}
}
// operand describes an operand: a column by what the schema says of it, a
// call by its name and what it returns, anything else by its type.
func (s *statement) operand(ctx context.Context, t text, start, end, depth int) analysis.Column {
tok := t.at(start)
if end-start == 1 && tok.kind == tkIdent {
return s.columnRef(ctx, t, tok, depth)
}
if tok.kind == tkVar {
return analysis.Column{}
}
ac := analysis.Column{}
if tok.kind == tkWord && t.at(start+1).kind == tkLParen && t.isCall(start) && !strings.HasPrefix(tok.text, "<") {
ac.Name = tok.text
}
ac.Type = s.eval(ctx, t, start, end)
return ac
}
// columnRef describes a column reference: a column of the schema, a column
// of a derived table by the expression its block projects, or a SELECT list
// alias by the expression it names.
func (s *statement) columnRef(ctx context.Context, t text, tok token, depth int) analysis.Column {
parts := tok.parts
if depth > 8 {
return analysis.Column{Name: parts[len(parts)-1]}
}
if len(parts) == 1 {
for _, it := range t.selectItems() {
if it.alias == parts[0] {
ac := s.operand(ctx, t, it.start, it.end, depth+1)
ac.Name = parts[0]
return ac
}
}
return analysis.Column{Name: parts[0]}
}
qualifier, name := parts[len(parts)-2], parts[len(parts)-1]
if rel, ok := s.resolve(parts); ok {
return s.column(rel, name)
}
if sel, ok := s.trace.derived[qualifier]; ok {
if block, ok := s.trace.blocks[sel]; ok {
for _, it := range block.selectItems() {
if it.alias == name {
ac := s.operand(ctx, block, it.start, it.end, depth+1)
ac.Name = name
return ac
}
}
}
}
return analysis.Column{Name: name}
}
// origin follows a column reference to the catalog column it reads, through
// any derived table or CTE whose block projects that column as it is.
func (s *statement) origin(t text, tok token, depth int) (column, bool) {
parts := tok.parts
if depth > 8 || len(parts) < 2 {
return column{}, false
}
qualifier, name := parts[len(parts)-2], parts[len(parts)-1]
if rel, ok := s.resolve(parts); ok {
if rel.schema == "information_schema" {
name = strings.ToLower(name)
}
for _, col := range s.a.catalog[rel] {
if col.name == name {
return col, true
}
}
return column{}, false
}
sel, ok := s.trace.derived[qualifier]
if !ok {
return column{}, false
}
block, ok := s.trace.blocks[sel]
if !ok {
return column{}, false
}
for _, it := range block.selectItems() {
if it.alias == name && it.end-it.start == 1 && block.at(it.start).kind == tkIdent {
return s.origin(block, block.at(it.start), depth+1)
}
}
return column{}, false
}
// column describes a column of a catalog table.
func (s *statement) column(rel relation, name string) analysis.Column {
if rel.schema == "information_schema" {
name = strings.ToLower(name)
}
for _, col := range s.a.catalog[rel] {
if col.name == name {
return analysis.Column{
Name: name,
Type: withNullable(col.typ, col.nullable),
Table: rel.name,
}
}
}
return analysis.Column{Name: name, Table: rel.name}
}
// eval finds the type of an expression by running it as a query of its
// own, over the tables its columns are read from under the aliases the
// statement gave them. An expression that reads a derived table cannot be
// run that way and gets no type.
func (s *statement) eval(ctx context.Context, t text, start, end int) *analysis.TypeExpr {
var from []string
seen := map[string]bool{}
for _, id := range t.idents(start, end) {
qualifier := id.parts[len(id.parts)-2]
if seen[qualifier] {
continue
}
seen[qualifier] = true
rel, ok := s.resolve(id.parts)
if !ok {
return nil
}
item := quote(rel.schema) + "." + quote(rel.name)
if qualifier != rel.name {
item += " " + quote(qualifier)
}
from = append(from, item)
}
query := "SELECT " + t.slice(start, end)
if len(from) > 0 {
query += " FROM " + strings.Join(from, ", ")
}
rows, err := s.a.conn.QueryContext(ctx, query+" LIMIT 0")
if err != nil {
return nil
}
defer rows.Close()
cts, err := rows.ColumnTypes()
if err != nil || len(cts) != 1 {
return nil
}
return typeOf(cts[0])
}
// insertTargets reads the table and columns an INSERT ... VALUES writes,
// as the note prints them: the columns listed, or every column of the
// table in order when none are.
func (s *statement) insertTargets(t text) (*relation, []string) {
for i, tok := range t.toks {
if tok.kind != tkWord || tok.text != "into" || t.at(i+1).kind != tkIdent {
continue
}
parts := t.at(i + 1).parts
schema := ""
if len(parts) >= 2 {
schema = parts[len(parts)-2]
}
rel, ok := s.a.table(schema, parts[len(parts)-1])
if !ok {
return nil, nil
}
var targets []string
if t.at(i+2).kind == tkLParen {
for _, m := range t.list(i + 2) {
if m[1]-m[0] == 1 && t.at(m[0]).kind == tkIdent {
parts := t.at(m[0]).parts
targets = append(targets, parts[len(parts)-1])
}
}
} else {
for _, col := range s.a.catalog[rel] {
targets = append(targets, col.name)
}
}
return &rel, targets
}
return nil, nil
}

150
internal/goldeneye/mysql/mysql.go Normal file
View File

@@ -0,0 +1,150 @@
// Package mysql generates the MySQL dialect seed under
// internal/engine/dolphin/dialect from a live server, and verifies the
// MySQL analyze cases under internal/endtoend/testdata against the same
// server.
//
// MySQL keeps no catalog of its types, functions or operators — the server
// describes a built-in function no further than its name, in the help
// tables — so types.jsonl, functions.jsonl and operators.jsonl are written
// by hand and are not this package's business. What it does describe is
// its data dictionary: relations.jsonl is every view of information_schema,
// read from information_schema itself.
//
// The package also verifies the MySQL analyze cases under
// internal/endtoend/testdata against the same server: each case's schema
// and fixture are loaded into a database of their own and its queries run
// there. Result columns come from the result set's metadata as the driver
// reports it, provenance and parameters from the optimizer trace's
// expanded_query and the note EXPLAIN leaves, since MySQL reports nothing
// about a parameter but its position. The answer is printed in the JSON
// shape sqlc analyze prints and compared with the case's committed
// stdout.json byte for byte.
//
// The server is named by MYSQL_SERVER_URI, in the form the go-sql-driver
// DSN takes, and has to be the major release in Major, since each release
// adds to information_schema.
package mysql
import (
"context"
"database/sql"
"errors"
"fmt"
"os"
"strconv"
"strings"
driver "github.com/go-sql-driver/mysql"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name sqlc knows the dialect by, and the name of the analyze
// case directories under internal/endtoend/testdata.
const Engine = "mysql"
// Dir is the name of the engine directory the dialect lives under, which
// is named after the parser sqlc reads MySQL with rather than the engine.
const Dir = "dolphin"
// Major is the MySQL major release the dialect is generated from. Bumping
// it is a deliberate change: every release adds tables and columns to the
// system schemas, so regenerate and review the dialect after changing it.
const Major = 26
// Locate returns the server to generate from, named by MYSQL_SERVER_URI.
func Locate() (string, error) {
dsn := os.Getenv("MYSQL_SERVER_URI")
if dsn == "" {
return "", errors.New("MYSQL_SERVER_URI is not set")
}
if _, err := driver.ParseDSN(dsn); err != nil {
return "", fmt.Errorf("MYSQL_SERVER_URI: %w", err)
}
return dsn, nil
}
// open connects to the server. The connection runs several statements at
// once, so that a case's schema loads as one script, and holds one
// session, since the optimizer trace and the warnings a statement leaves
// are both per session.
func open(ctx context.Context, dsn string) (*sql.Conn, error) {
cfg, err := driver.ParseDSN(dsn)
if err != nil {
return nil, err
}
cfg.MultiStatements = true
db, err := sql.Open("mysql", cfg.FormatDSN())
if err != nil {
return nil, err
}
conn, err := db.Conn(ctx)
if err != nil {
db.Close()
return nil, err
}
return conn, nil
}
// Version reports the release a server is.
func Version(ctx context.Context, dsn string) (string, error) {
conn, err := open(ctx, dsn)
if err != nil {
return "", err
}
defer conn.Close()
return version(ctx, conn)
}
func version(ctx context.Context, conn *sql.Conn) (string, error) {
var v string
if err := conn.QueryRowContext(ctx, "SELECT version()").Scan(&v); err != nil {
return "", err
}
return "MySQL " + v, nil
}
// checkVersion refuses a server of another major release than the dialect
// is generated from, whose system schemas would differ from the committed
// ones without anything being wrong.
func checkVersion(ctx context.Context, conn *sql.Conn) error {
var v string
if err := conn.QueryRowContext(ctx, "SELECT version()").Scan(&v); err != nil {
return err
}
head, _, _ := strings.Cut(v, ".")
major, err := strconv.Atoi(head)
if err != nil {
return fmt.Errorf("version %q: %w", v, err)
}
if major != Major {
return fmt.Errorf("the dialect is generated from MySQL %d, but the server is MySQL %d", Major, major)
}
return nil
}
// Generate reads the dialect from the server: relations.jsonl, the views of
// information_schema.
func Generate(ctx context.Context, dsn string) (dialect.Files, error) {
conn, err := open(ctx, dsn)
if err != nil {
return nil, err
}
defer conn.Close()
if err := checkVersion(ctx, conn); err != nil {
return nil, err
}
var relations []dialect.Relation
for _, schema := range systemSchemas {
rels, err := readRelations(ctx, conn, schema)
if err != nil {
return nil, err
}
relations = append(relations, rels...)
}
blob, err := dialect.JSONL(relations)
if err != nil {
return nil, err
}
return dialect.Files{dialect.RelationsFile: blob}, nil
}

66
internal/goldeneye/mysql/mysql_test.go Normal file
View File

@@ -0,0 +1,66 @@
package mysql
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// TestDialect verifies the committed MySQL dialect against what the server
// reports. It skips unless MYSQL_SERVER_URI names a server.
func TestDialect(t *testing.T) {
dsn, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, dsn)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, dsn)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Dir)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}
// TestAnalyzeCases verifies every MySQL analyze case under
// internal/endtoend/testdata against what the server reports. It skips
// unless MYSQL_SERVER_URI names a server.
func TestAnalyzeCases(t *testing.T) {
dsn, err := Locate()
if err != nil {
t.Skip(err)
}
cases, err := endtoend.Cases(Engine)
if err != nil {
t.Fatal(err)
}
if len(cases) == 0 {
t.Fatal("no mysql analyze cases found")
}
for _, c := range cases {
t.Run(c.Name, func(t *testing.T) {
diff, err := Check(context.Background(), dsn, c)
if err != nil {
t.Fatal(err)
}
if diff != "" {
t.Errorf("%s does not match what MySQL reports (-committed +mysql):\n%s", c.Output, diff)
}
})
}
}

99
internal/goldeneye/mysql/relations.go Normal file
View File

@@ -0,0 +1,99 @@
package mysql
import (
"context"
"database/sql"
"fmt"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// systemSchemas are the schemas whose relations are generated: the data
// dictionary's views, which are what a query of the server's catalog
// reads. The other schemas mysqld --initialize creates — mysql,
// performance_schema and sys — are tables rather than views, and every
// table a dialect seeds is one the analysis core hands codegen as a model,
// so they wait until codegen knows a system schema when it sees one.
var systemSchemas = []string{
"information_schema",
}
// relationQuery lists a schema's relations and their columns, in the order
// the columns were declared, so that the output is stable.
const relationQuery = `
SELECT t.TABLE_NAME, t.TABLE_TYPE,
c.COLUMN_NAME, c.DATA_TYPE, c.COLUMN_TYPE, c.IS_NULLABLE
FROM information_schema.TABLES t
JOIN information_schema.COLUMNS c
ON c.TABLE_SCHEMA = t.TABLE_SCHEMA AND c.TABLE_NAME = t.TABLE_NAME
WHERE t.TABLE_SCHEMA = ?
ORDER BY t.TABLE_NAME, c.ORDINAL_POSITION`
// readRelations reads a schema's tables and views. Their names are written
// in lower case: information_schema names its views in upper case and
// matches them in any case, MySQL matches every column name in any case,
// and sqlc's MySQL parser lowercases every identifier, so lower case is
// how a query reaches them. A column's type is spelled the way a
// declaration does, as COLUMN_TYPE reports it, arguments and UNSIGNED
// included: varchar(64), bigint unsigned.
func readRelations(ctx context.Context, conn *sql.Conn, schema string) ([]dialect.Relation, error) {
rows, err := conn.QueryContext(ctx, relationQuery, schema)
if err != nil {
return nil, fmt.Errorf("%s: %w", schema, err)
}
defer rows.Close()
var relations []dialect.Relation
var cur *dialect.Relation
for rows.Next() {
var name, kind, colName, dataType, columnType, nullable string
if err := rows.Scan(&name, &kind, &colName, &dataType, &columnType, &nullable); err != nil {
return nil, err
}
name = strings.ToLower(name)
if cur == nil || cur.Name != name {
rel := dialect.Relation{Catalog: "def", Schema: schema, Name: name}
if kind != "BASE TABLE" {
rel.Kind = "v"
}
relations = append(relations, rel)
cur = &relations[len(relations)-1]
}
cur.Columns = append(cur.Columns, dialect.Column{
Name: strings.ToLower(colName),
Type: typeName(dataType, columnType),
NotNull: nullable == "NO",
})
}
return relations, rows.Err()
}
// typeName spells a column's type the way a declaration does, which is
// COLUMN_TYPE as information_schema reports it: "varchar(64)", "bigint
// unsigned", "enum('a','b')". The family is spelled in lower case; the
// members of an enum or set are values and keep theirs.
func typeName(dataType, columnType string) string {
if columnType == "" {
return strings.ToLower(dataType)
}
var out strings.Builder
quoted := false
for i := 0; i < len(columnType); i++ {
c := columnType[i]
switch {
case quoted && c == '\\' && i+1 < len(columnType):
out.WriteByte(c)
i++
out.WriteByte(columnType[i])
continue
case c == '\'':
quoted = !quoted
}
if quoted {
out.WriteByte(c)
} else {
out.WriteString(strings.ToLower(string(c)))
}
}
return out.String()
}

678
internal/goldeneye/mysql/text.go Normal file
View File

@@ -0,0 +1,678 @@
package mysql
import (
"strings"
)
// MySQL prints a statement back with every name resolved, every table
// alias kept, every SELECT * expanded and every binary operation
// parenthesised — in the optimizer trace's expanded_query, once per query
// block, and in the note EXPLAIN leaves. Such a text is tokenised here and
// read for three things: the tables it names and their aliases, the
// columns each query block projects, and the operand a parameter is
// compared with or assigned to.
type tokKind int
const (
tkIdent tokKind = iota // `a`.`b`.`c`
tkVar // @`name`
tkNumber // 42, 1.5
tkString // 'text'
tkWord // a keyword, a function name or a <marker>
tkOp // =, <=>, +, like is a word
tkLParen
tkRParen
tkComma
)
type token struct {
kind tokKind
text string // the token as printed; a word is lower-cased
parts []string // an identifier's parts, unquoted; a variable's name
start int // offsets into the source
end int
}
// text is a tokenised statement.
type text struct {
src string
toks []token
match []int // the index of each parenthesis's partner, or -1
}
func tokenize(src string) text {
var toks []token
i := 0
emit := func(kind tokKind, start, end int, parts []string) {
toks = append(toks, token{kind: kind, text: src[start:end], parts: parts, start: start, end: end})
}
for i < len(src) {
c := src[i]
switch {
case c == ' ' || c == '\t' || c == '\n' || c == '\r':
i++
case strings.HasPrefix(src[i:], "/*"):
end := strings.Index(src[i:], "*/")
if end < 0 {
i = len(src)
} else {
i += end + 2
}
case c == '`':
start := i
var parts []string
for {
part, end := readQuoted(src, i)
parts = append(parts, part)
i = end
if i+1 < len(src) && src[i] == '.' && src[i+1] == '`' {
i++
continue
}
break
}
emit(tkIdent, start, i, parts)
case c == '@':
start := i
i++
for i < len(src) && src[i] == '@' {
i++
}
if i < len(src) && src[i] == '`' {
name, end := readQuoted(src, i)
i = end
emit(tkVar, start, i, []string{name})
} else {
for i < len(src) && isWordByte(src[i]) {
i++
}
emit(tkVar, start, i, []string{src[start+1 : i]})
}
case c == '\'' || c == '"':
start := i
i = skipString(src, i)
emit(tkString, start, i, nil)
case c >= '0' && c <= '9' || c == '.' && i+1 < len(src) && src[i+1] >= '0' && src[i+1] <= '9':
start := i
for i < len(src) && (src[i] >= '0' && src[i] <= '9' || src[i] == '.') {
i++
}
if i < len(src) && (src[i] == 'e' || src[i] == 'E') {
i++
if i < len(src) && (src[i] == '+' || src[i] == '-') {
i++
}
for i < len(src) && src[i] >= '0' && src[i] <= '9' {
i++
}
}
emit(tkNumber, start, i, nil)
case c == '<' && i+1 < len(src) && isWordByte(src[i+1]) && strings.IndexByte(src[i:], '>') > 0 && isMarker(src[i:i+strings.IndexByte(src[i:], '>')+1]):
// <cache>, <in_optimizer>, <exists>: what the optimizer wrapped
// an expression in. <cache> says nothing about the expression
// and is dropped; the others are read as calls.
end := i + strings.IndexByte(src[i:], '>') + 1
if src[i:end] != "<cache>" {
emit(tkWord, i, end, nil)
toks[len(toks)-1].text = src[i+1 : end-1]
}
i = end
case c == '(':
emit(tkLParen, i, i+1, nil)
i++
case c == ')':
emit(tkRParen, i, i+1, nil)
i++
case c == ',':
emit(tkComma, i, i+1, nil)
i++
case isWordByte(c):
start := i
for i < len(src) && isWordByte(src[i]) {
i++
}
emit(tkWord, start, i, nil)
toks[len(toks)-1].text = strings.ToLower(src[start:i])
default:
n := 1
for _, op := range operators {
if strings.HasPrefix(src[i:], op) {
n = len(op)
break
}
}
emit(tkOp, i, i+n, nil)
i += n
}
}
t := text{src: src, toks: toks, match: make([]int, len(toks))}
var open []int
for i, tok := range toks {
t.match[i] = -1
switch tok.kind {
case tkLParen:
open = append(open, i)
case tkRParen:
if len(open) > 0 {
j := open[len(open)-1]
open = open[:len(open)-1]
t.match[i] = j
t.match[j] = i
}
}
}
return t
}
// operators is every symbolic operator MySQL prints, longest first.
var operators = []string{
"<=>", "->>", "<>", "!=", ">=", "<=", "->", "<<", ">>", "||", "&&", ":=",
"=", "<", ">", "+", "-", "*", "/", "%", "!", "^", "&", "|", "~", ".",
}
func isMarker(s string) bool {
for _, c := range []byte(s[1 : len(s)-1]) {
if !isWordByte(c) {
return false
}
}
return true
}
func isWordByte(c byte) bool {
return c == '_' || c == '$' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'
}
// readQuoted reads the backticked identifier at i, unescaping a doubled
// backtick, and returns it with the index just past its closing quote.
func readQuoted(s string, i int) (string, int) {
var b strings.Builder
j := i + 1
for j < len(s) {
if s[j] == '`' {
if j+1 < len(s) && s[j+1] == '`' {
b.WriteByte('`')
j += 2
continue
}
return b.String(), j + 1
}
b.WriteByte(s[j])
j++
}
return b.String(), j
}
// skipString returns the index just past the string literal starting at i,
// honouring backslash escapes and doubled quotes.
func skipString(s string, i int) int {
q := s[i]
j := i + 1
for j < len(s) {
switch {
case s[j] == '\\' && j+1 < len(s):
j += 2
case s[j] == q && j+1 < len(s) && s[j+1] == q:
j += 2
case s[j] == q:
return j + 1
default:
j++
}
}
return len(s)
}
// wordOperators are the operators MySQL spells as words. and/or/not/is
// are left out: MySQL parenthesises each predicate, so a parameter that
// is a whole predicate has nothing to be compared with.
var wordOperators = map[string]bool{
"like": true, "in": true, "regexp": true, "rlike": true, "div": true, "mod": true,
"xor": true, "sounds": true, "escape": true, "between": true, "collate": true,
"member": true,
}
// keywords are the words that can precede a parenthesis without naming a
// function, so that "(a, b)" after one is a list rather than a call.
var keywords = map[string]bool{
"in": true, "values": true, "value": true, "and": true, "or": true, "not": true,
"where": true, "on": true, "from": true, "select": true, "having": true,
"set": true, "when": true, "then": true, "else": true, "case": true, "end": true,
"like": true, "between": true, "by": true, "group": true, "order": true,
"limit": true, "offset": true, "using": true, "join": true, "as": true, "is": true,
"xor": true, "div": true, "mod": true, "interval": true, "distinct": true,
"all": true, "any": true, "some": true, "escape": true, "regexp": true,
"rlike": true, "sounds": true, "collate": true, "union": true, "except": true,
"intersect": true, "with": true, "insert": true, "into": true, "update": true,
"delete": true, "replace": true, "key": true, "duplicate": true, "do": true,
"partition": true, "over": true, "window": true, "rows": true, "range": true,
"straight_join": true, "cross": true, "inner": true, "left": true, "right": true,
"natural": true, "lateral": true, "of": true, "for": true, "lock": true,
"share": true, "null": true, "true": true, "false": true, "unknown": true,
"binary": true, "row": true, "asc": true, "desc": true, "sql_calc_found_rows": true,
"high_priority": true, "low_priority": true, "ignore": true, "table": true,
"member": true, "exists": true,
}
// literalWords are the words that are operands on their own.
var literalWords = map[string]bool{
"null": true, "true": true, "false": true, "unknown": true,
"current_timestamp": true, "current_date": true, "current_time": true,
"current_user": true, "localtime": true, "localtimestamp": true, "utc_timestamp": true,
}
func (t text) at(i int) token {
if i < 0 || i >= len(t.toks) {
return token{kind: -1}
}
return t.toks[i]
}
func (t text) isWord(i int, word string) bool {
tok := t.at(i)
return tok.kind == tkWord && tok.text == word
}
// isCall reports whether the word at i names a function, given that a
// parenthesis follows it.
func (t text) isCall(i int) bool {
tok := t.at(i)
return tok.kind == tkWord && !keywords[tok.text] || tok.kind == tkWord && tok.text == "exists"
}
// isOperator reports whether the token at i joins two operands.
func (t text) isOperator(i int) bool {
tok := t.at(i)
switch tok.kind {
case tkOp:
return tok.text != "." && tok.text != "~" && tok.text != "!"
case tkWord:
return wordOperators[tok.text]
}
return false
}
// operandEnd returns the index just past the operand starting at i, or -1
// when no operand starts there: a name, a literal, a call with its
// arguments, or a parenthesised group.
func (t text) operandEnd(i int) int {
tok := t.at(i)
switch tok.kind {
case tkIdent, tkVar, tkNumber, tkString:
return i + 1
case tkLParen:
if t.match[i] < 0 {
return -1
}
return t.match[i] + 1
case tkWord:
if literalWords[tok.text] {
return i + 1
}
if t.at(i+1).kind == tkLParen && t.isCall(i) {
return t.operandEnd(i + 1)
}
if tok.text == "not" || tok.text == "binary" || tok.text == "interval" {
return t.operandEnd(i + 1)
}
case tkOp:
if tok.text == "-" || tok.text == "+" || tok.text == "~" || tok.text == "!" {
return t.operandEnd(i + 1)
}
}
return -1
}
// operandStart returns the index of the operand ending just before j, or
// -1 when none ends there.
func (t text) operandStart(j int) int {
k := j - 1
tok := t.at(k)
switch tok.kind {
case tkIdent, tkVar, tkNumber, tkString:
return k
case tkRParen:
open := t.match[k]
if open < 0 {
return -1
}
if t.at(open-1).kind == tkWord && t.isCall(open-1) {
return open - 1
}
return open
case tkWord:
if literalWords[tok.text] {
return k
}
}
return -1
}
// group returns the index of the parenthesis enclosing token i, or -1.
func (t text) group(i int) int {
depth := 0
for k := i - 1; k >= 0; k-- {
switch t.toks[k].kind {
case tkRParen:
depth++
case tkLParen:
if depth == 0 {
return k
}
depth--
}
}
return -1
}
// list splits the tokens strictly inside the group opened at open into its
// comma-separated members, each a token range.
func (t text) list(open int) [][2]int {
close := t.match[open]
if close < 0 {
return nil
}
var members [][2]int
start := open + 1
for i := open + 1; i < close; i++ {
switch t.toks[i].kind {
case tkLParen:
i = t.match[i]
case tkComma:
members = append(members, [2]int{start, i})
start = i + 1
}
}
if start < close {
members = append(members, [2]int{start, close})
}
return members
}
// item is one entry of a SELECT list: the expression and the name it is
// projected as.
type item struct {
start, end int
alias string
}
// selectItems reads the list of the first SELECT at the top level of the
// text, which is the statement's own for a query block's expanded_query
// and for a statement whose CTEs come first.
func (t text) selectItems() []item {
depth := 0
start := -1
for i, tok := range t.toks {
switch tok.kind {
case tkLParen:
depth++
case tkRParen:
depth--
case tkWord:
if depth == 0 && tok.text == "select" {
start = i + 1
}
}
if start >= 0 {
break
}
}
if start < 0 {
return nil
}
for t.at(start).kind == tkWord && (t.at(start).text == "distinct" || t.at(start).text == "all" || t.at(start).text == "sql_calc_found_rows" || t.at(start).text == "straight_join" || t.at(start).text == "high_priority") {
start++
}
end := len(t.toks)
for i := start; i < len(t.toks); i++ {
switch tok := t.toks[i]; tok.kind {
case tkLParen:
i = t.match[i]
if i < 0 {
return nil
}
case tkWord:
switch tok.text {
case "from", "where", "group", "having", "order", "limit", "union", "into", "window", "for", "lock", "except", "intersect":
end = i
}
}
if end != len(t.toks) {
break
}
}
var items []item
itemStart := start
flush := func(itemEnd int) {
it := item{start: itemStart, end: itemEnd}
if itemEnd-itemStart >= 3 && t.isWord(itemEnd-2, "as") && t.at(itemEnd-1).kind == tkIdent {
it.alias = t.toks[itemEnd-1].parts[0]
it.end = itemEnd - 2
}
items = append(items, it)
}
for i := start; i < end; i++ {
switch t.toks[i].kind {
case tkLParen:
i = t.match[i]
case tkComma:
flush(i)
itemStart = i + 1
}
}
if itemStart < end {
flush(end)
}
return items
}
// tableRef is a table as a FROM clause names it, with the schema when it
// spells one out.
type tableRef struct {
schema, name string
}
// aliases reads the table aliases the text declares: a table name followed
// by a bare identifier, as `users` `u` in a FROM clause. A SELECT list
// alias is always introduced by AS, so the shape means nothing else.
func (t text) aliases(into map[string]tableRef) {
for i := 0; i+1 < len(t.toks); i++ {
tok, next := t.toks[i], t.toks[i+1]
if tok.kind != tkIdent || next.kind != tkIdent || len(next.parts) != 1 || len(tok.parts) > 2 {
continue
}
if t.at(i-1).kind == tkOp && t.at(i-1).text == "." {
continue
}
ref := tableRef{name: tok.parts[len(tok.parts)-1]}
if len(tok.parts) == 2 {
ref.schema = tok.parts[0]
}
into[next.parts[0]] = ref
}
}
// find returns the index of the variable named name, or -1.
func (t text) findVar(name string) int {
for i, tok := range t.toks {
if tok.kind == tkVar && len(tok.parts) == 1 && tok.parts[0] == name {
return i
}
}
return -1
}
// findNumber returns the index of the number printed as lit, or -1.
func (t text) findNumber(lit string) int {
for i, tok := range t.toks {
if tok.kind == tkNumber && tok.text == lit {
return i
}
}
return -1
}
// A partner is what a parameter stands against in a statement.
type partnerKind int
const (
partnerNone partnerKind = iota
partnerOperand // the operand it is compared with, or the argument beside it
partnerProjection // the SELECT list item it is projected as
partnerInsert // the column of an INSERT ... VALUES row it is written to
partnerLimit // the count of a LIMIT or OFFSET
)
type partner struct {
kind partnerKind
start, end int // partnerOperand: the operand's tokens
index int // partnerProjection: the item; partnerInsert: the position in the row
}
// partner finds what the parameter at token index s stands against.
func (t text) partner(s int) partner {
lo, hi := s, s+1
if t.at(lo-1).kind == tkLParen && t.at(hi).kind == tkRParen {
lo, hi = lo-1, hi+1
}
if t.at(hi).kind == tkWord && t.at(hi).text == "as" {
for i, it := range t.selectItems() {
if it.start <= lo && hi <= it.end {
return partner{kind: partnerProjection, index: i}
}
}
}
// The count of a LIMIT or OFFSET, which is a number rather than a
// variable since MySQL allows nothing else there.
if t.at(lo-1).kind == tkWord && (t.at(lo-1).text == "limit" || t.at(lo-1).text == "offset") ||
t.at(lo-1).kind == tkComma && t.at(lo-2).kind == tkNumber && t.isWord(lo-3, "limit") {
return partner{kind: partnerLimit}
}
// Compared: the operand on the other side of the operator.
if t.isOperator(lo - 1) {
q := lo - 2
if t.isWord(q, "not") {
q--
}
if st := t.operandStart(q + 1); st >= 0 {
// The second bound of BETWEEN ... AND: the operand before BETWEEN.
if t.isWord(lo-1, "and") {
if t.isWord(st-1, "between") || t.isWord(st-1, "not") && t.isWord(st-2, "between") {
q = st - 2
if t.isWord(q, "not") {
q--
}
if st2 := t.operandStart(q + 1); st2 >= 0 {
return partner{kind: partnerOperand, start: st2, end: q + 1}
}
}
}
return partner{kind: partnerOperand, start: st, end: q + 1}
}
}
if t.isWord(lo-1, "and") {
if st := t.operandStart(lo - 1); st >= 0 && (t.isWord(st-1, "between") || t.isWord(st-1, "not") && t.isWord(st-2, "between")) {
q := st - 2
if t.isWord(q, "not") {
q--
}
if st2 := t.operandStart(q + 1); st2 >= 0 {
return partner{kind: partnerOperand, start: st2, end: q + 1}
}
}
}
if t.isOperator(hi) {
n := hi + 1
if t.isWord(n, "not") {
n++
}
if end := t.operandEnd(n); end >= 0 {
return partner{kind: partnerOperand, start: n, end: end}
}
}
// A member of a list: the row of an INSERT ... VALUES, the values of an
// IN, or the arguments of a call, where the argument beside it says
// what it is, a column above an expression above a constant.
if t.at(lo-1).kind == tkComma || t.at(lo-1).kind == tkLParen {
open := t.group(lo)
if open < 0 {
return partner{}
}
members := t.list(open)
index := -1
for i, m := range members {
if m[0] <= lo && hi <= m[1] {
index = i
}
}
head := open - 1
for t.at(head).kind == tkComma && t.at(head-1).kind == tkRParen {
head = t.match[head-1] - 1
}
switch {
case t.isWord(head, "values") || t.isWord(head, "value"):
return partner{kind: partnerInsert, index: index}
case t.isWord(head, "in"):
q := head - 1
if t.isWord(q, "not") {
q--
}
if st := t.operandStart(q + 1); st >= 0 {
return partner{kind: partnerOperand, start: st, end: q + 1}
}
case t.at(head).kind == tkWord && t.isCall(head):
best, bestRank := -1, 3
for i, m := range members {
if i == index {
continue
}
if r := t.rank(m[0], m[1]); r < bestRank {
best, bestRank = i, r
}
}
if best >= 0 {
return partner{kind: partnerOperand, start: members[best][0], end: members[best][1]}
}
}
}
return partner{}
}
// rank orders operands by how much they say about a parameter beside them.
func (t text) rank(start, end int) int {
switch tok := t.at(start); {
case end-start == 1 && tok.kind == tkIdent:
return 0
case tok.kind == tkVar:
return 3
case end-start == 1:
return 2
}
return 1
}
// idents lists the qualified identifiers in a token range: the columns an
// expression reads.
func (t text) idents(start, end int) []token {
var out []token
for i := start; i < end; i++ {
if tok := t.toks[i]; tok.kind == tkIdent && len(tok.parts) >= 2 {
out = append(out, tok)
}
}
return out
}
// slice is the source text of a token range.
func (t text) slice(start, end int) string {
if start >= end {
return ""
}
return t.src[t.toks[start].start:t.toks[end-1].end]
}
// quote backticks an identifier.
func quote(name string) string {
return "`" + strings.ReplaceAll(name, "`", "``") + "`"
}

87
internal/goldeneye/mysql/trace.go Normal file
View File

@@ -0,0 +1,87 @@
package mysql
import (
"encoding/json"
"fmt"
"sort"
"strings"
)
// trace is what the optimizer trace says about a statement: the text of
// each query block after name resolution and before optimisation, by the
// number MySQL gives the block, and the block each derived table or CTE is
// materialised from.
//
// The trace is the one place MySQL prints a query block as it was resolved
// rather than as it was optimised. The note EXPLAIN leaves is printed
// after optimisation, and a lookup on a unique key against an empty table
// folds to `NULL = (@x)` there, losing the column the parameter was
// compared with; the expanded_query of the block still has it.
type trace struct {
blocks map[int]text
derived map[string]int
}
// parseTrace reads the JSON of the OPTIMIZER_TRACE table's TRACE column.
func parseTrace(blob string) (trace, error) {
tr := trace{blocks: map[int]text{}, derived: map[string]int{}}
if strings.TrimSpace(blob) == "" {
return tr, nil
}
var v any
if err := json.Unmarshal([]byte(blob), &v); err != nil {
return tr, fmt.Errorf("reading the optimizer trace: %w", err)
}
tr.walk(v, 0)
return tr, nil
}
// walk collects the first expanded_query printed for each block — the one
// join_preparation prints, before join_optimization prints its rewrite —
// and the derived tables. Lists are walked in order and objects in key
// order, so the result does not depend on map iteration.
func (tr *trace) walk(v any, sel int) {
switch v := v.(type) {
case map[string]any:
if n, ok := v["select#"].(float64); ok {
sel = int(n)
}
if d, ok := v["derived"].(map[string]any); ok {
if table, ok := d["table"].(string); ok {
if n, ok := d["select#"].(float64); ok {
tr.derived[strings.Trim(table, "` ")] = int(n)
}
}
}
if q, ok := v["expanded_query"].(string); ok {
if _, seen := tr.blocks[sel]; !seen {
tr.blocks[sel] = tokenize(q)
}
}
keys := make([]string, 0, len(v))
for k := range v {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
tr.walk(v[k], sel)
}
case []any:
for _, c := range v {
tr.walk(c, sel)
}
}
}
// order lists the block numbers innermost first: a block's text includes
// its subqueries' text, and a subquery is numbered after the block it is
// in, so the block a parameter belongs to is the highest-numbered block
// whose text has it.
func (tr *trace) order() []int {
sels := make([]int, 0, len(tr.blocks))
for sel := range tr.blocks {
sels = append(sels, sel)
}
sort.Sort(sort.Reverse(sort.IntSlice(sels)))
return sels
}

364
internal/goldeneye/postgresql/postgresql.go Normal file
View File

@@ -0,0 +1,364 @@
// Package postgresql generates the PostgreSQL dialect seed under
// internal/engine/postgresql/dialect from a live server: pg_catalog's
// functions, the relations of pg_catalog and information_schema, and one
// directory per contrib extension holding the types and functions CREATE
// EXTENSION adds. types.jsonl and operators.jsonl at the top of the dialect
// are written by hand and are not this package's business.
//
// The server is named by POSTGRESQL_SERVER_URI and has to be the major
// release in Major, since each release adds to its catalogs.
package postgresql
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path"
"sort"
"strconv"
"strings"
"github.com/jackc/pgx/v5"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name of the engine directory the dialect lives under.
const Engine = "postgresql"
// Major is the PostgreSQL major release the dialect is generated from.
// Bumping it is a deliberate change: every release adds functions and
// catalog columns, so regenerate and review the dialect after changing it.
const Major = 16
// https://dba.stackexchange.com/questions/255412/how-to-select-functions-that-belong-in-a-given-extension-in-postgresql
//
// Extension functions are added to the public schema
const extensionFuncs = `
WITH extension_funcs AS (
SELECT p.oid
FROM pg_catalog.pg_extension AS e
INNER JOIN pg_catalog.pg_depend AS d ON (d.refobjid = e.oid)
INNER JOIN pg_catalog.pg_proc AS p ON (p.oid = d.objid)
INNER JOIN pg_catalog.pg_namespace AS ne ON (ne.oid = e.extnamespace)
INNER JOIN pg_catalog.pg_namespace AS np ON (np.oid = p.pronamespace)
WHERE d.deptype = 'e' AND e.extname = $1
)
SELECT p.proname as name,
format_type(p.prorettype, NULL),
array(select format_type(unnest(p.proargtypes), NULL)),
p.proargnames,
p.proargnames[p.pronargs-p.pronargdefaults+1:p.pronargs],
p.proargmodes::text[]
FROM pg_catalog.pg_proc p
JOIN extension_funcs ef ON ef.oid = p.oid
WHERE pg_function_is_visible(p.oid)
-- simply order all columns to keep subsequent runs stable
ORDER BY 1, 2, 3, 4, 5;
`
// The types an extension defines, in the order they were declared so that a
// type appears after the types it is built on. Array types are created
// implicitly rather than by the extension's script, so they carry an internal
// dependency and never show up here.
const extensionTypes = `
SELECT t.typname, t.typcategory
FROM pg_catalog.pg_extension AS e
INNER JOIN pg_catalog.pg_depend AS d ON (d.refobjid = e.oid)
INNER JOIN pg_catalog.pg_type AS t ON (t.oid = d.objid)
WHERE d.deptype = 'e'
AND d.classid = 'pg_catalog.pg_type'::regclass
AND e.extname = $1
AND t.typtype <> 'p'
ORDER BY t.oid;
`
// Locate returns the server to generate from, named by POSTGRESQL_SERVER_URI.
func Locate() (string, error) {
if url := os.Getenv("POSTGRESQL_SERVER_URI"); url != "" {
return url, nil
}
return "", errors.New("POSTGRESQL_SERVER_URI is not set")
}
// Version reports the release a server is.
func Version(ctx context.Context, url string) (string, error) {
conn, err := pgx.Connect(ctx, url)
if err != nil {
return "", err
}
defer conn.Close(ctx)
var version string
if err := conn.QueryRow(ctx, "SELECT version()").Scan(&version); err != nil {
return "", err
}
return version, nil
}
// checkVersion refuses a server of another major release than the dialect
// is generated from, whose catalogs would differ from the committed ones
// without anything being wrong.
func checkVersion(ctx context.Context, conn *pgx.Conn) error {
var num string
if err := conn.QueryRow(ctx, "SELECT current_setting('server_version_num')").Scan(&num); err != nil {
return err
}
n, err := strconv.Atoi(num)
if err != nil {
return fmt.Errorf("server_version_num %q: %w", num, err)
}
if major := n / 10000; major != Major {
return fmt.Errorf("the dialect is generated from PostgreSQL %d, but the server is PostgreSQL %d", Major, major)
}
return nil
}
func clean(arg string) string {
arg = strings.TrimSpace(arg)
arg = strings.Replace(arg, "\"any\"", "any", -1)
arg = strings.Replace(arg, "\"char\"", "char", -1)
arg = strings.Replace(arg, "\"timestamp\"", "char", -1)
return arg
}
// encodeFunctions encodes procs as JSONL, one function per line, in the form
// the seed package reads. Both the analysis core and the catalog the legacy
// compiler builds load the result.
func encodeFunctions(procs []Proc) ([]byte, error) {
funcs := make([]dialect.Function, 0, len(procs))
for _, proc := range procs {
fn := dialect.Function{Name: proc.Name, Returns: proc.ReturnTypeName()}
for _, arg := range proc.Args() {
a := dialect.Arg{Name: arg.Name, Type: arg.TypeName(), HasDefault: arg.HasDefault}
if arg.Mode != "" && arg.Mode != "i" {
a.Mode = arg.Mode
}
fn.Args = append(fn.Args, a)
}
funcs = append(funcs, fn)
}
return dialect.JSONL(funcs)
}
// readExtensionTypes reads the types an extension defines.
func readExtensionTypes(ctx context.Context, conn *pgx.Conn, extension string) ([]dialect.Type, error) {
rows, err := conn.Query(ctx, extensionTypes, extension)
if err != nil {
return nil, err
}
defer rows.Close()
var types []dialect.Type
for rows.Next() {
var t dialect.Type
if err := rows.Scan(&t.Name, &t.Category); err != nil {
return nil, err
}
types = append(types, t)
}
return types, rows.Err()
}
// writeRelations appends a schema's relations to out as JSONL, one relation
// per line, in the form the seed package reads.
func writeRelations(out *bytes.Buffer, schemaName string, relations []Relation) error {
enc := json.NewEncoder(out)
for _, relation := range relations {
rec := dialect.Relation{
Catalog: relation.Catalog,
Schema: schemaName,
Name: relation.Name,
}
for _, col := range relation.Columns {
c := dialect.Column{
Name: col.Name,
Type: col.Type,
NotNull: col.IsNotNull,
Array: col.IsArray,
}
if col.Length != nil {
c.Length = *col.Length
}
rec.Columns = append(rec.Columns, c)
}
if err := enc.Encode(rec); err != nil {
return err
}
}
return nil
}
// preserveLegacyCatalogBehavior maintain previous ordering and filtering
// that was manually done to the generated file pg_catalog.go.
// Some of the test depend on this ordering - in particular, function lookups
// where there might be multiple matching functions (due to type overloads)
// Until sqlc supports "smarter" looking up of these functions,
// preserveLegacyCatalogBehavior ensures there are no accidental test breakages
func preserveLegacyCatalogBehavior(allProcs []Proc) []Proc {
// Preserve the legacy sort order of the end-to-end tests
sort.SliceStable(allProcs, func(i, j int) bool {
fnA := allProcs[i]
fnB := allProcs[j]
if fnA.Name == "lower" && fnB.Name == "lower" && len(fnA.ArgTypes) == 1 && fnA.ArgTypes[0] == "text" {
return true
}
if fnA.Name == "generate_series" && fnB.Name == "generate_series" && len(fnA.ArgTypes) == 2 && fnA.ArgTypes[0] == "numeric" {
return true
}
return false
})
procs := make([]Proc, 0, len(allProcs))
for _, p := range allProcs {
// Skip generating pg_catalog.concat to preserve legacy behavior
if p.Name == "concat" {
continue
}
procs = append(procs, p)
}
return procs
}
// Generate reads the dialect from the server. It creates every contrib
// extension it describes, so the server's contrib package has to be
// installed.
func Generate(ctx context.Context, url string) (dialect.Files, error) {
conn, err := pgx.Connect(ctx, url)
if err != nil {
return nil, err
}
defer conn.Close(ctx)
if err := checkVersion(ctx, conn); err != nil {
return nil, err
}
files := dialect.Files{}
// The two schemas sqlc knows PostgreSQL by. pg_catalog's functions are
// the dialect's standard library; the relations of both share one file,
// keyed by schema.
procs, err := readProcs(ctx, conn, "pg_catalog", extensions)
if err != nil {
return nil, err
}
if files[dialect.FunctionsFile], err = encodeFunctions(preserveLegacyCatalogBehavior(procs)); err != nil {
return nil, err
}
var relations bytes.Buffer
for _, schema := range []string{"pg_catalog", "information_schema"} {
rels, err := readRelations(ctx, conn, schema)
if err != nil {
return nil, err
}
if err := writeRelations(&relations, schema, rels); err != nil {
return nil, err
}
}
files[dialect.RelationsFile] = relations.Bytes()
// Each extension is a directory of its own under the dialect, holding the
// functions CREATE EXTENSION adds to the catalog.
for _, extension := range extensions {
if _, err := conn.Exec(ctx, fmt.Sprintf("CREATE EXTENSION IF NOT EXISTS %q", extension)); err != nil {
return nil, fmt.Errorf("error creating %s: %s", extension, err)
}
rows, err := conn.Query(ctx, extensionFuncs, extension)
if err != nil {
return nil, err
}
procs, err := scanProcs(rows)
if err != nil {
return nil, err
}
types, err := readExtensionTypes(ctx, conn, extension)
if err != nil {
return nil, err
}
if len(procs) == 0 && len(types) == 0 {
continue
}
// Preserve the legacy sort order of the end-to-end tests
sort.SliceStable(procs, func(i, j int) bool {
fnA := procs[i]
fnB := procs[j]
if extension == "pgcrypto" {
if fnA.Name == "digest" && fnB.Name == "digest" && len(fnA.ArgTypes) == 2 && fnA.ArgTypes[0] == "text" {
return true
}
}
return false
})
extensionDir := path.Join(dialect.ExtensionsDir, extension)
if len(types) > 0 {
if files[path.Join(extensionDir, dialect.TypesFile)], err = dialect.JSONL(types); err != nil {
return nil, fmt.Errorf("error generating extension %s: %w", extension, err)
}
}
if files[path.Join(extensionDir, dialect.FunctionsFile)], err = encodeFunctions(procs); err != nil {
return nil, fmt.Errorf("error generating extension %s: %w", extension, err)
}
}
return files, nil
}
// https://www.postgresql.org/docs/current/contrib.html
var extensions = []string{
"adminpack",
"amcheck",
// "auth_delay",
// "auto_explain",
// "bloom",
"btree_gin",
"btree_gist",
"citext",
"cube",
"dblink",
// "dict_int",
// "dict_xsyn",
"earthdistance",
"file_fdw",
"fuzzystrmatch",
"hstore",
"intagg",
"intarray",
"isn",
"lo",
"ltree",
"pageinspect",
// "passwordcheck",
"pg_buffercache",
"pg_freespacemap",
"pg_prewarm",
"pg_stat_statements",
"pg_trgm",
"pg_visibility",
"pgcrypto",
"pgrowlocks",
"pgstattuple",
"postgres_fdw",
"seg",
// "sepgsql",
// "spi",
"sslinfo",
"tablefunc",
"tcn",
// "test_decoding",
// "tsm_system_rows",
// "tsm_system_time",
"unaccent",
"uuid-ossp",
"xml2",
}

37
internal/goldeneye/postgresql/postgresql_test.go Normal file
View File

@@ -0,0 +1,37 @@
package postgresql
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// TestDialect verifies the committed PostgreSQL dialect against what the
// server reports. It skips unless POSTGRESQL_SERVER_URI names a server.
func TestDialect(t *testing.T) {
url, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, url)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, url)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Engine)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}

163
internal/goldeneye/postgresql/proc.go Normal file
View File

@@ -0,0 +1,163 @@
package postgresql
import (
"context"
"strings"
pgx "github.com/jackc/pgx/v5"
)
// https://stackoverflow.com/questions/25308765/postgresql-how-can-i-inspect-which-arguments-to-a-procedure-have-a-default-valu
const catalogFuncs = `
SELECT p.proname as name,
format_type(p.prorettype, NULL),
array(select format_type(unnest(p.proargtypes), NULL)),
p.proargnames,
p.proargnames[p.pronargs-p.pronargdefaults+1:p.pronargs],
p.proargmodes::text[]
FROM pg_catalog.pg_proc p
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname::text = $1
AND pg_function_is_visible(p.oid)
-- A function that one of the extensions the dialect describes separately
-- put in the schema, as adminpack puts its own in pg_catalog, belongs to
-- that extension's directory: leaving it out here keeps the output the
-- same whether or not the extension is installed.
AND NOT EXISTS (
SELECT 1
FROM pg_catalog.pg_depend d
INNER JOIN pg_catalog.pg_extension e ON e.oid = d.refobjid
WHERE d.classid = 'pg_catalog.pg_proc'::regclass
AND d.objid = p.oid
AND d.deptype = 'e'
AND e.extname = ANY($2))
-- simply order all columns to keep subsequent runs stable
ORDER BY 1, 2, 3, 4, 5;
`
type Proc struct {
Name string
ReturnType string
ArgTypes []string
ArgNames []string
HasDefault []string
ArgModes []string
}
func (p *Proc) ReturnTypeName() string {
return clean(p.ReturnType)
}
func (p *Proc) Args() []Arg {
var args []Arg
defaults := map[string]bool{}
for _, name := range p.HasDefault {
defaults[name] = true
}
for i, argType := range p.ArgTypes {
mode := "i"
name := ""
if i < len(p.ArgModes) {
mode = p.ArgModes[i]
}
if i < len(p.ArgNames) {
name = p.ArgNames[i]
}
args = append(args, Arg{
Name: name,
Type: argType,
Mode: mode,
HasDefault: defaults[name],
})
}
// Some manual changes until https://github.com/sqlc-dev/sqlc/pull/1748
// can be completely implmented
if p.Name == "mode" {
return nil
}
if p.Name == "percentile_cont" && len(args) == 2 {
args = args[:1]
}
if p.Name == "percentile_disc" && len(args) == 2 {
args = args[:1]
}
return args
}
type Arg struct {
Name string
Mode string
Type string
HasDefault bool
}
func (a *Arg) TypeName() string {
return clean(a.Type)
}
func scanProcs(rows pgx.Rows) ([]Proc, error) {
defer rows.Close()
// Iterate through the result set
var procs []Proc
for rows.Next() {
var p Proc
err := rows.Scan(
&p.Name,
&p.ReturnType,
&p.ArgTypes,
&p.ArgNames,
&p.HasDefault,
&p.ArgModes,
)
if err != nil {
return nil, err
}
// TODO: Filter these out in SQL
if strings.HasPrefix(p.ReturnType, "SETOF") {
continue
}
// The internal pseudo-type is used to declare functions that are meant
// only to be called internally by the database system, and not by
// direct invocation in an SQL query. If a function has at least one
// internal-type argument then it cannot be called from SQL. To
// preserve the type safety of this restriction it is important to
// follow this coding rule: do not create any function that is declared
// to return internal unless it has at least one internal argument
//
// https://www.postgresql.org/docs/current/datatype-pseudo.html
var skip bool
for i := range p.ArgTypes {
if p.ArgTypes[i] == "internal" {
skip = true
}
}
if skip {
continue
}
if p.ReturnType == "internal" {
continue
}
procs = append(procs, p)
}
return procs, rows.Err()
}
// readProcs reads the functions of a schema, leaving out the ones that
// belong to any of the named extensions.
func readProcs(ctx context.Context, conn *pgx.Conn, schemaName string, extensions []string) ([]Proc, error) {
rows, err := conn.Query(ctx, catalogFuncs, schemaName, extensions)
if err != nil {
return nil, err
}
return scanProcs(rows)
}

118
internal/goldeneye/postgresql/relation.go Normal file
View File

@@ -0,0 +1,118 @@
package postgresql
import (
"context"
pgx "github.com/jackc/pgx/v5"
)
// Relations are the relations available in pg_tables and pg_views
// such as pg_catalog.pg_timezone_names
const relationQuery = `
with relations as (
select schemaname, tablename as name from pg_catalog.pg_tables
UNION ALL
select schemaname, viewname as name from pg_catalog.pg_views
)
select
relations.schemaname,
relations.name as tablename,
pg_attribute.attname as column_name,
attnotnull as column_notnull,
-- An array column's type is its element's, with the array flag set,
-- rather than pg_type's own _text spelling of the array type. Only the
-- _-prefixed array types count: int2vector and oidvector are in the
-- array category and carry an element, but are types of their own.
coalesce(element_type.typname, column_type.typname) as column_type,
nullif(coalesce(element_type.typlen, column_type.typlen), -1) as column_length,
element_type.oid is not null as column_isarray
from relations
inner join pg_catalog.pg_class on pg_class.relname = relations.name
left join pg_catalog.pg_attribute on pg_attribute.attrelid = pg_class.oid
inner join pg_catalog.pg_type column_type on pg_attribute.atttypid = column_type.oid
left join pg_catalog.pg_type element_type
on column_type.typcategory = 'A' and column_type.typname like '\_%'
and element_type.oid = column_type.typelem
where relations.schemaname = $1
-- Make sure these columns are always generated in the same order
-- so that the output is stable
order by
relations.schemaname ASC,
relations.name ASC,
pg_attribute.attnum ASC
`
type Relation struct {
Catalog string
SchemaName string
Name string
Columns []RelationColumn
}
type RelationColumn struct {
Name string
Type string
IsNotNull bool
IsArray bool
Length *int
}
func scanRelations(rows pgx.Rows) ([]Relation, error) {
defer rows.Close()
// Iterate through the result set
var relations []Relation
var prevRel *Relation
for rows.Next() {
var schemaName string
var tableName string
var columnName string
var columnNotNull bool
var columnType string
var columnLength *int
var columnIsArray bool
err := rows.Scan(
&schemaName,
&tableName,
&columnName,
&columnNotNull,
&columnType,
&columnLength,
&columnIsArray,
)
if err != nil {
return nil, err
}
if prevRel == nil || tableName != prevRel.Name {
// We are on the same table, just keep adding columns
r := Relation{
Catalog: "pg_catalog",
SchemaName: schemaName,
Name: tableName,
}
relations = append(relations, r)
prevRel = &relations[len(relations)-1]
}
prevRel.Columns = append(prevRel.Columns, RelationColumn{
Name: columnName,
Type: columnType,
IsNotNull: columnNotNull,
IsArray: columnIsArray,
Length: columnLength,
})
}
return relations, rows.Err()
}
func readRelations(ctx context.Context, conn *pgx.Conn, schemaName string) ([]Relation, error) {
rows, err := conn.Query(ctx, relationQuery, schemaName)
if err != nil {
return nil, err
}
return scanRelations(rows)
}

673
internal/goldeneye/spanner/analyze.go Normal file
View File

@@ -0,0 +1,673 @@
package spanner
import (
"context"
"fmt"
"hash/crc32"
"os"
"regexp"
"strconv"
"strings"
"cloud.google.com/go/spanner/apiv1/spannerpb"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// The analyze cases are checked against a live server, which compiles
// each query in PLAN mode: nothing runs, and the server reports the name
// and type of each result column, the type of each parameter the query
// leaves undeclared, and the query plan. The metadata types a value the
// way the wire does — STRING, ARRAY<INT64> — without the length a
// declaration gives a STRING(10) or whether the column can be NULL, so a
// result column the plan reads from a table column is described from the
// case's INFORMATION_SCHEMA instead, and a parameter the plan compares
// with or assigns to a table column is described as that column.
// placeholder is one parameter of a query as sqlc numbers them: each
// @name or sqlc.arg name once, at its first appearance, which is how
// Spanner numbers its named parameters too.
type placeholder struct {
Number int
Name string
}
var identRe = regexp.MustCompile(`^@([A-Za-z_][A-Za-z0-9_]*)`)
// bind rewrites the query so that every parameter is a Spanner @name, and
// lists the parameters in sqlc's order. The query's own @name references
// are kept, since that is how sqlc's GoogleSQL queries name their
// parameters; a sqlc.arg becomes one, and each ? in turn becomes one
// named after its position. A @@system_variable is not a parameter.
func bind(query string) (string, []placeholder) {
positional := 0
sql := endtoend.Rewrite(query, func(name, _ string) string {
if name == "" {
positional++
name = "p" + strconv.Itoa(positional)
}
return "@" + name
})
var phs []placeholder
numbers := map[string]int{}
i := 0
for i < len(sql) {
c := sql[i]
switch {
case c == '\'' || c == '"' || c == '`':
i = skipQuoted(sql, i)
case strings.HasPrefix(sql[i:], "--") || strings.HasPrefix(sql[i:], "#"):
end := strings.IndexByte(sql[i:], '\n')
if end < 0 {
i = len(sql)
} else {
i += end
}
case strings.HasPrefix(sql[i:], "/*"):
end := strings.Index(sql[i:], "*/")
if end < 0 {
i = len(sql)
} else {
i += end + 2
}
case c == '@' && i+1 < len(sql) && sql[i+1] == '@':
i += 2
for i < len(sql) && (isWordByte(sql[i])) {
i++
}
case c == '@' && identRe.MatchString(sql[i:]):
m := identRe.FindStringSubmatch(sql[i:])
if _, ok := numbers[m[1]]; !ok {
numbers[m[1]] = len(phs) + 1
phs = append(phs, placeholder{Number: len(phs) + 1, Name: m[1]})
}
i += len(m[0])
default:
i++
}
}
return sql, phs
}
func isWordByte(c byte) bool {
return c == '_' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'
}
// skipQuoted returns the index just past the quoted token starting at i,
// honouring backslash escapes.
func skipQuoted(s string, i int) int {
q := s[i]
j := i + 1
for j < len(s) {
switch {
case s[j] == '\\' && j+1 < len(s):
j += 2
case s[j] == q:
return j + 1
default:
j++
}
}
return len(s)
}
// splitStatements splits a script on the semicolons outside strings and
// comments, which is how a schema's DDL is handed to CREATE DATABASE and a
// fixture's DML to a transaction.
func splitStatements(src string) []string {
var stmts []string
flush := func(s string) {
if s = strings.TrimSpace(s); s != "" {
stmts = append(stmts, s)
}
}
start, i := 0, 0
for i < len(src) {
c := src[i]
switch {
case c == '\'' || c == '"' || c == '`':
i = skipQuoted(src, i)
case strings.HasPrefix(src[i:], "--") || strings.HasPrefix(src[i:], "#"):
end := strings.IndexByte(src[i:], '\n')
if end < 0 {
i = len(src)
} else {
i += end
}
case strings.HasPrefix(src[i:], "/*"):
end := strings.Index(src[i:], "*/")
if end < 0 {
i = len(src)
} else {
i += end + 2
}
case c == ';':
flush(src[start:i])
start = i + 1
i++
default:
i++
}
}
flush(src[start:])
return stmts
}
// column is what INFORMATION_SCHEMA says about a column of the case's
// database: its type as the schema declared it, and whether it can be
// NULL.
type column struct {
name string
typ *analysis.TypeExpr
nullable bool
}
// analyzer holds the session a case is compiled in and the catalog of the
// case's database, keyed by table name in the database's own schema.
type analyzer struct {
s *server
session string
catalog map[string][]column
keys map[string][]string // the primary key columns of each table
}
var dbNameRe = regexp.MustCompile(`[^a-z0-9_]+`)
// Analyze creates a database of the case's own from its schema, writes its
// fixture there, compiles its queries and returns what Spanner reports in
// the JSON shape sqlc analyze prints.
func Analyze(ctx context.Context, endpoint string, c endtoend.Case) ([]byte, error) {
s, err := open(ctx, endpoint)
if err != nil {
return nil, err
}
defer s.Close()
schema, err := os.ReadFile(c.Schema)
if err != nil {
return nil, err
}
var fixture []byte
if c.Fixture != "" {
if fixture, err = os.ReadFile(c.Fixture); err != nil {
return nil, err
}
}
queries, err := c.Queries()
if err != nil {
return nil, err
}
// A database id is lower-case letters, digits and underscores, at
// most 30 characters: the case's name, cut to fit, and a hash of the
// whole of it, so that two cases with the same head stay apart.
hash := fmt.Sprintf("%08x", crc32.ChecksumIEEE([]byte(c.Name)))
name := "goldeneye_" + dbNameRe.ReplaceAllString(strings.ToLower(c.Name), "_")
if max := 30 - 1 - len(hash); len(name) > max {
name = name[:max]
}
name = strings.TrimRight(name, "_") + "_" + hash
db := Instance + "/databases/" + name
if err := s.dropDatabase(ctx, db); err != nil {
return nil, err
}
if _, err := s.createDatabase(ctx, name, splitStatements(string(schema))); err != nil {
return nil, fmt.Errorf("loading %s: %w", c.Schema, err)
}
defer s.dropDatabase(context.WithoutCancel(ctx), db)
session, err := s.session(ctx, db)
if err != nil {
return nil, err
}
if stmts := splitStatements(string(fixture)); len(stmts) > 0 {
if err := s.write(ctx, session, stmts); err != nil {
return nil, fmt.Errorf("loading %s: %w", c.Fixture, err)
}
}
a := &analyzer{s: s, session: session}
if err := a.readCatalog(ctx); err != nil {
return nil, err
}
out := make([]analysis.Query, 0, len(queries))
for _, q := range queries {
aq, err := a.analyzeQuery(ctx, q)
if err != nil {
return nil, fmt.Errorf("%s: %w", q.Name, err)
}
out = append(out, aq)
}
return analysis.Encode(out)
}
// Check compares what Spanner reports for a case with the output the
// case committed, returning a diff when they differ.
func Check(ctx context.Context, endpoint string, c endtoend.Case) (string, error) {
got, err := Analyze(ctx, endpoint, c)
if err != nil {
return "", err
}
return c.Compare(got)
}
// write runs DML statements in a read-write transaction and commits it.
func (s *server) write(ctx context.Context, session string, stmts []string) error {
tx, err := s.data.BeginTransaction(ctx, &spannerpb.BeginTransactionRequest{
Session: session,
Options: &spannerpb.TransactionOptions{Mode: &spannerpb.TransactionOptions_ReadWrite_{ReadWrite: &spannerpb.TransactionOptions_ReadWrite{}}},
})
if err != nil {
return err
}
req := &spannerpb.ExecuteBatchDmlRequest{
Session: session,
Transaction: &spannerpb.TransactionSelector{Selector: &spannerpb.TransactionSelector_Id{Id: tx.Id}},
Seqno: 1,
}
for _, stmt := range stmts {
req.Statements = append(req.Statements, &spannerpb.ExecuteBatchDmlRequest_Statement{Sql: stmt})
}
resp, err := s.data.ExecuteBatchDml(ctx, req)
if err == nil && resp.Status != nil && resp.Status.Code != 0 {
err = fmt.Errorf("%s", resp.Status.Message)
}
if err != nil {
s.data.Rollback(context.WithoutCancel(ctx), &spannerpb.RollbackRequest{Session: session, TransactionId: tx.Id})
return err
}
_, err = s.data.Commit(ctx, &spannerpb.CommitRequest{
Session: session,
Transaction: &spannerpb.CommitRequest_TransactionId{TransactionId: tx.Id},
})
return err
}
const catalogQuery = `
SELECT TABLE_NAME, COLUMN_NAME, SPANNER_TYPE, IS_NULLABLE
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = ''
ORDER BY TABLE_NAME, ORDINAL_POSITION`
const keysQuery = `
SELECT TABLE_NAME, COLUMN_NAME
FROM INFORMATION_SCHEMA.INDEX_COLUMNS
WHERE TABLE_SCHEMA = '' AND INDEX_TYPE = 'PRIMARY_KEY'
ORDER BY TABLE_NAME, ORDINAL_POSITION`
// readCatalog reads the tables the schema created, and their keys.
func (a *analyzer) readCatalog(ctx context.Context) error {
rows, err := a.s.query(ctx, a.session, catalogQuery)
if err != nil {
return err
}
a.catalog = map[string][]column{}
for _, row := range rows {
table := row[0].GetStringValue()
a.catalog[table] = append(a.catalog[table], column{
name: row[1].GetStringValue(),
typ: parseType(row[2].GetStringValue()),
nullable: row[3].GetStringValue() == "YES",
})
}
rows, err = a.s.query(ctx, a.session, keysQuery)
if err != nil {
return err
}
a.keys = map[string][]string{}
for _, row := range rows {
table := row[0].GetStringValue()
a.keys[table] = append(a.keys[table], row[1].GetStringValue())
}
return nil
}
// tableName returns the catalog's spelling of a table's name, which
// Spanner matches in any case.
func (a *analyzer) tableName(table string) string {
for t := range a.catalog {
if strings.EqualFold(t, table) {
return t
}
}
return table
}
// lookup finds a column of a table of the case's database. Spanner
// matches table and column names in any case.
func (a *analyzer) lookup(table, name string) (column, bool) {
for t, cols := range a.catalog {
if !strings.EqualFold(t, table) {
continue
}
for _, col := range cols {
if strings.EqualFold(col.name, name) {
return col, true
}
}
}
return column{}, false
}
// parseType reads a type spelled the way SPANNER_TYPE and a declaration
// spell it — STRING(10), ARRAY<STRING(MAX)>, STRUCT<a INT64, b STRING> —
// into an expression, in lower case, with MAX an identifier.
func parseType(s string) *analysis.TypeExpr {
s = strings.TrimSpace(s)
lower := strings.ToLower(s)
if element, ok := strings.CutPrefix(lower, "array<"); ok && strings.HasSuffix(element, ">") {
return &analysis.TypeExpr{Name: "array", Args: []analysis.TypeArg{{Type: parseType(s[6 : len(s)-1])}}}
}
for _, kind := range []string{"proto", "enum"} {
if message, ok := strings.CutPrefix(lower, kind+"<"); ok && strings.HasSuffix(message, ">") {
m := s[len(kind)+1 : len(s)-1]
return &analysis.TypeExpr{Name: kind, Args: []analysis.TypeArg{{String: &m}}}
}
}
if fields, ok := strings.CutPrefix(lower, "struct<"); ok && strings.HasSuffix(fields, ">") {
t := &analysis.TypeExpr{Name: "struct"}
for _, f := range splitTop(s[7:len(s)-1], ',') {
f = strings.TrimSpace(f)
label, typ := "", f
if i := strings.IndexAny(f, " \t"); i > 0 && !strings.ContainsAny(f[:i], "<(") {
label, typ = f[:i], strings.TrimSpace(f[i+1:])
}
t.Args = append(t.Args, analysis.TypeArg{Label: label, Type: parseType(typ)})
}
return t
}
name, args := lower, ""
if open := strings.IndexByte(lower, '('); open >= 0 && strings.HasSuffix(lower, ")") {
name, args = strings.TrimSpace(lower[:open]), lower[open+1:len(lower)-1]
}
t := &analysis.TypeExpr{Name: name}
if args == "" {
return t
}
for _, arg := range strings.Split(args, ",") {
arg = strings.TrimSpace(arg)
if n, err := strconv.ParseInt(arg, 10, 64); err == nil {
t.Args = append(t.Args, analysis.TypeArg{Int: &n})
} else {
ident := arg
t.Args = append(t.Args, analysis.TypeArg{Ident: &ident})
}
}
return t
}
// splitTop splits on a separator outside angle brackets and parentheses.
func splitTop(s string, sep byte) []string {
var out []string
depth, start := 0, 0
for i := 0; i < len(s); i++ {
switch s[i] {
case '<', '(':
depth++
case '>', ')':
depth--
case sep:
if depth == 0 {
out = append(out, s[start:i])
start = i + 1
}
}
}
return append(out, s[start:])
}
// typeOf reads a type the way the wire spells it, which names the family
// and, for an array or a struct, what it holds, and nothing of a length.
func typeOf(t *spannerpb.Type) *analysis.TypeExpr {
if t == nil {
return nil
}
switch t.Code {
case spannerpb.TypeCode_ARRAY:
return &analysis.TypeExpr{Name: "array", Args: []analysis.TypeArg{{Type: typeOf(t.ArrayElementType)}}}
case spannerpb.TypeCode_STRUCT:
out := &analysis.TypeExpr{Name: "struct"}
if t.StructType != nil {
for _, f := range t.StructType.Fields {
out.Args = append(out.Args, analysis.TypeArg{Label: f.Name, Type: typeOf(f.Type)})
}
}
return out
case spannerpb.TypeCode_PROTO, spannerpb.TypeCode_ENUM:
// A proto or enum is named by its message, the way a declaration
// spells PROTO<p.M>.
fqn := t.ProtoTypeFqn
return &analysis.TypeExpr{Name: strings.ToLower(t.Code.String()), Args: []analysis.TypeArg{{String: &fqn}}}
}
return &analysis.TypeExpr{Name: strings.ToLower(t.Code.String())}
}
// withNullable copies a type with its nullability set.
func withNullable(t *analysis.TypeExpr, nullable bool) *analysis.TypeExpr {
if t == nil {
return nil
}
out := *t
out.Nullable = nullable
return &out
}
// isDML reports whether a statement writes, and so has to be compiled in a
// read-write transaction, which is begun for it and never committed. The
// statement's first word decides, past any comment it opens with.
func isDML(sql string) bool {
i := 0
for i < len(sql) {
switch {
case sql[i] == ' ' || sql[i] == '\t' || sql[i] == '\n' || sql[i] == '\r':
i++
case strings.HasPrefix(sql[i:], "--") || strings.HasPrefix(sql[i:], "#"):
end := strings.IndexByte(sql[i:], '\n')
if end < 0 {
return false
}
i += end
case strings.HasPrefix(sql[i:], "/*"):
end := strings.Index(sql[i:], "*/")
if end < 0 {
return false
}
i += end + 2
default:
head := strings.ToLower(sql[i:])
for _, kw := range []string{"insert", "update", "delete"} {
if strings.HasPrefix(head, kw) {
return true
}
}
return false
}
}
return false
}
// compile compiles a statement in PLAN mode and returns what the server
// reports about it.
func (a *analyzer) compile(ctx context.Context, sql string) (*spannerpb.ResultSet, error) {
req := &spannerpb.ExecuteSqlRequest{
Session: a.session,
Sql: sql,
QueryMode: spannerpb.ExecuteSqlRequest_PLAN,
}
if isDML(sql) {
req.Transaction = &spannerpb.TransactionSelector{Selector: &spannerpb.TransactionSelector_Begin{
Begin: &spannerpb.TransactionOptions{Mode: &spannerpb.TransactionOptions_ReadWrite_{ReadWrite: &spannerpb.TransactionOptions_ReadWrite{}}},
}}
req.Seqno = 1
}
rs, err := a.s.data.ExecuteSql(ctx, req)
if err != nil {
return nil, err
}
if rs.Metadata != nil && rs.Metadata.Transaction != nil && len(rs.Metadata.Transaction.Id) > 0 {
a.s.data.Rollback(context.WithoutCancel(ctx), &spannerpb.RollbackRequest{Session: a.session, TransactionId: rs.Metadata.Transaction.Id})
}
return rs, nil
}
// analyzeQuery compiles one query.
func (a *analyzer) analyzeQuery(ctx context.Context, q endtoend.Query) (analysis.Query, error) {
sql, phs := bind(q.SQL)
aq := analysis.Query{
Name: q.Name,
Cmd: q.Cmd,
Columns: []analysis.Column{},
Params: []analysis.Param{},
}
rs, err := a.compile(ctx, sql)
if err != nil {
return analysis.Query{}, err
}
p := newPlan(rs.Stats.GetQueryPlan())
partners := p.partners()
describe := func(o origin) (analysis.Column, bool) {
if o.param != "" {
o = partners[o.param]
}
if o.table == "" {
return analysis.Column{}, false
}
col, ok := a.lookup(o.table, o.column)
if !ok {
return analysis.Column{}, false
}
return analysis.Column{Name: col.name, Type: withNullable(col.typ, col.nullable), Table: a.tableName(o.table)}, true
}
var fields []*spannerpb.StructType_Field
if rs.Metadata != nil && rs.Metadata.RowType != nil {
fields = rs.Metadata.RowType.Fields
}
outputs := p.outputs()
table, operation := p.mutation()
if table != "" {
// A DML plan lists the columns it returns, then the values it
// writes: each of those is a parameter standing for the column
// it is written to.
written := a.written(sql, table, operation)
writes := outputs[min(len(fields), len(outputs)):]
for i, w := range written {
if i >= len(writes) {
break
}
if o := p.resolve(writes[i], map[int32]bool{}); o.param != "" {
if _, ok := partners[o.param]; !ok {
partners[o.param] = origin{table: table, column: w}
}
}
}
outputs = outputs[:min(len(fields), len(outputs))]
}
for i, f := range fields {
ac := analysis.Column{Name: f.Name, Type: typeOf(f.Type)}
var out *spannerpb.PlanNode
if i < len(outputs) {
out = outputs[i]
}
if col, ok := describe(p.resolve(out, map[int32]bool{})); ok {
ac.Type, ac.Table = col.Type, col.Table
} else if table != "" && (out == nil || out.DisplayName != "Function") {
// A THEN RETURN column the plan reads as a constant, such
// as the default a column not inserted takes, is the table's
// column of that name.
if col, ok := a.lookup(table, f.Name); ok {
ac.Type = withNullable(col.typ, col.nullable)
ac.Table = a.tableName(table)
}
}
aq.Columns = append(aq.Columns, ac)
}
declared := map[string]*spannerpb.Type{}
if rs.Metadata != nil && rs.Metadata.UndeclaredParameters != nil {
for _, f := range rs.Metadata.UndeclaredParameters.Fields {
declared[f.Name] = f.Type
}
}
for _, ph := range phs {
ac, ok := describe(origin{param: ph.Name})
if !ok {
ac = analysis.Column{Type: typeOf(declared[ph.Name])}
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}
var (
insertRe = regexp.MustCompile("(?is)^insert\\s+(?:or\\s+\\w+\\s+)?into\\s+[\\w.`]+\\s*(?:\\(([^)]*)\\))?")
setRe = regexp.MustCompile("(?is)^update\\s+[\\w.`]+(?:\\s+(?:as\\s+)?\\w+)?\\s+set\\s+")
)
// written lists the columns a DML statement writes, in the order the plan
// lists their values: for an INSERT the columns it inserts, which are the
// statement's column list or every column of the table, and for an
// UPDATE or DELETE the table's key columns, then the columns an UPDATE
// sets.
func (a *analyzer) written(sql, table, operation string) []string {
switch operation {
case "INSERT":
m := insertRe.FindStringSubmatch(sql)
if m == nil {
return nil
}
if strings.TrimSpace(m[1]) == "" {
var cols []string
for _, col := range a.catalog[a.tableName(table)] {
cols = append(cols, col.name)
}
return cols
}
var cols []string
for _, c := range strings.Split(m[1], ",") {
cols = append(cols, strings.Trim(strings.TrimSpace(c), "`"))
}
return cols
case "UPDATE":
cols := append([]string(nil), a.keys[a.tableName(table)]...)
if m := setRe.FindStringIndex(sql); m != nil {
for _, assignment := range setList(sql[m[1]:]) {
target, _, _ := strings.Cut(assignment, "=")
target = strings.TrimSpace(target)
if i := strings.LastIndexByte(target, '.'); i >= 0 {
target = target[i+1:]
}
cols = append(cols, strings.Trim(target, "`"))
}
}
return cols
case "DELETE":
return a.keys[a.tableName(table)]
}
return nil
}
// setList splits an UPDATE's SET list into its assignments: the text up
// to the WHERE outside strings and parentheses, split on the commas
// outside them.
func setList(s string) []string {
var out []string
depth, start := 0, 0
for i := 0; i < len(s); i++ {
c := s[i]
switch {
case c == '\'' || c == '"' || c == '`':
i = skipQuoted(s, i) - 1
case c == '(':
depth++
case c == ')':
depth--
case c == ',' && depth == 0:
out = append(out, s[start:i])
start = i + 1
case depth == 0 && (i == 0 || !isWordByte(s[i-1])) && len(s)-i >= 5 && strings.EqualFold(s[i:i+5], "where") && (len(s) == i+5 || !isWordByte(s[i+5])):
return append(out, s[start:i])
}
}
return append(out, s[start:])
}

220
internal/goldeneye/spanner/plan.go Normal file
View File

@@ -0,0 +1,220 @@
package spanner
import (
"regexp"
"strings"
"cloud.google.com/go/spanner/apiv1/spannerpb"
)
// The query plan is the one thing that says where a result column comes
// from and what a parameter stands for. It is a list of nodes: relational
// ones, such as a Scan of a table or the Serialize Result that produces the
// rows, and scalar ones, such as a Reference to a column or a variable, a
// Parameter, a Function. A relational node defines variables through its
// child links: a Scan of a table defines one per column it reads, named
// after the column, and any node may define one as another scalar, which
// a Reference names with a $. The children of Serialize Result after the
// relation it serializes are the result columns, in order; a DML plan
// lists its THEN RETURN columns first and the values it writes after
// them: for an INSERT the columns it inserts, in the statement's order,
// and for an UPDATE or DELETE the key columns of the table, then the
// columns an UPDATE sets. A comparison is a Function whose description
// reads ($col = @param).
// origin is what a scalar of the plan resolves to: a table column, a
// parameter, or nothing.
type origin struct {
table, column string
param string
}
// plan is a parsed query plan.
type plan struct {
nodes []*spannerpb.PlanNode
// vars maps a variable to the node that defines it and the scalar it
// is defined as.
vars map[string]definition
}
type definition struct {
owner *spannerpb.PlanNode
child *spannerpb.PlanNode
}
func newPlan(qp *spannerpb.QueryPlan) *plan {
p := &plan{vars: map[string]definition{}}
if qp == nil {
return p
}
p.nodes = qp.PlanNodes
for _, n := range p.nodes {
for _, l := range n.ChildLinks {
if l.Variable != "" {
p.vars[l.Variable] = definition{owner: n, child: p.node(l.ChildIndex)}
}
}
}
return p
}
func (p *plan) node(index int32) *spannerpb.PlanNode {
for _, n := range p.nodes {
if n.Index == index {
return n
}
}
return nil
}
func (p *plan) root() *spannerpb.PlanNode {
return p.node(0)
}
func meta(n *spannerpb.PlanNode, key string) string {
if n == nil || n.Metadata == nil {
return ""
}
if v, ok := n.Metadata.Fields[key]; ok {
return v.GetStringValue()
}
return ""
}
func description(n *spannerpb.PlanNode) string {
if n == nil || n.ShortRepresentation == nil {
return ""
}
return n.ShortRepresentation.Description
}
// serializeResult finds the node that produces the rows, the first
// Serialize Result reached from the root.
func (p *plan) serializeResult() *spannerpb.PlanNode {
var found *spannerpb.PlanNode
seen := map[int32]bool{}
var walk func(n *spannerpb.PlanNode)
walk = func(n *spannerpb.PlanNode) {
if n == nil || found != nil || seen[n.Index] {
return
}
seen[n.Index] = true
if n.DisplayName == "Serialize Result" {
found = n
return
}
for _, l := range n.ChildLinks {
walk(p.node(l.ChildIndex))
}
}
walk(p.root())
return found
}
// outputs lists the scalars Serialize Result produces, in order: every
// child after the relation it serializes.
func (p *plan) outputs() []*spannerpb.PlanNode {
sr := p.serializeResult()
if sr == nil {
return nil
}
var out []*spannerpb.PlanNode
for _, l := range sr.ChildLinks {
if c := p.node(l.ChildIndex); c != nil && c.Kind == spannerpb.PlanNode_SCALAR {
out = append(out, c)
}
}
return out
}
// mutation names the table a DML plan writes, or "" for a query.
func (p *plan) mutation() (table, operation string) {
for _, n := range p.nodes {
if n.DisplayName == "Apply Mutations" {
return meta(n, "table"), meta(n, "operation_type")
}
}
return "", ""
}
// resolve follows a scalar to what it stands for.
func (p *plan) resolve(n *spannerpb.PlanNode, seen map[int32]bool) origin {
if n == nil || seen[n.Index] {
return origin{}
}
seen[n.Index] = true
switch n.DisplayName {
case "Parameter":
return origin{param: meta(n, "name")}
case "Reference":
desc := description(n)
if name, ok := strings.CutPrefix(desc, "$"); ok {
return p.resolveVar(name, seen)
}
// A bare name is a column of the scan it is read by, or an input
// of a union, which names it in the link's type.
for _, m := range p.nodes {
for _, l := range m.ChildLinks {
if l.ChildIndex != n.Index {
continue
}
switch {
case m.DisplayName == "Scan" && meta(m, "scan_type") == "TableScan":
return origin{table: meta(m, "scan_target"), column: desc}
case m.DisplayName == "Scan":
// A batch scan reads the batch variable of that name.
target := strings.TrimPrefix(meta(m, "scan_target"), "$")
return p.resolveVar(target+".Batch."+desc, seen)
}
}
}
for _, m := range p.nodes {
if m.DisplayName != "Union Input" {
continue
}
for _, l := range m.ChildLinks {
if l.Type == desc {
return p.resolve(p.node(l.ChildIndex), seen)
}
}
}
}
return origin{}
}
// resolveVar follows a variable to what it is defined as.
func (p *plan) resolveVar(name string, seen map[int32]bool) origin {
d, ok := p.vars[name]
if !ok {
return origin{}
}
return p.resolve(d.child, seen)
}
var compareRe = regexp.MustCompile(`^\((.+) (=|!=|<>|<|<=|>|>=) (.+)\)$`)
// partners finds the table column each parameter is compared with, keyed
// by parameter name: the other operand of a comparison Function the
// parameter is an operand of. The first found wins.
func (p *plan) partners() map[string]origin {
out := map[string]origin{}
for _, n := range p.nodes {
if n.DisplayName != "Function" || len(n.ChildLinks) != 2 || !compareRe.MatchString(description(n)) {
continue
}
sides := []*spannerpb.PlanNode{p.node(n.ChildLinks[0].ChildIndex), p.node(n.ChildLinks[1].ChildIndex)}
for i, side := range sides {
o := p.resolve(side, map[int32]bool{})
if o.param == "" {
continue
}
other := p.resolve(sides[1-i], map[int32]bool{})
if other.table != "" {
if _, ok := out[o.param]; !ok {
out[o.param] = other
}
}
}
}
return out
}

96
internal/goldeneye/spanner/relations.go Normal file
View File

@@ -0,0 +1,96 @@
package spanner
import (
"context"
"fmt"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// relationQuery lists the relations of every schema but the database's
// own, unnamed one — INFORMATION_SCHEMA and SPANNER_SYS — with their
// columns in declared order, so that the output is stable. A column's
// type is spelled the way SPANNER_TYPE spells a declaration: STRING(MAX),
// ARRAY<INT64>.
const relationQuery = `
SELECT t.TABLE_SCHEMA, t.TABLE_NAME, t.TABLE_TYPE,
c.COLUMN_NAME, c.SPANNER_TYPE, c.IS_NULLABLE
FROM INFORMATION_SCHEMA.TABLES AS t
JOIN INFORMATION_SCHEMA.COLUMNS AS c
ON c.TABLE_SCHEMA = t.TABLE_SCHEMA AND c.TABLE_NAME = t.TABLE_NAME
WHERE t.TABLE_SCHEMA != ''
ORDER BY t.TABLE_SCHEMA, t.TABLE_NAME, c.ORDINAL_POSITION`
// readRelations reads the views of the system schemas. Their names are
// kept as the catalog spells them, in upper case, which is how a query
// names them; a column's type is spelled in lower case, the way the
// dialect's types.jsonl spells its types, with an ARRAY<T> written as T
// with the array flag, since that is how a seed spells an array.
func readRelations(ctx context.Context, s *server, session string) ([]dialect.Relation, error) {
rows, err := s.query(ctx, session, relationQuery)
if err != nil {
return nil, err
}
var relations []dialect.Relation
var cur *dialect.Relation
for _, row := range rows {
if len(row) != 6 {
return nil, fmt.Errorf("expected 6 columns, got %d", len(row))
}
schema, name, kind := row[0].GetStringValue(), row[1].GetStringValue(), row[2].GetStringValue()
if cur == nil || cur.Schema != schema || cur.Name != name {
rel := dialect.Relation{Schema: schema, Name: name}
if kind == "VIEW" {
rel.Kind = "v"
}
relations = append(relations, rel)
cur = &relations[len(relations)-1]
}
col := dialect.Column{
Name: row[3].GetStringValue(),
NotNull: row[5].GetStringValue() == "NO",
}
col.Type, col.Array = typeName(row[4].GetStringValue())
cur.Columns = append(cur.Columns, col)
}
return relations, nil
}
// typeName spells a SPANNER_TYPE the way a seed spells a column's type:
// in lower case, with an ARRAY<T> as its element and the array flag, a
// STRUCT<a T, b U> as struct(a: t, b: u) and a PROTO<p.M> or ENUM<p.E> as
// proto('p.M') or enum('p.E'), since a seed spells a type's arguments in
// parentheses.
func typeName(spannerType string) (string, bool) {
t := strings.TrimSpace(spannerType)
if element, ok := strings.CutPrefix(t, "ARRAY<"); ok && strings.HasSuffix(element, ">") {
name, _ := typeName(strings.TrimSuffix(element, ">"))
return name, true
}
if fields, ok := strings.CutPrefix(t, "STRUCT<"); ok && strings.HasSuffix(fields, ">") {
var args []string
for _, f := range splitTop(strings.TrimSuffix(fields, ">"), ',') {
f = strings.TrimSpace(f)
label, typ := "", f
if i := strings.IndexAny(f, " \t"); i > 0 && !strings.ContainsAny(f[:i], "<(") {
label, typ = f[:i], strings.TrimSpace(f[i+1:])
}
name, array := typeName(typ)
if array {
name += "[]"
}
if label != "" {
name = label + ": " + name
}
args = append(args, name)
}
return "struct(" + strings.Join(args, ", ") + ")", false
}
for _, kind := range []string{"PROTO<", "ENUM<"} {
if message, ok := strings.CutPrefix(t, kind); ok && strings.HasSuffix(message, ">") {
return strings.ToLower(strings.TrimSuffix(kind, "<")) + "('" + strings.TrimSuffix(message, ">") + "')", false
}
}
return strings.ToLower(t), false
}

224
internal/goldeneye/spanner/spanner.go Normal file
View File

@@ -0,0 +1,224 @@
// Package spanner generates the GoogleSQL dialect seed under
// internal/engine/googlesql/dialect from a live Spanner server, and
// verifies the GoogleSQL analyze cases under internal/endtoend/testdata
// against the same server. The server is Spanner Omni, the downloadable
// Spanner, run from its container image; the package writes into the
// googlesql directory, since sqlc's engine is named after the language
// Spanner speaks rather than the database.
//
// Spanner keeps no catalog of its types, functions or operators, so
// types.jsonl, functions.jsonl and operators.jsonl are written by hand and
// are not this package's business. What it does describe is its
// information schema: relations.jsonl is every view of INFORMATION_SCHEMA
// and SPANNER_SYS, read from INFORMATION_SCHEMA itself.
//
// The package also verifies the GoogleSQL analyze cases against the same
// server: each case's schema becomes a database of its own, its fixture is
// written there, and each query is compiled in PLAN mode, which runs
// nothing and reports the type of each result column and of each
// parameter the query leaves undeclared, and the query plan, which says
// which table column each result column and parameter stands for. The
// answer is printed in the JSON shape sqlc analyze prints and compared
// with the case's committed stdout.json byte for byte.
//
// The server is named by SPANNER_SERVER_URI, the gRPC endpoint of a
// Spanner Omni server such as localhost:15000, which is reached without
// TLS or credentials, as Omni is. Databases are created in the instance
// Omni's single server provides, projects/default/instances/default.
package spanner
import (
"context"
"errors"
"fmt"
"os"
"strings"
database "cloud.google.com/go/spanner/admin/database/apiv1"
"cloud.google.com/go/spanner/admin/database/apiv1/databasepb"
instance "cloud.google.com/go/spanner/admin/instance/apiv1"
"cloud.google.com/go/spanner/admin/instance/apiv1/instancepb"
spannerapi "cloud.google.com/go/spanner/apiv1"
"cloud.google.com/go/spanner/apiv1/spannerpb"
"google.golang.org/api/option"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/structpb"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name goldeneye knows the database by.
const Engine = "spanner"
// Dir is the name of the engine directory the dialect lives under, and
// Cases the name of the analyze case directories under
// internal/endtoend/testdata: both are named after the language, GoogleSQL,
// which is how sqlc knows the dialect.
const (
Dir = "googlesql"
Cases = "googlesql"
)
// Instance is the instance a Spanner Omni single server provides, which
// every database is created in.
const Instance = "projects/default/instances/default"
// Locate returns the server to generate from, named by SPANNER_SERVER_URI.
func Locate() (string, error) {
endpoint := os.Getenv("SPANNER_SERVER_URI")
if endpoint == "" {
return "", errors.New("SPANNER_SERVER_URI is not set")
}
if strings.Contains(endpoint, "://") {
return "", fmt.Errorf("SPANNER_SERVER_URI: %q should be a host:port, the gRPC endpoint of a Spanner Omni server", endpoint)
}
return endpoint, nil
}
// server holds the clients a run talks to the server through.
type server struct {
endpoint string
instances *instance.InstanceAdminClient
databases *database.DatabaseAdminClient
data *spannerapi.Client
}
// open connects to the server. Omni serves plaintext gRPC and asks for no
// credentials.
func open(ctx context.Context, endpoint string) (*server, error) {
opts := []option.ClientOption{
option.WithEndpoint(endpoint),
option.WithoutAuthentication(),
option.WithGRPCDialOption(grpc.WithTransportCredentials(insecure.NewCredentials())),
}
s := &server{endpoint: endpoint}
var err error
if s.instances, err = instance.NewInstanceAdminClient(ctx, opts...); err != nil {
return nil, err
}
if s.databases, err = database.NewDatabaseAdminClient(ctx, opts...); err != nil {
s.Close()
return nil, err
}
if s.data, err = spannerapi.NewClient(ctx, opts...); err != nil {
s.Close()
return nil, err
}
return s, nil
}
func (s *server) Close() {
if s.instances != nil {
s.instances.Close()
}
if s.databases != nil {
s.databases.Close()
}
if s.data != nil {
s.data.Close()
}
}
// Version describes the server: Spanner Omni reports no release of its
// own through the API, so the instance the databases are created in is
// described instead.
func Version(ctx context.Context, endpoint string) (string, error) {
s, err := open(ctx, endpoint)
if err != nil {
return "", err
}
defer s.Close()
inst, err := s.instances.GetInstance(ctx, &instancepb.GetInstanceRequest{Name: Instance})
if err != nil {
return "", fmt.Errorf("%s: %w", Instance, err)
}
return fmt.Sprintf("Spanner Omni at %s, instance %s (%s)", endpoint, inst.Name, inst.State), nil
}
// createDatabase creates a database in the instance, running the DDL
// statements in it, and returns its full name.
func (s *server) createDatabase(ctx context.Context, name string, ddl []string) (string, error) {
op, err := s.databases.CreateDatabase(ctx, &databasepb.CreateDatabaseRequest{
Parent: Instance,
CreateStatement: "CREATE DATABASE `" + name + "`",
ExtraStatements: ddl,
})
if err != nil {
return "", err
}
db, err := op.Wait(ctx)
if err != nil {
return "", err
}
return db.Name, nil
}
// dropDatabase drops a database by its full name. A database that does
// not exist is not an error, so that a run can clear the way for itself.
func (s *server) dropDatabase(ctx context.Context, db string) error {
err := s.databases.DropDatabase(ctx, &databasepb.DropDatabaseRequest{Database: db})
if status.Code(err) == codes.NotFound && strings.Contains(err.Error(), "Database") {
return nil
}
return err
}
// session opens a session on a database, multiplexed as Omni requires.
func (s *server) session(ctx context.Context, db string) (string, error) {
sess, err := s.data.CreateSession(ctx, &spannerpb.CreateSessionRequest{
Database: db,
Session: &spannerpb.Session{Multiplexed: true},
})
if err != nil {
return "", err
}
return sess.Name, nil
}
// query runs a statement and returns its rows as lists of values.
func (s *server) query(ctx context.Context, session, sql string) ([][]*structpb.Value, error) {
rs, err := s.data.ExecuteSql(ctx, &spannerpb.ExecuteSqlRequest{Session: session, Sql: sql})
if err != nil {
return nil, err
}
var rows [][]*structpb.Value
for _, row := range rs.Rows {
rows = append(rows, row.Values)
}
return rows, nil
}
// Generate reads the dialect from the server: relations.jsonl, the views of
// INFORMATION_SCHEMA and SPANNER_SYS, read from a database of their own,
// since every database has the same ones.
func Generate(ctx context.Context, endpoint string) (dialect.Files, error) {
s, err := open(ctx, endpoint)
if err != nil {
return nil, err
}
defer s.Close()
db := Instance + "/databases/goldeneye_dialect"
if err := s.dropDatabase(ctx, db); err != nil {
return nil, err
}
if _, err := s.createDatabase(ctx, "goldeneye_dialect", nil); err != nil {
return nil, err
}
defer s.dropDatabase(context.WithoutCancel(ctx), db)
session, err := s.session(ctx, db)
if err != nil {
return nil, err
}
relations, err := readRelations(ctx, s, session)
if err != nil {
return nil, err
}
blob, err := dialect.JSONL(relations)
if err != nil {
return nil, err
}
return dialect.Files{dialect.RelationsFile: blob}, nil
}

66
internal/goldeneye/spanner/spanner_test.go Normal file
View File

@@ -0,0 +1,66 @@
package spanner
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// TestDialect verifies the committed GoogleSQL dialect against what a
// Spanner server reports. It skips unless SPANNER_SERVER_URI names one.
func TestDialect(t *testing.T) {
endpoint, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, endpoint)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, endpoint)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Dir)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}
// TestAnalyzeCases verifies every GoogleSQL analyze case under
// internal/endtoend/testdata against what a Spanner server reports. It
// skips unless SPANNER_SERVER_URI names one.
func TestAnalyzeCases(t *testing.T) {
endpoint, err := Locate()
if err != nil {
t.Skip(err)
}
cases, err := endtoend.Cases(Cases)
if err != nil {
t.Fatal(err)
}
if len(cases) == 0 {
t.Fatal("no googlesql analyze cases found")
}
for _, c := range cases {
t.Run(c.Name, func(t *testing.T) {
diff, err := Check(context.Background(), endpoint, c)
if err != nil {
t.Fatal(err)
}
if diff != "" {
t.Errorf("%s does not match what Spanner reports (-committed +spanner):\n%s", c.Output, diff)
}
})
}
}

269
internal/goldeneye/sqlite/analyze.go Normal file
View File

@@ -0,0 +1,269 @@
package sqlite
import (
"context"
"errors"
"fmt"
"os"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// The analyze cases are checked against the analysis shell, which is asked
// three things about each query. What the library reports about the
// prepared statement, through `.stats stmt`: each result column's name and,
// for one read straight from a table, its declared type and which table
// column it is, with the column's NOT NULL looked up in the catalog. What
// the query returns, over the case's fixture and again over no rows, for
// the columns the library has no answer for: an expression's type is the
// storage class of its value, since SQLite types values rather than
// expressions, and any column is nullable when a NULL comes back — an
// aggregate over no rows, the far side of an outer join. And the bytecode
// the statement compiles to, which is where the parameters are found,
// since the library reports nothing about them but their number.
// placeholder is one parameter of a query as sqlc numbers them: each ? in
// turn, and each sqlc.arg name once, at its first appearance. SQLite
// numbers ?NNN the same way, so the query is rewritten with those.
type placeholder struct {
Number int
Name string
}
func bind(sql string) (string, []placeholder) {
var phs []placeholder
numbers := map[string]int{}
out := endtoend.Rewrite(sql, func(name, _ string) string {
n, ok := numbers[name]
if name == "" || !ok {
n = len(phs) + 1
phs = append(phs, placeholder{Number: n, Name: name})
if name != "" {
numbers[name] = n
}
}
return fmt.Sprintf("?%d", n)
})
return out, phs
}
// Analyze runs a case's queries through the analysis shell and returns what
// SQLite reports in the JSON shape sqlc analyze prints.
func Analyze(ctx context.Context, dir string, c endtoend.Case) ([]byte, error) {
if err := checkOptions(ctx, dir, analysisShell); err != nil {
return nil, err
}
binary := analysisShell.binary(dir)
schema, err := os.ReadFile(c.Schema)
if err != nil {
return nil, err
}
var fixture []byte
if c.Fixture != "" {
if fixture, err = os.ReadFile(c.Fixture); err != nil {
return nil, err
}
}
queries, err := c.Queries()
if err != nil {
return nil, err
}
out := make([]analysis.Query, 0, len(queries))
for _, q := range queries {
aq, err := analyzeQuery(ctx, binary, string(schema), string(fixture), q)
if err != nil {
return nil, fmt.Errorf("%s: %w", q.Name, err)
}
out = append(out, aq)
}
return analysis.Encode(out)
}
// Check compares what SQLite reports for a case with the output the case
// committed, returning a diff when they differ.
func Check(ctx context.Context, dir string, c endtoend.Case) (string, error) {
got, err := Analyze(ctx, dir, c)
if err != nil {
return "", err
}
return c.Compare(got)
}
func analyzeQuery(ctx context.Context, binary, schema, fixture string, q endtoend.Query) (analysis.Query, error) {
sql, phs := bind(q.SQL)
// What the library says: the catalog, the bytecode, and the statement
// prepared, with nothing bound to its parameters.
s := newScript()
s.sql(schema)
s.sql(fixture)
s.add(".mode json")
s.add(".explain off")
for _, cq := range catalogQueries {
s.section(cq.section)
s.sql(cq.sql)
}
s.section("explain")
s.sql("EXPLAIN " + sql)
s.section("query")
s.add(".stats stmt")
line := s.sql(sql)
s.section("end")
out, err := run(ctx, binary, s)
if err != nil {
return analysis.Query{}, err
}
if errs := out.errorsBefore(line); len(errs) > 0 {
return analysis.Query{}, errors.New(strings.Join(errs, "\n"))
}
query := out.sections["query"]
if query == nil || !query.prepared {
msg := strings.TrimSpace(out.stderr)
if msg == "" {
msg = "the statement was not prepared"
}
return analysis.Query{}, errors.New(msg)
}
cat, err := readCatalog(out)
if err != nil {
return analysis.Query{}, err
}
var prog []instr
if explain := out.sections["explain"]; explain != nil && len(explain.blocks) > 0 {
if err := explain.decode(0, &prog); err != nil {
return analysis.Query{}, fmt.Errorf("reading the bytecode: %w", err)
}
}
var names []string
for _, m := range query.columns {
names = append(names, m.Name)
}
t := newTracer(cat, names)
t.run(prog)
params := make([]analysis.Column, len(phs))
for i, ph := range phs {
params[i] = t.param(ph.Number)
}
// What the statement returns: over the fixture, with each parameter
// bound to a value of the column it stands in for, so that the rows
// come through the WHERE clause, and over no rows.
s = newScript()
s.sql(schema)
s.sql(fixture)
s.add(".parameter init")
for i, ph := range phs {
if v := sample(params[i]); v != "" {
s.sql(fmt.Sprintf("REPLACE INTO temp.sqlite_parameters(key, value) VALUES ('?%d', %s)", ph.Number, v))
}
}
s.add(".mode quote")
s.section("query")
s.sql(sql)
s.section("end")
bound, err := run(ctx, binary, s)
if err != nil {
return analysis.Query{}, err
}
s = newScript()
s.sql(schema)
s.add(".mode quote")
s.section("query")
s.sql(sql)
s.section("end")
empty, err := run(ctx, binary, s)
if err != nil {
return analysis.Query{}, err
}
var rows []string
for _, o := range []*output{bound, empty} {
if q := o.sections["query"]; q != nil {
rows = append(rows, q.rows...)
}
}
aq := analysis.Query{
Name: q.Name,
Cmd: q.Cmd,
Columns: []analysis.Column{},
Params: []analysis.Param{},
}
for i, m := range query.columns {
aq.Columns = append(aq.Columns, describeColumn(cat, m, classes(rows, i)))
}
for i, ph := range phs {
ac := params[i]
if ph.Name != "" {
ac.Name = ph.Name
}
aq.Params = append(aq.Params, analysis.Param{Number: ph.Number, Column: ac})
}
return aq, nil
}
// sample is an expression for a value to bind to a parameter: one of its
// column's values in the fixture, or a value of its type when all that is
// known is the type. Empty when nothing is known.
func sample(ac analysis.Column) string {
if ac.Table != "" && ac.Name != "" {
col, tbl := quoteIdent(ac.Name), quoteIdent(ac.Table)
return fmt.Sprintf("(SELECT %s FROM %s WHERE %s IS NOT NULL LIMIT 1)", col, tbl, col)
}
if ac.Type == nil {
return ""
}
switch ac.Type.Name {
case "integer", "numeric":
return "1"
case "real":
return "1.0"
case "text":
return "'a'"
case "blob":
return "x'00'"
}
return ""
}
func quoteIdent(s string) string {
return `"` + strings.ReplaceAll(s, `"`, `""`) + `"`
}
// classes lists the storage classes of a column's values across rows.
func classes(rows []string, i int) []string {
var out []string
for _, row := range rows {
if c := cells(row); i < len(c) {
out = append(out, storageClass(c[i]))
}
}
return out
}
// describeColumn describes a result column: as the table column it is read
// from when it is one, otherwise by the storage class of its values, and
// nullable when any of its values was NULL.
func describeColumn(cat *catalog, m columnMeta, classes []string) analysis.Column {
ac := analysis.Column{Name: m.Name}
if col := cat.lookup(m.Table, m.Origin); col != nil {
d := col.describe()
ac.Type, ac.Table = d.Type, d.Table
} else {
for _, class := range classes {
if class != "null" {
ac.Type = &analysis.TypeExpr{Name: class}
break
}
}
}
if ac.Type != nil {
for _, class := range classes {
if class == "null" {
ac.Type.Nullable = true
}
}
}
return ac
}

456
internal/goldeneye/sqlite/bytecode.go Normal file
View File

@@ -0,0 +1,456 @@
package sqlite
import (
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
)
// SQLite has nothing to say about a parameter's type: a bound value is
// whatever it is, and the column a parameter stands in for appears nowhere
// in what the library reports about a prepared statement. What it does
// report is the program it compiled the statement into. EXPLAIN prints the
// bytecode, in which a parameter is a Variable loading a register, and the
// column it stands in for is the other operand of the comparison the
// register reaches, the row the seek it drives lands on, or the position
// in the record that Insert writes. So each parameter is followed from the
// register it loads, through copies and through the expressions it is an
// argument of, to the first opcode that uses it against something the
// catalog can name, the way the ClickHouse check follows a placeholder
// through the query tree to the column it is compared with.
//
// Registers hold what the last opcode to write them wrote, and remember
// which parameters have been loaded into them, which a later constant does
// not clear: coalesce(?, 'x') overwrites the parameter's register with the
// fallback on the branch where the parameter is NULL, and the comparison
// that follows is still the parameter's. A parameter with no such use is
// described by what the program does to it before using it, when it does
// anything: MustBeInt says it has to be an integer, as LIMIT's does, and
// Affinity says which affinity it is coerced to.
// instr is one line of EXPLAIN output.
type instr struct {
Addr int `json:"addr"`
Opcode string `json:"opcode"`
P1 int `json:"p1"`
P2 int `json:"p2"`
P3 int `json:"p3"`
P4 *string `json:"p4"`
P5 int `json:"p5"`
}
func (in instr) p4() string {
if in.P4 == nil {
return ""
}
return *in.P4
}
type valueKind int
const (
vNone valueKind = iota
vParam // a Variable
vColumn // Column: a stored column of a cursor
vRowid // Rowid, IdxRowid: the rowid of a cursor
vConstant // a literal, with its storage class
vFunction // the result of a function call, with its name
vExpr // the result of an operator
vRecord // MakeRecord: the registers packed into a record
)
// value is what a register holds.
type value struct {
kind valueKind
cursor int
index int
class string // vConstant
fn string // vFunction
regs []int // vRecord
}
type register struct {
desc value
params []int // every parameter loaded into the register
}
// tracer follows the parameters of one statement through its bytecode.
type tracer struct {
cat *catalog
names []string // the statement's result columns
// cursors is what each open cursor is on; ephemeral holds, for the
// cursors on ephemeral tables and sorters, the parameters stored in
// each column, so that a parameter put into an IN list's table comes
// back out with the Column that reads it.
cursors map[int]object
ephemeral map[int]map[int][]int
regs map[int]*register
// found is the column each parameter was found to stand in for, and
// hint what the program said about a parameter's type.
found map[int]analysis.Column
hint map[int]string
}
func newTracer(cat *catalog, names []string) *tracer {
return &tracer{
cat: cat,
names: names,
cursors: map[int]object{},
ephemeral: map[int]map[int][]int{},
regs: map[int]*register{},
found: map[int]analysis.Column{},
hint: map[int]string{},
}
}
// run walks the program in the order it runs: the prologue Init jumps to,
// where the Variables of a query are loaded, and then the body.
func (t *tracer) run(prog []instr) {
order := prog
if len(prog) > 0 && prog[0].Opcode == "Init" && prog[0].P2 > 0 && prog[0].P2 < len(prog) {
order = append(append([]instr{}, prog[prog[0].P2:]...), prog[1:prog[0].P2]...)
}
for _, in := range order {
t.step(in)
}
}
// param describes what the parameter was found to stand in for.
func (t *tracer) param(number int) analysis.Column {
if ac, ok := t.found[number]; ok {
return ac
}
if typ := t.hint[number]; typ != "" {
return analysis.Column{Type: &analysis.TypeExpr{Name: typ}}
}
return analysis.Column{}
}
func (t *tracer) reg(n int) *register {
r := t.regs[n]
if r == nil {
r = &register{}
t.regs[n] = r
}
return r
}
func (t *tracer) params(n int) []int {
if r := t.regs[n]; r != nil {
return r.params
}
return nil
}
// set records a write to a register. A Variable is the parameter and
// nothing else; a column read replaces whatever the register held, since
// the compiler reuses registers for the columns it reads; a constant keeps
// the parameters, for the coalesce case above; anything else carries the
// parameters it was computed from.
func (t *tracer) set(n int, v value, params []int) {
r := t.reg(n)
r.desc = v
switch v.kind {
case vConstant:
default:
r.params = params
}
}
func (t *tracer) copy(from, to int) {
src := t.reg(from)
dst := t.reg(to)
dst.desc = src.desc
dst.params = append([]int{}, src.params...)
}
// assoc records the column the parameters in a register stand in for,
// keeping the first found for each.
func (t *tracer) assoc(params []int, ac analysis.Column) {
for _, p := range params {
if _, ok := t.found[p]; !ok {
t.found[p] = ac
}
}
}
// typeHint records what the program requires of a register's parameters.
func (t *tracer) typeHint(n int, typ string) {
if typ == "" {
return
}
for _, p := range t.params(n) {
if _, ok := t.hint[p]; !ok {
t.hint[p] = typ
}
}
}
// describe turns what a register holds into the column a parameter
// compared with it stands in for, when it can be named: a stored column
// or rowid, a constant's storage class, or a function's name.
func (t *tracer) describe(v value) (analysis.Column, bool) {
switch v.kind {
case vColumn:
return t.stored(v.cursor, v.index)
case vRowid:
if owner := t.cursors[v.cursor].owner(); owner != nil {
return owner.rowid(), true
}
case vConstant:
if v.class != "null" && v.class != "" {
return analysis.Column{Type: &analysis.TypeExpr{Name: v.class}}, true
}
case vFunction:
return analysis.Column{Name: v.fn}, true
}
return analysis.Column{}, false
}
// stored describes the ith stored column of a cursor.
func (t *tracer) stored(cursor, i int) (analysis.Column, bool) {
col, owner, ok := t.cursors[cursor].column(i)
switch {
case !ok:
return analysis.Column{}, false
case col != nil:
return col.describe(), true
default:
return owner.rowid(), true
}
}
// rank orders what a parameter's fellow operands may be described by: a
// column first, then a function, then a constant.
func rank(v value) int {
switch v.kind {
case vColumn, vRowid:
return 0
case vFunction:
return 1
case vConstant:
if v.class != "null" && v.class != "" {
return 2
}
}
return -1
}
// expression handles an opcode computing a result from operand registers:
// each parameter among the operands is described by the best other
// operand, and one with no other operand to be described by is carried
// into the result, for whatever the result is then compared with.
func (t *tracer) expression(args []int, dest int, result value) {
var carried []int
for _, a := range args {
params := t.params(a)
if len(params) == 0 {
continue
}
best := -1
for _, b := range args {
if b == a {
continue
}
if r := rank(t.reg(b).desc); r >= 0 && (best < 0 || r < rank(t.reg(best).desc)) {
best = b
}
}
if best >= 0 {
if ac, ok := t.describe(t.reg(best).desc); ok {
t.assoc(params, ac)
continue
}
}
carried = append(carried, params...)
}
t.set(dest, result, carried)
}
// compare handles a comparison of two registers: the parameters of each
// stand in for what the other holds.
func (t *tracer) compare(a, b int) {
if ac, ok := t.describe(t.reg(b).desc); ok {
t.assoc(t.params(a), ac)
}
if ac, ok := t.describe(t.reg(a).desc); ok {
t.assoc(t.params(b), ac)
}
}
// store handles a register written as the ith column of a cursor's row:
// an ephemeral table remembers the parameters for the reads to come, and
// a real one is the column the parameters stand in for.
func (t *tracer) store(cursor, i, reg int) {
params := t.params(reg)
if len(params) == 0 {
return
}
if cols, ok := t.ephemeral[cursor]; ok {
cols[i] = append(cols[i], params...)
return
}
if ac, ok := t.stored(cursor, i); ok {
t.assoc(params, ac)
}
}
// key handles registers used as a key into a cursor: n registers from
// start, or the record in start when n is zero.
func (t *tracer) key(cursor, start, n int) {
regs := t.span(start, n)
if n == 0 {
if r := t.reg(start); r.desc.kind == vRecord {
regs = r.desc.regs
}
}
for i, reg := range regs {
t.store(cursor, i, reg)
}
}
func (t *tracer) span(start, n int) []int {
regs := make([]int, 0, max(n, 0))
for i := 0; i < n; i++ {
regs = append(regs, start+i)
}
return regs
}
// affinity names the type an affinity character coerces to.
func affinity(c byte) string {
switch c {
case 'A':
return "blob"
case 'B':
return "text"
case 'C':
return "numeric"
case 'D':
return "integer"
case 'E':
return "real"
}
return ""
}
func (t *tracer) step(in instr) {
switch in.Opcode {
case "OpenRead", "OpenWrite", "ReopenIdx":
t.cursors[in.P1] = t.cat.root(in.P3, in.P2)
case "OpenDup":
t.cursors[in.P1] = t.cursors[in.P2]
if cols, ok := t.ephemeral[in.P2]; ok {
t.ephemeral[in.P1] = cols
}
case "OpenEphemeral", "OpenAutoindex", "SorterOpen", "OpenPseudo":
t.cursors[in.P1] = object{}
t.ephemeral[in.P1] = map[int][]int{}
case "Variable":
t.set(in.P2, value{kind: vParam}, []int{in.P1})
case "Column", "VColumn":
t.set(in.P3, value{kind: vColumn, cursor: in.P1, index: in.P2}, t.ephemeral[in.P1][in.P2])
case "Rowid", "IdxRowid":
t.set(in.P2, value{kind: vRowid, cursor: in.P1}, nil)
case "Copy":
for i := 0; i <= in.P3; i++ {
t.copy(in.P1+i, in.P2+i)
}
case "SCopy", "IntCopy":
t.copy(in.P1, in.P2)
case "Null", "SoftNull":
end := max(in.P2, in.P3)
if in.Opcode == "SoftNull" {
end = in.P1
in.P2 = in.P1
}
for r := in.P2; r <= end; r++ {
t.set(r, value{kind: vConstant, class: "null"}, nil)
}
case "Integer", "Int64":
t.set(in.P2, value{kind: vConstant, class: "integer"}, nil)
case "Real":
t.set(in.P2, value{kind: vConstant, class: "real"}, nil)
case "String8", "String":
t.set(in.P2, value{kind: vConstant, class: "text"}, nil)
case "Blob":
t.set(in.P2, value{kind: vConstant, class: "blob"}, nil)
case "Function", "PureFunc":
// P4 prints the function's arity, which for a variadic function
// is negative and says nothing about this call. Its arguments
// are the registers from P2 written so far, allocated together
// for the call.
name, n := parseCall(in.p4())
if n < 0 {
n = 0
for t.regs[in.P2+n] != nil {
n++
}
}
t.expression(t.span(in.P2, n), in.P3, value{kind: vFunction, fn: name})
case "Add", "Subtract", "Multiply", "Divide", "Remainder", "Concat", "BitAnd", "BitOr", "ShiftLeft", "ShiftRight":
t.expression([]int{in.P2, in.P1}, in.P3, value{kind: vExpr})
case "MakeRecord":
t.set(in.P3, value{kind: vRecord, regs: t.span(in.P1, in.P2)}, nil)
case "Insert":
if r := t.reg(in.P2); r.desc.kind == vRecord {
for i, reg := range r.desc.regs {
t.store(in.P1, i, reg)
}
}
if owner := t.cursors[in.P1].owner(); owner != nil {
t.assoc(t.params(in.P3), owner.rowid())
}
case "IdxInsert", "SorterInsert":
n, _ := strconv.Atoi(in.p4())
if n > 0 {
t.key(in.P1, in.P3, n)
} else {
t.key(in.P1, in.P2, 0)
}
case "Eq", "Ne", "Lt", "Le", "Gt", "Ge":
t.compare(in.P1, in.P3)
case "SeekRowid", "NotExists":
if owner := t.cursors[in.P1].owner(); owner != nil {
t.assoc(t.params(in.P3), owner.rowid())
}
case "SeekGE", "SeekGT", "SeekLE", "SeekLT":
// On a table with a rowid the seek is by the rowid in P3; on an
// index it is by the P4 registers from P3.
if o := t.cursors[in.P1]; o.table != nil && !o.table.withoutRowid {
t.assoc(t.params(in.P3), o.table.rowid())
break
}
n, _ := strconv.Atoi(in.p4())
t.key(in.P1, in.P3, n)
case "IdxGE", "IdxGT", "IdxLE", "IdxLT", "Found", "NotFound", "NoConflict":
n, _ := strconv.Atoi(in.p4())
t.key(in.P1, in.P3, n)
case "MustBeInt":
t.typeHint(in.P1, "integer")
case "Affinity":
for i, c := range []byte(in.p4()) {
if i < in.P2 {
t.typeHint(in.P1+i, affinity(c))
}
}
case "ResultRow":
for i := 0; i < in.P2; i++ {
if i < len(t.names) {
t.assoc(t.params(in.P1+i), analysis.Column{Name: t.names[i]})
}
}
}
}
// parseCall splits the `name(n)` a Function's P4 prints into the function's
// name and the number of arguments it is called with.
func parseCall(p4 string) (string, int) {
open := strings.LastIndexByte(p4, '(')
if open < 0 || !strings.HasSuffix(p4, ")") {
return p4, 0
}
n, _ := strconv.Atoi(p4[open+1 : len(p4)-1])
return p4[:open], n
}

294
internal/goldeneye/sqlite/catalog.go Normal file
View File

@@ -0,0 +1,294 @@
package sqlite
import (
"fmt"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/analysis"
)
// catalog is what the database says about a schema: its tables and their
// columns, its indexes, and the b-tree each is stored in, which is how the
// bytecode names them.
type catalog struct {
tables map[string]*table
roots map[int]object // by root page
}
type table struct {
name string
withoutRowid bool
cols []*tableColumn // by cid
// stored lists the columns in the order a row's record holds them,
// which is cid order less the generated columns that are not stored.
stored []*tableColumn
// pk is the PRIMARY KEY index of a WITHOUT ROWID table, whose order is
// the order the table's own b-tree stores a row in.
pk *index
}
type tableColumn struct {
table *table
cid int
name string
declType string
notNull bool
pk int // position in the primary key, 0 for none
}
type index struct {
name string
table *table
cids []int // by seqno: -1 for the rowid, -2 for an expression
}
// object is what a cursor is open on: a table, an index, or, with neither
// set, something the catalog does not describe — an ephemeral table, a
// sorter, a virtual table.
type object struct {
table *table
index *index
}
// The catalog is read with these, each in a section of its own, in json
// mode. A query that finds nothing prints nothing, so a section may be
// empty.
var catalogQueries = []struct{ section, sql string }{
{"tables", `SELECT name, type, wr FROM pragma_table_list WHERE schema = 'main' ORDER BY name`},
{"columns", `SELECT t.name AS tbl, c.cid, c.name, c.type, c."notnull" AS "notnull", c.pk, c.hidden
FROM pragma_table_list t, pragma_table_xinfo(t.name) c
WHERE t.schema = 'main' AND t.type IN ('table', 'virtual') ORDER BY t.name, c.cid`},
{"indexes", `SELECT t.name AS tbl, i.name AS idx, i.origin, x.seqno, x.cid
FROM pragma_table_list t, pragma_index_list(t.name) i, pragma_index_xinfo(i.name) x
WHERE t.schema = 'main' ORDER BY t.name, i.name, x.seqno`},
{"roots", `SELECT type, name, tbl_name, rootpage FROM sqlite_schema WHERE rootpage > 0`},
}
// readCatalog builds the catalog from the sections catalogQueries printed.
func readCatalog(out *output) (*catalog, error) {
var tables []struct {
Name string `json:"name"`
Type string `json:"type"`
WR int `json:"wr"`
}
var columns []struct {
Table string `json:"tbl"`
CID int `json:"cid"`
Name string `json:"name"`
Type string `json:"type"`
NotNull int `json:"notnull"`
PK int `json:"pk"`
Hidden int `json:"hidden"`
}
var indexes []struct {
Table string `json:"tbl"`
Index string `json:"idx"`
Origin string `json:"origin"`
Seqno int `json:"seqno"`
CID int `json:"cid"`
}
var roots []struct {
Type string `json:"type"`
Name string `json:"name"`
Table string `json:"tbl_name"`
Rootpage int `json:"rootpage"`
}
for name, v := range map[string]any{"tables": &tables, "columns": &columns, "indexes": &indexes, "roots": &roots} {
s := out.sections[name]
if s == nil {
return nil, fmt.Errorf("reading the catalog: no %s section in the output", name)
}
if len(s.blocks) > 0 {
if err := s.decode(0, v); err != nil {
return nil, fmt.Errorf("reading the catalog's %s: %w", name, err)
}
}
}
c := &catalog{tables: map[string]*table{}, roots: map[int]object{}}
for _, t := range tables {
if t.Type == "table" || t.Type == "virtual" {
c.tables[t.Name] = &table{name: t.Name, withoutRowid: t.WR == 1}
}
}
for _, col := range columns {
t := c.tables[col.Table]
if t == nil {
continue
}
tc := &tableColumn{table: t, cid: col.CID, name: col.Name, declType: col.Type, notNull: col.NotNull == 1, pk: col.PK}
for len(t.cols) <= col.CID {
t.cols = append(t.cols, nil)
}
t.cols[col.CID] = tc
// A generated column that is not stored — hidden is 2 — has no
// place in the record.
if col.Hidden != 2 {
t.stored = append(t.stored, tc)
}
}
byName := map[string]*index{}
for _, ic := range indexes {
t := c.tables[ic.Table]
if t == nil {
continue
}
ix := byName[ic.Index]
if ix == nil {
ix = &index{name: ic.Index, table: t}
byName[ic.Index] = ix
if ic.Origin == "pk" && t.withoutRowid {
t.pk = ix
}
}
for len(ix.cids) <= ic.Seqno {
ix.cids = append(ix.cids, -2)
}
ix.cids[ic.Seqno] = ic.CID
}
for _, r := range roots {
switch r.Type {
case "table":
if t := c.tables[r.Name]; t != nil {
c.roots[r.Rootpage] = object{table: t}
}
case "index":
if ix := byName[r.Name]; ix != nil {
c.roots[r.Rootpage] = object{index: ix}
}
}
}
return c, nil
}
// root returns what is stored in a b-tree of the main database.
func (c *catalog) root(db, page int) object {
if db != 0 {
return object{}
}
return c.roots[page]
}
// lookup finds a table column by the names `.stats stmt` prints.
func (c *catalog) lookup(tbl, col string) *tableColumn {
t := c.tables[tbl]
if t == nil {
return nil
}
for _, tc := range t.cols {
if tc != nil && tc.name == col {
return tc
}
}
return nil
}
// rowidAlias returns the column that is another name for the rowid: the
// lone INTEGER PRIMARY KEY of a table that has a rowid.
func (t *table) rowidAlias() *tableColumn {
if t.withoutRowid {
return nil
}
var alias *tableColumn
for _, tc := range t.cols {
if tc == nil || tc.pk == 0 {
continue
}
if alias != nil || !strings.EqualFold(tc.declType, "INTEGER") {
return nil
}
alias = tc
}
return alias
}
// describe is the column as sqlc analyze describes one: its declared type,
// nullable unless declared NOT NULL or the rowid, which is never NULL.
func (tc *tableColumn) describe() analysis.Column {
typ := parseType(tc.declType)
typ.Nullable = !tc.notNull && tc.table.rowidAlias() != tc
return analysis.Column{Name: tc.name, Type: typ, Table: tc.table.name}
}
// rowid describes the table's rowid: the column that aliases it, or the
// rowid itself.
func (t *table) rowid() analysis.Column {
if alias := t.rowidAlias(); alias != nil {
return alias.describe()
}
return analysis.Column{Name: "rowid", Type: &analysis.TypeExpr{Name: "integer"}, Table: t.name}
}
// column returns what the object's ith stored column is: a table column,
// or the rowid of a table. An index stores the columns it is on and then
// the rowid; a WITHOUT ROWID table is stored in the order of its primary
// key; any other table is stored in column order.
func (o object) column(i int) (*tableColumn, *table, bool) {
ix := o.index
if o.table != nil && o.table.withoutRowid {
ix = o.table.pk
}
switch {
case ix != nil:
if i < 0 || i >= len(ix.cids) {
return nil, nil, false
}
cid := ix.cids[i]
switch {
case cid == -1:
return nil, ix.table, true
case cid >= 0 && cid < len(ix.table.cols) && ix.table.cols[cid] != nil:
return ix.table.cols[cid], nil, true
}
case o.table != nil:
if i >= 0 && i < len(o.table.stored) {
return o.table.stored[i], nil, true
}
}
return nil, nil, false
}
// owner is the table the object stores rows of.
func (o object) owner() *table {
if o.index != nil {
return o.index.table
}
return o.table
}
// parseType reads a declared type the way sqlc's catalog does: the name
// lowercased, with whatever is in parentheses after it as arguments. A
// column declared with no type at all can hold anything.
func parseType(decl string) *analysis.TypeExpr {
decl = strings.TrimSpace(decl)
if decl == "" {
return &analysis.TypeExpr{Name: "any"}
}
name, args := decl, ""
if open := strings.IndexByte(decl, '('); open >= 0 && strings.HasSuffix(decl, ")") {
name, args = decl[:open], decl[open+1:len(decl)-1]
}
t := &analysis.TypeExpr{Name: strings.ToLower(strings.TrimSpace(name))}
if strings.TrimSpace(args) == "" {
return t
}
for _, a := range strings.Split(args, ",") {
a = strings.TrimSpace(a)
switch {
case strings.HasPrefix(a, "'") && strings.HasSuffix(a, "'") && len(a) >= 2:
s := strings.ReplaceAll(a[1:len(a)-1], "''", "'")
t.Args = append(t.Args, analysis.TypeArg{String: &s})
case strings.EqualFold(a, "true") || strings.EqualFold(a, "false"):
b := strings.EqualFold(a, "true")
t.Args = append(t.Args, analysis.TypeArg{Bool: &b})
default:
if n, err := strconv.ParseInt(a, 10, 64); err == nil {
t.Args = append(t.Args, analysis.TypeArg{Int: &n})
} else {
t.Args = append(t.Args, analysis.TypeArg{Type: &analysis.TypeExpr{Name: strings.ToLower(a)}})
}
}
}
return t
}

347
internal/goldeneye/sqlite/install.go Normal file
View File

@@ -0,0 +1,347 @@
package sqlite
import (
"archive/zip"
"context"
"crypto/sha3"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"runtime"
"slices"
"strings"
"sync"
)
// DefaultVersion is the SQLite release the dialect is generated from. It is
// the release the ncruces/go-sqlite3 driver in the main module embeds, so
// the functions the dialect knows are the ones the tests run against.
// Bumping it is a deliberate change: releases add functions and overloads,
// so regenerate and review the dialect after changing it, and add the new
// release's amalgamation to the table below.
const DefaultVersion = "3.53.4"
// defaultOptions are the compile options the base dialect is built with:
// the ones sqlite.org's own configure turns on by default, and so the ones
// a stock sqlite3 has. JSON is part of the library unless omitted, so only
// the math functions need naming.
var defaultOptions = []string{"SQLITE_ENABLE_MATH_FUNCTIONS"}
// extensions are the compile options that each become a directory under
// the dialect's extensions/, holding the functions a build with the option
// adds over the default build — the way each PostgreSQL contrib extension
// holds what CREATE EXTENSION adds. Each is named after its option as
// pragma compile_options spells it, in lower case, and lists the options
// its build needs: GEOPOLY lives inside the RTREE module, so enabling it
// alone adds nothing. Options that add virtual tables but no functions,
// such as SESSION and DBSTAT, are not listed, since the dialect has nothing
// to say about them, and ENABLE_FTS4 is the same module as ENABLE_FTS3.
var extensions = []build{
{"soundex", []string{"SQLITE_SOUNDEX"}},
{"enable_fts3", []string{"SQLITE_ENABLE_FTS3"}},
{"enable_fts5", []string{"SQLITE_ENABLE_FTS5"}},
{"enable_geopoly", []string{"SQLITE_ENABLE_RTREE", "SQLITE_ENABLE_GEOPOLY"}},
{"enable_offset_sql_func", []string{"SQLITE_ENABLE_OFFSET_SQL_FUNC"}},
{"enable_percentile", []string{"SQLITE_ENABLE_PERCENTILE"}},
{"enable_rtree", []string{"SQLITE_ENABLE_RTREE"}},
}
// analysisShell is the shell the analyze cases run through, which is no
// dialect build: nothing is generated from it. It is built with column
// metadata, so that `.stats stmt` can say which table column each result
// column of a statement comes from, and with every extension option at
// once, so that whatever a case's schema asks for is there.
var analysisShell = build{"analysis", append([]string{"SQLITE_ENABLE_COLUMN_METADATA"}, extensionOptions()...)}
// extensionOptions is every option an extension build turns on, once each.
func extensionOptions() []string {
var opts []string
for _, b := range extensions {
for _, opt := range b.options {
if !slices.Contains(opts, opt) {
opts = append(opts, opt)
}
}
}
return opts
}
// asset is one downloadable amalgamation of SQLite: the zip published on
// sqlite.org holding sqlite3.c, sqlite3.h and the shell's shell.c.
type asset struct {
Version string
// Path is the download's address under https://sqlite.org/, the
// release year included, as the download page's index lists it.
Path string
// SHA3 is the SHA3-256 of the zip, as the download page lists it.
SHA3 string
}
// assets lists every amalgamation Install knows how to fetch, with the
// SHA3-256 of the download. A version that is not in this table cannot be
// installed: verifying the download is the point of the table. The
// checksums are the ones in the index at the foot of
// https://sqlite.org/download.html.
var assets = []asset{
{"3.53.4", "2026/sqlite-amalgamation-3530400.zip", "628a44cfe82c66aed1ccbbe85a562d2e33ebe64b3288981ed76285612227934e"},
}
// sources are the files taken out of the amalgamation.
var sources = []string{"sqlite3.c", "sqlite3.h", "shell.c"}
func releaseAsset(version string) (asset, error) {
for _, a := range assets {
if a.Version == version {
return a, nil
}
}
return asset{}, fmt.Errorf("SQLite %s is not in the asset table; add its amalgamation and checksum to install.go", version)
}
func (a asset) url() string {
return "https://sqlite.org/" + a.Path
}
// cacheDir is where Install puts a version: the sources under src/, and one
// shell per build under default/, under each option's extension name and
// under analysis/.
func cacheDir(version string) (string, error) {
dir, err := os.UserCacheDir()
if err != nil {
return "", err
}
return filepath.Join(dir, "sqlc-sqlite", version), nil
}
// build is one compiled shell: its name, which is the directory it is
// under in the version's cache directory and, for an extension, under the
// dialect's extensions/, and the options it is built with beyond the
// default ones.
type build struct {
name string
options []string
}
// builds lists the dialect builds: the default build first, then one per
// extension.
func builds() []build {
return append([]build{{"default", nil}}, extensions...)
}
// shells lists every build Install makes: the dialect builds and the
// analysis shell.
func shells() []build {
return append(builds(), analysisShell)
}
// flags are every option a build is compiled with.
func (b build) flags() []string {
return append(append([]string{}, defaultOptions...), b.options...)
}
func (b build) binary(dir string) string {
return filepath.Join(dir, b.name, "sqlite3")
}
// Locate finds the directory of shells Install made for DefaultVersion.
func Locate() (string, error) {
dir, err := cacheDir(DefaultVersion)
if err != nil {
return "", err
}
for _, b := range shells() {
if _, err := os.Stat(b.binary(dir)); err != nil {
return "", fmt.Errorf("sqlite %s is not built with %s: run `go run ./cmd/goldeneye install sqlite` in internal/goldeneye", DefaultVersion, strings.Join(b.flags(), " "))
}
}
return dir, nil
}
// Install downloads the amalgamation for a version into the cache and
// compiles the shell from it once per build, returning the directory the
// shells are under. Builds already there are kept, so adding an option
// compiles only its shell. The download is checked against the table's
// SHA3-256 before it is unpacked. Compiling takes the compiler CC names, or
// cc, and a few seconds per build without optimisation, which a shell that
// only reads catalogs does not need.
//
// goos and goarch are what every installer is handed; the sources build the
// same everywhere a C compiler is, but no default compiler or link line is
// known for Windows.
func Install(ctx context.Context, version, goos, goarch string, progress io.Writer) (string, error) {
if goos == "windows" {
return "", errors.New("building SQLite from source is not supported on Windows")
}
dir, err := cacheDir(version)
if err != nil {
return "", err
}
var missing []build
for _, b := range shells() {
if _, err := os.Stat(b.binary(dir)); err != nil {
missing = append(missing, b)
}
}
if len(missing) == 0 {
return dir, nil
}
if err := download(ctx, version, dir, progress); err != nil {
return "", err
}
if err := compile(ctx, dir, missing, progress); err != nil {
return "", err
}
return dir, nil
}
// download fetches the amalgamation and unpacks the sources into src/,
// unless they are already there.
func download(ctx context.Context, version, dir string, progress io.Writer) error {
src := filepath.Join(dir, "src")
have := true
for _, name := range sources {
if _, err := os.Stat(filepath.Join(src, name)); err != nil {
have = false
}
}
if have {
return nil
}
a, err := releaseAsset(version)
if err != nil {
return err
}
if err := os.MkdirAll(src, 0o755); err != nil {
return err
}
url := a.url()
fmt.Fprintf(progress, "downloading %s\n", url)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("downloading %s: %s", url, resp.Status)
}
// The zip has to land on disk before it can be read — its directory is
// at the end — so download it whole and hash every byte on the way.
archive, err := os.CreateTemp(dir, "sqlite-amalgamation-*.partial")
if err != nil {
return err
}
defer os.Remove(archive.Name())
sum := sha3.New256()
if _, err := io.Copy(io.MultiWriter(archive, sum), resp.Body); err != nil {
archive.Close()
return err
}
if err := archive.Close(); err != nil {
return err
}
if got := hex.EncodeToString(sum.Sum(nil)); got != a.SHA3 {
return fmt.Errorf("downloading %s: SHA3-256 mismatch: got %s, want %s", url, got, a.SHA3)
}
zr, err := zip.OpenReader(archive.Name())
if err != nil {
return err
}
defer zr.Close()
for _, name := range sources {
if err := extract(zr, name, filepath.Join(src, name)); err != nil {
return fmt.Errorf("downloading %s: %w", url, err)
}
}
return nil
}
// extract copies the named file out of the zip. The amalgamation's files
// sit in one directory named after the release, so the name is matched on
// its base.
func extract(zr *zip.ReadCloser, name, dest string) error {
for _, f := range zr.File {
if filepath.Base(f.Name) != name || f.FileInfo().IsDir() {
continue
}
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
tmp, err := os.CreateTemp(filepath.Dir(dest), name+"-*.partial")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := io.Copy(tmp, rc); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmp.Name(), dest)
}
return fmt.Errorf("zip does not contain %s", name)
}
// compile builds the shells, as many at a time as there are CPUs.
func compile(ctx context.Context, dir string, todo []build, progress io.Writer) error {
cc := os.Getenv("CC")
if cc == "" {
cc = "cc"
}
if _, err := exec.LookPath(cc); err != nil {
return fmt.Errorf("no C compiler found: put cc on PATH or set CC to one")
}
var wg sync.WaitGroup
slots := make(chan struct{}, max(1, runtime.NumCPU()))
errs := make([]error, len(todo))
for i, b := range todo {
wg.Add(1)
go func() {
defer wg.Done()
slots <- struct{}{}
defer func() { <-slots }()
errs[i] = b.compile(ctx, cc, dir, progress)
}()
}
wg.Wait()
return errors.Join(errs...)
}
// compile builds one shell from the sources in src/, writing it beside its
// destination and renaming so a failed build never masquerades as one.
func (b build) compile(ctx context.Context, cc, dir string, progress io.Writer) error {
dest := b.binary(dir)
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return err
}
tmp := dest + ".partial"
defer os.Remove(tmp)
args := []string{"-O0"}
for _, opt := range b.flags() {
args = append(args, "-D"+opt)
}
src := filepath.Join(dir, "src")
args = append(args, filepath.Join(src, "shell.c"), filepath.Join(src, "sqlite3.c"), "-o", tmp, "-lm", "-ldl", "-lpthread")
fmt.Fprintf(progress, "building %s: %s %s\n", b.name, cc, strings.Join(args, " "))
cmd := exec.CommandContext(ctx, cc, args...)
out, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("building %s: %w\n%s", b.name, err, strings.TrimSpace(string(out)))
}
return os.Rename(tmp, dest)
}

254
internal/goldeneye/sqlite/shell.go Normal file
View File

@@ -0,0 +1,254 @@
package sqlite
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os/exec"
"regexp"
"strconv"
"strings"
)
// The analysis shell is driven with a script on its standard input, made
// of SQL and of the shell's own dot-commands: `.print` labels the sections
// of the output so that each statement's results can be found again,
// `.mode` chooses how rows print — json for the catalog and the bytecode,
// quote for a query's own rows, since quote mode is the one that tells a
// blob from a string and a real from an integer — and `.stats stmt` makes
// the shell print what it knows about each result column of the statement
// it has just run, which with column metadata compiled in is the column's
// name, declared type, database, table and origin column.
// script is a shell script under construction, counting its lines so that
// an error the shell reports "near line N" can be placed.
type script struct {
buf strings.Builder
line int // the number of the next line written
}
func newScript() *script {
return &script{line: 1}
}
// add writes text on lines of its own, and returns the line it starts on.
func (s *script) add(text string) int {
start := s.line
text = strings.TrimSpace(text)
if text == "" {
return start
}
s.buf.WriteString(text)
s.buf.WriteByte('\n')
s.line += strings.Count(text, "\n") + 1
return start
}
// sql writes statements, ending them with a semicolon when they lack one.
func (s *script) sql(text string) int {
text = strings.TrimRight(strings.TrimSpace(text), "; \t\r\n")
if text == "" {
return s.line
}
return s.add(text + ";")
}
// section labels what follows, up to the next label.
func (s *script) section(name string) {
s.add(".print @@" + name)
}
// columnMeta is what `.stats stmt` prints about one result column. Table
// and Origin name the table column a result column is read from, and are
// empty for an expression.
type columnMeta struct {
Name string
DeclType string
Database string
Table string
Origin string
}
// section is the output between two labels.
type section struct {
// blocks holds every JSON result set printed in the section.
blocks []json.RawMessage
// rows holds every other line printed before the statement's
// statistics: in quote mode, one row each.
rows []string
// prepared says a statement's statistics were printed, which the shell
// does only for a statement it could prepare.
prepared bool
columns []columnMeta
}
// decode reads the section's nth JSON result set.
func (s *section) decode(n int, v any) error {
if n >= len(s.blocks) {
return fmt.Errorf("expected at least %d result set(s), got %d", n+1, len(s.blocks))
}
return json.Unmarshal(s.blocks[n], v)
}
// output is what one run of the shell printed, by section.
type output struct {
sections map[string]*section
stderr string
}
var errLineRe = regexp.MustCompile(`near line (\d+):`)
// errorsBefore returns the errors the shell reported for statements before
// the given line: the ones that are not the statement under analysis
// failing to run, which it may, with nothing bound to its parameters.
func (o *output) errorsBefore(line int) []string {
var errs []string
for _, l := range strings.Split(o.stderr, "\n") {
if m := errLineRe.FindStringSubmatch(l); m != nil {
if n, _ := strconv.Atoi(m[1]); n < line {
errs = append(errs, strings.TrimSpace(l))
}
}
}
return errs
}
// run feeds a script to the shell over a fresh in-memory database and
// parses what it printed. A statement that fails does not stop the shell,
// so a failure is read from the output: a section whose statement has no
// statistics was not prepared, and stderr says why.
func run(ctx context.Context, binary string, s *script) (*output, error) {
cmd := exec.CommandContext(ctx, binary, ":memory:")
cmd.Stdin = strings.NewReader(s.buf.String())
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
var exit *exec.ExitError
if !errors.As(err, &exit) {
return nil, fmt.Errorf("sqlite3: %w", err)
}
}
out := &output{sections: map[string]*section{}, stderr: stderr.String()}
var (
cur *section
block []string // lines of the JSON result set being read
row []string // lines of the quote-mode row being read
)
for _, line := range strings.Split(stdout.String(), "\n") {
switch {
case block != nil:
block = append(block, line)
if strings.HasSuffix(line, "]") {
cur.blocks = append(cur.blocks, json.RawMessage(strings.Join(block, "\n")))
block = nil
}
case row != nil:
row = append(row, line)
if joined := strings.Join(row, "\n"); strings.Count(joined, "'")%2 == 0 {
cur.rows = append(cur.rows, joined)
row = nil
}
case strings.HasPrefix(line, "@@"):
cur = &section{}
out.sections[line[2:]] = cur
case cur == nil || line == "":
case strings.HasPrefix(line, "["):
if strings.HasSuffix(line, "]") {
cur.blocks = append(cur.blocks, json.RawMessage(line))
} else {
block = []string{line}
}
case strings.HasPrefix(line, "Number of output columns:"):
cur.prepared = true
case cur.prepared:
if i, field, value, ok := statLine(line); ok {
for len(cur.columns) <= i {
cur.columns = append(cur.columns, columnMeta{})
}
c := &cur.columns[i]
switch field {
case "name":
c.Name = value
case "declared type":
c.DeclType = value
case "database name":
c.Database = value
case "table name":
c.Table = value
case "origin name":
c.Origin = value
}
}
default:
// A quote-mode row, complete unless a string in it spans lines.
if strings.Count(line, "'")%2 == 0 {
cur.rows = append(cur.rows, line)
} else {
row = []string{line}
}
}
}
return out, nil
}
var statLineRe = regexp.MustCompile(`^Column (\d+) ([a-z ]+):`)
// statLine reads one line of `.stats stmt` column output, which the shell
// prints as the label padded to 36 columns, a space and the value; a value
// the library has no answer for prints as the C library prints a null
// string.
func statLine(line string) (int, string, string, bool) {
m := statLineRe.FindStringSubmatch(line)
if m == nil {
return 0, "", "", false
}
i, _ := strconv.Atoi(m[1])
label := m[0]
value := ""
if start := max(len(label), 36) + 1; start < len(line) {
value = line[start:]
}
if value == "(null)" {
value = ""
}
return i, m[2], value, true
}
// cells splits a quote-mode row into its values as printed.
func cells(row string) []string {
var out []string
start := 0
quoted := false
for i := 0; i < len(row); i++ {
switch {
case row[i] == '\'':
quoted = !quoted
case row[i] == ',' && !quoted:
out = append(out, row[start:i])
start = i + 1
}
}
return append(out, row[start:])
}
// storageClass reads the storage class of a value as quote mode prints
// it: a string is quoted, a blob is quoted with an X, NULL is spelled
// out, and a real has a point, an exponent or is infinite or not a number
// where an integer is digits.
func storageClass(cell string) string {
switch {
case cell == "NULL":
return "null"
case strings.HasPrefix(cell, "'"):
return "text"
case strings.HasPrefix(cell, "x'") || strings.HasPrefix(cell, "X'"):
return "blob"
case strings.ContainsAny(cell, ".eEIN"):
return "real"
default:
return "integer"
}
}

136
internal/goldeneye/sqlite/signatures.go Normal file
View File

@@ -0,0 +1,136 @@
package sqlite
import "github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
// signature is what a function returns and what its arguments hold, read
// from the amalgamation by source.signature. Args types the leading
// arguments in order; a position it does not cover is "any", which the
// analyzer resolves to the argument's own type. Variadic types the
// arguments an overload of variable arity repeats, "any" when empty. How
// many arguments an overload takes, and how many of them it requires,
// comes from the shell. Nullable is decided afterwards: for an aggregate by
// running it over no rows, for a scalar by the nullable list below.
// Numeric says the function returns an integer for an integer argument
// and a real for a real one, and Returns what it returns for anything
// else.
type signature struct {
Args []string
Variadic string
Returns string
Numeric bool
Nullable bool
}
// args builds an overload's parameters from the shell's argument count. A
// fixed count is that many parameters; a negative count is the required
// leading parameters, any further ones the signature types marked as
// having a default, and a variadic tail.
func (s signature) args(narg int) []dialect.Arg {
if narg >= 0 {
args := make([]dialect.Arg, narg)
for i := range args {
args[i] = dialect.Arg{Type: s.argType(i)}
}
return args
}
required := minArgs(narg)
n := max(required, len(s.Args))
args := make([]dialect.Arg, 0, n+1)
for i := 0; i < n; i++ {
args = append(args, dialect.Arg{Type: s.argType(i), HasDefault: i >= required})
}
tail := s.Variadic
if tail == "" {
tail = "any"
}
return append(args, dialect.Arg{Type: tail, Mode: "v"})
}
func (s signature) argType(i int) string {
if i < len(s.Args) {
return s.Args[i]
}
return "any"
}
// inlineReturns is what the functions the VDBE implements in bytecode
// return, by the INLINEFUNC_* constant their registration carries. All but
// one hand back one of their arguments, which is what the default of "any"
// says; sqlite_offset is a byte offset.
var inlineReturns = map[string]string{
"INLINEFUNC_sqlite_offset": "integer",
}
// realResults are the functions that return an integer or a real on
// something other than which their argument is, and so are typed real
// alone: unixepoch takes a time string and returns a real only with the
// 'subsec' modifier.
var realResults = map[string]bool{
"unixepoch": true,
}
// omitted are functions the dialect leaves out: ones that exist for their
// side effect and return nothing a query can use, and ones an extension
// uses to pass pointers to itself.
var omitted = map[string]bool{
// Loads a shared library and returns NULL.
"load_extension": true,
// Writes to the error log and returns NULL.
"sqlite_log": true,
// FTS3's and FTS5's ways of passing pointers to their virtual tables,
// not functions a query calls.
"fts3_tokenizer": true,
"fts5": true,
// A debugging aid of GEOPOLY's.
"geopoly_debug": true,
}
// nullable are the scalar functions that return NULL for arguments that
// are not: a lookup that finds nothing, an input that does not parse, a
// value with no sign. The source cannot say this — a SQLite function
// returns NULL as often by setting no result as by calling
// sqlite3_result_null — so it is listed by the documentation of each.
// Aggregates are not listed: whether one returns NULL over no rows is
// found by running it over none.
var nullable = map[string]bool{
// Core functions.
"nullif": true,
"sign": true,
"sqlite_compileoption_get": true,
"sqlite_offset": true,
"unhex": true,
// Date and time functions, for a time value they cannot parse.
"date": true,
"datetime": true,
"julianday": true,
"strftime": true,
"time": true,
"timediff": true,
"unixepoch": true,
// JSON functions, for a path that leads nowhere.
"->": true,
"->>": true,
"json_array_length": true,
"json_extract": true,
"json_type": true,
"jsonb_extract": true,
// Window functions, for a row outside the frame.
"first_value": true,
"lag": true,
"last_value": true,
"lead": true,
"nth_value": true,
// FTS5, for a table with no locale.
"fts5_get_locale": true,
// GEOPOLY functions, for an argument that is not a polygon.
"geopoly_area": true,
"geopoly_bbox": true,
"geopoly_blob": true,
"geopoly_ccw": true,
"geopoly_contains_point": true,
"geopoly_json": true,
"geopoly_overlap": true,
"geopoly_svg": true,
"geopoly_within": true,
"geopoly_xform": true,
}

521
internal/goldeneye/sqlite/source.go Normal file
View File

@@ -0,0 +1,521 @@
package sqlite
import (
"fmt"
"os"
"regexp"
"sort"
"strings"
)
// source is what the amalgamation says about how SQLite's functions answer.
// A function is registered in one of a few shapes — the FuncDef macros of
// the built-in tables, a sqlite3_create_function call, or a struct table an
// extension walks — each naming the C functions that implement it. Those
// implementations set their result through sqlite3_result_* and read their
// arguments through sqlite3_value_*, which is as close as SQLite comes to
// declaring a signature.
type source struct {
funcs map[string]cfunc
aliases map[string]string
regs map[string]*registration
}
// cfunc is one C function definition: its parameter list and its body.
type cfunc struct {
params string
body string
}
// registration is the union of everything the source registers under one
// SQL function name.
type registration struct {
// scalar implements a scalar overload: its results and its arguments.
scalar []string
// step and final implement an aggregate or window overload: step reads
// the arguments, final and value set the result.
step []string
final []string
// inline names the INLINEFUNC_* constant of a function the VDBE
// implements in bytecode, which has no C body to read.
inline string
// json records what a JSON function's registration says it returns,
// "text" or "blob", since the json_ and jsonb_ forms share an
// implementation; jsonAlways says the registration promises JSON text
// whatever the implementation might otherwise return, as -> does.
json string
jsonAlways bool
// table marks a registration found only in a struct table, which is
// consulted when nothing more direct registered the name.
table bool
}
// The FuncDef macros, with the positions of the C functions in each.
var macroEntry = regexp.MustCompile(`\b(FUNCTION2|FUNCTION|VFUNCTION|SFUNCTION|MFUNCTION|JFUNCTION|INLINE_FUNC|DFUNCTION|PURE_DATE|STR_FUNCTION|LIKEFUNC|WAGGREGATE|WINDOWFUNCX|WINDOWFUNCALL|WINDOWFUNCNOOP)\(`)
var (
createFunction = regexp.MustCompile(`\bsqlite3_create_(window_)?function\(`)
defineAlias = regexp.MustCompile(`(?m)^#define\s+(\w+)\s+(\w+)\s*$`)
definition = regexp.MustCompile(`(?m)^(?:static\s+|SQLITE_PRIVATE\s+)?(?:const\s+)?(?:unsigned\s+)?[A-Za-z_]\w*(?:\s*\*+\s*|\s+)([A-Za-z_]\w*)\(`)
initializer = regexp.MustCompile(`\{[^{}]*"([a-z_0-9>-]+)"[^{}]*\}`)
identifier = regexp.MustCompile(`\b([A-Za-z_]\w*)\b`)
resultCall = regexp.MustCompile(`\bsqlite3_result_(\w+)\(`)
valueCall = regexp.MustCompile(`\bsqlite3_value_(\w+)\(\s*(?:argv|apVal|apArg)\[(\w+)\]`)
callee = regexp.MustCompile(`\b([A-Za-z_]\w*)\s*\(`)
)
// readSource reads the amalgamation.
func readSource(path string) (*source, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
text := string(data)
s := &source{
funcs: map[string]cfunc{},
aliases: map[string]string{},
regs: map[string]*registration{},
}
for _, m := range defineAlias.FindAllStringSubmatch(text, -1) {
s.aliases[m[1]] = m[2]
}
s.readDefinitions(text)
s.readMacros(text)
s.readCreateFunctions(text)
s.readTables(text)
return s, nil
}
// readDefinitions indexes every function defined at the left margin.
func (s *source) readDefinitions(text string) {
for _, m := range definition.FindAllStringSubmatchIndex(text, -1) {
name := text[m[2]:m[3]]
params, end, ok := balanced(text, m[1]-1, '(', ')')
if !ok {
continue
}
i := end
for i < len(text) && (text[i] == ' ' || text[i] == '\n' || text[i] == '\r' || text[i] == '\t') {
i++
}
if i >= len(text) || text[i] != '{' {
continue
}
body, _, ok := balanced(text, i, '{', '}')
if !ok {
continue
}
if _, dup := s.funcs[name]; !dup {
s.funcs[name] = cfunc{params: params, body: body}
}
}
}
// balanced returns the text between the bracket at open and its match,
// skipping string and character literals and comments, and the index after
// the closing bracket.
func balanced(text string, open int, lb, rb byte) (string, int, bool) {
depth := 0
for i := open; i < len(text); i++ {
switch c := text[i]; {
case c == '"' || c == '\'':
for i++; i < len(text) && text[i] != c; i++ {
if text[i] == '\\' {
i++
}
}
case c == '/' && i+1 < len(text) && text[i+1] == '*':
end := strings.Index(text[i+2:], "*/")
if end < 0 {
return "", 0, false
}
i += end + 3
case c == '/' && i+1 < len(text) && text[i+1] == '/':
end := strings.IndexByte(text[i:], '\n')
if end < 0 {
return "", 0, false
}
i += end
case c == lb:
depth++
case c == rb:
depth--
if depth == 0 {
return text[open+1 : i], i + 1, true
}
}
}
return "", 0, false
}
// arguments splits a bracketed argument list at the commas of its own
// level, and reports the index after the closing bracket.
func arguments(text string, open int) ([]string, int, bool) {
inner, end, ok := balanced(text, open, '(', ')')
if !ok {
return nil, 0, false
}
var args []string
depth, start := 0, 0
for i := 0; i < len(inner); i++ {
switch inner[i] {
case '(', '[', '{':
depth++
case ')', ']', '}':
depth--
case '"':
for i++; i < len(inner) && inner[i] != '"'; i++ {
if inner[i] == '\\' {
i++
}
}
case ',':
if depth == 0 {
args = append(args, strings.TrimSpace(inner[start:i]))
start = i + 1
}
}
}
args = append(args, strings.TrimSpace(inner[start:]))
return args, end, true
}
func (s *source) reg(name string) *registration {
name = strings.ToLower(name)
r, ok := s.regs[name]
if !ok {
r = &registration{}
s.regs[name] = r
}
return r
}
// readMacros reads the FuncDef tables. The macro definitions themselves
// match too, and are told apart by their formal parameter names.
func (s *source) readMacros(text string) {
for _, m := range macroEntry.FindAllStringSubmatchIndex(text, -1) {
macro := text[m[2]:m[3]]
args, _, ok := arguments(text, m[1]-1)
if !ok || len(args) < 2 || args[0] == "zName" || args[0] == "name" {
continue
}
name := args[0]
switch macro {
case "FUNCTION", "FUNCTION2", "VFUNCTION", "SFUNCTION", "DFUNCTION", "PURE_DATE", "STR_FUNCTION":
if len(args) > 4 {
r := s.reg(name)
r.scalar = append(r.scalar, args[4])
}
case "MFUNCTION":
if len(args) > 3 {
r := s.reg(name)
r.scalar = append(r.scalar, args[3])
}
case "JFUNCTION":
// JFUNCTION(zName, nArg, bUseCache, bWS, bRS, bJsonB, iArg, xFunc)
if len(args) > 7 {
r := s.reg(name)
r.scalar = append(r.scalar, args[7])
r.json = "text"
if args[5] == "1" {
r.json = "blob"
}
r.jsonAlways = strings.Contains(args[6], "JSON_JSON")
}
case "INLINE_FUNC":
if len(args) > 2 {
s.reg(name).inline = args[2]
}
case "LIKEFUNC":
r := s.reg(name)
r.scalar = append(r.scalar, "likeFunc")
case "WAGGREGATE":
// WAGGREGATE(zName, nArg, arg, nc, xStep, xFinal, xValue, xInverse, f)
// The JSON aggregates use it too, told by their implementation,
// with JSON_BLOB as the user data of the jsonb_ forms.
if len(args) > 6 {
r := s.reg(name)
r.step = append(r.step, args[4])
r.final = append(r.final, args[5], args[6])
if strings.HasPrefix(args[4], "json") {
r.json = "text"
if strings.Contains(args[2], "JSON_BLOB") {
r.json = "blob"
}
}
}
case "WINDOWFUNCX", "WINDOWFUNCALL":
r := s.reg(name)
r.step = append(r.step, name+"StepFunc")
r.final = append(r.final, name+"ValueFunc")
case "WINDOWFUNCNOOP":
s.reg(name).inline = "bytecode"
}
}
}
// readCreateFunctions reads the functions extensions register by calling
// sqlite3_create_function or sqlite3_create_window_function with a literal
// name.
func (s *source) readCreateFunctions(text string) {
for _, m := range createFunction.FindAllStringSubmatchIndex(text, -1) {
window := m[2] != -1
args, _, ok := arguments(text, m[1]-1)
if !ok || len(args) < 8 || !strings.HasPrefix(args[1], `"`) {
continue
}
r := s.reg(strings.Trim(args[1], `"`))
if window {
// (db, zName, nArg, eTextRep, pApp, xStep, xFinal, xValue, xInverse, xDestroy)
r.step = given(r.step, args[5])
r.final = given(r.final, args[6], args[7])
} else {
// (db, zName, nArg, eTextRep, pApp, xFunc, xStep, xFinal)
r.scalar = given(r.scalar, args[5])
r.step = given(r.step, args[6])
r.final = given(r.final, args[7])
}
}
}
// given appends the implementations a registration names, leaving out the
// methods it passes as 0 or NULL.
func given(impls []string, names ...string) []string {
for _, name := range names {
if name != "0" && name != "NULL" {
impls = append(impls, name)
}
}
return impls
}
// readTables reads the struct tables extensions walk to register their
// functions — geopoly's aFunc, FTS5's aBuiltin, FTS3's aOverload — each row
// an initializer holding the function's name and the C functions that
// implement it. Any C function in the row counts as an implementation;
// which reads arguments and which sets the result comes out in the
// reading. A row is consulted only for a name nothing else registered.
func (s *source) readTables(text string) {
for _, m := range initializer.FindAllStringSubmatch(text, -1) {
var impls []string
for _, id := range identifier.FindAllStringSubmatch(m[0], -1) {
if _, ok := s.funcs[s.resolve(id[1])]; ok {
impls = append(impls, id[1])
}
}
if len(impls) == 0 {
continue
}
name := strings.ToLower(m[1])
r, ok := s.regs[name]
if ok && !r.table {
continue
}
r = s.reg(name)
r.table = true
r.scalar = append(r.scalar, impls...)
}
}
// resolve follows #define aliases from a C function name to the one that
// is defined.
func (s *source) resolve(name string) string {
for i := 0; i < 8; i++ {
alias, ok := s.aliases[name]
if !ok {
return name
}
name = alias
}
return name
}
// noop reports a C function that does nothing: the stand-in for a function
// the VDBE implements in bytecode, whose result is one of its arguments.
func noop(name string) bool {
return strings.HasPrefix(name, "noop")
}
// The kinds a result or argument call names, by the sqlite3_result_* or
// sqlite3_value_* suffix with any 64 dropped. Suffixes not listed — error,
// subtype, type, bytes, dup — say nothing about a type.
var (
resultKinds = map[string]string{
"int": "integer", "double": "real",
"text": "text", "text16": "text", "text16le": "text", "text16be": "text",
"blob": "blob", "zeroblob": "blob",
"value": "any", "pointer": "any",
}
valueKinds = map[string]string{
"int": "integer", "double": "real",
"text": "text", "text16": "text", "text16le": "text", "text16be": "text",
"blob": "blob",
}
)
func kindOf(kinds map[string]string, suffix string) (string, bool) {
k, ok := kinds[strings.TrimSuffix(suffix, "64")]
return k, ok
}
// results collects the kinds a C function sets its result to, following
// the helpers it calls a few levels down, since many functions hand their
// result to one.
func (s *source) results(name string, depth int, seen map[string]bool) map[string]bool {
kinds := map[string]bool{}
name = s.resolve(name)
if noop(name) {
kinds["any"] = true
return kinds
}
fn, ok := s.funcs[name]
if !ok || seen[name] || depth > 3 {
return kinds
}
seen[name] = true
for _, m := range resultCall.FindAllStringSubmatch(fn.body, -1) {
if k, ok := kindOf(resultKinds, m[1]); ok {
kinds[k] = true
}
}
for _, m := range callee.FindAllStringSubmatch(fn.body, -1) {
c := m[1]
if c == name || strings.HasPrefix(c, "sqlite3_") || strings.HasPrefix(c, "sqlite3Vdbe") {
continue
}
if _, ok := s.funcs[c]; ok {
for k := range s.results(c, depth+1, seen) {
kinds[k] = true
}
}
}
return kinds
}
// args collects the kinds a C function reads each of its arguments as, by
// position, and the kind it reads a run of arguments as under a loop
// index. An implementation called through the FTS5 extension API is handed
// the table as an implicit first argument, so its positions shift by one.
func (s *source) args(name string, positions map[int]map[string]bool, variadic map[string]bool) {
fn, ok := s.funcs[s.resolve(name)]
if !ok {
return
}
shift := 0
if strings.Contains(fn.params, "Fts5ExtensionApi") {
shift = 1
if positions[0] == nil {
positions[0] = map[string]bool{}
}
positions[0]["any"] = true
}
for _, m := range valueCall.FindAllStringSubmatch(fn.body, -1) {
k, ok := kindOf(valueKinds, m[1])
if !ok {
continue
}
var pos int
if _, err := fmt.Sscanf(m[2], "%d", &pos); err != nil {
variadic[k] = true
continue
}
pos += shift
if positions[pos] == nil {
positions[pos] = map[string]bool{}
}
positions[pos][k] = true
}
}
// single reduces the kinds seen at one position to a type: the one kind
// seen, or "any" for a mixture or nothing.
func single(kinds map[string]bool) string {
if len(kinds) == 1 {
for k := range kinds {
return k
}
}
return "any"
}
// signature derives what the source says a SQL function returns and takes.
// A result of one kind is that type. A function that returns one of its
// arguments, or a mixture of kinds, takes the type of its first argument,
// which the seed spells "any" — except that integer and real together are
// real, and Numeric, since abs, ceil and sum hand back an integer for an
// integer and a real for a real, and a real for the text or blob they
// convert, unless the function is one of the few whose choice turns on
// something else; and text and blob together are text, since a function
// that returns either is handing back the bytes it was given, and the
// legacy compiler cannot follow "any" to an argument.
func (s *source) signature(name string) (signature, error) {
r, ok := s.regs[strings.ToLower(name)]
if !ok {
return signature{}, fmt.Errorf("the amalgamation registers no function named %s", name)
}
var sig signature
switch {
case r.inline != "":
sig.Returns = inlineReturns[r.inline]
if sig.Returns == "" {
sig.Returns = "any"
}
default:
kinds := map[string]bool{}
for _, fn := range append(append([]string{}, r.scalar...), r.final...) {
for k := range s.results(fn, 0, map[string]bool{}) {
kinds[k] = true
}
}
jsonOnly := r.json != "" && len(kinds) <= 2 && !kinds["integer"] && !kinds["real"] && !kinds["any"]
switch {
case len(kinds) == 0:
return signature{}, fmt.Errorf("cannot tell what %s returns: no sqlite3_result call in %s", name, strings.Join(append(append([]string{}, r.scalar...), r.final...), ", "))
case r.jsonAlways:
sig.Returns = "text"
case kinds["any"]:
sig.Returns = "any"
case len(kinds) == 1:
for k := range kinds {
sig.Returns = k
}
case jsonOnly:
// The shared implementation writes JSON text or JSONB; the
// registration says which this form gets.
sig.Returns = r.json
case len(kinds) == 2 && kinds["integer"] && kinds["real"]:
sig.Returns = "real"
sig.Numeric = !realResults[strings.ToLower(name)]
case len(kinds) == 2 && kinds["text"] && kinds["blob"]:
sig.Returns = "text"
default:
sig.Returns = "any"
}
}
positions := map[int]map[string]bool{}
variadic := map[string]bool{}
for _, fn := range append(append([]string{}, r.scalar...), r.step...) {
s.args(fn, positions, variadic)
}
if len(positions) > 0 {
var order []int
for pos := range positions {
order = append(order, pos)
}
sort.Ints(order)
sig.Args = make([]string, order[len(order)-1]+1)
for i := range sig.Args {
sig.Args[i] = single(positions[i])
}
// Trailing positions nothing typed say nothing, unless a variadic
// tail follows them, when they keep it from starting early.
for len(variadic) == 0 && len(sig.Args) > 0 && sig.Args[len(sig.Args)-1] == "any" {
sig.Args = sig.Args[:len(sig.Args)-1]
}
}
if len(variadic) == 1 {
sig.Variadic = single(variadic)
}
return sig, nil
}

444
internal/goldeneye/sqlite/sqlite.go Normal file
View File

@@ -0,0 +1,444 @@
// Package sqlite generates the SQLite dialect seed under
// internal/engine/sqlite/dialect from sqlite3 shells built from the
// amalgamation sqlite.org publishes, run against in-memory databases that
// need no server, and from the amalgamation itself.
//
// SQLite describes its functions as far as their names, their kinds and the
// number of arguments each takes — pragma_function_list — and no further:
// it types values rather than columns or functions, so nothing in the
// database says what a function returns or what it expects. The source
// does, in its way. Every function is registered with the C functions that
// implement it, and those set their result through sqlite3_result_* and
// read their arguments through sqlite3_value_*, which is as close as SQLite
// comes to declaring a signature. So functions.jsonl is built from both:
// the shell says which functions exist, how many arguments each overload
// takes and whether it aggregates, and the amalgamation says what each
// returns and what its arguments hold. A function the shell reports that
// the source does not register, or whose implementation sets no result,
// fails generation rather than being guessed at. What neither can say —
// which scalar functions return NULL for arguments that are not — is a
// short list in signatures.go; for aggregates it is found by running each
// over no rows.
//
// Which functions a SQLite has is decided when it is compiled, so the
// dialect treats compile options the way the PostgreSQL dialect treats
// contrib extensions. functions.jsonl is what a build with the default
// options has, and each further option gets a directory under extensions/
// holding the functions a build with that option adds — found the way
// CREATE EXTENSION's additions are, by comparing the catalog with and
// without. SQLite has no catalog of types or operators, so types.jsonl and
// operators.jsonl are hand-written.
//
// The shells are built once per pinned version by Install.
package sqlite
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"maps"
"os/exec"
"path"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
)
// Engine is the name of the engine directory the dialect lives under.
const Engine = "sqlite"
// functionList is every function the connection knows, one row per
// overload: name, whether the library builds it in or the shell or an
// extension registered it, 's', 'a' or 'w' for how it is called, and how
// many arguments it takes. Fixed arities come before the variable one so
// that an exact match is found first by anything reading the file in
// order.
const functionList = `
SELECT name, builtin, type, narg
FROM pragma_function_list
ORDER BY name, narg < 0, narg`
type functionRow struct {
Name string `json:"name"`
Builtin int `json:"builtin"`
Type string `json:"type"`
NArg int `json:"narg"`
}
// key tells one overload from another.
func (r functionRow) key() string {
return r.Name + "/" + strconv.Itoa(r.NArg)
}
// Version reports the release the default shell is.
func Version(ctx context.Context, dir string) (string, error) {
out, err := exec.CommandContext(ctx, builds()[0].binary(dir), "--version").Output()
if err != nil {
return "", fmt.Errorf("sqlite3 --version: %w", err)
}
return "SQLite " + strings.TrimSpace(string(out)), nil
}
// query runs a SQL statement against an in-memory database and decodes the
// shell's JSON output into rows.
func query(ctx context.Context, binary, sql string, rows any) error {
cmd := exec.CommandContext(ctx, binary, "-json", "-bail", ":memory:", sql)
out, err := cmd.Output()
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
return fmt.Errorf("sqlite3: %s: %s", err, bytes.TrimSpace(exit.Stderr))
}
return fmt.Errorf("sqlite3: %w", err)
}
// An empty result set prints nothing rather than [].
if len(bytes.TrimSpace(out)) == 0 {
return nil
}
return json.Unmarshal(out, rows)
}
// minArgs decodes a negative argument count from pragma_function_list,
// which prints the count SQLite keeps for the function as it is: -1 for any
// number of arguments, and the two values reserved for built-ins, -3 for
// one or more and -4 for two or more — see matchQuality in the SQLite
// source. No function is counted as -2.
func minArgs(narg int) int {
if narg < -2 {
return -2 - narg
}
return 0
}
// functionKind maps the type the shell reports onto the seed's letters.
// The shell says 's' for a scalar function, 'a' for an aggregate and 'w'
// for one with a window implementation — but every one of SQLite's
// aggregates has one, so 'w' covers avg and count as well as row_number.
// The two are told apart by calling the function without an OVER clause,
// which only a window function refuses.
func functionKind(ctx context.Context, binary string, row functionRow) (string, error) {
switch row.Type {
case "s":
return "", nil
case "a":
return "a", nil
case "w":
window, err := isWindowFunction(ctx, binary, row)
if err != nil {
return "", err
}
if window {
return "w", nil
}
return "a", nil
}
return "", fmt.Errorf("sqlite: function %s has unknown type %q", row.Name, row.Type)
}
// isWindowFunction reports whether the shell refuses to call a function
// without an OVER clause. Any other complaint — an aggregate objecting to
// the NULLs it is handed — means the call was accepted as an aggregate.
func isWindowFunction(ctx context.Context, binary string, row functionRow) (bool, error) {
n := row.NArg
if n < 0 {
n = minArgs(n)
}
args := strings.TrimSuffix(strings.Repeat("NULL, ", n), ", ")
cmd := exec.CommandContext(ctx, binary, ":memory:", fmt.Sprintf("SELECT %s(%s)", row.Name, args))
var stderr bytes.Buffer
cmd.Stdout = io.Discard
cmd.Stderr = &stderr
err := cmd.Run()
if err == nil {
return false, nil
}
if strings.Contains(stderr.String(), "misuse of window function") {
return true, nil
}
var exit *exec.ExitError
if errors.As(err, &exit) {
return false, nil
}
return false, fmt.Errorf("sqlite3: %w", err)
}
// shell is one built sqlite3 and the functions it reports.
type shell struct {
build build
binary string
rows []functionRow
}
// readShell lists a build's functions, after checking that the shell was
// built with the options the build says — a cached shell from before the
// option lists changed would otherwise describe the wrong dialect.
func readShell(ctx context.Context, dir string, b build) (*shell, error) {
s := &shell{build: b, binary: b.binary(dir)}
if err := checkOptions(ctx, dir, b); err != nil {
return nil, err
}
if err := query(ctx, s.binary, functionList, &s.rows); err != nil {
return nil, err
}
return s, nil
}
// checkOptions asks a build's shell whether it was compiled with every
// option any build turns on, and fails when the answers do not match the
// build's own list.
func checkOptions(ctx context.Context, dir string, b build) error {
var used []struct {
Option string `json:"option"`
Used int `json:"used"`
}
known := map[string]bool{}
for _, b := range shells() {
for _, opt := range b.flags() {
known[opt] = true
}
}
var clauses []string
for _, opt := range slices.Sorted(maps.Keys(known)) {
clauses = append(clauses, fmt.Sprintf("SELECT '%s' AS option, sqlite_compileoption_used('%s') AS used", opt, opt))
}
if err := query(ctx, b.binary(dir), strings.Join(clauses, " UNION ALL "), &used); err != nil {
return err
}
for _, u := range used {
want := 0
if slices.Contains(b.flags(), u.Option) {
want = 1
}
if u.Used != want {
return fmt.Errorf("sqlite: the %s shell was not built with the options it should have been (%s is %d): remove %s and run `go run ./cmd/goldeneye install sqlite` again", b.name, u.Option, u.Used, filepath.Join(dir, b.name))
}
}
return nil
}
// generator accumulates the functions of every build, reading their
// signatures from the amalgamation, and remembers which names were
// reported so that the lists in signatures.go can be checked against the
// shells at the end.
type generator struct {
ctx context.Context
src *source
reported map[string]bool
// numeric holds every spelling types.jsonl gives integer and real, in
// its order, for the functions written once per spelling.
numeric map[string][]string
}
// overload is one row of a shell's list with what was found out about it.
type overload struct {
row functionRow
kind string
sig signature
}
// functions turns rows into records, one per overload. Every row has to be
// a function the amalgamation registers unless omitted. In the default
// build only what the library builds in counts: the rest is what the shell
// registers on top — edit, sha3, the extensions it bundles — which is not
// the dialect's business, and which a comparison with the default build has
// already removed from an option's rows.
func (g *generator) functions(s *shell, base bool) ([]dialect.Function, error) {
var overloads []overload
seen := map[string]bool{}
var missing []string
for _, row := range s.rows {
g.reported[row.Name] = true
if omitted[row.Name] || seen[row.key()] || base && row.Builtin == 0 {
continue
}
seen[row.key()] = true
sig, err := g.src.signature(row.Name)
if err != nil {
if !seen[row.Name] {
seen[row.Name] = true
missing = append(missing, err.Error())
}
continue
}
kind, err := functionKind(g.ctx, s.binary, row)
if err != nil {
return nil, err
}
overloads = append(overloads, overload{row: row, kind: kind, sig: sig})
}
if len(missing) > 0 {
return nil, fmt.Errorf("sqlite: %s build: %s", s.build.name, strings.Join(missing, "; "))
}
empty, err := overNoRows(g.ctx, s.binary, overloads)
if err != nil {
return nil, err
}
funcs := make([]dialect.Function, 0, len(overloads))
for _, o := range overloads {
isNullable := nullable[o.row.Name]
if o.kind == "a" {
isNullable = empty[o.row.key()] == "null"
}
fn := dialect.Function{
Name: o.row.Name,
Kind: o.kind,
Args: o.sig.args(o.row.NArg),
Returns: o.sig.Returns,
Nullable: isNullable,
}
funcs = append(funcs, fn)
// A function that returns an integer for an integer and a real for
// a real is written once more per spelling of each, the way
// PostgreSQL's catalog has a sum per numeric type. The overload over
// any comes first: the analysis core picks the overload whose
// parameter is the argument's type, but the legacy compiler takes
// the first of the right arity, and keeps what it had.
if o.sig.Numeric && len(fn.Args) > 0 {
for _, typ := range []string{"integer", "real"} {
for _, spelling := range g.numeric[typ] {
typed := fn
typed.Args = append([]dialect.Arg{{Type: spelling}}, fn.Args[1:]...)
typed.Returns = typ
funcs = append(funcs, typed)
}
}
}
}
return funcs, nil
}
// overNoRows runs every aggregate over no rows and reports the type of
// what each returns, "null" for the ones — avg, max, group_concat — that
// return NULL when there is nothing to aggregate, and not count or total.
// The probes go through one shell process, each statement labelled with
// its overload so that the answers can be told apart.
func overNoRows(ctx context.Context, binary string, overloads []overload) (map[string]string, error) {
var script strings.Builder
for _, o := range overloads {
if o.kind != "a" {
continue
}
n := o.row.NArg
if n < 0 {
n = minArgs(n)
}
args := strings.TrimSuffix(strings.Repeat("x, ", n), ", ")
fmt.Fprintf(&script, "SELECT '%s', typeof(%s(%s)) FROM (SELECT NULL AS x) WHERE 0;\n", o.row.key(), o.row.Name, args)
}
results := map[string]string{}
if script.Len() == 0 {
return results, nil
}
cmd := exec.CommandContext(ctx, binary, "-list", ":memory:", script.String())
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
var exit *exec.ExitError
if !errors.As(err, &exit) {
return nil, fmt.Errorf("sqlite3: %w", err)
}
}
for _, line := range strings.Split(strings.TrimSpace(stdout.String()), "\n") {
key, typ, ok := strings.Cut(line, "|")
if ok {
results[key] = typ
}
}
for _, o := range overloads {
if o.kind == "a" {
if _, ok := results[o.row.key()]; !ok {
return nil, fmt.Errorf("sqlite: cannot run %s over no rows: %s", o.row.Name, strings.TrimSpace(stderr.String()))
}
}
}
return results, nil
}
// added returns the rows of an option's shell that the default shell does
// not have: what the option adds.
func added(opt, base *shell) *shell {
have := map[string]bool{}
for _, row := range base.rows {
have[row.key()] = true
}
diff := &shell{build: opt.build, binary: opt.binary}
for _, row := range opt.rows {
if !have[row.key()] {
diff.rows = append(diff.rows, row)
}
}
return diff
}
// Generate reads the dialect from the shells under dir and the
// amalgamation they were built from.
func Generate(ctx context.Context, dir string) (dialect.Files, error) {
src, err := readSource(filepath.Join(dir, "src", "sqlite3.c"))
if err != nil {
return nil, err
}
g := &generator{ctx: ctx, src: src, reported: map[string]bool{}, numeric: map[string][]string{}}
dialectDir, err := dialect.Dir(Engine)
if err != nil {
return nil, err
}
types, err := dialect.ReadTypes(dialectDir)
if err != nil {
return nil, err
}
for _, t := range types {
if t.Name == "integer" || t.Name == "real" {
g.numeric[t.Name] = append([]string{t.Name}, t.Aliases...)
}
}
all := builds()
base, err := readShell(ctx, dir, all[0])
if err != nil {
return nil, err
}
funcs, err := g.functions(base, true)
if err != nil {
return nil, err
}
files := dialect.Files{}
if files[dialect.FunctionsFile], err = dialect.JSONL(funcs); err != nil {
return nil, err
}
for _, b := range all[1:] {
s, err := readShell(ctx, dir, b)
if err != nil {
return nil, err
}
funcs, err := g.functions(added(s, base), false)
if err != nil {
return nil, err
}
if len(funcs) == 0 {
return nil, fmt.Errorf("sqlite: %s adds no functions over the default build; drop it from the extensions in install.go", strings.Join(b.options, " "))
}
if files[path.Join(dialect.ExtensionsDir, b.name, dialect.FunctionsFile)], err = dialect.JSONL(funcs); err != nil {
return nil, err
}
}
var stale []string
for _, list := range []map[string]bool{omitted, nullable, realResults} {
for name := range list {
if !g.reported[name] {
stale = append(stale, name)
}
}
}
if len(stale) > 0 {
sort.Strings(stale)
return nil, fmt.Errorf("sqlite: signatures.go lists function(s) no shell reports: %s", strings.Join(stale, ", "))
}
return files, nil
}

66
internal/goldeneye/sqlite/sqlite_test.go Normal file
View File

@@ -0,0 +1,66 @@
package sqlite
import (
"context"
"testing"
"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
)
// TestDialect verifies the committed SQLite dialect against what the pinned
// sqlite3 shell reports. It skips unless the shell is installed.
func TestDialect(t *testing.T) {
binary, err := Locate()
if err != nil {
t.Skip(err)
}
ctx := context.Background()
version, err := Version(ctx, binary)
if err != nil {
t.Fatal(err)
}
files, err := Generate(ctx, binary)
if err != nil {
t.Fatal(err)
}
dir, err := dialect.Dir(Engine)
if err != nil {
t.Fatal(err)
}
report, err := dialect.Check(dir, files)
if err != nil {
t.Fatal(err)
}
if report != "" {
t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
}
}
// TestAnalyzeCases verifies every SQLite analyze case under
// internal/endtoend/testdata against what SQLite reports. It skips unless
// the shells are installed.
func TestAnalyzeCases(t *testing.T) {
dir, err := Locate()
if err != nil {
t.Skip(err)
}
cases, err := endtoend.Cases(Engine)
if err != nil {
t.Fatal(err)
}
if len(cases) == 0 {
t.Fatal("no sqlite analyze cases found")
}
for _, c := range cases {
t.Run(c.Name, func(t *testing.T) {
diff, err := Check(context.Background(), dir, c)
if err != nil {
t.Fatal(err)
}
if diff != "" {
t.Errorf("%s does not match what SQLite reports (-committed +sqlite):\n%s", c.Output, diff)
}
})
}
}