Add sqlc templates for arbitrary text substitution#792
Closed
Add sqlc templates for arbitrary text substitution#792
Conversation
d78d533 to
b054626
Compare
bgentry
reviewed
Mar 6, 2025
Contributor
bgentry
left a comment
There was a problem hiding this comment.
Solid, this should help with a few different areas including unlocking full support for schemas and some other stuff in the works. Let's get it over the line ![]()
Contributor
There was a problem hiding this comment.
I assume you're intending to fill this out a bit?
Contributor
Author
There was a problem hiding this comment.
Yeah, just wanted to get your reaction before I spent too much time on it in case we decided to go a different direction.
This one aims to give us a workable resolution to one of our most common
problems with sqlc. Namely, that although it allows substitution for
parameters that work with a prepared query, it can't replace arbitrary
parts of a SQL query, leading to operations that aren't possible so that
we either don't do them or end up degrading to raw SQL that's only
checked at runtime.
Here, we add a `sqlctemplate` package that's designed to be run from
inside custom implementations of sqlc's `DBTX` interface so that it it
runs after sqlc's generated code but before the query goes to Postgres.
In sqlc code, templates look like this:
-- name: JobCountByState :one
SELECT count(*)
FROM /* TEMPLATE: schema */river_job
WHERE state = @State;
The template replacement is modeled as a comment so that it doesn't
interfere with with sqlc's parsing of SQL syntax. The above is valid SQL
with or without the template, but with it, `sqlctemplate` can add an
arbitrary schema name to the queried table.
It also supports a form of syntax where a value is required for SQL to
be valid. For example, `WHERE` and `ORDER BY` clauses both require a
value for them to be valid. Here, a stand in value is provide between
template tags. It's processed by sqlc's parser, but then replace by the
template engine before the SQL is executed:
-- name: JobList :many
SELECT *
FROM river_job
WHERE /* TEMPLATE_BEGIN: where_clause */ 1 /* TEMPLATE_END */
ORDER BY /* TEMPLATE_BEGIN: order_by_clause */ id /* TEMPLATE_END */
LIMIT @max::int;
Template values are injected via context (don't love this, but there's
no other way in getting information down to a layer below `DBTX`):
ctx = sqlctemplate.WithTemplates(ctx, map[string]sqlctemplate.Replacement{
"order_by_clause": {Value: params.OrderByClause},
"where_clause": {Value: params.WhereClause},
}, params.NamedArgs)
jobs, err := dbsqlc.New().JobList(ctx, e.dbtx, params.Max)
The template engine is written to be root out as many error as possible
by noticing if a template replacement is passed that doesn't have an
equivalent template in SQL, or if a template in SQL is present for which
there's no replacement.
Named args are support in templates similar to how sqlc supports them.
This allows pgx's prepared statement cache to continue to operate as it
did before, thereby keeping everything fast.
Lastly, I should note that templates are meant as a utility of last
resort. All effort should be made to resolve problems via mainstream
sqlc, and only bring in templates when there's no other option.
b054626 to
a45b502
Compare
Contributor
Author
|
Going to keep this around for now as reference for a while longer, but superseded by #794. |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This one aims to give us a workable resolution to one of our most common
problems with sqlc. Namely, that although it allows substitution for
parameters that work with a prepared query, it can't replace arbitrary
parts of a SQL query, leading to operations that aren't possible so that
we either don't do them or end up degrading to raw SQL that's only
checked at runtime.
Here, we add a
sqlctemplatepackage that's designed to be run frominside custom implementations of sqlc's
DBTXinterface so that it itruns after sqlc's generated code but before the query goes to Postgres.
In sqlc code, templates look like this:
The template replacement is modeled as a comment so that it doesn't
interfere with with sqlc's parsing of SQL syntax. The above is valid SQL
with or without the template, but with it,
sqlctemplatecan add anarbitrary schema name to the queried table.
It also supports a form of syntax where a value is required for SQL to
be valid. For example,
WHEREandORDER BYclauses both require avalue for them to be valid. Here, a stand in value is provide between
template tags. It's processed by sqlc's parser, but then replace by the
template engine before the SQL is executed:
Template values are injected via context (don't love this, but there's
no other way in getting information down to a layer below
DBTX):The template engine is written to be root out as many error as possible
by noticing if a template replacement is passed that doesn't have an
equivalent template in SQL, or if a template in SQL is present for which
there's no replacement.
Named args are support in templates similar to how sqlc supports them.
This allows pgx's prepared statement cache to continue to operate as it
did before, thereby keeping everything fast.
Lastly, I should note that templates are meant as a utility of last
resort. All effort should be made to resolve problems via mainstream
sqlc, and only bring in templates when there's no other option.