Add ClickHouse Engine Support to sqlc #4220
Open
+17,434
−295
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces ClickHouse SQL parser integration using github.com/AfterShip/clickhouse-sql-parser. Includes AST converter to sqlc's internal representation, catalog initialization, and type resolver. Foundation for ClickHouse support.
Comprehensive test coverage for parser and converter: - Basic parsing functionality (parse_test.go) - AST conversion correctness (new_conversions_test.go) - Catalog initialization (catalog_test.go) - Edge cases and boundary conditions (parse_boundary_test.go) - Real-world queries (parse_actual_queries_test.go) - Type handling, identifiers, joins, arrays
Add engine registration in compiler pipeline and configuration schema. Enables compiler to recognize and process ClickHouse engine configurations.
Map ClickHouse types to Go types, update driver handling, and add import management for ClickHouse types.
Templates for generating Go database code: - Database interface template - Interface declarations - Query function implementations - Import resolution tests
Implement JOIN...USING clause support for ClickHouse and PostgreSQL. Refactor output_columns.go for improved type resolution. Add comprehensive tests for output columns and type resolution. Update quote character handling and catalog interface.
Docker and local connection adapters for integration testing. Enables end-to-end testing with real ClickHouse instances.
Getting started guide, configuration and datatype reference updates, example project with schema and queries, development guide updates, Docker Compose configuration for ClickHouse service.
Generated Go code output from running sqlc generate on the ClickHouse example project: database interface, model types, and query functions (1264 lines of generated code).
PostgreSQL test case validating JOIN USING feature works correctly with existing database engines.
Tests for basic ClickHouse functionality: SELECT queries, DML operations (INSERT/UPDATE/DELETE), DISTINCT, ORDER BY, LIMIT/OFFSET pagination, NULL handling, and string functions. Each test includes schema, queries, and generated Go code.
Tests for complex SQL constructs: JOIN operations (explicit and implicit), subqueries (including IN subqueries), common table expressions (CTEs), aggregate functions, GROUP BY with HAVING, and EXISTS operator. Each test includes schema, queries, and generated Go code.
Tests for type handling and built-in functions: array types, type casting, string/math functions, named parameters, DELETE/UPDATE statements, and advanced feature combinations. Includes test runner integration updates. Final comprehensive validation of ClickHouse engine support.
dosubot
bot
added
size:XXL
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces comprehensive support for ClickHouse as a database engine in sqlc. It enables users to write ClickHouse queries in
.sqlfiles and generate type-safe Go code using sqlc's familiar workflow.Key Features
github.com/AfterShip/clickhouse-sql-parserlibraryJOIN...USINGsyntax, working with both ClickHouse and PostgreSQLProduction Status
This implementation has been in production use for approximately one month. The core parser and code generation have proven stable for our queries, but I'm eager for community feedback on edge cases and other query patterns.
Large PR Notice
/internal/engine/clickhouse//internal/codegen/golang/templates/clickhouse/output_columns.go(backward compatible)JOIN...USINGsupport (benefits multiple engines)The majority of the PR size comes from test data and generated code examples, which are valuable for validation and user documentation.
Feedback Welcome
I'm committed to making this PR production-ready and welcome:
If you have ClickHouse queries that don't work with this implementation, please share them. I'm particularly interested in:
Commit Structure
The 13 commits are organized to facilitate review, with test data grouped at the end:
1. Core Engine Implementation (2 commits)
Add ClickHouse engine: parser, converter, and catalog- Foundation for ClickHouse SQL parsingAdd ClickHouse engine unit tests- Comprehensive unit tests for the parser2. Integration Layer (2 commits)
Register ClickHouse engine in compiler- Wires the engine into the compiler pipelineAdd ClickHouse type mapping for Go code generation- Maps ClickHouse types to Go types3. Code Generation (2 commits)
Add ClickHouse code generation templates- Templates for generating Go codeAdd JOIN USING support and refactor output column handling- New SQL feature and compiler improvements4. Infrastructure & Documentation (2 commits)
Add ClickHouse test database adapters- Docker and local test containersAdd ClickHouse documentation and example project- User-facing docs and complete working example5. Generated Code & Examples (1 commit)
Add ClickHouse example project generated code- Shows what users get fromsqlc generate6. End-to-End Tests (4 commits)
Add end-to-end test for JOIN...USING syntax- Validates feature works with PostgreSQLAdd end-to-end tests for ClickHouse core SQL features- SELECT, DML, pagination, etc.Add end-to-end tests for ClickHouse advanced SQL features- JOINs, subqueries, CTEs, aggregatesAdd end-to-end tests for ClickHouse types and functions- Type handling, arrays, functions, named parametersReview Strategy: Early commits establish the core logic before test data, making the implementation easy to understand and validate.
What's Included
/examples/clickhouse/Review suggestion
Looking forward to your feedback and excited to bring ClickHouse support to sqlc!