feat(clickpipes): BigQuery source support #413
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jkaflik
commented
Dec 2, 2025
This PR enables creating BigQuery ClickPipe. This is in private preview.
jkaflik
force-pushed
the
kk/clickpipes-bigquery
branch
from
7ae8fca to
c40f64f
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds support for BigQuery as a source for ClickPipe snapshots, currently in private preview. The implementation includes schema definitions, resource handling, and state synchronization for BigQuery configurations.
Key changes:
- Adds BigQuery source models and schema definitions with support for credentials, settings, and table mappings
- Updates source type detection and handling logic to treat BigQuery as a database pipe alongside Postgres
- Provides a complete example with GCP infrastructure setup for BigQuery ClickPipe integration
Reviewed changes
Copilot reviewed 11 out of 31 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/resource/models/clickpipe_resource.go | Defines model structures for BigQuery source, settings, table mappings, and service account credentials |
| pkg/resource/clickpipe.go | Implements BigQuery schema, extraction logic, and integrates it into the ClickPipe resource lifecycle |
| pkg/internal/api/clickpipe_models.go | Adds API model definitions for BigQuery source configuration and credentials |
| pkg/internal/api/clickpipe.go | Normalizes replication mode and table engine constants to be shared between Postgres and BigQuery |
| examples/resources/clickhouse_clickpipe/resource_bigquery_snapshot.tf | Provides inline documentation example for BigQuery ClickPipe configuration |
| examples/clickpipe/bigquery_snapshot/vars.tf | Defines input variables for the BigQuery example |
| examples/clickpipe/bigquery_snapshot/variables.tfvars.sample | Sample variable values for the BigQuery example |
| examples/clickpipe/bigquery_snapshot/provider.tf.template.alpha | Provider configuration template for the example |
| examples/clickpipe/bigquery_snapshot/provider.tf | Generated provider configuration with version pinning |
| examples/clickpipe/bigquery_snapshot/gcp.tf | GCP infrastructure setup including service account, IAM roles, and staging bucket |
| examples/clickpipe/bigquery_snapshot/clickpipe.tf | Complete example of BigQuery ClickPipe resource configuration |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
@jkaflik we have a short |
Member
Author
|
@EvandroLG yes. Forgot to push. |
EvandroLG
reviewed
Dec 5, 2025
EvandroLG
approved these changes
Dec 5, 2025
ilidemi
approved these changes
Dec 6, 2025
Member
ilidemi
left a comment
ilidemi
left a comment
There was a problem hiding this comment.
Agreed with others' comments, looks good otherwise.
Also Postgres has tests, does BQ need them too?
whites11
approved these changes
Dec 9, 2025
nunoadrego
approved these changes
Dec 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support for BigQuery snapshot ClickPipe creation, currently in private preview.
Comes with a self-contained example and a short, in-line documentation example.
Extra in this PR, I normalize how sample tfvars are organized for ClickPipes examples.