Skip to content

BigQuery Source Connector Code #599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

raheelkumarpersistent
Copy link

Created a Source Connector for BigQuery which takes a SELECT query from the user and a Service Account credentials file to fetch data from BigQuery and store in Quix Topics.

@tomas-quix tomas-quix requested a review from Copilot April 1, 2025 09:21
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a new BigQuery Source Connector that extracts data from BigQuery using a user-defined SQL query and publishes it to a Kafka topic using Quix Streams.

  • Implements custom JSON encoding and serialization for handling various BigQuery data types
  • Integrates with Quix Streams to publish data to Kafka
  • Adds a README with detailed usage, configuration, and setup instructions

Reviewed Changes

Copilot reviewed 2 out of 5 changed files in this pull request and generated no comments.

File Description
python/sources/BigQuery_source/main.py New implementation for BigQuery data extraction and publishing
python/sources/BigQuery_source/README.md Documentation detailing connector usage and setup
Files not reviewed (3)
  • python/sources/BigQuery_source/dockerfile: Language not supported
  • python/sources/BigQuery_source/library.json: Language not supported
  • python/sources/BigQuery_source/requirements.txt: Language not supported
Comments suppressed due to low confidence (2)

python/sources/BigQuery_source/main.py:138

  • The check 'header not in df.columns' is redundant since 'headers' is derived from df.columns. Consider removing this condition.
if pd.isna(value) and header not in df.columns:

python/sources/BigQuery_source/main.py:35

  • [nitpick] For consistency with the rest of the code, consider using the imported 'time' type (i.e., 'if isinstance(obj, time):') instead of 'datetime.time'.
if isinstance(obj, datetime.time):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant