Skip to main content

Documentation Index

Fetch the complete documentation index at: https://private-7c7dfe99-page-updates.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

Google Dataflow is a fully managed stream and batch data processing service. It supports pipelines written in Java or Python and is built on the Apache Beam SDK. There are two main ways to use Google Dataflow with ClickHouse, both of which leverage ClickHouseIO Apache Beam connector. These are:

​
Java runner

The Java runner allows you to implement custom Dataflow pipelines using the Apache Beam SDK ClickHouseIO integration. This approach provides full flexibility and control over the pipeline logic, enabling you to tailor the ETL process to specific requirements. However, this option requires knowledge of Java programming and familiarity with the Apache Beam framework.

​
Key features

  • High degree of customization.
  • Ideal for complex or advanced use cases.
  • Requires coding and understanding of the Beam API.

​
Predefined templates

ClickHouse offers predefined templates designed for specific use cases, such as importing data from BigQuery into ClickHouse. These templates are ready-to-use and simplify the integration process, making them an excellent choice if you prefer a no-code solution.

​
Key features

  • No Beam coding required.
  • Quick and easy setup for simple use cases.
  • Also suitable if you have minimal programming expertise.
Both approaches are fully compatible with Google Cloud and the ClickHouse ecosystem, offering flexibility depending on your technical expertise and project requirements.