Skip to main content

Documentation Index

Fetch the complete documentation index at: https://private-7c7dfe99-page-updates.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

Azure Synapse is an integrated analytics service that combines big data, data science and warehousing to enable fast, large-scale data analysis. Within Synapse, Spark pools provide on-demand, scalable Apache Spark clusters that let you run complex data transformations, machine learning, and integrations with external systems. This article will show you how to integrate the ClickHouse Spark connector when working with Apache Spark within Azure Synapse.

​
Add the connector’s dependencies

Azure Synapse supports three levels of packages maintenance:
  1. Default packages
  2. Spark pool level
  3. Session level

Follow the Manage libraries for Apache Spark pools guide and add the following required dependencies to your Spark application
  • clickhouse-spark-runtime-{spark_version}_{scala_version}-{connector_version}.jar - official maven
  • clickhouse-jdbc-{java_client_version}-all.jar - official maven
Please visit our Spark Connector Compatibility Matrix docs to understand which versions suit your needs.

​
Add ClickHouse as a catalog

There are a variety of ways to add Spark configs to your session:
  • Custom configuration file to load with your session
  • Add configurations via Azure Synapse UI
  • Add configurations in your Synapse notebook
Follow this Manage Apache Spark configuration and add the connector required Spark configurations. For instance, you can configure your Spark session in your notebook with these settings: 
%%configure -f
{
    "conf": {
        "spark.sql.catalog.clickhouse": "com.clickhouse.spark.ClickHouseCatalog",
        "spark.sql.catalog.clickhouse.host": "<clickhouse host>",
        "spark.sql.catalog.clickhouse.protocol": "https",
        "spark.sql.catalog.clickhouse.http_port": "<port>",
        "spark.sql.catalog.clickhouse.user": "<username>",
        "spark.sql.catalog.clickhouse.password": "password",
        "spark.sql.catalog.clickhouse.database": "default"
    }
}
Make sure it will be in the first cell as follows: Please visit the ClickHouse Spark configurations page for additional settings.
When working with ClickHouse Cloud Please make sure to set the required Spark settings.

​
Setup verification

To verify that the dependencies and configurations were set successfully, please visit your session’s Spark UI, and go to your Environment tab. There, look for your ClickHouse related settings:

​
Additional resources