Composable protocols

Composable protocols allows more flexible configuration of TCP access to the ClickHouse server.

Overview

Composable protocols allow more flexible configuration of TCP access to the ClickHouse server. This configuration can co-exist alongside, or replace, conventional configuration.

Configuring composable protocols

Composable protocols can be configured in an XML configuration file. The protocols section is denoted with protocols tags in the XML config file:

<protocols>

</protocols>

Configuring protocol layers

You can define protocol layers using basic modules. For example, to define an HTTP layer, you can add a new basic module to the protocols section:

<protocols>

  <!-- plain_http module -->
  <plain_http>
    <type>http</type>
  </plain_http>

</protocols>

Modules can be configured according to:

• plain_http - name which can be referred to by another layer
• type - denotes the protocol handler which will be instantiated to process data. It has the following set of predefined protocol handlers:
  • tcp - native clickhouse protocol handler
  • http - HTTP clickhouse protocol handler
  • tls - TLS encryption layer
  • proxy1 - PROXYv1 layer
  • mysql - MySQL compatibility protocol handler
  • postgres - PostgreSQL compatibility protocol handler
  • prometheus - Prometheus protocol handler
  • interserver - clickhouse interserver handler

gRPC protocol handler is not implemented for Composable protocols

Configuring endpoints

Endpoints (listening ports) are denoted by <port> and optional <host> tags. For example, to configure an endpoint on the previously added HTTP layer we could modify our configuration as follows:

<protocols>

  <plain_http>

    <type>http</type>
    <!-- endpoint -->
    <host>127.0.0.1</host>
    <port>8123</port>

  </plain_http>

</protocols>

If the <host> tag is omitted, then the <listen_host> from the root config is used.

Configuring layer sequences

Layers sequences are defined using the <impl> tag, and referencing another module. For example, to configure a TLS layer on top of our plain_http module we could further modify our configuration as follows:

<protocols>

  <!-- http module -->
  <plain_http>
    <type>http</type>
  </plain_http>

  <!-- https module configured as a tls layer on top of plain_http module -->
  <https>
    <type>tls</type>
    <impl>plain_http</impl>
    <host>127.0.0.1</host>
    <port>8443</port>
  </https>

</protocols>

Attaching endpoints to layers

Endpoints can be attached to any layer. For example, we can define endpoints for HTTP (port 8123) and HTTPS (port 8443):

<protocols>

  <plain_http>
    <type>http</type>
    <host>127.0.0.1</host>
    <port>8123</port>
  </plain_http>

  <https>
    <type>tls</type>
    <impl>plain_http</impl>
    <host>127.0.0.1</host>
    <port>8443</port>
  </https>

</protocols>

Defining additional endpoints

Additional endpoints can be defined by referencing any module and omitting the <type> tag. For example, we can define another_http endpoint for the plain_http module as follows:

<protocols>

  <plain_http>
    <type>http</type>
    <host>127.0.0.1</host>
    <port>8123</port>
  </plain_http>

  <https>
    <type>tls</type>
    <impl>plain_http</impl>
    <host>127.0.0.1</host>
    <port>8443</port>
  </https>

  <another_http>
    <impl>plain_http</impl>
    <host>127.0.0.1</host>
    <port>8223</port>
  </another_http>

</protocols>

Custom HTTP handlers per endpoint

By default, all type=http protocol entries share the same <http_handlers> configuration. You can override this by adding a <handlers> tag that points to a different configuration section. This allows each HTTP port to serve a different set of HTTP routing rules.

For example, to run an alternative HTTP API on port 8124 with its own handlers:

<protocols>

  <plain_http>
    <type>http</type>
    <host>127.0.0.1</host>
    <port>8123</port>
  </plain_http>

  <alt_http>
    <type>http</type>
    <host>127.0.0.1</host>
    <port>8124</port>
    <handlers>http_handlers_alt</handlers>
  </alt_http>

</protocols>

<!-- Default handlers used by plain_http (port 8123) -->
<http_handlers>
    <defaults/>
</http_handlers>

<!-- Alternative handlers used by alt_http (port 8124) -->
<http_handlers_alt>
    <rule>
        <url>/custom</url>
        <handler>
            <type>predefined_query_handler</type>
            <query>SELECT 'custom_endpoint'</query>
        </handler>
    </rule>
    <defaults/>
</http_handlers_alt>

In this example, requests to port 8123 use the standard <http_handlers> rules, while requests to port 8124 use the <http_handlers_alt> rules. If <handlers> is omitted, the endpoint falls back to the default <http_handlers>.

The custom handlers section follows the same format as <http_handlers>. Changes to the custom handlers section are detected during config reload, and the corresponding endpoint is automatically restarted.

Specifying additional layer parameters

Some modules can contain additional layer parameters. For example, the TLS layer allows a private key (privateKeyFile) and certificate files (certificateFile) to be specified as follows:

<protocols>

  <plain_http>
    <type>http</type>
    <host>127.0.0.1</host>
    <port>8123</port>
  </plain_http>

  <https>
    <type>tls</type>
    <impl>plain_http</impl>
    <host>127.0.0.1</host>
    <port>8443</port>
    <privateKeyFile>another_server.key</privateKeyFile>
    <certificateFile>another_server.crt</certificateFile>
  </https>

</protocols>