Monitoring ClickPipes

Monitor ClickPipes using Prometheus metrics.

In addition to in-console monitoring, ClickPipes exposes metrics to a Prometheus-compatible endpoint for scraping. These metrics are published with other ClickHouse Cloud service metrics, and allow you to integrate ClickPipes monitoring with your existing observability stack (e.g., Grafana, Datadog).

Metric labels

ClickPipes metrics use the standard service-level labels (clickhouse_org, clickhouse_service, clickhouse_service_name) plus the following ClickPipes-specific labels:

Label	Description
clickpipe_id	The unique identifier of the ClickPipe
clickpipe_name	The name of the ClickPipe
clickpipe_source	The source type (e.g., kafka, postgres, s3)
clickpipe_state	The state of the ClickPipe (e.g., Stopped, Provisioning, Running)

Sample response

The following shows what ClickPipes metrics look like in a Prometheus scrape response:

# HELP ClickPipes_Info Always equal to 1. Label "clickpipe_state" contains the current state of the pipe: Stopped/Provisioning/Running/Paused/Failed
# TYPE ClickPipes_Info gauge
ClickPipes_Info{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent",clickpipe_state="Running"} 1

# HELP ClickPipes_FetchedEvents_Total Total number of records fetched from the source.
# TYPE ClickPipes_FetchedEvents_Total counter
ClickPipes_FetchedEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5535376

# HELP ClickPipes_SentEvents_Total Total number of records sent to ClickHouse
# TYPE ClickPipes_SentEvents_Total counter
ClickPipes_SentEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5534250

# HELP ClickPipes_FetchedBytes_Total Total uncompressed bytes fetched from the source.
# TYPE ClickPipes_FetchedBytes_Total counter
ClickPipes_FetchedBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 873286202

# HELP ClickPipes_SentBytes_Total Total uncompressed bytes sent to ClickHouse.
# TYPE ClickPipes_SentBytes_Total counter
ClickPipes_SentBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 477187967

# HELP ClickPipes_Errors_Total Total errors ingesting data.
# TYPE ClickPipes_Errors_Total counter
ClickPipes_Errors_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 0

# HELP ClickPipes_Latency Time in milliseconds between when the record was produced to when it was written to ClickHouse.
# TYPE ClickPipes_Latency gauge
ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 340

Available metrics

Data transfer

Metric	Type	Description
ClickPipes_FetchedBytes_Total	Counter	Total uncompressed bytes fetched from the source.
ClickPipes_FetchedBytesCompressed_Total	Counter	Total compressed bytes fetched from the source. If the source data is uncompressed, equivalent to ClickPipes_FetchedBytes_Total.
ClickPipes_FetchedBytesInitialLoad_Total	Counter	CDC ClickPipes Total uncompressed bytes fetched during the initial load phase.
ClickPipes_FetchedBytesResync_Total	Counter	CDC ClickPipes Total uncompressed bytes fetched during a resync operation.
ClickPipes_SentBytes_Total	Counter	Total uncompressed bytes sent to ClickHouse.
ClickPipes_SentBytesCompressed_Total	Counter	Total compressed bytes sent to ClickHouse.

Events and records

Metric	Type	Description
ClickPipes_FetchedEvents_Total	Counter	Total number of records fetched from the source.
ClickPipes_SentEvents_Total	Counter	Total number of records sent to ClickHouse.
ClickPipes_FetchedEvents_PerTable	Gauge	CDC ClickPipes Number of records fetched per destination table and event type. Uses additional destination_table and event_type labels.

Errors

Metric	Type	Description
ClickPipes_Errors_Total	Counter	Total errors encountered during ingestion.

Latency

Metric	Type	Description
ClickPipes_Latency	Gauge (avg)	Time in milliseconds between when a record was produced at the source and when it was written to ClickHouse.
ClickPipes_SourceReplicationLatency_MiB	Gauge (avg)	CDC ClickPipes Replication lag at the source in MiB. For Postgres ClickPipes, this reflects the replication slot lag.

Resource usage

Metric	Type	Description
ClickPipes_Replica_CPUUsage	Gauge (avg)	Average CPU usage of the ingestion replicas, in millicores.
ClickPipes_Replica_CPULimit	Gauge (last)	Configured CPU limit of the ingestion replicas, in millicores.
ClickPipes_Replica_MemoryUsage	Gauge (avg)	Average memory usage of the ingestion replicas, in bytes.
ClickPipes_Replica_MemoryLimit	Gauge (last)	Configured memory limit of the ingestion replicas, in bytes.

State and progress

Metric	Type	Description
ClickPipes_Info	Gauge	Always equal to 1. Use the clickpipe_state label to filter or alert on pipe state.
ClickPipes_LastFetchedBatchId	Gauge	ID of the last batch fetched from the source.
ClickPipes_LastSentBatchId	Gauge	ID of the last batch sent to ClickHouse.

Related pages

• Prometheus endpoint — Endpoint reference, authentication, and setup for external tools (e.g., Grafana, Datadog)
• Error reporting — Where record and system errors are stored