system.kafka_consumers

System table containing information about Kafka consumers.

Querying in ClickHouse Cloud

The data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the clusterAllReplicas function. See here for further details.

Description

Contains information about Kafka consumers. Applicable for Kafka table engine (native ClickHouse integration)

Columns

• database (String) — Database of the table with Kafka Engine.
• table (String) — Name of the table with Kafka Engine.
• consumer_id (String) — Kafka consumer identifier. Note, that a table can have many consumers. Specified by kafka_num_consumers parameter.
• assignments.topic (Array(String)) — Kafka topic.
• assignments.partition_id (Array(Int32)) — Kafka partition id. Note, that only one consumer can be assigned to a partition.
• assignments.current_offset (Array(Int64)) — Current offset.
• assignments.intent_size (Array(Nullable(Int64))) — The number of pushed, but not yet committed messages in new StorageKafka.
• exceptions.time (Array(DateTime)) — Timestamp when the 10 most recent exceptions were generated.
• exceptions.text (Array(String)) — Text of 10 most recent exceptions.
• last_poll_time (DateTime) — Timestamp of the most recent poll.
• num_messages_read (UInt64) — Number of messages read by the consumer.
• last_commit_time (DateTime) — Timestamp of the most recent poll.
• num_commits (UInt64) — Total number of commits for the consumer.
• last_rebalance_time (DateTime) — Timestamp of the most recent Kafka rebalance.
• num_rebalance_revocations (UInt64) — Number of times the consumer was revoked its partitions.
• num_rebalance_assignments (UInt64) — Number of times the consumer was assigned to Kafka cluster.
• is_currently_used (UInt8) — The flag which shows whether the consumer is in use.
• last_used (DateTime64(6)) — The last time this consumer was in use.
• rdkafka_stat (String) — Library internal statistic. Set statistics_interval_ms to 0 disable, default is 3000 (once in three seconds).
• dependencies (Array(Array(String))) — Transitive database dependencies.
• missing_dependencies (Array(Array(String))) — Missing transitive database dependencies.

Example:

SELECT *
FROM system.kafka_consumers
FORMAT Vertical

Row 1:
──────
database:                      test
table:                         kafka
consumer_id:                   ClickHouse-instance-test-kafka-1caddc7f-f917-4bb1-ac55-e28bd103a4a0
assignments.topic:             ['system_kafka_cons']
assignments.partition_id:      [0]
assignments.current_offset:    [18446744073709550615]
exceptions.time:               []
exceptions.text:               []
last_poll_time:                2006-11-09 18:47:47
num_messages_read:             4
last_commit_time:              2006-11-10 04:39:40
num_commits:                   1
last_rebalance_time:           1970-01-01 00:00:00
num_rebalance_revocations:     0
num_rebalance_assignments:     1
is_currently_used:             1
rdkafka_stat:                  {...}
dependencies:                  [['test.mv2','test.target2'],['test.mv1','test.target1']]
missing_dependencies:          []