Lifecycle of a Postgres ClickPipe

Various pipe statuses and their meanings

Lifecycle of a Postgres ClickPipe

This is a document on the various phases of a Postgres ClickPipe, the different statuses it can have, and what they mean.

Provisioning

When you click on the Create ClickPipe button, the ClickPipe is created in a Provisioning state. The provisioning process is where we spin up the underlying infrastructure to run ClickPipes for the service, along with registering some initial metadata for the pipe. Since compute for ClickPipes within a service is shared, your second ClickPipe will be created much faster than the first one — as the infrastructure is already in place.

Setup

After a pipe is provisioned, it enters the Setup state. This state is where we create the destination ClickHouse tables. We also obtain and record the table definitions of your source tables here.

Snapshot

Once setup is complete, we enter the Snapshot state (unless it’s a CDC-only pipe, which would transition to Running). Snapshot, Initial Snapshot and Initial Load (more common) are interchangeable terms. In this state, we take a snapshot of the source database tables and load them into ClickHouse. This doesn’t use logical replication, but the replication slot is created at this step, therefore your max_slot_wal_keep_size and storage parameters should account for slot growth during initial load. For more information on initial load, see the parallel initial load documentation. The pipe will also enter the Snapshot state when a resync is triggered or when new tables are added to an existing pipe.

Running

Once the initial load is complete, the pipe enters the Running state (unless it’s a snapshot-only pipe, which would transition to Completed). This is where the pipe begins Change-Data Capture. In this state, we start logical replication from the source database to ClickHouse. For information on controlling CDC, see the doc on controlling CDC.

Paused

Once the pipe is in the Running state, you can pause it. This will stop the CDC process and the pipe will enter the Paused state. In this state, no new data is pulled from the source database, but the existing data in ClickHouse remains intact. You can resume the pipe from this state.

Pausing

This state is coming soon. If you’re using our OpenAPI, consider adding support for it now to ensure your integration continues working when it’s released.

When you click on the Pause button, the pipe enters the Pausing state. This is a transient state where we’re in the process of stopping the CDC process. Once the CDC process is fully stopped, the pipe will enter the Paused state.

Modifying

This state is coming soon. If you’re using our OpenAPI, consider adding support for it now to ensure your integration continues working when it’s released.

Currently, this indicates the pipe is in the process of removing tables.

Resync

This state is coming soon. If you’re using our OpenAPI, consider adding support for it now to ensure your integration continues working when it’s released.

This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the resync documentation.

Completed

This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there’s no more work to do.

Failed

If there is an irrecoverable error in the pipe, it will enter the Failed state. You can reach out to support or resync your pipe to recover from this state.