Generic MongoDB source setup guide

Set up any MongoDB instance as a source for ClickPipes

If you use MongoDB Atlas, please refer to the specific guide here.

Enable oplog retention

Minimum oplog retention of 24 hours is required for replication. We recommend setting the oplog retention to 72 hours or longer to ensure that the oplog isn’t truncated before the initial snapshot is completed.

You can check your current oplog retention by running the following command in the MongoDB shell (you must have clusterMonitor role to run this command):

db.getSiblingDB("admin").serverStatus().oplogTruncation.oplogMinRetentionHours

To set the oplog retention to 72 hours, run the following command on each node in the replica set as an admin user:

db.adminCommand({
    "replSetResizeOplog" : 1,
    "minRetentionHours": 72
})

For more details on the replSetResizeOplog command and oplog retention, see MongoDB documentation.

Configure a database user

Connect to your MongoDB instance as an admin user and execute the following command to create a user for MongoDB CDC ClickPipes:

db.getSiblingDB("admin").createUser({
    user: "clickpipes_user",
    pwd: "some_secure_password",
    roles: ["readAnyDatabase", "clusterMonitor"],
})

Make sure to replace clickpipes_user and some_secure_password with your desired username and password.

What’s next?

You can now create your ClickPipe and start ingesting data from your MongoDB instance into ClickHouse Cloud. Make sure to note down the connection details you used while setting up your MongoDB instance as you will need them during the ClickPipe creation process.