Allocation profiling for versions before 25.9

Page detailing allocation profiling in ClickHouse

ClickHouse uses jemalloc as its global allocator. Jemalloc comes with some tools for allocation sampling and profiling.
To make allocation profiling more convenient, SYSTEM commands are provided along with four letter word (4LW) commands in Keeper.

Sampling allocations and flushing heap profiles

If you want to sample and profile allocations in jemalloc, you need to start ClickHouse/Keeper with profiling enabled using the environment variable MALLOC_CONF:

MALLOC_CONF=background_thread:true,prof:true,prof_active:true

jemalloc will sample allocations and store the information internally.

You can tell jemalloc to flush the current profile by running:

ClickHouse

SYSTEM JEMALLOC FLUSH PROFILE

Keeper

echo jmfp | nc localhost 9181

By default, the heap profile file will be generated in /tmp/jemalloc_clickhouse._pid_._seqnum_.heap where _pid_ is the PID of ClickHouse and _seqnum_ is the global sequence number for the current heap profile.
For Keeper, the default file is /tmp/jemalloc_keeper._pid_._seqnum_.heap, and follows the same rules.

A different location can be defined by appending the MALLOC_CONF environment variable with the prof_prefix option.
For example, if you want to generate profiles in the /data folder where the filename prefix will be my_current_profile, you can run ClickHouse/Keeper with the following environment variable:

MALLOC_CONF=background_thread:true,prof:true,prof_prefix:/data/my_current_profile

The generated file will be appended to the prefix PID and sequence number.

Analyzing heap profiles

After heap profiles have been generated, they need to be analyzed.
For that, jemalloc’s tool called jeprof can be used. It can be installed in multiple ways:

• Using the system’s package manager
• Cloning the jemalloc repo and running autogen.sh from the root folder. This will provide you with the jeprof script inside the bin folder

jeprof uses addr2line to generate stacktraces which can be really slow.
If that’s the case, it is recommended to install an alternative implementation of the tool.

git clone https://github.com/gimli-rs/addr2line.git --depth=1 --branch=0.23.0
cd addr2line
cargo build --features bin --release
cp ./target/release/addr2line path/to/current/addr2line

There are many different formats to generate from the heap profile using jeprof. It is recommended to run jeprof --help for information on the usage and the various options the tool provides.

In general, the jeprof command is used as:

jeprof path/to/binary path/to/heap/profile --output_format [ > output_file]

If you want to compare which allocations happened between two profiles you can set the base argument:

jeprof path/to/binary --base path/to/first/heap/profile path/to/second/heap/profile --output_format [ > output_file]

Examples

• if you want to generate a text file with each procedure written per line:

jeprof path/to/binary path/to/heap/profile --text > result.txt

• if you want to generate a PDF file with a call-graph:

jeprof path/to/binary path/to/heap/profile --pdf > result.pdf

Generating a flame graph

jeprof allows you to generate collapsed stacks for building flame graphs.

You need to use the --collapsed argument:

jeprof path/to/binary path/to/heap/profile --collapsed > result.collapsed

After that, you can use many different tools to visualize collapsed stacks.

The most popular is FlameGraph which contains a script called flamegraph.pl:

cat result.collapsed | /path/to/FlameGraph/flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg

Another interesting tool is speedscope that allows you to analyze collected stacks in a more interactive way.

Controlling allocation profiler during runtime

If ClickHouse/Keeper is started with the profiler enabled, additional commands for disabling/enabling allocation profiling during runtime are supported. Using those commands, it’s easier to profile only specific intervals.

To disable the profiler:

ClickHouse

SYSTEM JEMALLOC DISABLE PROFILE

Keeper

echo jmdp | nc localhost 9181

To enable the profiler:

ClickHouse

SYSTEM JEMALLOC ENABLE PROFILE

Keeper

echo jmep | nc localhost 9181

It’s also possible to control the initial state of the profiler by setting the prof_active option which is enabled by default.
For example, if you don’t want to sample allocations during startup but only after, you can enable the profiler. You can start ClickHouse/Keeper with the following environment variable:

MALLOC_CONF=background_thread:true,prof:true,prof_active:false

The profiler can be enabled later.

Additional options for the profiler

jemalloc has many different options available, which are related to the profiler. They can be controlled by modifying the MALLOC_CONF environment variable. For example, the interval between allocation samples can be controlled with lg_prof_sample.
If you want to dump the heap profile every N bytes you can enable it using lg_prof_interval.

It is recommended to check jemallocs reference page for a complete list of options.

Other resources

ClickHouse/Keeper expose jemalloc related metrics in many different ways.

Warning

It’s important to be aware that none of these metrics are synchronized with each other and values may drift.

System table asynchronous_metrics

SELECT *
FROM system.asynchronous_metrics
WHERE metric LIKE '%jemalloc%'
FORMAT Vertical

Reference

System table jemalloc_bins

Contains information about memory allocations done via the jemalloc allocator in different size classes (bins) aggregated from all arenas.

Reference

Prometheus

All jemalloc related metrics from asynchronous_metrics are also exposed using the Prometheus endpoint in both ClickHouse and Keeper.

Reference

jmst 4LW command in Keeper

Keeper supports the jmst 4LW command which returns basic allocator statistics:

echo jmst | nc localhost 9181