Following on from my previous post, I’d like to share some more findings about Cassandra under heavy write load, and compare the results with Acunu’s distribution of Cassandra. In summary, our inserts are more consistent, with worst-case latency 2 orders of magnitude lower, and our range queries are in most cases 2 orders of magnitude faster.
Inserts
I ran long sustained insert tests, to see how or if performance degrades over time. The data was the same as in the previous post – 1 column per row with 50 byte random values. I wanted a larger scale result so I used a bigger box – this time with 24 GB RAM and 7 disks. I performed two tests – one for vanilla Cassandra and the other for Acunu’s distribution of Cassandra. For vanilla Cassandra (abbreviated to VC), one disk was a dedicated commit log disk, with 6 RAID0 data disks. For Acunu Cassandra (AC), all 7 disks were given to the filesystem. I inserted 3 billion rows, equating to about 400 GB of raw data. The results are shown on the following graph:
The red line is for VC, taking 26 hours to complete the inserts. The insert rate is initially about 60,000 per second but then starts to fall off. This is because when the heap starts to fill up, the JVM is spending more and more time garbage collecting. While it is doing this, inserts are blocked for significant periods of time. This can be seen more clearly in the latency graph following.
The blue line is for AC, taking 18 hours to complete. The initial rate is about 53,000 per second. This is slightly lower than for VC because the rate is limited to ensure read rates are not adversely affected by large batches of inserts, the effect of which can be seen in the range queries section later. The rate then decreases gradually again to ensure good read performance. The more data that has been inserted, the longer it takes to merge the results into the right place. The key difference between this decrease and the decrease for VC is that it is predictable. Although not shown here, the raw rate is very close to the average, whereas for VC it varies between 80,000 per second and long periods of zero. This is shown more clearly in the latency graph:
The latencies for AC are all below 300 ms, whereas for VC are often about 30 seconds. (The points plotted are 95th percentile latencies, meaning 5% of latencies are greater than the values plotted.) This is a key measure of predictability – if a user has to wait 30 seconds for an insert, they will most likely have gone elsewhere or their browser will have timed out. The maximum 300 ms latency for AC is a much better guarantee and shows an improvement of two orders of magnitude.
Range queries
In my previous post, I showed how range query performance improves over time after a batch of inserts with VC. I repeated this test for AC with results in this graph:
To recap, the test for VC was to insert 100 million rows, then every five minutes, perform one minute’s worth of small range queries and plot the rate. For AC, since the rate doesn’t change, I simply ran range queries continually once inserts had finished and plotted the rate. The rate for VC increases as compaction progresses, but only reaches about 12 per second; for AC, merges complete in a few minutes, and the rate is about 40 per second from shortly after the start. This is another example of AC giving more predictable performance.
Here are the slides from several talks I've given on this: