How a Single Line of Code Made a 24-core Server Slower Than a Laptop

Obviously you’d expect the more cores it runs on, the faster it is. You benchmark it on a laptop first and indeed you find out it scales nearly perfectly on all of the 4 available cores. Then you run it on a big, fancy, multiprocessor machine, expecting even better performance, only to see it actually runs slower than the laptop, no matter how many cores you give it. Uh. That has just happened to me recently.

I’ve been working recently on a Cassandra benchmarking tool which is probably the most efficient Cassandra benchmarking tool you can get, both in terms of CPU use and memory use. The whole idea is very simple: a small piece of code generates data and executes a bunch of asynchronous CQL statements against Cassandra. Latte calls that code in a loop and records how long each iteration took. Finally, it makes a statistical analysis and displays it in various forms.

Benchmarking seems to be a very pleasant problem to parallelize. As long as the code under benchmark is stateless, it can be fairly trivially called from multiple threads. I’ve blogged about how to achieve that in Rust already and .

However, at the time I wrote those earlier blog posts, Latte’s workload definition capabilities were nonexistent quite limited. It came with only two predefined, hardcoded workloads, one for reading and another one for writing. There were a few things you could parameterize, e.g. the number or the sizes of table columns, but nothing really fancy. No secondary indexes. No custom filtering clauses. No control over the CQL text. Really nothing. So, overall, Latte at that time was more of a proof-of-concept rather than a universal tool for doing real work. Surely, you could fork it and write a new workload in Rust, then compile everything from source. But who wants to waste time on learning the internals of a niche benchmarking tool?