Stability and reliability were the focus of the previous HBase versions, now it is time for performance! In this first post on the subject, I will describe the work done on the scanners.
As described in the Performance Evaluation page of the wiki, current numbers compared to Bigtable are not that great (though a lot better than it was). One big hotspot we have in HBase is remote procedure calls (RPC) and some profiling we did showed us that it's sometimes even bigger than the overhead of HDFS. So, our strategy for HBase 0.19.0 will be to batch as much rows as we can in a single RPC in order to diminish it's impact. It will require different implementations for different operations.
Implementation and tests
Scanners in HBase already provide the fastest way to retrieve data but compared to how a direct scan in a HDFS MapFile is, it is clear that something is wrong (3 737 rows per second instead of a staggering 58 054). HBASE-877 is about fixing that hotspot. The implementation is simple: since we know that we want many rows, why not get a bunch of them each time we communicate with a region server and then serve directly from cache? In other words, fetch x rows, serve from cache x times, repeat until rows are exhausted. That works for big jobs, but what if a user wants to scan only 2-3 rows (or even 1), can we do something so that it doesn't fetch 30 rows? If we make sure that that number is configurable, it will also work fine for small scans.
My first tests with my patch confirm that batching rows is efficient. In the following chart, I ran the Performance Evaluation test 30 times, going from batching 1 row to batching 30 rows:
As we can see, the test took around 108 seconds to scan all 1 million+ rows on my machine (2.66 Ghz, 2GB of memory, SATA disk). That's 9692 rows per second, better than the PE in the wiki but my machine is also stronger. But at 30 rows per fetch, it scanned 30 028 rows per second! That's a lot better for a very simple fix. I think this could be further optimized by fetching the rows in a second thread with some logic to make sure we don't load the full table right away... and that's for a future version of HBase.Configuration
There are many ways to configure how many rows are fetched. First, hbase-default.xml will define a default value for hbase.client.scanner.caching that you will be able to override in hbase-site.xml like any other value. The other way is to use HTable.setScannerCaching to define an instance-specific value. For example, you have to scan thousands of rows and right after that you only have to scan 2 rows and close the scanner (all on the same table), you should just pass 30 the first time and 2 the second time to setScannerCaching. That's a bit aggressive, but you can do it if you want.
Something you must verify before setting the caching to a high number of is the size of your rows. While testing the patch, I used 100MB rows to have many splits and kept the caching at 20. Bad idea, my machine began swapping like hell!
Conclusion
HBase 0.19.0 is all about speed and the first optimization is done on scanners. By fetching more rows, the cost of doing RPCs is greatly reduced. By keeping it configurable, it also fits all use cases. In my opinion, those who are using HBase as a source for MapReduce jobs will see a big improvement since it does full scans of tables (as long as the caching is configured at around 30 or less for very big rows).
If you want to use this sweet improvement right now, just download HBase 0.18.0 and patch it with my patch in HBASE-887 or wait until it gets committed.
Happy scanning!
