RocksDB

Resumable Remote Compaction

Tue, 19 May 2026 00:00:00 +0000

Background

RocksDB can offload compaction work to remote workers through the CompactionService API. In this model, the primary RocksDB instance selects the input files and sends a serialized CompactionServiceInput to a worker; the remote worker runs DB::OpenAndCompact(), writes output SSTs to output_directory, and returns a serialized CompactionServiceResult that the primary RocksDB instance installs into its LSM tree. See the Remote Compaction wiki for the full architecture. This lets operators scale compaction throughput with stateless workers while keeping the primary RocksDB instance’s CPU and I/O available for serving reads and writes. However, remote compaction jobs can be long-running—sometimes processing hundreds of gigabytes of input. When a worker crashes, gets preempted, or times out, the entire compaction must restart from scratch, wasting all output produced before the interruption and increasing compaction debt on the primary RocksDB instance.

How Resumable Remote Compaction Works

Resumable remote compaction introduces a checkpoint-and-resume mechanism. During a compaction, the worker periodically saves its progress to the output_directory. If the compaction is interrupted, a subsequent call to OpenAndCompact() with the same output directory can pick up from the last checkpoint rather than starting over.

Checkpointing

After each output SST file is completed, the worker persists a progress checkpoint to a compaction progress file in the output directory output_directory. The checkpoint records which internal key to resume from and the metadata of all completed output files. Progress records use delta encoding—each record only contains files completed since the last checkpoint—to keep serialization cost linear.

The worker skips checkpointing at boundaries where resuming could be unsafe or requires complicated handling: when range deletions span the file boundary or when adjacent output files share the same user key. These constraints ensure that resuming produces the same results as if the compaction was not interrupted.

Resuming

When OpenAndCompact() is called with allow_resumption = true, it scans the output directory for a valid progress file. If one is found, it loads the checkpointed state, seeks the input iterator to the recorded resume key, restores the output file state, and continues compaction from that point. If the progress file is corrupted or missing, the system falls back to a fresh compaction by cleaning the directory.

How to Enable It

On the primary RocksDB instance, set a CompactionService implementation on the DB options. On the remote worker, pass allow_resumption = true in OpenAndCompactOptions when calling DB::OpenAndCompact(). The output_directory must be the same across retries for resumption to work—each retry call with the same directory will automatically detect and resume from the previous checkpoint. The REMOTE_COMPACT_RESUMED_BYTES statistics ticker tracks the total bytes of output files reused from a previous interrupted run, giving visibility into how much work resumption saved.

// Primary RocksDB instance
DBOptions db_options;
db_options.compaction_service = std::make_shared<MyCompactionService>();

// Remote worker
OpenAndCompactOptions options;
options.allow_resumption = true;

std::string result;
Status s = DB::OpenAndCompact(
    options,
    db_path,             // source database path
    output_directory,    // where output SSTs and progress are stored
    compaction_input,    // serialized CompactionServiceInput
    &result,             // serialized CompactionServiceResult
    override_options);

Future Work

Today this feature targets remote compaction. The same checkpoint-and-resume mechanism could also support local compaction after a crash. The core persistence and resume logic is already in CompactionJob; the remaining work is to integrate it with local compaction scheduling and recovery.

Interpolation search for SST index blocks

Mon, 04 May 2026 00:00:00 +0000

For workloads with uniformly distributed keys, RocksDB now supports interpolation search for SST index blocks as an alternative to the default binary search.

The idea

Binary search always splits the remaining range in half:

mid = low + (high - low) / 2

That’s Θ(log n) probes regardless of the data. Interpolation search instead estimates where the target should land based on its value relative to the current boundaries:

probe = low + (target - key[low]) * (high - low) / (key[high] - key[low])

On uniformly distributed keys, that’s expected O(log log n) probes. The canonical example: for an index block with restart keys 0, 1, 2, ..., 1023 and a seek for 900, binary search needs about 10 hops; interpolation search lands on it in 1.

The catch is that pure interpolation search degrades to O(n) on badly skewed data.

Turning a key into a number

The interpolation formula needs numeric values, but index keys are variable-length byte slices. RocksDB extracts a uint64_t per key by reading the first 8 bytes after the common prefix shared by the block’s boundary keys, in big-endian, and zero-pads to the right if the remaining bytes are too short.

inline uint64_t ReadBe64FromKey(Slice s, bool is_user_key, size_t offset) {
  // ... strip internal seq/type bytes if needed ...
  if (s.size() - offset >= 8) {
    uint64_t val;
    memcpy(&val, s.data() + offset, sizeof(val));
    return port::kLittleEndian ? EndianSwapValue(val) : val;
  }
  // pad short tails with zeros on the right (preserves bytewise order)
}

Big-endian + zero-pad preserves bytewise ordering, so the linear interpolation formula stays consistent with the comparator. This is also why the feature requires BytewiseComparator.

Two distinct keys can still collapse to the same uint64_t once you go past the first 8 non-shared bytes. To avoid a divide-by-zero, we simply fall back to binary search in that case.

How to enable it

To force interpolation search on every index block:

rocksdb::BlockBasedTableOptions table_options;
table_options.index_block_search_type =
    rocksdb::BlockBasedTableOptions::kInterpolation;

kAuto: per-block selection

kAuto is the recommended way to use the feature. It chooses the search algorithm for each index block automatically, based on a uniformity hint written into the block footer at SST construction time:

table_options.index_block_search_type =
    rocksdb::BlockBasedTableOptions::kAuto;
table_options.uniform_cv_threshold = 0.2;

When uniform_cv_threshold >= 0, the SST writer scans each index block’s restart keys and computes the coefficient of variation (CV) of the gaps between consecutive numeric key values:

gap[i] = key_value[i + 1] - key_value[i]
CV     = stddev(gap) / mean(gap)

Lower CV means the gaps are more uniform — and the more likely interpolation search will outperform binary search. The CV is computed incrementally with Welford’s online algorithm, so the scan is one pass over the restart points.

If CV < uniform_cv_threshold, RocksDB sets an is_uniform bit in the block footer. At read time, kAuto resolves to kInterpolation only when that bit is set and the comparator is bytewise; otherwise it uses kBinary.

Write overhead

Computing the is_uniform bit is a cheap operation as it is only computed for the index blocks in a SST file. CPU profiling of db_bench -benchmarks=fillseq,compact -compression_type=none -disable_wal=1 attributes only ~0.08% of write-path CPU to ScanForUniformity.

After a few more releases, we plan to enable kAuto and uniform_cv_threshold by default.

Benchmarks

Setup — populate a DB and force a single-level shape so all reads hit the same index structure, then measure point-read throughput:

# Build a release binary
make clean && DEBUG_LEVEL=0 make db_bench

# Load + compact, varying the index_shortening_mode
./db_bench -benchmarks=fillrandom,compact \
           -index_shortening_mode=1

# Then point-read against the populated DB
./db_bench -use_existing_db=true -benchmarks=readrandom \
           -index_block_search_type=binary_search   # or interpolation_search / auto_search

index_shortening_mode=1 (kShortenSeparators) keeps the file’s last index key intact, which preserves a roughly uniform numeric distribution for the benchmark.

Results, averaged over multiple runs:

Mode	ops/s	vs `binary_search`
`binary_search`	335,749	baseline
`interpolation_search`	366,598	+9.2%
`auto_search`	366,832	+9.2%

Compatibility

The is_uniform bit reuses a previously-reserved bit in the data block footer. SSTs written by older RocksDB never set it and decode as is_uniform = false, so they read with binary search under kAuto. However, after the bit is set, if read by an older RocksDB version < 11.0.0, it will read it as a corruption error.

Future work

Some future opportunities can involve extending interpolation search to data blocks, as well as supporting other comparators such as ReverseBytewiseComparator.

RocksDB development finds a CPU bug

Tue, 17 Feb 2026 00:00:00 +0000

This is the story of how a RocksDB unit test I added four years ago, a mini-stress test you might call it, revealed a novel hardware bug in a newer CPU. It was scary enough to be assigned a “high severity” CVE.

Background: Unique Identifiers

About four years ago, we added unique identifiers to SST files to give them stable identifiers across different filesystems for caching purposes. Part of the motivation here was to eliminate our dependence on the uniqueness and non-recycling of unique identifiers on files provided by the OS filesystem. (Some filesystems were only guaranteeing uniqueness among existing files, not among all files even in recent history.) I would call this dependency problem the great tension between reusing existing solutions and code self-reliance. You don’t want to duplicate others’ work but you also don’t want to be subject to their bugs or changing / misaligned requirements. Striking this balance can be tricky, but in this case it was clear to us that we didn’t want to rely on all the possible filesystems providing quality unique identifiers.

If you’re comfortable with large random numbers (e.g. 128 bits), you probably agree that persisting random identifiers (or quasi-random, which I helped formalize in a paper, also on arXiv) with each file would be safer and more predictable than relying so crucially on a minor feature of OS filesystems.

High Quality Randomness

However, that assumes we have access to high quality random numbers (at least a good one or two to start from - see the paper). Because RocksDB intends to be cross-platform, we want to minimize platform-specific dependencies and prefer cross-platform dependencies. But that could easily land us back where we didn’t want to be: susceptible to a bug or hiccup in one implementation of what we needed.

Fortunately, the nature of random entropy allows combining sources so that your result is as good as your best input source, so even if one is bad, you only have a problem if they’re all bad. And we had the advantages that (a) we only needed uniqueness, not security, which reduced the need for extra scrutiny and allowed us to use the quasi-random approach, and (b) the quasi-random approach minimized the amount of entropy needed, so the performance cost of acquiring each unit of entropy was almost inconsequential. Therefore, I combined these sources of entropy:

C++11’s std::random_device which is supposed to provide high quality but is allowed not to.
A hash of various environment parameters including hostname, process id, thread id, and various macro and micro time readings.
Platform-specific UUID generator (Linux and Windows only)

Trust But Verify

To verify the quality of each of these sources on an ongoing basis, I added unit tests that used many threads to create thousands of unique identifiers based on one of the above sources at a time and verified their uniqueness. For a high quality source, the probability of any duplicate 128-bit IDs among thousands is negligible, even if running these tests continuously for decades.

That’s Weird

That was pretty much the story until some months ago the test based on std::random_device failed, once. It was quite suspicious because the number of unique IDs was not just one short of expectation, it was dozens or hundreds short. However, even that could be explained by a random CPU hiccup or bit flip in which we generated fewer IDs to begin with. (You might have noticed an increasing amount of RocksDB development effort and portion of CPU time going into checks that are logically redundant but exist to detect CPU miscalculations before the corruption propagates too far.)

But then it failed again about a month later. No failures for four years, then two failures in two months. This smelled really bad. Digging into the details I noticed a crucial correlation: both of the failed test jobs had run on the same type of hardware, though in completely different data centers.

From there I did the natural thing for an engineer: scale it up to try to reproduce the failure. And that was remarkably easy. By increasing the number of threads in the job to around the number of cores it would fail quickly and consistently on all systems using the same type of newer CPU, and pass on everything else. I tested some variants of this to establish some more details, including

std::random_device using “rdrand” and “/dev/urandom” sources were not affected, and
libc++ (from clang) was not affected, only libstdc++ (from GCC)

Root Cause Analysis

From there Meta colleagues investigated the low-level details. They found the problem to be that the RDSEED instruction on this type of processor would return 0 and “success” much more often than would randomly be expected, but only on some cores and only under “complex micro-architectural conditions reproducible under memory-load,” as a colleague describes it. A mitigating Linux kernel patch was developed to signal that RDSEED was unavailable on these processors, with the intention of rolling it out internally at Meta to avoid problems until a fix came from the OEM. AMD quickly acknowledged the issue and announced planned mitigation, including a CPU microcode update.

With Apologies

Although I worked to keep the information confidential until the OEM publicly acknowledged the issue, the uncoordinated disclosure via the Linux mailing list was due to zealous remediation efforts that crossed multiple infrastructure teams at Meta. We regret the mistake and are working to improve controls on the processes that failed to coordinate with the OEM first.

Key Takeaways

Test what you depend on.
Have redundancies and/or sanity checks for what you depend on.
Even CPUs can have bugs, usually flaky individual units but occasionally a bug affecting all units.

BitFields API: Type-Safe Bit Packing for Lock-Free Data Structures

Wed, 31 Dec 2025 00:00:00 +0000

Modern concurrent data structures increasingly rely on atomic operations to avoid the overhead of locking. A valuable but under-utilized technique for maximizing the effectiveness of atomic operations is bit packing—fitting multiple logical fields into a single atomic variable for algorithmic simplicity and efficiency. However, language support for bit packing does not guarantee dense packing, and manually managing bit manipulation quickly becomes error-prone, especially when dealing with complex state machines.

To address this in RocksDB, we have developed a reusable BitFields API, a type-safe, zero-overhead abstraction for bit packing in C++. This works in conjunction with clean wrappers for std::atomic for powerful and relatively safe bit-packing of atomic data. For broader use, a variant of the code has been proposed for adding to folly.

The Problem: Managing Packed Bit Fields

Consider HyperClockCache, an essentially lock-free cache implementation in RocksDB, which was refactored to use this BitFields API. It is a hash table built on slots that can each hold a cache entry and relevant metadata. For atomic simplicity and efficiency, all the essential metadata for each slot is packed into a single 64-bit value:

The reference count and eviction metadata are together encoded into acquire and release counters, 30 bits each.
The possible states of {empty, under construction/destruction, occupied+visible, and occupied+invisible} are encoded into three state bits (instead of two, for easier decoding and manipulation).
A hit bit is used for secondary cache integration.

Traditionally, you might write code like this:

// Old approach: manual bit manipulation
constexpr uint64_t kAcquireCounterShift = 0;
constexpr uint64_t kReleaseCounterShift = 30;
constexpr uint64_t kCounterMask = 0x3FFFFFFF;
constexpr uint64_t kHitBitShift = 60;
constexpr uint64_t kOccupiedShift = 61;
constexpr uint64_t kShareableShift = 62;
constexpr uint64_t kVisibleShift = 63;
constexpr uint64_t kStateShift = kOccupiedShift;

std::atomic<uint64_t> meta_;

bool IsUnderConstruction(uint64_t meta) const {
    return (meta & (uint64_t{1} << kOccupiedShift)) && !(meta & (uint64_t{1} << kShareableShift));
}

// Getting fields
uint64_t meta = meta_.load(std::memory_order_acquire);
if (IsUnderConstruction(meta)) {
  // ...
} else if ((meta >> kVisibleShift) & 1) {
  uint32_t refcount =
      static_cast<uint32_t>(((meta >> kAcquireCounterShift) -
                             (meta >> kReleaseCounterShift)) & kCounterMask);
  // ...
}


// Setting fields

// Set the hit bit (relaxed)
meta_.fetch_or(uint64_t{1} << kHitBitShift, std::memory_order_relaxed);

// Set both counters to `new_count` (as in eviction processing)
uint64_t meta = meta_.load(std::memory_order_relaxed);
uint64_t new_meta =
    (meta & ((uint64_t{1} << kHitBitShift) | (uint64_t{7} << kStateShift))) |
    (new_count << kReleaseCounterShift) |
    (new_count << kAcquireCounterShift);
bool success = meta_.compare_exchange_strong(meta, new_meta,
                                             std::memory_order_acq_rel);

// Increment acquire counter by initial_countdown
old_meta = meta_.fetch_add((uint64_t{1} << kAcquireCounterShift) * initial_countdown,
                           std::memory_order_acq_rel);

This approach has several problems:

Error-prone: Easy to get masks and shifts wrong
Maintenance burden: Changes to field sizes require updating multiple constants
Abstraction challenges: Even if writing a full set of well-tested getters and setters to hide all the details, details can leak in to do things like update multiple fields in one non-CAS (compare-and-swap) atomic operation.

New Solution: BitFields API

The BitFields API provides a declarative, type-safe way to define bit-packed structures. Here’s how the same example looks with BitFields:

// New approach: declarative bit fields. (Each field must reference the
// previous, so that the declaration machinery is simply stateless.)
struct SlotMeta : public BitFields<uint64_t, SlotMeta> {
  using AcquireCounter = UnsignedBitField<SlotMeta, 30, NoPrevBitField>;
  using ReleaseCounter = UnsignedBitField<SlotMeta, 30, AcquireCounter>;
  using HitFlag = BoolBitField<SlotMeta, ReleaseCounter>;
  using OccupiedFlag = BoolBitField<SlotMeta, HitFlag>;
  using ShareableFlag = BoolBitField<SlotMeta, OccupiedFlag>;
  using VisibleFlag = BoolBitField<SlotMeta, ShareableFlag>;

  // Convenience helpers
  bool IsUnderConstruction() const {
    return Get<OccupiedFlag>() && !Get<ShareableFlag>();
  }
};

BitFieldsAtomic<SlotMeta> meta_;

// Getting fields
SlotMeta state = meta_.Load();
if (state.IsUnderConstruction()) {
  // ...
} else if (state.Get<SlotMeta::VisibleFlag>()) {
  uint32_t refcount = state.Get<SlotMeta::AcquireCounter>() -
                      state.Get<SlotMeta::ReleaseCounter>();
  // ...
}

// Setting fields

// Set the hit bit (relaxed)
meta_.ApplyRelaxed(SlotMeta::HitFlag::SetTransform());

// Set both counters to `new_count` (as in eviction processing)
SlotMeta meta = meta_.LoadRelaxed();
SlotMeta new_meta = meta;
new_meta.Set<SlotMeta::ReleaseCounter>(new_count);
new_meta.Set<SlotMeta::AcquireCounter>(new_count);
meta_.CasStrongRelaxed(meta, new_meta);

// Increment acquire counter by initial_countdown
auto add_acquire =
    AcquireCounter::PlusTransformPromiseNoOverflow(initial_countdown);
meta_.Apply(add_acquire, &old_meta);

// Bonus: Atomic multi-field updates without compare-exchange
auto transform = AcquireCounter::PlusTransformPromiseNoOverflow(1) +
                 ReleaseCounter::PlusTransformPromiseNoOverflow(1);
meta_.Apply(transform);

Key Features

Type Safety and Self-Documentation

Each field has a specific type (bool for BoolBitField, appropriately-sized unsigned int for UnsignedBitField) and clear semantic meaning. The field definitions are self-documenting: you can immediately see how many bits each field occupies and in what order.

Zero Overhead

Because of heavy use of templates and constexpr operations and the ability to satisfy multiple field reads or writes from a single atomic operation, we have seen no runtime overhead vs. hand-written bit manipulation, in RocksDB. In one case, we verified the assembly code was identical.

For folly’s LifoSem, there was one case where an optimization hack with detected overflow from one field to another couldn’t be replicated as efficiently with the BitFields API because it would violate overflow checking. For that case I dove into the underlying representation to bypass the BitFields overflow check.

Atomic Operations with Transforms

One of the most powerful features is the ability to combine multiple field updates into a single atomic operation using “transforms”, if they are all either (a) some combination of addition and subtraction, (b) bitwise-and, or (c) bitwise-or. For example:

// Clear several but not all fields atomically
auto and_transform = Field1::AndTransform(0) +
                 Field2::ClearTransform() +
                 Field4::ClearTransform();
atomic_bitfields.Apply(and_transform, &old_state, &new_state);
...
// Set more than one boolean field atomically
auto or_transform = Field2::SetTransform() +
                 Field4::SetTransform();
atomic_bitfields.Apply(or_transform, &old_state, &new_state);
...
auto add_transform = Field1::PlusTransformPromiseNoOverflow(1) +
                     Field3::MinusTransformPromiseNoUnderflow(1);
atomic_bitfields.Apply(add_transform, &old_state, &new_state);

Each Apply() generates a single atomic operation (e.g., fetch_add or fetch_or) that updates all the specified fields, and optionally returns both the old and new values. This enables a number of hacks for atomic updates without CAS.

Overflow Protection

The API includes built-in overflow detection in debug builds:

// An assertion will fail in debug builds if the counter overflows
auto transform = Counter::PlusTransformPromiseNoOverflow(value);
atomic.Apply(transform);

For fields at the top of the underlying representation (where overflow doesn’t affect other fields), overflow is explicitly ignored. (A compile time error is generated if you try to use PlusTransformPromiseNoOverflow on a field at the top of the representation or PlusTransformIgnoreOverflow on a field not at the top of the representation.)

// For wraparound counters
auto transform = Counter::PlusTransformIgnoreOverflow(value);

This capability is used in a folly data structure called LifoSem, which I have proposed to refactor to a proposed BitFields API variant for folly.

Compare-and-Swap (CAS) Support

The atomic wrappers provide full CAS support for lock-free algorithms:

SlotMeta expected = current_state;
SlotMeta desired = expected.With<Field1>(new_value).With<Field2>(true);
if (meta_.CasStrong(expected, desired)) {
  // Successfully updated
  ...
}

Atomic wrappers

The BitFields API includes two atomic wrappers: RelaxedBitFieldsAtomic and BitFieldsAtomic. However, RocksDB also has versions of these wrappers for regular std::atomic variables that help with memory ordering discipline: RelaxedAtomic and Atomic in util/atomic.h.

These wrappers help in a couple of ways:

Self-document intended memory order: An atomic field generally has a single memory order that all or most operations should use, typically either std::memory_order_relaxed or std::memory_order_acq_rel.
More intentional memory orders and atomic operations: The standard library’s implicit conversions and default memory ordering (memory_order_seq_cst) make it easy to accidentally use sequential consistency with acquire/release ordering or even relaxed, which could hurt performance, and tend to hide where atomic operations are actually happening (e.g. implicit vs. explicit load).

For example, instead of writing:

std::atomic<uint64_t> stat_counter;
stat_counter++;  // Uses memory_order_seq_cst implicitly - maybe inefficient

You write:

RelaxedAtomic<uint64_t> stat_counter;
stat_counter.FetchAddRelaxed(1);  // Explicitly relaxed - appropriate for a diagnostic counter

Or for data providing synchronization:

Atomic<size_t> refcount;
refcount.FetchAdd(1);  // Standard acquire-release semantics for coordinating with other threads

These wrappers complement the BitFields atomic wrappers by providing the same ordering discipline for non-packed atomic variables throughout much of RocksDB, creating a more readable and less clunky approach to concurrent programming. Migrating remaining uses of std::atomic is an ongoing effort.

Real-World Usage in RocksDB

The BitFields API was developed along with the revamped parallel compression in RocksDB, but with the intention to also clean up the HyperClockCache (HCC) implementation. With that migration complete, we can see the benefits. Specifically, by packing more of the state machine into a single atomic value, the parallel algorithms became both simpler and more efficient. Concurrent algorithms that could have blown up in their state space with elaborate interleavings between threads trying not to block each other, e.g. because of multi-step consensus on work assignments, were instead able to quickly and more easily make progress, e.g. with atomically clear work assignments.

Before: Manual Bit Manipulation

The old HCC code was difficult to read and maintain. Many of the common read and update operations had manually written helper functions, but it was not practical to develop the full set of functions needed for rare cases. Consider this code that clears the “visible” flag on a slot when an entry is erased from subsequent lookups but might still be referenced:

// Old HCC code, without atomic wrappers
uint64_t old_meta =
        h->meta.fetch_and(~(uint64_t{ClockHandle::kStateVisibleBit}
                                   << ClockHandle::kStateShift), std::memory_order_acq_rel);
// Apply update to local copy
uint64_t new_meta = old_meta & ~(uint64_t{ClockHandle::kStateVisibleBit}
                            << ClockHandle::kStateShift);

// New HCC code
SlotMeta old_meta, new_meta;
h->meta.Apply(SlotMeta::VisibleFlag::ClearTransform(), &old_meta, &new_meta);

Or this assertion that the acquire and release counters are different:

// Old HCC code
uint64_t old_meta = ...;
assert(((old_meta >> ClockHandle::kAcquireCounterShift) &
        ClockHandle::kCounterMask) !=
        ((old_meta >> ClockHandle::kReleaseCounterShift) &
        ClockHandle::kCounterMask));

// New HCC code without single-purpose helper functions
SlotMeta old_meta = ...;
assert(old_meta.Get<SlotMeta::AcquireCounter>() !=
       old_meta.Get<SlotMeta::ReleaseCounter>());

// New HCC code, with single-purpose helper functions
SlotMeta old_meta = ...;
assert(old_meta.GetAcquireCounter() != old_meta.GetReleaseCounter());

Some hand-written helper functions or using directives are still useful for brevity, but even without them all the bit manipulation details are hidden in the BitFields implementation.

Future Directions

We hope the proposed folly version is accepted to make the BitFields API available for broader usage. Additionally, some quality-of-life improvements are likely possible, perhaps including easier declaration and usage syntax, hopefully without delving into boost-like macro hell. Better runtime and compile time checks might also be possible.

Conclusion

The BitFields API demonstrates that zero-overhead abstractions can significantly improve code quality without sacrificing performance. By providing type safety, self-documentation, and convenience features around bit manipulation and atomic operations, it makes lock-free programming more accessible and maintainable. Bit-packed atomics are arguably essential for slaying the complexity dragon of efficient lock-free and low-lock algorithms, because they reduce explosion in algorithm states.

For RocksDB specifically, the migration to BitFields has made the HyperClockCache implementation substantially easier to understand and modify, while maintaining the same high-performance characteristics. Combined with the recent parallel compression revamp, these improvements showcase our ongoing commitment to writing clean, efficient, and maintainable code.

The BitFields API is available in RocksDB’s util/bit_fields.h and can be adapted for use in other projects requiring efficient, type-safe bit packing. For those building high-performance concurrent systems, it offers a compelling alternative to manual bit manipulation—proving that safe abstractions and peak performance are not mutually exclusive.

Parallel Compression Revamp: Dramatically Reduced CPU Overhead

Wed, 08 Oct 2025 00:00:00 +0000

The upcoming RocksDB 10.7 release includes a major revamp of parallel compression that dramatically reduces the feature’s CPU overhead by up to 65% while maintaining or improving throughput for compression-heavy workloads. We expect this to broaden the set of workloads that could benefit from parallel compression, especially for bulk SST generation and remote compaction use cases that are less sensitive to CPU responsiveness.

Background

Parallel compression in RocksDB (CompressionOptions::parallel_threads > 1) allows multiple threads to compress different blocks simultaneously during SST file generation, which can significantly improve compaction throughput for workloads where compression is a bottleneck. However, the original implementation had substantial CPU overhead that often outweighed the benefits, limiting its practical adoption.

What’s New: A Complete Reimplementation

The parallel compression framework has been completely rewritten from the ground up in pull request #13910 to address the core inefficiencies:

Ring Buffer Architecture

Instead of separate compression and write queues with complex thread coordination, the new implementation uses a ring buffer of blocks-in-progress that enables efficient work distribution across threads. This bounds working memory while enabling high throughput with minimal cross-thread synchronization.

Work-Stealing Design

Previously, the calling thread could only generate uncompressed blocks, dedicated compression threads could only compress, and a writer thread could only write the SST file to storage. Now, all threads can participate in compression work in a quasi-work-stealing manner, dramatically reducing the need for threads to block waiting for work. While only one thread (the calling thread or “emit thread”) can generate uncompressed SST blocks in the new implementation, feeding compression work to other threads and itself, all other threads are compatible with writing compressed blocks to storage.

Auto-Scaling Thread Management

The ring buffer enables another key feature: auto-scaling of active threads based on ring buffer utilization. The framework intelligently wakes up idle worker threads only when there’s sufficient work to justify the overhead, achieving near-maximum throughput while minimizing CPU waste from unnecessary thread wake-ups.

Lock-Free Synchronization

The entire framework is now lock-free (and wait-free as long as compatible work units are available for each thread), based primarily on atomic operations. To cleanly pack and leverage many data fields into a single atomic value, I’ve developed a new BitFields utility API. This is proving useful for cleaning up the HyperClockCache implementation as well, and will be the topic of a later blog post.

Semaphores are used for lock-free management of idle threads (assuming a lock-free semaphore implementation, which is likely the case with ROCKSDB_USE_STD_SEMAPHORES but that is untrustworthy; see below).

Performance Improvements

The results speak for themselves. Here’s a comparison using db_bench fillseq benchmarks with various compression configurations:

ZSTD Compression (Default Level)

Note:

“throughput” = how quickly a given CPU-bound flush or compaction can complete
“CPU increase” = total CPU usage in amount of time that each core was used
“PT” = parallel_threads setting.

Before:

PT=3: ~38% throughput increase for ~73% CPU increase
PT=6: No throughput increase for ~70% CPU increase

After:

PT=3: ~58% throughput increase for ~25% CPU increase
PT=6: ~58% throughput increase for ~28% CPU increase

High Compression Scenarios

For ZSTD compression level 8, the improvements are even more dramatic:

Before:

PT=4: 2.6x throughput increase for 139% CPU increase
PT=8: 3.6x throughput increase for 135% CPU increase

After:

PT=4: 2.8x throughput increase for 114% CPU increase
PT=8: 3.7x throughput increase for 116% CPU increase

Compression Algorithm Optimizations

Alongside the parallel compression revamp, some optimizations have gone into the underlying compression implementations/integrations. Most notably, LZ4HC received dramatic performance improvements through better reuse of internal data structures between compression calls (detailed in pull request #13805). A small regression in LZ4 performance from that change was fixed in pull request #14017.

While ZSTD remains the gold standard for medium-to-high compression ratios in RocksDB, these LZ4HC optimizations make it an increasingly attractive option for read-heavy workloads where LZ4’s faster decompression can provide overall performance benefits.

Production Ready

With these efficiency improvements, parallel compression is now considered production-ready. The feature has been thoroughly tested in both unit tests and stress testing, including validation on high-load scenarios with hundreds of concurrent compression jobs and thousands of threads.

Some notes on current limitations:

Parallel compression is currently incompatible with UserDefinedIndex and with the deprecated decouple_partitioned_filters=false setting
Maximum performance is available with -DROCKSDB_USE_STD_SEMAPHORES at compile time, though this is not currently recommended due to reported bugs in some implementations of C++20 semaphores

Configuration Recommendations

The dramatically reduced CPU overhead means parallel compression is now viable for a broader range of workloads, particularly those using higher compression levels or compression-heavy scenarios like time-series data. However, simply enabling parallel compression could result in more spiky CPU loads for hosts serving live DB data. Parallel compression might be most useful for bulk SST file generation and/or remote compaction workloads because they are less sensitive to CPU responsiveness. In these scenarios there is little danger in setting parallel_threads=8 even with the possibility of over-subscribing CPU cores, though the potentially safer “sweet spot” is typically around parallel_threads=3, depending on compression level, etc.

Limitations and Future

Although this offers a great improvement in the implementation of an existing option, we recognize that this setup is suboptimal in a number of ways:

There is no work sharing / thread pooling for these SST compression/writer threads among compactions in the same process, so not well able to fit the workload to available CPU cores and not able to use other SST file compression work to avoid a worker thread going to sleep.
We are not (yet) using a framework that would allow micro-work sharing with things other than SST generation on a set of threads. That would be a good direction for effective sharing of CPU resources without spikes in usage, but might incur intolerable CPU overhead in managing work. With this “hand optimized” and specialized framework, we can at least evaluate such future endeavors against a perhaps ideal framework in terms of parallelizing with minimal overhead.

Try It Out

Parallel compression revamp will be available in RocksDB 10.7. As always, we recommend testing in your specific environment to determine the optimal configuration for your workload.

IO Activity Tagging

Thu, 25 Sep 2025 00:00:00 +0000

Context

RocksDB performs a variety of IO operations—user reads, background compactions, flushes, database opens, and verification tasks. Treating all these operations the same makes it difficult for file system implementers to optimize performance, prioritize latency-sensitive IOs, and diagnose bottlenecks. To solve that, RocksDB internally tags every IO operation with its activity type using the IOActivity enum. This automatic tagging provides precise context for each IO, enabling file systems to make smarter, context-aware decisions for scheduling, caching, and resource management.

How Internal IO Tagging Works

RocksDB automatically assigns an IOActivity tag to each IO operation. This tag is propagated through the storage stack and included in the IO options passed to the file system.

enum class IOActivity : uint8_t {
    kFlush = 0,                        // IO for flush operations (background write)
    kCompaction = 1,                   // IO for compaction (background read/write)
    kDBOpen = 2,                       // IO during database open (read/write)
    kGet = 3,                          // User Get() read
    kMultiGet = 4,                     // User MultiGet() read
    kDBIterator = 5,                   // User iterator read
    kVerifyDBChecksum = 6,             // Verification: DB checksum
    kVerifyFileChecksums = 7,          // Verification: file checksums
    kGetEntity = 8,                    // Entity Get (e.g., wide-column)
    kMultiGetEntity = 9,               // Entity MultiGet
    kGetFileChecksumsFromCurrentManifest = 10, // Manifest checksum reads
    // 0x80–0xFE: Reserved for custom/internal use
    kUnknown = 0xFF                    // Unknown/unspecified activity
};

Access IO Tag in File System

Custom file systems can access the IOActivity tag via the IO options structure provided by RocksDB. This allows them to optimize behavior based on the specific IO activity.

Status CustomFileSystem::Append(uint64_t offset, const Slice& data, const IOOptions& io_opts, ...) {
    switch (io_opts.io_activity) {
        case Env::IOActivity::kGet:
            // Prioritize or cache user reads
            break;
        case Env::IOActivity::kCompaction:
            // Throttle or deprioritize background compaction IO
            break;
        case Env::IOActivity::kDBOpen:
            // Track or optimize DB open IO
            break;
        // ... handle other activities ...
        default:
            // Default handling
            break;
    }
}

IO Activity Statistics in RocksDB

RocksDB provides detailed histograms for IO activities, allowing you to analyze both the aggregate time spent (in microseconds) and the count of IOs for each activity type.

// Read Histograms
FILE_READ_FLUSH_MICROS
FILE_READ_COMPACTION_MICROS
FILE_READ_DB_OPEN_MICROS
FILE_READ_GET_MICROS
FILE_READ_MULTIGET_MICROS
FILE_READ_DB_ITERATOR_MICROS
FILE_READ_VERIFY_DB_CHECKSUM_MICROS
FILE_READ_VERIFY_FILE_CHECKSUMS_MICROS

// Write Histograms
FILE_WRITE_FLUSH_MICROS
FILE_WRITE_COMPACTION_MICROS
FILE_WRITE_DB_OPEN_MICROS

Thanks to Maciej Szeszko and Andrew Chang from the RocksDB team for their contributions in expanding and maintaining the IOActivity enum.

Unified Memory Tracking

Wed, 24 Sep 2025 00:00:00 +0000

Context / Problem

Modern RocksDB deployments often run in environments with strict memory constraints—cloud VMs, containers, or hosts with hundreds of DB instances. Unpredictable memory usage can lead to out-of-memory (OOM) errors, degraded performance, or even service outages. Historically, while the block cache was the main source of memory usage, other components—such as memtables, table readers, file metadata, and temporary buffers—could consume significant memory outside the block cache’s control. This made it difficult for users to set a single memory limit and guarantee resource usage stays within expectations.

Goal

The goal of recent memory tracking work in RocksDB is to enable users to cap the total memory usage of RocksDB instances under a single, configurable limit—the block cache capacity. This is achieved by:

Tracking and charging all major memory consumers (memtables, table readers, file metadata, compression buffers, filter construction) to the block cache.
Evicting data blocks or other memory when the total tracked usage exceeds the configured limit.
Providing a fixed memory footprint for RocksDB, making it easier to run in resource-constrained environments and avoid OOMs.

Memtable Memory Charging

A major source of memory usage in RocksDB is the memtable. To ensure memtable memory is tracked and capped under a single limit, RocksDB provides the WriteBufferManager (WBM). When WBM is configured with a block cache, memtable memory usage is charged to the block cache. This helps prevent OOM errors and simplifies resource management.

std::shared_ptr<Cache> cache = HyperClockCacheOptions(capacity).MakeSharedCache();;
DBOptions db_options;
db_options.write_buffer_manager = std::make_shared<WriteBufferManager>(.., cache);

Other Memory Charging

Beyond memtables, RocksDB allows users to control memory charging for other internal roles using the cache_usage_options API. This provides fine-grained control over how memory is tracked for components like table readers, file metadata, compression dictionary buffers (CompressionOptions::max_dict_buffer_bytes:) and filter construction.

struct CacheEntryRoleOptions {
  enum class Decision {
    kEnabled,
    kDisabled,
    kFallback,
  };
  Decision charged = Decision::kFallback;
};
struct CacheUsageOptions {
  CacheEntryRoleOptions options;
  std::map<CacheEntryRole, CacheEntryRoleOptions> options_overrides;
};

...
BlockBasedTableOptions table_options;
table_options.cache_usage_options.options.charged = CacheEntryRoleOptions::Decision::kFallback;
table_options.cache_usage_options.options_overrides[CacheEntryRole::kTableBuilder] = {
  .charged = CacheEntryRoleOptions::Decision::kEnabled,
};

Default (Decision::kFallback) behavior for each memory type:

CacheEntryRole::kCompressionDictionaryBuildingBuffer: kEnabled
CacheEntryRole::kFilterConstruction: kDisabled
CacheEntryRole::kBlockBasedTableReader: kDisabled
CacheEntryRole::kFileMetadata: kDisabled

Monitoring and Observability

RocksDB provides built-in statistics to help users monitor memory usage and cache behavior. The DB::Properties::kBlockCacheEntryStats exposes detailed statistics about block cache entries, including breakdowns by each CacheEntryRole. These statistics are essential for understanding memory consumption and tuning cache configuration.

Addressing a Mitigated Misconfig Bug in the RocksDB OSS Repository

Fri, 07 Feb 2025 00:00:00 +0000

Dear RocksDB Community,

We want to share an update about the bug that allowed our bug bounty researcher to update the release note title in August 2024 involving the RocksDB open-source repository on GitHub. This issue was found and responsibly disclosed to us by an external bug bounty researcher through our Meta Bug Bounty program and quickly mitigated by our teams. We have not seen any evidence of malicious exploitation. Please note that no action is required from our community, as we have taken all necessary steps to remediate the issue.

Background

RocksDB is a high-performance storage engine library widely used in various large-scale applications. On August 21, 2024, a bug was reported to us by one of our bug bounty researchers. They were able to demonstrate the ability to obtain the GITHUB_TOKEN used in GitHub Actions workflows. This token provides write access to the metadata of the repository, and the researcher used it to change the title of the release note 9.5.2 as proof of concept. The researcher also unsuccessfully attempted to merge a change to the main branch of the repository; however, we had access controls set up to prevent it from going through.

Key Details

Incident Discovery: After the bug bounty researcher changed the open source release note title to demonstrate the vulnerability, external users noticed this change and notified RocksDB. RocksDB then reached out to the Bug Bounty program to confirm this was the result of security research.
No Malicious Abuse: The investigation confirmed that no code or data was compromised. The change was public and visible on GitHub.
Tag Reversion Clarification: On August 21, a tag named “v9.5.2” was initially published pointing to an incorrect commit. This was unrelated to the bug described here and was promptly corrected by pointing the tag to the correct commit. The release binary remains safe to use, and this correction does not impact the security or integrity of the release.

We’ve taken the following steps to mitigate and remediate the issue:

The release note title was corrected.
The workflow running on the self-hosted runner was disabled immediately.
It was confirmed that the GITHUB_TOKEN expired and is no longer in use.
The binary tagged for public release was examined to confirm that it was not compromised.
Action logs were cross-checked to ensure no other actions were taken with the compromised token, other than the release note title change and the failed attempts to merge self-approved pull requests to the main branch.
We have scoped down the access level of tokens generated for workflows to prevent similar issues. Additionally, we are developing better guidelines for bug bounty researchers to minimize disruptions during their research.

Thank you for your continued support and trust in RocksDB.

Sincerely,

The RocksDB Team

Java Foreign Function Interface

Tue, 20 Feb 2024 00:00:00 +0000

Java Foreign Function Interface (FFI)

Evolved Binary has been working on several aspects of how the Java API to RocksDB can be improved. The recently introduced FFI features in Java provide significant opportunities for improving the API. We have investigated this through a prototype implementation.

Java 19 introduced a new FFI Preview which is described as an API by which Java programs can interoperate with code and data outside of the Java runtime. By efficiently invoking foreign functions (i.e., code outside the JVM), and by safely accessing foreign memory (i.e., memory not managed by the JVM), the API enables Java programs to call native libraries and process native data without the brittleness and danger of JNI.

If the twin promises of efficiency and safety are realised, then using FFI as a mechanism to support a future RocksDB API may be of significant benefit.

Remove the complexity of JNI access to C++ RocksDB
Improve RocksDB Java API performance
Reduce the opportunity for coding errors in the RocksDB Java API

Here’s what we did. We have

created a prototype FFI branch
updated the RocksDB Java build to use Java 19
implemented an FFI Preview API version of core RocksDB feature (get())
Extended the current JMH benchmarks to also benchmark the new FFI methods. Usefully, JNI and FFI can co-exist peacefully, so we use the existing RocksDB Java to do support work around the FFI-based get() implementation.

Implementation

How JNI Works

JNI requires a preprocessing step during build/compilation to generate header files which are linked into by Pure Java code. C++ implementations of the methods in the headers are implemented. Corresponding native methods are declared in Java and the whole is linked together.

Code in the C++ methods uses what amounts to a JNI library to access Java values and objects and to create Java objects in response.

How FFI Works

FFI provides the facility for Java to call existing native (in our case C++) code from Pure Java without having to generate support files during compilation steps. FFI does support an external tool (jextract) which makes generating common boilerplate easier and less error prone, but we choose to start prototyping without it, in part better to understand how things really work.

FFI does its job by providing 2 things

A model for allocating, reading and writing native memory and native structures within that memory
A model for discovering and calling native methods with parameters consisting of native memory references and/or values

The called C++ is invoked entirely natively. It does not have to access any Java objects to retrieve data it needs. Therefore existing packages in C++ and other sufficiently low level languages can be called from Java without having to implement stubs in the C++.

Our Approach

While we could in principle avoid writing any C++, C++ objects and classes are not easily defined in the FFI model, so to begin with it is easier to write some very simple C-like methods/stubs in C++ which can immediately call into the object-oriented core of RocksDB. We define structures with which to pass parameters to and receive results from the C-like method(s) we implement.

`C++` Side

The first method we implement is

extern "C" int rocksdb_ffi_get_pinnable(
    ROCKSDB_NAMESPACE::DB* db, ROCKSDB_NAMESPACE::ReadOptions* read_options,
    ROCKSDB_NAMESPACE::ColumnFamilyHandle* cf, rocksdb_input_slice_t* key,
    rocksdb_pinnable_slice_t* value);

our input structure is

typedef struct rocksdb_input_slice {
  const char* data;
  size_t size;
} rocksdb_input_slice_t;

and our output structure is a pinnable slice (of which more later)

typedef struct rocksdb_pinnable_slice {
  const char* data;
  size_t size;
  ROCKSDB_NAMESPACE::PinnableSlice* pinnable_slice;
  bool is_pinned;
} rocksdb_pinnable_slice_t;

`Java` Side

We implement an FFIMethod class to advertise a java.lang.invoke.MethodHandle for each of our helper stubs

  public static MethodHandle GetPinnable; // handle which refers to the rocksdb_ffi_get_pinnable method in C++
  public static MethodHandle ResetPinnable; // handle which refers to the rocksdb_ffi_reset_pinnable method in C++

We also implement an FFILayout class to describe each of the passed structures (rocksdb_input_slice , rocksdb_pinnable_slice and rocksdb_output_slice) in Java terms

 public static class InputSlice {
  static final GroupLayout Layout = ...
  static final VarHandle Data = ...
  static final VarHandle Size =  ...
 };

 public static class PinnableSlice {
  static final GroupLayout Layout = ...
  static final VarHandle Data = ...
  static final VarHandle Size =  ...
  static final VarHandle IsPinned =  ...
 };

 public static class OutputSlice {
  static final GroupLayout Layout = ...
  static final VarHandle Data = ...
  static final VarHandle Size =  ...
 };

The FFIDB class, which implements the public Java FFI API methods, makes use of FFIMethod and FFILayout to make the code for each individual method as idiomatic and efficient as possible. This class also contains java.lang.foreign.MemorySession and java.lang.foreign.SegmentAllocator objects which control the lifetime of native memory sessions and allow us to allocate lifetime-limited native memory which can be written and read by Java, and passed to native methods.

At the user level, we then present a method which wraps the details of use of FFIMethod and FFILayout to implement our single, core Java API get() method

 public GetPinnableSlice getPinnableSlice(final ReadOptions readOptions,
      final ColumnFamilyHandle columnFamilyHandle, final MemorySegment keySegment,
      final GetParams getParams)

The flow of implementation of getPinnableSlice(), in common with any other core RocksDB FFI API method becomes:

Allocate MemorySegments for C++ structures using Layouts from FFILayout
Write to the allocated structures using VarHandles from FFILayout
Invoke the native method using the MethodHandle from FFIMethod and addresses of instantiated MemorySegments, or value types, as parameters
Read the call result and the output parameter(s), again using VarHandles from FFILayout to perform the mapping.

For the getPinnableSlice() method, on successful return from an invocation of rocksdb_ffi_get(), the PinnableSlice object will contain the data and size fields of a pinnable slice (see below) containing the requested value. A MemorySegment referring to the native memory of the pinnable slice is then constructed, and used by the client to retrieve the value in whatever fashion they choose.

Pinnable Slices

RocksDB offers core (C++) API methods using the concept of a PinnableSlice to return fetched data values while reducing copies to a minimum. We take advantage of this to base our central get() method(s) on PinnableSlices. Methods mirroring the existing JNI-based API can then be implemented in pure Java by wrapping the core getPinnableSlice().

So we implement

public record GetPinnableSlice(Status.Code code, Optional<FFIPinnableSlice> pinnableSlice) {}

public GetPinnableSlice getPinnableSlice(
      final ColumnFamilyHandle columnFamilyHandle, final byte[] key)

and we wrap that to provide

public record GetBytes(Status.Code code, byte[] value, long size) {}

public GetBytes get(final ColumnFamilyHandle columnFamilyHandle, final byte[] key)

Benchmark Results

We extended existing RocksDB Java JNI benchmarks with new benchmarks based on FFI. Full benchmark run on Ubuntu, including new benchmarks.

java --enable-preview --enable-native-access=ALL-UNNAMED -jar target/rocksdbjni-jmh-1.0-SNAPSHOT-benchmarks.jar -p keyCount=100000 -p keySize=128 -p valueSize=4096,65536 -p columnFamilyTestType="no_column_family" -rf csv org.rocksdb.jmh.GetBenchmarks

Discussion

We have plotted the performance (more operations is better) of a selection of benchmarks,

q "select Benchmark,Score from ./plot/jmh-result-fixed.csv where \"Param: keyCount\"=100000 and \"Param: valueSize\"=65536 -d, -H

JNI versions of benchmarks are previously implemented jmh benchmarks for measuring the performance of the current RocksDB Java interface.
FFI versions of benchmarks are equivalent benchmarks (as far as possible) implemented using the FFI mechanisms.

We can see that for all benchmarks which have equivalent FFI and JNI pairs, the JNI version is only very marginally faster. FFI has successfully optimized away most of the extra safety-checking of the new invocation mechanism.

Our initial implementation of FFI benchmarks lagged the JNI benchmarks quite significantly, but we have received extremely helpful support from Maurizio Cimadamore of the Panama Dev team, to help us optimize the performance of our FFI implementation. We consider that the small remaining performance gap is a feature of the remaining extra bounds checking of FFI.

For basic get() the result buffer is allocated by the method, so that there is a cost of allocation associated with each request.

ffiGet vs get
The JNI version is very marginally faster than FFI

For preallocated get() where the result buffer is supplied to the method, we avoid an allocation of a fresh result buffer on each call, and the test recycles its result buffers. Then the same small difference persists

JNI is very marginally faster than FFI
preallocatedGet() is a lot faster than basic get()

We implemented some methods where the key for the get() is randomized, so that any ordering effects can be accounted for. The same differences persisted.

The FFI interface gives us a natural way to expose RocksDB’s pinnable slice mechanism. When we provide a benchmark which accesses the raw PinnableSlice API, as expected this is the fastest method of any; however we are not comparing like with like:

ffiGetPinnableSlice() returns a handle to the RocksDB memory containing the slice, and presents that as an FFI MemorySegment. No copying of the memory in the segment occurs.

As noted above, we implement the new FFI-based get() methods using the new FFI-based getPinnableSlice() method, and copying out the result. So the ffiGet and ffiPreallocatedGet benchmarks use this mechanism underneath.

In an effort to discover whether using the Java APIs to copy from the pinnable slice backed MemorySegment was a problem, we implemented a separate ffiGetOutputSlice() benchmark which copies the result into a (Java allocated native memory) segment at the C++ side.

ffiGetOutputSlice() is faster than ffiPreallocatedGet() and is in fact at least as fast as preallocatedGet(), which is an almost exact analogue in the JNI world.

So it appears that we can build an FFI-based API with equal performance to the JNI-based one.

Thinking about the (very small, but probably statistically significant) difference between our ffiGetPinnableSlice()-based FFI calls and the JNI-based calls, it is reasonable to expect that some of the cost is the extra FFI call to C++ to release the pinned slice as a separate operation. A null FFI method call is extremely fast, but it does take some time.

We would recommend looking again the performance of the FFI-based implementation when Panama is release post-Preview in Java 21. It seems that at least with Java 20 the performance is of our FFI benchmarks is not significantly different from that of the Java 19 version.

Copies versus Calls

The second method call over the FFI boundary to release a pinnable slice has a cost. We compared the ffiGetOutputSlice() and ffiGetPinnableSlice() benchmarks in order to examine this cost. We ran it with a fixed ky size (128 bytes); the key size is likely to be pretty much irrelevant anyway; we varied the value size read from 16 bytes to 16k, and we found a crossover point between 1k and 4k for performance:

ffiGetOutputSlice() is faster when values read are 1k in size or smaller. The cost of an extra copy in the C++ side from the pinnable slice buffer into the supplied buffer allocated by Java Foreign Memory API is less than the cost of the extra call to release a pinnable slice.
ffiGetPinnableSlice() is faster when values read are 4k in size, or larger. Consistent with intuition, the advantage grows with larger read values.

The way that the RocksDB API is constructed means that of the 2 methods compared, ffiGetOutputSlice() will always make exactly 1 more copy than ffiGetPinnableSlice(). The underlying RocksDB C++ API will always copy into its own temporary buffer if it decides that it cannot pin an internal buffer, and that will be returned as the pinnable slice. There is a potential optimization where the temporary buffer could be replaced by an output buffer, such as that supplied by ffiGetOutputSlice(); in practice that is a hard fix to hack in. Its effectiveness depends on how often RocksDB fails to pin an internal buffer.

A solution which either filled a buffer or returned a pinnable slice would give us the best of both worlds.

Other Conclusions

Build Processing

It is easier to implement an interface using FFI than JNI. No intermediate build processing or code generation steps were needed to implement this protoype.
For a production version, we would urge using jextract to automate the process of generating Java API methods from the set of supporting stubs we generate.

Safety

The use of jextract will give a similar level of type security to the use of JNI, when crossing the language boundary. But we do not believe FFI is significantly more type-safe than JNI for method invocation. Neither is it less safe, though.

Native Memory

Panama’s Foreign-Memory Access API appears to us to be the most significant part of the whole project. At the Java side of RocksDB it gives us a clean mechanism (a MemorySegment) for holding RocksDB data (e.g. as from the result of a get()) call pending its forwarding to client code or network buffers.

We have taken advantage of this mechanism to provide the core FFIDB.getPinnableSlice() method in our Panama-based API. The rest of our prototype get() API, duplicating the existing JNI-based API, is then a Pure Java library on top of FFIDB.getPinnableSlice() and FFIPinnableSlice.reset().

The common standard for foreign memory opens up the possibility of efficient interoperation between RocksDB and Java clients (e.g. Kafka). We think that this is really the key to higher performing, more integrated Java-based systems:

This could result in data never being copied into Java memory, or a significant reduction in copies, as native MemorySegments are handed off between co-operating Java clients of fundamentally native APIs. This extra potential performance can be extremely useful when 2 or more clients are interoperating; we still need to provide a simplest possible API wrapping these calls (like our prototype get()), which operates at a similar level to the current Java API.
Some thought should be applied to how this architecture would interact with the cache layer(s) in RocksDB, and whether it can be accommodated within the present RocksDB architecture. How long can 3rd-party applications pin pages in the RocksDB cache without disrupting RocksDB normal behaviour (e.g. compaction) ?

Summary

Panama/FFI (in Preview) is a highly capable technology for (re)building the RocksDB Java API, although the supported language level of RocksDB and the planned release schedule for Panama mean that it could not replace JNI in production for some time to come.
Panama/FFI would seem to offer comparable performance to JNI; there is no strong performance argument for a re-implementation of a standalone RocksDB Java API. But the opportunity to provide a natural pinnable slice-based API gives a lot of flexibility; not least because an efficient API could be built mostly in Java with only a small underlying layer implementing the pinnable slice interface.
Panama/FFI can remove some boilerplate (native method declarations) and allow Java programs to access C libraries without stub code, but calling a C++-based library still requires C stubs; a possible approach would be to use the RocksDB C API as the basis for a rebuilt Java API. This would allow us to remove all the existing JNI boilerplate, and concentrate support effort on the C API. An alternative approach would be to build a robust API based on Reference Counting, but using FFI.
Panama/FFI really shines as a foreign memory standard for a Java API that can allow efficient interoperation between RocksDB Java clients and other (Java and native) components of a system. Foreign Memory gives us a model for how to efficiently return data from RocksDB; as pinnable slices with their contents presented in MemorySegments. If we focus on designing an API for native interoperability we think this can be highly productive in opening RocksDB to new uses and opportunities in future.

Appendix

Code and Data

The Experimental Pull Request contains the source code implemented, together with further data plots and the source CSV files for all data plots.

Running

This is an example run; the jmh parameters (after -p) can be changed to measure performance with varying key counts, and key and value sizes.

java --enable-preview --enable-native-access=ALL-UNNAMED -jar target/rocksdbjni-jmh-1.0-SNAPSHOT-benchmarks.jar -p keyCount=100000 -p keySize=128 -p valueSize=4096,65536 -p columnFamilyTestType="no_column_family" -rf csv org.rocksdb.jmh.GetBenchmarks -wi 1 -to 1m -i 1

Processing

Use q to select the csv output for analysis and graphing.

Note that we edited the column headings for easier processing

q "select Benchmark,Score,Error from ./plot/jmh-result.csv where keyCount=100000 and valueSize=65536" -d, -H -C readwrite

Java 19 installation

We followed the instructions to install Azul. Then select the correct instance of java locally:

sudo update-alternatives --config java
sudo update-alternatives --config javac

And set JAVA_HOME appropriately. In my case, sudo update-alternatives --config java listed a few JVMs thus:

          /usr/lib/jvm/bellsoft-java8-full-amd64/bin/java   20803123  auto mode
          /usr/lib/jvm/bellsoft-java8-full-amd64/bin/java   20803123  manual mode
          /usr/lib/jvm/java-11-openjdk-amd64/bin/java       1111      manual mode
* 3            /usr/lib/jvm/zulu19/bin/java                      2193001   manual mode

For our environment, we set this:

export JAVA_HOME=/usr/lib/jvm/zulu19

The default version of Maven avaiable on the Ubuntu package repositories (3.6.3) is incompatible with Java 19. You will need to install a later Maven, and use it. I used 3.8.7 successfully.

Java 20, 21, 22 and subsequent versions

The FFI version we used was a preview in Java 19, and the interface has changed through to Java 22, where it has been finalized. Future work with this prototype will need to update the code to use the changed interface.

Java API Performance Improvements

Mon, 06 Nov 2023 00:00:00 +0000

RocksDB Java API Performance Improvements

Evolved Binary has been working on several aspects of how the Java API to RocksDB can be improved. Two aspects of this which are of particular importance are performance and the developer experience.

We have built some synthetic benchmark code to determine which are the most efficient methods of transferring data between Java and C++.
We have used the results of the synthetic benchmarking to guide plans for rationalising the API interfaces.
We have made some opportunistic performance optimizations/fixes within the Java API which have already yielded noticable improvements.

Synthetic JNI API Performance Benchmarks

The synthetic benchmark repository contains tests designed to isolate the Java to/from C++ interaction of a canonical data intensive Key/Value Store implemented in C++ with a Java (JNI) API layered on top.

JNI provides several mechanisms for allowing transfer of data between Java buffers and C++ buffers. These mechanisms are not trivial, because they require the JNI system to ensure that Java memory under the control of the JVM is not moved or garbage collected whilst it is being accessed outside the direct control of the JVM.

We set out to determine which of multiple options for transfer of data from C++ to Java and vice-versa were the most efficient. We used the Java Microbenchmark Harness to set up repeatable benchmarks to measure all the options.

We explore these and some other potential mechanisms in the detailed results (in our Synthetic JNI performance repository)

We summarise this work here:

The Model

In C++ we represent the on-disk data as an in-memory map of (key, value) pairs.
For a fetch query, we expect the result to be a Java object with access to the contents of the value. This may be a standard Java object which does the job of data access (a byte[] or a ByteBuffer) or an object of our own devising which holds references to the value in some form (a FastBuffer pointing to com.sun.unsafe.Unsafe unsafe memory, for instance).

Data Types

There are several potential data types for holding data for transfer, and they are unsurprisingly quite connected underneath.

Byte Array

The simplest data container is a raw array of bytes (byte[]).

There are 3 different mechanisms for transferring data between a byte[] and C++

At the C++ side, the method JNIEnv.GetArrayCritical() allows access to a C++ pointer to the underlying array.
The JNIEnv methods GetByteArrayElements() and ReleaseByteArrayElements() fetch references/copies to and from the contents of a byte array, with less concern for critical sections than the critical methods, though they are consequently more likely/certain to result in (extra) copies.
The JNIEnv methods GetByteArrayRegion() and SetByteArrayRegion() transfer raw C++ buffer data to and from the contents of a byte array. These must ultimately do some data pinning for the duration of copies; the mechanisms may be similar or different to the critical operations, and therefore performance may differ.

Byte Buffer

A ByteBuffer abstracts the contents of a collection of bytes, and was in fact introduced to support a range of higher-performance I/O operations in some circumstances.

There are 2 types of byte buffers in Java, indirect and direct. Indirect byte buffers are the standard, and the memory they use is on-heap as with all usual Java objects. In contrast, direct byte buffers are used to wrap off-heap memory which is accessible to direct network I/O. Either type of ByteBuffer can be allocated at the Java side, using the allocate() and allocateDirect() methods respectively.

Direct byte buffers can be created in C++ using the JNI method JNIEnv.NewDirectByteBuffer() to wrap some native (C++) memory.

Direct byte buffers can be accessed in C++ using the JNIEnv.GetDirectBufferAddress() and measured using JNIEnv.GetDirectBufferCapacity()

Unsafe Memory

The call com.sun.unsafe.Unsafe.allocateMemory() returns a handle which is (of course) just a pointer to raw memory, and can be used as such on the C++ side. We could turn it into a byte buffer on the C++ side by calling JNIEnv.NewDirectByteBuffer(), or simply use it as a native C++ buffer at the expected address, assuming we record or remember how much space was allocated.

A custom FastBuffer class provides access to unsafe memory from the Java side.

Allocation

For these benchmarks, allocation has been excluded from the benchmark costs by pre-allocating a quantity of buffers of the appropriate kind as part of the test setup. Each run of the benchmark acquires an existing buffer from a pre-allocated FIFO list, and returns it afterwards. A small test has confirmed that the request and return cycle is of insignificant cost compared to the benchmark API call.

GetJNIBenchmark Performance

Benchmarks ran for a duration of order 6 hours on an otherwise unloaded VM, the error bars are small and we can have strong confidence in the values derived and plotted.

Comparing all the benchmarks as the data size tends large, the conclusions we can draw are:

Indirect byte buffers add cost; they are effectively an overhead on plain byte[] and the JNI-side only allows them to be accessed via their encapsulated byte[].
SetRegion and GetCritical mechanisms for copying data into a byte[] are of very comparable performance; presumably the behaviour behind the scenes of SetRegion is very similar to that of declaring a critical region, doing a memcpy() and releasing the critical region.
GetElements methods for transferring data from C++ to Java are consistently less efficient than SetRegion and GetCritical.
Getting into a raw memory buffer, passed as an address (the handle of an Unsafe or of a netty ByteBuf) is of similar cost to the more efficient byte[] operations.
Getting into a direct nio.ByteBuffer is of similar cost again; while the ByteBuffer is passed over JNI as an ordinary Java object, JNI has a specific method for getting hold of the address of the direct buffer, and using this, the get() cost with a ByteBuffer is just that of the underlying C++ memcpy().

At small(er) data sizes, we can see whether other factors are important.

Indirect byte buffers are the most significant overhead here. Again, we can conclude that this is due to pure overhead compared to byte[] operations.
At the lowest data sizes, netty ByteBufs and unsafe memory are marginally more efficient than byte[]s or (slightly less efficient) direct nio.Bytebuffers. This may be explained by even the small cost of calling the JNI model on the C++ side simply to acquire a direct buffer address. The margins (nanoseconds) here are extremely small.

Post processing the results

Our benchmark model for post-processing is to transfer the results into a byte[]. Where the result is already a byte[] this may seem like an unfair extra cost, but the aim is to model the least cost processing step for any kind of result.

Copying into a byte[] using the bulk methods supported by byte[], nio.ByteBuffer have comparable performance.
Accessing the contents of an Unsafe buffer using the supplied unsafe methods is inefficient. The access is word by word, in Java.
Accessing the contents of a netty ByteBuf is similarly inefficient; again the access is presumably word by word, using normal Java mechanisms.

PutJNIBenchmark

We benchmarked Put methods in a similar synthetic fashion in less depth, but enough to confirm that the performance profile is similar/symmetrical. As with get() using GetElements is the least performant way of implementing transfers to/from Java objects in C++/JNI, and other JNI mechanisms do not differ greatly one from another.

Lessons from Synthetic API

Performance analysis shows that for get(), fetching into allocated byte[] is equally as efficient as any other mechanism, as long as JNI region methods are used for the internal data transfer. Copying out or otherwise using the result on the Java side is straightforward and efficient. Using byte[] avoids the manual memory management required with direct nio.ByteBuffers, which extra work does not appear to provide any gain. A C++ implementation using the GetRegion JNI method is probably to be preferred to using GetCritical because while their performance is equal, GetRegion is a higher-level/simpler abstraction.

Vitally, whatever JNI transfer mechanism is chosen, the buffer allocation mechanism and pattern is crucial to achieving good performance. We experimented with making use of netty’s pooled allocator part of the benchmark, and the difference of getIntoPooledNettyByteBuf, using the allocator, compared to getIntoNettyByteBuf using the same pre-allocate on setup as every other benchmark, is significant.

Equally importantly, transfer of data to or from buffers should where possible be done in bulk, using array copy or buffer copy mechanisms. Thought should perhaps be given to supporting common transformations in the underlying C++ layer.

API Recommendations

Of course there is some noise within the results. but we can agree:

Don’t make copies you don’t need to make
Don’t allocate/deallocate when you can avoid it

Translating this into designing an efficient API, we want to:

Support API methods that return results in buffers supplied by the client.
Support byte[]-based APIs as the simplest way of getting data into a usable configuration for a broad range of Java use.
Support direct ByteBuffers as these can reduce copies when used as part of a chain of ByteBuffer-based operations. This sort of sophisticated streaming model is most likely to be used by clients where performance is important, and so we decide to support it.
Support indirect ByteBuffers for a combination of reasons:
- API consistency between direct and indirect buffers
- Simplicity of implementation, as we can wrap byte[]-oriented methods
Continue to support methods which allocate return buffers per-call, as these are the easiest to use on initial encounter with the RocksDB API.

High performance Java interaction with RocksDB ultimately requires architectural decisions by the client

Use more complex (client supplied buffer) API methods where performance matters
Don’t allocate/deallocate where you don’t need to
- recycle your own buffers where this makes sense
- or make sure that you are supplying the ultimate destination buffer (your cache, or a target network buffer) as input to RocksDB get() and put() calls

We are currently implementing a number of extra methods consistently across the Java fetch and store APIs to RocksDB in the PR Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() according to these principles.

Optimizations

Reduce Copies within API Implementation

Having analysed JNI performance as described, we reviewed the core of RocksJNI for opportunities to improve the performance. We noticed one thing in particular; some of the get() methods of the Java API had not been updated to take advantage of the new PinnableSlice methods.

Fixing this turned out to be a straightforward change, which has now been incorporated in the codebase Improve Java API get() performance by reducing copies

Performance Results

Using the JMH performances tests we updated as part of the above PR, we can see a small but consistent improvement in performance for all of the different get method variants which we have enhanced in the PR.

java -jar target/rocksdbjni-jmh-1.0-SNAPSHOT-benchmarks.jar -p keyCount=1000,50000 -p keySize=128 -p valueSize=1024,16384 -p columnFamilyTestType="1_column_family" GetBenchmarks.get GetBenchmarks.preallocatedByteBufferGet GetBenchmarks.preallocatedGet

The y-axis shows ops/sec in throughput, so higher is better.

Analysis

Before the invention of the Pinnable Slice the simplest RocksDB (native) API Get() looked like this:

Status Get(const ReadOptions& options,
                           ColumnFamilyHandle* column_family, const Slice& key,
                           std::string* value)

After PinnableSlice the correct way for new code to implement a get() is like this

Status Get(const ReadOptions& options,
                    ColumnFamilyHandle* column_family, const Slice& key,
                    PinnableSlice* value)

But of course RocksDB has to support legacy code, so there is an inline method in db.h which re-implements the former using the latter. And RocksJava API implementation seamlessly continues to use the std::string-based get()

Let’s examine what happens when get() is called from Java

jint Java_org_rocksdb_RocksDB_get__JJ_3BII_3BIIJ(
   JNIEnv* env, jobject, jlong jdb_handle, jlong jropt_handle, jbyteArray jkey,
   jint jkey_off, jint jkey_len, jbyteArray jval, jint jval_off, jint jval_len,
   jlong jcf_handle)

Create an empty std::string value
Call DB::Get() using the std::string variant
Copy the resultant std::string into Java, using the JNI SetByteArrayRegion() method

So stage (3) costs us a copy into Java. It’s mostly unavoidable that there will be at least the one copy from a C++ buffer into a Java buffer.

But what does stage 2 do ?

Create a PinnableSlice(std::string&) which uses the value as the slice’s backing buffer.
Call DB::Get() using the PinnableSlice variant
Work out if the slice has pinned data, in which case copy the pinned data into value and release it.
..or, if the slice has not pinned data, it is already in value (because we tried, but couldn’t pin anything).

So stage (2) costs us a copy into a std::string. But! It’s just a naive std::string that we have copied a large buffer into. And in RocksDB, the buffer is or can be large, so an extra copy something we need to worry about.

Luckily this is easy to fix. In the Java API (JNI) implementation:

Create a PinnableSlice() which uses its own default backing buffer.
Call DB::Get() using the PinnableSlice variant of the RocksDB API
Copy the data indicated by the PinnableSlice straight into the Java output buffer using the JNI SetByteArrayRegion() method, then release the slice.
Work out if the slice has successfully pinned data, in which case copy the pinned data straight into the Java output buffer using the JNI SetByteArrayRegion() method, then release the pin.
..or, if the slice has not pinned data, it is in the pinnable slice’s default backing buffer. All that is left, is to copy it straight into the Java output buffer using the JNI SetByteArrayRegion() method.

In the case where the PinnableSlice has succesfully pinned the data, this saves us the intermediate copy to the std::string. In the case where it hasn’t, we still have the extra copy so the observed performance improvement depends on when the data can be pinned. Luckily, our benchmarking suggests that the pin is happening in a significant number of cases.

On discussion with the RocksDB core team we understand that the core PinnableSlice optimization is most likely to succeed when pages are loaded from the block cache, rather than when they are in memtable. And it might be possible to successfully pin in the memtable as well, with some extra coding effort. This would likely improve the results for these benchmarks.

RocksDB

Resumable Remote Compaction

Background

How Resumable Remote Compaction Works

Checkpointing

Resuming

How to Enable It

Future Work

Interpolation search for SST index blocks

The idea

Turning a key into a number

How to enable it

kAuto: per-block selection

Write overhead

Benchmarks

Compatibility

Future work

RocksDB development finds a CPU bug

Background: Unique Identifiers

High Quality Randomness

Trust But Verify

That’s Weird

Root Cause Analysis

With Apologies

Key Takeaways

BitFields API: Type-Safe Bit Packing for Lock-Free Data Structures

The Problem: Managing Packed Bit Fields

New Solution: BitFields API

Key Features

Type Safety and Self-Documentation

Zero Overhead

Atomic Operations with Transforms

Overflow Protection

Compare-and-Swap (CAS) Support

Atomic wrappers

Real-World Usage in RocksDB

Before: Manual Bit Manipulation

Future Directions

Conclusion

Parallel Compression Revamp: Dramatically Reduced CPU Overhead

Background

What’s New: A Complete Reimplementation

Ring Buffer Architecture

Work-Stealing Design

Auto-Scaling Thread Management

Lock-Free Synchronization

Performance Improvements

ZSTD Compression (Default Level)

High Compression Scenarios

Compression Algorithm Optimizations

Production Ready

Configuration Recommendations

Limitations and Future

Try It Out

IO Activity Tagging

Context

How Internal IO Tagging Works

Access IO Tag in File System

IO Activity Statistics in RocksDB

Unified Memory Tracking

Context / Problem

Goal

Memtable Memory Charging

Other Memory Charging

Monitoring and Observability

Addressing a Mitigated Misconfig Bug in the RocksDB OSS Repository

Background

Key Details

Java Foreign Function Interface

Java Foreign Function Interface (FFI)

Implementation

How JNI Works

How FFI Works

Our Approach

C++ Side

Java Side

Pinnable Slices

Benchmark Results

Discussion

Copies versus Calls

`C++` Side

`Java` Side