用户您好!请先登录!

KeyDB: Low TCO Redis™ Compatible Key Value Store

KeyDB: Low TCO Redis™ Compatible Key Value Store

What is KeyDB?

KeyDB is an in-memory, high performance database with support for numerous data types, on-disk persistence and high availability. Compatible with Redis™ API and protocols, KeyDB is a drop-in replacement for your existing Redis™ deployments with numerous enhancements, higher performance, and lower Total Cost of Ownership (TCO).

Designed for use as both a cache as well as use as a primary database, KeyDB is a fast and versatile NOSQL database useful in many applications. Its low resource requirements and high-speed mean fewer instances to manage and much lower costs. KeyDB looks to fully utilize the hardware provided to your instances enabling a higher throughput and efficiency.

Why KeyDB?

KeyDB is a fully opensource database that aims to make use of all hardware resources. KeyDB makes it possible to breach boundaries often dictated by price and complexity.

  • 100% Compatible with Redis protocols, API’s and modules
  • Eliminate the need for complex sharding and clustering
  • Double your resources turning replicas into masters while maintaining high availability
  • Not limited to small data sizes or bound by expensive RAM
  • Enable lightning fast speeds with a simple setup
  • Be part of a community where your input is heard and valued

Why fork Redis?

KeyDB has a different philosophy on how the codebase should evolve. We feel that ease of use, high performance, and a “batteries included” approach is the best way to create a good user experience. While we have great respect for the Redis maintainers it is our opinion that the Redis approach focusses too much on simplicity of the code base at the expense of complexity for the user. This results in the need for external components and workarounds to solve common problems – resulting in more complexity overall.

Because of this difference of opinion features which are right for KeyDB may not be appropriate for Redis. A fork allows us to explore this new development path and implement features which may never be a part of Redis. KeyDB keeps in sync with upstream Redis changes, and where applicable we upstream bug fixes and changes. It is our hope that the two projects can continue to grow and learn from each other.

How is KeyDB Different?

1. LOWER OPERATIONAL COSTS

KeyDB uses a unique multi-tier data structure to share information between memory and Non-Volatile Memory (NVM) such as flash storage. Unlike other memory only databases which keep data in expensive RAM, KeyDB can offload less frequently used data to cheaper storage mediums.

Past attempts at merging fast in-memory databases with FLASH merely bolted on a separate disk-based database. This approach requires two queries to access disk-based data, and prevents the storage of keys in non-volatile memory. As a result, the non-volatile storage cannot be fully utilized. The net effect of this approach is higher resource consumption, higher latency, and lower throughput.

In contrast, KeyDB is a fully integrated dual-tier database. KeyDB does not execute additional operations to access flash-based data, and is limited only by the throughput of the underlying hardware. As a result, KeyDB is able to achieve near-RAM performance from inexpensive non-volatile memory.

2. EFFICIENT USE OF SYSTEM RESOURCES

KeyDB contains memory and performance enhancements over vanilla Redis. KeyDB saves up to 75% less memory than Redis™1, and supporting 10%2 higher throughput vs Redis™ on the same hardware. This unique combination of performance enhancements and efficient use of tiered storage is designed to improve performance while reducing costs.

3. CLOUD FIRST DESIGN

From inception KeyDB was designed with the cloud in mind. Designed to take advantage of inexpensive ephemeral storage, KeyDB dramatically lowers cost vs traditional databases using IOP provisioned EBS storage. KeyDB is capable of backing up data directly to Amazon S3™ ensuring the lowest costs possible.

Annotation 2019-10-11 225658
KeyDB’s unique cloud optimized architecture can reduce costs by over 40%.

4. MAKING USE OF REDUNDANT INSTANCES

A big downside of maintaining a high availability setup is launching replica instances that might do very little to handle increased loads. Even if you take time to set up a replica that accept reads, your ability to handle traffic and loads has not necessarily doubled. The Active Replication and Multi-Master configurations allow replica instances to function as a combined force accepting reads and writes to each instance. This operates on the last write wins principle.

KeyDB can handle split brain scenarios where the connection between masters is severed, but writes continue to be made. Each write is timestamped and when the connection is restored each master will share their new data. The newest write will win. This prevents stale data from overwriting new data written after the connection was severed.

Active-Replica refers to a single replica node of an instance where both nodes are set to be replicas of each other. Multi-Master configuration is similar, however a variety of setups are possible. Each master can be a master of several instances or all of the other instances allowing different topologies. With timestamped writes the failed nodes can be enabled to reboot automatically and will sync to the latest values when returning.

On boot each KeyDB instance will compute a dynamic UUID. This UUID is not saved and exists only for the life of the process. When a replica connects to its master it will inform the master of its UUID and the master will reply with its own. The UUIDs are compared to inform the server if two clients are from the same KeyDB instance (IPs and ports are insufficient as there may be different processes on the same machine). The UUIDs are used to prevent rebroadcasting changes to the master that sent them if it is also our replica. A new configuration option is added to enable this mode, and when enabled also makes KeyDB writable even if it is a replica (by default this is disabled). Except for extra logic to prevent infinitely bouncing queries between clients in a loop the replication code executes as it normally would.

5. BENCHMARKING 

Annotation 2019-10-11 230003

The first chart shows that throughput is more than doubled when multi-thread support is added. However Redis™ is used not just where throughput is required but also in applications that are highly sensitive to latency. We’ve been able to achieve great results there too:

Annotation 2019-10-11 230151

For the purposes of our benchmark we are using two m5a.2xlarge instances on AWS. One to run the server, and the other to run our bench-marking tool. We decided upon memtier as our benchmark tool for its ability to generate higher levels of traffic. KeyDB is running with 4 threads, while memtier is set to use 8 threads for all tests. All traffic is within the same AWS Availability Zone using private IPs.

Introduction to KeyDB

KeyDB is an open source, in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps), hyperloglogs, geospatial indexes with radius queries and streams. KeyDB has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Active-Replication or Sentinel and automatic partitioning with KeyDB Cluster.

You can run atomic operations on these types, like appending to a string; incrementing the value in a hash; pushing an element to a list; computing set intersection, union and difference; or getting the member with highest ranking in a sorted set.

In order to achieve its outstanding performance, KeyDB works with an in-memory dataset. Depending on your use case, you can persist it either by dumping the dataset to disk every once in a while, or by appending each command to a log. Persistence can be optionally disabled, if you just need a feature-rich, networked, in-memory cache.

KeyDB also supports trivial-to-setup master-slave asynchronous replication, with very fast non-blocking first synchronization, auto-reconnection with partial resynchronization on net split.

Other features include:

  • Transactions
  • Pub/Sub
  • Lua scripting
  • Keys with a limited time-to-live
  • LRU eviction of keys
  • Automatic failover

You can use KeyDB from most programming languages out there.

KeyDB is written in ANSI C and works in most POSIX systems like Linux, *BSD, OS X without external dependencies. Linux and OS X are the two operating systems where KeyDB is developed and more tested, and we recommend using Linux for deploying. KeyDB may work in Solaris-derived systems like SmartOS, but the support is best effort. There is no official support for Windows builds, but Microsoft develops and maintains a Win-64 port of KeyDB.

Additional Resources

 

行走的code
行走的code

要发表评论,您必须先登录