Skip to content

Redis

homepage-banner

Redis is an open-source, in-memory data structure store that is commonly used as a database, cache, and message broker. It is known for its high performance, scalability, and flexibility. Redis supports a wide range of data structures, including strings, hashes, sets, sorted sets, and lists, and provides a variety of operations for each data structure.

Redis is often used in web applications to cache frequently accessed data, improve performance, and reduce database load. It is also used as a message broker in real-time systems and as a database for storing session data.

This document aims to provide a simple introduction to Redis for those who are new to it. It covers various aspects of Redis, such as data types, implementation details, best practices, and operations, to help you get started with Redis and make the most of its features.

Commands and Configuration

Common Command Query

  • Redis Command Reference (https://redis.io/commands)

Master-Slave Configuration

slaveof 10.4.44.71 3904
slaveof no one
# Set the read-only / writable of the slave
config set slave-read-only yes
config set slave-read-only no

Memory Recycling Policy

config get maxmemory-policy

When the maxmemory limit is reached, Redis will take precise action as configured by the maxmemory-policy configuration command. The following policies are available:

  • noeviction: Returns an error when the memory limit is reached. Attempting to execute commands by the client will cause more memory usage (most write commands, except DEL and some exceptions). If the data is very sensitive and data loss is not allowed, it is recommended to use the noeviction policy.
  • allkeys-lru: Recycles the least recently used (LRU) keys to make room for new data. If you have hot and cold data, it is recommended to use the allkeys-lru strategy.
  • volatile-lru: Recycles the least recently used (LRU) keys, but only recycles keys with expiration set to make room for new data.
  • allkeys-random: Recycles random keys to make room for new data. If you need to read and write all keys in a loop, or the access frequency of each key is similar, you can use the allkeys-random strategy, where the probability of reading and writing all elements is similar.
  • volatile-random: Recycles random keys, but only recycles keys with expiration set to make room for new data.
  • volatile-ttl: Recycles keys with expiration set and tries to recycle the key with the shortest time to live (TTL) to make room for new data.

Replication

Adjusting the replication timeout

config get repl-timeout
config set repl-timeout 1800

Adjusting the replication buffer size

The last two parameters after slave represent the hard limit and soft limit, respectively. The hard limit cannot be exceeded and any attempts to write beyond it will fail. The soft limit indicates that the limit has been continuously reached for 60 seconds.

config set client-output-buffer-limit "slave 2684354560 536870912 0"

Setting the size of the replication backlog buffer

config get repl-backlog-size
config set repl-backlog-size xx
(unit: bytes)

Persistence

Set the rewrite size to 80GB

config set auto-aof-rewrite-percentage 200
config set auto-aof-rewrite-min-size 80108864000

Replay commands through AOF

tail -n +6 /opt/redis_dump/appendonly.aof | pv -L 10m | nc 10.13.56.15 3707 > ~/run.log

Turn off AOF

config set appendonly no

When AOF full

When the aof file fills up the disk, its write operation fails and there is no other space on the disk to be released. In such cases, you need to clear the aof file and then execute bgrewrite. After clearing the aof file, use the following command to clear the file handle:

>appendonly.aof

If the aof file is deleted directly, the disk space will not be released because the file handle is not released. In this case, use the following command to clear the deleted file.

>/proc/${pid}/fd

Cluster

View the cluster nodes topology distribution

redis-cli -h 10.3.23.27 -p 3920 cluster nodes

Adding a new node to an existing cluster

To add a new node to an existing cluster, select any node from the current cluster (10.3.23.35:3920) and then add the new node (10.3.20.213:3920).

redis-cli -h 10.3.23.35 -p 3920 CLUSTER MEET 10.3.20.213 3920

Setting up slave nodes

To set 10.3.20.213 3920 as a slave node for node-id (fd276dc8e818f53239f988dc2bb3ec85504f9096), follow these steps:

redis-cli -h 10.3.20.213 -p 3920
 CLUSTER REPLICATE fd276dc8e818f53239f988dc2bb3ec85504f9096

Resetting a Node in the Cluster

To reset a node in the cluster, first remove it from the cluster topology, then clear and reset its data, and finally add it back to the cluster. For example, if the node to be reset is 10.3.20.213:3920 and its node-id is fd276dc8e818f53239f988dc2bb3ec85504f9096, follow these steps:

  1. Remove the node from the cluster topology.
  2. Clear and reset the node data.
  3. Add the node back to the cluster.
# Send forget command to all nodes in the cluster for the node
redis-cli -h 10.3.1.1 -p 3920 
 CLUSTER FORGET fd276dc8e818f53239f988dc2bb3ec85504f9096
# Connect to the node, clear the data and reset it
redis-cli -h 10.3.20.213 -p 3920
  FLUSHALL
  CLUSTER RESET
# Connect to any node and add the new node
redis-cli -h 10.3.1.1 -p 3920 CLUSTER MEET 10.3.20.213 3920

View the number of keys in the specified slot in the cluster

redis-cli -h 10.3.23.35 -p 3920 CLUSTER COUNTKEYSINSLOT 11587

Set the slot to a specific node

redis-cli -h 10.3.21.50 -p 3920 CLUSTER SETSLOT 11587 NODE f68aa3484f2340a9586632b2696f5a8fe0b30f40

Set the timeout for the cluster

The default is 5s.

config set cluster-node-timeout 30000

cluster rebalance

redis-cli --cluster rebalance ip:port

  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace

cluster failover

If both the master and the slave are active, switch the slave to become the master.

slave-node-cli> CLUSTER FAILOVER

If the master server goes down, switch the slave server to become the master.

slave-node-cli> CLUSTER FAILOVER [FORCE|TAKEOVER]
  • The FORCE option does not handshake with the master. Instead, it broadcasts information, communicates with other masters in the cluster, and obtains authorized votes to become the new master.
  • The TAKEOVER option does not handshake with the master and does not need to get other master votes. It is directly promoted to the master when a large number of slaves are set as the master during data center switching, or when most of the masters are not available due to a brain split.

Note: TAKEOVER violates the last-failover-wins principle of Redis Cluster. Refer to https://redis.io/commands/cluster-failover for more information.

Slot Migration Process

I. Set the destination node to the “importing state” using the following command:

CLUSTER SETSLOT <slot> IMPORTING <source-node-id>.
  • Directly request the key of this slot and return MOVED redirection.
  • First, send the ASKING command, then request the key of this slot, and process the request.

II. Set the source node to the “migrating state” using the following command:

CLUSTER SETSLOT <slot> MIGRATING <destination-node-id>.
  • Requests for existing keys are treated the same as the original processing logic.
  • If the request does not include a key, the client returns an ask redirection, and the client requests the importing node.
  • If the request contains multiple keys, some exist and some do not, TRYAGAIN is returned until all keys are migrated.

III. Read the key from the source node in a loop and transfer them to the destination node using this command:

CLUSTER GETKEYSINSLOT
MIGRATE host port key "" destination-db timeout REPLACE KEYS key [key ...]

IV. Set the slots status to normal using the following command:

 CLUSTER SETSLOT <slot> NODE <destination-node-id>
 # in the source and destination

The CLUSTER SETSLOT <slot> NODE <node-id> command is typically used to send a command to the two instances in the migration status after the migration slot is completed.

Effects:

  • Executing this command on the owner of the slot will result in an error.
  • Executing this command on the instance in the migrating state of the slot will clear the migrating state.
  • Executing this command on the instance in the importing state of the slot will clear the importing state, and check the epoch id. If the epoch id is not the largest in the cluster, a new maximum id will be generated (note that this does not communicate with the cluster, so you will need to manually reset it to the maximum).

Note: If the migration process is interrupted and the status is modified, execute the following command on the node to be repaired.

redis-cli  -h <host> -p <port> cluster setslot <slot> stable

Note: just clear migrating / importing state https://redis.io/commands/cluster-setslot

client tools

View the list of clients currently connected to the server

127.0.0.1:6379> client list
addr=127.0.0.1:52555 fd=5 name= age=855 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
addr=127.0.0.1:52787 fd=6 name= age=6 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

Each line represents various information of a connection:

  • addr: The TCP address of the client, including the IP and port.
  • fd: The file descriptor handle number corresponding to the client connection socket.
  • name: The name of the connection, which is empty by default and can be set through CLIENT SETNAME.
  • age: The number of seconds the client has been alive.
  • idle: The number of seconds the client has been idle.

View the currently executing command

monitor

Big Keys

Large keys not only reduce the throughput of the proxy, but also easily trigger blocking, and the response time of long-tail requests will increase. It is best to compress them before storing.

View

slowlog get {N}
memory usage KEY
debug object key
## Serializedlength does not represent the actual byte size. It returns the length of the object serialized using RDB encoding, which may be smaller. However, it has some auxiliary effects for troubleshooting bigkey.
redis-cli --stat
redis-cli -h {ip} -p {port} bigkeys
redis-cli -h {ip} -p {port} --latency
  • The size of string type should not exceed 10KB.
  • The number of elements in HASH, SET, ZSET, and LIST should not exceed 1000.
  • It is not recommended to use hgetall, lrange, mget, or mset to obtain thousands of elements at once.
  • When the element or number of elements is too large, it is not recommended to query all elements. For example, the zrange key 0 -1 command will result in slow queries.
  • It is not recommended to have more than 500 keys under pipeline. It is best to keep it within 100.
  • It is recommended to keep the size of string type within 1k. For json strings larger than 10k, it is recommended to store them using compression.
  • It is recommended to distribute the expiration time to prevent expiration blocking due to concentration of data.
  • Redis has a compression algorithm for data with fewer elements and smaller values, which can reduce the data structure overhead of Redis itself.
Type Maximum number of elements
Hash 512
Set 512
Zset 128
List 512

Data Types and Implementation

/images/redis-data-type.webp

Best Practice

/images/redis-best-practice.webp

/images/redis-dev.png

/images/redis-ops.webp

Reference

  • https://mp.weixin.qq.com/s/Fz1EbsmJP5k2Rh6ir_a1pQ
Small world. Big idea!
  • Welcome to visit the knowledge base of SRE and DevOps!
  • License under CC BY-NC 4.0
  • No personal information is collected
  • Made with Material for MkDocs and generative AI tools
  • Copyright issue feedback me#imzye.com, replace # with @
  • Get latest SRE news and discuss on Discord Channel