Redis

homepage-banner

Introduction

Redis is an open-source, in-memory data structure store that is commonly used as a database, cache, and message broker. It is known for its high performance, scalability, and flexibility. Redis supports a wide range of data structures, including strings, hashes, sets, sorted sets, and lists, and provides a variety of operations for each data structure.

Redis is often used in web applications to cache frequently accessed data, improve performance, and reduce database load. It is also used as a message broker in real-time systems and as a database for storing session data.

This document aims to provide a simple introduction to Redis for those who are new to it. It covers various aspects of Redis, such as data types, implementation details, best practices, and operations, to help you get started with Redis and make the most of its features.

Redis User Cases

User cases

Commands and Configuration

Common Command Query

Redis Command Reference (https://redis.io/commands)

Master-Slave Configuration

slaveof 10.4.44.71 3904
slaveof no one
# Set the read-only / writable of the slave
config set slave-read-only yes
config set slave-read-only no

Memory Recycling Policy

config get maxmemory-policy

When the maxmemory limit is reached, Redis will take precise action as configured by the maxmemory-policy configuration command. The following policies are available:

noeviction: Returns an error when the memory limit is reached. Attempting to execute commands by the client will cause more memory usage (most write commands, except DEL and some exceptions). If the data is very sensitive and data loss is not allowed, it is recommended to use the noeviction policy.
allkeys-lru: Recycles the least recently used (LRU) keys to make room for new data. If you have hot and cold data, it is recommended to use the allkeys-lru strategy.
volatile-lru: Recycles the least recently used (LRU) keys, but only recycles keys with expiration set to make room for new data.
allkeys-random: Recycles random keys to make room for new data. If you need to read and write all keys in a loop, or the access frequency of each key is similar, you can use the allkeys-random strategy, where the probability of reading and writing all elements is similar.
volatile-random: Recycles random keys, but only recycles keys with expiration set to make room for new data.
volatile-ttl: Recycles keys with expiration set and tries to recycle the key with the shortest time to live (TTL) to make room for new data.

Replication

Adjusting the replication timeout

config get repl-timeout
config set repl-timeout 1800

Adjusting the replication buffer size

The last two parameters after slave represent the hard limit and soft limit, respectively. The hard limit cannot be exceeded and any attempts to write beyond it will fail. The soft limit indicates that the limit has been continuously reached for 60 seconds.

config set client-output-buffer-limit "slave 2684354560 536870912 0"

Setting the size of the replication backlog buffer

config get repl-backlog-size
config set repl-backlog-size xx
(unit: bytes)

Persistence

Set the rewrite size to 80GB

config set auto-aof-rewrite-percentage 200
config set auto-aof-rewrite-min-size 80108864000

Replay commands through AOF

tail -n +6 /opt/redis_dump/appendonly.aof | pv -L 10m | nc 10.13.56.15 3707 > ~/run.log

Turn off AOF

config set appendonly no

When AOF full

When the aof file fills up the disk, its write operation fails and there is no other space on the disk to be released. In such cases, you need to clear the aof file and then execute bgrewrite. After clearing the aof file, use the following command to clear the file handle:

>appendonly.aof

If the aof file is deleted directly, the disk space will not be released because the file handle is not released. In this case, use the following command to clear the deleted file.

>/proc/${pid}/fd

Cluster

View the cluster nodes topology distribution

redis-cli -h 10.3.23.27 -p 3920 cluster nodes

Adding a new node to an existing cluster

To add a new node to an existing cluster, select any node from the current cluster (10.3.23.35:3920) and then add the new node (10.3.20.213:3920).

redis-cli -h 10.3.23.35 -p 3920 CLUSTER MEET 10.3.20.213 3920

Setting up slave nodes

To set 10.3.20.213 3920 as a slave node for node-id (fd276dc8e818f53239f988dc2bb3ec85504f9096), follow these steps:

redis-cli -h 10.3.20.213 -p 3920
 CLUSTER REPLICATE fd276dc8e818f53239f988dc2bb3ec85504f9096

Resetting a Node in the Cluster

To reset a node in the cluster, first remove it from the cluster topology, then clear and reset its data, and finally add it back to the cluster. For example, if the node to be reset is 10.3.20.213:3920 and its node-id is fd276dc8e818f53239f988dc2bb3ec85504f9096, follow these steps:

Remove the node from the cluster topology.
Clear and reset the node data.
Add the node back to the cluster.

# Send forget command to all nodes in the cluster for the node
redis-cli -h 10.3.1.1 -p 3920 
 CLUSTER FORGET fd276dc8e818f53239f988dc2bb3ec85504f9096
# Connect to the node, clear the data and reset it
redis-cli -h 10.3.20.213 -p 3920
  FLUSHALL
  CLUSTER RESET
# Connect to any node and add the new node
redis-cli -h 10.3.1.1 -p 3920 CLUSTER MEET 10.3.20.213 3920

View the number of keys in the specified slot in the cluster

redis-cli -h 10.3.23.35 -p 3920 CLUSTER COUNTKEYSINSLOT 11587

Set the slot to a specific node

redis-cli -h 10.3.21.50 -p 3920 CLUSTER SETSLOT 11587 NODE f68aa3484f2340a9586632b2696f5a8fe0b30f40

Set the timeout for the cluster

The default is 5s.

config set cluster-node-timeout 30000

cluster rebalance

redis-cli --cluster rebalance ip:port

  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace

cluster failover

If both the master and the slave are active, switch the slave to become the master.

slave-node-cli> CLUSTER FAILOVER

If the master server goes down, switch the slave server to become the master.

slave-node-cli> CLUSTER FAILOVER [FORCE|TAKEOVER]

The FORCE option does not handshake with the master. Instead, it broadcasts information, communicates with other masters in the cluster, and obtains authorized votes to become the new master.
The TAKEOVER option does not handshake with the master and does not need to get other master votes. It is directly promoted to the master when a large number of slaves are set as the master during data center switching, or when most of the masters are not available due to a brain split.

Note: TAKEOVER violates the last-failover-wins principle of Redis Cluster. Refer to https://redis.io/commands/cluster-failover for more information.

Slot Migration Process

I. Set the destination node to the “importing state” using the following command:

CLUSTER SETSLOT <slot> IMPORTING <source-node-id>.

Directly request the key of this slot and return MOVED redirection.
First, send the ASKING command, then request the key of this slot, and process the request.

II. Set the source node to the “migrating state” using the following command:

CLUSTER SETSLOT <slot> MIGRATING <destination-node-id>.

Requests for existing keys are treated the same as the original processing logic.
If the request does not include a key, the client returns an ask redirection, and the client requests the importing node.
If the request contains multiple keys, some exist and some do not, TRYAGAIN is returned until all keys are migrated.

III. Read the key from the source node in a loop and transfer them to the destination node using this command:

CLUSTER GETKEYSINSLOT
MIGRATE host port key "" destination-db timeout REPLACE KEYS key [key ...]

IV. Set the slots status to normal using the following command:

 CLUSTER SETSLOT <slot> NODE <destination-node-id>
 # in the source and destination

The CLUSTER SETSLOT <slot> NODE <node-id> command is typically used to send a command to the two instances in the migration status after the migration slot is completed.

Effects:

Executing this command on the owner of the slot will result in an error.
Executing this command on the instance in the migrating state of the slot will clear the migrating state.
Executing this command on the instance in the importing state of the slot will clear the importing state, and check the epoch id. If the epoch id is not the largest in the cluster, a new maximum id will be generated (note that this does not communicate with the cluster, so you will need to manually reset it to the maximum).

Note: If the migration process is interrupted and the status is modified, execute the following command on the node to be repaired.

redis-cli  -h <host> -p <port> cluster setslot <slot> stable

Note: just clear migrating / importing state https://redis.io/commands/cluster-setslot

client tools

View the list of clients currently connected to the server

127.0.0.1:6379> client list
addr=127.0.0.1:52555 fd=5 name= age=855 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
addr=127.0.0.1:52787 fd=6 name= age=6 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

Each line represents various information of a connection:

addr: The TCP address of the client, including the IP and port.
fd: The file descriptor handle number corresponding to the client connection socket.
name: The name of the connection, which is empty by default and can be set through CLIENT SETNAME.
age: The number of seconds the client has been alive.
idle: The number of seconds the client has been idle.

View the currently executing command

monitor

Big Keys

Large keys not only reduce the throughput of the proxy, but also easily trigger blocking, and the response time of long-tail requests will increase. It is best to compress them before storing.

View

slowlog get {N}
memory usage KEY
debug object key
## Serializedlength does not represent the actual byte size. It returns the length of the object serialized using RDB encoding, which may be smaller. However, it has some auxiliary effects for troubleshooting bigkey.
redis-cli --stat
redis-cli -h {ip} -p {port} bigkeys
redis-cli -h {ip} -p {port} --latency

Not Recommended

The size of string type should not exceed 10KB.
The number of elements in HASH, SET, ZSET, and LIST should not exceed 1000.
It is not recommended to use hgetall, lrange, mget, or mset to obtain thousands of elements at once.
When the element or number of elements is too large, it is not recommended to query all elements. For example, the zrange key 0 -1 command will result in slow queries.
It is not recommended to have more than 500 keys under pipeline. It is best to keep it within 100.

It is recommended to keep the size of string type within 1k. For json strings larger than 10k, it is recommended to store them using compression.
It is recommended to distribute the expiration time to prevent expiration blocking due to concentration of data.
Redis has a compression algorithm for data with fewer elements and smaller values, which can reduce the data structure overhead of Redis itself.

Type	Maximum number of elements
Hash	512
Set	512
Zset	128
List	512

Data Types and Implementation

/images/redis-data-type.webp

Best Practice

/images/redis-best-practice.webp

/images/redis-dev.png

/images/redis-ops.webp

Q & A

Redis Metrics: Monitoring, Performance, and Best Practices

https://last9.io/blog/redis-metrics-monitoring/

How to Set Up Redis in Node.js Server

https://shivangyadav.hashnode.dev/setting-up-redis-in-nodejs-server

Redis Configurations and Persistent Setting of the Key Parameters

https://www.percona.com/blog/valkey-redis-configurations-and-persistent-setting-of-the-key-parameters/

Demystifying Redis Messaging

https://medium.com/@tarunannapareddy1997/demystifying-redis-messaging-0448e4dba9e0

Client library support for client-side caching

https://redis.io/blog/faster-redis-client-library-support-for-client-side-caching/

Using Redis for real-time RAG goes beyond a Vector Database

https://redis.io/blog/using-redis-for-real-time-rag-goes-beyond-a-vector-database/

Utilizing RedisShake for Redis Data Migration

https://www.bboy.app/2024/01/02/utilizing-redisshake-for-redis-data-migration/

Migrate from Redis to Valkey Without Downtime

https://www.percona.com/blog/migrate-from-redis-to-valkey-without-downtime/

Exploring the 7 common scenarios of using Redis in work

https://levelup.gitconnected.com/exploring-the-7-common-scenarios-of-using-redis-in-work-0109d8423291

How to Fix the “Error establishing a Redis connection” error

https://www.rosehosting.com/blog/how-to-fix-the-error-establishing-a-redis-connection-error/

Comparing Redis and Cassandra

https://blog.devops.dev/comparing-redis-and-cassandra-a-non-relational-database-showdown-fc44d201a7eb

RabbitMQ vs Kafka vs Redis

https://vvekpandey.medium.com/rebbitmq-vs-kafka-vs-redis-pub-sub-how-are-they-different-0f01544f6fd1

Valkey/Redis: Setting Up Replication

https://www.percona.com/blog/valkey-redis-setting-up-replication/

sabledb-io/sabledb: Ultra fast, persistent database supporting Redis API

https://github.com/sabledb-io/sabledb

MongoDB vs Redis

https://dev.to/kigazon/mongodb-vs-redis-a-comprehensive-comparison-5cn0

Redis Data Persistence: Mastering AOF and RDB Configuration

https://dev.to/asifzcpe/a-thorough-guide-to-redis-data-persistence-mastering-aof-and-rdb-configuration-a3f

Building a vector embedding injection pipeline with Redis and VectorFlow

https://redis.io/blog/building-a-vector-embedding-injection-pipeline-with-redis-and-vectorflow/

Deploy a Redis cluster on OpenShift Virtualization

https://developers.redhat.com/articles/2024/09/18/deploy-redis-cluster-openshift-virtualization#create_the_openshift_services

Retrieving All Keys in Redis: Commands & Best Practices

https://last9.io/blog/retrieving-all-keys-in-redis/

Rate Limiting with Spring Boot, Bucket4j, and Redis

https://www.innoq.com/en/blog/2024/03/distributed-rate-limiting-with-spring-boot-and-redis/

Reding List

Setting Up Redis in Node.js Server (https://shivangyadav.hashnode.dev/setting-up-redis-in-nodejs-server)
Valkey/Redis Configurations and Persistent Setting of the Key Parameters (https://www.percona.com/blog/- valkey-redis-configurations-and-persistent-setting-of-the-key-parameters/)
Valkey/Redis: The Hash Datatype (https://www.percona.com/blog/valkey-redis-the-hash-datatype/)
Kafka on Medium: Demystifying Redis Messaging (https://medium.com/@tarunannapareddy1997/demystifying-redis-messaging-0448e4dba9e0)
Bboysoul’s Blog: Utilizing RedisShake for Redis Data Migration (https://www.bboy.app/2024/01/02/utilizing-redisshake-for-redis-data-migration/)
Stack Overflow: How to set a client timeout for Redis Cloud db (https://stackoverflow.com/questions/77852802/- how-to-set-a-client-timeout-for-redis-cloud-db)
Hacker News: Caching with Redis (https://nintyzeros.substack.com/p/power-of-redis-a-deep-dive-into-caching)
Linux Foundation Forks the Open Source Redis as ‘Valkey’ (https://thenewstack.io/linux-foundation-forks-the-open-source-redis-as-valkey/)
Deploy GenAI apps faster with Redis and NVIDIA NIM (https://redis.io/blog/use-redis-with-nvidia-nim-to-deploy-genai-apps-faster/)
Redis Acquires Speedb to Supercharge End-to-End Application Performance at Lower Cost (https://redis.com/press/- redis-acquires-speedb-to-supercharge-end-to-end-application-performance-at-lower-cost/)
Rediscovering DevOps with Secure Cloud Development Environments (https://thenewstack.io/rediscovering-devops-with-secure-cloud-development-environments/)
Exploring the 7 common scenarios of using Redis in work (https://levelup.gitconnected.com/- exploring-the-7-common-scenarios-of-using-redis-in-work-0109d8423291)
KeyDB - The Faster Redis Alternative (https://docs.keydb.dev/)
Managing Concurrent Database Writes in a Distributed System with Redis or PostgreSQL (https://dev.to/nikl/- managing-concurrent-database-writes-in-a-distributed-system-with-redis-or-postgresql-4nc9)
The Practical Developer: Can Redis be used as a Primary database? (https://dev.to/nnomier/can-redis-be-used-as-a-primary-database-5d4b)
IO Multiplexing: The Secret Sauce Behind Redis’s Efficiency (https://medium.com/@sumitsagar_20050/- io-multiplexing-the-secret-sauce-behind-rediss-efficiency-1e48a9397f8c)
RebbitMQ vs Kafka vs Redis Pub/sub: How are they different? (https://vvekpandey.medium.com/- rebbitmq-vs-kafka-vs-redis-pub-sub-how-are-they-different-0f01544f6fd1)
Valkey/Redis: Migrating to Valkey (https://www.percona.com/blog/valkey-redis-migrating-to-valkey/)
Valkey/Redis: Setting Up Replication (https://www.percona.com/blog/valkey-redis-setting-up-replication/)
sabledb-io/sabledb: Ultra fast, persistent database supporting Redis API (https://github.com/sabledb-io/sabledb)
terraform - Stack Overflow: AWS redis with ustom conf file (https://stackoverflow.com/questions/77692468/aws-redis-with-ustom-conf-file)
MongoDB vs Redis: A Comprehensive Comparison (https://dev.to/kigazon/mongodb-vs-redis-a-comprehensive-comparison-5cn0)
Redis Labs: Redis-driven Dataflow for Clickstream Aggregation with ByteWax (https://redis.com/blog/- redis-driven-dataflow-for-clickstream-aggregation-with-bytewax/)
Caching with DynamoDB or Redis (https://xebia.com/blog/caching-with-dynamodb-or-redis/)
What’s New in Two with Redis – May Edition (https://redis.io/blog/whats-new-in-two-with-redis-may-edition/)
Operator Workshop: Redis Persistence Explained (https://www.youtube.com/watch?v=MTVcScW423w)
Quick and Easy Integration of GitHub Action with Your Golang, PostgreSQL, and Redis Projects (https://surajshetty.hashnode.dev/- quick-and-easy-integration-of-github-action-with-your-golang-postgresql-and-redis-projects)
Now Available: Scrapable metrics for Managed PostgreSQL, MySQL, Redis, and Kafka (https://www.digitalocean.com/blog/- digitalocean-scrapable-metrics-for-postgresql-mysql-redis-kafka)
A Thorough Guide to Redis Data Persistence: Mastering AOF and RDB Configuration (https://dev.to/asifzcpe/- a-thorough-guide-to-redis-data-persistence-mastering-aof-and-rdb-configuration-a3f)
Semaphore CI: Build a Caching Layer in Node.js With Redis (https://semaphoreci.com/blog/nodejs-caching-layer-redis)
Understanding Redis Memory Management (https://rahu1.medium.com/understanding-redis-memory-management-73358e361427)
Redis Pipeline: The Way of Sending Redis Commands in One Shot (https://dev.to/nghtslvr/redis-pipeline-the-way-of-sending-redis-commands-in-one-shot-2j0)
What’s New in Two with Redis – February Edition (https://redis.com/blog/whats-new-in-two-with-redis-february-edition/)
Mastering Dynamic Task Scheduling with Redis: How We Actually Solved Our SAAS Problem? (https://dev.to/nikl/- mastering-dynamic-task-scheduling-with-redis-how-we-actually-solved-our-saas-problem-41mp)
Multi-Tenancy in Redis Enterprise (https://redis.io/blog/multi-tenancy-redis-enterprise/)
Valkey/Redis: Configuration Best Practices (https://www.percona.com/blog/valkey-redis-configuration-best-practices/)
Amazon Web Services Blog: Amazon ElastiCache Serverless for Redis and Memcached is now available (https://aws.amazon.com/blogs/aws/- amazon-elasticache-serverless-for-redis-and-memcached-now-generally-available/)
Valkey/Redis Replication and Auto-Failover With Sentinel Service (https://www.percona.com/blog/- valkey-redis-replication-and-auto-failover-with-sentinel-service/)
Redis, Valkey, and Percona’s Ongoing Support of Open Source (https://www.percona.com/blog/redis-valkey-and-perconas-ongoing-support-of-open-source/)
CodeProject: RedisProvider for .NET (https://www.codeproject.com/Articles/5269781/RedisProvider-for-NET)
Valkey/Redis Sharding Using the Native Clustering Feature (https://www.percona.com/blog/valkey-redis-sharding-using-the-native-clustering-feature/)
Building a RAG application with Redis and Spring AI (https://redis.io/blog/building-a-rag-application-with-redis-and-spring-ai/)
Caching RESTful API requests with Heroku’s Redis Add-on (https://dev.to/heroku/caching-restful-api-requests-with-herokus-redis-add-on-3bg8)
Redis Adopts Dual Source-Available Licensing (https://redis.com/blog/redis-adopts-dual-source-available-licensing/)
Redis Labs: Building a vector embedding injection pipeline with Redis and VectorFlow (https://redis.com/blog/- building-a-vector-embedding-injection-pipeline-with-redis-and-vectorflow/)
Valkey/Redis: Sets and Sorted Sets (https://www.percona.com/blog/valkey-redis-sets-and-sorted-sets/)
Laravel Redis Throttle In Details: Tutorial (https://laravel.io/articles/laravel-redis-throttle-in-details-tutorial)
Install Redis Stack on Linux | Docs (https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/linux/)
Lua Scripts Are Not Atomic in Redis (https://joaojunior.org/posts/lua-scripts-are-not-atomic-in-redis/)
Mastering Kredis in Ruby: Your Essential Guide (https://dev.to/daviducolo/mastering-kredis-in-ruby-your-essential-guide-4b6l)
Valkey/Redis: Not-So-Good Practices (https://www.percona.com/blog/valkey-redis-not-so-good-practices/)
gitconnected Sharpen Your Redis Skills with the AlgoRedis Challenge (https://levelup.gitconnected.com/- sharpen-your-redis-skills-with-the-algoredis-challenge-01f64fa5da84?source=rss----5517fd7b58a6---4)
How Uber Uses Integrated Redis Cache to Serve 40M Reads/Second? (https://blog.bytebytego.com/p/how-uber-uses-integrated-redis-cache)
Redict: A New Still-in-Development Redis Fork (http://lxer.com/module/newswire/ext_link.php?rid=339093)
InfoQ: lastminute.com Improves Search Scalability Using Microservices with RabbitMQ and Redis (https://www.infoq.com/news/2024/01/- lastminute-search-rabbitmq-redis/)
Synchronising Micro Services with Redis: A Kubernetes Case Study (https://medium.com/@rahulptl1997/- synchronising-micro-services-with-redis-a-kubernetes-case-study-af36298cbb91)
Rate Limiting with Spring Boot, Bucket4j, and Redis (https://www.innoq.com/en/blog/2024/03/distributed-rate-limiting-with-spring-boot-and-redis/)
Deploy Amazon ElastiCache for Redis clusters using AWS CDK and TypeScript (https://aws.amazon.com/blogs/database/- deploy-amazon-elasticache-for-redis-clusters-using-aws-cdk-and-typescript/)

Reference

https://redis.io/
https://mp.weixin.qq.com/s/Fz1EbsmJP5k2Rh6ir_a1pQ
https://redis.io/glossary/distributed-caching/
https://redis.io/blog/what-is-data-replication/

Buy me a coffee

Leave Your Message