More Than Just a Cache
Redis Data Structures
Andy Snell
@andrewsnell
Longhorn PHP
October 16, 2021
Setting Expectations
Expect wildly inconsistent pronunciation
Expectation #1
"Data"
Setting Expectations
Expect oversimplification and some handwaving
Expectation #2
This is an hour long talk
About Me
- Contract PHP Developer and Consultant
- Currently Living in Dallas
- Never Intended to Be a Developer
- Consider that my real PHP career started in 2016
wkdb.ty/redis
wkdb.ty/redis
Fast Foward: Rediscovering Redis
Tasked to Store Phone Numbers to Not Call
MySQL Not Performant Enough
From >12 GB (on disk) to < 400 mb (in memory)
wkdb.ty/redis
What is Redis?
REmote DIctionary Server
wkdb.ty/redis
Why Redis?
Redis is an open-source, in-memory, persistent data storage engine that associates string keys with values using a variety of data structures and atomic operations.
wkdb.ty/redis
Why Redis?
Great Documentation => Happy Developer
wkdb.ty/redis
Why Redis
Redis is a great cache
wkdb.ty/redis
If you just used Redis as a NoSQL key-value store with key expiration as a cache, there is still significant value there.
Why Redis
- PHP Session Handler
- Server Clusters
- Pub/Sub Message Brokering
- Transactions & Command Pipeline
- Lua Scripting
- Extensibility with C Modules
- High Availability with Redis Sentinel
- Replication & Partitioning of Data
- Access Control Lists
- etc...
wkdb.ty/redis
Why Redis
php -r "echo 'Hello, World';"
redis-cli SET foo "Hello World"
redis-cli GET foo
Redis Does Not Let Its Features Get in Your Way
wkdb.ty/redis
Setup & Connect to Redis
1. Spin Up a Redis Server
2. Run Commands on the Command Line
docker exec redis-server redis-cli ping
docker run --rm --name redis-server -d -p 6379:6379 redis
wkdb.ty/redis
Setup & Connect to Redis
3. Install PHP Extension with PECL
4. Instantiate and Connect the Redis Client
pecl install redis
$redis = new Redis();
$redis->connect('127.0.0.1');
$pong = $redis->ping();
var_dump($pong);
wkdb.ty/redis
Docker Compose Stack & Examples
wkdb.yt/redis
wkdb.ty/redis
RedisInsight
localhost:8001
wkdb.ty/redis
RedisInsight
localhost:8001
wkdb.ty/redis
Data Structures
A data structure defines how to organize a collection of data and what operations can be performed on it.
wkdb.ty/redis
wkdb.ty/redis
Data Structures
A single collection of data could be represented by several data structures, as needed to provide the most efficient operations for accessing and manipulating the data.
wkdb.ty/redis
Measuring Complexity
wkdb.ty/redis
Redis Data Structures
string
hash
list
set
sorted set
stream
Basic Structures
wkdb.ty/redis
Redis Data Structures
bitmap
geospatial
hyperloglog
Derived Structures
wkdb.ty/redis
Redis Data Structures
hash
list
set
sorted set
stream
=>
=>
=>
=>
=>
mapping
sequences
membership
ranking
log-like
Optimizations
wkdb.ty/redis
Keys
- Keys are binary-safe strings
- Maximum Key Size: 512 Mb
- Maximum Number of Keys: 4,294,967,296
- Keys can either persist or expire
- Keyspacing is left up to you.
Key | Value |
---|---|
user.12343.greeting | Hello, Andy |
user.12343.lastlogin | 1566423852 |
import.532 | {"import_id":53233,"import_time":"2019-08-23... |
wkdb.ty/redis
wkdb.ty/redis
String
A singular value of any kind of binary data.
Value |
---|
Hello, Andy |
greeting:andy
Value |
---|
50 4b 03 04 14 00 00 00 08 00 39 18 f9 4e 8f 59 95 1d... |
files:archive.zip
Value |
---|
2503234 |
usercount
Value |
---|
{"host":"192.168.1.70","last_updated": "2021-10-16... |
cache:config
- Operations that get the length or a substring are highly optimized
- Integer and float values are can be easily incremented/decremented
- Each string value can be up to 512Mb in size
wkdb.ty/redis
String
$redis->set('foo', 'hello, world');
echo $redis->get('foo');
wkdb.ty/redis
hello, world
String
$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found
String: Get & Set
$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found
String: Debounce
if(!$redis->get('api_debouncer')){
$redis->set('api_debouncer', true, [
'nx',
'ex' => 2,
]);
// do api stuff...
}
wkdb.ty/redis
String: Lock
$quicklock = new class($redis) {
public function __construct(private Redis $redis){}
public function lock($name): string
{
$random_bytes = \random_bytes(16);
$this->redis->set('lock.' . $name, $random_bytes, [
'nx',
'ex' => 2,
]);
return $random_bytes;
}
public function unlock($name, $key = null): void
{
$lock = $this->redis->get('lock.' . $name);
if (!$lock || $lock === $key) {
$this->redis->del('lock.' . $name);
}
}
};
wkdb.ty/redis
Bitmap
wkdb.ty/redis
Act upon a String value as if it were packed binary data
- Allows extremely compact storage of continuous or time-series data
- Redis works with arbitrary bitfield sizes and does the math for you.
Value |
---|
111110011000001000000000000000000001... |
user:23453:flags
Value |
---|
111110011000001 |
user:23454:flags
Value |
---|
...02 01 02 03 00 00 00 01 00 02 00 ... |
user:23453:site:perminute:20210823
unoptimized worse case: 1.44KB per user per day
Value |
---|
...000111101110000110101000001111101... |
user:23453:site:engaged:20210823
unoptimized worse case: 0.14KB per user per day
Bitmap
class CheckZipCode
{
public function __construct(private Redis $redis)
{
}
public function add(string $zip_code): void
{
$key = substr($zip_code, 0, 5);
$offset = (int)substr($zip_code, 5, 4);
$this->redis->setBit($key, $offset, 1);
}
public function check(string $zip_code): bool
{
$key = substr($zip_code, 0, 5);
$offset = (int)substr($zip_code, 5, 4);
return $this->redis->getBit($key, $offset);
}
}
wkdb.ty/redis
Hash
wkdb.ty/redis
A collection of field and value pairs
Field | Value |
---|---|
firstname lastname lastlogin logincount |
Andy Snell andy@example.com 2021-08-23T13:10:56+00:00 1232 |
user:12343
- Can be used to map objects when the values need to be used
- Hashes are like sub-key/value pairs and have string-like methods
- Field lookups occur at constant time
- Caution: hashes expire based on the key and are schema-less
Hash
// Set a field on a hash
$redis->hSet('import:33342', 'job_id', 12345); // 1
$redis->hSet('import:33342', 'import_count', 0); // 1
// Set multiple fields on a hash
$redis->hMSet('import:73723', ['job_id' => 234323, 'import_count' => 0]); // true
// Get the value of a hash field
$redis->hGet('import:73723', 'job_id'); // '234323'
// Get all the fields/values of a hash -- caution: order is not guaranteed.
$redis->hGetAll('import:73723'); // [ 'import_count' => '0', 'job_id' => '234323']
// Increment the value of a hash field
$redis->hIncrBy('import:73723', 'import_count', 1); // 1
wkdb.ty/redis
List
wkdb.ty/redis
A collection of ordered values
- A deque data structure, allowing push/pop operations from both sides
- You can get a value by index, but probably should use a Hash instead
- Lists can pop-push directly to other lists.
- A connection can block pop or block push on multiple lists at a time
Index | Value |
---|---|
0 1 2 3 4 5 |
12324422 23423432 23432343 23432345 23423432 23423432 |
user:12343:next_action_id
Index | Value |
---|---|
0 1 2 3 |
{"job_id":53233,"email":"andy@ex... {"job_id":53234,"email":"jeff@p... {"job_id":53235,"email":"megan@... {"job_id":53236,"email":"sarah@... |
emailqueue
List
// Add values to either side of the list
$redis->lPush('queued_email', 8976454); // 1 (length of list)
$redis->rPush('queued_email', 3423432); // 2
$redis->rPush('queued_email', 3423432); // 3 duplicates are ok
$redis->rPush('queued_email', 5675456); // 4
$redis->rPush('queued_email', 3423411); // 5
// Get the length of the list
$redis->llen('queued_email'); // 5
// Remove from either side of the list
$redis->lPop('queued_email'); // '8976454'
$redis->rPop('queued_email'); // '3423411'
// Remove from either side while blocking for up to 30 seconds
$redis->blPop('queued_email', 30); // '3423432'
$redis->brPop('queued_email', 30); // '5675456'
// Remove from right side of one list and push it onto a different list.
$redis->rpoplpush('queued_contacts', 'dequeued_email'); // 3423432
$redis->brpoplpush(queued_contacts, dequeued_email, 30); // false
wkdb.ty/redis
Set
wkdb.ty/redis
An unordered collection of unique values
Value |
---|
andy@example.com me@example.net andysnell@example.com |
contact:435433:emails
- Sets are used when membership and cardinality are important.
- Operations return intersection, union, and difference of multiple sets
Value |
---|
andy@example.com kevin@example.com |
user:234329:emails
Value |
---|
andy@example.com me@example.net andysnell@example.com kevin@example.com |
union:emails
Value |
---|
andy@example.com |
inter:emails
Value |
---|
me@example.net andysnell@example.com |
diff:emails
Set
$redis->sAdd('emails', 'andy@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 0 (already exists)
// Add more items to the set using an Array
$redis->sAddArray('emails', ['rachel@example.com', 'jc@example.com']); // true
// Get the cardinality (count) of a set
$redis->sCard('emails'); // 5
// Check if an element is a member of the set
$redis->sIsMember('emails', 'kevin@example.com'); // true
$redis->sIsMember('emails', 'jack@example.com'); // false
// Get all members of the set
$redis->sMembers('emails'); // [ array containing all the emails ]
wkdb.ty/redis
Set
// Get the intersection of multiple sets
$redis->sInter('emails', 'other_emails');
$redis->sInterStore('inter_list', 'emails', 'other_emails');
// Get the union of multiple sets
$redis->sUnion('emails', 'other_emails');
$redis->sUnionStore('union_list', 'emails', 'other_emails');
// Get the difference of multiple sets
$redis->sDiff('emails', 'other_emails');
$redis->sDiffStore('diff_list', 'emails', 'other_emails');
wkdb.ty/redis
Set
class CombinedPosts
{
public function __construct(private Redis $redis) {}
public function add(int $user_id, int $post_id): bool
{
$personal_key = "user:{$user_id}:posts";
return (bool)$this->redis->sAdd($personal_key, $post_id);
}
public function check(int $user_id, int $team_id, int $post_id): bool
{
$personal_key = "user:{$user_id}:posts";
$team_key = "team:{$team_id}:posts";
$union_key = "temp:allposts:{$user_id}";
$pipe = $this->redis->multi();
$pipe->sInterStore($union_key, $personal_key, $team_key);
$pipe->sIsMember($union_key, $post_id);
$pipe->del($union_key);
$result = $pipe->exec();
return (bool)$result[1];
}
}
wkdb.ty/redis
Sorted Set
wkdb.ty/redis
A collection of unique values ordered by a user defined score
Score | Value |
---|---|
1 1 2 2 3 |
+13145551234 +13145553943 +13433431232 +19193332347 +12165553211 |
user:12343:callcount
Score | Value |
---|---|
1 1 1 1 1 |
Adam Beth Charlie Dan Erica |
user:12343:names
- The score can be any 64-bit float/integer, including a Unix timestamp
- When scores are identical, members are sorted lexicographically
- Provides combined functionality similar to both Sets and Lists
- Scores can be aggregated between set operations (min, max, sum)
Score | Value |
---|---|
1566448434 1566448304 1566448134 1566448027 1566447927 |
+13145551234 +13145553943 +13433431232 +19193332347 +12165553211 |
user:12343:recentcalls
Sorted Set
class RedisRateLimit
{
public function __construct(private Redis $redis) {}
public function check(string $request_source_id, $requests_per_minute = 120): void
{
$key = "ratelimit:" . $request_source_id;
$pipe = $this->redis->multi(Redis::PIPELINE);
$pipe->zAdd($key, time(), bin2hex(\random_bytes(16)));
$pipe->expire($key, 60);
$pipe->zRemRangeByScore($key, 0, time() - 60);
$pipe->zCard($key);
$pipe_results = $pipe->exec(); // [1, true, 0, 1]
if ($pipe_results[3] > $requests_per_minute) {
throw new RateLimitExceeded();
}
}
}
wkdb.ty/redis
Stream
wkdb.ty/redis
Stream is a fairly recent addition to Redis
An append-only log data structure used for cooperative message stream consumption, with applications for event sourcing.
If the "pub/sub" functionality is a push of message data to the consumer,
using the stream type is a pull.
Conclusion
wkdb.ty/redis
If Redis is so great, should I replace my existing relational database with it?
Can you:
Maybe?
Should you:
Probably Not.
Conclusion
wkdb.ty/redis
Use the best tool you can for the job at hand.
Using Redis to its fullest, as a data structure server, gives you a toolbox from which to build solutions that work together with your existing infrastructure.
Please rate our talks...
joind.in/talk/83825
wkdb.ty/redis
Additional Resources
wkdb.yt/redis
wkdb.ty/redis
More Than Just a Cache
By Andy Snell
More Than Just a Cache
Redis is a popular key-value store, commonly used as a cache or message broker service. However, it can do so much more than just hold string values in memory! -- Redis is a full featured “data structure server”. As PHP developers, we typically don’t think about data structures other than our jack-of-all-trades array, but Redis can store hashes, lists, sets, and sorted sets, in addition to operating on string values. In this talk, we’ll explore these basic data structures in Redis and look at how we can apply them to solve problems like rate limiting, creating distributed locks, or efficiently checking membership in a massive set of data.
- 367