Redis Data Structures
Andy Snell
@andrewsnell
Longhorn PHP
October 16, 2021
Expect wildly inconsistent pronunciation
Expect oversimplification and some handwaving
wkdb.ty/redis
wkdb.ty/redis
Tasked to Store Phone Numbers to Not Call
MySQL Not Performant Enough
From >12 GB (on disk) to < 400 mb (in memory)
wkdb.ty/redis
REmote DIctionary Server
wkdb.ty/redis
Redis is an open-source, in-memory, persistent data storage engine that associates string keys with values using a variety of data structures and atomic operations.
wkdb.ty/redis
Great Documentation => Happy Developer
wkdb.ty/redis
Redis is a great cache
wkdb.ty/redis
If you just used Redis as a NoSQL key-value store with key expiration as a cache, there is still significant value there.
wkdb.ty/redis
php -r "echo 'Hello, World';"
redis-cli SET foo "Hello World"
redis-cli GET foo
Redis Does Not Let Its Features Get in Your Way
wkdb.ty/redis
1. Spin Up a Redis Server
2. Run Commands on the Command Line
docker exec redis-server redis-cli ping
docker run --rm --name redis-server -d -p 6379:6379 redis
wkdb.ty/redis
3. Install PHP Extension with PECL
4. Instantiate and Connect the Redis Client
pecl install redis
$redis = new Redis();
$redis->connect('127.0.0.1');
$pong = $redis->ping();
var_dump($pong);
wkdb.ty/redis
wkdb.yt/redis
wkdb.ty/redis
localhost:8001
wkdb.ty/redis
localhost:8001
wkdb.ty/redis
A data structure defines how to organize a collection of data and what operations can be performed on it.
wkdb.ty/redis
wkdb.ty/redis
A single collection of data could be represented by several data structures, as needed to provide the most efficient operations for accessing and manipulating the data.
wkdb.ty/redis
wkdb.ty/redis
string
hash
list
set
sorted set
stream
wkdb.ty/redis
bitmap
geospatial
hyperloglog
wkdb.ty/redis
hash
list
set
sorted set
stream
=>
=>
=>
=>
=>
mapping
sequences
membership
ranking
log-like
wkdb.ty/redis
Key | Value |
---|---|
user.12343.greeting | Hello, Andy |
user.12343.lastlogin | 1566423852 |
import.532 | {"import_id":53233,"import_time":"2019-08-23... |
wkdb.ty/redis
wkdb.ty/redis
A singular value of any kind of binary data.
Value |
---|
Hello, Andy |
greeting:andy
Value |
---|
50 4b 03 04 14 00 00 00 08 00 39 18 f9 4e 8f 59 95 1d... |
files:archive.zip
Value |
---|
2503234 |
usercount
Value |
---|
{"host":"192.168.1.70","last_updated": "2021-10-16... |
cache:config
wkdb.ty/redis
$redis->set('foo', 'hello, world');
echo $redis->get('foo');
wkdb.ty/redis
hello, world
$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found
$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found
if(!$redis->get('api_debouncer')){
$redis->set('api_debouncer', true, [
'nx',
'ex' => 2,
]);
// do api stuff...
}
wkdb.ty/redis
$quicklock = new class($redis) {
public function __construct(private Redis $redis){}
public function lock($name): string
{
$random_bytes = \random_bytes(16);
$this->redis->set('lock.' . $name, $random_bytes, [
'nx',
'ex' => 2,
]);
return $random_bytes;
}
public function unlock($name, $key = null): void
{
$lock = $this->redis->get('lock.' . $name);
if (!$lock || $lock === $key) {
$this->redis->del('lock.' . $name);
}
}
};
wkdb.ty/redis
wkdb.ty/redis
Act upon a String value as if it were packed binary data
Value |
---|
111110011000001000000000000000000001... |
user:23453:flags
Value |
---|
111110011000001 |
user:23454:flags
Value |
---|
...02 01 02 03 00 00 00 01 00 02 00 ... |
user:23453:site:perminute:20210823
unoptimized worse case: 1.44KB per user per day
Value |
---|
...000111101110000110101000001111101... |
user:23453:site:engaged:20210823
unoptimized worse case: 0.14KB per user per day
class CheckZipCode
{
public function __construct(private Redis $redis)
{
}
public function add(string $zip_code): void
{
$key = substr($zip_code, 0, 5);
$offset = (int)substr($zip_code, 5, 4);
$this->redis->setBit($key, $offset, 1);
}
public function check(string $zip_code): bool
{
$key = substr($zip_code, 0, 5);
$offset = (int)substr($zip_code, 5, 4);
return $this->redis->getBit($key, $offset);
}
}
wkdb.ty/redis
wkdb.ty/redis
A collection of field and value pairs
Field | Value |
---|---|
firstname lastname lastlogin logincount |
Andy Snell andy@example.com 2021-08-23T13:10:56+00:00 1232 |
user:12343
// Set a field on a hash
$redis->hSet('import:33342', 'job_id', 12345); // 1
$redis->hSet('import:33342', 'import_count', 0); // 1
// Set multiple fields on a hash
$redis->hMSet('import:73723', ['job_id' => 234323, 'import_count' => 0]); // true
// Get the value of a hash field
$redis->hGet('import:73723', 'job_id'); // '234323'
// Get all the fields/values of a hash -- caution: order is not guaranteed.
$redis->hGetAll('import:73723'); // [ 'import_count' => '0', 'job_id' => '234323']
// Increment the value of a hash field
$redis->hIncrBy('import:73723', 'import_count', 1); // 1
wkdb.ty/redis
wkdb.ty/redis
A collection of ordered values
Index | Value |
---|---|
0 1 2 3 4 5 |
12324422 23423432 23432343 23432345 23423432 23423432 |
user:12343:next_action_id
Index | Value |
---|---|
0 1 2 3 |
{"job_id":53233,"email":"andy@ex... {"job_id":53234,"email":"jeff@p... {"job_id":53235,"email":"megan@... {"job_id":53236,"email":"sarah@... |
emailqueue
// Add values to either side of the list
$redis->lPush('queued_email', 8976454); // 1 (length of list)
$redis->rPush('queued_email', 3423432); // 2
$redis->rPush('queued_email', 3423432); // 3 duplicates are ok
$redis->rPush('queued_email', 5675456); // 4
$redis->rPush('queued_email', 3423411); // 5
// Get the length of the list
$redis->llen('queued_email'); // 5
// Remove from either side of the list
$redis->lPop('queued_email'); // '8976454'
$redis->rPop('queued_email'); // '3423411'
// Remove from either side while blocking for up to 30 seconds
$redis->blPop('queued_email', 30); // '3423432'
$redis->brPop('queued_email', 30); // '5675456'
// Remove from right side of one list and push it onto a different list.
$redis->rpoplpush('queued_contacts', 'dequeued_email'); // 3423432
$redis->brpoplpush(queued_contacts, dequeued_email, 30); // false
wkdb.ty/redis
wkdb.ty/redis
An unordered collection of unique values
Value |
---|
andy@example.com me@example.net andysnell@example.com |
contact:435433:emails
Value |
---|
andy@example.com kevin@example.com |
user:234329:emails
Value |
---|
andy@example.com me@example.net andysnell@example.com kevin@example.com |
union:emails
Value |
---|
andy@example.com |
inter:emails
Value |
---|
me@example.net andysnell@example.com |
diff:emails
$redis->sAdd('emails', 'andy@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 0 (already exists)
// Add more items to the set using an Array
$redis->sAddArray('emails', ['rachel@example.com', 'jc@example.com']); // true
// Get the cardinality (count) of a set
$redis->sCard('emails'); // 5
// Check if an element is a member of the set
$redis->sIsMember('emails', 'kevin@example.com'); // true
$redis->sIsMember('emails', 'jack@example.com'); // false
// Get all members of the set
$redis->sMembers('emails'); // [ array containing all the emails ]
wkdb.ty/redis
// Get the intersection of multiple sets
$redis->sInter('emails', 'other_emails');
$redis->sInterStore('inter_list', 'emails', 'other_emails');
// Get the union of multiple sets
$redis->sUnion('emails', 'other_emails');
$redis->sUnionStore('union_list', 'emails', 'other_emails');
// Get the difference of multiple sets
$redis->sDiff('emails', 'other_emails');
$redis->sDiffStore('diff_list', 'emails', 'other_emails');
wkdb.ty/redis
class CombinedPosts
{
public function __construct(private Redis $redis) {}
public function add(int $user_id, int $post_id): bool
{
$personal_key = "user:{$user_id}:posts";
return (bool)$this->redis->sAdd($personal_key, $post_id);
}
public function check(int $user_id, int $team_id, int $post_id): bool
{
$personal_key = "user:{$user_id}:posts";
$team_key = "team:{$team_id}:posts";
$union_key = "temp:allposts:{$user_id}";
$pipe = $this->redis->multi();
$pipe->sInterStore($union_key, $personal_key, $team_key);
$pipe->sIsMember($union_key, $post_id);
$pipe->del($union_key);
$result = $pipe->exec();
return (bool)$result[1];
}
}
wkdb.ty/redis
wkdb.ty/redis
A collection of unique values ordered by a user defined score
Score | Value |
---|---|
1 1 2 2 3 |
+13145551234 +13145553943 +13433431232 +19193332347 +12165553211 |
user:12343:callcount
Score | Value |
---|---|
1 1 1 1 1 |
Adam Beth Charlie Dan Erica |
user:12343:names
Score | Value |
---|---|
1566448434 1566448304 1566448134 1566448027 1566447927 |
+13145551234 +13145553943 +13433431232 +19193332347 +12165553211 |
user:12343:recentcalls
class RedisRateLimit
{
public function __construct(private Redis $redis) {}
public function check(string $request_source_id, $requests_per_minute = 120): void
{
$key = "ratelimit:" . $request_source_id;
$pipe = $this->redis->multi(Redis::PIPELINE);
$pipe->zAdd($key, time(), bin2hex(\random_bytes(16)));
$pipe->expire($key, 60);
$pipe->zRemRangeByScore($key, 0, time() - 60);
$pipe->zCard($key);
$pipe_results = $pipe->exec(); // [1, true, 0, 1]
if ($pipe_results[3] > $requests_per_minute) {
throw new RateLimitExceeded();
}
}
}
wkdb.ty/redis
wkdb.ty/redis
Stream is a fairly recent addition to Redis
An append-only log data structure used for cooperative message stream consumption, with applications for event sourcing.
If the "pub/sub" functionality is a push of message data to the consumer,
using the stream type is a pull.
wkdb.ty/redis
Can you:
Maybe?
Should you:
Probably Not.
wkdb.ty/redis
joind.in/talk/83825
wkdb.ty/redis
wkdb.yt/redis
wkdb.ty/redis