More Than Just a Cache

Redis Data Structures

Andy Snell

@andrewsnell

Longhorn PHP

October 16, 2021

Setting Expectations

Expect wildly inconsistent  pronunciation

Expectation #1

"Data"

Setting Expectations

Expect oversimplification and some handwaving

Expectation #2

This is an hour long talk

About Me

  • Contract PHP Developer and Consultant
  • Currently Living in Dallas
  • Never Intended to Be a Developer
  • Consider that my real PHP career started in 2016
wkdb.ty/redis
wkdb.ty/redis

Fast Foward: Rediscovering Redis

Tasked to Store Phone Numbers to Not Call 

MySQL Not Performant Enough

 

From >12 GB (on disk) to < 400 mb (in memory)

 

 

wkdb.ty/redis

What is Redis?

REmote DIctionary Server
wkdb.ty/redis

Why Redis?

Redis is an open-source, in-memory, persistent data storage engine that associates string keys with values using a variety of data structures and atomic operations.

 

 

toolbox outline icon toolbox outline icon illustration
wkdb.ty/redis

Why Redis?

Great Documentation => Happy Developer

 

wkdb.ty/redis

Why Redis

Redis is a great cache

wkdb.ty/redis

If you just used Redis as a NoSQL key-value store with key expiration as a cache, there is still significant value there.

 

Why Redis

  • PHP Session Handler
  • Server Clusters
  • Pub/Sub Message Brokering
  • Transactions & Command Pipeline
  • Lua Scripting
  • Extensibility with C Modules
  • High Availability with Redis Sentinel
  • Replication & Partitioning of Data
  • Access Control Lists
  • etc...
wkdb.ty/redis

Why Redis

php -r "echo 'Hello, World';"
redis-cli SET foo "Hello World"
redis-cli GET foo

Redis Does Not Let Its Features Get in Your Way

wkdb.ty/redis

Setup & Connect to Redis

1. Spin Up a Redis Server

2. Run Commands on the Command Line

docker exec redis-server redis-cli ping
docker run --rm --name redis-server -d -p 6379:6379 redis 
wkdb.ty/redis

Setup & Connect to Redis

3. Install PHP Extension with PECL

4. Instantiate and Connect the Redis Client

pecl install redis
$redis = new Redis();
$redis->connect('127.0.0.1');
$pong = $redis->ping();
var_dump($pong);
wkdb.ty/redis

Docker Compose Stack & Examples

wkdb.yt/redis
wkdb.ty/redis

RedisInsight

localhost:8001
wkdb.ty/redis

RedisInsight

localhost:8001
wkdb.ty/redis

Data Structures

A data structure defines how to organize a collection of data and what operations can be performed on it.

wkdb.ty/redis
wkdb.ty/redis

Data Structures

A single collection of data could be represented by several data structures, as needed to provide the most efficient operations for accessing and manipulating the data.

wkdb.ty/redis

Measuring Complexity

wkdb.ty/redis

Redis Data Structures

string
hash
list
set
sorted set
stream

Basic Structures

wkdb.ty/redis

Redis Data Structures

bitmap
geospatial
hyperloglog

Derived Structures

wkdb.ty/redis

Redis Data Structures

hash
list
set
sorted set
stream
=>
=>
=>
=>
=>
mapping
sequences
membership
ranking
log-like

Optimizations

wkdb.ty/redis

Keys

  • Keys are binary-safe strings
  • Maximum Key Size: 512 Mb
  • Maximum Number of Keys: 4,294,967,296
  • Keys can either persist or expire
  • Keyspacing is left up to you.
Key Value
user.12343.greeting Hello, Andy
user.12343.lastlogin 1566423852
import.532 {"import_id":53233,"import_time":"2019-08-23...
wkdb.ty/redis
wkdb.ty/redis

String

A singular value of any kind of binary data.

Value
Hello, Andy
greeting:andy
Value
50 4b 03 04 14 00 00 00 08 00 39 18 f9 4e 8f 59 95 1d...
files:archive.zip
Value
2503234
usercount
Value
{"host":"192.168.1.70","last_updated": "2021-10-16...
cache:config
  • Operations that get the length or a substring are highly optimized
  • Integer and float values are can be easily incremented/decremented
  • Each string value can be up to 512Mb in size
wkdb.ty/redis

String

$redis->set('foo', 'hello, world');
echo $redis->get('foo');
wkdb.ty/redis
hello, world

String

$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found

String: Get & Set

$redis->setex('foo', 2, 'hello, world');
echo $redis->get('foo'), PHP_EOL;
sleep(3);
echo ($redis->get('foo') ?: 'Not Found'), PHP_EOL;
wkdb.ty/redis
hello, world
Not Found

String: Debounce

if(!$redis->get('api_debouncer')){
    $redis->set('api_debouncer', true, [
        'nx',
        'ex' => 2,
    ]);

    // do api stuff...
}
wkdb.ty/redis

String: Lock

$quicklock = new class($redis) {
    public function __construct(private Redis $redis){}

    public function lock($name): string
    {
        $random_bytes = \random_bytes(16);
        $this->redis->set('lock.' . $name, $random_bytes, [
            'nx',
            'ex' => 2,
        ]);
        return $random_bytes;
    }

    public function unlock($name, $key = null): void
    {
        $lock = $this->redis->get('lock.' . $name);
        if (!$lock || $lock === $key) {
            $this->redis->del('lock.' . $name);
        }
    }
};
wkdb.ty/redis

Bitmap

wkdb.ty/redis

Act upon a String value as if it were packed binary data

  • Allows extremely compact storage of continuous or time-series data
  • Redis works with arbitrary bitfield sizes and does the math for you.
Value
111110011000001000000000000000000001...
user:23453:flags
Value
111110011000001
user:23454:flags
Value
...02 01 02 03 00 00 00 01 00 02 00 ...
user:23453:site:perminute:20210823

 unoptimized worse case: 1.44KB per user per day

Value
...000111101110000110101000001111101...
user:23453:site:engaged:20210823

 unoptimized worse case: 0.14KB per user per day

Bitmap

class CheckZipCode
{
    public function __construct(private Redis $redis)
    {
    }

    public function add(string $zip_code): void
    {
        $key = substr($zip_code, 0, 5);
        $offset = (int)substr($zip_code, 5, 4);
        $this->redis->setBit($key, $offset, 1);
    }

    public function check(string $zip_code): bool
    {
        $key = substr($zip_code, 0, 5);
        $offset = (int)substr($zip_code, 5, 4);
        return $this->redis->getBit($key, $offset);
    }
}
wkdb.ty/redis

Hash

wkdb.ty/redis

A collection of field and value pairs

Field Value
firstname
lastname
email
lastlogin
logincount
Andy
Snell
andy@example.com
2021-08-23T13:10:56+00:00
1232
user:12343
  • Can be used to map objects when the values need to be used
  • Hashes are like sub-key/value pairs and have string-like methods
  • Field lookups occur at constant time
  • Caution: hashes expire based on the key and are schema-less

Hash

// Set a field on a hash
$redis->hSet('import:33342', 'job_id', 12345); // 1
$redis->hSet('import:33342', 'import_count', 0); // 1

// Set multiple fields on a hash
$redis->hMSet('import:73723', ['job_id' => 234323, 'import_count' => 0]); // true

// Get the value of a hash field
$redis->hGet('import:73723', 'job_id'); // '234323'

// Get all the fields/values of a hash -- caution: order is not guaranteed.
$redis->hGetAll('import:73723'); // [ 'import_count' => '0', 'job_id' => '234323']

// Increment the value of a hash field 
$redis->hIncrBy('import:73723', 'import_count', 1); // 1
wkdb.ty/redis

List

wkdb.ty/redis

A collection of ordered values

  • A deque data structure, allowing push/pop operations from both sides
  • You can get a value by index, but probably should use a Hash instead
  • Lists can pop-push directly to other lists.
  • A connection can block pop or block push on multiple lists at a time 
Index Value
0
1
2
3
4
5
 
12324422
23423432
23432343
23432345
23423432
23423432
user:12343:next_action_id
Index Value
0
1
2
3
 
{"job_id":53233,"email":"andy@ex...
{"job_id":53234,"email":"jeff@p...
{"job_id":53235,"email":"megan@...
{"job_id":53236,"email":"sarah@...
 
emailqueue

List

// Add values to either side of the list
$redis->lPush('queued_email', 8976454); // 1 (length of list)
$redis->rPush('queued_email', 3423432); // 2
$redis->rPush('queued_email', 3423432); // 3 duplicates are ok
$redis->rPush('queued_email', 5675456); // 4
$redis->rPush('queued_email', 3423411); // 5

// Get the length of the list
$redis->llen('queued_email'); // 5

// Remove from either side of the list
$redis->lPop('queued_email'); // '8976454'
$redis->rPop('queued_email'); // '3423411'

// Remove from either side while blocking for up to 30 seconds
$redis->blPop('queued_email', 30); // '3423432'
$redis->brPop('queued_email', 30); // '5675456'

// Remove from right side of one list and push it onto a different list.
$redis->rpoplpush('queued_contacts', 'dequeued_email'); // 3423432
$redis->brpoplpush(queued_contacts, dequeued_email, 30); // false
wkdb.ty/redis

Set

wkdb.ty/redis

An unordered collection of unique values

Value
andy@example.com
me@example.net
andysnell@example.com
contact:435433:emails
  • Sets are used when membership and cardinality are important.
  • Operations return intersection, union, and difference of multiple sets
Value
andy@example.com
kevin@example.com
user:234329:emails
Value
andy@example.com
me@example.net
andysnell@example.com
kevin@example.com
union:emails
Value
andy@example.com
inter:emails
Value
me@example.net
andysnell@example.com
diff:emails

Set

$redis->sAdd('emails', 'andy@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 1
$redis->sAdd('emails', 'kevin@example.com'); // 0 (already exists)

// Add more items to the set using an Array
$redis->sAddArray('emails', ['rachel@example.com', 'jc@example.com']); // true

// Get the cardinality (count) of a set
$redis->sCard('emails'); // 5

// Check if an element is a member of the set
$redis->sIsMember('emails', 'kevin@example.com'); // true
$redis->sIsMember('emails', 'jack@example.com'); // false

// Get all members of the set
$redis->sMembers('emails'); // [ array containing all the emails ]
wkdb.ty/redis

Set

// Get the intersection of multiple sets
$redis->sInter('emails', 'other_emails');
$redis->sInterStore('inter_list', 'emails', 'other_emails');

// Get the union of multiple sets
$redis->sUnion('emails', 'other_emails');
$redis->sUnionStore('union_list', 'emails', 'other_emails');

// Get the difference of multiple sets
$redis->sDiff('emails', 'other_emails');
$redis->sDiffStore('diff_list', 'emails', 'other_emails');
wkdb.ty/redis

Set

class CombinedPosts
{
    public function __construct(private Redis $redis) {}

    public function add(int $user_id, int $post_id): bool
    {
        $personal_key = "user:{$user_id}:posts";
        return (bool)$this->redis->sAdd($personal_key, $post_id);
    }

    public function check(int $user_id, int $team_id, int $post_id): bool
    {
        $personal_key = "user:{$user_id}:posts";
        $team_key = "team:{$team_id}:posts";
        $union_key = "temp:allposts:{$user_id}";
        $pipe = $this->redis->multi();
        $pipe->sInterStore($union_key, $personal_key, $team_key);
        $pipe->sIsMember($union_key, $post_id);
        $pipe->del($union_key);
        $result = $pipe->exec();
        return (bool)$result[1];
    }
}
wkdb.ty/redis

Sorted Set

wkdb.ty/redis

A collection of unique values ordered by a user defined score

Score Value
1
1
2
2
3
+13145551234
+13145553943
+13433431232
+19193332347
+12165553211
user:12343:callcount
Score Value
1
1
1
1
1
Adam
Beth
Charlie
Dan
Erica
user:12343:names
  • The score can be any 64-bit float/integer, including a Unix timestamp
  • When scores are identical, members are sorted lexicographically
  • Provides combined functionality similar to both Sets and Lists
  • Scores can be aggregated between set operations (min, max, sum)
Score Value
1566448434
1566448304
1566448134
1566448027
1566447927
+13145551234
+13145553943
+13433431232
+19193332347
+12165553211
user:12343:recentcalls

Sorted Set

class RedisRateLimit
{
    public function __construct(private Redis $redis) {}

    public function check(string $request_source_id, $requests_per_minute = 120): void
    {
        $key = "ratelimit:" . $request_source_id;

        $pipe = $this->redis->multi(Redis::PIPELINE);
        $pipe->zAdd($key, time(), bin2hex(\random_bytes(16)));
        $pipe->expire($key, 60);
        $pipe->zRemRangeByScore($key, 0, time() - 60);
        $pipe->zCard($key);
        $pipe_results = $pipe->exec(); // [1, true, 0, 1]
        if ($pipe_results[3] > $requests_per_minute) {
            throw new RateLimitExceeded();
        }
    }
}
wkdb.ty/redis

Stream

wkdb.ty/redis

Stream is a fairly recent addition to Redis

An append-only log data structure used for cooperative message stream consumption, with applications for event sourcing.

If the "pub/sub" functionality is a push of message data to the consumer,

using the stream type is a pull.

Conclusion

wkdb.ty/redis

If Redis is so great, should I replace my existing relational database with it?

Can you:

Maybe?

Should you:

Probably Not.

Conclusion

wkdb.ty/redis

Use the best tool you can for the job at hand.

 

Using Redis to its fullest, as a data structure server, gives you a toolbox from which to build solutions that work together with your existing infrastructure.

Please rate our talks...

joind.in/talk/83825
wkdb.ty/redis

Additional Resources

wkdb.yt/redis
wkdb.ty/redis