Highly Available

Matt Gray

  • Freelance consultant and maker of things
  • PHP and GoLang
  • Craft CMS user for about 2 years

WhY HA?

  • Downtime
  • Slow response times
  • Data loss

We need to avoid:

  • Duplication
  • Duplication
  • Duplication

We can do this using:

Elastic Scaling

Very popular in cloud environments.

 

Copies of an application can be created and destroyed as required with traffic routed to each of them intelligently.

 

Minimises Cost

 

Maximises Resiliency

The State Complication

If we duplicate our application naively, we might also be duplicating our state.

Changes made to one state might not be reflected in another.

 

Which state is the correct one?

 

How do we make copies when creating new copies of the application?

Identifying State

Craft CMS has multiple different things which define its current state.

  • Content Structure Configuration
  • Content
  • Administrative Data
  • Sessions
  • Cache
  • Assets
  • CPResources

Database

Content Structure
Content
Admin Data

File System

Sessions
Cache
Assets
CPResources

App Server

Database

Content Structure
Content
Admin Data

File System

Sessions
Cache
Assets
CPResources

App Server

Database

Content Structure
Content
Admin Data

File System

Sessions
Cache
Assets
CPResources

App Server

Extract the database

Run the database on a separate server.

 

Connect to it from our app servers over the network.

 

The database might also use a built-in HA solution.

return [
    'driver' => 'mysql',
    'server' => 'external.db.host',
    'port' => 3306,
    ...
];

config/db.php

File System

Sessions
Cache
Assets
CPResources

App Server

Content Structure
Content
Admin Data

Database Server

File System

Sessions
Cache
Assets
CPResources

App Server

Load Balancer

External CachING

Sessions and general cache data can both be stored in an external, in-memory Key:Value store.

return [
    'components' => [
        'redis' => [
            'class' => yii\redis\Connection::class,
            'hostname' => 'external.redis.host',
            'port' => 6379,
            'database' => 0,
        ],
        'session' => [
            'class' => yii\redis\Session::class,
            'as session' => [
                'class' => craft\behaviors\SessionBehavior::class,
            ],
        ],
        'cache' => [
            'class' => yii\redis\Cache::class,
            'defaultDuration' => 86400,
        ],
    ]
];

config/app.php

File System

Assets
CPResources

App Server

Content Structure
Content
Admin Data

Database Server

File System
Assets
CPResources

App Server

Load Balancer

Cache
Sessions

Redis Server

External Assets

We can use plugins to store assets remotely.

Bonus

If you need to create asset sources automatically (such as if you're deploying a project to a zero-config hosting environment)

you can do so using Craft migrations.

public function safeUp()
{
    $vol = new craft\googlecloud\Volume([
        "name" => "Assets",
        "handle" => "project-name",
        "hasUrls" => true,
        "url" => "https://s3.aws.com/project-name",
        "settings" => [
            "subfolder" => "public",
            "projectId" => "1234",
            "keyFileContents" => "some-long-string",
            "expires" => "5 hours",
            "bucket" => "my-asset-bucket"
        ]
    ]);	        
    Craft::$app->volumes->saveVolume($volume);
}

File System
CPResources

App Server

Content Structure
Content
Admin Data

Database Server

File System
CPResources

App Server

Load Balancer

Cache
Sessions

Redis Server

Assets

Cloud Storage

User

CPResources

Bundles of assets (JS, CSS, Images etc) which are created on-the-fly as they are needed by Craft or plugin templates.

  • Usually created using template directives

  • Stored in 'web/cpresources' by default

  • Link inserted into HTML once created

What happens in a HA environment?

File System
CPResources

App Server

File System
CPResources

App Server

Load Balancer

Assets

Cloud Storage

User

1.

  1. Page Request
  2. Received, assets created
  3. HTML returned
  4. Request for asset
  5. 404!!!!

2.

3.

4.

5.

CPResources

An update was applied to Craft to fix this issue. If a request for an asset is received that doesn't exist it'll be re-published and then piped back through PHP.

 

​Piping files through PHP is generally considered bad practice - can have memory issues and locks processes as the client downloads.

File System
CPResources

App Server

File System
CPResources

App Server

Load Balancer

Assets

Cloud Storage

User

1.

  1. Page Request
  2. Received, assets created
  3. HTML returned
  4. Request for asset
  5. Assets created
  6. Files streamed back to user via PHP from file system

2.

3.

4.

5.

6.

CPResources

So we're all good?

 

Maybe.

On rare occasions when a user visits the Craft control panel it'll generate multiple asset bundles (10s) and each of these can contain multiple files. It's not unusual for many of these to be created simultaneously and for their combined file size to be in the MB range.

 

A user could cause many PHP child processes to stream files from the filesystem, locking them until the download is complete and consuming memory.

CPResources

Can we optimise further?

 

In a well optimised HA environment we'd prefer to run each of our application components in an individually scalable manner.

 

This gives us memory and CPU allocation benefits and allows us to ease bottlenecks more efficiently.

 

We should separate our web server and PHP servers.

File System
CPResources

PHP Server

Load Balancer

Assets

Cloud Storage

User

1.

  1. Page Request, forwarded to PHP
  2. Received, assets created
  3. HTML returned
  4. Request for asset, forwarded to PHP because it doesn't exist within webserver's filesystem
  5. Files streamed back to user via PHP from file system
  6. This will happen for every cpresources request!

File System
Static Files

Nginx

2.

3.

4.

5.

CPResources

Solutions?

 

  • Sync cpresources with web server - eventual consistency. Would need to watch for file system changes or hook into Craft's bundle publishing code. How to sync up a newly created webserver? 
  • Shared file system - slow, very fragile in an elastically scaling environment. Potential problems when multiple PHP servers try to publish bundles simultaneously.
  • Just offload it to an external store - ...

File System
 

PHP Server

Load Balancer

Assets

CPResources

Cloud Storage

User

1.

  1. Page Request, forwarded to PHP
  2. Received, assets created on remote storage
  3. HTML returned containing remote storage URLs
  4. Assets requested from remote storage
  5. Asset returned, everyone is happy

File System
Static Files

Nginx

2.

3.

4.

5.

CPResources

  • Asset bundles are only created once
  • No piping files through PHP
  • No confused nginx servers

 

BUT

 

  • PHP servers need to be aware of which bundles have already been published

CPResources

How do we make this happen?

 

A plugin. All the asset bundle publishing logic is held within a single class called AssetManager. If we can replace the logic in there we can do whatever we want.

Craft::$app->set('assetManager', function () {
    $generalConfig = Craft::$app->getConfig()->getGeneral();
    $config        = [
        'class'           => CustomSubclassOfAssetManager::class,
        'basePath'        => $generalConfig->resourceBasePath,
        'baseUrl'         => $generalConfig->resourceBaseUrl,
        'fileMode'        => $generalConfig->defaultFileMode,
        'dirMode'         => $generalConfig->defaultDirMode,
        'appendTimestamp' => false,
    ];
    return Craft::createObject($config);
});

Content Structure
Content
Admin Data

Database Server

PHP Server

Load Balancer

Cache
Sessions

Redis Server

Assets

CPResources

Cloud Storage

User

PHP Server

Nginx

Nginx

1..n

1..n

THanks!

A quick favour...

@mattgrayisok

Made with Slides.com