Resilient PHP applications with Phystrix

Resilience...

...the capacity to recover quickly from difficulties; toughness.

Hystrix

Long time ago Netflix developed Hystrix, a Java library to make Netflix API more resilient.

The problem...

  • Systems fails (it is inevitably).
  • Dependencies fails produce cascade failures.

On a system with 30 dependencies, each with a 99.99% uptime, in 1 billion request means 3,000,000 are failures.

The problem...

The system receives requests that depends on third party services to be resolved (database, API, queue, ...)

The problem...

If one of the dependencies is down the requests is active until a timeout is triggered

The problem...

More request arrives and are waiting the response of a system that is failing.

  • Our system could collapse by the volume of pending request.
  • Setting timeouts is not a solution (which value to use? 1sec, 5secs, ...)
  • Why to continue asking a system we know is down

What problem does Hystrix solve?

  • Fail fast and rapidly recover.
  • Stop cascading failures in a complex distributed system.
  • Protect from and control over latency and failure from dependencies (typically over the network and via third-party client libraries).
  • Fallback and gracefully degrade when possible.
  • Enable near real-time monitoring, alerting, and operational control.

Phystrix

Phystrix is a PHP Hystrix port made by oDesk (now rebranded to Upwork).

Phystrix

Circuit breaker pattern

Phystrix

Circuit breaker pattern

Phystrix

  • Actions to third party dependencies must be implemented as commands:
    • Subclass from AbstractCommand
    • Implement your logic within the run() method.
    • Use getFallback() as alternative logic when run() fails.
  • Metrics are stored through APC.

Phystrix

use Odesk\Phystrix\AbstractCommand;

class PrintNameCommand extends AbstractCommand
{
    // Metrics will be grouped by this name
    protected $commandKey = "print_name_command";

    private $name;

    public function __construct($name)
    {
        $this->name = $name;
    }

    /**
     * This function is called internally by Phystrix, only if the request is allowed
     */
    protected function run()
    {
        return 'Hello ' . $this->name;
    }

    /**
     * Optionally we implemented a fallback method.
     */
    protected function getFallback()
    {
        return 'Hello no-name :(';
    }
}

Custom command implementation:

Phystrix

$phystrixFactory = ...; // Must be a CommandFactory reference

// Create a command instance passing parameters to the constructor
$command = $phystrixFactory->getCommand('PrintNameCommand', 'Batman');

$result = $command->execute();    // Hello Batman

Command invokation:

phystrix-bundle: Easily integrate Phystrix within Symfony2 applications.

Phystrix

  • Configure thresholds to define when a circuit must be opened and closed.
  • Configure thresholds per command.
odesk_phystrix:
    default:
        # Whether fallback logic of the phystrix command is enabled
        fallback: false
        circuitBreaker:
            # How many failed request it might be before we open the circuit (disallow consecutive requests)
            errorThresholdPercentage: 5
            # If true, the circuit breaker will always be open regardless the metrics
            forceOpen: false
            # If true, the circuit breaker will always be closed, allowing all requests, regardless the metrics
            forceClosed: false
            # How many requests we need minimally before we can start making decisions about service stability
            requestVolumeThreshold: 1
            # For how long to wait before attempting to access a failing service
            sleepWindowInMilliseconds: 5000
        metrics:
            # This is for caching metrics so they are not recalculated more often than needed
            healthSnapshotIntervalInMilliseconds: 1000
            # The period of time within which we the stats are collected
            rollingStatisticalWindowInMilliseconds: 50000
            # The more buckets the more precise and actual the stats and slower the calculation.
            rollingStatisticalWindowBuckets: 10
        # Request cache, if enabled and a command has getCacheKey implemented caches results within current http request
        requestCache: false
        # Request log collects all commands executed within current http request
        requestLog: true

Let's make a break

  • Command? It is nice but most of my code that access third party services is implemented within a repository.
  • Possible solution: Create a proxy phystrixized repository that runs the source repository operations as commands.

Your source repository:

class UserRepository implements UserRepositoryInterface
{
    private $databaseClient;

    public function __construct(SomeDatabaseClient $databaseClient)
    {
        $this->databaseClient = $databaseClient;
    }

    public function find($userId)
    {
        return $this->databaseClient->findMagically($userId);
    }
}

This is a trait:

trait PhystrixCommandExecutorTrait
{
    private function executeCommand($callback)
    {
        /** @var PhystrixCommand $command */
        $command = $this->commandFactory->getCommand(
            PhystrixCommand::class,
            $callback,
            static::COMMAND_NAME
        );

        return $command->execute();
    }
}

This is the phystrixized repository:

class PhystrixUserRepository implements UserPhystrixRepositoryInterface
{
    use PhystrixCommandExecutorTrait;

    const COMMAND_NAME = 'user_repository';

    private $commandFactory;
    private $userRepository;

    public function __construct(CommandFactory $commandFactory, UserRepositoryInterface $userRepository)
    {
        $this->commandFactory = $commandFactory;
        $this->userRepository = $userRepository;
    }

    public function find($userId)
    {
        $callback = function () use ($userId) {
            return $this->userRepository->find($userId);
        };

        return $this->executeCommand($callback);
    }
}

Hystrix Dashboard

  • Java application that reads event-stream data from hystrix clients and visualizes (hystrix dashboard)

Hystrix Dashboard

data: {
  "type": "HystrixCommand",
  "name": "PlaylistGet",
  "group": "PlaylistGet",
  "currentTime": 1355239617628,
  "isCircuitBreakerOpen": false,
  "errorPercentage": 0,
  "errorCount": 0,
  "requestCount": 121,
  ...
}

phystrix-dashboard: Utilities to create a page that serves monitoring information to Hystrix-Dashboard.

Hystrix Dashboard

  • What if you have N instances for the same app reporting event-stream data? Turbine, an stream aggregator.

Phystrix Dashboard

  • Phystrix stores data in APC.
  • phystrix-dashboard serves data from APC.
  • Someone has made a little controller for Symfony (gist)
<?php
namespace MyBundle\Controller;
use Odesk\PhystrixDashboard\MetricsEventStream\ApcMetricsPoller;
use Odesk\PhystrixDashboard\MetricsEventStream\MetricsServer;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpFoundation\StreamedResponse;
use Zend\Config\Config;
class PhystrixController extends Controller
{
    /**
     * Streaming response to return Phystrix statistics.
     *
     * @Route("/phystrix_status", name="phystrix_status")
     */
    public function statusAction()
    {
        // Made this end point accesible from local host in production environment
        if ('prod' === $this->get('kernel')->getEnvironment()) {
            if (isset($_SERVER['HTTP_CLIENT_IP'])
                || isset($_SERVER['HTTP_X_FORWARDED_FOR'])
                || !(in_array(@$_SERVER['REMOTE_ADDR'], array('127.0.0.1', 'fe80::1', '::1')) || php_sapi_name() === 'cli-server')
            ) {
                header('HTTP/1.0 403 Forbidden');
                exit('You are not allowed to access this file.');
            }
        }
        $response = new StreamedResponse();
        $response->setCallback(function () {
            // Get phystrix data configuration
            $configData = $this->getParameter('phystrix.configuration.data');
            $config = new Config($configData);
            $metricsPoller = new ApcMetricsPoller($config);
            $metricsServer = new MetricsServer($metricsPoller);
            $metricsServer->run();
        });
        $response->send();
    }
}

Questions?