Resilient PHP applications with Phystrix
Resilience...
...the capacity to recover quickly from difficulties; toughness.
Hystrix
Long time ago Netflix developed Hystrix, a Java library to make Netflix API more resilient.
The problem...
- Systems fails (it is inevitably).
- Dependencies fails produce cascade failures.
On a system with 30 dependencies, each with a 99.99% uptime, in 1 billion request means 3,000,000 are failures.
The problem...
![](https://github.com/Netflix/Hystrix/wiki/images/soa-1-640.png)
The system receives requests that depends on third party services to be resolved (database, API, queue, ...)
The problem...
![](https://github.com/Netflix/Hystrix/wiki/images/soa-2-640.png)
If one of the dependencies is down the requests is active until a timeout is triggered
The problem...
![](https://github.com/Netflix/Hystrix/wiki/images/soa-3-640.png)
More request arrives and are waiting the response of a system that is failing.
- Our system could collapse by the volume of pending request.
- Setting timeouts is not a solution (which value to use? 1sec, 5secs, ...)
- Why to continue asking a system we know is down
What problem does Hystrix solve?
- Fail fast and rapidly recover.
- Stop cascading failures in a complex distributed system.
- Protect from and control over latency and failure from dependencies (typically over the network and via third-party client libraries).
- Fallback and gracefully degrade when possible.
- Enable near real-time monitoring, alerting, and operational control.
Phystrix
Phystrix
Circuit breaker pattern
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/3389071/88.jpg)
Phystrix
Circuit breaker pattern
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/2712303/phystrix-circuit-breaker-diagram.png)
Phystrix
- Actions to third party dependencies must be implemented as commands:
- Subclass from AbstractCommand
- Implement your logic within the run() method.
- Use getFallback() as alternative logic when run() fails.
- Metrics are stored through APC.
Phystrix
use Odesk\Phystrix\AbstractCommand;
class PrintNameCommand extends AbstractCommand
{
// Metrics will be grouped by this name
protected $commandKey = "print_name_command";
private $name;
public function __construct($name)
{
$this->name = $name;
}
/**
* This function is called internally by Phystrix, only if the request is allowed
*/
protected function run()
{
return 'Hello ' . $this->name;
}
/**
* Optionally we implemented a fallback method.
*/
protected function getFallback()
{
return 'Hello no-name :(';
}
}
Custom command implementation:
Phystrix
$phystrixFactory = ...; // Must be a CommandFactory reference
// Create a command instance passing parameters to the constructor
$command = $phystrixFactory->getCommand('PrintNameCommand', 'Batman');
$result = $command->execute(); // Hello Batman
Command invokation:
phystrix-bundle: Easily integrate Phystrix within Symfony2 applications.
Phystrix
- Configure thresholds to define when a circuit must be opened and closed.
- Configure thresholds per command.
odesk_phystrix:
default:
# Whether fallback logic of the phystrix command is enabled
fallback: false
circuitBreaker:
# How many failed request it might be before we open the circuit (disallow consecutive requests)
errorThresholdPercentage: 5
# If true, the circuit breaker will always be open regardless the metrics
forceOpen: false
# If true, the circuit breaker will always be closed, allowing all requests, regardless the metrics
forceClosed: false
# How many requests we need minimally before we can start making decisions about service stability
requestVolumeThreshold: 1
# For how long to wait before attempting to access a failing service
sleepWindowInMilliseconds: 5000
metrics:
# This is for caching metrics so they are not recalculated more often than needed
healthSnapshotIntervalInMilliseconds: 1000
# The period of time within which we the stats are collected
rollingStatisticalWindowInMilliseconds: 50000
# The more buckets the more precise and actual the stats and slower the calculation.
rollingStatisticalWindowBuckets: 10
# Request cache, if enabled and a command has getCacheKey implemented caches results within current http request
requestCache: false
# Request log collects all commands executed within current http request
requestLog: true
Let's make a break
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/3391726/break.gif)
- Command? It is nice but most of my code that access third party services is implemented within a repository.
- Possible solution: Create a proxy phystrixized repository that runs the source repository operations as commands.
Your source repository:
class UserRepository implements UserRepositoryInterface
{
private $databaseClient;
public function __construct(SomeDatabaseClient $databaseClient)
{
$this->databaseClient = $databaseClient;
}
public function find($userId)
{
return $this->databaseClient->findMagically($userId);
}
}
This is a trait:
trait PhystrixCommandExecutorTrait
{
private function executeCommand($callback)
{
/** @var PhystrixCommand $command */
$command = $this->commandFactory->getCommand(
PhystrixCommand::class,
$callback,
static::COMMAND_NAME
);
return $command->execute();
}
}
This is the phystrixized repository:
class PhystrixUserRepository implements UserPhystrixRepositoryInterface
{
use PhystrixCommandExecutorTrait;
const COMMAND_NAME = 'user_repository';
private $commandFactory;
private $userRepository;
public function __construct(CommandFactory $commandFactory, UserRepositoryInterface $userRepository)
{
$this->commandFactory = $commandFactory;
$this->userRepository = $userRepository;
}
public function find($userId)
{
$callback = function () use ($userId) {
return $this->userRepository->find($userId);
};
return $this->executeCommand($callback);
}
}
Hystrix Dashboard
- Java application that reads event-stream data from hystrix clients and visualizes (hystrix dashboard)
Hystrix Dashboard
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/3391479/dashboard-annoted-circuit-640.png)
data: {
"type": "HystrixCommand",
"name": "PlaylistGet",
"group": "PlaylistGet",
"currentTime": 1355239617628,
"isCircuitBreakerOpen": false,
"errorPercentage": 0,
"errorCount": 0,
"requestCount": 121,
...
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/3391519/hystrix-dashboard-single-row-640.png)
phystrix-dashboard: Utilities to create a page that serves monitoring information to Hystrix-Dashboard.
Hystrix Dashboard
- What if you have N instances for the same app reporting event-stream data? Turbine, an stream aggregator.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/3391529/dashboard-direct-vs-turbine-640.png)
Phystrix Dashboard
- Phystrix stores data in APC.
- phystrix-dashboard serves data from APC.
- Someone has made a little controller for Symfony (gist)
<?php
namespace MyBundle\Controller;
use Odesk\PhystrixDashboard\MetricsEventStream\ApcMetricsPoller;
use Odesk\PhystrixDashboard\MetricsEventStream\MetricsServer;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpFoundation\StreamedResponse;
use Zend\Config\Config;
class PhystrixController extends Controller
{
/**
* Streaming response to return Phystrix statistics.
*
* @Route("/phystrix_status", name="phystrix_status")
*/
public function statusAction()
{
// Made this end point accesible from local host in production environment
if ('prod' === $this->get('kernel')->getEnvironment()) {
if (isset($_SERVER['HTTP_CLIENT_IP'])
|| isset($_SERVER['HTTP_X_FORWARDED_FOR'])
|| !(in_array(@$_SERVER['REMOTE_ADDR'], array('127.0.0.1', 'fe80::1', '::1')) || php_sapi_name() === 'cli-server')
) {
header('HTTP/1.0 403 Forbidden');
exit('You are not allowed to access this file.');
}
}
$response = new StreamedResponse();
$response->setCallback(function () {
// Get phystrix data configuration
$configData = $this->getParameter('phystrix.configuration.data');
$config = new Config($configData);
$metricsPoller = new ApcMetricsPoller($config);
$metricsServer = new MetricsServer($metricsPoller);
$metricsServer->run();
});
$response->send();
}
}
Questions?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/337684/images/3391720/questions.gif)