Spring Cloud Overview

What

We'll be taking a look at Spring Cloud, the open source microservices framework from Pivotal.

Why

• Spring Cloud embodies the collective knowledge of the JVM community on how microservices should be constructed.
• Non-JVM and non-Spring projects can borrow ideas from the stack and look for analogues.
• In some cases, the technology is non-JVM specific and can be shared.
• The hope is to start a discussion about how we borrow some of their ideas and implementation for non-JVM usages

How

• We'll be looking at examples from working code.
• We'll be taking a brief look at the various pieces that underpin Spring Cloud.
• We'll speculate as to how well the tools and techniques put forth by Spring Cloud might fit our situation.

Spring Cloud

Spring Cloud provides tools for developers to quickly build some of the common patterns in distributed systems (e.g. configuration management, service discovery, circuit breakers, intelligent routing, micro-proxy, control bus, one-time tokens, global locks, leadership election, distributed sessions, cluster state). Coordination of distributed systems leads to boiler plate patterns, and using Spring Cloud developers can quickly stand up services and applications that implement those patterns. They will work well in any distributed environment, including the developer's own laptop, bare metal data centres, and managed platforms such as Cloud Foundry.

Spring Cloud

• open source -- everything is in GitHub
• follows the familiar Spring programming model
• in general, integrates with best-of-breed technologies, inventing new ones only in rare cases
• many times, multiple integration options are available
• is easily incorporated into existing Spring projects
• it is possible to leverage some of the tools in non-JVM applications via "side cars".
• follows many of the 12 Factor App guidelines

Features

• distributed/versioned configuration
• service registration and discovery
• load balancing
• routing
• distributed messaging
• service-to-service calls
• circuit-breakers
• metrics
• distributed tracing
• global locks
• leadership election and cluster state

Distributed Configuration

• Spring already has a rich configuration mechanism: local file, system property, environment property, command-line switch
• Spring Cloud augments the existing mechanism with an external configuration service where changes can be version controlled and peer reviewed, separate from the code or build artifact.
• Changes can be reflected in all running servers without having to "bounce" them.
• Git is a popular backing store but there are other options, including Consul.

Distributed Configuration

• REST API supports multiple ways of obtaining configurations
• configurations are composable and overridable

/{application}/{profile}[/{label}]
/{application}-{profile}.yml
/{label}/{application}-{profile}.yml
/{application}-{profile}.properties
/{label}/{application}-{profile}.properties

• application = spring.application.name
• profile = spring.active.profiles
• label is Git branch, tag, label or commit id

Distributed Configuration

# src/main/resources/config/boostrap.yml

spring:
    application:
        name: example
    cloud:
        config:
            uri: ${SPRING_CONFIG_URI:http://localhost:2020}

# it is possible to use service discovery instead of a well-known URI

Distributed Configuration

# application.yml in the Git repository
# shared configuration for all applications

applications: example
endpoints:
    health:
        time-to-live: 1000
        sensitive: false
logging:
    config: classpath:logback.xml
management:
    contextPath: /operations
    security:
        enabled: false
        role: admin
        sessions: stateless
security:
    user:
        name: developer
        password: developer
    basic:
        enabled: false
        realm: example
server:
    contextPath: /
    port: 8080
    useForwardHeaders: true
    tomcat:
        portHeader: X-Forwarded-Port
        protocolHeader: X-Forwarded-Protocol-Header
        remoteIpHeader: X-Forwarded-Remote-IP-Header
spring:
    cloud:
        consul:
            host: localhost
            port: 8500
            discovery:
                healthCheckInterval: 15s
                healthCheckPath: ${management.contextPath}/health
                instanceId: ${spring.application.name}:${random.value}
                preferAgentAddress: true
                preferIpAddress: true
    groovy:
        template:
            check-template-location: false
    inetutils:
        timeoutSeconds: 1
        defaultHostname: localhost
        localhost: 127.0.0.1
        ignoredInterfaces:
            - docker0
    jackson:
        serialization:
            indent_output: true
        serialization-inclusion: non_empty
    main:
        banner-mode: console
    rabbitmq:
        host: localhost
        password: guest
        port: 5672
        virtualHost: /
        username: guest
turbine:
    aggregator:
        clusterConfig: ${applications}
    appConfig: ${applications}

Distributed Configuration

# example-default.yml in the Git repository
# configuration for the example application when using the default profile

example:
    foo: default
    exchangeName: some-exchange
    queueName: some-queue
    deadLetterExchangeName: dead-letter
    deadLetterQueueName: dead-letter
    messageRetryAttempts: 3

Distributed Configuration

# example-bamboo.yml in the Git repository
# configuration for the example application when using the bamboo profile

example:
  foo: bamboo

Distributed Configuration

// The only code needed to spin up a configuration server

@SpringBootApplication
@EnableConfigServer     <----- magic incantation
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }

}

Distributed Configuration

# src/main/resources/config/application.yml

spring:
    cloud:
        config:
            server:
                git:
                    uri: https://github.com/kurron/spring-configuration-files.git

• Consul can be used in place of a dedicated server
• requires a mechanism to watch Git and push the changes to Consul
• git2Consul is a Javascript tool that can do that

Distributed Configuration

# application.yml
nginx:
    server:
        name: example.com

# nginx.conf
server {
    listen              80;
    server_name         ${nginx.server.name};
}

http localhost:8080/CONFIGSERVER/example/default/master/nginx.conf

HTTP/1.1 200 OK
Content-Disposition: inline;filename=f.txt
Content-Type: text/plain;charset=UTF-8
Date: Wed, 30 Mar 2016 01:39:02 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080

server {
    listen              80;
    server_name         example.com; <---- two files are combined
}

Service Registration

• Eureka, Zookeeper and Consul supported
• Etcd?
• I used Consul

// src/main/groovy/org/kurron/example/Application.groovy

@SpringBootApplication
@EnableCircuitBreaker
@EnableHystrixDashboard
@EnableTurbine
@EnableDiscoveryClient <---- magic incantation
@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }
}

Service Registration

# pulled down as part of the shared configuration
spring:
    cloud:
        consul:
            host: localhost
            port: 8500
            discovery:
                healthCheckInterval: 15s
                healthCheckPath: ${management.contextPath}/health
                instanceId: ${spring.application.name}:${random.value}
                preferAgentAddress: true
                preferIpAddress: true

Service Registration

Service Discovery

// src/main/groovy/org/kurron/example/Application.groovy

@SpringBootApplication
@EnableCircuitBreaker
@EnableHystrixDashboard
@EnableTurbine
@EnableDiscoveryClient <---- magic incantation
@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }
}

Service Discovery

def 'call service by hand'() {
    given: 'a proper testing environment'
    assert serviceName      <----- injected by Spring
    assert discoveryClient  <----- injected by Spring

    when: 'we call checkTheTime'
    def template = new TestRestTemplate()
    //WARNING: this call can come back empty/null!
    def instances = discoveryClient.getInstances( serviceName )
    def chosen = randomElement( instances ) as ServiceInstance
    def uri = constructURI( chosen, '/descriptor/application' )
    ResponseEntity<String> response = template.getForEntity( uri, String )

    then: 'we get a proper response'
    response.statusCode == HttpStatus.OK
}

URI constructURI( ServiceInstance service, String path ) {
    UriComponentsBuilder.newInstance()
                        .scheme( 'http' )
                        .host( service.host )
                        .port( service.port )
                        .path( path )
                        .build().toUri()
}

Service Discovery

def 'call service using load balancer'() {
    given: 'a proper testing environment'
    assert serviceName  <--- we use a logical name, not a host name
    assert loadBalancer <-- watches Consul for health changes

    when: 'we call checkTheTime'
    def template = new TestRestTemplate()
    def chosen = Optional.ofNullable( loadBalancer.choose( serviceName ) )
    def uri = constructURI( chosen.orElseThrow( unavailableLogic ), '/descriptor/application' )
    ResponseEntity<String> response = template.getForEntity( uri, String )

    then: 'we get a proper response'
    response.statusCode == HttpStatus.OK
}

def unavailableLogic = { new RuntimeException( 'No service instances!' ) }

URI constructURI( ServiceInstance service, String path ) {
    UriComponentsBuilder.newInstance()
                        .scheme( 'http' )
                        .host( service.host )
                        .port( service.port )
                        .path( path )
                        .build().toUri()
}

Load Balancing

• Ribbon is Netflix's client-side load balancer.
• Feign is Netflix's delcarative REST client that bundles in load-balancing, retry logic, circuit-breakers and RxJava support.
• Using RestTemplate means that we have to bake in all those features ourselves.  Options include Groovy meta-programming, Java Aspects or by hand implementation of the RestOperations interface.

Routing

• Feign is a Netflix abstraction that combines best practices into a single solution.
• service discovery
• load balancing
• circuit-breakers
• timeouts
• bulk-heading
• delcarative HTTP client

Routing

@Category( OutboundIntegrationTest )
@IntegrationTest
@ContextConfiguration( classes = FeignIntegrationTestConfiguration, 
                       loader = SpringApplicationContextLoader )
class FeignIntegrationTest extends Specification implements GenerationAbility {

    @Autowired
    private RestGatewayClient client  <---- resource-specific client

    def 'call happy path'() {
        given: 'a proper testing environment'
        assert client

        when: 'we call happyPath'
        def results = client.happyPath()  <---- convenience method

        then: 'we get a proper response'
        results
        println results
    }
}

Routing

@FeignClient( name = 'example', <---- logical service name
              configuration = RestGatewayClientConfiguration.
              fallback = RestGatewayClientFallback ) <---- fallback logic
interface RestGatewayClient {

    @RequestMapping( method = RequestMethod.GET,
                     path = '/descriptor/application',
                     consumes = 'application/json' )
    String happyPath()

    @RequestMapping( method = RequestMethod.GET,
                     path = '/descriptor/fail',
                     consumes = 'application/json' )
    String systemFailure()

    @RequestMapping( method = RequestMethod.GET,
                     path = '/descriptor/fail/application',
                     consumes = 'application/json' )
    String applicationFailure()
}

Routing

class RestGatewayClientFallback implements RestGatewayClient {

    @Override
    String happyPath() {
        'happyPath fallback returned'
    }

    @Override
    String systemFailure() {
        'systemFailure fallback returned'
    }

    @Override
    String applicationFailure() {
        'applicationFailure fallback returned'
    }
}

Routing

@Configuration
class RestGatewayClientConfiguration {

    @Bean
    RestGatewayClientFallback restGatewayClientFallback() {
        new RestGatewayClientFallback()
    }

    @Bean
    Logger.Level feignLoggerLevel() {
        Logger.Level.FULL
    }

    @Bean
    CustomErrorDecoder customErrorDecoder() {
        new CustomErrorDecoder()
    }

    @Bean
    Request.Options requestOptions() {
        int connectionTimeout = 1000
        int readTimeout = 1000
        new Request.Options( connectionTimeout, readTimeout  )
    }

    @Bean
    RequestInterceptor customRequestInterceptor() {
        { RequestTemplate template ->
            template.header( 'X-Custom-Header', Instant.now().toString() )
        } as RequestInterceptor
    }
}

Routing

Zuul is the front door for all requests from devices and web sites to the backend of the Netflix streaming application. As an edge service application, Zuul is built to enable dynamic routing, monitoring, resiliency and security. It also has the ability to route requests to multiple Amazon Auto Scaling Groups as appropriate.

Routing

• Authentication and Security - identifying authentication requirements for each resource and rejecting requests that do not satisfy them.
• Insights and Monitoring - tracking meaningful data and statistics at the edge in order to give us an accurate view of production.
• Dynamic Routing - dynamically routing requests to different backend clusters as needed.
• Stress Testing - gradually increasing the traffic to a cluster in order to gauge performance.

Routing

• Load Shedding - allocating capacity for each type of request and dropping requests that go over the limit.
• Static Response handling - building some responses directly at the edge instead of forwarding them to an internal cluster
• Multiregion Resiliency - routing requests across AWS regions in order to diversify our ELB usage and move our edge closer to our members

Routing

Routing

@SpringBootApplication
@EnableZuulProxy <---- embeds the proxy
@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }

}

Routing

zuul:
    routes:
        users:
            path: /myusers/**  <---- the path getting proxied
            serviceId: users_service  <---- service name in Consul
            sensitiveHeaders: Cookie,Set-Cookie,Authorization

# more elaborate configuration is required if circuit-breakers are desired

Routing

# Example of API strangulation
zuul:
    routes:
        first:  <---- send traffic to external URL
            path: /first/**
            url: http://first.example.com
        second:  <---- send traffic to internal /second service
            path: /second/**
            url: forward:/second
        third:  <---- send traffic to internal /3rd service
            path: /third/**
            url: forward:/3rd
        legacy:  <---- default to the old system
            path: /**
            url: http://legacy.example.com

Distributed Messaging

Spring Cloud Bus links nodes of a distributed system with a lightweight message broker. This can then be used to broadcast state changes (e.g. configuration changes) or other management instructions. A key idea is that the Bus is like a distributed Actuator for a Spring Boot application that is scaled out, but it can also be used as a communication channel between apps. The only implementation currently is with an AMQP broker as the transport, but the same basic feature set (and some more depending on the transport) is on the roadmap for other transports.

Distributed Messaging

• Spring Cloud Bus integrates with GitHub, GitLab and Bitbucket to monitor changes in the configuration repository and automatically push out notifications to applications that they should refresh their configurations
• The bus currently supports sending messages to all nodes listening or all nodes for a particular service.

Distributed Messaging

Spring Cloud Stream is a framework for building message-driven microservices. Spring Cloud Stream builds upon Spring Boot to create DevOps friendly microservice applications and Spring Integration to provide connectivity to message brokers. Spring Cloud Stream provides an opinionated configuration of message brokers, introducing the concepts of persistent pub/sub semantics, consumer groups and partitions across several middleware vendors. This opinionated configuration provides the basis to create stream processing applications.

Distributed Messaging

• RabbitMQ
• Redis
• Kafka
• Gemfire
• Amazon SQS?
• Simple to set up and use
• a consumer group deals with the competing consumer problem -- only one instance in the group will get the message
• partitioning is "sticky" routing of messages where stateful processing is important
• Spring Integration under the hood

Distributed Messaging

@Slf4j
@OutboundGateway
@EnableBinding( Source ) <---- magic incantation
class StreamProducer implements GenerationAbility{

    @InboundChannelAdapter( Source.OUTPUT )
    public String send() {
        def message = "Hello, World. It is ${Instant.now().toString()}"
        log.info( 'Sending {}', message )
        message
    }
}

// we could have sent an object and had Spring auto-transform it into JSON
// we can also assemble messages by hand if we need more control, e.g. headers

Distributed Messaging

Slf4j
@InboundGateway
@EnableBinding( Sink )
class StreamGateway {

    @StreamListener( Sink.INPUT )
    void processMessage( String  request ) {
        log.info( 'Hearing {}', request )
    }
}

// we could have accepted an object and had Spring auto-transform it from JSON

Distributed Messaging

# example-default.yml on the configuration server

spring:
    cloud:
        stream:
            bindings:
                input:  <---- this is the channel name in the code
                    destination: rendezous  <---- RabbitMQ exchange to use
                    group: example
                    contentType: text/plain
                    consumer:
                        concurrency: 1
                        partitioned: false
                        maxAttempts: 3
                        backOffInitialInterval: 1000
                        backOffMaxInterval: 10000
                        backOffMultiplier: 2.0
                        acknowledgeMode: AUTO
                        autoBindDlq: true
                        durableSubscription: true
                        maxConcurrency: 1
                        prefetch: 1
                        prefix: ${spring.application.name}.
                        requeueRejected: true
                        republishToDlq: true
                        transacted: false
                        txSize: 1
                output:  <---- this is the channel name in the code
                    destination: rendezous  <---- RabbitMQ exchange to use
                    contentType: text/plain
                    producer:
                        autoBindDlq: true
                        batchingEnabled: false
                        batchSize: 100
                        batchBufferLimit: 1000
                        batchTimeout: 5000
                        compress: false
                        deliveryMode: PERSISTENT
                        prefix: ${spring.application.name}.

Distributed Messaging

http localhost:8080/operations/health

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Mon, 28 Mar 2016 23:54:03 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: example:8080
X-B3-Sampled: 0
X-B3-SpanId: 64740575bcecba80
X-B3-TraceId: 64740575bcecba80

{
    "binders": {  <---- all the brokers Spring Cloud Stream is bound to
        "rabbit": {  
            "binderHealthIndicator": {
                "status": "UP", 
                "version": "3.6.1"
            }, 
            "status": "UP"
        }, 
        "status": "UP"
    }, 
    "configServer": {
        "propertySources": [
            "https://github.com/kurron/spring-configuration-files.git/application.yml"
        ], 
        "status": "UP"
    }, 
    "description": "Spring Cloud Consul Discovery Client", 
    "discoveryComposite": {
        "description": "Spring Cloud Consul Discovery Client", 
        "discoveryClient": {
            "description": "Spring Cloud Consul Discovery Client", 
            "services": [
                "consul", 
                "example"
            ], 
            "status": "UP"
        }, 
        "status": "UP"
    }, 
    "diskSpace": {
        "free": 25699893248, 
        "status": "UP", 
        "threshold": 10485760, 
        "total": 39233855488
    }, 
    "hystrix": {
        "status": "UP"
    }, 
    "rabbit": {
        "status": "UP", 
        "version": "3.6.1"
    }, 
    "refreshScope": {
        "status": "UP"
    }, 
    "status": "UP"
}

Service-To-Service Calls

While Spring Cloud Stream makes it easy for individual boot apps to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-app pipelines, where microservice apps are sending data to each other. This can be achieved by correlating the input and output destinations of adjacent apps.

• instance counting and instance indexing help deal with scale out
• partition targeting ensures data arrives at its intended destination, regardless of the size of the cluster

Circuit-Breakers

Circuit-Breakers

Circuit-Breakers

Uses Netflix's Hystrix library

Circuit-Breakers

Circuit-Breakers

Bulkheading provides isolation

Circuit-Breakers

• load shedding
• timeout

Circuit-Breakers

• load shedding via counters
• no timeouts

Circuit-Breakers

• request caching
• deduplication

Circuit-Breakers

@Slf4j
@SpringBootApplication
@EnableCircuitBreaker    <---- magic incantation
@EnableHystrixDashboard
@EnableTurbine
@EnableDiscoveryClient
@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }
}

Circuit-Breakers

@OutboundGateway
class RemoteTimeGateway implements TimeService {

    @HystrixCommand( fallbackMethod = 'defaultTime' )  <--- magic
    Instant checkTheTime() {
        // force an error to trigger the circuit-breaker
        throw = new UnsupportedOperationException( 'checkTime' )
    }

    // You don't have to provide a fallback
    private Instant defaultTime() {
        // can't reach the time server, use local time instead
        Instant.now()
    }
}

We're jumping threads so some thought has to be given if thread-specific context needs to be propogated

Circuit-Breakers

{
    "hystrix": {
        "openCircuitBreakers": [
            "RemoteTimeGateway::checkTheTime"
        ], 
        "status": "CIRCUIT_OPEN"  <---- not getting to the service
    }, 
    "rabbit": {
        "status": "UP", 
        "version": "3.6.1"
    }, 
    "status": "UP"  <---- up because we have fallbacks in place
}

• Health check reflects any open breakers
• Notice overall health is still UP

Circuit-Breakers

Each service can host its own dashboard

Circuit-Breakers

Turbine aggregates breaker data into one panel

Security

Spring Cloud Security offers a set of primitives for building secure applications and services with minimum fuss. A declarative model which can be heavily configured externally (or centrally) lends itself to the implementation of large systems of co-operating, remote components, usually with a central indentity management service. It is also extremely easy to use in a service platform like Cloud Foundry. Building on Spring Boot and Spring Security OAuth2 we can quickly create systems that implement common patterns like single sign on, token relay and token exchange.

Security

@SpringBootApplication
@EnableOAuth2Sso  <---- Turns on OAuth2 single sign on
@EnableResourceServer <---- Makes this application a resource server
@EnableAuthorizationServer <---- Make this application an authorization server

@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }
}

Security

• never got the GitHub example fully working
• can run our own OAuth2 servers, if desired
• Spring has an OAuth2RestTemplate that automatically does the token management for us

Metrics

http http://10.0.2.15:8080/operations/metrics --pretty=format

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Sun, 27 Mar 2016 22:41:26 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: example:8080

{
    "classes": 12101, 
    "classes.loaded": 12101, 
    "classes.unloaded": 0, 
    "currentTimeCounter(type=NORMALIZED)": 0.016666666666666666, 
    "currentTimeDistribution(statistic=count)": 13.0, 
    "currentTimeDistribution(statistic=totalAmount)": 4.2259077013282903e+18, 
    "currentTimeTimer(statistic=count)": 0.016666666666666666, 
    "currentTimeTimer(statistic=max)": 0.365126288, 
    "currentTimeTimer(statistic=totalOfSquares)": 0.13331720618865894, 
    "currentTimeTimer(statistic=totalTime)": 0.006085438133333334, 
    "gc.ps_marksweep.count": 3, 
    "gc.ps_marksweep.time": 336, 
    "gc.ps_scavenge.count": 17, 
    "gc.ps_scavenge.time": 225, 
    "heap": 1359872, 
    "heap.committed": 550400, 
    "heap.init": 96256, 
    "heap.used": 350533, 
    "httpsessions.active": 0, 
    "httpsessions.max": -1, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.0()": 2.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.100()": 12.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.25()": 2.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.50()": 4.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.75()": 7.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.90()": 11.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.95()": 11.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.99()": 12.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.99.5()": 12.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.currentConcurrentExecutionCount()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.errorCount()": 8.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.errorPercentage()": 100.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.latencyExecute_mean()": 5.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.latencyTotal_mean()": 5.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_circuitBreakerErrorThresholdPercentage()": 50.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_circuitBreakerRequestVolumeThreshold()": 20.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_circuitBreakerSleepWindowInMilliseconds()": 5000.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_executionIsolationSemaphoreMaxConcurrentRequests()": 10.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_executionIsolationThreadTimeoutInMilliseconds()": 1000.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_executionTimeoutInMilliseconds()": 1000.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests()": 10.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.propertyValue_metricsRollingStatisticalWindowInMilliseconds()": 10000.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.reportingHosts()": 1.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.requestCount()": 8.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountBadRequests()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountCollapsedRequests()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountEmit()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountExceptionsThrown()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountFailure()": 8.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountFallbackEmit()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountFallbackFailure()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountFallbackMissing()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountFallbackRejection()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountFallbackSuccess()": 8.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountResponsesFromCache()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountSemaphoreRejected()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountShortCircuited()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountSuccess()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountThreadPoolRejected()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingCountTimeout()": 0.0, 
    "hystrix.HystrixCommand.RemoteTimeGateway.checkTheTime.rollingMaxConcurrentExecutionCount()": 1.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentActiveCount()": 0.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentCompletedTaskCount()": 13.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentCorePoolSize()": 10.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentLargestPoolSize()": 10.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentMaximumPoolSize()": 10.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentPoolSize()": 10.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentQueueSize()": 0.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.currentTaskCount()": 13.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.propertyValue_metricsRollingStatisticalWindowInMilliseconds()": 10000.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.propertyValue_queueSizeRejectionThreshold()": 5.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.reportingHosts()": 1.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.rollingCountCommandRejections()": 0.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.rollingCountThreadsExecuted()": 8.0, 
    "hystrix.HystrixThreadPool.RemoteTimeGateway.rollingMaxActiveThreads()": 1.0, 
    "instance.uptime": 480861, 
    "mem": 644575, 
    "mem.free": 199866, 
    "nonheap": 0, 
    "nonheap.committed": 96384, 
    "nonheap.init": 2496, 
    "nonheap.used": 94175, 
    "processors": 2, 
    "response.descriptor.application()": 8.0, 
    "response.operations.health()": 248.0, 
    "response.operations.metrics()": 30.0, 
    "systemload.average": 0.4, 
    "threads": 48, 
    "threads.daemon": 36, 
    "threads.peak": 48, 
    "threads.totalStarted": 79, 
    "uptime": 491444
}

Metrics

• Netflix has a metrics library called Spectator
• It provides counters, timers, gauges and distribution summaries
• Netflix has a metrics back end called Atlas

An Atlas standalone node running on an r3.2xlarge (61GB RAM) can handle roughly 2 million metrics per minute for a given 6 hour window.

Metrics

@SpringBootApplication
@EnableCircuitBreaker
@EnableHystrixDashboard
@EnableTurbine
@EnableDiscoveryClient
@EnableFeignClients
@EnableAtlas  <---- magic incantation
@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }

    @Bean
    AtlasTagProvider atlasCommonTags( @Value( '${spring.application.name}' ) String appName ) {
        [ 'defaultTags': { ['app': appName] } ] as AtlasTagProvider
    }
}

Metrics

/**
 * Metrics collector.
*/
private final Registry registry

@Override
Instant currentTime() {
    // the timer also counts so this is redundant
    def counter = registry.counter( 'currentTimeCounter' )
    counter.increment()

    // contrived example: normally this is used to record some incoming value
    def distribution = registry.distributionSummary( 'currentTimeDistribution' )
    distribution.record( randomLong() )

    // gauges, which track the current number of something, like queue size, are also available

    def timer = registry.timer( 'currentTimeTimer' )
    // in a real implementation we would interact with multiple services and take
    // the best result but this is only an example
    // The timer simultaneously records 4 statistics: count, max, totalOfSquares, and totalTime.
    timer.record( { gateway.checkTheTime() } as Callable<Instant> )
}

Metrics

# shared application.yml
netflix:
    atlas:
        uri: http://localhost:7101 <---- location of the Atlas instance

Metrics

/api/v1/graph?
  e=2012-01-01T00:00
  &no_legend=1
  &q=
    name,sps,:eq,
    (,nf.cluster,),:by,
    :pct,
    :stack
  &s=e-1w

Distributed Tracing

• distributed tracing is complicated
• Google wrote "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure"
• Twitter implemented it in their Zipkin framework
• Spring Sleuth is a Dapper compatible abstraction that can integrate with a couple back-ends, including Zipkin

Distributed Tracing

Distributed Tracing

Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of. Spans also have other data, such as descriptions, timestamped events, key-value annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).

Spans are started and stopped, and they keep track of their timing information. Once you create a span, you must stop it at some point in the future.

Distributed Tracing

Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed big-data store, a trace might be formed by a put request.

Distributed Tracing

Annotation: is used to record existence of an event in time. Some of the core annotations used to define the start and stop of a request are:

• cs - Client Sent - The client has made a request. This annotation depicts the start of the span.

• sr - Server Received - The server side got the request and will start processing it. If one subtracts the cs timestamp from this timestamp one will receive the network latency.

Distributed Tracing

• ss - Server Sent - Annotated upon completion of request processing (when the response got sent back to the client). If one subtracts the sr timestamp from this timestamp one will receive the time needed by the server side to process the request.
• cr - Client Received - Signifies the end of the span. The client has successfully received the response from the server side. If one subtracts the cs timestamp from this timestamp one will receive the whole time needed by the client to receive the response from the server.

Distributed Tracing

Distributed Tracing

All this is under a single trace id

Distributed Tracing

• Spring automatically places interceptors on all inbound and outbound gateways
• Circuit-breakers
• HTTP filters
• RestTemplate
• Feign client
• Spring Integration
• Rabbit template?
• Zuul

Distributed Tracing

• log output can be searched via Kibana and Splunk
• Logstash grok pattern provided
• can output Zipkin compatible format
• can output to Spring Cloud Stream
• default output is to SLF4J
• can use sampling for high volume scenarios
• there is a manual API for special circumstances
• appears to be very customizable

Distributed Tracing

HTTP/1.1 200 OK
Content-Type: application/json; type=FIXME; version=1.0.0;charset=UTF-8
Date: Sun, 27 Mar 2016 23:15:57 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: example:8080
X-B3-Sampled: 1
X-B3-SpanId: 6fca525591c416be
X-B3-TraceId: 6fca525591c416be

{
    "path": "/descriptor/application", 
    "status": 200, 
    "time": "2016-03-27T23:15:57.917Z", 
    "timestamp": "2016-03-27T23:15:57.562Z"
}

HTTP/1.1 200 OK
Content-Type: application/json; type=FIXME; version=1.0.0;charset=UTF-8
Date: Sun, 27 Mar 2016 23:16:00 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: example:8080
X-B3-Sampled: 1
X-B3-SpanId: 29882e8f3c43d283
X-B3-TraceId: 29882e8f3c43d283

{
    "path": "/descriptor/application", 
    "status": 200, 
    "time": "2016-03-27T23:16:00.066Z", 
    "timestamp": "2016-03-27T23:16:00.048Z"
}

HTTP/1.1 200 OK
Content-Type: application/json; type=FIXME; version=1.0.0;charset=UTF-8
Date: Sun, 27 Mar 2016 23:22:18 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: example:8080
X-B3-Sampled: 1
X-B3-SpanId: 6db06cf2872894e
X-B3-TraceId: 6db06cf2872894e

{
    "path": "/descriptor/application", 
    "status": 200, 
    "time": "2016-03-27T23:22:18.887Z", 
    "timestamp": "2016-03-27T23:22:18.881Z"
}

No code changes required

Global Locks

Never found documentation on how to do global locks.

Leadership Election

Spring Cloud Cluster offers a set of primitives for building "cluster" features into a distributed system. Example are leadership election, consistent storage of cluster state, global locks and one-time tokens.

• Zookeper
• Etcd
• Hazelcast
• Redis
• Consul?

Leadership Election

@InboundGateway
class ClusterEventListener implements ApplicationListener<AbstractLeaderEvent> {

    @Override
    void onApplicationEvent( AbstractLeaderEvent event ) {

        switch ( event ) {
            case OnGrantedEvent:
                log.info( 'A new leader has been elected for role {}', event.role )
                break
            case OnRevokedEvent:
                break
                log.info( 'An old leader has been removed for role {}', event.role )
                break
            default:
                log.info( 'Heard an unclassified event of type {}', event.class.name )
                break
        }
    }
}

// Never got it to work with Redis

Cluster State

Never found any documentation on how to manage cluster state.

Polyglot Usage

• standalone JVM process
• service registration and discovery (Consul)
• load balancing (Ribbon)
• remote configuration
• if you need to have the sidecar register the service with Consul, you'll need a companion sidecar for each service

Polyglot Usage

@SpringBootApplication
@EnableSidecar  <---- magic incantation
@EnableConfigurationProperties( ApplicationProperties )
class Application {

    static void main( String[] args ) {
        SpringApplication.run( Application, args )
    }
}

dependencies {
    compile('org.springframework.cloud:spring-cloud-starter-feign')
    compile('org.springframework.cloud:spring-cloud-starter-ribbon')
    compile('org.springframework.cloud:spring-cloud-starter-zuul')
    compile('org.springframework.cloud:spring-cloud-netflix-sidecar') {
        exclude module: 'spring-cloud-netflix-eureka-client'
        exclude module: 'eureka-client'
        exclude module: 'ribbon-eureka'
    }
}

Polyglot Usage

# src/main/resources/config/application.yml

info:
    app:
        name: ${name}
        description: ${description}
        version: ${version}

spring:
    application:
        name: configuration-service  <---- non-JVM service name

sidecar:
    port: 2020
    health-uri: http://localhost:2020/operations/health <---- Consul checks this

Polyglot Usage

Polyglot Usage

http localhost:8080/hosts/configuration-service

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Tue, 29 Mar 2016 21:30:18 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080

[
    {
        "host": "127.0.0.1", 
        "port": 8080, 
        "secure": false, 
        "serviceId": "configuration-service", 
        "uri": "http://127.0.0.1:8080"  <---- points back to sidecar process
    }
]

Polyglot Usage

http localhost:8080/example/descriptor/application  <---- Zuul proxy

HTTP/1.1 200 OK
Content-Type: application/json; type=FIXME; version=1.0.0;charset=UTF-8
Date: Tue, 29 Mar 2016 21:44:05 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080
X-B3-Sampled: 1
X-B3-SpanId: 5dffc13065fde452
X-B3-TraceId: 5dffc13065fde452

{
    "path": "/descriptor/application", 
    "status": 200, 
    "time": "2016-03-29T21:44:05.716Z", 
    "timestamp": "2016-03-29T21:44:05.370Z"
}

Polyglot Usage

http localhost:8080/hosts/example/

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Wed, 30 Mar 2016 02:14:15 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080

[
    {
        "host": "127.0.0.1", 
        "port": 7070, 
        "secure": false, 
        "serviceId": "example",
        "uri": "http://127.0.0.1:7070"  <---- no proxy
    }, 
    {
        "host": "127.0.0.1", 
        "port": 9090, 
        "secure": false, 
        "serviceId": "example", 
        "uri": "http://127.0.0.1:9090"  <---- no proxy
    }
]

Polyglot Usage

http localhost:8080/CONFIGSERVER/example/default

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Tue, 29 Mar 2016 22:36:51 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080

{
    "name": "example", 
    "profiles": [
        "default"
    ], 
    "propertySources": [
        {
            "name": "https://github.com/kurron/spring-configuration-files.git/example-default.yml", 
            "source": {
                "example.deadLetterExchangeName": "dead-letter", 
                "example.deadLetterQueueName": "dead-letter", 
                "example.exchangeName": "some-exchange", 
                "example.foo": "default", 
                "example.inputChannelName": "my-input-channel", 
                "example.messageRetryAttempts": 3, 
                "example.queueName": "some-queue", 
                "sidecar.health-uri": "http://localhost:2020/operations/health", 
                "sidecar.port": 2020, 
                "spring.cloud.stream.bindings.input.consumer.acknowledgeMode": "AUTO", 
                "spring.cloud.stream.bindings.input.consumer.autoBindDlq": true, 
                "spring.cloud.stream.bindings.input.consumer.backOffInitialInterval": 1000, 
                "spring.cloud.stream.bindings.input.consumer.backOffMaxInterval": 10000, 
                "spring.cloud.stream.bindings.input.consumer.backOffMultiplier": 2.0, 
                "spring.cloud.stream.bindings.input.consumer.concurrency": 1, 
                "spring.cloud.stream.bindings.input.consumer.durableSubscription": true, 
                "spring.cloud.stream.bindings.input.consumer.maxAttempts": 3, 
                "spring.cloud.stream.bindings.input.consumer.maxConcurrency": 1, 
                "spring.cloud.stream.bindings.input.consumer.partitioned": false, 
                "spring.cloud.stream.bindings.input.consumer.prefetch": 1, 
                "spring.cloud.stream.bindings.input.consumer.prefix": "${spring.application.name}.", 
                "spring.cloud.stream.bindings.input.consumer.republishToDlq": true, 
                "spring.cloud.stream.bindings.input.consumer.requeueRejected": true, 
                "spring.cloud.stream.bindings.input.consumer.transacted": false, 
                "spring.cloud.stream.bindings.input.consumer.txSize": 1, 
                "spring.cloud.stream.bindings.input.contentType": "text/plain", 
                "spring.cloud.stream.bindings.input.destination": "rendezous", 
                "spring.cloud.stream.bindings.input.group": "example", 
                "spring.cloud.stream.bindings.output.contentType": "text/plain", 
                "spring.cloud.stream.bindings.output.destination": "rendezous", 
                "spring.cloud.stream.bindings.output.producer.autoBindDlq": true, 
                "spring.cloud.stream.bindings.output.producer.batchBufferLimit": 1000, 
                "spring.cloud.stream.bindings.output.producer.batchSize": 100, 
                "spring.cloud.stream.bindings.output.producer.batchTimeout": 5000, 
                "spring.cloud.stream.bindings.output.producer.batchingEnabled": false, 
                "spring.cloud.stream.bindings.output.producer.compress": false, 
                "spring.cloud.stream.bindings.output.producer.deliveryMode": "PERSISTENT", 
                "spring.cloud.stream.bindings.output.producer.prefix": "${spring.application.name}.", 
                "zuul.ignoredServices": "*", 
                "zuul.routes.first.path": "/first/**", 
                "zuul.routes.first.url": "http://first.example.com", 
                "zuul.routes.second.path": "/second/**", 
                "zuul.routes.second.url": "forward:/second", 
                "zuul.routes.third.path": "/third/**", 
                "zuul.routes.third.url": "forward:/3rd", 
                "zuul.routes.users.path": "/myusers/**", 
                "zuul.routes.users.sensitiveHeaders": "Cookie,Set-Cookie,Authorization", 
                "zuul.routes.users.serviceId": "users_service"
            }
        }, 
        {
            "name": "https://github.com/kurron/spring-configuration-files.git/application.yml", 
            "source": {
                "applications": "${spring.application.name}", 
                "endpoints.health.sensitive": false, 
                "endpoints.health.time-to-live": 1000, 
                "logging.config": "classpath:logback.xml", 
                "management.contextPath": "/operations", 
                "management.security.enabled": false, 
                "management.security.role": "admin", 
                "management.security.sessions": "stateless", 
                "netflix.atlas.uri": "http://localhost:7101", 
                "security.basic.enabled": false, 
                "security.basic.realm": "example", 
                "security.user.name": "developer", 
                "security.user.password": "developer", 
                "server.contextPath": "/", 
                "server.port": 8080, 
                "server.tomcat.portHeader": "X-Forwarded-Port", 
                "server.tomcat.protocolHeader": "X-Forwarded-Protocol-Header", 
                "server.tomcat.remoteIpHeader": "X-Forwarded-Remote-IP-Header", 
                "server.useForwardHeaders": true, 
                "spring.cloud.cluster.leader.enabled": true, 
                "spring.cloud.cluster.leader.id": "Sponge Bob", 
                "spring.cloud.cluster.leader.role": "Master Of The Universe", 
                "spring.cloud.config.allowOverride": true, 
                "spring.cloud.config.failFast": true, 
                "spring.cloud.config.overrideNone": false, 
                "spring.cloud.config.overrideSystemProperties": false, 
                "spring.cloud.consul.discovery.healthCheckInterval": "15s", 
                "spring.cloud.consul.discovery.healthCheckPath": "${management.contextPath}/health", 
                "spring.cloud.consul.discovery.instanceId": "${spring.application.name}:${random.value}", 
                "spring.cloud.consul.discovery.preferAgentAddress": true, 
                "spring.cloud.consul.discovery.preferIpAddress": true, 
                "spring.cloud.consul.host": "localhost", 
                "spring.cloud.consul.port": 8500, 
                "spring.cloud.stream.rabbit.binder.addresses[0]": "localhost", 
                "spring.cloud.stream.rabbit.binder.adminAdresses[0]": "localhost", 
                "spring.cloud.stream.rabbit.binder.password": "guest", 
                "spring.cloud.stream.rabbit.binder.username": "guest", 
                "spring.cloud.stream.rabbit.binder.vhost": "/", 
                "spring.groovy.template.check-template-location": false, 
                "spring.inetutils.defaultHostname": "localhost", 
                "spring.inetutils.ignoredInterfaces[0]": "docker0", 
                "spring.inetutils.localhost": "127.0.0.1", 
                "spring.inetutils.timeoutSeconds": 1, 
                "spring.jackson.serialization-inclusion": "non_empty", 
                "spring.jackson.serialization.indent_output": true, 
                "spring.main.banner-mode": "console", 
                "spring.oauth2.client.accessTokenUri": "https://github.com/login/oauth/access_token", 
                "spring.oauth2.client.clientAuthenticationScheme": "form", 
                "spring.oauth2.client.clientId": "a1c5113ec74d4fc16c69", 
                "spring.oauth2.client.clientSecret": "aac122e5814095dc3525f1a19e5274353330f09d", 
                "spring.oauth2.client.userAuthorizationUri": "https://github.com/login/oauth/authorize", 
                "spring.oauth2.resource.preferTokenInfo": false, 
                "spring.oauth2.resource.userInfoUri": "https://api.github.com/user", 
                "spring.rabbitmq.host": "localhost", 
                "spring.rabbitmq.password": "guest", 
                "spring.rabbitmq.port": 5672, 
                "spring.rabbitmq.username": "guest", 
                "spring.rabbitmq.virtualHost": "/", 
                "turbine.aggregator.clusterConfig": "${applications}", 
                "turbine.appConfig": "${applications}"
            }
        }
    ], 
    "version": "527c752ddbfdd5b9f2f2c4862883fc5fd83668df"
}

Polyglot Usage

http localhost:8080/CONFIGSERVER/example-default.yml

HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
Date: Wed, 30 Mar 2016 01:21:32 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080

spring:
  cloud:
    stream:
      rabbit:
        binder:
          addresses:
          - localhost
          adminAdresses:
          - localhost
          password: guest
          username: guest
          vhost: /
      bindings:
        input:
          consumer:
            acknowledgeMode: AUTO
            autoBindDlq: true
            backOffInitialInterval: 1000
            backOffMaxInterval: 10000
            backOffMultiplier: 2.0
            concurrency: 1
            durableSubscription: true
            maxAttempts: 3
            maxConcurrency: 1
            partitioned: false
            prefetch: 1
            prefix: ${spring.application.name}.
            republishToDlq: true
            requeueRejected: true
            transacted: false
            txSize: 1
          contentType: text/plain
          destination: rendezous
          group: example
        output:
          contentType: text/plain
          destination: rendezous
          producer:
            autoBindDlq: true
            batchBufferLimit: 1000
            batchSize: 100
            batchTimeout: 5000
            batchingEnabled: false
            compress: false
            deliveryMode: PERSISTENT
            prefix: ${spring.application.name}.
    cluster:
      leader:
        enabled: true
        id: Sponge Bob
        role: Master Of The Universe
    config:
      allowOverride: true
      failFast: true
      overrideNone: false
      overrideSystemProperties: false
    consul:
      discovery:
        healthCheckInterval: 15s
        healthCheckPath: /operations/health
        instanceId: ${spring.application.name}:${random.value}
        preferAgentAddress: true
        preferIpAddress: true
      host: localhost
      port: 8500
  inetutils:
    ignoredInterfaces:
    - docker0
    defaultHostname: localhost
    localhost: 127.0.0.1
    timeoutSeconds: 1
  groovy:
    template:
      check-template-location: false
  jackson:
    serialization-inclusion: non_empty
    serialization:
      indent_output: true
  main:
    banner-mode: console
  oauth2:
    client:
      accessTokenUri: https://github.com/login/oauth/access_token
      clientAuthenticationScheme: form
      clientId: a1c5113ec74d4fc16c69
      clientSecret: aac122e5814095dc3525f1a19e5274353330f09d
      userAuthorizationUri: https://github.com/login/oauth/authorize
    resource:
      preferTokenInfo: false
      userInfoUri: https://api.github.com/user
  rabbitmq:
    host: localhost
    password: guest
    port: 5672
    username: guest
    virtualHost: /
applications: ${spring.application.name}
endpoints:
  health:
    sensitive: false
    time-to-live: 1000
example:
  deadLetterExchangeName: dead-letter
  deadLetterQueueName: dead-letter
  exchangeName: some-exchange
  foo: default
  inputChannelName: my-input-channel
  messageRetryAttempts: 3
  queueName: some-queue
logging:
  config: classpath:logback.xml
management:
  contextPath: /operations
  security:
    enabled: false
    role: admin
    sessions: stateless
netflix:
  atlas:
    uri: http://localhost:7101
security:
  basic:
    enabled: false
    realm: example
  user:
    name: developer
    password: developer
server:
  contextPath: /
  port: 8080
  tomcat:
    portHeader: X-Forwarded-Port
    protocolHeader: X-Forwarded-Protocol-Header
    remoteIpHeader: X-Forwarded-Remote-IP-Header
  useForwardHeaders: true
sidecar:
  health-uri: http://localhost:2020/operations/health
  port: 2020
turbine:
  aggregator:
    clusterConfig: ${spring.application.name}
  appConfig: ${spring.application.name}
zuul:
  ignoredServices: '*'
  routes:
    first:
      path: /first/**
      url: http://first.example.com
    second:
      path: /second/**
      url: forward:/second
    third:
      path: /third/**
      url: forward:/3rd
    users:
      path: /myusers/**
      sensitiveHeaders: Cookie,Set-Cookie,Authorization
      serviceId: users_service

Polyglot Usage

http localhost:8080/CONFIGSERVER/example-bamboo.json

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Wed, 30 Mar 2016 01:22:31 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
X-Application-Context: configuration-service:8080

{
    "applications": "${spring.application.name}", 
    "endpoints": {
        "health": {
            "sensitive": false, 
            "time-to-live": 1000
        }
    }, 
    "example": {
        "foo": "bamboo"
    }, 
    "logging": {
        "config": "classpath:logback.xml"
    }, 
    "management": {
        "contextPath": "/operations", 
        "security": {
            "enabled": false, 
            "role": "admin", 
            "sessions": "stateless"
        }
    }, 
    "netflix": {
        "atlas": {
            "uri": "http://localhost:7101"
        }
    }, 
    "security": {
        "basic": {
            "enabled": false, 
            "realm": "example"
        }, 
        "user": {
            "name": "developer", 
            "password": "developer"
        }
    }, 
    "server": {
        "contextPath": "/", 
        "port": 8080, 
        "tomcat": {
            "portHeader": "X-Forwarded-Port", 
            "protocolHeader": "X-Forwarded-Protocol-Header", 
            "remoteIpHeader": "X-Forwarded-Remote-IP-Header"
        }, 
        "useForwardHeaders": true
    }, 
    "spring": {
        "cloud": {
            "cluster": {
                "leader": {
                    "enabled": true, 
                    "id": "Sponge Bob", 
                    "role": "Master Of The Universe"
                }
            }, 
            "config": {
                "allowOverride": true, 
                "failFast": true, 
                "overrideNone": false, 
                "overrideSystemProperties": false
            }, 
            "consul": {
                "discovery": {
                    "healthCheckInterval": "15s", 
                    "healthCheckPath": "/operations/health", 
                    "instanceId": "${spring.application.name}:${random.value}", 
                    "preferAgentAddress": true, 
                    "preferIpAddress": true
                }, 
                "host": "localhost", 
                "port": 8500
            }, 
            "stream": {
                "rabbit": {
                    "binder": {
                        "addresses": [
                            "localhost"
                        ], 
                        "adminAdresses": [
                            "localhost"
                        ], 
                        "password": "guest", 
                        "username": "guest", 
                        "vhost": "/"
                    }
                }
            }
        }, 
        "groovy": {
            "template": {
                "check-template-location": false
            }
        }, 
        "inetutils": {
            "defaultHostname": "localhost", 
            "ignoredInterfaces": [
                "docker0"
            ], 
            "localhost": "127.0.0.1", 
            "timeoutSeconds": 1
        }, 
        "jackson": {
            "serialization": {
                "indent_output": true
            }, 
            "serialization-inclusion": "non_empty"
        }, 
        "main": {
            "banner-mode": "console"
        }, 
        "oauth2": {
            "client": {
                "accessTokenUri": "https://github.com/login/oauth/access_token", 
                "clientAuthenticationScheme": "form", 
                "clientId": "a1c5113ec74d4fc16c69", 
                "clientSecret": "aac122e5814095dc3525f1a19e5274353330f09d", 
                "userAuthorizationUri": "https://github.com/login/oauth/authorize"
            }, 
            "resource": {
                "preferTokenInfo": false, 
                "userInfoUri": "https://api.github.com/user"
            }
        }, 
        "rabbitmq": {
            "host": "localhost", 
            "password": "guest", 
            "port": 5672, 
            "username": "guest", 
            "virtualHost": "/"
        }
    }, 
    "turbine": {
        "aggregator": {
            "clusterConfig": "${spring.application.name}"
        }, 
        "appConfig": "${spring.application.name}"
    }
}

Closing

• Are there any tools or techniques we want to be using?
• In the JVM world, it would be a shame to not use this stuff
• Would side cars help or do we want to write our own implementations?