CI/CD Overview & Workshop

What Will I Learn?

Understand CI / CD
Understand how Jenkins and VSTS aids in DevOps CI/CD process
Implement jobs that integrate with Git source control repository, fetch code, build and deploy artifacts in public cloud in a fully automated way
Hand-on real-time experience with Jenkins & VSTS.

CI- Continuous Integration

"Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily leading to multiple integrations per day." --Martin Fowler

CD- Continuous Delivery

"Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time" --Martin Fowler

CD- Continuous Deployment

Continuous Deployment is a third term that's sometimes confused with Continuous Delivery. Where Continuous Delivery provides a process to create frequent releases but not necessarily deploy them, Continuous Deployment means that every change you make automatically gets deployed through the deployment pipeline.

Best practices

Maintain a code repository
Automate the build
Make the build self-testing
Everyone commits to the baseline every day
Every commit (to baseline) should be built
Keep the build fast
Test in a clone of the production environment
Make it easy to get the latest deliverables
Everyone can see the results of the latest build
Automate deployment

What is pipeline?

From monolith to

What are challenge with modern application architecture?

Gated application deployment Pattern

Independent service deployment pattern

Velocity & Stability

Stability: Testing NFRs in the build pipeline

Consumer drive Contract WorkFlow
Stability: Testing NFRs in the build pipeline
- SonarQube / Code Climate
Security testing
- OWASP Dependency check / bdd-security
- Docker Bench for Security / CoreOS Clair
Performance and Load testing
- Jmeter , gatling & ab -Apache http server benchmarking tool
Unit Testing Architecture
- archunit
Building Sample Dashboard to make things more visible.

Talking of (service) contracts…

APIs are service contracts
Consumer-driven Contracts
martinfowler.com/articles/consumerDrivenContracts.html

Talking about (service) contracts…

APIs are service contracts
Consumer-driven Contracts
martinfowler.com/articles/consumerDrivenContracts.html

CDC Workflow

Consumer writes a contract that defines an interaction with the API.
- For HTTP RPC this is simply requested with acceptable params and response
- Often the contract can be autogenerated from a test
Consumer issues a pull request to producer containing the contract

Producer runs the SUT (via pipeline) and tests if the contract is valid

If yes, then simply accept the pull request

If no, then modify the SUT to meet the contract
(this often involves inter-team communication), 
and then accept the pull request

Producer deploys (via pipeline), and consumer deploys (via pipeline)
- Take care in regards to backwards compatibility

package au.com.dius.pact.consumer.examples;

import au.com.dius.pact.consumer.Pact;
import au.com.dius.pact.consumer.PactProviderRuleMk2;
import au.com.dius.pact.consumer.PactVerification;
import au.com.dius.pact.consumer.dsl.PactDslWithProvider;
import au.com.dius.pact.consumer.exampleclients.ConsumerClient;
import au.com.dius.pact.model.RequestResponsePact;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import static org.junit.Assert.assertEquals;

public class ExampleJavaConsumerPactRuleTest {

    @Rule
    public PactProviderRuleMk2 provider = new PactProviderRuleMk2("test_provider", this);

    @Pact(provider="test_provider", consumer="test_consumer")
    public RequestResponsePact createFragment(PactDslWithProvider builder) {
        Map<String, String> headers = new HashMap<String, String>();
        headers.put("testreqheader", "testreqheadervalue");

        return builder
            .given("test state")
            .uponReceiving("ExampleJavaConsumerPactRuleTest test interaction")
                .path("/")
                .method("GET")
                .headers(headers)
            .willRespondWith()
                .status(200)
                .headers(headers)
                .body("{\"responsetest\": true, \"name\": \"harry\"}")
            .given("test state")
            .uponReceiving("ExampleJavaConsumerPactRuleTest second test interaction")
                .method("OPTIONS")
                .headers(headers)
                .path("/second")
                .body("")
            .willRespondWith()
                .status(200)
                .headers(headers)
                .body("")
            .toPact();
    }

    @Test
    @PactVerification("test_provider")
    public void runTest() throws IOException {
        Assert.assertEquals(new ConsumerClient(provider.getUrl()).options("/second"), 200);
        Map expectedResponse = new HashMap();
        expectedResponse.put("responsetest", true);
        expectedResponse.put("name", "harry");
        assertEquals(new ConsumerClient(provider.getUrl()).getAsMap("/", ""), expectedResponse);
    }
}

Contract Talks ...

docs.pact.io

org.springframework.cloud.contract.spec.Contract.make {
  request {
    method 'PUT'
    url '/fraudcheck'
    body("""
    {
      "clientId":"1234567890",
      "loanAmount":99999
    }
    """)
    headers {
      header('Content-Type', 'application/vnd.fraud.v1+json')
    }
  }
response {
  status 200
  body("""
  {
    "fraudCheckStatus": "FRAUD",
    "rejectionReason": "Amount too high"
  }
  """)
  headers {
    header('Content-Type': 'application/vnd.fraud.v1+json')
  }
 }
}

cloud.spring.io/spring-cloud-contract/

Messaging Contract:
1. presentations/contracts-streaming-microservices
2. docs.confluent.io/current/schema-registry/docs/maven-plugin.html

Unit Testing Architecture

public class DaoRulesIntegrationTest extends DaoRulesTest {
    @Rule
    public final ExpectedViolation expectedViolation = ExpectedViolation.none();

    @Test
    @Override
    public void DAOs_must_reside_in_a_dao_package() {
        expectedViolation.ofRule("DAOs should reside in a package '..dao..'")
                .by(javaPackageOf(InWrongPackageDao.class).notMatching("..dao.."));

        super.DAOs_must_reside_in_a_dao_package();
    }

    @Test
    @Override
    public void entities_must_reside_in_a_domain_package() {
        expectedViolation.ofRule("Entities should reside in a package '..domain..'")
                .by(javaPackageOf(EntityInWrongPackage.class).notMatching("..domain.."));

        super.entities_must_reside_in_a_domain_package();
    }
}

public class ThirdPartyRulesIntegrationTest extends ThirdPartyRulesTest {
    private static final String RULE_TEXT = "classes should " + THIRD_PARTY_CLASS_RULE_TEXT;

    @Rule
    public final ExpectedViolation expectedViolation = ExpectedViolation.none();

    @Test
    @Override
    public void third_party_class_should_only_be_instantiated_via_workaround() {
        expectedViolation.ofRule(RULE_TEXT)
                .by(callFrom(ClassViolatingThirdPartyRules.class, "illegallyInstantiateThirdPartyClass")
                        .toConstructor(ThirdPartyClassWithProblem.class)
                        .inLine(9))
                .by(callFrom(ClassViolatingThirdPartyRules.class, "illegallyInstantiateThirdPartySubClass")
                        .toConstructor(ThirdPartySubClassWithProblem.class)
                        .inLine(17));

        super.third_party_class_should_only_be_instantiated_via_workaround();
    }
}

Security Visibility: Static Package Scanning

Serverless security

Security Visibility: Dependencies

owasp.org/index.php/OWASP_Dependency_Check

Testing: Core concepts

martinfowler.com/articles/microservice-testing/#testing-progress-3

martinfowler.com/articles/practical-test-pyramid.html

What are we going to Do?

Pipeline to try

Build pipeline in Jenkins.
Replicate in VSTS.
Try reusing lib and play around.

VSTS Pipeline

Understand differences between VSTS and TFS

VSTS and Team Foundation Server (TFS) both provide an integrated, collaborative environment that supports Git, continuous integration, and Agile tools for planning and tracking work.

VSTS is the cloud offering that provides a scalable, reliable, and globally available hosted service. It is backed by a 99.9% SLA, monitored by our 24-7 operations team, and available in local data centers around the world.

Team Foundation Server is the on-premises offering built on a SQL Server back end. Enterprises typically choose on-premises TFS when they need their data to stay within your network, or they want access to SharePoint sites and SQL Server reporting services that integrate with TFS data and tools.

Key feature differences between VSTS and TFS

Even though VSTS is a hosted version of TFS, there are some differences between the features available in the two products. Some TFS features are not supported in VSTS at all—for example, VSTS does not support integration with SharePoint or Project Server.

Continuous Integration pipeline

Create New org
Import a repository :
```
https://github.com/manoharanRajesh/test-ci
```
Setup Build set.
- Add Maven tasks
Enable triggers
Change source and test Build trigger

Continuous Deployment with PCF

Goto Market place and install following Extensions:
- Cloud foundry
- Download and extract
Create service end point with pivotal account detail.
Create Pipeline(release) with following:
- use previous artifices.
- create environment and do following task
  - Download cf cli to the agent
  - deploy artifices to pcf

Create Lib

Group task
Export the task
Import the task

Sample: Build Group Task

PCFDeploy Group Task

https://github.com/manoharanRajesh/cicd/tree/dev/vsts/task_groups

https://api.run.pivotal.io

https://packages.cloudfoundry.org/stable?release=windows64-exe&source=github

Code as Pipeline

Enable preview yml config
Update the yml file in the repo.
- use copy snippet
create new build and release pipeline
select the yml file.

Note:

Only Build task is supported in yml configuration.
About system variable -TODO.
Need to use " / " Escape character.
Some time needs to add "/n".

Jenkins Pipeline

Prerequisite :

Working version of Jenkins.
- Build slaves if required.
- You can use Docker images or OVA virtual box image
Test if you are able to logging, install plugin & right to add build job.

CI/CD Overview & Workshop

What Will I Learn?

Best practices

From monolith to

Gated application deployment Pattern

Independent service deployment pattern

Velocity & Stability

Stability: Testing NFRs in the build pipeline

Talking of (service) contracts…

Talking about (service) contracts…

Security Visibility: Static Package Scanning

Serverless security

Security Visibility: Dependencies

What are we going to Do?

Pipeline to try

VSTS Pipeline

Create Lib

Code as Pipeline

Jenkins Pipeline

Add MultiPipeline job

Update Jenkins file

Copy of Workshop

Copy of Workshop

Rajesh Manoharan

CI/CD Overview & Workshop

What Will I Learn?

Best practices

From monolith to

Gated application deployment Pattern

Independent service deployment pattern

Velocity & Stability

Stability: Testing NFRs in the build pipeline

Talking of (service) contracts…

Talking about (service) contracts…

Security Visibility: Static Package Scanning

Serverless security

Security Visibility: Dependencies

What are we going to Do?

Pipeline to try

VSTS Pipeline

Create Lib

Code as Pipeline

Jenkins Pipeline

Add MultiPipeline job

Update Jenkins file

Copy of Workshop

More from Rajesh Manoharan