Henri Gomez
OSS activist, ASF Member, former Tomcat contributor, founder of JPackage, DevOps Incubator and OBuildFactory projects. Dev, QA and Ops
from CI TO CD&D
LavaJUG*
November 8, 2012
TOC
SOFTWARE CHALLENGES
CONTINUOUS INTEGRATION
CONTINUOUS DEPLOYMENT
NATIVE PACKAGING
ABOUT ME - @HGOMEZ
TODAY - CI Architect @ Axway*
TODAY Software Challenges
Software projects are larger and complex
More and more component interoperability
Many teams and often remote teams
Tight schedules
Frequent needs and requirements changes
EARLY AGES (PREHISTORIC)
Build / Release on a “per request” basis -> GO
Project in proper state -> yes it build
Well documented per standards -> doc is not an option
Properly reviewed -> someone tested product
BUT IT FAILED TOO OFTEN ;(
THEN CAME AGILE
TOOLING NEEDED FOR AGILE
Standardize Build, Test and Deployment
Provide Continuous Builds, Test and Deployment
Ensures solid, reproducible and analyzable steps
Visible to all actors (Dev, QA, Product Owner)
AGILE IDEAL CYCLE :-)
CONTINUOUS INTEGRATION
"Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of integration hell".
GOAL
THE ULTIMATE GOAL IS TO REDUCE TIMELY REWORK AND THUS REDUCE COST AND TIME.
WHEN DONE WELL, CONTINUOUS INTEGRATION HAS BEEN SHOWN TO ACHIEVE THESE GOALS.
WORKLOAD - HUMAN
Team members work on code
Code and Unit tests
Push on SCM after local tests passed
WORKLOAD - MACHINE
POLLS SCM AND CHECK OUT IF CHANGES
BUILD ALL CODE
RUN ALL UNIT TESTS
DEPLOY SNAPSHOTS TO ARTIFACTS REPOSITORY
WORKFLOW
CONTINUOUS INTEGRATION STEPS
Fetch FROM SCM
BUILD CODE
RUN UNIT TESTS
RUN INTEGRATION TESTS
DEPLOY ARTIFACTS
RUN Acceptance tests
PRODUCE REPORTS
NOTES ABOUT TESTS
TESTS SHOULD COVERS A WIDE SCOPE FROM DEV UP TO RUNTIME
UNIT TESTING
INTEGRATION TESTING
ACCEPTANCE TESTING
PERFORMANCES TESTING
UNIT TESTING
HERE WE TEST CODE
Ensures the code is doing what we intend it to do
Many extremely quick and small tests
UT ensure we make the software right
INTEGRATION TESTING
USED TO VALIDATE CONTRACTS BETWEEN PARTICIPANTS
SHOULD BE PORTABLE (NOT PLATFORM DEPENDANTS)
"External Components" SHOULD BE embeddedABLE or mocked
IT ensure we make the right software
ACCEPTANCE TESTING
KEEP THEM SIMPLE
REAL TARGET COMPONENTS SHOULD BE USED
PREFER PRE-REQUISITE (DONT EMBED SQL DB ENGINES)
DATASETS IN USE SHOULD BE MODERATE
PREPARE FOR PRODUCTION - ENROLL YOUR OPS
PERFORMANCE TESTING
Fix quickly performance regressionS
Prepare for Capacity Planning
proTECT YOURSELF FROM IN-PROD DISCOVERY
CONTINUOUS DEPLOYMENT
CONTINUOUS DEPLOYMENT IS CONTINUOUS INTEGRATION ON STEROIDS
BIG PICTURE
WELCOME to Ops LAND
CONTINUOUS DEPLOYMENT FOR ?
YOUR DEVs
YOUR QAs
YOUR SUPPORT
YOUR SALES
AND OF COURSE YOUR CUSTOMERS
ANY MAGIC RECIPES ?
KEEP IT SIMPLE
THINK COMPONENT
ASSEMBLE THEM ALL !
LEGO MODE
Mandate component approach
Assemble Artifacts
Components Assembly guidelines
TAKE Special care ON Components configurations
ASSEMBLE THEM ALL
DEPLOY USABLE ARTIFACTS
READY TO USE BY UPSTREAM TEAMS LIKE QA-OPS
SHOULD BE CONFIGURABLE FROM THE OUTSIDE
HEY, DON'T FORGET
CONTINUOUS DELIVERY !
THERE IS CONTINUOUS DEPLOYMENT
BUT DON't FORGET CONTINUOUS DELIVERY
DELIVERY
ProvidE product to end user
User could then install or update it on his own systems
DEPLOYMENT
PROVIDE OPERATIONAL SOLUTION TO END USER
USER COULD DIRECTLY USE IT ON HIS OWN OR LEASED SYSTEMS
EXTERNAL APPROACH
Many good providers, Cloud Based
Cloud bees*
heroku*
openshift*
BUT you MAY don't have choiceS and HAD do it IN-HOUSE
In-HOUSE APPROACH
Materialize 3 chains
SOFTWARE FACTORY
QUALITY FACTORY
DELIVERY&DEPLOYMENT FACTORY
AND DONT FORGET SERIOUS MONITORING FOR ALL OF THEM
SOFTWARE FACTORY
FROM SCM TO ARTIFACTS
BUILD
UNIT TESTS
INTEGRATION TESTS
CODE QA REPORTS (SONAR)
DEPLOY ARTIFACTS
QUALITY FACTORY
FROM ARTIFACTS TO RUNTIME REPORTS
CONSUME ARTIFACTS FROM SOFTWARE FACTORY
RUN ACCEPTANCE TESTS
PERFORM PERFORMANCE TESTS
PROVISION CAPACITY PLANNING
REPORTS GENERATION
DELIVERY&DEPLOYMENT FACTORY
FROM ARTIFACTS TO RUNNABLE
BUILD NATIVE PACKAGES
BUILD IMAGES (VMDK, AMI, DVD ISO)
PROVIDE RUN INFRASTRUCTURE (INSITE)
INTERFACE TO RUN INFRASTRUCTURE (OFF-SITE/PUBLIC)
CATALOG & PROVISION RUNTIMES
INTERACTIONS BETWEEN FACTORIES
INTERACTIONS
SWF PRODUCES SOFTWARE MATERIALS
QAF CONSUME SWF MATERIALS
QAF GIVE 'GO FOR PRODUCTION' IN D&DF
D&DF PROVIDES QAF RUN INFRASTRUCTURE
D&DF COULD PROVIDES DEV RUN INFRASTRUCTURE
TOOLING
SCM : SUBVERSIOn, GIT, MERCURIAL (CVS sic)
ORCHESTRATiON : JENKINS* (What else ?)
CONTENT REPOSITORIES : ARCHIVA*, ARTIFACTORY*, NEXUS*, HTTPD*
QUALITY : SONAR*
RUN FORCE : JENKINS AGENTS
RUN MIDDLEWARE : ESXi*, VirtualBOX*, VAGRANT*, OPENSTACK*/CLOUDSTACK*
MONITORING : Zabbix*, GRAPHITE*, COLLECTD*, JMXTRANS*
NATIVE PACKAGING
Widely used in Unix world
OS Componentisation approach
Runtime dependencies aware
POWERFUL life-cycle support
Web based packages repositories for modern delivery
Widely used in Unix world
YOUR OPS KNOWS THEM
YOUR CUSTOMER OPS KNOWS THEM
AVOID "NOT INVENTED HERE SYNDROM", BRING MORE CONFIDENCE
OS Componentisation approach
YOUR APP SHOULD BE A COMPONENT LIKE THOSE BUNDLED BY OS
EASY TO MANAGE AS SYSTEM SERVICE
TAKE CARE OF SYSTEM HABBIT (USER BY SERVICE)
Runtime dependencies aware
PROVIDES OR DEPENDS ON OTHERS OS PACKAGES
HELP SPLIT COMPLEX SOLUTIONS IN SET OF SIMPLE PACKAGES
INJECT SQL ENGINES, FRONT-ENDs CONFIGURATION VIA HOOKS
PUPPET/CHEF MANAGEABLE
Powerfull life-cycle support
EASY INSTALLATION
EASY UPDATE
EASY REMOVAL
INTERACT WITH OTHER COMPONENTS LIFE-CYCLE
modern delivery
YUM, ZYPPER, APT ARE USING Web based packages repositories
BUILD / PROVIDE YOUR OWN COMPANY WEB REPOSITORIES
SAME SOURCE FOR YOUR DEV, QA, SUPPORT, SALES and CUSTOMERS
100% CLOUD-READY WITHOUT EFFORT
SMART CD WITH JENKINS & RPM
- BUILD (maven)
- PACKAGE (RPMBUILD)
- UPDATE REPO
- DEPLOY !
DEPLOYMENT STEPS IN PIPELINE VIEW
DEMONSTRATION AT VIMEO
IN A NUTSHELL
CONTINUOUS INTEGRATION MOTORIZE AGILITY
THEN CAME CONTINUOUS DELIVERY AND DEPLOYMENT
CONTINUOUS D&D COULD BE DONE IN HOUSE
CONSIDER NATIVE PACKAGING FOR Continuous DEPLOYMENT
from ci to cd
By Henri Gomez
