Henri Gomez
OSS activist, ASF Member, former Tomcat contributor, founder of JPackage, DevOps Incubator and OBuildFactory projects. Dev, QA and Ops
introduction to continuous integration and delivery
OBJECTIVES
Continuous integration and Delivery are one of the most important aspects of Agile Software Development.
In this introduction course, you will learn the concepts, why CI/CD is important and how to set up and use the various tools.
This course will be based on tooling used in Axway such as Jenkins, Subversion, Artifactory, Nexus, Sonar.
TOC
Axway Enterprise Continuous Delivery
Continuous Integration
Some CIs @ Axway
Source Code Management (SCM)
Jenkins
Sonar
Artifacts Repositories
Continuous Delivery
Axway
Enterprise Continuous Delivery
Introduction TO ECD program
A global program within Axway R&D organization
Provide Software Factory, Services & Product Delivery
Support Project team & Axway Development Methodology (ADM)
WHY ECD ?
Improve Software Products Quality
Improve time to market and efficiency
Standardize Infrastructure, tooling, processes
Provide off the shelf Software delivery infrastructure & services
ECD SCOPE
Software Factory (Continuous Integration/Delivery)
Software Delivery (Installer, ESD, ISO, AMI, PXE, …)
Global CI support and expertise
ECD GLOBAL CI
3 MAIN ECD TEAMS :
D2D, SYNCHRONY & SOFIA CI
EMEA AND USA ZONES
WORLD-WIDE SUPPORT (TIME-ZONE)
ECD PRACTICES & KNOWLEDGE SHARING
ECD Global support
ECD Maturity Dashboard
How mature are we from an ECD stand point ?
Where can we get the most improvement ?
How are other projects solving these same problems?
ECD MATURITY OBJECTIVES
Provide visibility on Project/Product CD maturity
Provide feedback to Project team and CI for improvement
ECD MATURITY CATEGORIES
SW Factory : Quality, Reliability, Scalability, Trust on CI infra
Building : Software Compilation Best Practices Compliance
Testing : Tests Coverage & Automation (unit, system)
Releasing : Smart, Trustable and Replicable Process
Deploying : Automation and availability
Reporting : Monitoring and visibility
MATURITY LEVELS
- Minimal: Grade 1-2 based on criteria evaluation
- Beginner: Grade 3-4 based on criteria evaluation
- Intermediate: Grade 5-6 based on criteria evaluation
- Advanced: Grade 7-8 based on criteria evaluation
- Extreme: Grade 9-10 based on criteria evaluation
MATURITY MATRIX
MATURITY RESULTS
Continuous Integration
challenges of Modern Software
SOFTWARE PROJECTS ARE LARGER AND COMPLEX
MORE AND MORE COMPONENT INTEROPERABILITY
MANY TEAMS AND OFTEN REMOTE TEAMS
TIGHT SCHEDULES
FREQUENT NEEDS AND REQUIREMENTS CHANGES
EARLY AGES (PREHISTORIC)
BUILD / RELEASE ON A “PER REQUEST” BASIS -> GO
PROJECT IN PROPER STATE -> YES IT BUILD
WELL DOCUMENTED PER STANDARDS -> DOC IS NOT AN OPTION
PROPERLY REVIEWED -> SOMEONE TESTED PRODUCT
BUT IT FAILED TOO OFTEN ;(
THEN CAME AGILE
TOOLING NEEDED FOR AGILE
STANDARDIZE BUILD, TEST AND DEPLOYMENT
PROVIDE CONTINUOUS BUILDS, TEST AND DEPLOYMENT
ENSURES SOLID, REPRODUCIBLE AND ANALYZABLE STEPS
VISIBLE TO ALL ACTORS (DEV, QA, PRODUCT OWNER)
CI system for and who does it serve
DEV TEAMS
QA TEAMS
OPS TEAMS
Moving to Continuous Integration
SOURCES SHOULD BE IN SCM
BUILDS SHOULD BE FULLY AUTOMATED
BINARY ARTIFACTS SHOULD BE EXPORTABLE
TESTS SHOULD BE FULLY AUTOMATED
business case for Continuous Integration
DEV TEAMS WORKS ON CODE
CI BUILD, TEST AND PRODUCE BINARIES
QA TEAMS USE BINARIES
QA TEAMS VALIDATE BINARIES AND PROVIDE GO
The basic CI flow
CONTINUOUS INTEGRATION STEPS
FETCH FROM SCM
BUILD CODE
RUN UNIT & INTEGRATION TESTS
DEPLOY ARTIFACTS
RUN ACCEPTANCE & PERFORMANCES TESTS
PRODUCE REPORTS
TYPICal STEPS ON D2D
NOTES ABOUT TESTS
Tests should covers A WIDE SCOPE FROM Dev UP TO Runtime
UNIT TESTING
INTEGRATION TESTING
ACCEPTANCE TESTING
PERFORMANCE TESTING
UNIT TESTING
HERE WE TEST CODE
ENSURES THE CODE IS DOING WHAT WE INTEND IT TO DO
MANY EXTREMELY QUICK AND SMALL TESTS
UT ENSURE WE MAKE THE SOFTWARE RIGHT
INTEGRATION TESTING
USED TO VALIDATE CONTRACTS BETWEEN PARTICIPANTS
SHOULD BE PORTABLE (NOT PLATFORM DEPENDANTS)
"EXTERNAL COMPONENTS" SHOULD LIGHTLY EMBEDDEDABLE OR MOCKED
IT ENSURE WE MAKE THE RIGHT SOFTWARE
ACCEPTANCE TESTING
KEEP THEM SIMPLE
REAL TARGET COMPONENTS SHOULD BE USED
PREFER PRE-REQUISITE (DONT EMBED SQL DB ENGINES)
DATASETS IN USE SHOULD BE MODERATE
PREPARE FOR PRODUCTION - ENROLL YOUR OPS
PERFORMANCE TESTING
FIX QUICKLY PERFORMANCE REGRESSIONS
PREPARE FOR CAPACITY PLANNING
PROTECT YOURSELF FROM IN-PROD DISCOVERY
ECD FACTORIES
SOFTWARE FACTORY
QUALITY FACTORY
DELIVERY&DEPLOYMENT FACTORY
SOFTWARE FACTORY
FETCH FROM SCM
BUILD CODE
RUN UNIT & INTEGRATION TESTS
DEPLOY ARTIFACTS
QUALITY FACTORY
RUN ACCEPTANCE & PERFORMANCES TESTS
CAPACITY PLANNING
PRODUCE REPORTS
DELIVERY & DEPLOYMENT FACTORY
ENTERING CUSTOMER LAND
OPS MINDED
DELIVERY
PROVIDE PRODUCT TO END USER
USER COULD THEN INSTALL OR UPDATE IT ON HIS OWN SYSTEMS
DEPLOYMENT
PROVIDE OPERATIONAL SOLUTION TO END USER
USER COULD DIRECTLY USE IT ON HIS OWN OR LEASED SYSTEMS
Some CI @ Axway
Synchrony CI
D2D CI
Synchrony CI
BUCHAREST BASED Team
SYNCHRONY CI MISSIONS
IN CHARGE OF SYNCHRONY SUITE PRODUCTS
MAINLY LEGACY/ NATIVE CODE BASED
LARGE SET OF OS AND HARDWARE TO BE SUPPORTED
SYNCHRONY CI INFRASTRUCTURE MANAGED BY IT
D2D
FORMER DARWINCI
D2D STANDS FOR 'Development to Delivery'
Annecy based Team (BEARS TEAM)
D2D MISSIONS
IN CHARGE OF FTD anD CG PRODUCTS
R&D & METHODS ON CI/CD
PLATFORMS MAINLY LINUX AND WINDOWS
D2D INFRASTRUCTURE MANAGED BY D2D TEAM
D2D MAIN ACTIVITIES
Source Code Management (SCM)
QUICK LOOK ON SCM
SUBVERSION
open source version control system
centralized version control
STRONG AND ROBUST
AVAILABLE IN ALL IDE
ENGINEERING RULES
ALL SOURCES SHOULD LIVE IN SCM
COMMIT OFTEN
TAG OFTEN
NO RELEASE WITHOUT ASSOCIATED TAG
SCM++ - DVCS*
decentralized version control
peer-to-peer approach to version control
richer branch/revision features
LOCAL & REMOTE COMMITS
VERY WELL SUITED FOR DISTRIBUTED DEVELOPMENTS
REQUIRE SOLID TRAINING FOR BOTH USERS AND ADMINS
MOST USED DVCS
Jenkins
CI/CD ORCHESTRATOR
OVERVIEW
Jenkins* is AN INTEGRATION SERVER
BUT MUCH more than just a build engine
FREE AND OPEN SOURCE
Features
RICH ECOSYSTEM (700+ PLUGINS)
NATIVE SUPPORT FOR ANT, MAVEN, UNIT TESTING,...
MAJOR SCM/DVCS SUPPORTED, CVS, SVN, GIT, MERCURIAL
POWERFUL SCHEDULER (not just cron)
Distributed builds
VIRTUALISATION AWARE (VB, VMWARE, EC2, CLOUD)
...
Architecture
Dashboard
Jobs
Project page
PROJECT SIDE PANEL
Project relationship
Plugins
Running Automated tests
Tooling Integration
Sonar
YES, Code Quality MATTER
Code Quality
FROM SONARSOURCE* :
"Improve software quality to increase the efficiency of your development teams and the longevity of your application with our unique Continuous Inspection approach.'
Overview
open platform to manage code quality
IT covers the 7 axes of code quality
AND OF COURSE FREE AND OPENSOURCE
Features
Rules, alerts, thresholds, exclusions configured online
JAVA, C, C#, Flex, Natural, PHP, PL/SQL, Cobol and Visual Basic 6 COVERAGE
PLUGIN BASED FOR MORE FLEXIBILITY AND FEATURES
CONFIGURABLE DASHBOARDS & VIEWS
dashboard, TimeMachine and defect hunting tools
DASHBOARD
PROJECT VIEW
PROJECTS DETAILED METRICS
CONFIGURABLE RULES
Plugins
Artifacts Repositories
SCM STORE SOURCES, ARTIFACTS REPOSITORIES HANDLE BINARIES
DEFINITION
INITIALLY FOR MAVEN USAGE :
'A Maven repository provides a platform for the storage, retrieval, and management of binary software artifacts and metadata' *
THERE ARE NO MORE RESTRICTED TO MAVEN CONTENTS
ARTIFACTS REPOSITORIES SUPPORT ANT/IVY, GROOVY, P2/OBR, YUM/ZYPPER
APACHE ARCHIVA
Apache Archiva* WAS one of the FIRST IMPLEMENTATION
WAS A BIT SLEEPING BUT NOW VERY ACTIVELY DEVELOPED
VERY FAST ENGINE (2.5K req/s)
MODERN UI SINCE 1.4-M3
MAVEN REPOSITORY FEATURES
Free and OPENSOURCE
JFrog Artifactory
JFROG ARTIFACTORY* IS A LONG TIME PLAYER IN REPOSITORIES ARENA
MORE THAN JUST MAVEN WITH FEATURES LIKE P2, YUM, PROVISIONING
RICH API ALLOWING CUSTOM PLUGINS DEVELOPMENT IN GROOVY
BEING JCR BASED, IT's TRICKY TO TUNE IT FINELY
NOT VERY THE FASTER IMPLEMENTATION AROUND (500 REQ/S)
PUSH/PULL REPLICATION MODES
OSS AND COMMERCIAL VERSION
SONATYPE NEXUS
SONATYPE NEXUS* IS PROBABLY THE MOST WIDELY USED (maven central, jboss...)
MORE THAN JUST MAVEN WITH NUGET, p2, INSIGHT, YUM, PROMOTION...
NO JCR OR SQL REQUIRED, STOCK FILE SYSTEM BASED
PRETTY FAST (1.5-2k REQ/S)
OSS AND COMMERCIAL VERSIONS
Repositories & Artifacts utilization
THIRDPTY CACHING* TO LOWER BUILD TIME (AND SAVE BANDWIDTH)
STORAGE OF AXWAY ARTIFACTS PRODUCTION
PROMOTION mECANISMS ENABLER
LICENCES TRACKING AND REPORTING (ARTIFACTORY)
THIRDPTY SECURITY AND VULNERABILITY CHECK (NEXUS)
Artifacts replication
MIRROR MODE
PUSH/PULL
Build
KEEP LINKS BETWEEN YOUR JOBS AND ARTIFACTS
Using The UI
MAIN WINDOW
SEARCH ARTIFACT
Continuous D&D
GOALS
CONTINUOUS DELIVERY AND DEPLOYMENT FOR :
DEVS, QAS, SUPPORT, SALES
AND OF COURSE CUSTOMERS
CONTINUOUS Delivery
PROVIDE PRODUCT TO END USER
USER COULD THEN INSTALL OR UPDATE IT ON HIS OWN SYSTEMS
CONTINUOUS DEPLOYMENT
PROVIDE OPERATIONAL SOLUTION TO END USER
USER COULD DIRECTLY USE IT ON HIS OWN OR LEASED SYSTEMS
Think Cloud
CLOUD OPERATIONS ARE USING LINUX IN 99,9% CASE
LINUX HAS POWERFUL NATIVE PROVISIONING MECANISMS
TO BE FAST, ONLY NEW/UPDATED CONTENTS SHOULD BE PROVIDED
Smart ways
ALWAYS PREFER NATIVE PACKAGES
SAME SOURCES FOR DEV, QA, SUPPORT AND CUSTOMERS
PACKAGES REPOSITORIES FEEDED IN CONTINUOUS
HEY, DO YOU WANT A DEMO ?
SMART WAY SAMPLE IN 1-2-3-4
- BUILD A MAVEN
- PACKAGE IT IN RPM
- UPDATE RPM REPO
- DEPLOY ON DEST SYSTEM
AND USING JENKINS...
IN A NUTSHELL
AXWAY ENTERPRISE CONTINUOUS DELIVERY
PROGRAM COVERING CONTINUOUS INTEGRATION UP TO DELIVERY
USING STATE OF ART TOOLING TO MOTORIZE AGILITY
ECD IS MORE THAN JUST BUILD
METHODS & QUALITY MATTER
introduction to continuous integration and delivery
By Henri Gomez
