Engineering@Nuxeo

Sneak peak in the sausage factory

 

 

Thierry Delprat
tdelprat@nuxeo.com

https://github.com/tiry/

Nuxeo Engineering Team

People behind the code

1 team - 360 view

the Nuxeo Engineering Team handles
 

  • development of new features in the product
    • ​design, development, testing
       
  • customer support
    • reproduce, investigate, fix
       
  • consulting
    • ​design workshops, training, project bootstrap ...

get feedback from customers

efficient support for
customer developers

always up-to-date with the platform

Distributed team

  • 18 engineers in France
    • 14 engineers in Paris
    • 3 engineers in Lille
    • 1 engineer in Lyon
       
  • 7 engineers in the US
    • 4 engineers in NYC
    • 3 engineers in California
  • 8 engineers in EU
    • 7 engineers in Portugal
    • 1 engineer in Germany
       
  • Product Team in Paris
    • 1 Product Manager
    • 1 Developer advocate
    • 2 Technical writers
    • 1 Technical consultant

Specialized SQUADS

  • dedicated support team
    • 2 engineers in Paris
    • 1 engineer in California
    • 1 engineer in Iowa
       
  • consulting team
    • 1 project manager in Paris
    • 1 project manager in NYC
    • 1 dedicated engineer in Paris
    • all US engineers are mixed between Product/Project

Nuxeo Developer profile

  • Experienced and passionate developers

    • they want to do quality code

    • they will complain if something reduces quality

  • Expectations are high

    • Best practices to ensure QA and maintainability

      • Coding rules, Issues Management, CI, Code review

    • Doing bad code has direct consequences

      • Other developers get angry

      • Support duty  becomes a burden

they defend the QA proces

Developers work hard to learn and adapt or they leave quickly

Nuxeo DeveloperS

  • Integrate new developers 
    • 6 new developers in the last 12 months
       
  • ​Keep the good developers

    • in average developers are with us for 5+ years
       

  • ​Keep growing expertise

    • ​our 8 top experts are with us for 10+ years

Work Organization

  • most developers are grouped in 4 teams
    • work on related tasks in iterations
    • Scrum boards (in JIRA)
       
  • some experts developers work alone

    • focus on complex technical tasks

    • Kanban style or Scrum board (Jira)
       

  • Sprint Planning / Sprint review every 2 or 3 weeks
     
  • Backlogs filled conjointly 
    • Product Manager & Head of Engineering
    • Head of Support
    • CTO

Technical coordination: Techleads

  • Technical leadership
    • each component of the platform has an owner
    • each team as a Technical Lead
       
  • Technical leads
    • are responsible for proposing design
    • are responsible for raising issues
       
  • Techleads meeting every 2 weeks
    • discuss issues and define solutions
    • prioritize infrastructure tasks (technical debt)
    • feed backlogs

No StupiP coding

  • Developers are responsible
    • for good design
    • for QA
    • for performances
    • for handing support
       
  • Techleads are here to help
    • review design
    • review code
    • discuss performances

Technical CommunicaTION

  • Standup meeting 3 times a week
    • share day to day tasks and issues
       
  • Slack Channels
    • discuss about issues
    • share new ideas or technologies
    • intra-team channels
       
  • Techleads reports
    • share ongoing tasks, issues & concerns
    • share knowledge
       

Coding @ Nuxeo

Principles and Rules

QA & Testing part of our DNA

  • TDD by nature
    • we were already doing TDD in 2005
    • we have build a versatile Unit Testing Framework
      • ​no excuse
  • ​Testing at all levels
    • ​Unit tests
    • Functional testing
    • Performances testing
  • ​CI supports the testing work
    • ​each developer has his own Test&Push job
    • Jenkins runs the performances and functional tests
    • Jenkins runs the matrix testing (OS / DB)​

History and Code Tracking

  • Task tracking
    • All issues are created in JIRA
    • Issues are assigned to Iteration & Agile Boards
    • Status indicate progress
    • Tasks links associated commits
       
  • ​Commit History
    • Every single commit is associated with a Jira Issue
      • ​tracability (support)
    • Git History is clean
      • re-base policy
      • feature development is done in separated branches

Coding Rules & Best Practices

  • Nuxeo Core Developer Guide is public

  • This guide includes

    • guidelines for using all the tools

      • ​Maven, Git, Npm, Bower ...

    • coding rules

    • guidelines for avoiding security issues

  • Code analysis

    • For now the most efficient remains Code review by peers

      • we have a lot of experienced developers

      • a lot of them review commits and raise hand when something is wrong

CI & DEvops

  • Jenkins is used extensively
    • JUnit, WebDriver & Selenium testing
    • Gatling & FunkLoad testing
    • Packaging (jar, Nuxeo Packages, Distributions ...)
    • Deployment (Maven, Marketplace, Connect, Production)
       
  • ​DevOps teams helps maintaining the process
    • ​Sys-admin + developer skills (3 persons)

CI Role

some Figures

  • 2+ Million lines of code
  • 15,000 commits
     
  • 500+ Jenkins Jobs declared
  • ​3,000+ build per 24h
  • 500,000 tests run per 24h
     
  • Standard Build = 5,500 unit tests + 6,000 functional tests
     
  • 15,000 NXP Tasks completed
  • 15,000 SUPNXP Issues solved

Source Code Management

Releases cycle, Feature branches, HotFixes ...

Release a new FastTrack every 2 to 3 months

Release a new FastTrack every 2 to 3 months

Give early access to new features for ongoing projects

4 to 5 FastTrack releases per year

Release LTS once a year

LTS          Long Term Support

LTS          Maintenance Branch

New Features are developed

in isolated "feature branch"

we can work on several features at the same time without any interactions

CI does continuous testing of the branch
detect issues early

when ready changes are integrated in a
Pull Request

Pull Request is reviewed by peers

when changes are accepted
the new feature is merged
 

feature branch
is automatically deleted

Fixes are done in a branch too

Fixes benefit from CI Services too
including PR & Review

Fixes may be back-ported to LTS
(Cherry pick)

Backported-Fixes are tested

Backported-Fixes
are reviewed

Backported-Fixes
are reviewed

Backported-Fixes
eventually become
Hot Fixes packages

About HotFixes

Released every 2 to 4 weeks for each of the 3 LTS

  • quickly integrate fixes required by customers
  • unlock new configurations option when needed
  • make LTS benefit from performances improvements

34 HotFixes released in 2015

  • LTS 2015 - 2   (LTS is released in October)
  • LTS 2014 - 21
  • LTS 2013 - 9
  • LTS 2012 - 2