Engineering@Nuxeo
Sneak peak in the sausage factory

Thierry Delprat
tdelprat@nuxeo.com
https://github.com/tiry/
Nuxeo Engineering Team
People behind the code

1 team - 360 view
the Nuxeo Engineering Team handles
-
development of new features in the product
-
design, development, testing
-
design, development, testing
-
customer support
-
reproduce, investigate, fix
-
reproduce, investigate, fix
-
consulting
- design workshops, training, project bootstrap ...
get feedback from customers
efficient support for
customer developers
always up-to-date with the platform
Distributed team
-
18 engineers in France
- 14 engineers in Paris
- 3 engineers in Lille
-
1 engineer in Lyon
-
7 engineers in the US
- 4 engineers in NYC
- 3 engineers in California
-
8 engineers in EU
- 7 engineers in Portugal
-
1 engineer in Germany
-
Product Team in Paris
- 1 Product Manager
- 1 Developer advocate
- 2 Technical writers
- 1 Technical consultant
Specialized SQUADS
-
dedicated support team
- 2 engineers in Paris
- 1 engineer in California
- 1 engineer in Iowa
-
consulting team
- 1 project manager in Paris
- 1 project manager in NYC
- 1 dedicated engineer in Paris
- all US engineers are mixed between Product/Project
Nuxeo Developer profile
-
Experienced and passionate developers
-
they want to do quality code
-
they will complain if something reduces quality
-
-
Expectations are high
-
Best practices to ensure QA and maintainability
-
Coding rules, Issues Management, CI, Code review
-
-
Doing bad code has direct consequences
-
Other developers get angry
-
Support duty becomes a burden
-
-
they defend the QA proces
Developers work hard to learn and adapt or they leave quickly
Nuxeo DeveloperS
- Integrate new developers
-
6 new developers in the last 12 months
-
6 new developers in the last 12 months
-
Keep the good developers
-
in average developers are with us for 5+ years
-
-
Keep growing expertise
-
our 8 top experts are with us for 10+ years
-
Work Organization
- most developers are grouped in 4 teams
- work on related tasks in iterations
-
Scrum boards (in JIRA)
-
some experts developers work alone
-
focus on complex technical tasks
-
Kanban style or Scrum board (Jira)
-
- Sprint Planning / Sprint review every 2 or 3 weeks
- Backlogs filled conjointly
- Product Manager & Head of Engineering
- Head of Support
- CTO
Technical coordination: Techleads
- Technical leadership
- each component of the platform has an owner
-
each team as a Technical Lead
- Technical leads
- are responsible for proposing design
-
are responsible for raising issues
- Techleads meeting every 2 weeks
- discuss issues and define solutions
- prioritize infrastructure tasks (technical debt)
- feed backlogs
No StupiP coding
- Developers are responsible
- for good design
- for QA
- for performances
-
for handing support
- Techleads are here to help
- review design
- review code
- discuss performances
Technical CommunicaTION
- Standup meeting 3 times a week
-
share day to day tasks and issues
-
share day to day tasks and issues
- Slack Channels
- discuss about issues
- share new ideas or technologies
-
intra-team channels
- Techleads reports
- share ongoing tasks, issues & concerns
-
share knowledge
Coding @ Nuxeo
Principles and Rules

QA & Testing part of our DNA
-
TDD by nature
- we were already doing TDD in 2005
-
we have build a versatile Unit Testing Framework
- no excuse
-
Testing at all levels
- Unit tests
- Functional testing
- Performances testing
-
CI supports the testing work
- each developer has his own Test&Push job
- Jenkins runs the performances and functional tests
- Jenkins runs the matrix testing (OS / DB)
History and Code Tracking
-
Task tracking
- All issues are created in JIRA
- Issues are assigned to Iteration & Agile Boards
- Status indicate progress
-
Tasks links associated commits
-
Commit History
-
Every single commit is associated with a Jira Issue
- tracability (support)
-
Git History is clean
- re-base policy
- feature development is done in separated branches
-
Every single commit is associated with a Jira Issue
Coding Rules & Best Practices
-
Nuxeo Core Developer Guide is public
-
This guide includes
-
guidelines for using all the tools
-
Maven, Git, Npm, Bower ...
-
-
coding rules
-
guidelines for avoiding security issues
-
-
Code analysis
-
For now the most efficient remains Code review by peers
-
we have a lot of experienced developers
-
a lot of them review commits and raise hand when something is wrong
-
-
CI & DEvops
-
Jenkins is used extensively
- JUnit, WebDriver & Selenium testing
- Gatling & FunkLoad testing
- Packaging (jar, Nuxeo Packages, Distributions ...)
-
Deployment (Maven, Marketplace, Connect, Production)
-
DevOps teams helps maintaining the process
- Sys-admin + developer skills (3 persons)
CI Role

some Figures
- 2+ Million lines of code
- 15,000 commits
- 500+ Jenkins Jobs declared
- 3,000+ build per 24h
- 500,000 tests run per 24h
- Standard Build = 5,500 unit tests + 6,000 functional tests
- 15,000 NXP Tasks completed
- 15,000 SUPNXP Issues solved

Source Code Management
Releases cycle, Feature branches, HotFixes ...



Release a new FastTrack every 2 to 3 months

Release a new FastTrack every 2 to 3 months
Give early access to new features for ongoing projects

4 to 5 FastTrack releases per year

Release LTS once a year

LTS Long Term Support

LTS Maintenance Branch

New Features are developed
in isolated "feature branch"
we can work on several features at the same time without any interactions

CI does continuous testing of the branch
detect issues early

when ready changes are integrated in a
Pull Request


Pull Request is reviewed by peers


when changes are accepted
the new feature is merged
feature branch
is automatically deleted


Fixes are done in a branch too

Fixes benefit from CI Services too
including PR & Review

Fixes may be back-ported to LTS
(Cherry pick)

Backported-Fixes are tested

Backported-Fixes
are reviewed

Backported-Fixes
are reviewed

Backported-Fixes
eventually become
Hot Fixes packages
About HotFixes
Released every 2 to 4 weeks for each of the 3 LTS
- quickly integrate fixes required by customers
- unlock new configurations option when needed
- make LTS benefit from performances improvements
34 HotFixes released in 2015
- LTS 2015 - 2 (LTS is released in October)
- LTS 2014 - 21
- LTS 2013 - 9
- LTS 2012 - 2
Branches & dev cycles
By Thierry Delprat
Branches & dev cycles
- 2,240