Let's talk about Workflow !

Sylvain Fargier, CERN

07/11/2023

Introduction

Introduction

Introduction

• About the presentation, 3 parts:
  • When you should take the initiative to talk about workflow
  • Why it should matter to you as a software developer
  • How you can implement it

When ?

When should you take the initiative to talk about workflow

When ?

• When to talk about workflow and methodology ?

• If your project can fit one of those:

When ?

• Then it's already late !

• If you hear the following :

When ?

• Then someone missed the point, this is your workflow, it needs to be tailored to your need !

• Final answer :
  • It's really important to head-up from time to time !
  • You're the best fitted to know how you should work, advocating for a sane workflow and methodology is your responsibility
  • You're not being asked to "piss lines code" but to build and implement robust features in products
• And side info, I've really heard all those sentences at least once !

When ?

Why ?

Why should it matter to you as a software developer

• Why should you worry about your workflow ?
• For any of those reasons (amongst others):
  • You are concerned about maintainability, robustness or quality.
  • You work in a team and efficiency matters.
  • You prefer to spend time improving your workflow rather than losing it on repetitive tasks.
  • You may be managed by non-software-developer, so the initiative is yours

Why ?

• A workflow is: how you get work done

• Always Keep It Simple Stupid (KISS), It must be easy to understand and jump-in for anyone !

• You can call it Agile, Scrum, XP, LEAN or whatever, it has to be yours and tailored for your need !

Why ?

• A workflow to get better code-quality, through :
  • Style : fasten readability and reuse
  • Lint : detect miss-constructs and errors with a first glance
  • Unit-Tests ("White-box testing") : ensure it does what it's meant for

Why ?

• That's not sufficient, enforce this through :
  • documentation : explain tricky parts, make your work nice to users
    • even if you're your own user !
  • reviews : external overview of your work, ensures it's readable and neat
    • must always be with good intent !
  • Test Driven Development (TDD) :
    • Tests are painful, you'd rather write it first

Why ?

• A workflow to get better maintainability, through :
  • Modularity : enforces code reuse
    • you've spent time writing clean high-quality code, let's maximize its reuse !
    • Always keep in mind: KISS
  • Continuous Integration (CI)
    • build to detect environmental changes asap
    • ensures your code-quality criteria are respected
    • Makes your efforts visible to others

Why ?

• A workflow to get better product-quality, through :
  • Integration tests ("gray-box testing") : 
    • Ensures everything works altogether, focus on communication and core features
  • Functional tests ("black-box testing") :
    • Ensures features and product is behaving as expected

Why ?

• A workflow to release the pressure, through :
  • Continuous Delivery (CD) :
    • Separate devs from admins
    • Any dev can make it to production not having to understand the full stack

Why ?

Style, Lint, (type)check : because you like it shinny

Documentation and KISS : because you like it crystal-clear

Unit-/Integration-/Functional-Tests : because you like when it works

CI/CD : because you like when it rolls on its own (and you're a lazy cat)

All this because it needs to scale

Conclusion

How ?

How can you implement it

• This is not the answer on "how to implement your workflow" this is our custom solution
  • take it as a simple example !
• It is initially based on Scrum with XP practices but is evolving depending on the team's needs
• Your workflow doesn't have to stick with literature, take the best, drop the rest and make it tailored to your need !

Introduction

• Homogeneous
  • On GitLab
  • CI and tools rules
  • Containerized build environment
  • Directory layout, naming, styling ...
  • Consistent techs and frameworks
• Documented
  • templates & boilerplates (cpp/web)
  • guidelines & readmes
  • playgrounds & live-documentation
  • Formal documentation streams and tooling

Projects & Repositories

• Backlog and sprints
  • Jira board to aggregate several projects
  • Using per-project releases
  • Using a planning-poker
• KanBan board (see Jira board above)
• Daily-Scrum
  • On Mattermost In addition to on-site stand-up dailies (on Monday)
• Homogeneous workflow rules
  • Large number of repositories and users
  • Enforced using automated tools

Project management

• Peer-review oriented workflow
  • All work must be reviewed before being merged
  • Using CERN/GitLab merge-requests
  • Relying on CI tests and analysis results
• Per component maintainers
  • To empower and give sense of responsibility to developers
• Spikes and brainstorming
  • When you don't know yet how to achieve something, a white-board is essential !

Team work

• Weekly code-reviews
  • Show your work to others
  • Open discussions and exchanges
• Weekly builds
  • Rebuild and validate everything
  • Dependencies, platforms, environment evolves
  • "Better fix sooner than later"
• Pair coding and reviews
  • Since software is all about learning

Quality & reviews

• Automated deployments
  • For web-applications using CERN/OpenShift
  • For middleware using packages (deb, RPMs) and Puppet
• Roles separation
  • Developers are not admins but can trigger software deployment
• Traceability and reproducibility
  • Version tag based (v1.x.x)
  • Deployed from versioned build environments

Deployments & Production

Example deployments 1/2

Example deployments 2/2

Example pipeline: libccut

MRO applications
(TIM, RadMon, SSVG...)

SAMbuCa & SCT functional test-benchs (937 motors-lab)

• Setting a workflow up and writing tests may feel overwhelming
  • But you'd rather start slow and reach a fast pace than the opposite
  • How many teams gets stale being overwhelmed by maintenance and repetitive tasks
  • Unit-tests represents about 50% of the development time, but it saves way more on the maintenance and fixes
  • technical debt and cost of software errors is hard to measure, but all this is to keep it low

Conclusion 1/2

• Pickup your tools and make a workflow tailored to your needs
  • You can rely on existing methodologies and adapt or mix depending on your experience and projects
  • Even being a lone developer that will secure your job and make your life easier
• Securing your production path is vital
  • there's no context where it wouldn't apply (being embedded or tiny is not an excuse)
• (Almost) everybody can write code, but deploying and maintaining it at large scale is more of an art

Conclusion 2/2

Questions ?