Package Quality Assurance

Sergey Dolganov @ Evil Martians

FOSDEM '18. Package Management Track

Problem definition

How do people think about open source?

Probe Downloads Projects Contributors
Total 5B 124K 90K
99% 0.5B 123K 70K
90% 0.1B 112K 40K

Quality problem in open-source

  • High demand

  • Lack of participants

  • Risk of enterprise failures

  • High pressure over maintainers

Happy Accident

Ossert open-source quality experiment

  • Intention

  • Success

  • Failure

Demo

Current issues in open source

  • Lack of measurable quality definition

  • Lack of risks model

  • Lack of actionable model

What was known before I start?

Existing research

  • Mostly outdated *

  • Too specific **

* Understanding software quality assurance in Open Source Software

** GitHub Projects. Quality Analysis of Open-Source Software

Existing research

  • Focus on similarities with appropriate software

  • Not applicable (especially to mid-size projects)

QA in software is business oriented

  • Money = Obligations, Requirements, Resources
  • No money = No Obligations, No Requirements, Lack of Resources

Possible Solution

QA in Manufacturing

  • Six Sigma

  • DMAIC
  • Lean process

"Six Sigma":

Improvement of product quality is a function of improvement in production processes

DMAIC

  • Define

  • Measure

  • Analyze

  • Improve

  • Control

"Lean process":

Improvement of product quality is a function of reducing resource wastage

What I've Learned So Far

Key Product Quality Factors

  • Efficient process
  • Measurable quality definition
  • Risks analysis
  • Waste optimization

Under the Hood of Mid-size Project

Surviving the Framework Hype Cycle by Brandon Hayes

Trigger stage

Properties:

  • Innovation

  • Hard to discover

Trigger stage

Quality =

  • "Customer" = Developer

  • 100% customer satisfaction

  • No resource wastage

Hype stage

Properties:

  • Trending in media

  • No proof or guarantees

  • Innovative

Hype stage

Quality =

  • 100 "Customers"

  • 1 Developer

  • 99 defects

  • 1% customer satisfaction

Hype stage (optimization)

What is critical for growing
(covering the risks)		 Way to optimize
(fighting resource wastage)
- Cultivate interest (Lack of application)
- Minimize bugs (Proof of work)		 Define initial requirements (Absence of requirements)

Crystallization stage

Properties:

  • Proved solution

  • Ready for optimizations

  • Modest adoption

  • Contributions growth

Crystallization stage

Quality =

  • 1000 "Customers"

  • 1 Developer

  • Improved features processing

  • Source Quality

  • Solution optimization

Crystallization stage (optimization)

What is critical for growing
(covering the risks)		 Way to optimize
(fighting resource wastage)
- Write source documentation (Sustainability)
- "Make it right" (Non-optimal solution)		 Define strict feature requirements (Overgrown scope)

Productivity stage

  • High adoption

  • Maintainers team

  • Low defects level

  • Stable contributions

Definition of Quality

Quality = Fn(

Community, Docs, Implementation, Support)

  • Implement = Fn(Community*)

  • Docs            = Fn(Community*)

  • Support      = Fn(Community*)

  * Community behavior actually

Hypothesis:

Quality = Fn(Community*) 

 

Who makes up an open source project community?

Community is

  • Spectators

  • End-users

  • Bug Reporters

  • Contributors

  • Maintainers

bundler    Conversion

Year Downloads Participants
2017 80M 1014
2016 51M 901
Revenue 36% 11%

ruby-openid Conversion

Year Downloads Participants
2017 735K 14
2016 725K 18
Revenue 1% -22%

Are we able to improve number of people involved?

Multi-agent systems are semiautonomous decision makers which cooperate to optimize the manufacturing process.

Rule of thumb:

Open-source gives us freedom to start and to quit whenever you like

Smart Open-source Agent Behavior

The Open-Source Enthusiast Playbook

Playbook Overview

Problem Empathy

Efficiency

Innovation

Focus \ Role Spec User Reporter Contrib Maintainer
Pioneer (Trigger/Hype)
Settler (Crystallization)
Town Planner (Productivity)

Curiosity

Spectator Playbook

Stage  Actions
Trigger Discover and share innovations
Hype Analyze and discuss the problem demand
Crystallization Analyze and discuss requirements development and refactoring process in valuable open-source project
Productivity Analyze and discuss best practices and quality of the mature open-source

User Playbook

Stage  Actions
Trigger Discover and share application experience
Hype Analyze and discuss the solution conflicts based on your experience
Crystallization Analyze and discuss how new requirements and refactoring process influenced on your user experience 
Productivity Analyze and discuss your experience about using a tool with such level of quality 

Reporter Playbook

Stage  Actions
Trigger Discover and share application experience with contributors (e.g. in social or DM)
Hype Analyze and express the solution conflicts based on your experience. Add here any idea of how to overcome the problem and share with contributors
Crystallization Analyze pitfalls of new requirements and refactoring process upon your experience. Add here any idea of how to overcome the problem and share with contributors 
Productivity Learn the best practices of bug reporting and create high quality reports.
Create reports about development process issues.

Contributor Playbook

Stage  Actions
Trigger Learn about innovative tech by contribution
Hype Earn some experience about trending topic by contribution
Crystallization Take an attempt to optimize or refactor valuable open-source project
Productivity Contributions of any kind are welcome at the stage (especially documentation, which is the most simple way to start contributing)
Also you'll learn about being a part of the efficient and grown up developer team

Maintainer Playbook

Stage  Actions
Trigger Make innovation
Hype Verify and clarify your solution
Crystallization Make your solution stable
(Congrats! You almost raised mature open-source project.)
Productivity Fight with complexity (be open for alternative solutions).
Make the community around your problem stable.

Participants Conversion

Efficiency

Innovation

Stage \ Role Spec User Reporter Contrib Maintainer
Trigger MW*
Hype MW*
Crystallization MW* MW*
Productivity MW* MW*

* Most Wanted group of participants

Problem Empathy

Curiosity

Summary

Thanks

Visit the open-source-quality.github.io

 

Tweet me @ss_dolganov

Current Package Quality Assurance

By Sergey Dolganov

Made with Slides.com

Current Package Quality Assurance

Story behind one research about what Open-Source quality is and how to measure it.

  • 506

More from Sergey Dolganov