Move fast and break things?

The impact of rapid releases on the rejection of software changes

Rodrigo Souza

<rodrigo@dcc.ufba.br>

Title Text

Subtitle

Salvador, Bahia, Brazil

Assistant Professor at the

Federal University of Bahia

Research interests

  • Empirical software engineering
  • Mining software repositories
  • Release engineering
  • Software quality

This talk

  • Rejected changes
    • issue/bug reopening
    • supplementary changes
    • negative code reviews
  • Rapid releases
    • Facebook's early motto:
      "Move fast and break things"
    • Mozilla: 6-week releases
  • Are changes more likely to be rejected when we move faster?

Rejected changes

Software changes

  • Software has to change
  • Not all changes are created equal
    • Good enough :)
    • Inappropriate :(

Undetected

Bugs are shipped to users!

Detected

Rejection

 

 

 

Rework
 

 

  • the review is negative
  • the commit is reverted
  • the issue is reopened
  • supplementary change
  • re-testing
  • (i.e., rework)

Inappropriate changes

(the earlier, the better)

Inappropriate changes

how to detect them from existing repos?

how much rework do them cause?

are they more likely under rapid releases?

Data: Mozilla Foundation

  • Main projects:
    • Firefox
    • Core (reusable code)
  • Mercurial (commits) and
    Bugzilla (issues/bugs)
  • All non-trivial code is reviewed
  • All inappropriate commits are reverted
  • Closed issues may be reopened

Inappropriate changes

how to detect them from existing repos?

  • negative code reviews
  • reverted commits
  • supplementary commits
  • reopened issues

Negative code reviews

  • Very early rejection
  • Problems in the code
    • coding style
    • design
    • performance
    • potential bugs
  • Not all projects perform reviews on all changes

Reverted commits

  • A commit that reverts a previous commit
  • In Mercurial, revert = "backout"
  • Can be early or late in the process
  • Rejection due to problems found during automated or manual testing
  • Very few projects actually revert commits
changeset: 0a93ae68184e
summary:   Bug 787078 - Backout 43dd8252f52d

Supplementary commits

  • 2+ commits for the same issue
  • Can be early or late in the process
  • Not always a sign of problems
    • Multipart commits
  • Easily detectable
changeset: 0a93ae68184e
summary:   Bug 1635 - add info button

changeset: 394bf8a8e9fa
summary:   Bug 1635 - fix button behavior

Issue reopening

  • Problem found after issue was closed
  • Late in the process
  • Not always because of problems in the change itself
    • Tester used the wrong version
    • Issue was prematurely closed
      (e.g., duplicate, wontfix)
  • In some projects, a new issue is created

supplementary commits

reverted commits

Core

supplementary commits

reverted commits

Part 1: ...

Part 2: ...

Core

reopened issues

commits reverted after issue is closed

Core

Inappropriate changes

how much rework do them cause?

rejected changes

negative reviews

reverted commits

reopened issues

21%

12%

8%

5%

(among changes associated with issues, reviews, and commits)

Core

5 rejections

per day

1 rejection

per day

How often are changes rejected?

Inappropriate changes

are they more likely under rapid releases?

Rapid Releases

Release approach

Feature-based

Time-based

Traditional releases

Rapid releases
(short cycles)

Rapid releases
and quality

Frequent delivery => more feedback

Tighter deadlines => less time for testing

Rapid releases
and time pressure

Hurry to ship a feature in the next release

"If I don't ship now, I can ship in a few weeks"

vs.

Traditional

Before March 2011

12+ months btw releases

Rapid

After March 2011

6 weeks btw releases

Transition to rapid releases

  • Annoying update notifications!
  • My extensions are broken!
  • My company needs stability!

(users' viewpoint)

What about rejections?

issues w/ reverted commits

the rejection rate is increasing!

... or is it?

.

.

.

Early vs late rejections

early reverts

problems are detected earlier

late reverts

You have to understand the process

main branch

pull

push

automated build:

wait for build...

or

(broken)

revert and push

Before

main branch

pull

push

revert and push

After

staging branch

build sheriff

merge

Conclusions

  • Naïve conclusion: with rapid releases...
    • developers are breaking the system more often
    • hence, they perform more rework
  • After talking to developers: with rapid releases...
    • they perform less testing before pushing changes
    • the main branch is rarely broken
  • For users:
    • late rejections decreased (fewer bugs are shipped)
    • updates arrive frequently
    • corporate users have extended support releases

References

R. SOUZA. Inappropriate Software Changes: Rejection and Rework (PhD Thesis)

http://rodrigorgs.github.io/files/rodrigo-thesis-final.pdf

SOUZA, R. ; CHAVEZ, C. ; BITTENCOURT, R. A.
Rapid Releases and Patch Backouts: A Software Analytics Approach
In: Software, IEEE, 2015.

http://online.qmags.com/ISW0315#pg91&mode2

Other related publications

http://rodrigorgs.github.io/publications

Move fast and break things?

The impact of rapid releases on the rejection of software changes

Rodrigo Souza

<rodrigo@dcc.ufba.br>

(extra slides)

Before

  • All changes were pulled from and pushed to one single branch
  • Each change triggered a build (compile + test), which lasted ~ 4h
  • A broken build affected all developers
  • The developers was responsible for reverting his/her commit

After

  • Automated builds are still run and are still slow
  • Changes are pulled from the main branch and pushed to a staging branch
  • Upon a succesful build, sheriffs merge the two branches
  • Upon a broken build, sheriffs revert the offending commits

Broad questions

  • How to detect inappropriate changes before they reach end users?
  • How to avoid creating inappropriate changes in the first place?
  • what are the characteristics of changes that end up being rejected?
  • what characteristics of the process lead to increased rejection?
Made with Slides.com