1 BUG

20k CRASHES

About me

Alexandre Moureaux

Tech Lead  @

@almouro

github.com/almouro

Check out our open-source (github.com/bamlab) and our blog (blog.bam.tech)

ONCE UPON A TIME...

A team of 9 people

Happy and Proud to release v4.3 of our app

OCTOBER 9TH

Ready for Oct 11th Live event

WHEN

SUDDENLY

OUR SENTRY IS UNDER 

HEAVY FIRE

JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'paddingTop' in shadow node of type: RCTView
Error while updating property 'fill' of a view managed by: RNSVGPath
JSApplicationIllegalArgumentException 
Error while updating property 'paddingBottom' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'right' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'height' in shadow node of type: RCTView
Error while updating property 'fill' of a view managed by: RNSVGPath
JSApplicationIllegalArgumentException 
Error while updating property 'paddingBottom' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'paddingTop' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'right' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'height' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'right' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'height' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'paddingTop' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView
JSApplicationIllegalArgumentException 
Error while updating property 'left' in shadow node of type: RCTView

OUR SENTRY IS UNDER 

HEAVY FIRE

JSApplicationIllegalArgumentException 
Error while updating property 'paddingTop' in shadow node of type: RCTView

ALL USERS ARE AFFECTED

LET'S

REPRODUCE

THE BUG

 SENTRY TO THE RESCUE

ANY DEVICE

CRASH ON STARTUP

ANY USER

CAN'T REPRODUCE??

ANALYZE THE

STACKTRACE

Buuuut...

we have several different errors

OK, LET'S PICK ONE

java.lang.ArrayIndexOutOfBoundsException: length=10; index=-1
    at android.support.v4.util.Pools$SimplePool.release(Pools.java:116)
    at com.facebook.react.bridge.DynamicFromMap.recycle(DynamicFromMap.java:40)
    at com.facebook.react.uimanager.LayoutShadowNode.setHeight(LayoutShadowNode.java:168)
    at java.lang.reflect.Method.invoke(Method.java)
    ...

java.lang.reflect.InvocationTargetException: null
    at java.lang.reflect.Method.invoke(Method.java)
    ...

com.facebook.react.bridge.JSApplicationIllegalArgumentException: Error while updating property 'height' in shadow node of type: RNSVGSvgView
    at com.facebook.react.uimanager.ViewManagersPropertyCache$PropSetter.updateShadowNodeProp(ViewManagersPropertyCache.java:113)
    ...

What is SimplePool?

Why am I in the Android source code?

SHOULD WE ROLL BACK

OUR RELEASE?

  1. We need to deploy the old code
  2. We now know crash rate is about 10%
  3. This release has great value for our users

SHOULD WE ROLL BACK

OUR RELEASE?

NO

BATTLE PLAN

  • 6 Android devices
  • 10 app launch per device
  • Should get ~5-10 crashes
  • We'd have reproduction

RESULT

0 CRASHES

WHAT CHANGED?

TWO NATIVE CHANGES SINCE LAST RELEASE

  • upgraded native-navigation

  • upgraded react-native-svg

ANDROID ROLLOUT

NEW BATTLE PLAN

  1. Downgrade native-navigation
  2. Roll out for 10% users
  3. Check back
  4. If success? Full rollout!

NEW BATTLE PLAN

  1. Downgrade native-navigation
  2. Roll out for 10% users
  3. Check back
  4. What if the crash still remains?

NEW BATTLE PLAN

  1. Downgrade native-navigation
  2. Roll out for 10% users
  3. Check back
  4. If failed, downgrade svg lib
  5. Roll out for 10% users
  6. Check back
  7. If success, full rollout!

NEW BATTLE PLAN

  1. Downgrade native-navigation
  2. Roll out for 10% users
  3. Check back
  4. If failed, downgrade svg lib
  5. Roll out for 10% users
  6. Check back
  7. And if the crash remains?

THAT PLAN

IS

DUMB

java.lang.ArrayIndexOutOfBoundsException: length=10; index=-1
 mPoolSize=-1

UNDERSTANDING THE ENEMY

OUR ULTIMATE WEAPON

THE DEBUGGER

Text

BREAKPOINT HIT 1

Text

BREAKPOINT HIT 2

AFTER 34 CLICKS ON THE PLAY BUTTON

REVELATION

HIT 1 - 33

HIT 34

HIT 34

Soooo....

It was the SVG upgrade all along!

react-native-svg started using this code in v7

WHAT CAN HAPPEN

WHAT CAN HAPPEN

LET'S

FIX

REACT NATIVE PR

FINAL BATTLE PLAN

  1. Patch react-native
  2. Roll out for 10% users
  3. Check back
  4. Success?

VICTORY

Takeaways

Use your crash reporting

Know what the user is doing

Use your crash reporting

Add details about the user

Use your crash reporting

Monitor your release health

Protect your users

Learn by digging deeper

Thank you!

:   

)   

@almouro

almouro

Debugging a non reproducible bug (with 20k crashes)

By Alexandre Moureaux

Made with Slides.com

Debugging a non reproducible bug (with 20k crashes)

1 bug. 20k crashes. Impossible to reproduce. This is the story of an epic fight against that vicious and cruel bug.

  • 1,481

More from Alexandre Moureaux