Tech from startup to enterprise

Miro Svrtan (@msvrtan)

15y PHP veteran

ZgPHP & WebCamp co-organizer

@Njuškalo for 5y

Technical analyst
Development team leader
Architect
Developer

Njuskalo.hr introduction

• online classifieds platform
• based in Zagreb, Croatia
• founded in 2007
• startup inside a large media corporation
  

Team organization

Small development team

• 3 backend devs
• 1 frontend dev
• small but dedicated & focused team
  

Platform: software

• LAMP stack
• Nginx proxy
• Sphinx search
• Memcached + Redis
  

Platform: code

• in house MVC PHP framework
  
• 2009: too many flavours of OS FWs
• 2014: IMHO loosing a battle to OS FWs
• small team vs vast community
• learning best practices
  
• introduction to new developers
• existing expirience
• better documentation
• tutorials/howto's
  

Page views in last 5y

Unique visitors in last 5y

2009 vs 2014

Visitors

425.000 (06/2009)

1.000.000 (01/2014)

Source: Gemius Audience

PageViews

25.000.000 (06/2009)

295.000.000 (01/2014)

Source: Gemius Audience

Platform: Physical location

• local data center
• low network latency -> 90% local traffic
  
• (possible) legal issues

Platform: Hardware

• bare metal servers
• enterprise grade storage
  

Resources used: 2009 vs 2014

< 30GB to ~10TB

< 30Mb/s to > 450Mb/s

DB

~1GB  to ~360GB

Living on the edge

• close to maximum resource utilization
  
• developing features with no room to spare
• concentrating on performance instead of feature
  
• hard & exhausting

We got room

• new shiny server is here
• vastly oversized -> easy living
  
• developers stop thinking about performances

Multi feature development

• test server hosts >10 Njuškalo applications
• test1.example.com
• test2.example.com
• ..
  
• test10.example.com
• most locations serve only 1 new feature/bug-fix
• helps to cherry pick what is ready for production

Multi feature development #2

• using git
• moved from subversion in 2010
  
• always working on rebased feature branch
  
• master branch + that feature changes
  

Startup

Enterprise

Becoming an enterprise

• transformation of goals/mindsets
  
• not something in a road map

When?

• mistakes "cost" more than
  
• better/longer preparation
• more testing
• additional hardware

Cost of doing errors

• not just money!
• brand name recognition
• loosing trust of users
  
• personal pride
  

Journey

• started with ~0 scaling experience
• hard to come by
  
• there is no manual
• listen & read what/how others did it
• outside PHP || web world too!!
  

Monitoring

• started with
• dstat
  
• gemius
• added
• cacti
• Google Analytics real time
• Graphite
• NewRelic
  

Autoload

• irritated with requires/includes
  
• different naming schemes
• before PSR-0
• parsing PHP files to generate map
  
• file array vs APC user cache
• after 4 years found a CRITICAL bug
• logical flaw
• production crashes
  

So when can you be sure your 100 lines of code are bug free?

If it worked first 10.000 times?

.. 100.000 times?

... 1.000.000 times?

... 1.000.000.000 times?

Well 8.000.000.000 times was not enough for this one :)

Tips n tricks:

Linux desktop

• development team switched to Linux desktop
  
• easier environment to setup
  
• get to know terminal
  
• similar to production
• reduces fear when working on production

Tips n tricks:

face 2 face discussions

• improves communication
  
• meetings outside office when ever possible
• less interruptions
  
• more relaxed atmosphere
• change of scenery
• boosts creativity & engagement

Tips n tricks:

analyze seasonality

• locate
• peaks
• low times
  
• predict user behavior -> scaling
  
• we found patterns based on
• hours in day
  
• day of week
• months in year
• weather!
  

Tips n tricks:

deployment guidelines

• not after 4pm
  
• avoid fridays
• major upgrades during night
  

Thank you

Miro Svrtan

@msvrtan

https://joind.in/11315