…adding some microservices still worked…

…relying on remote instances worked for a while…

…but ultimately running the whole system locally failed

…testing a single service isolated became a very important option…

…delivering much less ambiguous test results due to the limited scope

…allowing to run service tests prior to deployment to a test system…

…eventually merging back to master and only then deploying it anywhere…

…is not a good option with multiple microservices involved…

…in separate repositories…

…owned by different teams…

…under different workloads…

Feature toggles helped us to isolate unfinished features and allow real continuous integration

…since merges would need to be synced!

Bug assignment with a small number of teams worked okay…

…but with more teams and more possible root causes (microservices) the bugs started piling up.

As a consequence we created a team of service managers to route bugs…

…improving initial assignment and kept an eye on bugs & incidents.

Running for

>3 h !!!

…teams kept adding more "special interest" tests to it.

Eventually we ignored several tests, asking the teams to adopt them…

…reducing the suite to the minimum common suite covering the money paths.

We also extracted a test framework…

…to make service specific tests easier to write.

…but it actually caught a lot of bugs!

Testing each microservice before deployment would be hard…

…and would ruin a lot of the benefits.

…integration issues became a more prominent problem.

To compensate the release triggered sign-off tests, we added continuous exploratory testing…

…guided by our recently finished user stories

and continuous risk assessments.

We create a changelog chat rooms to compensate…

…filled automatically by the deployment system…

…and also test results of the common system test suite…

…but also open to human comments!

…so teams started to monitor services in production.

Traffic

Errors

Latency

Saturation

+ business metrics

So the teams do not only monitor their services during office hours…

…but also after 5 pm…

…being truly responsible for that deployment on Friday

…who is able to detect and manage global incidents.

Knowing which of the 150 services might cause the incident…

…which other services might be affected…

…which of the 25 teams might be able to help…

…and manage information between these teams, the stakeholders, customer service etc.