Release Review
(29.07 - 18.08)
Infrastructure Team

ChainDrop attack

• Problem: On August 4th, Microsoft published information about a new supply chain attack targeting NPM.

• Solution: We scanned all repositories for infected libraries, conducted a security review, and, with Frontend Infrastructure, added extra checks.

• Effect: Within 24 hours, we were back to normal operations without long, expensive follow-ups.

Bot attack

• Problem: LLM bots from China crawled our site, creating unnecessary load and problematic requests.

• Solution: We identified the traffic source and took countermeasures using Cloudflare.

• Effect: In the last 12 hours, we blocked ~350k requests. We will introduce more rules to stop future crawling before it happens.

Bot attack

• Problem: LLM bots from China crawled our site, creating unnecessary load and problematic requests.

• Solution: We identified the traffic source and took countermeasures using Cloudflare.

• Effect: In the last 12 hours, we blocked ~350k requests. We will introduce more rules to stop future crawling before it happens.

Sentinels desync

• Problem: Our Sentinels mixed stage and production configurations. Some mastersets were not resilient to downtime.

• Solution: We moved some masterset configurations between Sentinels and filled configuration gaps.

• Effect: No Redis Cluster problem since fix.

Sentinels desync

Sentinels desync

QA environment manager

• Problem: QA deployment has become more complicated as the number of services has grown. We use a Slack channel to reserve machines. People tend to forget to shut machines down, which leads to unnecessary costs.

• Solution: We introduced a centralized place for everyone to manage QA machines. No K8S knowledge required; SSO; web GUI.

• I hope, you will love it.

QA environment manager

QA environment manager