Architecture Overview & Improvements
Agenda
- What is DTech Group about?
- Requirements
- Architecture and Infrastructure
- Discuss NFRs and Improvements
Requirements
- SaaS platform to access data from multiple social media platforms
- MVP: Twitter only
- Future: Facebook, LinkedIn, etc.
-
Multitenancy, Billing
-
Data synchronization from social media platforms
-
Complex searching queries
-
Email notifications
-
Screenshots of social media posts
Non-Functional Requirements
- Extensive requests and queues logging
- Optimize Twitter Enterprise API request count
- Configurable variables to tweak cost/performance
- MVP: Low infrastructure cost
- Future proof
- Easy scalability
- Performance (Response time)
- Security
Architecture
Infrastructure
Performance
- Handles 1200+ Twitter profiles
- Searching over 10 million+ documents
- Have 500MB+ worth of screenshots
- API response time < 600ms for 85th percentile
- Collected over 1million tweets in a day
Improvements
- Use caching
- Optimize some poor performing SQL queries
- Use database read-replicas
- Shard Elasticsearch data
Scalability
- Uses ECS Fargate
- Stateless backend services
- Uses CloudMap for Service Discovery
Improvements
- Breakdown some consumers/tasks to Lambdas
- Perform automatic scaling of containers
Availability
- Serving FE over CloudFront
Improvements
- Use multi AZ for services
- Use multi AZ for databases
Security
- TLS v1.3
- CORS Enabled
- JWT Authentication
- IAM Roles / Policies
- Security Groups
Improvements
- Use Cognito
- Perform penetration testing
- Secure application configurations with secret manager
Maintainability / Extensibility
- Dev Env: mimics 100% infrastructure using containers
- Modular structure to support more social media platforms
- Replaceable services
- API Documentation / Consistent response
- Extensive logging
Improvements
- Increase test coverage
- Add support for internationalization in FE
- Use Cognito