Fcookie : a Health monitor for a fortune cookie service
Requirements :
- Fortune cookie service
- Health monitor to provide status of the service
- Manage several instances of the service
- Dynamic load balancer
- Dynamic load balancing through http proxy : can add servers manually or automatically in case of overload
- Simple fortune cookie web service, delivers raw json structure, no interface
- Simple monitoring interface displaying the state of the web services running
- Home-made push/pull monitoring system with sort of MIBs
Work done :
OS : Linux (ubuntu) Language : Python 2.7
Architecture
Architecture
Main idea : create a process for each interfacing with the monitor software
First goal : distributed implementation, with a tree structure, parents leaves aggregating children data
Current implementation : a node writes informations in a table shared with the interface process, the latter sending the informations to the monitoring software
Load balancer
- Manages lists of servers to distribute the received requests to available servers among the "in use" servers
( Configuration file)
Monitoring system
- The monitoring software is aware of the load balancer
- It sends a request to have the list of running web-services
- Upon reception of this request, the load balancer informs the web-services about the monitor address
- Heart-beat protocol started between web-services and monitoring software
Monitoring data :
- Alive monitoring interfaces ( heart-beat)
- Alive web-services ( http requests)
- Last exception raised by the web-service
- When web-service fails, request for the last logs
Main problems :
- Time : tried to focus on important points and global architecture
- Couldn't implement all I wanted to do in time
- Troubles with the ports
Improvements
- Heartbeat protocol between the Load balancer MI and each WS
- Visual upgrades of the web interfaces
- Booting instances from the balance loader, instead of using available servers or booting them manually
- Security : some data should be protected from external access, and depending on their importance, an handshake protocol could be implemented to exchange keys and encrypt the communications. Certificates could be used to identify the member of the system and avoid spoofing or forgery.
- Add more monitoring data
- HAproxy / Chef