Jalex Chang
2019.06.18
Epoll
The Hero Behind One Million WebSocket Connections in Go
Jalex Chang
- Backend Engineer @ Umbo Computer Vision
- Taiwan Data Engineering Association Member
- Golang Taiwan Member
Agenda
- Introduction
- Linux Network I/O
- Epoll in Go
- Demo and Discussion
- Summary
References
[1] Going Infinite, handling 1M websockets connections in Go, https://github.com/eranyanay/1m-go-websockets
[2] A Million WebSockets and Go, https://www.freecodecamp.org/news/million-websockets-and-go-cc58418460bb/
[3] WebSocket implementation in Go: WS, https://github.com/gobwas/ws
[4] UNP - I/O Multiplexing: The select and poll Functions, https://notes.shichao.io/unp/ch6/
Going Infinite, handling 1 million WebSocket connections in Go, Eran Yanay
- It is a talk shared in Gophercon Israel 2019.
- Use one single Go program serving 1 million WebSocket connections with less than 1 GB memory.
- This sharing does not claim a better way to build a production service.
- This sharing does want to demonstrate the capability of combing state-of-the-art software technologies
- How to handle persistent connections (network resources) efficiently.
The whole project materials in https://github.com/eranyanay/1m-go-websockets
What is WebSocket?
- WebSocket is a network protocol providing a duplex channel over a TCP connection.
- Usually used between websites and servers for needs of persistent connections.
- Feed notifications
- Collaborating editing
- Chat applications
WebSocket in GO
golang.org/x/net/websocket (X)
gorilla/websocket (O)
var upgrader = websocket.Upgrader{} // use default options
func ws(w http.ResponseWriter, r *http.Request) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Print("upgrade:", err)
return
}
defer conn.Close()
for {
mt, message, err := conn.ReadMessage()
log.Printf("recv: %s", message)
conn.WriteMessage(mt, message)
}
}
func main() {
http.HandleFunc("/", ws) // Will create a goroutine to execute the function
log.Fatal(http.ListenAndServe(":8080", nil))
}
- In some use cases, WebSockets are almost idled.
- However, we still need to allocate basic resources for existed connections (goroutines).
What is the problem?
- For 1 million connections, there are about 20GB memory uasge.
Optimizations
If we could know which connections are active (read-ready/write-ready), we can only allocate resources to serve the connections in need.
So, all optimization strategies talked about how to manage built WebSocket connections efficiently.
TCP socket in Linux
- Sockets are interface indicating used network I/O resource in the Linux kernel.
-Every time the TCP server call accept(), the kernel will give it a file descriptor (fd) for the built socket.
- We use fd as a token to communicate with the kernel to check the status of the socket connection.
Blocking I/O
Nonblocking I/O
I/O multiplexing
Signal driven I/O
Asynchronous I/O
Summary of I/O models
Solutions of I/O multiplexing
select
- Have a limited size of fds: 1024.
- Should scan all fds: O(N).
poll
- Without size limitation of fds.
- Should scan all fds: O(N).
epoll
- Without size limitation of fds.
- Implemented by red-black-tree: search in O(log N).
Epoll in GO
golang.org/x/sys/unix
// All methods can be mapped to Linux epoll commands (http://man7.org/linux/man-pages/man7/epoll.7.html)
// Create an epoll with initial space for fds.
// Since Linux 2.6.8, the size argument is ignored, but must be greater than zero for backward compatible.
func EpollCreate(size int) (epfd int, err error) {......}
// Create an epoll with indicating flag.
// If flag is 0, then, other than the fact that the obsolete size argument is dropped
func EpollCreate1(flag int) (epfd int, err error) {......}
// EpollCtl is used to add, modify, or remove entries in the interest list of the epoll.
// Valid values for the op argument are: EPOLL_CTL_ADD, EPOLL_CTL_MOD, and EPOLL_CTL_DEL.
// EpollEvent is an object telling the epoll what events we want to monitor for the fd.
// E.g. unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)}
func EpollCtl(epfd int, op int, fd int, event *EpollEvent) (err error) {......}
// EpollWait call system call waits for events on the epoll.
// Ready events will be injected to events
func EpollWait(epfd int, events []EpollEvent, msec int) (n int, err error) {......}
Epoller: a simple implementation
type epoll struct {
fd int
connections map[int]net.Conn // net.Conn implements io.ReadWriter
lock *sync.RWMutex
}
func MkEpoll() (*epoll, error) {
fd, err := unix.EpollCreate1(0)
if err != nil {
return nil, err
}
return &epoll{
fd: fd,
lock: &sync.RWMutex{},
connections: make(map[int]net.Conn),
}, nil
}
func socketFD(conn net.Conn) int {
tcpConn := reflect.Indirect(reflect.ValueOf(conn)).FieldByName("conn")
fdVal := tcpConn.FieldByName("fd")
pfdVal := reflect.Indirect(fdVal).FieldByName("pfd")
return int(pfdVal.FieldByName("Sysfd").Int())
}
func (e *epoll) Add(conn net.Conn) error {
// Extract file descriptor associated with the connection
fd := socketFD(conn)
event := unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)}
err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_ADD, fd, &event)
if err != nil {
return err
}
e.connections[fd] = conn // connections are protected by RWMutex
return nil
}
func (e *epoll) Remove(conn net.Conn) error {
fd := socketFD(conn)
err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_DEL, fd, nil)
if err != nil {
return err
}
delete(e.connections, fd)
return nil
}
func (e *epoll) Wait() ([]net.Conn, error) {
events := make([]unix.EpollEvent, 100) // The max size of events retrieving from epoll
n, err := unix.EpollWait(e.fd, events, 100)
if err != nil {
return nil, err
}
var connections []net.Conn
for i := 0; i < n; i++ {
conn := e.connections[int(events[i].Fd)]
connections = append(connections, conn)
}
return connections, nil
}
var epoller *epoll
func wsHandler(w http.ResponseWriter, r *http.Request) {
conn, _, _, err := ws.UpgradeHTTP(r, w) // Upgrade connection
if err != nil {
return
}
// When conn is added to epoller, this goroutine is done.
if err := epoller.Add(conn); err != nil {
log.Printf("Failed to add connection %v", err)
conn.Close()
}
}
func main() {
// Increase resources limitations......
// Enable pprof hooks.......
// Start epoll
var err error
epoller, err = MkEpoll()
if err != nil {
panic(err)
}
go HandleConnection()
http.HandleFunc("/", wsHandler)
if err := http.ListenAndServe("0.0.0.0:8000", nil); err != nil {
log.Fatal(err)
}
}
WebSocket server
func HandleConnection() {
for {
connections, err := epoller.Wait()
if err != nil {
log.Printf("Failed to epoll wait %v", err)
continue
}
for _, conn := range connections {
if conn == nil {
break
}
// ReadClientData reads next data message from conn
if msg, _, err := wsutil.ReadClientData(conn); err != nil {
if err := epoller.Remove(conn); err != nil {
log.Printf("Failed to remove %v", err)
}
conn.Close()
} else {
log.Printf("msg: %s", string(msg))
}
}
}
Use one goroutine to serve whole active sockets.
Demo
Before
After
Discussion
Q1: What is the drawback of using epoll?
A trade-off between memory consumption and latency/throughput.
Q2: What use cases are suitable for mentioned optimizations?
For applications with non-busy socket connections or regardless of latency.
Q3: If my production fit the use case, can I adopt these optimizations?
Sure, but should be aware of connection storm and event burst.
Summary
- We went through the details of WebSocket application optimization.
- It is all about network resource management efficiency.
- So, we introduced the network I/O in Linux and adopted epoll.
- We illustrated how to use epoll in Go.
- Use a single goroutine to monitor epoll, and then serve socket if need.
- Epoll is only useful in the network I/O, but the concept (red-black-tree) is helpful anywhere.
We are either progressing or retrograding all the while. There is no such thing as remaining stationary in this life. – James Freeman Clarke
Thanks for listening!
Epoll: The Hero Behind One Million WebSocket Connections in Go
By Jalex Chang
Epoll: The Hero Behind One Million WebSocket Connections in Go
Introduce the core methodology behind serving 1 million WebSocket connections in limited memory.
- 3,572