Jalex Chang
2019.06.18
Epoll
The Hero Behind One Million WebSocket Connections in Go

Jalex Chang
- Backend Engineer @ Umbo Computer Vision
- Taiwan Data Engineering Association Member
- Golang Taiwan Member

Agenda
- Introduction
- Linux Network I/O
- Epoll in Go
- Demo and Discussion
- Summary
References
[1] Going Infinite, handling 1M websockets connections in Go, https://github.com/eranyanay/1m-go-websockets
[2] A Million WebSockets and Go, https://www.freecodecamp.org/news/million-websockets-and-go-cc58418460bb/
[3] WebSocket implementation in Go: WS, https://github.com/gobwas/ws
[4] UNP - I/O Multiplexing: The select and poll Functions, https://notes.shichao.io/unp/ch6/
Going Infinite, handling 1 million WebSocket connections in Go, Eran Yanay
- It is a talk shared in Gophercon Israel 2019.
- Use one single Go program serving 1 million WebSocket connections with less than 1 GB memory.
- This sharing does not claim a better way to build a production service.
- This sharing does want to demonstrate the capability of combing state-of-the-art software technologies
- How to handle persistent connections (network resources) efficiently.
The whole project materials in https://github.com/eranyanay/1m-go-websockets
What is WebSocket?

- WebSocket is a network protocol providing a duplex channel over a TCP connection.
- Usually used between websites and servers for needs of persistent connections.
- Feed notifications
- Collaborating editing
- Chat applications
WebSocket in GO
golang.org/x/net/websocket (X)
gorilla/websocket (O)
var upgrader = websocket.Upgrader{} // use default options
func ws(w http.ResponseWriter, r *http.Request) {
	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		log.Print("upgrade:", err)
		return
	}
	defer conn.Close()
	for {
		mt, message, err := conn.ReadMessage()
		log.Printf("recv: %s", message)
		conn.WriteMessage(mt, message)
	}
}
func main() {
	http.HandleFunc("/", ws) // Will create a goroutine to execute the function
        log.Fatal(http.ListenAndServe(":8080", nil))
}- In some use cases, WebSockets are almost idled.
- However, we still need to allocate basic resources for existed connections (goroutines).
What is the problem?
- For 1 million connections, there are about 20GB memory uasge.
Optimizations
If we could know which connections are active (read-ready/write-ready), we can only allocate resources to serve the connections in need.
So, all optimization strategies talked about how to manage built WebSocket connections efficiently.
TCP socket in Linux

- Sockets are interface indicating used network I/O resource in the Linux kernel.
-Every time the TCP server call accept(), the kernel will give it a file descriptor (fd) for the built socket.
- We use fd as a token to communicate with the kernel to check the status of the socket connection.
Blocking I/O

Nonblocking I/O

I/O multiplexing

Signal driven I/O

Asynchronous I/O

Summary of I/O models

Solutions of I/O multiplexing
select
- Have a limited size of fds: 1024.
- Should scan all fds: O(N).
poll
- Without size limitation of fds.
- Should scan all fds: O(N).
epoll
- Without size limitation of fds.
- Implemented by red-black-tree: search in O(log N).
Epoll in GO
golang.org/x/sys/unix
// All methods can be mapped to Linux epoll commands (http://man7.org/linux/man-pages/man7/epoll.7.html)
 
// Create an epoll with initial space for fds.
// Since Linux 2.6.8, the size argument is ignored, but must be greater than zero for backward compatible.
func EpollCreate(size int) (epfd int, err error) {......}
// Create an epoll with indicating flag. 
// If flag is 0, then, other than the fact that the obsolete size argument is dropped
func EpollCreate1(flag int) (epfd int, err error) {......}
// EpollCtl is used to add, modify, or remove entries in the interest list of the epoll.
// Valid values for the op argument are: EPOLL_CTL_ADD, EPOLL_CTL_MOD, and EPOLL_CTL_DEL.
// EpollEvent is an object telling the epoll what events we want to monitor for the fd.
// E.g. unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)}
func EpollCtl(epfd int, op int, fd int, event *EpollEvent) (err error) {......}
// EpollWait call system call waits for events on the epoll.
// Ready events will be injected to events
func EpollWait(epfd int, events []EpollEvent, msec int) (n int, err error) {......}
Epoller: a simple implementation
type epoll struct {
	fd          int
	connections map[int]net.Conn // net.Conn implements io.ReadWriter
	lock        *sync.RWMutex
}
func MkEpoll() (*epoll, error) {
	fd, err := unix.EpollCreate1(0)
	if err != nil {
		return nil, err
	}
	return &epoll{
		fd:          fd,
		lock:        &sync.RWMutex{},
		connections: make(map[int]net.Conn),
	}, nil
}
func socketFD(conn net.Conn) int {
	tcpConn := reflect.Indirect(reflect.ValueOf(conn)).FieldByName("conn")
	fdVal := tcpConn.FieldByName("fd")
	pfdVal := reflect.Indirect(fdVal).FieldByName("pfd")
	return int(pfdVal.FieldByName("Sysfd").Int())
}
func (e *epoll) Add(conn net.Conn) error {
	// Extract file descriptor associated with the connection
	fd := socketFD(conn)
        event := unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)}
	err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_ADD, fd, &event)
	if err != nil {
		return err
	}
	e.connections[fd] = conn // connections are protected by RWMutex
	return nil
}
func (e *epoll) Remove(conn net.Conn) error {
	fd := socketFD(conn)
	err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_DEL, fd, nil)
	if err != nil {
		return err
	}
	delete(e.connections, fd)
	return nil
}
func (e *epoll) Wait() ([]net.Conn, error) {
	events := make([]unix.EpollEvent, 100) // The max size of events retrieving from epoll
	n, err := unix.EpollWait(e.fd, events, 100)
	if err != nil {
		return nil, err
	}
	var connections []net.Conn
	for i := 0; i < n; i++ {
		conn := e.connections[int(events[i].Fd)]
		connections = append(connections, conn)
	}
	return connections, nil
}var epoller *epoll
func wsHandler(w http.ResponseWriter, r *http.Request) {
	conn, _, _, err := ws.UpgradeHTTP(r, w) // Upgrade connection
	if err != nil {
		return
	}
        // When conn is added to epoller, this goroutine is done.
	if err := epoller.Add(conn); err != nil {
		log.Printf("Failed to add connection %v", err)
		conn.Close()
	}
}
func main() {
	// Increase resources limitations......
        // Enable pprof hooks.......
	// Start epoll
	var err error
	epoller, err = MkEpoll()
	if err != nil {
		panic(err)
	}
	go HandleConnection()
	http.HandleFunc("/", wsHandler)
	if err := http.ListenAndServe("0.0.0.0:8000", nil); err != nil {
		log.Fatal(err)
	}
}WebSocket server
func HandleConnection() {
	for {
		connections, err := epoller.Wait()
		if err != nil {
			log.Printf("Failed to epoll wait %v", err)
			continue
		}
		for _, conn := range connections {
			if conn == nil {
				break
			}
                        // ReadClientData reads next data message from conn
			if msg, _, err := wsutil.ReadClientData(conn); err != nil {
				if err := epoller.Remove(conn); err != nil {
					log.Printf("Failed to remove %v", err)
				}
				conn.Close()
			} else {
				log.Printf("msg: %s", string(msg))
			}
		}
	}
Use one goroutine to serve whole active sockets.
Demo

Before
After

Discussion
Q1: What is the drawback of using epoll?
A trade-off between memory consumption and latency/throughput.
Q2: What use cases are suitable for mentioned optimizations?
For applications with non-busy socket connections or regardless of latency.
Q3: If my production fit the use case, can I adopt these optimizations?
Sure, but should be aware of connection storm and event burst.
Summary
- We went through the details of WebSocket application optimization.
- It is all about network resource management efficiency.
- So, we introduced the network I/O in Linux and adopted epoll.
- We illustrated how to use epoll in Go.
- Use a single goroutine to monitor epoll, and then serve socket if need.
- Epoll is only useful in the network I/O, but the concept (red-black-tree) is helpful anywhere.
We are either progressing or retrograding all the while. There is no such thing as remaining stationary in this life. – James Freeman Clarke
Thanks for listening!


Epoll: The Hero Behind One Million WebSocket Connections in Go
By Jalex Chang
Epoll: The Hero Behind One Million WebSocket Connections in Go
Introduce the core methodology behind serving 1 million WebSocket connections in limited memory.
- 3,929
 
   
   
  