Jalex Chang

2019.06.18

Epoll

The Hero Behind One Million WebSocket Connections in Go

 

Jalex Chang

- Backend Engineer @ Umbo Computer Vision

- Taiwan Data Engineering Association Member

- Golang Taiwan Member

Contact:

- jalex.cpc @ Gmail

- jalex.chang @ Facebook

- JalexChang @ GitHub

Agenda

- Introduction 

- Linux Network I/O

- Epoll in Go

- Demo and Discussion

- Summary

References

[1] Going Infinite, handling 1M websockets connections in Go, https://github.com/eranyanay/1m-go-websockets

[2] A Million WebSockets and Go, https://www.freecodecamp.org/news/million-websockets-and-go-cc58418460bb/

[3]  WebSocket implementation in Go: WS, https://github.com/gobwas/ws

[4] UNP - I/O Multiplexing: The select and poll Functions, https://notes.shichao.io/unp/ch6/

Going Infinite, handling 1 million WebSocket connections in Go, Eran Yanay

 

- It is a talk shared in Gophercon Israel 2019.

- Use one single Go program serving 1 million WebSocket connections with less than 1 GB memory.

 

- This sharing does not claim a better way to build a production service.

- This sharing does want to demonstrate the capability of combing state-of-the-art software technologies

    - How to handle persistent connections (network resources) efficiently.

 

The whole project materials in https://github.com/eranyanay/1m-go-websockets

What is WebSocket?

- WebSocket is a network protocol providing a duplex channel over a TCP connection.

 

- Usually used between websites and servers for needs of persistent connections.

    - Feed notifications

    - Collaborating editing

    - Chat applications

 

 

WebSocket in GO

golang.org/x/net/websocket (X)

gorilla/websocket (O)

var upgrader = websocket.Upgrader{} // use default options

func ws(w http.ResponseWriter, r *http.Request) {
	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		log.Print("upgrade:", err)
		return
	}
	defer conn.Close()
	for {
		mt, message, err := conn.ReadMessage()
		log.Printf("recv: %s", message)
		conn.WriteMessage(mt, message)
	}
}


func main() {
	http.HandleFunc("/", ws) // Will create a goroutine to execute the function
        log.Fatal(http.ListenAndServe(":8080", nil))
}

- In some use cases, WebSockets are almost idled.

- However, we still need to allocate basic resources for existed connections (goroutines).

 
 

What is the problem?

Mem = conns * (goroutine + buf_{net/http} + buf_{gorilla/ws})
\simeq 20KB

- For 1 million connections, there are about 20GB memory uasge.

Optimizations 

If we could know which connections are active (read-ready/write-ready), we can only allocate resources to serve the connections in need

 

So, all optimization strategies talked about how to manage built WebSocket connections efficiently.

 

TCP socket in Linux

- Sockets are interface indicating used network I/O resource in the Linux kernel.

 

-Every time the TCP server call accept(), the kernel will give it a file descriptor (fd) for the built socket.

 

- We use fd as a token to communicate with the kernel to check the status of the socket connection.

Blocking I/O

Nonblocking I/O

I/O multiplexing

Signal driven I/O

Asynchronous I/O

Summary of I/O models

Solutions of I/O multiplexing

select

- Have a limited size of fds: 1024.

- Should scan all fds: O(N).

poll

- Without size limitation of fds.

- Should scan all fds: O(N).

epoll

- Without size limitation of fds.

- Implemented by red-black-tree: search in O(log N).

Epoll in GO

golang.org/x/sys/unix

// All methods can be mapped to Linux epoll commands (http://man7.org/linux/man-pages/man7/epoll.7.html)
 
// Create an epoll with initial space for fds.
// Since Linux 2.6.8, the size argument is ignored, but must be greater than zero for backward compatible.
func EpollCreate(size int) (epfd int, err error) {......}

// Create an epoll with indicating flag. 
// If flag is 0, then, other than the fact that the obsolete size argument is dropped
func EpollCreate1(flag int) (epfd int, err error) {......}

// EpollCtl is used to add, modify, or remove entries in the interest list of the epoll.
// Valid values for the op argument are: EPOLL_CTL_ADD, EPOLL_CTL_MOD, and EPOLL_CTL_DEL.
// EpollEvent is an object telling the epoll what events we want to monitor for the fd.
// E.g. unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)}
func EpollCtl(epfd int, op int, fd int, event *EpollEvent) (err error) {......}

// EpollWait call system call waits for events on the epoll.
// Ready events will be injected to events
func EpollWait(epfd int, events []EpollEvent, msec int) (n int, err error) {......}

Epoller: a simple implementation

type epoll struct {
	fd          int
	connections map[int]net.Conn // net.Conn implements io.ReadWriter
	lock        *sync.RWMutex
}

func MkEpoll() (*epoll, error) {
	fd, err := unix.EpollCreate1(0)
	if err != nil {
		return nil, err
	}
	return &epoll{
		fd:          fd,
		lock:        &sync.RWMutex{},
		connections: make(map[int]net.Conn),
	}, nil
}

func socketFD(conn net.Conn) int {
	tcpConn := reflect.Indirect(reflect.ValueOf(conn)).FieldByName("conn")
	fdVal := tcpConn.FieldByName("fd")
	pfdVal := reflect.Indirect(fdVal).FieldByName("pfd")
	return int(pfdVal.FieldByName("Sysfd").Int())
}
func (e *epoll) Add(conn net.Conn) error {
	// Extract file descriptor associated with the connection
	fd := socketFD(conn)
        event := unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)}
	err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_ADD, fd, &event)
	if err != nil {
		return err
	}
	e.connections[fd] = conn // connections are protected by RWMutex
	return nil
}

func (e *epoll) Remove(conn net.Conn) error {
	fd := socketFD(conn)
	err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_DEL, fd, nil)
	if err != nil {
		return err
	}
	delete(e.connections, fd)
	return nil
}

func (e *epoll) Wait() ([]net.Conn, error) {
	events := make([]unix.EpollEvent, 100) // The max size of events retrieving from epoll
	n, err := unix.EpollWait(e.fd, events, 100)
	if err != nil {
		return nil, err
	}
	var connections []net.Conn
	for i := 0; i < n; i++ {
		conn := e.connections[int(events[i].Fd)]
		connections = append(connections, conn)
	}
	return connections, nil
}
var epoller *epoll

func wsHandler(w http.ResponseWriter, r *http.Request) {
	conn, _, _, err := ws.UpgradeHTTP(r, w) // Upgrade connection
	if err != nil {
		return
	}
        // When conn is added to epoller, this goroutine is done.
	if err := epoller.Add(conn); err != nil {
		log.Printf("Failed to add connection %v", err)
		conn.Close()
	}
}

func main() {
	// Increase resources limitations......
        // Enable pprof hooks.......

	// Start epoll
	var err error
	epoller, err = MkEpoll()
	if err != nil {
		panic(err)
	}

	go HandleConnection()

	http.HandleFunc("/", wsHandler)
	if err := http.ListenAndServe("0.0.0.0:8000", nil); err != nil {
		log.Fatal(err)
	}
}

WebSocket server

func HandleConnection() {
	for {
		connections, err := epoller.Wait()
		if err != nil {
			log.Printf("Failed to epoll wait %v", err)
			continue
		}
		for _, conn := range connections {
			if conn == nil {
				break
			}
                        // ReadClientData reads next data message from conn
			if msg, _, err := wsutil.ReadClientData(conn); err != nil {
				if err := epoller.Remove(conn); err != nil {
					log.Printf("Failed to remove %v", err)
				}
				conn.Close()
			} else {
				log.Printf("msg: %s", string(msg))
			}
		}
	}

Use one goroutine to serve whole active sockets.

Demo

Before

After

Discussion

Q1: What is the drawback of using epoll?

A trade-off between memory consumption and latency/throughput.

 

Q2: What use cases are suitable for mentioned optimizations?

For applications with non-busy socket connections or regardless of latency.

 

Q3: If my production fit the use case, can I adopt these optimizations?

Sure, but should be aware of connection storm and event burst.

Summary

- We went through the details of WebSocket application optimization.

    - It is all about network resource management efficiency.

    - So, we introduced the network I/O in Linux and adopted epoll.

 

- We illustrated how to use epoll in Go.

    - Use a single goroutine to monitor epoll, and then serve socket if need.

 

- Epoll is only useful in the network I/O, but the concept (red-black-tree) is helpful anywhere.

We are either progressing or retrograding all the while. There is no such thing as remaining stationary in this life. – James Freeman Clarke

Thanks for listening!

Epoll: The Hero Behind One Million WebSocket Connections in Go

By Jalex Chang

Epoll: The Hero Behind One Million WebSocket Connections in Go

Introduce the core methodology behind serving 1 million WebSocket connections in limited memory.

  • 3,018