Why NSQ

Data Bus Arch

Light

Golang

Can run on ARM

Usage based on Message

Stable

Message order(1/2)

Fault Tolerance

Backup

Simple

UDP,Nanomsg,Websocket,XMPP

More Rules

Get Started

NSQ.IO

A realtime  messaging platform

Distributed and decentralized topologies

Message Robust Backup and Transmit

Rich Library

Get 

Start NSQ  =>

<= Start CMD Reader

Send Msg  =>

Started

Client Library

Go

Javascript

Python

Java

.NET

Erlang

Perl

C

Ruby

Closure

Scala

Haskell

C++:Boost

C++:Qt

Go-nsq

        cfg := nsq.NewConfig()

	consumer, err := nsq.NewConsumer("hello", "world", cfg)

	if err != nil {
		fmt.Println("Error")
	}

	consumer.AddHandler(nsq.HandlerFunc(func(m *nsq.Message) error {
		// handle the message
		fmt.Println("%v", m)
		return nil
	}))

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	for {
		err = consumer.ConnectToNSQD("localhost:4150")
		if err != nil {
			fmt.Println("Error connecting,reconnecting in 15s")
			time.Sleep(time.Second * 15)
			continue
		} else {
			break
		}
	}
	for {
		select {
		case <-consumer.StopChan:
			return
		case <-sigChan:
			consumer.Stop()
		}
	}

NSQ's Design

Goroutine&Channel

Using independent worker to manage msg in every topic and channel

Using Channel as msg inner pipe

Topic & Channel

Topic

a topic is a distinct stream of messages

a topic has one or more channels

topics and channels are created at runtime

Channel

a channel is a distinct stream of messages

a channel can be connected by multi client

channels are created at runtime

Same Channel

broadcast

Diff Channel

loadbalance

Client_1

Client_2

Publisher

for msg := range c.incomingMsgChan {
	select {
	case c.memoryMsgChan <- msg:
	default:
		err := WriteMessageToBackend(&msgBuf, msg, c.backend)
		if err != nil {
			// ... handle errors ...
		}
	}
}

<= Backup & Robust

select {
		case msg = <-memoryMsgChan:
		case buf = <-backendChan:
			msg, err = decodeMessage(buf)
			if err != nil {
				t.ctx.nsqd.logf("ERROR: failed to decode message - %s", err)
				continue
			}
		case <-t.channelUpdateChan:
			chans = chans[:0]
			t.RLock()
			for _, c := range t.channelMap {
				chans = append(chans, c)
			}
			t.RUnlock()
			if len(chans) == 0 || t.IsPaused() {
				memoryMsgChan = nil
				backendChan = nil
			} else {
				memoryMsgChan = t.memoryMsgChan
				backendChan = t.backend.ReadChan()
			}
			continue
		case pause := <-t.pauseChan:
			if pause || len(chans) == 0 {
				memoryMsgChan = nil
				backendChan = nil
			} else {
				memoryMsgChan = t.memoryMsgChan
				backendChan = t.backend.ReadChan()
			}
			continue
		case <-t.exitChan:
			goto exit
		}

Order

Topic 

 Channel

 Client

Nsqd

＊

1

／

／

／

／

1

＊

\

\

1

＊

Lookupd & NsqAdmin

Nsqadmin

Topic 

 Channel

 Channel

 Channel

Client

|

Client

/

\

|

/

\

Client

Topic 

***

/

/

-

\

-

\

\

Nsqd

Nsqd

Nsqd

Nsqd

Lookupd

\

\

-

-

|

/

/

-

-

***

\

***

\

***

/

\

Tools

Nsq_tail 

-channel="": NSQ channel
-consumer-opt=: option to passthrough to nsq.Consumer (may be given multiple times
               , http://godoc.org/github.com/nsqio/go-nsq#Config)
-lookupd-http-address=: lookupd HTTP address (may be given multiple times)
-max-in-flight=200: max number of messages to allow in flight
-n=0: total messages to show (will wait if starved)
-nsqd-tcp-address=: nsqd TCP address (may be given multiple times)
-reader-opt=: (deprecated) use --consumer-opt
-topic="": NSQ topic
-version=false: print version string

Nsq_to_File

-channel string: nsq channel (default "nsq_to_file")
-consumer-opt value: option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-datetime-format string: strftime compatible format for <DATETIME> in filename format (default "%Y-%m-%d_%H")
-filename-format string: output filename format (<TOPIC>, <HOST>, <PID>, <DATETIME>, <REV> are replaced. <REV> is increased when file already exists) (default "<TOPIC>.<HOST><REV>.<DATETIME>.log")
-gzip: gzip output files.
-gzip-compression int: (deprecated) use --gzip-level, gzip compression level (1 = BestSpeed, 2 = BestCompression, 3 = DefaultCompression) (default 3)
-gzip-level int: gzip compression level (1-9, 1=BestSpeed, 9=BestCompression) (default 6)
-host-identifier string: value to output in log filename in place of hostname. <SHORT_HOST> and <HOSTNAME> are valid replacement tokens
-lookupd-http-address value: lookupd HTTP address (may be given multiple times)
-max-in-flight int: max number of messages to allow in flight (default 200)
-nsqd-tcp-address value: nsqd TCP address (may be given multiple times)
-output-dir string: directory to write output files to (default "/tmp")
-reader-opt value: (deprecated) use --consumer-opt
-rotate-interval duration: rotate the file every duration
-rotate-size rotate-size: rotate the file when it grows bigger than rotate-size bytes
-skip-empty-files: Skip writing empty files
-topic value: nsq topic (may be given multiple times)
-topic-pattern string: Only log topics matching the following pattern (default ".*")
-topic-refresh duration: how frequently the topic list should be refreshed (default 1m0s)
-version: print version string

Nsq_to_Http

-channel="nsq_to_http": nsq channel
-consumer-opt=: option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-content-type="application/octet-stream": the Content-Type used for POST requests
-get=: HTTP address to make a GET request to. '%s' will be printf replaced with data (may be given multiple times)
-http-timeout=20s: timeout for HTTP connect/read/write (each)
-http-timeout-ms=20000: (deprecated) use --http-timeout=X, timeout for HTTP connect/read/write (each)
-lookupd-http-address=: lookupd HTTP address (may be given multiple times)
-max-backoff-duration=2m0s: (deprecated) use --consumer-opt=max_backoff_duration,X
-max-in-flight=200: max number of messages to allow in flight
-mode="round-robin": the upstream request mode options: multicast, round-robin, hostpool
-n=100: number of concurrent publishers
-nsqd-tcp-address=: nsqd TCP address (may be given multiple times)
-post=: HTTP address to make a POST request to.  data will be in the body (may be given multiple times)
-reader-opt=: (deprecated) use --consumer-opt
-round-robin=false: (deprecated) use --mode=round-robin, enable round robin mode
-sample=1: % of messages to publish (float b/w 0 -> 1)
-status-every=250: the # of requests between logging status (per handler), 0 disables
-throttle-fraction=1: (deprecated) use --sample=X, publish only a fraction of messages
-topic="": nsq topic
-version=false: print version string

Nsq_to_Nsq

-channel="nsq_to_nsq": nsq channel
-consumer-opt=: option to passthrough to nsq.Consumer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-destination-nsqd-tcp-address=: destination nsqd TCP address (may be given multiple times)
-destination-topic="": destination nsq topic
-lookupd-http-address=: lookupd HTTP address (may be given multiple times)
-max-backoff-duration=2m0s: (deprecated) use --consumer-opt=max_backoff_duration,X
-max-in-flight=200: max number of messages to allow in flight
-mode="round-robin": the upstream request mode options: round-robin (default), hostpool
-nsqd-tcp-address=: nsqd TCP address (may be given multiple times)
-producer-opt=: option to passthrough to nsq.Producer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-reader-opt=: (deprecated) use --consumer-opt
-require-json-field="": for JSON messages: only pass messages that contain this field
-require-json-value="": for JSON messages: only pass messages in which the required field has this value
-status-every=250: the # of requests between logging status (per destination), 0 disables
-topic="": nsq topic
-version=false: print version string
-whitelist-json-field=: for JSON messages: pass this field (may be given multiple times)

Nsq_to_nanomsg

Nsq_to_udp

Nsq_to_websocket

Nsq_to_mongo

Nsq_to_hls

Pipeline Game

With Docker

TCP Protocol

CMD for TCP

Pub

Sub

Identify

Fin

Req

What will contribe

libnsqpp

libnsqt

nsq-blend

nanomsg ,udp,websocket protocol

.Net client

libnsq

Thanks

Q/A