Kotlin Coroutines in Practice:

writing asynchronous RabbitMQ client

Vyacheslav Artemyev

Agenda

  • RabbitMQ introduction
  • Declare exchange and queue in different manners
  • Reactive vs Coroutines
  • Coroutines base concepts
  • Publisher implementation
  • Consumer implementation
  • Conclusion

RabbitMQ architecture

RabbitMQ architecture

The problem #1

channel.exchangeDeclare("exchange", "direct")

channel.queueDeclare("queue", true, true, false, emptyMap())

channel.queueBind("queue", "exchange", "")


/** Some useful actions **/

Declare exchange & queue

Solution v1

val exchangeDeclaration = CompletableFuture.supplyAsync {
    channel.exchangeDeclare("exchange", "direct")
}

val queueDeclaration = CompletableFuture.supplyAsync {
    channel.queueDeclare("queue", false, false, false, emptyMap())
}

CompletableFuture
    .allOf(exchangeDeclaration, queueDeclaration)
    .thenApply { channel.queueBind("queue", "exchange", "") }

Solution v2

val exchangeDeclaration = CompletableFuture.supplyAsync {
    channel.exchangeDeclare("exchange", "direct")
}

val queueDeclaration = CompletableFuture.supplyAsync {
    channel.queueDeclare("queue", false, false, false, emptyMap())
}

val tasks = listOf(exchangeDeclaration, queueDeclaration)

val allFutures = CompletableFuture.allOf(*tasks.toTypedArray())

val results = allFutures.thenApply { v -> tasks.map { it.join() } }.get()

Solution v3

val exchangeDeclaration: CompletableFuture<Command> = channel
    .asyncCompletableRpc(
        AMQP.Exchange.Declare.Builder()
            .exchange("exchange")
            .type("direct")
            .build()
    )

val queueDeclaration: CompletableFuture<Command> = channel
    .asyncCompletableRpc(
        AMQP.Queue.Declare.Builder()
            .queue("queue")
            .durable(true)
            .exclusive(true)
            .autoDelete(false)
            .build()
    )

Reactive solution

val exchangeDeclaration = sender.declareExchange(ExchangeSpecification.exchange())
val queueDeclaration = sender.declareQueue(QueueSpecification.queue())

Flux.concat(exchangeDeclaration, queueDeclaration)
    .then(sender.bind(BindingSpecification.binding()))
    .doOnError { e -> LOGGER.error("Boom", e) }
    .subscribe { r -> LOGGER.info("Done") }

Reactive under the hood

AMQP.Queue.Declare declare = new AMQImpl.Queue.Declare.Builder()
    .queue(specification.getName())
    .durable(specification.isDurable())
    .exclusive(specification.isExclusive())
    .autoDelete(specification.isAutoDelete())
    .arguments(specification.getArguments())
    .build();

return channelMono.map(channel -> {
    try {
        return channel.asyncCompletableRpc(declare);
    } catch (IOException e) {
        throw new RabbitFluxException("Error during RPC call", e);
    }
})
.flatMap(future -> Mono.fromCompletionStage(future))
.flatMap(command -> Mono.just((AMQP.Queue.DeclareOk) command.getMethod()))
.publishOn(resourceManagementScheduler);

Asynchronous programming

Asynchrony, in computer programming, refers to the occurrence of events independent of the main program flow and ways to deal with such events. These may be "outside" events such as the arrival of signals, or actions instigated by a program that take place concurrently with program execution, without the program blocking to wait for results.

https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)

Coroutines

Coroutines are computer-program components that generalize subroutines for non-preemptive multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations.

Use cases

You do not want threads:

  • You have a lot of mutable states
  • Mobile/Desktop UI apps, etc

You cannot afford threads:

  • They are expensive to keep and to switch
  • High-load server-side, micro-tasks, etc

You cannot do threads:

  • Your code is a single thread only
  • JS/web target, μC, etc

Why Coroutines?

Reactive:

  • You need to change return type and this type should be propagated

Coroutines:

  • You don't need to change signature*
fun publishWithConfirm(message: OutboundMessage): Boolean

Function signatures

fun publishWithConfirm(message: OutboundMessage): Observable<Boolean>

* Just a bit

suspend

Why Coroutines?

Functions stay "the same"

Localize the changes to a single entry point 

(instead of changing how you do thing everywhere)

ONE LESS THING TO LEARN

Why Coroutines?

Reactive:

  • Not easy to debug

Coroutines:

  • Easier to debug

Debugging

Why Coroutines?

for ((message) in list) {
    publishWithConfirm(message)
}

Direct loops

Why Coroutines?

try {
    publishWithConfirm(message)
} catch (e: Exception) {
    ...
}

Direct exception handing

Why Coroutines?

Direct higher-order functions

(1..times).map {
     async { 
        publisher.publishWithConfirm(createMessage("Hello #$it")) 
    }
}.awaitAll()

//forEach, let, apply, repeat, filter, map, use etc

Why Coroutines?

Easy to manage

withContext(Dispatchers.IO) {
    publisher.publishWithConfirm(createMessage("Hello #$it")) 
}

Reactive:

Coroutines

...          
.subscribeOn(Schedulers.single())
.observeOn(Schedulers.elastic())
...

[main @coroutine#2] DEBUG c.v.t.p.PublisherTest - Coroutine #1 started 
[main @coroutine#3] DEBUG c.v.t.p.PublisherTest - Coroutine #2 started 
[main @coroutine#4] DEBUG c.v.t.p.PublisherTest - Coroutine #3 started 
[main @coroutine#5] DEBUG c.v.t.p.PublisherTest - Coroutine #4 started 
[main @coroutine#6] DEBUG c.v.t.p.PublisherTest - Coroutine #5 started 
[main @coroutine#2] DEBUG c.v.t.p.ConfirmPublisher - The message Sequence Number: 1 
[rabbitmq-nio] DEBUG c.v.t.p.AckListener - deliveryTag = [1], multiple = [false], positive = [true] 
[main @coroutine#2] INFO  c.v.t.p.ConfirmPublisher - Message Hello #1 has sent 
[main @coroutine#3] DEBUG c.v.t.p.ConfirmPublisher - The message Sequence Number: 2 
[main @coroutine#3] INFO  c.v.t.p.ConfirmPublisher - Message Hello #2 has sent 
[main @coroutine#4] DEBUG c.v.t.p.ConfirmPublisher - The message Sequence Number: 3 
[rabbitmq-nio] DEBUG c.v.t.p.AckListener - deliveryTag = [2], multiple = [false], positive = [true]

-Dkotlinx.coroutines.debug

CoroutineName("my_coroutine")

Why Coroutines?

Easy to debug

Coroutine basics

Suspending function

A suspending function — a function that is marked with suspend modifier. It may suspend execution of the code without blocking the current thread of execution by invoking other suspending functions.

suspend fun publishWithConfirm(message: OutboundMessage): Boolean {
    // some logic is here
}

fun publishWithConfirm(message: OutboundMessage, callback: Continuation<Boolean>): Any? {
    // some logic is here
}

Continuation

Being passed

Continuation

Coroutines (like futures!) use callbacks at their low level, but allow asynchronous programming in direct style.

interface Continuation<in T> {
    val context: CoroutineContext
    fun resumeWith(result: Result<T>)
}

Continuation

Coroutines (like futures!) use callbacks at their low level, but allow asynchronous programming in direct style.

suspend fun method(): Boolean {
  return suspendCoroutine { continuation ->
	//some actions here
    continuation.resume(true)
    
    //another logic here
    continuation.resumeWithException(RuntimeException("Boom"))
  }
}

Let's try it

/**
 * Asynchronously send a method over this channel.
 * ...
 */
CompletableFuture<Command> asyncCompletableRpc(Method method) throws IOException;

/**
 * Awaits for completion of the completion stage without blocking a thread.
 *
 * This suspending function is cancellable.
 */
public suspend fun <T> CompletionStage<T>.await(): T {
    // fast path when CompletableFuture is already done (does not suspend)
    ...
    // slow path -- suspend
    return //The magic is here
}

Creating a queue

suspend fun Channel.declareQueue(queueSpecification: QueueSpecification): DeclareOk {
    val queueDeclaration = AMQP.Queue.Declare.Builder()
        .queue(queueSpecification.name)
        .durable(queueSpecification.durable)
        .exclusive(queueSpecification.exclusive)
        .autoDelete(queueSpecification.autoDelete)
        .arguments(queueSpecification.arguments)
        .build()

    return this.asyncCompletableRpc(queueDeclaration).await().method as DeclareOk
}

Let's try it!

@Test
fun `declare queue test`() {
    val queueName = "declare_queue_test"
    factory.newConnection().use { connection ->
        connection.createChannel().use { channel ->
            channel.declareQueue(QueueSpecification(queueName))
            assertTrue { getQueues().find { it.name == queueName } != null }
        }
    }
}

Doesn't compile: suspend function should be called only from a coroutine or another suspend function

GlobalScope

public object GlobalScope : CoroutineScope {
    override val coroutineContext: CoroutineContext
        get() = EmptyCoroutineContext
}

Returns immediately, coroutine works in a background thread pool

Combine it!

@Test
fun `bind queue test`() {
    factory.newConnection().use { connection ->
        connection.createChannel().use { channel ->
            runBlocking {

                channel.declareExchange(ExchangeSpecification("exchange")) //#1
    
                channel.declareQueue(QueueSpecification("queue")) //#2
    
                channel.bindQueue(BindQueueSpecification("queue", "exchange")) //#3

            }
        }
    }
}

Concurrency in Coroutines

Concurrency in Coroutines

Expectations

Coroutine code

"Normal" code

Coroutine builder

Multi async

Proceeding linearly

Concurrency in Coroutines

Reality

Coroutine code

"Normal" code

Coroutine builder

Proceeding linearly

Proceeding 

linearly

Under the hood

Coroutines: State machine object

suspend fun initialize(channel: Channel) {

    channel.declareExchange(ExchangeSpecification("exchange"))

    channel.declareQueue(QueueSpecification("queue"))

    channel.bindQueue(BindQueueSpecification("queue", "exchange"))

}

Under the hood

Coroutines: State machine object

Data for each state

Index of current state

suspend fun initialize(channel: Channel, continuation: Continuation) {

    val sm = object : CoroutineImpl { ... }

    switch(sm.lable) {
        case 0:
            channel.declareExchange(ExchangeSpecification("exchange"), sm)
        case 1:
            channel.declareQueue(QueueSpecification("queue"), sm)
        case 2:
            channel.bindQueue(BindQueueSpecification("queue", "exchange"), sm)
    }

}

Call stack 

Coroutine builder

Action 1

Action 2

Suspend coroutine

unwind

Continuation in heap

Coroutine builders

Synchronous code

Suspendable code

Coroutine builders

public fun CoroutineScope.launch(
    context: CoroutineContext = EmptyCoroutineContext,
    start: CoroutineStart = CoroutineStart.DEFAULT,
    block: suspend CoroutineScope.() -> Unit
): Job {
    ...
}

Launch

public fun <T> CoroutineScope.async(
    context: CoroutineContext = EmptyCoroutineContext,
    start: CoroutineStart = CoroutineStart.DEFAULT,
    block: suspend CoroutineScope.() -> T
): Deferred<T> {
    ...
}

Async

Runnable??

Callable??

Inside Coroutine

public interface CoroutineScope {
    public val coroutineContext: CoroutineContext
}

CoroutineScope

interface CoroutineContext {
    abstract operator fun <E : Element> get(key: Key<E>): E?
    abstract fun minusKey(key: Key<*>): CoroutineContext
    open operator fun plus(context: CoroutineContext): CoroutineContext
    ...
}

CoroutineContext

interface Element : CoroutineContext

interface Key<E : Element>

Improve it!

@Test
fun `bind queue test`() {
    factory.newConnection().use { connection ->
        connection.createChannel().use { channel ->
            runBlocking {
                val exchange: Deferred = async { channel.declareExchange(ExchangeSpecification("exchange")) }
    
                val queue: Deferred = async { channel.declareQueue(QueueSpecification("queue")) }
    
                awaitAll(exchange, queue)
    
                channel.bindQueue(BindQueueSpecification("queue", "exchange"))
            } 
        }
    }
}

Structured concurrency

...

coroutineScope {

    val exchange = async { channel.declareExchange(ExchangeSpecification("exchange")) }

    val queue = async { channel.declareQueue(QueueSpecification("queue")) }

    awaitAll(exchange, queue)

    channel.bindQueue(BindQueueSpecification("queue", "exchange"))
}

...

Improve it again!

suspend fun Channel.declareQueue(queueSpecification: QueueSpecification): AMQP.Queue.DeclareOk {
    val channel = this
    val queueDeclaration = AMQP.Queue.Declare.Builder()
        .queue(queueSpecification.name)
        .durable(queueSpecification.durable)
        .exclusive(queueSpecification.exclusive)
        .autoDelete(queueSpecification.autoDelete)
        .arguments(queueSpecification.arguments)
        .build()

    return withContext(resourceManagementDispatcher) {
        channel.asyncCompletableRpc(queueDeclaration).await().method as AMQP.Queue.DeclareOk
    }
}

val resourceManagementDispatcher = newSingleThreadContext("ResourceManagementDispatcher")

Dispatcher

GlobalScope.launch(context = /** default **/ Dispatchers.Default) {}
  • Dispatchers.Default - CPU bound work
  • Dispatchers.IO - intensive IO blocking operations
  • Dispatchers.Unconfined - Risky business
  • Private thread pools can be created with newSingleThreadContext and newFixedThreadPoolContext.
  • An arbitrary Executor can be converted to a dispatcher with asCoroutineDispatcher extension function.

Coroutine dispatchers direct traffic

(to threads)

Dispatcher

SubscribeOn

Initial Context

ObserveOn

withContext

Recap: CompletableFuture

val exchangeDeclaration = CompletableFuture.supplyAsync {
    channel.exchangeDeclare("exchange", "direct")
}.exceptionally(ex -> {
  LOGGER.error("Boom", e)
  return null;
});

val queueDeclaration = CompletableFuture.supplyAsync {
    channel.queueDeclare("queue", false, false, false, emptyMap())
}.exceptionally(ex -> {
  LOGGER.error("Boom", e)
  return null;
});

val allOf = CompletableFuture.allOf(exchangeDeclaration, queueDeclaration)
allOf.get()

channel.queueBind("queue", "exchange", "")

Recap: Reactive

val exchangeDeclaration = sender.declareExchange(ExchangeSpecification.exchange())

val queueDeclaration = sender.declareQueue(QueueSpecification.queue())

Flux.concat(exchangeDeclaration, queueDeclaration)
    .then(sender.bind(BindingSpecification.binding()))
    .doOnError { e -> LOGGER.error("Boom", e) }
    .subscribe { r -> LOGGER.info("Done") }


coroutineScope {

    val exchange = async { channel.declareExchange(ExchangeSpecification()) }

    val queue = async { channel.declareQueue(QueueSpecification()) }

    awaitAll(exchange, queue)

    channel.bindQueue(BindQueueSpecification())
}

Recap: Coroutines

The problem #2

String message = "Hello World!";
channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");

Publish a message *

* You must not lose messages

Transactions

ch.txSelect();
for (int i = 0; i < MSGCOUNT; ++i) {
        ch.basicPublish("", 
                        QUEUENAME,
                        PERSISTENTBASIC,
                        "Hello".getBytes());
        ch.txCommit();
}
  • Transactions are needlessly heavy: every commit requires a fsync(), which takes a lot of time to complete

  • Transactions are blocking: the publisher has to wait for the broker to process each message

Problems:

Enter Confirms

volatile SortedSet<Long> unconfirmedSet =
    Collections.synchronizedSortedSet(new TreeSet());


ch.setConfirmListener(new ConfirmListener() {
    public void handleAck(long seqNo, boolean multiple) {
        if (multiple) {
            unconfirmedSet.headSet(seqNo+1).clear();
        } else {
            unconfirmedSet.remove(seqNo);
        }
    }
    public void handleNack(long seqNo, boolean multiple) {
        // handle the lost messages somehow
    }
});

Enter Confirms

ch.confirmSelect();

for (long i = 0; i < MSGCOUNT; ++i) {
     unconfirmedSet.add(ch.getNextPublishSeqNo());
     ch.basicPublish("", 
                     QUEUENAME, 
                     PERSISTENT_BASIC,
                     "Hello".getBytes());
}

Confirm channel

class ConfirmChannel internal constructor(private val channel: Channel) : Channel by channel {
    init {
        channel.confirmSelect()
    }

    fun publisher() = ConfirmPublisher(this)
}

fun Connection.createConfirmChannel(): ConfirmChannel = ConfirmChannel(this.createChannel())

Publisher

class ConfirmPublisher internal constructor(private val channel: Channel) {
    private val continuations = ConcurrentHashMap<Long, Continuation<Boolean>>()

    init {
        channel.addConfirmListener(AckListener(continuations))
    }

    ...
}

Listener

class AckListener(
        private val continuations: ConcurrentHashMap<Long, Continuation<Boolean>>
) : ConfirmListener {

    ... 

    private fun handle(deliveryTag: Long, multiple: Boolean, ack: Boolean) {
        if (multiple) {
            ...
            continuations.remove(tag)?.resume(ack)
            ...
        } else {
            continuations.remove(deliveryTag)?.resume(ack)
            ...
        }
    }
}

Publisher

class ConfirmPublisher internal constructor(private val channel: Channel) {

    ...

    suspend fun publishWithConfirm(message: OutboundMessage): Boolean {
        val messageSequenceNumber = channel.nextPublishSeqNo
        logger.debug { "The message Sequence Number: $messageSequenceNumber" }
        return suspendCancellableCoroutine { continuation ->
            continuations[messageSequenceNumber] = continuation
            continuation.invokeOnCancellation { continuations.remove(messageSequenceNumber) }
            cancelOnIOException(continuation) {
                message.run { 
                    channel.basicPublish(exchange, routingKey, properties, msg.toByteArray()) 
                }
            }
        }
    }

}

Coroutine

val channel = connection.createConfirmChannel()

val publisher = channel.publisher()
channel.declareQueue(QueueSpecification("queue"))

val acks = (1..times).map {
    async { 
        publisher.publishWithConfirm(createMessage("Hello #$it"))
        LOGGER.info("Message {} sent successfully", 
                            String(r.getOutboundMessage().getBody()))
    }
}.awaitAll()

JavaRx

val confirmations = sender.sendWithPublishConfirms(Flux.range(1, count)
    .map({ i -> OutboundMessage("", queue, "Message_$i".toByteArray()) }))

sender.declareQueue(QueueSpecification.queue(queue))
    .thenMany(confirmations)
    .doOnError({ e -> LOGGER.error("Send failed", e) })
    .subscribe({ r ->
        if (r.isAck()) {
            LOGGER.info("Message {} sent successfully", 
                            String(r.getOutboundMessage().getBody()))
        }
    })

Results

The White Rabbit

Benchmark                                         (numberOfMessages)  Mode  Cnt        Score   Error  Units
ConfirmPublisherBenchmark.sendWithPublishConfirm                   1  avgt    2      104.172          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                  10  avgt    2      598.625          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                 100  avgt    2     3845.833          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm                1000  avgt    2    36108.709          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm               10000  avgt    2   392132.353          us/op
ConfirmPublisherBenchmark.sendWithPublishConfirm              100000  avgt    2  4098567.349          us/op

Reactor Rabbitmq

Benchmark                                               (nbMessages)  Mode  Cnt        Score   Error  Units
SenderBenchmark.sendWithPublishConfirms                            1  avgt    2      697.424          us/op
SenderBenchmark.sendWithPublishConfirms                           10  avgt    2     1306.490          us/op
SenderBenchmark.sendWithPublishConfirms                          100  avgt    2     4819.441          us/op
SenderBenchmark.sendWithPublishConfirms                         1000  avgt    2    39597.671          us/op
SenderBenchmark.sendWithPublishConfirms                        10000  avgt    2   373226.865          us/op
SenderBenchmark.sendWithPublishConfirms                       100000  avgt    2  3900685.520          us/op

Consumer

public interface Channel extends ShutdownNotifier, AutoCloseable {

    ...

    String basicConsume(
        String queue, 
        boolean autoAck, 
        DeliverCallback deliverCallback, 
        CancelCallback cancelCallback
    );

    void basicAck(
        long deliveryTag, 
        boolean multiple
    );
}

Consumer v1

class ConfirmConsumer 
        internal constructor(private val amqpChannel: Channel, amqpQueue: String) {
    private val continuations = LinkedBlockingQueue<Continuation<Delivery>>()

    init {
        amqpChannel.basicConsume(amqpQueue, false,
                DeliverCallback { consumerTag, message -> continuations.take().resume(message) },
                CancelCallback { logger.info { "Cancelled" } }
        )
    }

    suspend fun consumeWithConfirm(
        handler: (Delivery) -> Unit) = 
        coroutineScope {
            val delivery = suspendCancellableCoroutine<Delivery> { continuations.put(it) }
            handler(delivery)
            amqpChannel.basicAck(delivery.envelope.deliveryTag, false)
        }
}

Channel "quality of service"

public interface Channel ... {

    ... 

    void basicQos(
        int prefetchCount, 
        boolean global
    );
}

Consumer v1

class ConfirmConsumer 
       constructor(private val amqpChannel: Channel, amqpQueue: String, prefetchSize: Int) {
    private val continuations = LinkedBlockingQueue<Continuation<Delivery>>(prefetchSize)

    init {
        amqpChannel.basicConsume(amqpQueue, false,
                DeliverCallback { consumerTag, message -> continuations.take().resume(message) },
                CancelCallback { logger.info { "Cancelled" } }
        )
    }

    suspend fun consumeWithConfirm(
        handler: (Delivery) -> Unit) = 
        coroutineScope {
            val delivery = suspendCancellableCoroutine<Delivery> { continuations.put(it) }
            handler(delivery)
            amqpChannel.basicAck(delivery.envelope.deliveryTag, false)
        }
}

Channels

Deferred values provide a convenient way to transfer a single value between coroutines. Channels provide a way to transfer a stream of values

https://github.com/Kotlin/kotlinx.coroutines/blob/master/docs/channels.md

suspending

suspending

Channels

private val continuations = Channel<Delivery>()
private lateinit var consTag: String

Compilation error

suspend fun consumeWithConfirm(
    handler: (Delivery) -> Unit)
= coroutineScope {
    val delivery = continuations.receive()
    handler(delivery)
    AMQPChannel.basicAck(delivery.envelope.deliveryTag, false)
}

init {
    consTag = amqpChannel.basicConsume(amqpQueue, false,
        { consumerTag, message ->
            if (consumerTag == consTag) {
                continuations.send(message)
            }
        },
        { consumerTag ->
            if (consumerTag == consTag) {
                logger.info { "Consumer $consumerTag has been cancelled" }
                continuations.cancel()
            }
        }
    )
}

Consumer v2

init {
    consTag = AMQPChannel.basicConsume(AMQPQueue, false,
        { consumerTag, message ->
            if (consumerTag == consTag) {
                GlobalScope.launch { continuations.send(message) }
            }
        },
        { consumerTag ->
            if (consumerTag == consTag) {
                logger.info { "Consumer $consumerTag has been cancelled" }
                continuations.cancel()
            }
        }
    )
}

Consumer v3

init {
    consTag = AMQPChannel.basicConsume(AMQPQueue, false,
        { consumerTag, message ->
            if (consumerTag == consTag) {
                continuations.sendBlocking(message)
            }
        },
        { consumerTag ->
            if (consumerTag == consTag) {
                logger.info { "Consumer $consumerTag has been cancelled" }
                continuations.cancel()
            }
        }
    )
}

Recap: Reactive

val queueDeclaration = sender.declareQueue(QueueSpecification.queue(queue))
val messages = receiver.consumeAutoAck(queue)
queueDeclaration
    .thenMany(messages)
    .take(3)
    .subscribe({ m ->
        LOGGER.info("Received message {}", String(m.getBody()))
    })

subscribeOn ?

observeOn ?

Reacp: Coroutine

val handler = { m -> LOGGER.info("Received message {}", String(m.getBody())) }
channel.declareQueue(QueueSpecification())
val consumer = channel.consumer()
for (i in 1..3) consumer.consumeWithConfirm(handler)

Conclusion

  • The code is more readable and maintainable
  • An ecosystem is big and is getting bigger
  • Easy to integrate to your codebase
  • "Free optimisation"
  • Upcoming integrations with Spring and so on

Questions?

The White Rabbit

Twitter

Telegram

Made with Slides.com