A very simple intro to AWS Lambda & Scala

by Pawel Dolega @pdolega

Expectations

Serverless

http://www.bullshitbingo.net/

(... ) manifestation of how the intelligent combination of hosted services and self-managing infrastructure can result in significant improvements in development time and operating cost.

Serverless Architecture on AWS by Peter Sbarski

(... ) manifestation of how the intelligent combination of hosted services and self-managing infrastructure can result in significant improvements in development time and operating cost.

Serverless Architecture on AWS by Peter Sbarski

Moving away from servers and infrastructure concerns, as well as allowing the developer to primarily focus on code, is the ultimate goal behind serverless.

Serverless Architecture on AWS by Peter Sbarski

Someone else's problem

C = Ce + Ca

Complexity (C)

Ce - essential complexity

Ca - accidental complexity

stability, performance, reliability, availability, temporal coupling

Problems

scalability, elasticity, memory consumption, concurrency

state sharing, unreliable networking, complexity

readability, testability, correlation, throughput, latency

time to release, congestion, noisy neighbour, security

auditability, velocity, technical debt, runtime dependencies

distribution, reusability, dependency hell, spatial coupling

graceful degradation, transactions, data loss, repeatability

reproducibility, callback hell, mutable state access

hardware failure, network partition​, synchronization

...

Focus (more) on your

core business

No server is easier to manage than no server

Werner Vogels

Stateless

Event-driven

Compute code

Stateless

Why container may be not reused

It got scrapped

Existing one is squeezed under load (scalling)

Any other reason

Why container may be not reused

Why container got scrapped

Has not been used for some time

It failed miserably (unreachable?)

Any other reason

Why container got scrapped

Scalling down

Stateless

It keeps state (up to a point)

...but you should not rely on it

Event-driven

Pay per execution (100 ms)

One more thing

Free tier

1 mln requests

400,000 TB-seconds

Pricing example

384 MB / avg time 1 s

Why now

Google App Engine

Initial release: April 2008

Knowledge

by Jonas Bonér

Adopting Microservices at Netflix: Lessons for Architectural Design

Create a Separate Data Store for Each Microservice

Keep Code at a Similar Level of Maturity

Do a Separate Build for Each Microservice

Deploy in Containers

Treat Servers as Stateless

Technology

Containers

Services

Network speed

What serverless is not

Function

as

a

Service

Landscape

AWS Lambda

Azure Functions

Google Cloud Functions

Bluemix OpenWhisk

Let's see how to build something

*

HelloLambda.scala


class HelloLambda extends RequestStreamHandler {
  override def handleRequest(inputStr: InputStream, 
                             output: OutputStream, 
                             context: Context): Unit = {
    context.getLogger.log(s"Remaining time: " +
                           "${context.getRemainingTimeInMillis}ms")

    val writer = new OutputStreamWriter(output, "UTF-8")

    val input = Source.fromInputStream(inputStr).mkString
    val result = write(s"Input: ${input} fetched with ID: " +
        "${context.getAwsRequestId} (at ${formattedNow})")

    writer.write(result)
    context.getLogger.log(result)

    writer.flush()
  }
}

UberJar to S3

plugins.sbt










    addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

build.sbt


    ...
    scalaVersion := "2.12.2",
    retrieveManaged := true,
    libraryDependencies ++= Seq(
    
        // AWS api
        "com.amazonaws" % "aws-lambda-java-core" % "1.1.0",
        "com.amazonaws" % "aws-lambda-java-events" % "1.3.0",
        "org.json4s" %% "json4s-jackson" % "3.5.1",
        
        "org.scalatest" %% "scalatest" % "3.0.0" % Test,
        "com.amazonaws" % "aws-java-sdk-lambda" % "1.11.123" % Test
    )
    ...

    assemblyMergeStrategy in assembly := {
      case PathList("META-INF", xs @ _*) => MergeStrategy.discard
      case _ => MergeStrategy.first
    }

Generate and push jar

    
$ sbt bareHello/assembly  
[warn] Executing in batch mode.
[warn]   For better performance, hit [ENTER] to switch to interactive mode, or
[warn]   consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading global plugins from /home/pdolega/.sbt/0.13/plugins
[info] Loading project definition from /home/pdolega/projects/lambda/lambda-samples/project
...
[success] Total time: 6 s, completed Jun 17, 2017 12:54:37 AM


$ aws s3 mb s3://bare-lambda  
make_bucket: bare-lambda


$ aws s3 cp \
        ./target/scala-2.12/bare-hello-assembly-1.0-SNAPSHOT.jar \
        s3://bare-lambda/app.jar

Role & policy

    aws iam attach-role-policy \
        --role-name bare-hello-role \
        --policy-arn \
            arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

Configure function

$ aws lambda create-function \
    --function-name bare-lambda \
    --runtime java8 \
    --role arn:aws:iam::123456789012:role/bare-hello-role \
    --timeout 10 \
    --memory 256 \
    --handler com.virtuslab.lambda.hello.HelloLambda::handleRequest \
    --code S3Bucket=bare-lambda,S3Key=app.jar

$ aws lambda invoke --function-name bare-lambda output.txt
{
    "StatusCode": 200
}

$ cat output.txt 
"Input: {} fetched with ID: \
    b696b53a-52f3-11e7-8daa-8d98cda88cfb \
    (at 17-06-2017 12:27:15)"

Client call

LambdaClientSpec.scala






trait HelloLambdaClient {
  @LambdaFunction(functionName = "bare-lambda")
  def hello(param: String): String

  @LambdaFunction(functionName = "bare-lambda", 
                  invocationType = Event)
  def helloAsync(param: String): String
}

LambdaClientSpec.scala



class LambdaClientSpec  extends WordSpec with MustMatchers {
  "Invoking lambda function" should {
    "work for sync mode" in {

      val helloLambda = LambdaInvokerFactory.builder
        .lambdaClient(AWSLambdaClientBuilder.defaultClient)
        .build(classOf[HelloLambdaClient])

      val returnValue = helloLambda.hello("sync call")
      println(s">> Retrieved result is: ${returnValue}")

      returnValue must not be(null)
      returnValue must startWith("Input: ")

      val asyncReturn = helloLambda.helloAsync("a-sync call")
      println(s">> Retrieved result is: ${asyncReturn}")
    }
  }
}

Testing started at 9:45 AM ...
>> Retrieved result is: Input: "sync call" \
    fetched with ID: f4d4ade1-57e7-11e7-baae-8db3097073aa (at 23-06-2017 07:45:44)
>> Retrieved result is: null

But hey, I want to have REST

HttpLambda.scala








case class Response(body: Option[String] = None,
                    statusCode: Int = 200,
                    headers: Map[String, Any] = Map.empty)

HttpLambda.scala

    ...
    ...
    override def handleRequest(input: InputStream, 
                               output: OutputStream, 
                               context: Context): Unit = {
      context.getLogger.log(Source.fromInputStream(input).mkString)
        
      Try {
        val response = write(
          Response(body = Some(s"Hello lambda via API gateway ..."))
        )
        context.getLogger.log(s"Generated response is: ${response}")
    
        val writer = new OutputStreamWriter(output, "UTF-8")
        writer.write(response)
        writer.flush()
      }.recover {
        case e: Throwable => context.getLogger.log(s"exception? -> ${e}")
      }
    }

# API resource
$ aws apigateway create-rest-api --name lambda-api
$ aws apigateway get-rest-apis
$ aws apigateway get-resources --rest-api-id 184i4c0pjk
$ aws apigateway create-resource --rest-api-id 184i4c0pjk \
        --parent-id tecp4zmc0k --path-part hello

# API method
$ aws apigateway put-method --rest-api-id 184i4c0pjk 
    \--resource-id iwku45 
    \--http-method ANY 
    \--authorization-type NONE

# method integration with Lambda
$ aws apigateway put-integration --rest-api-id 184i4c0pjk \
        --resource-id iwku45 --http-method ANY --type AWS_PROXY \
        --integration-http-method POST \
        --uri arn:aws:apigateway:eu-west-1:lambda:path/2015-03-31/functions/\
              arn:aws:lambda:eu-west-1:123456789012:function:bare-lambda/invocations

# permissions
$ aws lambda add-permission --function-name http-lambda --statement-id "API-accesss" \
        --action lambda:InvokeFunction --principal apigateway.amazonaws.com \
        --source-arn arn:aws:execute-api:eu-west-1:123456789012:184i4c0pjk/*/*/hello

How to make your life easier

https://serverless.com/

Supports many langs but focused mainly on NodeJS

serverless.yml


service: aws-java-simple-http-endpoint

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: java8
  
package:
  artifact: build/distributions/aws-java-simple-http-endpoint.zip

functions:
  currentTime:
    handler: com.serverless.Handler
    events:
      - http:
          path: ping
          method: get

RequestHandler.java

package com.serverless;

import java.util.Date;
import java.util.HashMap;
import java.util.Map;

import org.apache.log4j.Logger;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

public class Handler implements RequestHandler<Map<String, Object>, ApiGatewayResponse> {

	private static final Logger LOG = Logger.getLogger(Handler.class);

	@Override
	public ApiGatewayResponse handleRequest(Map<String, Object> input, Context context) {
		LOG.info("received: " + input);
		Response responseBody = new Response("Hello, the current time is " + new Date());
		Map<String, String> headers = new HashMap<>();
		headers.put("X-Powered-By", "AWS Lambda & Serverless");
		headers.put("Content-Type", "application/json");
		return ApiGatewayResponse.builder()
				.setStatusCode(200)
				.setObjectBody(responseBody)
				.setHeaders(headers)
				.build();
	}
}

$ serverless deploy

Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading service .zip file to S3...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..............................
Serverless: Stack update finished...
Service Information
service: aws-java-simple-http-endpoint
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  GET - https://XXXXXXX.execute-api.us-east-1.amazonaws.com/dev/ping
functions:
  aws-java-simple-http-endpoint-dev-currentTime: \
  arn:aws:lambda:us-east-1:XXXXXXX:function:aws-java-simple-http-endpoint-dev-currentTime

Challice (Python)

https://github.com/awslabs/chalice

$ pip install chalice
$ chalice new-project helloworld && cd helloworld
$ cat app.py

from chalice import Chalice

app = Chalice(app_name="helloworld")

@app.route("/")
def index():
    return {"hello": "world"}



$ chalice deploy
...
Your application is available at: https://endpoint/dev

$ curl https://endpoint/dev
{"hello": "world"}

Quaich

https://github.com/quaich-serverless

*

Simple Lambda with Quaich

plugins.sbt





    addSbtPlugin(
        "codes.bytes" % "sbt-quartercask-lambda" % "0.0.4-SNAPSHOT"
    )

parameters:


    val s3Bucket = settingKey[String]("...")

    val s3KeyPrefix = settingKey[String]("...")

    val lambdaName = settingKey[String]("...")

    val role = settingKey[String]("...")

    val region = settingKey[String]("Required. ...")

    val awsLambdaTimeout = settingKey[Int]("...")

    val awsLambdaMemory = settingKey[Int]("...")

    val handlerName = settingKey[String]("...")

Optimize for developer's experience

create-automatically:




val createAutomatically = settingKey[Boolean](
    "Flag indicating if AWS infrastructure should be created " + 
    "automatically. If yes - objects like bucket, " +
    "lambda definition, api gateway would be automatically created. " +
    "Defaults to: false"
)

build.sbt

lazy val plainDemo = (project in file("plain-demo")).
  settings(commonSettings: _*).
  settings(quaichMacroSettings: _*).
  settings(
    name := "plain-demo",
    createAutomatically := true,

    awsLambdaMemory := 192,
    awsLambdaTimeout := 30,
    region := "eu-west-1",

    publishArtifact in (Compile, packageDoc) := false,

    libraryDependencies ++= Seq(
      // Lambda client libs
      "com.amazonaws" % "aws-java-sdk-lambda" % "1.11.123" % Test
    )
  ).
  dependsOn(util).
  enablePlugins(AWSLambdaPlugin)

PlainLambda.scala

@DirectLambda
class PlainLambda extends DirectLambdaHandler {

  val randomValue = Random.nextLong()

  var value = 0

  override protected def handleEvent(json: JValue, output: OutputStream)
                                    (implicit ctx: LambdaContext) {
    Thread.sleep(1500)
    value = value + 1

    writeJson(output, s"Success returned from request: ${ctx.awsRequestId}. " +
      s"Random value is: ${randomValue} " +
      s"remaining ms: ${ctx.remainingTimeInMillis} ms " +
      s"VALUE IS: ${value}")
  }
}

sbt deployLambda

$ sbt plainSample/deployLambda
[info] Loading global plugins from /home/pdolega/.sbt/0.13/plugins                                                                                                                                                  
[info] Loading project definition from /home/pdolega/projects/.../project                                                                                                                   
[info] Set current project to quaich-other-samples (in build file:/home/pdolega/...
...
[info] Packaging /home/pdolega/projects/.../plain-sample-assembly-0.0.4-SNAPSHOT.jar ...
[info] Done packaging.
[info] Inferred lambda handlers are: \
    HandlerName(codes.bytes.quaich.samples.plain.PlainLambda::handleRequest)
[info] Role plain-sample was not found. It will be now created...
[info] Role plain-sample has been created \
    [ arn:aws:iam::123456789012:role/plain-sample ]...
[info] Created role policy [ plain-sample ]...
[info] Role policy successfully attached...
[info] Bucket codes.bytes.plain-sample doesn't exists, attempting to create it
[=========================================]   100%   (18.89/18.89 MB) Lambda JAR -> S3S3
[info] Creating new AWS Lambda function 'plain-sample'
[info] Created Lambda: arn:aws:lambda:eu-west-1:123456789012:function:plain-sample
[success] Total time: 261 s, completed Jun 19, 2017 12:21:00 PM

First run

$ sbt plainDemo/deployLambda
[info] Loading global plugins from /home/pdolega/.sbt/0.13/plugins                                                                                                                                                  
[info] Loading project definition from /home/pdolega/projects/.../project                                                                                                                   
[info] Set current project to quaich-other-samples (in build file:/home/pdolega/...
...
[info] Packaging /home/pdolega/projects/.../plain-demo-assembly-0.0.4-SNAPSHOT.jar ...
[info] Done packaging.
[info] Inferred lambda handlers are: \
    HandlerName(codes.bytes.quaich.samples.plain.PlainLambda::handleRequest)
[info] Role arn:aws:iam::123456789012:role/plain-demo has been found and will be used...
[info] Bucket codes.bytes.plain-demo exists and is accessible
[=========================================]   100%   (18.89/18.89 MB) Lambda JAR -> S3S3
[info] Updating existing AWS Lambda function 'plain-demo'
[info] Successfully updated function code: \
    arn:aws:lambda:eu-west-1:123456789012:function:plain-demo
[info] Successfully updated function configuration: \
    arn:aws:lambda:eu-west-1:123456789012:function:plain-demo
[info] Updated lambda arn:aws:lambda:eu-west-1:123456789012:function:plain-demo
[success] Total time: 231 s, completed Jun 19, 2017 11:10:19 AM

Next runs

$ aws lambda invoke --function-name plain-sample output.txt                                                                                                                                    
{                                                                                                                                                                                                                   
    "StatusCode": 200                                                                                                                                                                                               
}    

"Success returned from request: bf003465-57b8-11e7-82c9-3b4dce1f0c8e. \
Random value is: 2317584354818852822 remaining ms: 24874 ms VALUE IS: 1"

PlainClientSpec Demo

PlainClientSpec.scala: 

    "multiple calls - one after the other" in {
      val helloLambda = LambdaInvokerFactory.builder
        .lambdaClient(AWSLambdaClientBuilder.defaultClient)
        .build(classOf[HelloLambdaClient])

      println(helloLambda.fetchNames())       // 1
      println(helloLambda.fetchNames())       // 2
      println(helloLambda.fetchNames())       // 3
      println(helloLambda.fetchNames())       // 4
      println(helloLambda.fetchNames())       // 5
      println(helloLambda.fetchNames())       // 6
      println(helloLambda.fetchNames())       // 7

      val start2 = System.currentTimeMillis()
      println(helloLambda.fetchNamesAsyncNonsense()) // 8
      println(s"Execution (async) took: ${System.currentTimeMillis() - start2} ms")
    }

PlainLambda.scala

@DirectLambda
class PlainLambda extends DirectLambdaHandler {

  val randomValue = Random.nextLong()

  var value = 0

  override protected def handleEvent(json: JValue, output: OutputStream)
                                    (implicit ctx: LambdaContext) {
    Thread.sleep(1500)
    value = value + 1

    writeJson(output, s"Success returned from request: ${ctx.awsRequestId}. " +
      s"Random value is: ${randomValue} " +
      s"remaining ms: ${ctx.remainingTimeInMillis} ms " +
      s"VALUE IS: ${value}")
  }
}

Success returned: bb7d0d1... Random value is: 7502495642 ... ms VALUE IS: 3
Success returned: bc8e2e8... Random value is: 7502495642 ... ms VALUE IS: 4
Success returned: bdb326d... Random value is: 7502495642 ... ms VALUE IS: 5
Success returned: bec7559... Random value is: 7502495642 ... ms VALUE IS: 6
Success returned: bfe4acb... Random value is: 7502495642 ... ms VALUE IS: 7
Success returned: c11760d... Random value is: 7502495642 ... ms VALUE IS: 8
Success returned: c251b5c... Random value is: 7502495642 ... ms VALUE IS: 9
()
Execution (async) took: 617 ms

What about hAkking?

NameActor.scala:


sealed trait NameMsg
final case class AddNew(lang: String, 
                        replyTo: ActorRef[Array[String]]) 
                        extends NameMsg


class NameActor extends Actor.MutableBehavior[NameMsg] {

  private var names: List[String] = Nil

  override def onMessage(msg: NameMsg): Behavior[NameMsg] = {
    msg match {
      case AddNew(lang, replyTo) =>
        names = lang :: names
        replyTo ! names.toArray
    }

    this
  }
}

HTTP API with Quaich

Simplistic Slack integration

plugins.sbt:






    
    addSbtPlugin(
        "codes.bytes" % "sbt-quartercask-lambda" % "0.0.4-SNAPSHOT"
    )
    addSbtPlugin(
        "codes.bytes" % "sbt-quartercask-api-gateway" % "0.0.4-SNAPSHOT"
    )

build.sbt:

lazy val commonSettings = Seq(
  libraryDependencies ++= Seq(
    "codes.bytes" %% "quaich-http" % projectVersion
  ),
  ...
)

lazy val slackbotDemo = (project in file("slackbot-demo")).
  settings(commonSettings: _*).
  settings(quaichMacroSettings: _*).
  settings(
    name := "slackbot-demo",
    createAutomatically := true,

    awsLambdaMemory := 192,
    awsLambdaTimeout := 30,
    region := "eu-west-1",
  ).
  dependsOn(util).
  enablePlugins(AWSLambdaPlugin, AwsApiGatewayPlugin)

@LambdaHTTPApi
class SlackbotLambda {
  post[String]("/ping") { reqCtx =>
    val params = (for {
      body <- reqCtx.request.body.toSeq
      paramValue <- body.split("&")
      keyValue <- toKeyValue(paramValue.split("="))
    } yield {
      keyValue
    }).toMap

    val userHandlePart = params
        .get("user_name")
        .map(u => s"@${u}").getOrElse("")

    complete(s"Great! It works ${userHandlePart}")
  }

  get("/ping") { reqCtx =>
    complete("Great! Dummy ping works via GET")
  }
  ...
}

sbt deployHttpApi

$ sbt slackbotDemo/deployHttpApi
[info] Loading global plugins from /home/pdolega/.sbt/0.13/plugins
[info] Initiating API deployment...
[info] Given stack does not exist on AWS: slackbot-demo, it will be created...
[info] Progress: CREATE_IN_PROGRESS...
...
[info] Progress: CREATE_IN_PROGRESS...
[info] Progress: CREATE_IN_PROGRESS...
[info] Stack sync finished with status: CREATE_COMPLETE
[info] Detailed log of events during update of stack: slackbot-demo
[info]  AWS::CloudFormation::Stack: CREATE_IN_PROGRESS (User Initiated)...
[info]  AWS::ApiGateway::RestApi: CREATE_IN_PROGRESS...
[info]  AWS::ApiGateway::RestApi: CREATE_IN_PROGRESS (Resource creation Initiated)...
[info]  AWS::ApiGateway::RestApi: CREATE_COMPLETE...
[info]  AWS::ApiGateway::Resource: CREATE_IN_PROGRESS...
[info]  AWS::ApiGateway::Resource: CREATE_IN_PROGRESS (Resource creation Initiated)...
[info]  AWS::Lambda::Permission: CREATE_IN_PROGRESS...
[info]  AWS::ApiGateway::Resource: CREATE_COMPLETE...
[info]  AWS::Lambda::Permission: CREATE_IN_PROGRESS (Resource creation Initiated)...
[info]  AWS::ApiGateway::Method: CREATE_IN_PROGRESS...
[info]  AWS::ApiGateway::Method: CREATE_IN_PROGRESS (Resource creation Initiated)...
[info]  AWS::ApiGateway::Method: CREATE_COMPLETE...
[info]  AWS::ApiGateway::Deployment: CREATE_IN_PROGRESS...
[info]  AWS::ApiGateway::Deployment: CREATE_IN_PROGRESS (Resource creation Initiated)...
[info]  AWS::ApiGateway::Deployment: CREATE_COMPLETE...
[info]  AWS::Lambda::Permission: CREATE_COMPLETE...
[info]  AWS::CloudFormation::Stack: CREATE_COMPLETE...
[info] Stack arn:aws:cloudformation:eu-west-1:123456789012:stack/... successfully deployed...
[info] ========================================================================
[info] >>> Your API has been deployed at:
[info] >>> https://dogvhqiy64.execute-api.eu-west-1.amazonaws.com/dev/
[info] ========================================================================
[info]          
[success] Total time: 29 s, completed Jun 20, 2017 6:55:42 PM

SlackbotLambda demo

HTTP DSL

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

Cloudformation

Huge possibilities here

https://github.com/tptodorov/sbt-cloudformation

Cloudformation

Lambda backed web app ?

Possibilities are endless

Thank you

Contact

Software Engineer /

Entrepreneur

twitter: @pdolega

github: github.com/pdolega​

Quaich project:

https://github.com/quaich-serverless

Quaich samples:

https://github.com/quaich-serverless/quaich-samples

Samples of dev features:

https://github.com/pdolega/quaich-other-samples

Samples not using Quaich (plain Lambda):

https://github.com/pdolega/lambda-samples