Cloud computing demo

CCFE PhD Coding Discussion Group

Jonathan Shimwell

By the end of this demo you should be more familiar with creating cloud resources, connect to and terminating instances.

Buy what you need, when you need it

Why chose cloud computing over traditional computing

Access to huge scale (both computing and storage)

Access to global infrastructure

Grow and shrink services with demand

Rent

"Computers in the future may weigh no more than 1.5 tons"

1949, Popular Mechanics.

Regions, availability zones, edge locations

EC2

Compute

Lambda

Function

S3

Storage

Dynamo

Database

EC2

Compute

How to create an EC2 instance - GUI method

ec2

EC2

Compute

How to create an EC2 instance - script method

1. Setup the Command Line Interface

 

2. Create the instance

 

3. Connect to the instance

EC2

Compute

Setting up command line interface

1:   conda install -c conda-forge awscli

2:   aws configure (get your access keys from the IAM users security credentials, text )

3. aws ec2 create-security-group 
      --group-name devenv-sg --description "EC2 dev"

                returns security group "sg-0a78c970"

5. aws ec2 authorize-security-group-ingress 
     --group-name devenv-sg 
     --protocol tcp --port 22 
     --cidr 0.0.0.0/0
6. aws ec2 create-key-pair 
     --key-name devenv-key 
     --query 'KeyMaterial' 
     --output text > devenv-key.pem

Change to current ip for extra security

7. chmod 400 devenv-key.pem

4.   aws_sg=sg-0a78c970

EC2

Compute

Using the command line interface to

create an EC2 instance

1: Find an AMI Amazon Machine Image (AMI)

    Ubuntu 16.04 based in us-west-2 region is

    ami-835b4efa

2: aws ec2 run-instances 
    --image-id ami-835b4efa 
    --security-group-ids $aws_sg 
    --count 1 --instance-type t2.micro 
    --key-name devenv-key 
    --query 'Instances[0].InstanceId'

returns instance id "i-007bd56fd6b98ed71"

3: aws_id=i-007bd56fd6b98ed71

EC2

Compute

Using the command line interface to

connect to the instance

5. ssh -i devenv-key.pem ubuntu@$aws_ip

1:aws ec2 describe-instances 
   --instance-ids $aws_id 
   --query 'Reservations[0].Instances[0].PublicIpAddress'

                  returns public ip address "54.183.247.192"

3. scp -i devenv-key.pem test.txt ubuntu@$aws_ip:~/test.txt

7. aws ec2 terminate-instances  
     --instance-ids $aws_id

4. scp -i devenv-key.pem ubuntu@$aws_ip:~/test.txt test.txt

2. aws_ip=54.183.247.192

6. exit

EC2

Compute

Great I now know how to create a EC2 instance automatically, but what can I do ...

EC2

Compute

Python web frameworks and endpoints

Flask is a Python Microframwork

 

The entire idea of Flask is to map URL paths to some logic that you will run

from flask import Flask
from flask import Flask, jsonify, render_template, request, Response


app = Flask(__name__)


@app.route('/helloworld')
def helloworld():
    return 'Hello world, Flask is working'


@app.route('/hello/<name>')
def hello(name):
    return 'Hello'+name

@app.route('/index')
def index():
    return render_template('index.html')

EC2

Compute

Example web apps

http://ec2-52-40-223-220.us-west-2.compute.amazonaws.com/transmuteandplot/0Ti48_2.73281839873e-14_Ti49_9.99492430306e-11_Ti50_5.16735478117e-09_Ti51_1.53321848768e-15_Ti52_1.19212760776e-23_V50_4.29181961298e-13_V51_1.97358404743e-09_V52_3.46691934117e-13_V53_8.30432327022e-14_V54_2.31092912555e-16_Cr50_1.60............

EC2

Compute

Making an MPI cluster then terminate it

The creation of MPI clusters based on AWS is demonstration using StarCluster

EC2

Compute

Creating the cluster, logging in and testing

Once StarCluster is installed and the config file is setup the commands for cluster management are ...

starcluster start myfirstcluster

starcluster sshmaster --user sgeadmin myfirstcluster

startcluster stop myfirstcluster

starcluster terminate myfirstcluster

Perhaps this is a good time to talk about prices

Lambda Function

AWS Lambda is a computation service that lets you run code without servers.

 

 


from random import randint
import time


def lambda_handler(event, context):
    time.sleep(2)    
    return  randint(0, 9)

Scaling Up Lambda Function

Lambda functions can be triggered by a url call.

 

Asynchronously

 

 

import requests
import eventlet
import urllib3
import asyncio
from aiohttp import ClientSession

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.read()

async def run(r):
    url = "https://0vi2dvwp0c.execute-api.us-east-1.amazonaws.com/prod/number"
    tasks = []
    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(fetch(url.format(i), session))
            tasks.append(task)
        responses = await asyncio.gather(*tasks)
        # you now have all response bodies in this variable
        print(responses)

def print_responses(result):
    print(result)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(10000))
loop.run_until_complete(future)

source activate novapy

subl /media/jshim/Data/vboxshare.lambda

Amazon Simple Storage Service is storage for the Internet.

 

S3 buckets can be accessed from EC2 instances, Lambda functions, websites and url endpoints.

 

Read and write access can be configured to suit the needs of the bucket

S3 Storage buckets

https://s3-us-west-2.amazonaws.com/shimwellstorage/xs/n/li/6/jeff3.2/105

Cloud

By Jonathan Shimwell