Cloud computing demo
CCFE PhD Coding Discussion Group
Jonathan Shimwell
By the end of this demo you should be more familiar with creating cloud resources, connect to and terminating instances.
Buy what you need, when you need it
Why chose cloud computing over traditional computing
Access to huge scale (both computing and storage)
Access to global infrastructure
Grow and shrink services with demand
Rent
"Computers in the future may weigh no more than 1.5 tons"
1949, Popular Mechanics.
Regions, availability zones, edge locations
EC2
Compute
Lambda
Function
S3
Storage
Dynamo
Database
EC2
Compute
How to create an EC2 instance - GUI method
ec2
EC2
Compute
How to create an EC2 instance - script method
1. Setup the Command Line Interface
2. Create the instance
3. Connect to the instance
EC2
Compute
Setting up command line interface
1: conda install -c conda-forge awscli
2: aws configure (get your access keys from the IAM users security credentials, text )
3. aws ec2 create-security-group
--group-name devenv-sg --description "EC2 dev"
returns security group "sg-0a78c970"
5. aws ec2 authorize-security-group-ingress
--group-name devenv-sg
--protocol tcp --port 22
--cidr 0.0.0.0/0
6. aws ec2 create-key-pair
--key-name devenv-key
--query 'KeyMaterial'
--output text > devenv-key.pem
Change to current ip for extra security
7. chmod 400 devenv-key.pem
4. aws_sg=sg-0a78c970
EC2
Compute
Using the command line interface to
create an EC2 instance
2: aws ec2 run-instances
--image-id ami-835b4efa
--security-group-ids $aws_sg
--count 1 --instance-type t2.micro
--key-name devenv-key
--query 'Instances[0].InstanceId'
returns instance id "i-007bd56fd6b98ed71"
3: aws_id=i-007bd56fd6b98ed71
EC2
Compute
Using the command line interface to
connect to the instance
5. ssh -i devenv-key.pem ubuntu@$aws_ip
1:aws ec2 describe-instances --instance-ids $aws_id --query 'Reservations[0].Instances[0].PublicIpAddress'
returns public ip address "54.183.247.192"
3. scp -i devenv-key.pem test.txt ubuntu@$aws_ip:~/test.txt
7. aws ec2 terminate-instances --instance-ids $aws_id
4. scp -i devenv-key.pem ubuntu@$aws_ip:~/test.txt test.txt
2. aws_ip=54.183.247.192
6. exit
EC2
Compute
Great I now know how to create a EC2 instance automatically, but what can I do ...
EC2
Compute
Python web frameworks and endpoints
Flask is a Python Microframwork
The entire idea of Flask is to map URL paths to some logic that you will run
from flask import Flask
from flask import Flask, jsonify, render_template, request, Response
app = Flask(__name__)
@app.route('/helloworld')
def helloworld():
return 'Hello world, Flask is working'
@app.route('/hello/<name>')
def hello(name):
return 'Hello'+name
@app.route('/index')
def index():
return render_template('index.html')
EC2
Compute
Example web apps
http://ec2-52-40-223-220.us-west-2.compute.amazonaws.com/transmuteandplot/0Ti48_2.73281839873e-14_Ti49_9.99492430306e-11_Ti50_5.16735478117e-09_Ti51_1.53321848768e-15_Ti52_1.19212760776e-23_V50_4.29181961298e-13_V51_1.97358404743e-09_V52_3.46691934117e-13_V53_8.30432327022e-14_V54_2.31092912555e-16_Cr50_1.60............
EC2
Compute
Making an MPI cluster then terminate it
The creation of MPI clusters based on AWS is demonstration using StarCluster
EC2
Compute
Creating the cluster, logging in and testing
Once StarCluster is installed and the config file is setup the commands for cluster management are ...
starcluster start myfirstcluster
starcluster sshmaster --user sgeadmin myfirstcluster
startcluster stop myfirstcluster
starcluster terminate myfirstcluster
Perhaps this is a good time to talk about prices
Lambda Function
AWS Lambda is a computation service that lets you run code without servers.
from random import randint
import time
def lambda_handler(event, context):
time.sleep(2)
return randint(0, 9)Scaling Up Lambda Function
Lambda functions can be triggered by a url call.
Asynchronously
import requests
import eventlet
import urllib3
import asyncio
from aiohttp import ClientSession
async def fetch(url, session):
async with session.get(url) as response:
return await response.read()
async def run(r):
url = "https://0vi2dvwp0c.execute-api.us-east-1.amazonaws.com/prod/number"
tasks = []
async with ClientSession() as session:
for i in range(r):
task = asyncio.ensure_future(fetch(url.format(i), session))
tasks.append(task)
responses = await asyncio.gather(*tasks)
# you now have all response bodies in this variable
print(responses)
def print_responses(result):
print(result)
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(10000))
loop.run_until_complete(future)source activate novapy
subl /media/jshim/Data/vboxshare.lambda
Amazon Simple Storage Service is storage for the Internet.
S3 buckets can be accessed from EC2 instances, Lambda functions, websites and url endpoints.
Read and write access can be configured to suit the needs of the bucket
S3 Storage buckets
Cloud
By Jonathan Shimwell
