Everything I Know

about S3 pre-signed URLs

Luciano Mammino (@loige)

2023-09-28, Manchester

META_SLIDE!

𝕏 loige

😎📱 "Hey, Luciano. I have a startup idea..."

Cool non-techy friend giving you a call!

𝕏 loige

𝕏 loige

𝕏 loige

𝕏 loige

𝕏 loige

WTF?

𝕏 loige

Sorry, I still have to implement the profile picture upload feature... 😅

𝕏 loige

$ ~ whoami

👋 I'm Luciano (🇮🇹🍕🍝🤌)

👨‍💻 Senior Architect @ fourTheorem (Dublin 🇮🇪)

📔 Co-Author of Node.js Design Patterns  👉

🤝 Let's connect 👉 linktr.ee/loige

Always re-imagining

We are a pioneering technology consultancy focused on AWS and serverless

We can help with:
Cloud Migrations
Training & Cloud enablement
Building serverless applications
Cutting cloud costs

𝕏 loige

𝕏 loige

🤔
"upload" feature
in a web app...

𝕏 loige

What's an upload, really?

𝕏 loige

OK, what protocol?

𝕏 loige

Structure of an HTTP request

POST /profilepic HTTP/1.1
Host: api.meower.com
Content-Type: text/plain
Content-Length: 9
 
Some data

 Method

 Path

 Version

Headers

Body

𝕏 loige

What if it's a binary (like a picture)?

PUT /profilepic HTTP/1.1
Host: api.meower.com
Content-Type: image/jpeg
Content-Length: 2097852

����JFIFHH������"��
���Dl��FW�'6N�()H�'p��FD3 [...]

read 2097852 bytes

𝕏 loige

Using Lambda

Lambda proxy integration*

* JSON Based protocol mapping to HTTP (JSON Request / JSON Response)

𝕏 loige

Lambda proxy integration (request)

{
  "resource": "/profilepic",
  "path": "/profilepic",
  "httpMethod": "POST",
  "headers": {
    "Content-Type": "text/plain",
    "Content-Length": "9",
    "Host": "api.meower.com",
  },
  "body": "Some data"
}

𝕏 loige

Lambda proxy integration (request - picture)

{
  "resource": "/profilepic",
  "path": "/profilepic",
  "httpMethod": "PUT",
  "headers": {
    "Content-Type": "image/jpeg",
    "Content-Length": "2097852",
    "Host": "api.meower.com",
  },
  "body": "????????"
}

𝕏 loige

Lambda proxy integration (request - picture)

{
  "resource": "/profilepic",
  "path": "/profilepic/upload",
  "httpMethod": "PUT",
  "headers": {
    "Content-Type": "image/jpeg",
    "Content-Length": "2097852",
    "Host": "api.meower.com",
  },
  "isBase64Encoded": true,
  "body": "/9j/4AAQSkZJRgABAQEASABIAAD/2w[...]"
}

𝕏 loige

1. Parse request (JSON)

2. Decode body (Base64)

3. Validation / resize

4. ...

Lambda proxy integration request

/profilepic

𝕏 loige

1. Parse request (JSON)

2. Decode body (Base64)

3. Validation / resize

4. ...

Lambda proxy integration request

/profilepic

😎

𝕏 loige

Is this a good solution? 🙄

𝕏 loige

Limitations...

  • API Gateway requests timeout: 30 sec
  • API Gateway payload: max 10 MB
  • Lambda timeout: max 15 mins
  • Lambda payload size: max 6 MB

Upload: 6 MB / 30 sec

𝕏 loige

Is this a good solution? 🙄

... not really!

What about supporting big images or even videos?

𝕏 loige

An alternative approach

✅ Long lived connection

✅ No size limit

𝕏 loige

🤔 SERVERS...

𝕏 loige

S3 pre-signed URLs 😱

An S3 built-in feature

to authorize operations (download, upload, etc) on a bucket / object

using time-limited authenticated URLs

𝕏 loige

Using S3 pre-signed URLs for upload

* yeah, this can be a Lambda as well 😇

*

𝕏 loige

Using S3 pre-signed URLs for upload

* yeah, this can be a Lambda as well 😇

*

𝕏 loige

Using S3 pre-signed URLs for upload

* yeah, this can be a Lambda as well 😇

*

𝕏 loige

Using S3 pre-signed URLs for upload

* yeah, this can be a Lambda as well 😇

*

𝕏 loige

... and we can also use it for downloads! 🤩

𝕏 loige

Using S3 pre-signed URLs for download

𝕏 loige

Using S3 pre-signed URLs for download

𝕏 loige

Using S3 pre-signed URLs for download

𝕏 loige

Using S3 pre-signed URLs for download

𝕏 loige

⚠️ VERY important details!

The server never really talks with S3!

The server actually creates the signed URL by itself!

We will see later what's the security model around this idea!

𝕏 loige

Is this a good solution? 🙄

✅ It's a managed feature (a.k.a. no servers to manage)

✅ We can upload and download arbitrarily big files with no practical limits*

✅ Reasonably simple and secure

👍 Seems good to me!

* objects in S3 are "limited" to 5TB (when using multi-part upload), 5 GB otherwise.

𝕏 loige

Generating our first pre-signed URL

$ aws s3 presign \ 
    s3://finance-department-bucket/2022/tax-certificate.pdf
https://s3.amazonaws.com/finance-department-bucket/2022/tax-certificate.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3SGQVQG7FGA6KKA6%2F20221104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221104T140227Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4751b4d9787314fd6da4d55

Whoever has this URL can download the tax certificate!

𝕏 loige

What's in a pre-signed URL

https://s3.amazonaws.com/finance-department-bucket/2022/tax-certificate.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3SGQVQG7FGA6KKA6%2F20221104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221104T140227Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4751b4d9787314fd6da4d55

𝕏 loige

What's in a pre-signed URL

  • https://s3.amazonaws.com

  • /finance-department-bucket

  • /2022/tax-certificate.pdf

  • ?X-Amz-Algorithm=AWS4-HMAC-SHA256

  • &X-Amz-Credential=AKIA3SGQXQG7XXXYKKA6%2F20221104...

  • &X-Amz-Date=20221104T140227Z

  • &X-Amz-Expires=3600

  • &X-Amz-SignedHeaders=host

  • &X-Amz-Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4...

What if I change this to /passwords.txt?

𝕏 loige

👿

𝕏 loige

Pre-signed URLs validation

  • https://s3.amazonaws.com

  • /finance-department-bucket

  • /2022/tax-certificate.pdf

  • ?X-Amz-Algorithm=AWS4-HMAC-SHA256

  • &X-Amz-Credential=AKIA3SGQXQG7XXXYKKA6%2F20221104...

  • &X-Amz-Date=20221104T140227Z

  • &X-Amz-Expires=3600

  • &X-Amz-SignedHeaders=host

  • &X-Amz-Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4...

Are these credentials still valid?

Is the URL expired?

Does the signature match the data in the request?

𝕏 loige

🤓
Once a pre-signed URL is generated you cannot edit it without breaking it

Photo by CHUTTERSNAP on Unsplash

⚠️ Also note that you can use a pre-signed URL as many times as you want until it expires

𝕏 loige

🔐 Permissions

Anyone with valid credentials can create a pre-signed URL (client side)

valid credentials = Role, User, or Security Token

The generated URL inherits the permissions of the credentials used to generate it

This means you can generate pre-signed URLs for things you don't have access to 😅

𝕏 loige

aws s3 presign s3://manchester/hello.txt
https://manchester.s3.amazonaws.com/hello.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3ABCVQG7FGA6KKA6%2F20221115%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20221115T182036Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=75749c92d94d03e411e7bbf64419f2af09301d1791b0df54c639137c715f7888

😱

I swear I don't even know if this bucket exists or who owns it!

𝕏 loige

Pre-signed URLs are validated at request time

𝕏 loige

👩‍💻

Creating a pre-signed URL programmatically

 

AWS SDK for JavaScript v3

𝕏 loige

import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

const s3Client = new S3Client()

const command = new GetObjectCommand({
  Bucket: "some-bucket",
  Key: "some-object"
})

const preSignedUrl = await getSignedUrl(s3Client, command, {
  expiresIn: 3600
})

console.log(preSignedUrl)

𝕏 loige

📦
Uploading a file

using pre-signed URLs

𝕏 loige

2 Options: PUT & POST 🤨

𝕏 loige

PUT Method

PUT <preSignedURL> HTTP/1.1
Host: <bucket>.s3.<region>.amazonaws.com
Content-Length: 2097852

����JFIFHH������"��
���Dl��FW�'6N�()H�'p��FD3 [...]

𝕏 loige

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

const s3Client = new S3Client()

const command = new PutObjectCommand({
  Bucket: "some-bucket",
  Key: "some-object"
})

const preSignedUrl = await getSignedUrl(s3Client, command, {
  expiresIn: 3600
})

console.log(preSignedUrl)

Only difference with the previous example

𝕏 loige

PUT Method - Limitations

You cannot set a limit on the upload size (max of 5 TB)! *

You can limit the Content-Type but you can specify exactly one

* Unless you know the exact size in advance

𝕏 loige

POST method

It uses the multipart/form-data encoding (form upload)

Gives more freedom to the client to shape the request (Content-Type, file name, etc)

It uses a policy mechanism to define the "rules" of what can be uploaded

E.g. you can limit the supported mime types and provide a maximum file size

You can use it to upload from a web form and even configure the redirect URL

It's not really a URL but more of a pre-signed form!

𝕏 loige

POST / HTTP/1.1
Host: <bucket>.s3.amazonaws.com
Content-Type: multipart/form-data; boundary=9431149156168
Content-Length: 2097852

--9431149156168
Content-Disposition: form-data; name="key"

picture.jpg
--9431149156168
Content-Disposition: form-data; name="X-Amz-Credential"

AKIA3SGABCDXXXA6KKA6/20221115/eu-west-1/s3/aws4_request
--9431149156168
Content-Disposition: form-data; name="Policy"

eyJleHBpcmF0aW9uIjoiMjAyMi0xMS0xNVQyMDo0NjozN1oiLCJjb25kaXRpb25zIjpbWyJj[...]
--9431149156168
Content-Disposition: form-data; name="X-Amz-Signature"

2c1da0001dfec7caea1c9fb80c7bc8847f515a9e4483d2942464f48d2f827de7
--9431149156168
Content-Disposition: form-data; name="file"; filename="MyFilename.jpg"
Content-Type: image/jpeg

����JFIFHH������"��
���Dl��FW�'6N�()H�'p��FD3[...]
--9431149156168--

𝕏 loige

POST method Policy

A JSON object (Base64 encoded) that defines the upload rules (conditions) and the expiration date

This is what gets signed: you cannot alter the policy without breaking the signature

{
  "expiration": "2022-11-15T20:46:37Z",
  "conditions": [
    ["content-length-range", 0, 5242880],
    ["starts-with", "$Content-Type", "image/"],
    {"bucket": "somebucket"},
    {"X-Amz-Algorithm": "AWS4-HMAC-SHA256"},
    {"X-Amz-Credential": "AKIA3SGABCDXXXA6KKA6/20221115/eu-west-1/s3/aws4_request"},
    {"X-Amz-Date": "20221115T194637Z"},
    {"key": "picture.jpg"}
  ]
}

𝕏 loige

import { S3Client } from '@aws-sdk/client-s3'
import { createPresignedPost } from '@aws-sdk/s3-presigned-post'

const { BUCKET_NAME, OBJECT_KEY } = process.env
const s3Client = new S3Client()

const { url, fields } = await createPresignedPost(s3Client, {
  Bucket: 'somebucket',
  Key: 'someobject',
  Conditions: [
    ['content-length-range', 0, 5 * 1024 * 1024] // 5 MB max
  ],
  Fields: {
    success_action_status: '201',
    'Content-Type': 'image/png'
  },
  Expires: 3600
})

console.log({ url, fields })

𝕏 loige

// you can use `url` and `fields` to generate an HTML form

const code = `<h1>Upload an image to S3</h1>
  <form action="${url}" method="post" enctype="multipart/form-data">
    ${Object.entries(fields).map(([key, value]) => {
      return `<input type="hidden" name="${key}" value="${value.replace(/"/g, '&quot;')}">`
    }).join('\n')}
    <div><input type="file" name="file" accept="image/png"></div>
    <div><input type="submit" value="Upload"></div>
  </form>`

𝕏 loige

Limitations and quirks 🤷‍♀️

It supports only 1 file (cannot upload multiple files in one go)

The file field must be the last entry in the form
(S3 will ignore every other field after the file) 

From the browser (AJAX) you need to enable CORS on the bucket

𝕏 loige

Should I PUT or should I POST? 🎸

PUT is simpler but definitely more limited

POST is slightly more complicated (and less adopted) but it's more flexible

You should probably put some time into learning POST and use that!

𝕏 loige

Pre-signed URLs for other operations

S3 pre-signed URLs are not limited to GET, PUT or POST operations

You can literally create pre-signed URLs for any command
(DeleteObject, ListBuckets, MultiPartUpload, etc...)

𝕏 loige

Do you need moar examples? 😼

𝕏 loige

... In summary 👩‍🏫

S3 pre-signed URLs are a great way to authorise operations on S3

They are generally used to implement upload/download features

The signature is created client-side so you can sign anything. Access is validated at request time

This is not the only solution, you can also use the JavaScript SDK from the frontend and get limited credentials from Cognito (Amplify makes that process simpler)

For upload you can use PUT and POST, but POST is much more flexible

💬 PS: Meower.com doesn't really exist... but... do you want to invest?! It's a great idea, trust me!

𝕏 loige

Cover photo by Kelly Sikkema on Unsplash

THANKS! 🙌

It's a wrap!

𝕏 loige

Everything I know about S3 pre-signed URLs - AWS ComSum Manchester 2023

By Luciano Mammino

Everything I know about S3 pre-signed URLs - AWS ComSum Manchester 2023

Almost every web application at some point needs a way to upload or download files… and no one seems to enjoy building reliable and scalable upload/download servers… and for good reasons too! In fact, you’ll probably need to manage long-running connections and handle files that can be quite large (i.e videos). If you are running a fully serverless backend using API Gateway and Lambda, you probably know that you are limited in terms of payload size and execution time, so things get even more complicated there. In all these cases you should consider offloading this problem to S3 by using S3 pre-signed URLs. Pre-signed URLs are a fantastic tool to handle file download and upload directly in S3 in a managed and scalable fashion. But all that glitters is not gold and S3 pre-signed URLs come with quite a few gotchas… So in this talk, we will explore some use cases, see some potential implementations of S3 pre-signed URLs and uncover some of the gotchas that I discovered while using them. By the end of this talk, you should know exactly when to use pre-signed URLs and how to avoid most of the many mistakes I made with them!

  • 732