How to send gzipped requests
with boto3
Luciano Mammino (@loige)
AWS UserGroup Dublin
2021-07-01
Get these slides!
Let me introduce myself first...
I'm Luciano (🇮🇹🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin 🇮🇪) 👨💻
Co-Author of Node.js Design Patterns 👉
We are business focused technologists that deliver.
Accelerated Serverless | AI as a Service | Platform Modernisation
We are hiring: do you want to work with us?
In the previous episodes...
SLIC WATCH: fth.link/slic-watch
SLIDES: fth.link/o11y-no-pain
⚠️ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland)
Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).
@wake_me_up_bot
This guy was failing... a lot!
for event in payload['logEvents']:
# ...
cw_client.put_metric_data(...)Sending 1 metric per log line... 🙈
[
{
'MetricName': 'SomeMetric1',
'Dimensions': [
{
'Name': 'Dimension1Name',
'Value': 'Dimension1Value'
},
# Up to other 9 dimensions here ...
],
'Unit': 'Count',
'Values': [217, 220, 221], # Up to 150 items here ...
'Timestamp': 1624290910000
},
# Up to other 19 metric items here ...
]⚠️ Payload size
limit: 40 KB
🤔
Maybe boto3 gzips the data automatically! 😏
import boto3
import gzip
endpoint_url = "http://localhost:8000/"
cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False)
cw_client.put_metric_data(
MetricData = [
{
'MetricName': 'TEST_BOTO',
'Dimensions': [
{
'Name': 'APP_VERSION',
'Value': '1.0'
},
],
'Unit': 'None',
'Value': 17
},
],
Namespace='BotoTest'
)
... and there is no magic flag like GzipPayload=True! 😭
Can we extend boto3 somehow? 🤔
boto3 has an event system! 🤩
import boto3
lambda_client = boto3.client('lambda')
# our event handler
def add_xtrace_header(request, **kwargs):
request.headers.add_header('x-trace-id', 'trace-trace')
# get the event system for the lambda_client
event_system = lambda_client.meta.events
# attach an event handler to the client for
event_system.register('before-sign.lambda.Invoke', add_xtrace_header)
# invoke a lambda function
lambda_client.invoke(FunctionName='my-function')Event naming convention
<event-type>.<service-name>.<operation-name>
provide-client-params request-created before-sign before-send response-received
s3 cloudwatch lambda ...
ListBuckets PutMetricData Invoke ...
You can be a 🌟 too!
* (every event) after-call.*.* (all responses) after-call.lambda.* (all responses for lambda)
OK, now we know enough to write our own event handler for gzip compression! 🤓
import boto3
import gzip
cw_client = boto3.client('cloudwatch')
event_system = cw_client.meta.events
# Gzip handler
def gzip_request_body(request, **kwargs):
request.headers.add_header('Content-Encoding', 'gzip')
gzipped_body = gzip.compress(request.body)
request.data = gzipped_body
# registers the custom handler
event_system.register('before-sign.cloudwatch.PutMetricData',
gzip_request_body)
cw_client.put_metric_data(...)
Other potential use cases
- Log data before/after it is sent to AWS
- Add additional headers to AWS requests (tracing)
- Additional input validation (your custom rules)
- Payload enrichment (e.g. make sure certain tags exist)
What if you don't use Python?
What did we learn?
-
Always have alarms for your lambdas!
(have you checked SLIC Watch already? 😉) - You can use Gzip compression to send big payloads to AWS
- Boto3 is extensible through its event system!
Do you think boto3 should make it easier to send gzipped payloads?
👍 UPVOTE this issue: github.com/boto/botocore/issues/2425
Do you want more details?
Cover picturea by Moritz Mentges on Unsplash
Thanks doggie by Stefanie Shank
