从工作中找开源专案
以S3RS为例

 
 

Antonio Yang 楊伯安

2019年11月2-11月3日 中国·上海

COSCon ’ 2019

前言 Preface

 

我只是一个开源软体的支持者

Open Source

  • 自由的使用
  • 自由的学习
  • 自由的修改
  • freedom   →  without fear
  • 自由  →  無懼的

    •  
 

Antonio Yang

  • Software Engineer
    • Web
    • System
    • Python
  • Open-source user
    • Archlinux
    • Qtile
  • Outdoor lover
    • Hiking
    • Climbing
    • River tracing

S3RS

Side project @ Bigtera

 

  • 对象储存云
    服務用户端程序
  • 易于试调
  • 支持多种配置
  • S3RS@Github

Object Storage

对象存储

 

[1]

对象存储

  • 非結構化資料（Unstructured data）
  • 存储空间（Bucket）
    • /photo
  • 对象（Object）
    • /2019/11/02/CosCon.jpg

"Four Objects in a Diagonal Row" by byzantiumbooks is licensed under CC BY 2.0

云服務

  • 支持標準
  • 跨装置
  • 跨平台
  • 易取得

"Bir Paragliding & Hiking 2017" by fredibach is licensed under CC BY 2.0

对象储存云服務

  • 公有云
    • 亚马逊（AWS）
    • 阿里云（OSS）
 
  • HTTP REST request signature
    • Version 2
    • Version 4
  • URL style
    • Path style
    • Virtual hosting style
  • Response Format
    • XML
    • JSON
    • TEXT
 
  • 私有云
    • Bigtera
    • 星辰天合
 

Object Storage API

对象存储API

 

[2]

"The right fit" by Ian D. Keating is licensed under CC BY 2.0

RESTful API

Content

  • GET /{Bucket}/{Key}
  • PUT /{Bucket}/{Key}
  • DELETE /{Bucket}/{Key}

 

Configure

For example ACL

  • GET /{Bucket}/{Key}?acl
  • PUT /{Bucket}/{Key}?acl

 

 

 

 

  • DELETE /{Bucket}/{Key}?acl

<?xml version="1.0" encoding="UTF-8"?>
<AccessControlPolicy
    xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
        <ID>54bbddd7c9c485b696f5b188467d4bec889b83d3862d0a6db526d9d17aadcee2</ID>
        <DisplayName>yanganto</DisplayName>
    </Owner>
    <AccessControlList>
        <Grant>
            <Grantee
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
                <ID>Canonical-user-id</ID>
                <DisplayName>yanganto</DisplayName>
            </Grantee>
            <Permission>FULL_CONTROL</Permission>
        </Grant>
    </AccessControlList>
</AccessControlPolicy>

 

 

PERMISSION: FULL_CONTROL, WRITE, WRITE_ACP,READ, READ_ACP

ACL can on Bucket or Object

CEPH 自带的API

  • /admin/usage?uid=user_id
  • /admin/user
  • /admin/user?subuser
  • /admin/user?key
  • /admin/user?quota
  • /admin/user?cap
  • /admin/bucket
  • /admin/bucket?object=object
  • /admin/bucket?quota

Security

訊息安全

 

[3]

Security

Confidentiality  → HTTPS, Access key

Security

          Availability →

Redundant、Redirect

 

              ​          Integrity  →  

Message authentication code (MAC)

Signature

"Signatures and wax seals" by aehdeschaine is licensed under CC BY-ND 2.0

Signature V2

Authorization: Algorithm Access-Key:Signature

Algorithm: OSS、AWS

Signature: Base64(HMAC-SHA1( Secret-Key, UTF-8-Encoding-Of( StringToSign ) ) )

CanonicalizedSpecialHeaders:  X-OSS-、X-AMZ-

 
 

StringToSign =

    HTTP-Verb + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Date + "\n" +
    CanonicalizedSpecialHeaders +
    CanonicalizedResource;

PUT

c8fdb181845a4ca6b8fec737b3581d76

text/html

Thu, 17 Nov 2005 18:49:58 GMT

x-oss-magic:abracadabra

x-oss-meta-author:foo@bar.com

/oss-example/nelso

Example :

 

OSS: RFC 822

https://github.com/yanganto/s3rs/issues/55

Signature V4

Authorization: AWS4-HMAC-SHA256
                          Credential=ACCESS_KEY/20150830/us-east-1/s3/aws4_request,                                                  SignedHeaders=content-type;host;x-amz-date,
                          Signature= Hex(HMAC-SHA256(SigningKey, StringToSign ))

 

 
 

StringToSign =

 HTTP-Verb + "\n" +
 Canonical URI + "\n" +
 Canonical Query String + "\n" +
 Canonical Headers + "\n" +
 Signed Headers + "\n" +
 Hex(SHA256Hash(payload))

 

DateKey = HMAC-SHA256("AWS4 SecreteKey", "yyyymmdd" )

RegionKey = HMAC-SHA256(DateKey, "region" )

ServiceKey = HMAC-SHA256(RegionKey, "service" )

SigningKey =  HMAC-SHA256(ServiceKey, "aws4_request" )

 

High Availability

高可用

 

[4]

多重地区Multi-Region

Multiple cluster share the same bucket namespace

  • 同名空间 →  同样的meta
  • Unitary executive theory
  • 行政一体

多重区Multi-Zone

Cluster  Mirror

  • Active-Standby
  • Active-Active (Ceph after Krakan)

Redirect

"Prism trials" by byzantiumbooks is licensed under CC BY 2.0

https://bucket.s3.region1.tw/the/key/of/some/obj

https://bucket.s3.region2.tw/the/key/of/some/obj

301 Move Permanently 

URL 变动 → StringToSign 變動

Region 变动 → SignKey 變動

 

要重新产生一次signature

Endpoint

端點

 

[5]

URL Style

"Project Dance République" by Christophe Becker is licensed under CC BY-ND 2.0

Storage Zone/Region

  • Bucket 1
  • Bucket 2
  • Bucket 3

"Server Icon" by David Yim is licensed under CC BY-NC-ND 4.0

  • Bucket a
  • Bucket b
  • Bucket c

Path style

https://s3.region1.tw/bucket/the/key/of/some/obj

 

Virtual Host style

https://bucket.s3.region1.tw/the/key/of/some/obj

23.20.0.0

DNS query

DNS query

23.20.1.1

23.20.1.1

Get data from

Big File

大文件

 

[6]

Multipart

"Ein Haus aus LEGO Steinen" by koelnblogging.com is licensed under CC BY 2.0

Multipart Upload

1. POST /{Bucket}/{Key}?uploads

→  UploadId

2. PUT /{Bucket}/{Key}?uploadId=UploadId&partNumber=Partnumber

→ etag

3. POST /{Bucket}/{Key}?uploadId=UploadId

 

每一个 part 5 MB to 5 GB，用户端定义​

  •  

<CompleteMultipartUpload >

   <Part>

      <ETag>etag</ETag>

      <PartNumber>int</PartNumber>

   </Part> ...

</CompleteMultipartUpload>

https://github.com/yanganto/s3rs/issues/57

Multipart Download

GET /Key HTTP/1.1
Host: bucket.s3.awsamazone.com
Range: bytes=0-1023
 

HTTP Range Header

Request

Response

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-1023/146515
Content-Length: 1024
...

UX

客户体验

 

[7]

"Two men talking on a bench in Glasgow by the River Clyde" by CherryTherapies.com is licensed under CC BY 2.0

Access Key, Secrete Key

ACL

?

"paycheck" by owaief89 is licensed under CC BY-NC 2.0

Presign

One URL Makes

Life Easier

Presigned URL

https://{bucket}.s3.amazonaws.com/{key}?

X-Amz-Algorithm=AWS4-HMAC-SHA256&

X-Amz-Expires={second}&

X-Amz-Credential={AccessKey}{region}%2Fs3%2Faws4_request&

X-Amz-SignedHeaders=host&

X-Amz-Date={iso 8601 time string}&

X-Amz-Signature=xxxxxxx

https://{bucket}.s3.amazonaws.com/{key}?

AWSAccessKeyId={AccessKey}&

Expires={utc time stampe}&

Signature=xxxxxxxxxxx

Signature V2

Signature V4

https://github.com/yanganto/s3rs/issues/54

 

Signature V2

StringToSign =

    HTTP-Verb + "\n" +
    Host + "\n" +
    URI + "\n" +
    QueryString

PUT

{bucket}.s3.amazonaws.com

/{key}

AWSAccessKeyId={AccessKey}Expires={utc time stamps}

Example :

https://{bucket}.s3.amazonaws.com/{key}?

AWSAccessKeyId={AccessKey}&

Expires={utc time stampe}&

Signature=xxxxxxxxxxx

Signature V4

StringToSign =

    HTTP-Verb + "\n" +
    Host + "\n" +
    URI + "\n" +
    QueryString

DateKey = HMAC-SHA256("AWS4 SecreteKey", "yyyymmdd" )

RegionKey = HMAC-SHA256(DateKey, "region" )

ServiceKey = HMAC-SHA256(RegionKey, "service" )

SigningKey =  HMAC-SHA256(ServiceKey, "aws4_request" )

 

https://{bucket}.s3.amazonaws.com/{key}?

X-Amz-Algorithm=AWS4-HMAC-SHA256&

X-Amz-Expires={second}&

X-Amz-Credential={AccessKey}{region}%2Fs3%2Faws4_request&

X-Amz-SignedHeaders=host&

X-Amz-Date={iso 8601 time string}&

X-Amz-Signature=xxxxxxx

Do it !

 
 

[8]

Yes, It should be easier !

简化设定

  • 设定档越简单越好→ s3cfg 81行设定档
  • Object Storage →  Access Key / Secret Key
  • 本同的服务提供者 → Endpoint / type / region
  • 一站一档案，易于管理

[[credential]]

s3_type = "ceph"
host = "10.1.13.98"
user = "admin"
access_key = "XXXXX"
secret_key = "XXXXX"
region = "us-east-1"
 

易于试调

  • ERROR - 服務器異常
  • INFO - HTTP body, headers
  • DEBUG - Signature
  • TRACE - Chunk Detail
 

盡可能的方便使用

  • support AWS
  • support CEPH
  • support redirect
  • single executable binay

开源参与

 
 

[9]

我不是大神、公司也没有开源计划的打算

大项目没法参与，但我们总会找到自己可以做的事情~~~

怎麼開始的?

  • Linux 沒有S3 Browser
  • S3 Browser 沒有debug log
  • support：我们S3的那个XXX可不可以用?
  • support：我们S3怎麽查?
  • 公司：我们只做服务器端
  • 公司：我们要开发安全删除功能

... ...

想办法知道、
想办法解决

  • 写笔记不如实作
  • 实作不如实用

工作已经很忙了~~~~

  • 从手边找个可以做的tool先，至少自己就是user
  • 上班的时候，边用自己的工具，
    • 开发公司的产品当RD
    • 边当QA Debug tool
  • 下班的时候，回头来开发自己的工具
    • 因为实作过两边的问题都会更清楚
    • 反而更能體會客戶的感覺

工作已经很丫雜了~~~~

那还不做些有趣的开源专案~~~~

哪来的时间？

下班时间週末，心情好就写一点

心情不好也写一点

我写的不完美、怕丢脸~~~

别怕，自由是無懼的，多嘗試

开源文化

共享／自由／学习

祝大家找到自己的兴趣

Have Fun

  

Let’s Cross the Boundaries Together!

PR is welcome

谢谢大家

S3RS - COScon'19

By Antonio Yang

S3RS - COScon'19

The talk about S3RS in COSCon'19 in Shanghai（上海）

  • 1,624

More from Antonio Yang