Open Source

• 自由的使用
• 自由的学习
• 自由的修改

• freedom   →  without fear
• 自由  →  無懼的

 

Antonio Yang

• Software Engineer
  • Web
  • System
  • Python
• Open-source user
  • Archlinux
  • Qtile
• Outdoor lover
  • Hiking
  • Climbing
  • River tracing

S3RS

Side project @ Bigtera

• 对象储存云
  服務用户端程序
• 易于试调
• 支持多种配置
• S3RS@Github

对象储存云服務

• 公有云
  • 亚马逊（AWS）
  • 阿里云（OSS）

• HTTP REST request signature
  • Version 2
  • Version 4
• URL style
  • Path style
  • Virtual hosting style
• Response Format
  • XML
  • JSON
  • TEXT

• 私有云
  • Bigtera
  • 星辰天合

Content

• GET /{Bucket}/{Key}
• PUT /{Bucket}/{Key}
• DELETE /{Bucket}/{Key}

Configure

For example ACL

• GET /{Bucket}/{Key}?acl
• PUT /{Bucket}/{Key}?acl

• DELETE /{Bucket}/{Key}?acl

<?xml version="1.0" encoding="UTF-8"?>
<AccessControlPolicy
    xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
        <ID>54bbddd7c9c485b696f5b188467d4bec889b83d3862d0a6db526d9d17aadcee2</ID>
        <DisplayName>yanganto</DisplayName>
    </Owner>
    <AccessControlList>
        <Grant>
            <Grantee
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
                <ID>Canonical-user-id</ID>
                <DisplayName>yanganto</DisplayName>
            </Grantee>
            <Permission>FULL_CONTROL</Permission>
        </Grant>
    </AccessControlList>
</AccessControlPolicy>

PERMISSION: FULL_CONTROL, WRITE, WRITE_ACP,READ, READ_ACP

ACL can on Bucket or Object

CEPH 自带的API

• /admin/usage?uid=user_id
• /admin/user
• /admin/user?subuser
• /admin/user?key
• /admin/user?quota
• /admin/user?cap
• /admin/bucket
• /admin/bucket?object=object
• /admin/bucket?quota

Security

Confidentiality  → HTTPS, Access key

Security

          Availability →

Redundant、Redirect

              ​          Integrity  →  

Message authentication code (MAC)

Signature V2

Authorization: Algorithm Access-Key:Signature

Algorithm: OSS、AWS

Signature: Base64(HMAC-SHA1( Secret-Key, UTF-8-Encoding-Of( StringToSign ) ) )

CanonicalizedSpecialHeaders:  X-OSS-、X-AMZ-

StringToSign =

    HTTP-Verb + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Date + "\n" +
    CanonicalizedSpecialHeaders +
    CanonicalizedResource;

PUT

c8fdb181845a4ca6b8fec737b3581d76

text/html

Thu, 17 Nov 2005 18:49:58 GMT

x-oss-magic:abracadabra

x-oss-meta-author:foo@bar.com

/oss-example/nelso

Example :

OSS: RFC 822

Signature V4

Authorization: AWS4-HMAC-SHA256
                          Credential=ACCESS_KEY/20150830/us-east-1/s3/aws4_request,                                                  SignedHeaders=content-type;host;x-amz-date,
                          Signature= Hex(HMAC-SHA256(SigningKey, StringToSign ))

StringToSign =

 HTTP-Verb + "\n" +
 Canonical URI + "\n" +
 Canonical Query String + "\n" +
 Canonical Headers + "\n" +
 Signed Headers + "\n" +
 Hex(SHA256Hash(payload))

DateKey = HMAC-SHA256("AWS4 SecreteKey", "yyyymmdd" )

RegionKey = HMAC-SHA256(DateKey, "region" )

ServiceKey = HMAC-SHA256(RegionKey, "service" )

SigningKey =  HMAC-SHA256(ServiceKey, "aws4_request" )

多重地区Multi-Region

Multiple cluster share the same bucket namespace

• 同名空间 →  同样的meta
• Unitary executive theory
• 行政一体

多重区Multi-Zone

Cluster  Mirror

• Active-Standby
• Active-Active (Ceph after Krakan)

https://bucket.s3.region1.tw/the/key/of/some/obj

https://bucket.s3.region2.tw/the/key/of/some/obj

301 Move Permanently 

URL 变动 → StringToSign 變動

Region 变动 → SignKey 變動

 

要重新产生一次signature

Storage Zone/Region

• Bucket 1
• Bucket 2
• Bucket 3

"Server Icon" by David Yim is licensed under CC BY-NC-ND 4.0

• Bucket a
• Bucket b
• Bucket c

Path style

https://s3.region1.tw/bucket/the/key/of/some/obj

Virtual Host style

https://bucket.s3.region1.tw/the/key/of/some/obj

23.20.0.0

DNS query

DNS query

23.20.1.1

23.20.1.1

Get data from

Multipart

"Ein Haus aus LEGO Steinen" by koelnblogging.com is licensed under CC BY 2.0

Multipart Upload

1. POST /{Bucket}/{Key}?uploads

→  UploadId

2. PUT /{Bucket}/{Key}?uploadId=UploadId&partNumber=Partnumber

→ etag

3. POST /{Bucket}/{Key}?uploadId=UploadId

每一个 part 5 MB to 5 GB，用户端定义​

<CompleteMultipartUpload >

   <Part>

      <ETag>etag</ETag>

      <PartNumber>int</PartNumber>

   </Part> ...

</CompleteMultipartUpload>

https://github.com/yanganto/s3rs/issues/57

Multipart Download

GET /Key HTTP/1.1
Host: bucket.s3.awsamazone.com
Range: bytes=0-1023

HTTP Range Header

Request

Response

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-1023/146515
Content-Length: 1024
...

One URL Makes

Life Easier

Presigned URL

https://{bucket}.s3.amazonaws.com/{key}?

X-Amz-Algorithm=AWS4-HMAC-SHA256&

X-Amz-Expires={second}&

X-Amz-Credential={AccessKey}{region}%2Fs3%2Faws4_request&

X-Amz-SignedHeaders=host&

X-Amz-Date={iso 8601 time string}&

X-Amz-Signature=xxxxxxx

https://{bucket}.s3.amazonaws.com/{key}?

AWSAccessKeyId={AccessKey}&

Expires={utc time stampe}&

Signature=xxxxxxxxxxx

Signature V2

Signature V4

https://github.com/yanganto/s3rs/issues/54

Signature V2

StringToSign =

    HTTP-Verb + "\n" +
    Host + "\n" +
    URI + "\n" +
    QueryString

PUT

{bucket}.s3.amazonaws.com

/{key}

AWSAccessKeyId={AccessKey}Expires={utc time stamps}

Example :

https://{bucket}.s3.amazonaws.com/{key}?

AWSAccessKeyId={AccessKey}&

Expires={utc time stampe}&

Signature=xxxxxxxxxxx

Signature V4

StringToSign =

    HTTP-Verb + "\n" +
    Host + "\n" +
    URI + "\n" +
    QueryString

DateKey = HMAC-SHA256("AWS4 SecreteKey", "yyyymmdd" )

RegionKey = HMAC-SHA256(DateKey, "region" )

ServiceKey = HMAC-SHA256(RegionKey, "service" )

SigningKey =  HMAC-SHA256(ServiceKey, "aws4_request" )

https://{bucket}.s3.amazonaws.com/{key}?

X-Amz-Algorithm=AWS4-HMAC-SHA256&

X-Amz-Expires={second}&

X-Amz-Credential={AccessKey}{region}%2Fs3%2Faws4_request&

X-Amz-SignedHeaders=host&

X-Amz-Date={iso 8601 time string}&

X-Amz-Signature=xxxxxxx

Yes, It should be easier !

简化设定

• 设定档越简单越好→ s3cfg 81行设定档
• Object Storage →  Access Key / Secret Key
• 本同的服务提供者 → Endpoint / type / region
• 一站一档案，易于管理

[[credential]]

s3_type = "ceph"
host = "10.1.13.98"
user = "admin"
access_key = "XXXXX"
secret_key = "XXXXX"
region = "us-east-1"
 

易于试调

• ERROR - 服務器異常
• INFO - HTTP body, headers
• DEBUG - Signature
• TRACE - Chunk Detail

盡可能的方便使用

• support AWS
• support CEPH
• support redirect
• single executable binay

我不是大神、公司也没有开源计划的打算

大项目没法参与，但我们总会找到自己可以做的事情~~~

怎麼開始的?

• Linux 沒有S3 Browser
• S3 Browser 沒有debug log
• support：我们S3的那个XXX可不可以用?
• support：我们S3怎麽查?
• 公司：我们只做服务器端
• 公司：我们要开发安全删除功能

... ...

想办法知道、
想办法解决

• 写笔记不如实作
• 实作不如实用

工作已经很忙了~~~~

• 从手边找个可以做的tool先，至少自己就是user
• 上班的时候，边用自己的工具，
  • 开发公司的产品当RD
  • 边当QA Debug tool
• 下班的时候，回头来开发自己的工具
  • 因为实作过两边的问题都会更清楚
  • 反而更能體會客戶的感覺

工作已经很丫雜了~~~~

那还不做些有趣的开源专案~~~~

哪来的时间？

下班时间週末，心情好就写一点

心情不好也写一点

我写的不完美、怕丢脸~~~

别怕，自由是無懼的，多嘗試

开源文化

共享／自由／学习

祝大家找到自己的兴趣

Have Fun

Let’s Cross the Boundaries Together!

PR is welcome