Design Lesson 2

http://www.zhitongguigu.com

Design pieces

Accuracy
Performance
Robustness
Scalability
Test

http://www.zhitongguigu.com

Knowledge pieces

Database
Network,Server,TCP/IP
Security, Encryption/Decryption
Distributed System
Backup System
*Hadoop

http://www.zhitongguigu.com

BBS System

User could sign up by username, email
User could edit its own profile
User could post
User could reply to the post
User could like some post
User could go to someone's profile to see all the posts

http://www.zhitongguigu.com

First Question:

http://www.zhitongguigu.com

What is the key point for this question?

Database Design

BBS System

http://www.zhitongguigu.com

What do we need?

First: Extract basic elements

User, Post, Profile, Reply

http://www.zhitongguigu.com

Most Important question

http://www.zhitongguigu.com

Design endpoint to fulfill these requirements

Backend: How to join the table to give the results

endpoint to fulfill the requirements

http://www.zhitongguigu.com

Sign up
Sign in
Profile Edit
Post
Reply
Like
Unlike
Edit Post/Reply

How to send the request

http://www.zhitongguigu.com

Apache

Nginx

Lighttpd

IIS

Tomcat

Jetty

Netty

How to join the table to give the results

http://www.zhitongguigu.com

Sign up
Sign in
Profile Edit
Post
Reply
Like
Unlike
Edit Post/Reply

Basic Operation of SQL

http://www.zhitongguigu.com

Select
From
Where
Join
- Left Join
- Right Join
- Outer Join

Delete
Create
Insert Into
Update
Distinct

Count
Min
Max
Avg

w3c w3c_cn

Follow up question

If the user can see who likes the post?
If some one could reply other's reply? (like wechat discovery)

http://www.zhitongguigu.com

Scalability (System Design)

How to improve the system if there are thousands of replies?
How to improve the system if there are thousands of users?

http://www.zhitongguigu.com

System Design

Database Sharding

Vertical partitioning. Normalization is one way of vertical partition. We split columns into different tables. Or even further, we have frequent used table of data stored on a faster machine.
Horizontal partitioning. (Shard). Instead of vertical split the columns, we split the rows. The tables are divided and distributed into different server.

http://www.zhitongguigu.com

System Design

Scaling up

Vertical Scaling
- CPU, Disk(PATA, SATA, SSD), RAM
Horizontal Scaling
- Load Balancer
- Database Replication

http://www.zhitongguigu.com

System Design

Load Balancer

load balancer has public IP address for DNS to return
backend worker servers has private IP address for safety
How to divide work?
- determine which is busy
- divide by work category
- round robin(different ip address at different times)
  - login again and again
    - Write session to hard disk
    - Sticky Session

http://www.zhitongguigu.com

System Design

Server & Database Replication

Master-Slave
- Good for read-heavy websites.
- Free backup
- Slave become master if master is down
Master-Master
- Prevent master down

http://www.zhitongguigu.com

System Design

Caching

Fetching from DB is slow

Caching queries
- Most commonly used
- Query (hash) -> result
- If one cell is updated, delete all cached queries
Caching objects (Redis)
- See data as an object
- Build dataset from DB and store the whole. (page)
- Activity steams
- A smaller but faster DB in mem.

http://www.zhitongguigu.com

System Design

CDN

Content Distributed Network

CDNs replicate content to multiple places.
Closer, less hops
Amazon CloudFront

http://www.zhitongguigu.com

How to improve the system if there are thousands of replies/users?

http://www.zhitongguigu.com

Pagination: fetch the data as pages
Async: preload the most important data
Sharding: horizontal partition of data in a database or search engine

Short URL

http://www.zhitongguigu.com

Short URL

http://www.zhitongguigu.com

In Twitter, Weibo, you have charactor limit. Also normal URL will contain some invalid ASCII code that you cannot put into the content.

So a lot of company has its own Short URL

Example: http://t.cn/RGKGR3T

Short URL

http://www.zhitongguigu.com

Design A System that can generate this Short URL

The user input an URL, the server returns a short one
Other user click this short URL, it will redirect to the real URL
If the URL has not been used (no one clicks for 72 hours, configurable), it will be removed

How to generate a short URL

http://www.zhitongguigu.com

UUID

MD5

Are these too long? Could we find a better way?

Think about the requirement

one URL could be used released after some day

How to generate a short URL

http://www.zhitongguigu.com

We do not generate a short URL everytime, we just lookup the table

How to send request

http://www.zhitongguigu.com

Apache Tomcat

Design a Database

http://www.zhitongguigu.com

Short URL -> Long URL, time

Pretty Easy? If we have a high QPS?

What DB we should use?

http://www.zhitongguigu.com

Relational Database: Oracle, MySql

Non-Relational Database (NO Sql): MangoDB, CouchDB, Couchbase, Redis

RDBMS vs NoSQL

http://www.zhitongguigu.com

RDBMS(Relational Database Management System)

Strong mathematical basis
Declarative syntax
A well-known language in Structured Query Language (SQL)
Real-time Read
Read/Write Consistency

RDBMS vs NoSQL

http://www.zhitongguigu.com

NoSQL (Not Only SQL)

It does not have schema
Read/Write throughput is very high
Horizontal scalability can be achieved easily
Will support Bigdata in volumes of Terra Bytes & Peta Bytes
Faster development life cycles for developers
Can be hosted in cheaper hardware machines
In memory caching option is available to increase the performance of queries

Lots of companies combine these two together

http://www.zhitongguigu.com

For this question

http://www.zhitongguigu.com

WE could just use NoSQL

Reason: we do not have complicated query to fetch data, we only need original URL and short URL (timestamp maybe)

How to remove unused links?

http://www.zhitongguigu.com

How to update the Store?

How to remove them?

How to remove unused links?

http://www.zhitongguigu.com

short -> Long

short -> time

time -> short

Is it okay?

How to remove unused links?

http://www.zhitongguigu.com

If we could stretch the time

short -> Long

short -> time

Daily data purge job

If the QPS is very high

Everything is different

http://www.zhitongguigu.com

Do we really need to be accurate to seconds to purge the data?

Since we have a daily job, it seems not necassary

Messaging System

Apache Kafka

RDBMS + NOSQL

http://www.zhitongguigu.com

Relational Database cannot handle high reading QPS

NOSQL has a lot limit in relation and maintenance

So sometimes we just hybrid them together

RDBMS + NOSQL

http://www.zhitongguigu.com

For common get queries, we store them in NOSQL

For specific queries and write operation, we still use RDBMS

How to update NOSQL? Event Listener

Summary

http://www.zhitongguigu.com

Database: Relational and NoSql -> consider to use which one

Network: Apache, Tomcat, etc -> Rest API, JSON

Distributed System: Load balancing, Sticky Session

Security: HTTPS, Encryption/Decryption