CEPH
Introduction and Beyond
# whoami
- Author of "Learning Ceph" - Packt Publication , January 2015
- http://karan-mj.blogspot.fi/ --- Tune in for latest Ceph blogs
Karan Singh
System specialist cloud
CSC - IT Center for Science , Espoo
karan.singh@csc.fi
CSC-IT Center for Science
- Founded in 1971
- Finnish, Non Profit Organization
- Funded by Ministry of Education
- Connected Finland to Internet in 1988
- Most powerful academic computing facility in the Nordics.
- ISO27001:2013 Certification
- Production Cloud : http://pouta.csc.fi
Our Need for a Storage System
- To build our own storage - Not to Buy
- Software Defined Storage
- Unified - File , block , object
- Open Source
- Ultra scalable & No SPOF
- Cloud ready
- No hardware lockin
- Low cost & Future ready
Challenges with proprietary systems
- Really expensive
- Vendor lock in
- Block,File & Object : Mostly separate product line
- Difficult to scale out
- Mostly closed source
- Hardware & Software tied together
How to overcome these challenges ??
Ceph is The Future of Storage
&
Answer to your storage problems
Ceph :: Philosophy & Design
The History
- 2003 : Sage Weil developed Ceph as a PhD project
- 2003 - 2007 : Research period for Ceph
- 2006 : Open sourced under LGPL
- 2007 - 2011 : DreamHost incubated Ceph
- April 2012 : Sage Weil founded InkTank
- April 2014 : RehHat acquired InkTank
Ceph releases
- Quick release cycles
- Every alternate release is LTS
Ceph :: Requirements
Servers with several DISKS
Networking b/w these servers
Operating System
How Ceph Stack looks like
Ceph :: Building Blocks
Ceph :: Simplified
Ceph is Software Defined Storage
and every component in Ceph
is a daemon
* Ceph MON ( Monitor )
* Ceph OSD ( Object Storage Device )
* Ceph MDS ( Ceph Filesystem )
* Ceph RGW ( Object Storage Service )
Ceph Components :: Quick Overview
- Tracks & Monitor the health of entire cluster
- Maintains the map of cluster state
- "Do Not" store DATA
- Stores the actual data on physical disks
- One OSD per HDD ( recommended )
- Stores everything in the form of "Object"
MON
OSD
- The Filesystem Part of Ceph
- Tracks file hierarchy and stores metadata
- Provides distributed File-system of any size
- The Object Storage part of Ceph
- Provides RESTful gateway to Ceph cluster
- Amazon S3 and OpenStack Swift compatible API
MDS
RGW
Ceph Components :: Quick Overview
Live Demonstration
Getting Your First Ceph Cluster
Up & Running Is Wicked Easy
Step-1 : # git clone https://github.com/ksingh7/Try-Ceph.git
Step-2 : # vagrant up
And you are done
Recorded session available at http://youtu.be/lki1fQptIRk
How Do We Do Ceph @ CSC - [1]
- 15 x HP DL 380e
- Intel Xeon 2.10 GHz
- 24GB Memory
- 12 x 3 TB SATA Disk
- 40Gb Infiniband
- 4 x HP SL4540
- Intel Xeon 2 x 2.30 GHz
- 192GB Memory
- 60 x 4 TB SATA Disk
- 10Gb Infiniband
- 2 x Mellanox IB SW
- Ceph Firefly 0.80.7
- Erasure Coding
- CentOS 6.5 , 3.10.X
- OpenStack ICH
- Gangila Monitoring
- Ansible for Deployment & Management
Total Raw Capacity
1.5 PB
How Do We Do Ceph @ CSC - [2]
- 4 x HP SL4540
- Intel Xeon 2 x 2.30 GHz
- 192GB Memory
- 60 x 4 TB SATA Disk
- 10Gbe
- 2 x 10Gbe SW
- Ceph Firefly 0.80.7
- CentOS 6.5 , 3.10.X
- OpenStack ICH
- Ansible for Deployment & Management
Total Raw Capacity
960 TB
How Do We Do Ceph @ CSC - [3]
- 4 x Primergy RX300 Storage Nodes
- 4 x Eternus JX40 JBOD
- 1 x Primergy Management Node
- 1 x Brocade 1G Mgmt. S/W
- 1 x Mellanox 40G IB Cluster N/W S/W
Total Raw
Capacity
250 TB
Fujitsu Storage ETERNUS CD10000
- Hyperscale Storage System
- H/w by Fujitsu , Ceph S/w from RedHat
- Fujitsu's GUI / CLI Storage Management S/w
- CSC is the first company to use this for POC testing
- Product comes with Fujitsu Enterprise Support
For more information on this product check out
http://www.fujitsu.com/global/products/computing/storage/eternus-cd/index.html
Lessons Learnt
- Never use RAID underneath Ceph OSD. Ceph is very smart and has its own way of doing things .
- Use smaller machines and use as many of them
- Design your cluster for I/O not for Raw TB
- Test recovery situations - Go & literally pull the power cables
- Reboot your machine regularly - uptime XXXX days is no longer cool.
- Use SSD's if you can
Questions ?
- Email : Karan.Singh@csc.fi , Karan_Singh1@live.com
- Twitter : @karansingh0101
- Github : https://github.com/ksingh7
- Blog : http://karan-mj.blogspot.fi/
https://www.packtpub.com/big-data-and-business-intelligence/learning-ceph
Want more about Ceph ? Read the Book
