CEPH

Introduction and Beyond

# whoami

  • Author of "Learning Ceph" - Packt Publication , January 2015
  • http://karan-mj.blogspot.fi/   --- Tune in for latest Ceph blogs

Karan Singh

System specialist cloud

CSC - IT Center for Science , Espoo

karan.singh@csc.fi

CSC-IT Center for Science

  • Founded in 1971
  • Finnish, Non Profit Organization
  • Funded by Ministry of Education
  • Connected Finland to Internet in 1988
  • Most powerful academic computing facility in the Nordics.
  • ISO27001:2013 Certification
  • Production Cloud : http://pouta.csc.fi

Our Need for a Storage System

  • To build our own storage - Not to Buy
  • Software Defined Storage
  • Unified - File , block , object
  • Open Source 
  • Ultra scalable & No SPOF
  • Cloud ready
  • No hardware lockin
  • Low cost & Future ready

Challenges with proprietary systems

  • Really expensive
  • Vendor lock in
  • Block,File & Object : Mostly separate product line 
  • Difficult to scale out
  • Mostly closed source
  • Hardware & Software tied together

How to overcome these challenges ??

Ceph is The Future of Storage

& 

Answer to your storage problems

Ceph :: Philosophy & Design

The History

  • 2003 : Sage Weil  developed Ceph as a PhD project
  • 2003 - 2007 : Research period for Ceph
  • 2006 : Open sourced under LGPL
  • 2007 - 2011 : DreamHost incubated Ceph
  • April 2012 : Sage Weil founded InkTank
  • April 2014 : RehHat acquired InkTank

Ceph releases

  • Quick release cycles
  • Every alternate release is LTS

Ceph :: Requirements

Servers with several DISKS

Networking b/w these servers

Operating System

How Ceph Stack looks like

Ceph :: Building Blocks

Ceph :: Simplified

Ceph is Software Defined Storage

and every component in Ceph

is a daemon

 

* Ceph MON  ( Monitor )

* Ceph OSD   ( Object Storage Device )

* Ceph MDS   ( Ceph Filesystem )

* Ceph RGW  ( Object Storage Service )

Ceph Components :: Quick Overview

  • Tracks & Monitor the health of entire cluster
  • Maintains the map of cluster state
  • "Do Not" store DATA
  • Stores the actual data on physical disks
  • One OSD per HDD ( recommended )
  • Stores everything in the form of "Object"

MON

OSD

  • The Filesystem Part of Ceph
  • Tracks file hierarchy and stores metadata
  • Provides distributed File-system of any size
  • The Object Storage part of Ceph
  • Provides RESTful gateway to Ceph cluster
  • Amazon S3 and OpenStack Swift compatible API

MDS

RGW

Ceph Components :: Quick Overview

Live Demonstration

Getting Your First Ceph Cluster

Up & Running Is Wicked Easy

Step-1 :  # git clone https://github.com/ksingh7/Try-Ceph.git

 

Step-2 : # vagrant up

 

And you are done

Recorded session available at  http://youtu.be/lki1fQptIRk

How Do We Do Ceph @ CSC - [1]

  • 15 x HP DL 380e
  • Intel Xeon 2.10 GHz
  • 24GB Memory
  • 12 x 3 TB SATA Disk
  • 40Gb Infiniband
  •  4 x HP SL4540
  • Intel Xeon 2 x 2.30 GHz
  • 192GB Memory
  • 60 x 4 TB SATA Disk
  • 10Gb Infiniband
  •  2 x Mellanox IB SW
  • Ceph Firefly 0.80.7
  • Erasure Coding
  • CentOS 6.5 , 3.10.X
  • OpenStack ICH
  • Gangila Monitoring
  • Ansible for Deployment & Management

 Total Raw Capacity

1.5 PB

How Do We Do Ceph @ CSC - [2]

  •  4 x HP SL4540
  • Intel Xeon 2 x 2.30 GHz
  • 192GB Memory
  • 60 x 4 TB SATA Disk
  • 10Gbe
  •  2 x 10Gbe SW
  • Ceph Firefly 0.80.7
  • CentOS 6.5 , 3.10.X
  • OpenStack ICH
  • Ansible for Deployment & Management

 Total Raw Capacity

960 TB

How Do We Do Ceph @ CSC - [3]

  • 4 x Primergy RX300 Storage Nodes
  • 4 x Eternus JX40 JBOD
  • 1  x Primergy Management Node
  • 1  x Brocade 1G Mgmt. S/W
  • 1  x Mellanox 40G IB  Cluster N/W S/W

 Total Raw

Capacity

250 TB

Fujitsu Storage ETERNUS CD10000

  • Hyperscale Storage System
  • H/w by Fujitsu , Ceph S/w from RedHat
  • Fujitsu's GUI / CLI Storage Management S/w
  • CSC is the first company to use this for  POC testing
  • Product comes with Fujitsu Enterprise Support

For more information on this product check out 

http://www.fujitsu.com/global/products/computing/storage/eternus-cd/index.html

Lessons Learnt

  • Never use RAID underneath Ceph OSD. Ceph is very smart and has its own way of doing things .
  • Use smaller machines and use as many of them
  • Design your cluster for I/O not for Raw TB
  • Test recovery situations - Go & literally pull the power cables
  • Reboot your machine regularly - uptime XXXX days is no longer cool.
  • Use SSD's if you can

Questions ?

  • Email   :  Karan.Singh@csc.fi , Karan_Singh1@live.com
  • Twitter :  @karansingh0101
  • Github  :  https://github.com/ksingh7
  • Blog      :  http://karan-mj.blogspot.fi/

https://www.packtpub.com/big-data-and-business-intelligence/learning-ceph

Want more about Ceph ?  Read the Book