Ruby 

and 

Hadoop



Alan Gardner


solutions architect at Pythian


@alanctgardner

alanctgardner@gmail.com

Github

Hadoop



a collection of daemons for clustering systems


provides redundant storage and parallel computation


handles petabytes of data, scales linearly


not suitable for high velocity data (runs batch jobs)

WhY hadoop?



Volume



Variety




Development environment


You don't need a cluster


Cloudera's single-node VM



developing for hadoop


Map Reduce - HADOOP STREAMING


IMPALA - SQL on Hadoop

map reduce



it's not just for java developers


hadoop streaming interface 


one script each for the map and reduce phases


read key/value pairs from stdin, write to stdout


mappers


operate on a portion of the input files


don't share state, can't see the whole set of input records


write intermediate key-value pairs for reducers


typical cases: extracting fields, filtering records

reducers



receive a partition of the map output 

see all the records for one key

emit key-value pairs which are stored as output

typical uses: SORTING, 

counting, joins

an example


running the  code
hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.2.0.jar -input /user/cloudera/in -output /user/cloudera/out7/ -mapper map.rb -reducer reduce.rb -file map.rb -file reduce.rb

map
#!/usr/bin/ruby

ARGF.each do |line|
	line.split(' ').each do |word|
    	puts "#{word}\t1"
	end
end

an example

reduce
#!/usr/bin/ruby

count=0
ARGF.each do |line|
    w = line.split("\t")[0]
    if w == word
        count +=1    else
        if count != 0
            puts "#{word}\t#{count+1}"
    	end
        word = w
        count = 0
    end  	 
end

CAVEATS




keys must be text or numeric

any dependencies must exist on all nodes

install gems with a tool like Ansible or Chef first


impala


complements MapReduce

provides real-time SQL queries on data in Hadoop

easy to connect from Ruby with the impala gem:

 you have to define a schema on your data

An Example

Hive
# Create the tableCREATE TABLE soccer ( sid int, ts bigint, x double, y double, ... )ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';# Load the dataLOAD DATA LOCAL INPATH 'full-game' INTO TABLE soccer;
Ruby (Sinatra)
get '/data' do
  content_type 'application/json'
  data=[]
  Impala.connect(IMPALA_SERVER, 21000) do |conn|
    data = conn.query("select (round(-1*x/2624)*25)+563 as y, (round(-1*y/3400)*39)+472 as x, count(*)/200 as val from soccer_part WHERE sid =#{params[:sid]} AND ts <= #{params[:max]} AND ts >= #{params[:min]} AND x > 0 group by x,y")
  end
  return JSON.dump data
end




Impala

In action




Questions?

Big dataRuby

By Alan Gardner

Made with Slides.com

Big dataRuby

  • 1,768

More from Alan Gardner