Scala & FP
Produced by
Distributed Computing, corp.
@Noootsab
Proud husband and father
Glasses caused by Maths graduation
Fashion victim after CS graduation
Lost in Geomatics and GIS
Data evangelist via NextLab
Co-founded Wajug
Helper and organizer of Devoxx4Kids
Scala trainer
Oh, and I wrote a Packt book about Play! 2
Why I mean it!
and others do...
Scala has a reputation to be accessible
It eases the maths (mostly [matrix] algebra)
The CS world is changing (fast)
It shifts from the cloud to analyses
That is, from IT needs to Market opportunities
Reused YOUR knowledge
Fact: syntax close to Java, C#, Ruby, ...
Cause: Object Oriented
case class Person( name:String, first:String,
age:Double, gender:Gender,
father:Option[Person],
mother:Option[Person],
children:List[Child]=Nil
) {
def incAge(n:Int):Person = copy(age = age+n)
def newSon(child:Person):(Person, Person) = {
val newChild = this.gender match {
case Male => child.copy(father = Some(this))
case Female => child.copy(mother = Some(this))
}
(newChild, this.copy(children = newChild :: children)
}
}val _Noah = Person("Petrella", "Noah",
age=4, Male,
mother=Some(Sandrine)
father=None)
val boringNoootsab = Person("Petrella", "Andy",
32, Male,
father=Some(Arcangelo),
mother=Some(Nadine))Following the wave
Fact: Functional Programming ftw
Cause: Scalable Language
Please, bear with me...
WHO
a
LOT
mainly data fans
Coursera
10⁶ online students
Slightly moving from PHP→Scala
From Blue Whale to Analytic Platform
Netflix
Billion devices
Historical events
Real-time analytics
and counting...
Azavea
Socrata
AirBNB
Virdata
eBay
PowerInbox
BBC
Snips
Ooyala
Functional Programming
in a nutshell
source wikipedia: http://en.wikipedia.org/wiki/Function_(mathematics)
Input x
can be a function...
Defines a general process
that could behave differently
listOfNames map { name => DB.getByName(name) }
listOfPersons flatMap { person => person.friends }
listOfFriends filter { (f:Friend) => f.met moreThan (10 years) }
listOfOldFriends.count(_.person.gender != me.gender)OUTPUT f(X)
bah... can be a function as well..
Prepares a process
that will be available for later usage
def authentication(manager:SecurityManager): User=>Authentication
def source(url:String): Authentication=>DataRepo=>Data
//[...]
val authenticate = authentication(FakeSecurityManager)
val settings = source("/settings")
def request = {
val user = //...
val auth = authenticate(user)
val settingsFetcher = settings(auth)
// and so on
}Show me!
// linear regression: returns the coefficients and the `prediction` function
def lm(x:List[Double], y:List[Double]):((Double, Double), Double=>Double) = {
val n = x.size
val ẍ = x.sum.toDouble / n
val ÿ = y.sum.toDouble / n
val Sp = ((x ∙- ẍ) ∙* (y ∙- ÿ) sum) / (n-1)
val Sx2 = ((x ∙- ẍ) ∙^ 2 sum) / (n-1)
val ß1 = Sp / Sx2
val ß0 = ÿ - ß1 * ẍ
val coefs = (ß0, ß1)
val predict = (d:Double) => ß0 + ß1 * d
(coefs, predict)
}
def test(ß0:Double = 18.1d, ß1:Double = 6d, error:Int=>List[Double]) = {
val n = 10000
val x:List[Double] = -n.toDouble to n by 1 toList
val e = error(2*n+1)
val y:List[Double] = ß0 ∙+: (ß1 ∙*: x) ∙+: e
lm(x, y)
}
val error = rnorm(mean=0, sigma=5) // gen gaussian nbs
val model = test(103, 7, error)
So, even hard things could be done simply
keeping readability
conciseness
Honestly,
showing the OO version isn't worthwhile...
Lazy
yeah yeah... I'll do it
Avoid computations
Delayed initialization
// delay initialization of the application at usage time
lazy val app:App = initializeApp()
// only compute the string `m` when accessing it, hence with debug is enabled
def logDebug(m: => String)= if (LOG.debugEnabled) LOG.debug(m) else ()Sooo laaazy
Come back... in a potential future
TL; DW
val app:Future[App] = initializeApp()
val http:Future[HttpClient] = app.map( appInst => appInst.http.client )
def isOk(url:String):Future[Boolean] =
http.flatMap { client =>
client.get(url) // execute request
}
.map{ resp =>
resp.code // get the HTTP code only
}
.filter(code => code == 200) // only cares about 200
.recoverWith {
// catches the comm' prob. → and retries
case x:CommunicationException => isOk(url)
}.recover {
// if fails → not ok
case e: Throwable => false
}Code... now
(I promised)
// Build an **INFINITE** List of `A`s
class LazyCons[+A](a:A, t: => Lazy[A]) extends Lazy[A] {
val head = Some(a)
lazy val tail = t
}// read a file in a LAZY manner
// ... more helpful when using NIO of course
def fetch(file:String):Lazy[Future[String]] = {
val texts = io.Source.fromFile(new java.io.File(file)).getLines
def readLine(texts:Iterator[String]):Lazy[Future[String]] = //...
readLine(texts)
}val fibs:Stream[Int] = 0 #:: 1 #:: ((fibs zip fibs.drop(1)) map ((_:Int) + (_:Int)).tupled)Mashup
A function could either
→ be called on data (method, sync)
→ be sent to the data (message, async)
A function composes
A function is a delayed computation
...
What if
I compose several computations
WHAT IF
I send the composition to where the data is?
We've created Spark!
(back to Gerard's talk)
DO YA Wanna funk?
trait Data {
def dependent:List[Double]
def observed:Matrix
def bootstrap(proportion:Double):Future[Data]
}trait Model {
type Coefs
def apply(data:Data):Future[(Coefs, List[Double]=>Future[Double])]
}def bagging(model:Model)(agg:Aggregation[model.Coefs], n:Int)(data:Data):Future[model.Coefs] ={
def exec:Future[model.Coefs] = for {
sample <- data.bootstrap(0.6)
(coefs, _) <- model(sample)
} yield coefs
val execs:List[Future[model.Coefs]] = List.fill(n)(exec)
val coefsList:Future[List[model.Coefs]] = Future.sequence(execs)
val result:Future[model.Coefs] = coefsList map agg
result
}Q.E.D.
Thanks for stayin'
Join me on Twitter @Noootsab
Head to @NextLab_be: http://nextlab.be
Check http://ftts.nextlab.be for future Scala trainings
Scala & Functional Programming in Distributed Computing World
By andy petrella
