Introduction to Big Data

Explosion of Data

Explosion of Data (continued)

• The increase in amount of data we are generating opens up huge possibilities
   
• But it comes with problems too:
  -  where do we have to store all this data?
  -  and process it too?​

What is big data?

• How will you define big data?
   
• It's a very subjective term, and there is not one definition for big data
   
• Most people would certainly consider a dataset of several TB to be big data
   
• A reasonable definition:
  Data that are too big to be processed on a single machine

Challenges with Big Data

• Not just about the size
   
• Data is created fast
   
• Data from different sources in various formats

Volume

• Store the data
  -  which one(s) of the following data do you think worth storing?
  transactions, logs, business, user, sensor, medical, social
  
  
  
  
  
  
   
• Read and process the data efficiently

Variety

• For a long time people stored data in relational databases
   
• The problem is that to store data in such databases, the data need to be able to fit in pre-defined tables
   
• Many data we are dealing nowadays are unstructured or semi-structured data
   
• We want to store the data in its original format so we are not throwing any information away
  -  e.g. transcribe a call center conversation into text
  -  we have what people said to customer services representatives
  -  but if we had the actual recording, later we may able to develop algorithms to interpret the tone of voice

Velocity

• We need to be able to store the data even it's coming in at a rate of TB per day
  -  if we can't store as it arrives, we have to discard some of the data
   
• Many recommendation systems were developed based on a variety of data stored which arrives with a high velocity
  -  Amazon will recommend products to you
  -  Netflix will recommend movies you might be interested in

Hadoop

• In around 2003, google published papers about their internally used distributed file systems and their processing framework, MapReduce.
   
• A group tried to reimplement these in open source, and they developed hadoop, which later becomes the core part of today's big data processing platforms
   
• Hadoop:
  -  store data: HDFS (Hadoop distributed file system)
  -  process data: MapReduce
   
• Data are split and stored in a collection of machines (a cluster), and then the data are processed in where they are stored rather than retrieving the data to a central server

Hadoop Ecosystem

• Some softwares are intended to make it easier to load data into the Hadoop cluster
• Many are designed to make Hadoop easier to use

HDFS and MapReduce

HDFS

• Data are stored in HDFS: Hadoop distributed file system

Is there a problem?

blk_1

blk_2

blk_3

• Network failure
   
• Disk failure on datanodes
   
• Disk failure on the namenode

Data Redundancy

blk_1

blk_2

blk_3

Hadoop replicates each block three times

What if there is a problem with the namenode?

blk_1

blk_2

blk_3

• If a network failure: data inaccessible
   
• If a disk failure: data lost forever (no way to know which block belongs to which file)

blk_1

blk_1

blk_2

blk_2

blk_3

blk_3

MapReduce

Designed to process data in parallel

Example

• National hospital inpatient cost
  - want to calculate the cost per city for the year

• Key-value pairs (Hashtable)
   
• What are the problems if you have 1 TB data?
   
• You may run out of memory, and computers will need a long time to read and process the data.

Example (continued)

Mappers

Example (continued)

Spark

Why Spark?

• Shortcomings of Hadoop MapReduce:
  -  only for map and reduce based computations
  -  relies on reading data from HDFS (many machine learning uses iterative algorithms that require several reads of the data, which result in a performance bottleneck due to I/O)
  -  native support for Java only
  -  no interactive shell support
  -  no support for streaming
   
• Advantages of Spark:
  -  expressive programming model
  -  in-memory processing
  -  support for diverse workloads
  -  interactive shell

The Spark Stack

What is in memory processing?

Resilient Distributed Datasets

Resilient Distributed Datasets

• Resilient:
  -  recover from errors: e.g. node failure
  -  track history of each partition, re-run
   
• Distributed:
  -  distributed across the cluster of machines
  -  divided in partitions, atomic chunks of data
   
• Dataset:
  -  data storage created from HDFS, S3, HBase, JSON, text, local hierarchy of folders
  -  or created transforming another RDD (RDD is immutable)

Big Data and Health Disparities

Opportunities Identified by NIMHD

• To incorporate social determinants information and improve quality of care for underserved populations
   
• To improve public health surveillance and address health disparities
   
• To understand etiology and to guide interventions to reduce disparities

Disease registries and EHRs collect limited information on social determinants

Use social media data in public health surveillance and research

• Estimate influenza prevalence:
  -  some of the earliest studies used web data, e.g. search query volumes (Google's Flu Trends)
  -  while search queries were the original data sources, social media has since become a popular data source, e.g. blogs and microblogs especially Twitter
   
• The original motivation for using social media data to estimate influenza prevalence is that it can be estimated in real-time, in contrast to traditional government systems

• Obtained data from Spinn3r during October 2008 to March 2009
   
• A total of 158,497,700 web and social media items were pulled
  -  when a lexical match exists to the term "influenze" and "flu" anywhere in its content

• The authors hypothesized that the frequency of blog-world flu posts correlate with a patient reporting influenza-like-illness and the US flu season
  -  they compared the web and social media data to CDC surveillance reports

• r=0.545

Pharmacovigilance

• Primarily involves the monitoring of adverse reactions caused by medications
   
• People discuss their use of prescription drugs, side effects and treatments on social media, which makes social media data unique and robust sources of information about health, drugs, and treatments
   
• Adverse reaction detection:
  -  automatic classification of user posts to determine if adverse reactions are mentioned
  -  data imbalance in social media data is challenging to train supervised learning models
   
• Discovering drug-adverse reaction associations:
  -  extract specific adverse reaction mentions and identify association between specific drugs and adverse reactions

Behavioral health

• Another rapidly expanding area is using social media data to understand behaviors that affect health
  -  behavioral medicine may play a prominent role in the digital surveillance revolution given the large knowledge gap in many areas of behavioral medicine
   
• Smoking and substance abuse:
  -  to understand availability of and interest in various nicotine and tobacco products such as electronic cigarettes
  -  to understand smoking cessation and online social support for cessation
  -  trends in alcohol use, problem drinking, prescription drug abuse
   
• Diet and fitness:
  -  analyze food consumption patterns using data from Instagram and Twitter
  -  study physical activities in Twitter by measuring outcomes of fitness goals

All of Us