Abdullah Fathi
Set of techniques/processes used to discover relationships, recognize patterns, predict trends, and find associations in your data
Explains What Happened
Descriptive analytics juggles raw data from multiple data sources to give valuable insights into the past
Describing, summarizing, and identifying patterns through calculations of existing data, like mean, median, mode, percentage, frequency, and range
The baseline from which other data analysis begins.
It is only concerned with statistical analysis and absolute numbers, it can’t provide the reason or motivation for why and how those numbers developed
Minumum | Maximum | Mean | Standard Deviation | |
---|---|---|---|---|
Total Amount (RM) | 12 | 500 | 51 | 56 |
As we can see, each customer has spent an average of RM51, but some people have spent up to RM500
As we can see, most of the customers are between
30 and 40 years old
Explains Why Did Something Happened
Like descriptive analytics, diagnostic analytics also focus on the past
Look for cause and effect to illustrate why something happened
The objective is to compare past occurrences to determine causes
Provided in the context of probability, likelihood, or a distributed outcome.
Scatter charts might help to discover dependencies between the output variables and the input variables
The chart above shows that as the value of Feature 1 increases, the value of Target decreases
The maximum correlation (-0.287) is a yield between the recency and the conversion. Such a high correlation indicates that we have to study this variable more thoroughly
What is Likely To Happen
Uses the findings of descriptive and diagnostic analytics to detect clusters and exceptions and to predict future trends
Based on Machine Learning or Deep Learning
The more data points you have, the more accurate the prediction is likely to be
No analytics will be able to tell you exactly what WILL happen in the future.
Predictive analytics put in perspective what MIGHT happen, providing respective probabilities of likelihoods
given the variables that are being looked at
K-nearest neighbors is a very simple method used for classification and approximation
The outputs from the neural network depend on the inputs fed to it and the different parameters within the neural network
The graph above shows a neural network with four inputs (feature 1, 2, 3, and 4). When we introduce the values of the four features in the neural network, we get an output
What Action Should be Taken
Combines all of your data and analytics, then outputs a model prescription: What action to take
Analyze multiple scenarios, predict the outcome of each, and decide which is the
best course of action based on the findings
Prescribe what action to take to eliminate a
future problem or take full advantage of a promising trend
Uses Machine Learning, Algorithms, Artificial Intelligence (AI)
In this phase, we not only is predicted what will happen in the future using our predictive model but also is shown to the decision-maker the implications of each option
Traditional Analytics (BI) | Big Data Analytics | |
---|---|---|
Focus On | - Descriptive Analytics - Diagnostic Analytics |
- Predictive Analytics - Prescriptive Analytics |
Data Sets | - Limited Data Sets - Cleansed Data - Simple Model |
- Large Scale Data Sets - More types of data - Raw data - Complex data models |
Supports | Causation: What happened and why? | Correlation: New insight. More accurate answers |
A good data analyst will spend around 60-80% of their time cleaning the data. Focusing on the wrong data points will severly impact your analytical result
Wrong data class prevents calculations to be performed
Missing data prevents
functions to work properly
Outliers corrupt the output and produce bias
The size of the data requires too much computation
Deleting and exchanging methods have their drawbacks with small samples
Assumption: The data point is faulty
An outlier totally out of scale might be a wrong measurement
Outliers can be valid measurements and they might reveal hidden potentials
3 Methods applied:
Exploratory Data Analysis (EDA): To discover relationships between measures in data and to gain insight on the trends, pattern and relationship among various entities with the help of statistic and visualisation tools
Uni means one and variate means variable
Reflects how often an occurrence has taken place in the data. It gives a brief idea of the data and make it easier to find a pattern
IQ Range | Number |
---|---|
118-125 | 3 |
126-133 | 7 |
134-141 | 4 |
142-149 | 2 |
150-157 | 1 |
Example: The list of IQ scores is: 118, 139, 124, 125, 127, 128, 129, 130, 130, 133, 136, 138, 141, 142, 149, 130, 154
The bar graph is very convenient while comparing categories of data or different groups of data. It helps to track changes over time. Best to visualizing discrete data (variable that only store certain value)
Bar chart is a great way to display categorical variables in the x-axis. This type of graph denotes two aspects in the y-axis.
Similar to bar charts. Represent the group of variables with values in the y-axis
Mainly used to comprehend how a group is broken down into smaller pieces. The whole pie represents 100%.
Bi means two and variate means variable. Relationship between two variables
Represents individual pieces of data using dots. These plots make it easier to see if two variables are related to each other. The resulting pattern indicates the type (linear or non-linear) and strength of the relationship between two variables
Represents the strength of linear relationship between two numerical variables
Use for determining the association between categorical variables
A t-test is a statistical test that is used to compare the means of two groups. It is often used in hypothesis testing to determine whether a process or treatment actually has an effect on the population of interest, or whether two groups are different from one another.
You want to know whether the mean petal length of iris flowers differs according to their species. You find two different species of irises growing in a garden and measure 25 petals of each species. You can test the difference between these two groups using a t-test.
From the output table, we can see that the difference in means for our sample data is -4.084 (1.456 – 5.540), and the confidence interval shows that the true difference in means is between -3.836 and -4.331. So, 95% of the time, the true difference in means will be different from 0. Our p-value of 2.2e-16 is much smaller than 0.05, so we can reject the null hypothesis of no difference and say with a high degree of confidence that the true difference in means is not equal to zero.
Required when more than two variables have to be analyzed simultaneously. It is hard to visualize a relationship among 4 variables in graph.
Classify different objects into clusters in a way that the similarity between two objects from the same group is maximum and minimal otherwise
Reducing dimensionality of a data table with large number of interrelated measures.
Use various algorithms to build predictive models
Check the efficiency of our model
Are people who purchase tea more or less likely to purchase carbonated drinks?
Which categories does this document belong to?
Statistical technique used to identify trends and cycles over time
How well is our new return policy being received?
List of unsupervised learning algorithms
CART
Classification And Regression Trees
Classification Trees
Regression
Trees
X1 and X2 is Independent Variable
Y is our Dependent Variable which we could not see because it is in another dimension (z-axis)
Calculate Mean/Average for each leaf
We are not just predicting based on 1 Tree, We are predicting based on forest of trees. It will improve the accuracy of prediction because we take the average of many prediction
We can use other Distance as well such as Manhattan Distance. But Euclidean is the commonly used for geometry
Mach1: 30 wrenches/hr
Mach2: 20 wrenches/hr
Out of all produced parts:
We can SEE that 1% are defective
Out of all defective parts:
We can SEE that 50% came from mach1 And 50% cam from mach2
Question:
What is the probability that a part produced by mach2 is defective = ?
Assign class based on probability
Text
0.75 VS 0.25
0.75 > 0.25
CART
Classification And Regression Trees
Classification Trees
Regression
Trees
Clustering is similar to classification, but the basis is different. In Clustering we don’t know what we are looking for, and we are trying to identify some segments or clusters in our data. When we use clustering algorithms on our dataset, unexpected things can suddenly pop up like structures, clusters and groupings we would have never thought of otherwise.
We can apply K-Means for different purposes:
People who bought also bought ...