Optum 

SCAO Business Intelligence Review

SCAO
Business Intelligence

Data Warehousing

SCAO
Business Intelligence

Data Warehousing

 

Providers

Optum 

SCAO Business Intelligence Review

Optum 

SCAO Business Intelligence Review

Providers

SCAO DW

Courts

Dept of Corrections

State
Police

Health and Human Services

Future
Data Sources

Optum 

SCAO Business Intelligence Review

Providers

Who provides data to
SCAO Data Warehouse?

  • SCAO data warehouse is 100-130GB in size
  • Approximately 760 million rows
  • Daily refresh takes 5 hours
  • Future data sources both internal and external

Consumers

Optum 

SCAO Business Intelligence Review

Consumers

Optum 

SCAO Business Intelligence Review

Consumers

SCAO DW

Named Search Application (NSA)

NSA Dashboards

Application

Server

BI-Query
Users

CoreTech

Teradata

SQL Assistant

State
Police

Health and Human Services

Optum 

SCAO Business Intelligence Review

Consumers

Who uses the data in the
SCAO Data Warehouse?

  • 2400 Name Search Application (NSA) users
  • 1000 NSA Dashboard users
  • 20-30 BI-Query users
  • Data feeds to other State Departments

Query Tool

Optum 

SCAO Business Intelligence Review

Query Tool

Optum 

SCAO Business Intelligence Review

Query Tool

SCAO DW

BI-Query
Users

SCAO
Model

Ad-Hoc queries

Optum 

SCAO Business Intelligence Review

Query Tool

SCAO DW

BI-Query
Designer

SCAO
Model

Designing the Model

Optum 

SCAO Business Intelligence Review

Query Tool

BI-Query
Designer

SCAO
Model

Distribution
Server

Publishing the Model

SCAO
Model

Optum 

SCAO Business Intelligence Review

Query Tool

BI-Query
Users

SCAO
Model

SCAO
Model

Distribution
Server

Consuming the Model

Optum 

SCAO Business Intelligence Review

Query Tool

SCAO DW

BI-Query
Users

SCAO
Model

Big Picture

BI-Query
Designer

SCAO
Model

Distribution
Server

SCAO
Model

Optum 

SCAO Business Intelligence Review

Query Tool

BI-Query (Open Text)

  • Ad-Hoc detail and summary queries
  • No query server required
  • Data model client-server distribution
  • No real Business Intelligence functions

BI-Tool Columnar

Optum 

SCAO Business Intelligence Review

BI-Tool Columnar

Optum 

SCAO Business Intelligence Review

Columnar

ID Name Court Disp Date
16 Brown, Jim C05 Guilty 03/15/16
9 Adams, Douglas D12 Not Guilty 02/10/16
26 Swank, Craig C07 Guilty 12/15/15

Typical results from a data warehouse query

ID Key Value
16 Name Brown, Jim
16 Court C05
16 Dist Guilty
16 Date 03/15/16
9 Name Adams, Douglas
9 Court D12
9 Dist Not Guilty
9 Date 02/10/16
26 Name Swank, Craig
26 Court C07
26 Dist Guilty
26 Date 12/15/15

BI Tools Prefer

are transformed...

Optum 

SCAO Business Intelligence Review

Columnar

ID Name Court Dist Date
16 Brown, Jim C05 Guilty 03/15/16
9 Adams, Douglas D12 Not Guilty 02/10/16
26 Swank, Craig C07 Guilty 12/15/15
ID Key Value
16 Name Brown, Jim
16 Court C05
16 Dist Guilty
16 Date 03/15/16
9 Name Adams, Douglas
9 Court D12
9 Dist Not Guilty
9 Date 02/10/16
26 Name Swank, Craig
26 Court C07
26 Dist Guilty
26 Date 12/15/15

BI Tools Prefer

Optum 

SCAO Business Intelligence Review

Columnar

How do BI-Tools prefer data

  • BI-Tools prefer columnar over rows
  • Rows may need to be pivoted
  • Where is the pivoting done, cost?
  • Doesn't favor detail data

BI-Tool Aggregates

Optum 

SCAO Business Intelligence Review

BI-Tool Aggragates

Optum 

SCAO Business Intelligence Review

Aggregates

Possibly
millions
of rows

Court Month Disp Count Sum
C07 03/2016 Guilty 45 $45,600
C07 03/2016 Not Guilty 33 $0
C07 03/2016 Dismiss 8 $0
C07 04/2016 Guilty 56 $89,800
C07 04/2016 Not Guilty 41 $0
C07 04/2016 Dismiss 14 $0
D67 03/2016 Guilty 112 $456,900
D67 03/2016 Not Guilty 78 $0
D67 03/2016 Dismiss 45 $0
D67 04/2016 Guilty 98 $390,500
D67 04/2016 Not Guilty 56 $0
D67 04/2016 Dismiss 23 $0
... ... ... ... ...
... ... ... ... ...
... ... ... ... ...

Summary Measures

...baked down into

Optum 

SCAO Business Intelligence Review

Aggregates

Court Month Disp Count Sum
C07 03/2016 Guilty 45 $45,600
C07 03/2016 Not Guilty 33 $0
C07 03/2016 Dismiss 8 $0
C07 04/2016 Guilty 56 $89,800
C07 04/2016 Not Guilty 41 $0
C07 04/2016 Dismiss 14 $0
D67 03/2016 Guilty 112 $456,900
D67 03/2016 Not Guilty 78 $0
D67 03/2016 Dismiss 45 $0
D67 04/2016 Guilty 98 $390,500
D67 04/2016 Not Guilty 56 $0
D67 04/2016 Dismiss 23 $0
... ... ... ... ...
... ... ... ... ...
... ... ... ... ...

Dimensions

Measures / Facts

Dates

Locations

Statuses

Teams

Hierarchies

Count

Sum

Average

Min - Max

others...

STDEV

others...

Optum 

SCAO Business Intelligence Review

Aggregates

How do BI-Tools handle
Summary Data?

  • Tons of details baked down into summaries
  • Focus on dimensions
  • Aggregate numbers into measures
  • Where is aggregation done, cost?

BI-Tool Share-It

Optum 

SCAO Business Intelligence Review

BI-Tool Share-It

Optum 

SCAO Business Intelligence Review

Share-It

SCAO DW

BI-Designer

Static
Dashboards

Business Intelligence Users

Measure
Summaries

Process

Measure
Summaries

Existing

Server

Static
Dashboards

Update

Process

Static
Dashboards

Daily

Optum 

SCAO Business Intelligence Review

Share-It

BI Solution based on sharing data

  • Measure-Summaries process done on SCAO DW
  • Dashboards are static-dated
  • Summary to detail drill-down possible
  • Where are summary and update process done, cost?

BI-Tool Move-It

Optum 

SCAO Business Intelligence Review

Optum 

SCAO Business Intelligence Review

Move-IT

SCAO DW

BI

Designer

Semantic

Database

Business Intelligence

Users

Rules

Detail
Data

Daily

New BI
Servers

Optum 

SCAO Business Intelligence Review

Move-IT (Cloud)

SCAO DW

BI Vendor Server(s)

BI

Designer

Semantic

Database

Business Intelligence

Users

Rules

Detail
Data

Vendors  Secure

Cloud

SOM

Firewall

SOM

Firewall

New BI
Servers

Daily

Optum 

SCAO Business Intelligence Review

Move-IT (Cloud)

BI Solution based on moving data

  • Detail data copied to New BI-Servers daily      
  • BI-Server handles Measure-Summaries process
  • Summary to detail drill-down varies
  • Frequency, how much data copied,
    where saved, cost?

BI-Tool Hybrid

Optum 

SCAO Business Intelligence Review

BI-Tool Hybrid

 

Optum 

SCAO Business Intelligence Review

Hybrid

SCAO DW

Existing

Server

Business Intelligence Users

Measure

Summeries

Measure
Summaries

Process

Existing NSA

REST

Graphics

Charting

JSON

Drill Down

Detail

Data

The web
is built on
JavaScript

Optum 

SCAO Business Intelligence Review

Hybrid

REST

Graphics

Charting

JSON

Drill Down

Bootstrap

Node

Bootstrap is the most popular HTML, CSS, and JS framework for developing responsive, mobile first projects on the web.

Node is open-source, cross-platform runtime environment for developing server-side Web applications, based on powerful Chrome V8 engine

REST (REpresentational State Transfer) is an approach to communications that is often used in the development of Web services.  REST is often preferred  because it does not leverage as much bandwidth, which makes it a better fit for use over the Internet

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you to visualize data. D3’s emphasis on web standards gives you the full capabilities of modern browsers.  

C3.JS makes it easy to generate D3-based charts by wrapping the code required to construct the entire chart. Hundreds of reusable chart templates are available.

Teradata SQL provides both detail and measure-summary access via the REST interface and JSON data exchange to JavaScript.  It's the SQL you know and love.

Optum 

SCAO Business Intelligence Review

Hybrid

More

Optum 

SCAO Business Intelligence Review

Hybrid

BI Solution based on
something old & something new

  • Measure-Summaries process done on Teradata  
  • Extend existing Name Search Application (NSA)
  • Popular current and powerful JavaScript libraries
  • Summary to drill-down very easy

BI-Tool New Desktop

Optum 

SCAO Business Intelligence Review

BI-Tool New Desktop

Optum 

SCAO Business Intelligence Review

New Desktop

SCAO DW

Business Intelligence Users

Measure
Summaries

Measure
Summaries

Process

REST

Graphics

Charting

JSON

Drill Down

Detail

Data

The web
is built on
JavaScript

Electron

React

Node

Optum 

SCAO Business Intelligence Review

New Desktop

Who writes "Desktop" applications anymore?

Optum 

SCAO Business Intelligence Review

New Desktop

BI Solution based on
new desktop
methodologies

  • Measure-Summaries process done on Teradata
  • Desktop application that behaves like a web application
  • Extends JavaScript to Windows, Apple and Linux
  • Ad-Hoc queries, dashboards, reporting and big data

SCAO DW-BI

By John H Schuster

SCAO DW-BI

Presentation to be embedded in pwc-lms.com/JDW-BI

  • 1,840