Amir Gal-or

10+ yr in software dev.

But first - a Dad

For 4.5 years

Some titles...

Senior SW dev

FrontEnd Tech Lead

Scale Platform Team & Tech Lead

DESTROYER OF BUILDS

Panaya is all about disruptive SAAS (Cloud Based) solutions for ERP lifecycle process...

?!

Who here worked with AngularJS ?

you have an Angular 1.x app

you want to upgrade to Angular 2.0 app

What do YOU need to do for that to happen ?

Is it a simple process ?

Relatively yes...

• Read Documentation ? 
• Do tutorials ?
• Take deep breaths ?
• Fix Code
• Compile

Good luck finding...

250 million lines of vanilla code !!!

did you read my previous bullet ?

did you read my previous bullet ?

Relative to what ?

Divide and Conquer - Large Scale !

Divide into small "computational" units

Each "unit" is sent to "Computational Engine"

Coordinate

Coordinate

Thread

Process

Machine

In Cloud - Over EC2

We have our own Framework for Elastic Cloud Computing

Written ~ 8 years ago

Today - you have

Over

We're Agile

I'm aware - today every company say they are AGILE

But they are all doing it differently!

Logicaly ?! can they all be right ?

PERHAPS...

When we started - it was more like ADHD

Now... It's better

(with room for improvement)

We have our ceremonies

PLANNING

DEMO

RETROSPECTIVE

DAILY

BACKLOG GROOMING

But - It's actually not about the ceremonies

It's a mindset !

more than a methodology

It's a culture

more than a way to manage

Continuous Improvment

Value to Customer

Trust

Transparancy

Priority

TEAM WORK

Intuition ?

Premenition ?

Experience ?

We have non of those

Sit with experienced people, developers, architects

Listen

Do homework

Research

Understand your expectations & limitations

Listen to Amir

NO!!!

Understand your "Expectations"

Understand your "Limitations"

What do you need to support for the next year (or 2)

Capture "rate"

SLA

SLA

SLA

SLA

Curation "period"

Processing time

OR

Do we have the required knowledge ?

Dev Support

DevOps support

Ops support

+

Troubleshooting

Understand principals in Scalability & Big-Data

There are allot of good options

Choices we make now might (and should) be invalidated in the future.

SO - you can't plan for everything - but you can try

Why ?

• Product ?
• Pricing ?
• Better options ?

Is the devil we know ;)

S3 became the standard "de facto" for scaling data curation, it is cheap, high availability, easy to use, and has extension in many processing Frameworks

Spark Over EMR is currently one of the best contenders as a "Big Data Processing FW" - it continues to remain so due to a large community of users and feature developers - relentlesly making it better

High security requirements - in all aspects.

AWS maintain security standards and has a built-in encryption and key management solution we're currently researching into.

Server

DB

Lambda

Kinesis

Firehose

S3

encrypted

Other Events

RAW

Supervised by

IAM + KMS

API Gateway

Lambda

Kinesis

Firehose

S3

encrypted

API Gateway

"front door" for applications to access data, BL, functionality in the BackEnd

event-driven function, code run in response to events from API Gateway

auto-magically buffers, than "dump" to S3 (every MB / seconds)

It's not a file-storage, It's a Key-Value storage

This is a requirement - "Key Per Customer"

IAM + KMS

Identity / Auth Management including Encrypt/Decrypt Key Management

Lambda

Kinesis

Firehose

S3

encrypted

API Gateway

"Single Point Of entrance" - will allow us not to bind code of "monitor" to AWS (by SDK). Good practice to control traffic and "Versioning"

Handling of incoming data for uses cases such as "License Validation" && / || "BlackList", as well as JSON validity and more.

As S3 is a by Key-Value storage (and not an FS) - there's no support for ops like "Append", so to generate a large file, a buffer is required

It's a Key-Value storage - sensitive data should be encrypted

This is a requirement - "Key Per Customer"

IAM + KMS

Identity / Auth Management including Encrypt/Decrypt Key Management