Event Driven BigQuery Consumer

Track BigQuery table updates and turn them into events

The Problem

What if we can track changes?

Introducing Two Special Information Schemas

Track changes using Metadata not Data

information-schema-streaming

information-schema-partitions

The Design Concept

Checkpointing

Start

End

Start

End

Gap

Correct

Assume, scheduler runs every minute

Code Example

Let's see how it works

Summary & Next Steps

Event Driven BigQuery Consumer

By Richard He

Event Driven BigQuery Consumer

Track BigQuery table updates and turn them into events

139

Richard He