django-vector

the data access layer

Huh?

Why do we need another data access layer?

  • REST API ( calculate, expand)
  • Glossary

Issues with current data access

  • Slow
  • Don't give you full control over data
  • Payload format

Design goals

  • Fast
  • Flat schema
  • Self-documenting

Fast

1 query

+ vectorized calculations

 

supports pagination

Flat, Rich

 

REST API

{
  quote: {
    system: {
      site: {
        address_1: '564 Market St'
      }
    }
  }
}

 

Glossary

{
  solar_pmts_net_sum_yrs: 9155.44,
  solar_pmts_net_yr0: 0.0,
  solar_pmts_net_yr1: 610.36
  ...
}

 

django-vector

 

Self-documenting

No Errors*

Reads from database in a standard way - less room for error

* in theory

How does it work?

apps/solar/product/metrics.py


class ProductNameMetric(FieldMetric):
    model = SolarProduct
    identifier = 'product_name'
    field = 'name'


class ProductNamesMetric(ListAggregationMetric):
    identifier = 'product_names'
    dependencies = [ProductNameMetric]
def _build_solarsite_dataframe(solarsites):
    metrics = [
        AddressLine1Metric,
        CityNameMetric,
        ContactNameMetric,
        SolarSiteLastUpdatedMetric,
        SolarSiteIDMetric,
        StateAbbreviationMetric,
        ZipcodeMetric
    ]

    return django_vector.build_dataframe(
        solarsites,
        metrics
    )
build_dataframe
+ build_single

Metric Types

  # django_vector/metrics.py


  class FieldMetric

  class ConcatenationMetric

  class ListAggregationMetric

More Metric Types

# coming soon: more aggregation

class AverageAggregationMetric
class CountMetric

# future work

class DateFieldChangedMetric

class DateQuoteSignedMetric(DateFieldChangedMetric):
   model = Quote
   identifier = 'date_quote_signed'
   field = 'status'
   value = 'SIGN'

Metric interface

class Metric():
    """Base class which defines the Metric interface."""

    dependencies = []

    def calculate(self, df):
        raise NotImplementedError

    def get_dependency_fields(self):
        raise NotImplementedError

Use cases

Sales & Ops Pipelines

Custom API endpoints

  • When we calculate things like savings by default on the public api, things can get slow

 

"Hey customer can we turn off this field"

Why not use

  • graphql                            

 

  • django-rest-framework

Future integrations

  • Unify the filter service with metrics
  • Product rules could use metrics
  • External-facing "glossary"

Links

https://engine.sighten.io/api/data/             

https://github.com/sighten/django-vector

Made with Slides.com