migrating thumbnail solution to thumbor

@namngology

 

Software Developer @ Kogan.com

By: Nam Ngo

Agenda

  • Current infrastructure and the problem
  • Scalable solution? Microservices?
  • More about Thumbor
  • Codebase migration
  • Go live, feature flags

current setup & The problem

App server

App server

App server

GlusterFS cluster

Volume

Volume

Volume

Issues

  • 100,000s of products  x  2-4 images x 10-20 thumb sizes
  • GlusterFS + small files =  :(
  • Thumbnailing code lives in main codebase 
  • Tightly coupled to Ecommerce platform
  • Cost

scalable solution? Microservice?

  • Thumbnailing service
  • Secure
  • Extensible
  • Large community & active development
  • Well documented
  • Storage backend: Amazon S3
  • Multiple engine support: OpenCV, PIL
  • Extra features (face recognition, webp) = plus
  • Written in Python = plus
  • Reusable across different projects

More about thumbor

http://server/unsafe/300x200/smart/www.mydomain.com/myfile.jpg

Thumbor server

size & filters

url to source image

  • Loaders: where to load the thumbnail from?
  • Storages: where to store the original image?
  • Result storages: where to store the thumbnail?

Codebase migration - Django integration

settings.py

    
    THUMBOR_SECURITY_KEY = 'supersecurekey'

    THUMBOR_SERVER 'http://thumbor.kogan.com'

    THUMBOR_MEDIA_URL = 'http://www.kogan.com/media'

pip install libthumbor


    from libthumbor import CryptoURL
    
    def get_thumbnail(image_path, **kwargs):
        crypto = CryptoURL(key=THUMBOR_SECURITY_KEY)
        image_url = '%s%s' % (THUMBOR_MEDIA_URL, image_path)
        thumbor_url = crypto.generate(image_url=image_url, **kwargs)
        
        return '%s/%s' % (THUMBOR_SERVER, thumbor_url)
    

sorl-thumbnail template tags


    {% load thumbnail %}
    {% thumbnail product.image "600x400" as image_file %}
        <img src="{{ image_file.url }}" width="{{ image_file.width }}" />
    {% endthumbnail %}

django-thumbor library


    {% load thumbor_tags %}
    <img src="{% thumbor_url product.image.url width=600 height=400 %}" 
    width="600" />

Backwards compatible templatetags

Context variable 


        from collections import namedtuple
        from core.thumbor import get_thumbnail


        Thumbnail = namedtuple('Thumbnail', 'url, width, height')

        thumbnail_url = get_thumbnail(url, **options)
        thumbnail = Thumbnail(thumbnail_url, **options)

        thumbnail.url
        thumbnail.width
        thumbnail.height

roll-your-own template tags


    {% load thumbor %}
    {% thumbnail product.image "600x400" as image_file %}
        <img src="{{ image_file.url }}" width="{{ image_file.width }}" />
    {% endthumbnail %}

FEATURE FLAg

pip install django-waffle


    from sorl.thumbnail.templatetags.thumbnail import ThumbnailNode
    from thumbor.templatetags import ThumborNode
    
    import waffle
    
    @register.tag
    def thumbnail(parser, token):
        if waffle.flag_is_active(request, 'use_thumbor'):
            return ThumborNode(parser, token)
        return ThumbnailNode(parser, token)

Incremental Migration  >  Big Bang Migration 

Kogan.com

is hiring

Developers

Hackers

Engineers

Scientists

 

careers.kogan.com

Q & A

Migrating thumbnail solution to Thumbor

By Nam Ngo

Migrating thumbnail solution to Thumbor

  • 1,543