Making Media

Awesome

with

Machine Learning

Making the world a better place by using machine learning to extract rich information from images to categorise and process visual data

Steve Temple

Technical Director

@steve_gibe

A Bristol based digital agency specialising in Umbraco & ecommerce sites

Looking at how users actually use the media section

Media WTFs

90% of media files just in the root of the tree
The same image uploaded lots of times
10,000s of images organised in different ways depending on who uploaded them
- Date folders
- By section

Symptoms of a bigger problem?

There is no standard way of organising media
Relies on corporate policy/agreement
Or more often than not, no policy
Why should we have to worry about this?

Content Tree

Makes sense as a tree

Hierarchical structure mirrors sitemap

List view for when the metaphor breaks down

Media Tree

Doesn't mirror site structure

What folder structure makes sense?

Works OK for smaller sites,

but not so well for bigger sites with

10,000s or more images or

lots of content editors

The 2 types of user

The organised, everything in a folder kind

Their house probably looks like this

But it's OK because they're probably no fun at parties, but will remember to buy beers and snacks

The 2 types of user

The everything in the root kind and find it by searching

Their house probably looks like this

But that's also OK because they're probably fun at parties, but will have forgotten to bring beers and snacks

Faceted media search

Instead of a tree, use search to find media

Make that search rich and faceted

For that to be useful we would need to add tags, descriptions and other meta data to images

This is going to add a lot of overhead to adding media, we're going to have to get users to fill in that content for us

Unless we could generate that dynamically...

Machine learning Vision APIs

Machine learning has just now become available and affordable to everyone

APIs available from Azure, Google, AWS, IBM, etc.

Often free below generous usage limits

Automatically tag and categorise

Can also detect faces, moods, celebrities, colours and generate descriptions

Azure Vision API

Free for up to 5,000 transactions per month, then up to £1.864 per 1,000 transactions

Simple REST API

Supports:

Tagging
Face detection
Automatically cropping thumbnail
Colors
Image type
OCR
Adult image detection
Celebrity

Google Vision API

Free for up to 1,000 transactions per month, after that, $1.50 per 1,000 transactions

Simple REST API

Supports:

Labels (Tags)
OCR
Crop hints
Faces
Image properties
Landmarks (not what you think)
Logos
Safe search
Web

Demo

How it works

User uploads an image
The image is sent to Azure and Google
The responses from the APIs are merged
The additional data is stored on the image
The media item is saved in Azure Search for easy faceted searching

Where can I get this?

Intelligent Media package is available on Nuget

Media Search package available as source, but not ready for production...

Can see the rough and ready source code here:

https://github.com/Gibe/Gibe.Umbraco.MediaSearch

Install-Package Gibe.Umbraco.IntelligentMedia

Will this work for my site?

It depends...

It might help but I don't think it's there yet to replace the main media section

Very domain specific, for example, if you have a site selling cars, everything will have pretty much the same tag

How can we make it better?

Context aware facets:

Search by page they're included in
Search keywords based on where they're used

Image meta data:

Geolocation - Where the picture was taken
When picture was taken
Exposure/camera type/focal length etc

Support for more APIs:

IBM, AWS etc.

How can we make it better?

Custom Vision API, training for specific sites:

Has a feedback look based on modifications
Initially trained from other Vision APIs
Requires user involvement but relatively simple

More information

Azure

https://azure.microsoft.com/en-gb/services/cognitive-services/computer-vision/

Google

https://cloud.google.com/vision/