Making Media 

Awesome

with

Machine Learning

or

Making the world a better place by using machine learning to extract rich information from images to categorise and process visual data

Steve Temple

Technical Director

@steve_gibe

A Bristol based digital agency specialising in Umbraco & ecommerce sites

Looking at how users actually use the media section

Media WTFs

• 90% of media files just in the root of the tree
• The same image uploaded lots of times
• 10,000s of images organised in different ways depending on who uploaded them
  • Date folders
  • By section

Symptoms of a bigger problem?

• There is no standard way of organising media
• Relies on corporate policy/agreement
• Or more often than not, no policy
• Why should we have to worry about this?

Content Tree

Makes sense as a tree

Hierarchical structure mirrors sitemap

List view for when the metaphor breaks down

Media Tree

Doesn't mirror site structure

What folder structure makes sense?

Works OK for smaller sites,

but not so well for bigger sites with

10,000s or more images or

lots of content editors

The 2 types of user

The organised, everything in a folder kind

Their house probably looks like this

But it's OK because they're probably no fun at parties, but will remember to buy beers and snacks

The 2 types of user

The everything in the root kind and find it by searching

Their house probably looks like this

But that's also OK because they're probably fun at parties, but will have forgotten to bring beers and snacks

Faceted media search

Instead of a tree, use search to find media

Make that search rich and faceted

For that to be useful we would need to add tags, descriptions and other meta data to images

This is going to add a lot of overhead to adding media, we're going to have to get users to fill in that content for us

Unless we could generate that dynamically...

Machine learning Vision APIs

Machine learning has just now become available and affordable to everyone

APIs available from Azure, Google, AWS, IBM, etc.

Often free below generous usage limits

Automatically tag and categorise

Can also detect faces, moods, celebrities, colours and generate descriptions

Azure Vision API

Free for up to 5,000 transactions per month, then up to £1.864 per 1,000 transactions

Simple REST API

Supports:

• Tagging
• Face detection
• Automatically cropping thumbnail
• Colors
• Image type
• OCR
• Adult image detection
• Celebrity

Google Vision API

Free for up to 1,000 transactions per month, after that, $1.50 per 1,000 transactions

Simple REST API

Supports:

• Labels (Tags)
• OCR
• Crop hints
• Faces
• Image properties
• Landmarks (not what you think)
• Logos
• Safe search
• Web

Demo

How it works

1. User uploads an image
2. The image is sent to Azure and Google
3. The responses from the APIs are merged
4. The additional data is stored on the image
5. The media item is saved in Azure Search for easy faceted searching

Where can I get this?

Intelligent Media package is available on Nuget

Media Search package available as source, but not ready for production...

Can see the rough and ready source code here:

https://github.com/Gibe/Gibe.Umbraco.MediaSearch

Install-Package Gibe.Umbraco.IntelligentMedia

Will this work for my site?

It depends...

It might help but I don't think it's there yet to replace the main media section

Very domain specific, for example, if you have a site selling cars, everything will have pretty much the same tag

How can we make it better?

Context aware facets:

• Search by page they're included in
• Search keywords based on where they're used

Image meta data:

• Geolocation - Where the picture was taken
• When picture was taken
• Exposure/camera type/focal length etc

Support for more APIs:

• IBM, AWS etc.

How can we make it better?

Custom Vision API, training for specific sites:

• Has a feedback look based on modifications
• Initially trained from other Vision APIs
• Requires user involvement but relatively simple

More information

Azure

https://azure.microsoft.com/en-gb/services/cognitive-services/computer-vision/

Google

https://cloud.google.com/vision/