How we can use AI to enhance Umbraco sites?

Steve Temple

Technical Director

About me

Technical Director @ Gibe

Will be using AI and Machine Learning interchangeably

Approaching Machine Learning as a developer

I am not a data scientist

Have been playing with Azure ML since it came out

First experiments were predicting horse racing results

Point of the tale

No finished out of the box solutions

Intention is to spark ideas and inspire packages

There is a massive learning curve to some of these

But for some solutions the hard work has been done an it's up to us to figure out how to use it

Machine Learning tools

On the shoulders of giants

 

Services available from amongst others:

Microsoft Azure, AWS,

Google, IBM

 

 

What can you use Machine Learning for?

Classification

Prediction

Anomaly detection

How can we use these to solve problems in Umbraco sites?

Classification

Categorising & tagging media or content

Problem to solve

How can we use AI to improve Umbraco's media section?

  1. No standard location for media
  2. Doesn't scale to 10,000s of images 
  3. Hard to find the exact image you want

 

The Media Tree

Does a tree make sense for Media?

Has some downsides:

Needs more information on each image/video/file

Editors would have to tag/categorise all media including existing media

Could we search instead?

Tagging of media

Use Azure Vision API to auto tag/categorise/detect features in images

 

DEMO

Great demo

But rubbish IRL

Custom Vision API

We can train our own model with our own tags which we want to detect

 

DEMO

 

What other APIs

could we use?

Focal point?

Video analysis?

Image cropper?

Visual search?

What about content?

Auto tagging / subject identification

Automated translation

https://our.umbraco.com/packages/backoffice-extensions/translation-manager/

Optical Character Recognition (OCR)

https://our.umbraco.com/packages/backoffice-extensions/ocr-content-app/

Prediction

Recommending other content

Problem to solve

How can we use AI to recommend content on a site?

Azure Machine Learning Studio

Azure Machine Learning Service

Replaces Azure ML Studio

Can use VS Code or Jupyter Notebooks

Azure Databricks

Very powerful

Very complicated!

Works with Azure ML Services

Solutions/Examples

https://gallery.azure.ai

Recommendations Solution

Prebuilt solution template that deploys Azure resources pre-configured for performing recommendations

 

Needs usage information and catalog information to train the model

 

https://github.com/Microsoft/Product-Recommendations

 

 

Recommendations solution

Needs to be supplied with 2 models to create recommendations

Usage

Models

How users are using the site

Catalog

Models

Product or content information

Configuring and using the API 

REST API with Swagger UI

  1. Create a trained model
  2. Make requests to the API for recommendations

 

DEMO

Suggesting media to editors

You might also like / related content

Estimated time to read

Personalisation

...

How about?

Anomaly Detection

Making sites more secure and easier for the users

Problem to solve

Can we use AI to make user/member logins simpler while being more secure

Login

Could we serve different user or member login experiences based on probability of the user being legitimate

Factors we might consider

IP Address

Country

Region

Time

Browser / cookie recognised

Previous logins

Recent failed attempts

Privileges

Score the attempt

> 98% 

Auto login?

Remember me, etc

75% +

Regular login

 < 75%

Show Recaptcha

Require 2FA

This all sounds super complex

How to get started

So much potential

Get a free azure account & experiment

There are multiple competitions/challenges all come with brilliant data sets to get started with

Steep learning curve, but look at sample solutions and work from there

 

Resources

https://studio.azureml.net/

https://azure.microsoft.com/en-gb/services/cognitive-services/

https://gallery.azure.ai/

https://github.com/Microsoft/Product-Recommendations

https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice

Thank you

Enhancing Umbraco with AI

By Steve Temple

Enhancing Umbraco with AI

  • 1,270