Full-Text Search in Django

Marco Alabruzzo

Lead Developer @ We Got Pop

http://marcoala.com - marco.alabruzzo@gmail.com

Django Day Copenhagen

25 September 2020

👋

Hello I'm Marco

🐣 Italian born 🇮🇹

🏠 London based 🇬🇧

📐 Lead developer 👨‍💻

We Got POP

❤️ Co-organiser @ Django London ❤️

https://www.meetup.com/djangolondon/

I play

Dungeons & Dragons

Full Text Search + DnD = 🤯

Why is search so important?

🤔

Different kind of search in Django

🛡
The Fighter
Standard textual queries
🧙‍♀️
The Wizard 
​External services
🎸
The Bard
Postgre Full-Text Search
⚔️
The Barbarian
Postgre Trigram

Standard textual queries

The simplest form of search

Zero Setup: comes with the database

Good for MVP

It's binary: something is there or is not

No ranking

The fighter

Intuitive, easy to get started, effective at low level

Potential at high levels is low

Pros

Cons

🛡

🛡

Standard textual queries

Good use case

Filters in the admin interface

Bad use case

Search for content in the homepage

🛡

Standard textual queries

External services

Can manage millions of records

Search related features

(spell checking, faceting, weighted search, special ranking, etc.)

Initial development investment

Increase the complexity of the architecture

You will have to manage a copy of your data

The Wizard

Very hard to get started, you need to know all the mechanics

Once you master it your possibilities are limitless

Pros

Cons

🧙‍♀️

Eleasticsearch, Apache Solr, Algolia

🧙‍♀️

External services

💰 💸

This is the high-cost/high-value solution

Hard to implement, but offer the best solution

Usual implementation

Define a schema of the search index

Index all the existing record of your database

Keep the index up to date with the DB

(post_save and post_delete signals)

🧙‍♀️

External services

Good use case

Big dataset, specific requirements on ranking, heavy request load

Bad use case

Having constraints around development time and/or architecture complexity

🧙‍♀️

External services

Postgre Full Text Search

Language dependent

Semantic(ish) search system

Advanced Ranking System

Search in more than one field

Highly configurable

The Bard

Diplomacy and Deception

Cons

Pros

🎸

How does it work?

 Normalise words 

Remove stop words

Apply a frequency score

🎸 Postgre Full Text Search

 Normalise words 

Remove stop words

Apply a frequency score

deadlines => deadlin

The, for, with

Deadline (7) that is less common than Web (3)

🎸 Postgre Full Text Search

🎸 Postgre Full Text Search

Ranking system

Multiple matches in the same record

Proximity of matches in the same record

Importance of the field

🎸 Postgre Full Text Search

Good use case

Search in the articles' content

Bad use case

Search for a list of movie titles

🎸 Postgre Full Text Search

Postgre Trigram

On large text size can be less precise than Full-Text Search

Language-Independent

Will find result even in case of misspelling

The Barbarian

Big, Fast, Strong, Angry

They can't read or write

(but they don't need to...)

Cons

Pros

⚔️

⚔️

Postgre Trigram

How does it work?

Spaces are prepended and appended to the string

The string is dived in group of 3 characters (trigrams)

The list of trigram is filtered from duplication and ordered

How does it work?

Spaces are prepended and appended to the string

The string is dived in group of 3 characters (trigrams)

⚔️

Postgre Trigram

The list of trigram is filtered from duplication and ordered

⚔️

Postgre Trigram

Misspells

⚔️

Postgre Trigram

Good use case

Search in a list of unique words

(names, movie titles, song titles, band names)

Bad use case

Search in the articles' content

⚔️

Postgre Trigram

Use a GIN Index

Apply a GIN index to your database before performing searches to optimise this two types of queries.

🎸⚔️

Performances on

Full Text and Trigram

Enjoy having different options

🛡
The Fighter
Standard textual queries
🧙‍♀️
The Wizard 
​External services
🎸
The Bard
Postgre Full-Text Search
⚔️
The Barbarian
Postgre Trigram

🙏 Thank you 🙏

Questions? 🤔

Full-Text search in Django

By Marco Alabruzzo

Full-Text search in Django

Full-Text Search is an old problem. There are many solutions: the classic SQL LIKE statement, external search index, 3rd party SaaS, PostgreSQL’s trigram… This talk will go through the difference between the various approaches, providing elements to identify which one is the best for each case.

  • 25
Loading comments...

More from Marco Alabruzzo