Nuxeo
&
Elasticsearch
NUXEO AND LUCENE
About Nuxeo
- Since 2001
- A team of 25 developers
- Making Opensource software since the very beginning
- Providing a software platform to build Content Applications
- Document Repository, Document Managament
- Digital Asset Management, Media Library
- Case Management, Document centric business process
- All business application that needs to manage Content
(i.e. : structured business object and not just files) - Trusted by Electonic Arts, JCDecaux, Boeing, Sharp ...
Nuxeo & ContenT
- Store Content
- Manage versions
- Secure access to Content
- Manage relations between Content objects
- Render content
- Convert content
- Manage worfklows
- Forms, Layouts and Tasks
- ...
- Index and Query
Nuxeo & Lucene
That's a long story :
Nuxeo integrated Lucene
with various technologies over the years
Is lucene the super-hero we looking for ?
Nuxeo / Lucene History
- 2006 : Nuxeo CPS 3.6 (Python / Zope based)
- Built-in index (Z-Catalog) was too weak
- Build a Lucene based index server accessed via XML-RPC
- pyLucene (Lucene re-built with GCJ + python bindings)
- Twisted as server framework
-
Complex setup
- 2007 : Nuxeo Platform 5.1 (Java + JCR)
- JCR is not good for making queries
- Integrate a Lucene based index via Compass Core
- provides transactional wrapper
- provides storage abstraction
- Concurrency issues
Nuxeo / Lucene now
- Since 2009 :
- VCS : Homebrew SQL based repository
- Everything is indexed inside the database (ACID)
- But :
- Full text capabilities are not so good
- Scaling for complex queries is a challenge
- Handling very big repositories is a challenge
- 2014 : Nuxeo 5.9.3 + elasticsearch
- Distribute indexes and queries on multiple nodes
- Relief the database
- Learn from our past mistakes !
RDBMS Limits
Static schemas
- RDBMS
- Schemas are defined beforehand
- not always possible
-
Schemas choice impacts performances
- Schemas migration are painful
Query performances
- SQL DB issues
- normalized data
- generate complex join and filtering
- several round trips to fetch a Document
- poor performances on unselective multi-criteria queries
- tweaking is possible but painful
- medium or poor full text support
-
vary depending on DB vendor
- Workarounds
- Stored procedures and triggers
(maintenance issues)
Query exAmple
- Some queries are nearly impossible to optimize
SELECT "hierarchy"."id" AS "_C1" FROM "hierarchy"
JOIN "fulltext" ON "fulltext"."id" = "hierarchy"."id"
LEFT JOIN "misc" "_F1" ON "hierarchy"."id" = "_F1"."id"
LEFT JOIN "dublincore" "_F2" ON "hierarchy"."id" = "_F2"."id"
WHERE
("hierarchy"."primarytype" IN ('Video', 'Picture', 'File', 'Audio'))
AND ((TO_TSQUERY('english', 'sydney') @@NX_TO_TSVECTOR("fulltext"."fulltext")))
AND ("hierarchy"."isversion" IS NULL)
AND ("_F1"."lifecyclestate" <> 'deleted')
AND ("_F2"."created" IS NOT NULL )
ORDER BY "_F2"."created" DESC
LIMIT 201 OFFSET 0;
Write overhead
- RDBMS engine constraints create overhead
-
buffer management
- locking & latching
- transaction isolation
- transaction log
- Application and model level constraints
- constraints checking
- type, unique, fk
- Triggers and associated stored procedures
Nuxeo ExAmple
- Triggers and constraints impact performances
-
ex : compute ACLs, compute path ...
- Most database don't handle correctly cascades
- ex: ON DELETE CASCADE
Scale out
-
RDBMS don't scale out easily
-
multi-master is very expensive and not very efficient
- ensuring ACID across the network is a real problem!
-
single master + multi-read is easy by not always applicable
- need to flag read only transactions at startup
- data partitioning is a solution
- but it impacts user experience (no global index)
Why Elasticsearch wins ?
- No Static Schemas
- essentially schema-less
- mapping can be adapted as needed
- Super query performances
- query on term using revert index
- returning document and score
- native full text support
- one query to fetch them all
- Fast indexing
- No ACID constraints
- Easy scale out
- Native distributed architecture
Why keep RDBMS ?
ACID can be useful
- Ensuring immediate consistency may be required
-
some user don't understand otherwise anyway
- ACID model ensure that data
- is always good
-
is auditable
- ACID RDBMS are safe and reliable
- people trust RDBMS to store their data
RDBMS are here for a long time
- Customers have Databases and DBAs
- Customers have BI tools that use RDBMS
- RDBMS are use for data interoperability
Hybrid model
- Use each storage for what it does the best
- RDBMS
- store content in an ACID way
(store & retrieve)
- elasticsearch
- provide powerful and scalable queries
- do the heavy lifting that the RDBMS can not do
Nuxeo
&
elasticsearch
Nuxeo Challenges
- Security filtering
- filtering may be complex and must be fast
- Hierarchy
-
recursive re-indexing needed for some operations
- Ensuring consistency
-
keep repository and index in sync
- User operation should be indexed real-time
- UI must be rendered using ES index
- ES index must be updated in pseudo real time
ES/Lucene challenges
- No support for update
- Document must be re-indexed completely
- No Join
- can not split hierarchy, security and data
in separated indexes
- No transaction management
Nuxeo Cluster
Nuxeo and SHARDING
ES to Lighten DataBase Load
ES and Sharding
Nuxeo 5.9.3 integration
- Export Nuxeo Document as JSON
- JSON is already the format of our REST API
-
Index mapping is configurable
- Asynchronous persistent tasks for indexing
- survive server restart and can be retried as needed
- Manage near-real time for UI operations
- indexing tasks triggered by UI in Post Commit
- NXQL query language and PageProvider model
- mapped to ES query language
Nuxeo 5.9.3 integration
- Security
- Previously tested approaches
- use a custom Lucene Matcher
- use index joins inside SOLR
- simplified ACLs is part of the indexed document
- Document fetching
- for now only the entry is taken from ES
- for now the Document is fetched from repository
- Integration
- run with embedded ES (test) or external server
Async indexing
Sync indexing
First results
Does it works ?
Yes, it does work !
Perspectives
Improving ES integration
- Improve integration
- load Documents from ES rather than repository
- UpdateByQuery for recursive updates
- Leverage elasticsearch features
- integrate Aggregates system with Nuxeo PageProviders
- move Audit Log to elasticsearch
- use Kibana for analytics
Nuxeo and Lucene
By Thierry Delprat
