Daemons

NLP

GEO

DUP

ETL

dem_db

Code Layout

Libs

Lokku::Daemon::*

Lokku::NLP::KeywordExtractor

Lokku::Geo::GeoCoder

Lokku::Listings::Deduper

Scripts

code/bin/daemons

Services

dem-nlpd

dem-geod

dem-dupd

ListingETL

• Moose object
• ORM-lite-lite
• Read-only
• JOINs ALL tables to represent a single listing
• Uses Lokku::DB::ListingDB::ListingsRole
• Enforces types
• No local logic
• Abstracts attributes as hashes (eg. keywords)

Lokku::Listings::ListingETL

Database

| Field           | Type                                      |
| id              | int(10) unsigned                          |
| status          | enum('READY','STARTED','FAIL','COMPLETE') |
| worker          | varchar(10)                               |
| insert_time     | timestamp                                 |
| mtime           | timestamp                                 |
| comment         | varchar(1024)                             |
| orig_listingetl | blob                                      |

Metrics

http://metrics.lokku.net/cgi/dem_workers.cgi

 

http://metrics.lokku.net/cgi/dem_metrics.cgi

Doing Stuff

Process a single listing

lokku@eb1:~ > ~/code/bin/daemons/dem_geod --country uk --listingid 151892333

 

Reset a listing or listings

~/code/bin/daemons/dem_db  --reset --id 128438625 --country uk --type nlp

 

Run a bunch of daemons to help process a long queue

~/code/bin/daemons/start-daemon dem_dupd 5

(Be careful not to clash with current daemons)