Systems
Thinking...
And ImageSys
ImageSys
- AU - 810,000 (1,701,874 files)
- BR - 7,400,000
- DE - 3,550,000
- ES - 4,200,000
- FR - 5,500,000
- IN - 1,000,00
- IT - 6,000,000
- MX - 400,000
- UK - 3,000,000
- Total: Lots
DIKW Pyramid
Peter Drucker
"Efficiency is doing things right;
effectiveness is doing the right things"
http://en.wikipedia.org/wiki/Peter_Drucker
If you improve the efficiency of something wrong, you make it MORE WRONG!
"It's better to do the right thing wrong, than the wrong thing right."
When you add that hack, are you making the wrong thing wronger?
More Ackoff
"There are two kinds of mistakes…[errors of commission and errors of omission]
Errors of omission are much more important than errors of commission"
Functional Requirement
Produce thumbnails for the Nestoria website
Non-functional requirements
- Process all new images every day
- Ensure sync across n1, n2, n3
- Purge old images
- Be quick
- Produce 'lean' images
- Resiliance
- Redundancy
- Diagnostic tools
- Monitoring
- Metrics
- Play nicely with our servers
- Play nicely with our databases
- Low maintainance
- Disk space management
- Log file rotation
- ...
Elevator PItch
Lokku ImageSys works by accepting URLs from ETL and processing them by downloading, thumbnailing and then uploading the thumbnails to the frontend (n1, etc) servers.
Goals
- Simple
- Robust
- Generalised
- As limited customisation as possible
- No close coupling with current Lokku systems
Testing!
"Opinions are like ..."
http://twiki.lon.lokku.net/twiki/bin/view/Lokku/ImageSysThumbnail
Simplicity
- ImageSys knows almost nothing about Lokku
- We broke that rule when we needed to
-
mysql> describe lis_live_images;+------------------+ | Field | +------------------+ | image_url | | image_hash | | datasource_id | | image_data | | insert_time | | human_classified | | image_class | +------------------+ 7 rows in set (0.01 sec) -
Lokku::* only knows of Lokku::ImageSys::Interface
Examples
Search/Listings.pm
use Lokku::ImageSys::Interface qw(lis_get_searchapi_data_for_hash);[...]foreach my $image ( split(',', $rh_record->{image_data}) ) {## ImageSys START my $ra_image_data = lis_get_searchapi_data_for_hash( $image ); if (defined($ra_image_data)) { $rh_listing->{image_data} = $ra_image_data; last IMAGEDATA; ## we know only one image per listing atm } ## ImageSYS END }
Examples
Listings/NormalizedListings.pm
lisinterface => Lokku::ImageSys::Interface->new({
product => $product,
}),
my $ra_images = $self->{lisinterface}->lis_images_for_si_build($raw_listing->{photo_url});
Vincent's Crazy
# Die if we're called from *modules* outside the image system
my $caller = caller;
if ($caller !~ m/^Lokku::ImageSys/ && $caller ne "main") {
croak "Lokku::ImageSys::Thumbnail is private code [...]";
}
Logging
Loggu!
Simple configuration!
No log file management!
Simple
loggu_init({ module => 'imagesys',
component => 'downloadd',
verbosity => defined($opt_verbose) ? $opt_verbose : 0,
screen => defined($opt_daemon) ? 0 : 1,
email_threshold => $ERROR,
});Logs
lokku@eb1:~ > ( cd common/logs/imagesys && ls downloadd* )
downloadd.error.log downloadd.log.2
downloadd.error.log.1 downloadd.log.3
downloadd.error.log.2 downloadd.log.4
downloadd.log downloadd.log.5
downloadd.log.1
Single Daemons
Daemons are perl scripts run directly from wigwam
No shell scripts running perl scripts running perl scripts
Daemons can be run from the command line!
Scripts are considered mostly harmless
Self-healing
Manual Run - Cron
# Run at 05:00 each Saturday0 5 * * sat : Error running weekly ImageSys lis_cron_frontend; $WIGDO $LOCKRUN --lockfile=$LOCKFILE_DIR/daily_lis_cron_frontend.lock -- nice -n 20 /home/lokku/code/bin/imagesys/lis_cron_frontend --all --verbose
lokku@n1:~ > lis_cron_frontend --help/home/lokku/code/bin/imagesys/lis_cron_frontend [options]
Commands:
--all Run all checks
--fs_check Run the file-system check
--db_scan Run the full live_images DB scan
Options:
--country Run only for this country (country name: uk, de, etc)
--verbose Run verbosely
--help This help
lokku@n1:~ > lis_cron_frontend --fs_check --country uk Jun 19 09:37:14 INFO Cleaner::perform_daily_fs_check[15422]:uk Starting filesystem check for 'uk' - /home/lokku/common/imagesys/limg/uk
Manual Run - Daemons
lokku@eb1:~ > servicectl stop imagesys-downloaddservicectl stop: stopping imagesys-downloadd...
Waiting for 12899 to die...dead.
done.
lokku@eb1:~ > ~/code/bin/imagesys/lis_downloadd --country uk --verbose Jun 19 09:11:26 INFO Utils::lis_worker[6881]: Worker: eb1 Jun 19 09:11:26 INFO Download::_get_chunk_of_work[6881]:UK Doing chunk of work for UK (size 500) Jun 19 09:11:26 INFO DB::lis_gain_locks[6881]:UK UPDATE returned '0E0' Jun 19 09:11:26 INFO DB::lis_gain_locks[6881]:UK No rows gained. No work to do. Jun 19 09:11:26 INFO Daemon::END[6881]:UK Exiting(time_to_die: 0) lokku@eb1:~ >
Self Healing
- Daemon, heal thyself
- Live Continuous Testing
Manager.pm
sub _check_stuck_jobs { }
sub _check_bad_retry_jobs { }
sub _check_bad_datasources { }Lokku::ImageSys::DB::lis_worker_report
Metrics / Monitoring
What shall we monitor?
No idea.
Monitor everything - we have data for all!
lis_metrics
To Object or To Not Object
- Objects
- DB
- Download
- Pusher
- Interface
- Rsync (!?)
- Thumbnail (!?)
- Not Objects
- Utils
Arg Passing
my $image_hash = lis_hash_for_url( "http://a.com/b.jpg" );my $tree_path = lis_path_for_hash( $image_hash );
my $rh_process_details = lis_datasource_process_details('uk', 16);
my $rh_result = $Thumbnailer->create_thumbnails_of_image({
http_response => $Response, # HTTP::Response objectimage_hash => $image_hash, process_details => {sharpen => '0x1', # a string gamma => 1.0, # 0-1 quality => 90, # 0-100 }, });
new()
- Always takes arrayref
my $Download = Lokku::ImageSys::Download->new({
worker => 'workerid', # Optional
country => 'uk', # Optional
noproxy => 0, # Optional
});
Fin
Systems View
By chris_lokku
Systems View
- 656

