Systems

Thinking...


And ImageSys

ImageSys


  • AU - 810,000                  (1,701,874 files)
  • BR - 7,400,000
  • DE - 3,550,000
  • ES - 4,200,000
  • FR - 5,500,000
  • IN - 1,000,00
  • IT - 6,000,000
  • MX - 400,000
  • UK - 3,000,000
  • Total: Lots




http://en.wikipedia.org/wiki/Russell_L._Ackoff

DIKW Pyramid

Peter Drucker



"Efficiency is doing things right;
effectiveness is doing the right things"




http://en.wikipedia.org/wiki/Peter_Drucker




If you improve the efficiency of something wrong, you make it MORE WRONG!


"It's better to do the right thing wrong, than the wrong thing right."






When you add that hack, are you making the wrong thing wronger?

More Ackoff


"There are two kinds of mistakes…[errors of commission and errors of omission] 


Errors of omission are much more important than errors of commission"

Functional Requirement


Produce thumbnails for the Nestoria website

Non-functional requirements

  • Process all new images every day
  • Ensure sync across n1, n2, n3
  • Purge old images
  • Be quick
  • Produce 'lean' images
  • Resiliance
  • Redundancy
  • Diagnostic tools
  • Monitoring
  • Metrics
  • Play nicely with our servers
  • Play nicely with our databases
  • Low maintainance
  • Disk space management
  • Log file rotation
  • ...

Elevator PItch


Lokku ImageSys works by accepting URLs from ETL and processing them by downloading, thumbnailing and then uploading the thumbnails to the frontend (n1, etc) servers.

Goals


    • Simple
    • Robust
    • Generalised
    • As limited customisation as possible
    • No close coupling with current Lokku systems 


Testing!

"Opinions are like ..."


http://twiki.lon.lokku.net/twiki/bin/view/Lokku/ImageSysThumbnail


Simplicity

  • ImageSys knows almost nothing about Lokku
  • We broke that rule when we needed to
  •  mysql> describe lis_live_images;            +------------------+
    | Field            |
    +------------------+
    | image_url        |
    | image_hash       |
    | datasource_id    |
    | image_data       |
    | insert_time      |
    | human_classified |
    | image_class      |
    +------------------+
    7 rows in set (0.01 sec)
  • Lokku::* only knows of Lokku::ImageSys::Interface

Examples

Search/Listings.pm

use Lokku::ImageSys::Interface qw(lis_get_searchapi_data_for_hash);
[...]
foreach my $image ( split(',', $rh_record->{image_data}) ) { ## ImageSys START my $ra_image_data = lis_get_searchapi_data_for_hash( $image ); if (defined($ra_image_data)) { $rh_listing->{image_data} = $ra_image_data; last IMAGEDATA; ## we know only one image per listing atm } ## ImageSYS END }

Examples

Listings/NormalizedListings.pm

        lisinterface      => Lokku::ImageSys::Interface->new({
            product       => $product,
        }),


    my $ra_images =
        $self->{lisinterface}->lis_images_for_si_build($raw_listing->{photo_url});

Vincent's Crazy


# Die if we're called from *modules* outside the image system 

my $caller = caller;
if ($caller !~ m/^Lokku::ImageSys/ && $caller ne "main") {
    croak "Lokku::ImageSys::Thumbnail is private code [...]";
}

Logging

Loggu!

Simple configuration!
No log file management!

Simple


 loggu_init({    module          => 'imagesys',
    component       => 'downloadd',
    verbosity       => defined($opt_verbose) ? $opt_verbose : 0,
    screen          => defined($opt_daemon) ? 0 : 1,
    email_threshold => $ERROR,
});

Logs


lokku@eb1:~ > ( cd common/logs/imagesys && ls downloadd* )
downloadd.error.log    downloadd.log.2
downloadd.error.log.1  downloadd.log.3
downloadd.error.log.2  downloadd.log.4
downloadd.log          downloadd.log.5
downloadd.log.1

Single Daemons


Daemons are perl scripts run directly from wigwam

No shell scripts running perl scripts running perl scripts

Daemons can be run from the command line!

Scripts are considered mostly harmless

Self-healing

Manual Run - Cron

# Run at 05:00 each Saturday0 5 * * sat : Error running weekly ImageSys lis_cron_frontend; $WIGDO $LOCKRUN --lockfile=$LOCKFILE_DIR/daily_lis_cron_frontend.lock -- nice -n 20 /home/lokku/code/bin/imagesys/lis_cron_frontend --all --verbose
lokku@n1:~ > lis_cron_frontend --help/home/lokku/code/bin/imagesys/lis_cron_frontend [options]
Commands:
            --all        Run all checks

            --fs_check   Run the file-system check
            --db_scan    Run the full live_images DB scan
Options:
            --country    Run only for this country (country name: uk, de, etc)
            --verbose    Run verbosely
            --help       This help
lokku@n1:~ > lis_cron_frontend --fs_check --country uk
Jun 19 09:37:14 INFO Cleaner::perform_daily_fs_check[15422]:uk Starting filesystem check for 'uk' - /home/lokku/common/imagesys/limg/uk


Manual Run - Daemons

 lokku@eb1:~ > servicectl stop imagesys-downloaddservicectl stop: stopping imagesys-downloadd... 
Waiting for 12899 to die...dead.
done.


lokku@eb1:~ > ~/code/bin/imagesys/lis_downloadd --country uk --verbose
Jun 19 09:11:26 INFO Utils::lis_worker[6881]: Worker: eb1
Jun 19 09:11:26 INFO Download::_get_chunk_of_work[6881]:UK Doing chunk of work for UK (size 500)
Jun 19 09:11:26 INFO DB::lis_gain_locks[6881]:UK UPDATE returned '0E0'
Jun 19 09:11:26 INFO DB::lis_gain_locks[6881]:UK No rows gained.  No work to do.
Jun 19 09:11:26 INFO Daemon::END[6881]:UK Exiting(time_to_die: 0)
lokku@eb1:~ > 


Self Healing


  • Daemon, heal thyself
  • Live Continuous Testing


Manager.pm

sub _check_stuck_jobs { }
sub _check_bad_retry_jobs { }
sub _check_bad_datasources { }

Lokku::ImageSys::DB::lis_worker_report

Metrics / Monitoring

What shall we monitor?

No idea.

Monitor everything - we have data for all!


lis_metrics

To Object or To Not Object

  • Objects
    • DB
    • Download
    • Pusher
    • Interface
    • Rsync (!?)
    • Thumbnail (!?)
  • Not Objects
    • Utils

Arg Passing

my $image_hash = lis_hash_for_url( "http://a.com/b.jpg" );

my $tree_path = lis_path_for_hash( $image_hash );

my $rh_process_details = lis_datasource_process_details('uk', 16);


my $rh_result = $Thumbnailer->create_thumbnails_of_image({
 http_response => $Response, # HTTP::Response object
image_hash => $image_hash, process_details => {
sharpen => '0x1', # a string gamma => 1.0, # 0-1 quality => 90, # 0-100 }, });


new()


  • Always takes arrayref
 

 my $Download = Lokku::ImageSys::Download->new({
                  worker  => 'workerid',        # Optional
                  country => 'uk',              # Optional
                  noproxy => 0,                 # Optional
                });






Fin

Systems View

By chris_lokku

Systems View

  • 656