The journey of
the Unofficial
Brunei Times
Archive

https://slides.com/thewheat/brunei-times-archive/

6th November 2016

Website - wget

https://www.guyrutenberg.com/2014/05/02/make-offline-mirror-of-a-site-using-wget/

Text

wget --mirror --convert-links 
  --adjust-extension 
  --page-requisites 
  --no-parent http://example.org

ePaper

How to get the files?

Live demo (fingers crossed): http://digital.nstp.com.my/

Been down this road before

Brunei Times Downloader Android app source

Download all the things!

mkdir 161001
wget -c "$URL/getZoom.jsp?id=160101bruneitimes&file=Zoom-1.jpg" -O 1.jpg
wget -c "$URL/getZoom.jsp?id=160101bruneitimes&file=Zoom-2.jpg" -O 2.jpg
wget -c "$URL/getZoom.jsp?id=160101bruneitimes&file=Zoom-3.jpg" -O 3.jpg
wget -c "$URL/getZoom.jsp?id=160101bruneitimes&file=Zoom-4.jpg" -O 4.jpg
wget -c "$URL/getZoom.jsp?id=160101bruneitimes&file=Zoom-5.jpg" -O 5.jpg
wget -c "$URL/getZoom.jsp?id=160101bruneitimes&file=Zoom-6.jpg" -O 6.jpg
...

zip -r 2016_10.zip 1610*

Offload a backup!

• Telegram 1.5GB files!

Bandwidth + Quota

telegram-cli

https://github.com/vysheng/tg

send_document <peer> <file>

If peer has a space, replace with _

./telegram-cli --enable-msg-id

history <peer>

load_document <msg-id>

Where to host?

Digital Ocean?

Amazon S3?

Google Photos!

Nothing official

https://productforums.google.com/forum/#!topic/photos/nMdGWGFSNeI

We love big red warnings

https://developers.google.com/picasa-web/

Still have a chance!

We have upload code!

Public Gist with upload code

    "require": {
        "google/apiclient": "^2.0"
    }

...

// Get a service account key from https://console.developers.google.com/ 
putenv('GOOGLE_APPLICATION_CREDENTIALS=./auth.json');

// email address of the user to upload data with
$user_to_impersonate = "EMAIL@DOMAIN.COM"; 

More about Google Oauth2 authentication 

and service accounts

Google Photos!

But.....

Wrong EXIF

Back to the docs!

😞

Time to share!

Back to the drawing board

Right the EXIF wrongs

exiftool '-datetimeoriginal=2016:01:18 12:00:00' \
   -overwrite_original_in_place test.jpg

Thank you @rjames86 for this wonderful reference gist on exiftool

Oldie but a goodie

API Docs with Libraries!

Thank you good API

More solutions more problems

API drops - 1 or 2 months per session

<!— status code : 502 —>
<!— Server Connection Closed —>
<!— host machine: r11.ycpi.tw1.yahoo.net —>
<!— timestamp: 1481898655.000 —>
<!— url: https://api.flickr.com/services/upload/-->

More solutions more problems

Out of ordered uploads (drag and drop...)

Internet to the rescue

Wee does rock me, Thanks Wee

  def order_photosets_by_title
    ordered_list = load_photoset_list.sort { |a,b| 
       (b[:title] || "") <=> (a[:title] || "") }
    ordered_ids = ordered_list.map { |set| 
       set[:id] }.join(',')
    @flickr.send_request('flickr.photosets.orderSets', 
       { :photoset_ids => ordered_ids }, :post)
  end

Thank you again good API

To the public!

Thank you yet again good API

Cover view creation!

33MB but... 1125 requests!

css.spritegen.com => 8 requests

Hosting part deux

• Github pages is my friend https://thewheat.github.io/brunei-times-archive/
• Free, HTTPS, backup for free

Lessons

• A good API is good
• Server side command line stuff is great
• Research end to end first!
• A good API is great
• Old tech is still good tech
• Google API servers are fast and robust
• Flickrs API servers are not so...
• A good API is fantastic