Sign in

I split this part in a library(gem).

Check out amazon_auth gem.

• Simple and reversible conversion of credentials
  • Store it in dotenv(.env)
• Initialize Capybara session
• Basic commands to sign in and common utility

Sign in Amazon

Convert credentials

Download pages

I build a library for this part.
Check out kindle_manager gem.

I suppose users update data periodically(daily). This library will

• Have limit on downloading
  • It isn’t practical to Download all data every time
• Store(append) new pages to existing download directory
  • ‘session.save_page(html_path)’
  • ‘session.save_screenshot(image_path)’ (Not necessary)
• Don’t care duplicates of data among pages
  • Output will be the same if there is no update(idempotence)

Fetch kindle books list

Capybara or Nokogiri

def number_of_fetched_books
  # Capybara method
  wait_for_selector('.contentCount_myx')
  # Nokogiri method
  text = doc.css('.contentCount_myx').text
  ...
end

• 1st, ensure dom with capybara methods
  • capybara waits page loading in some way
• 2nd, read dom with nokogiri methods
  • Because it’s faster

Example of downloading

class BooksAdapter < BaseAdapter
  def load_next_kindle_list
    wait_for_selector('.contentCount_myx')
    current_loop = 0
    while current_loop <= max_scroll_attempts
      if limit && limit < number_of_fetched_books
        break
      elsif has_more_button?
        snapshot_page
        current_loop = 0
        log "Clicking 'Show More'"
        show_more_button.click
      else
        log "Loading books with scrolling #{current_loop+1}"
        session.execute_script "window.scrollBy(0,10000)"
      end
      sleep fetching_interval
      current_loop += 1
    end
    snapshot_page
  end

Implementation of downloading

Code is’t clear because

• Hard to test downloading part
• It could fail depending on machine
  • height of browser, network
• Response time from the site could be random
• Many code for logging
• Should use less instance variables

Output data

@parser.parse.first.asin
#=> "B004YW6M6G"
@parser.parse.first.title
#=> "Design Patterns in Ruby"

puts @parser.parse.to_json
#=> [{"asin":"B004YW6M6G","title":"Design Patterns in Ruby", ...

Print data on console

Troubles

• Security of amazon
  • You could be asked security question
• Compatibility of libraries
  • It’s getting harder to find compatible libraries for recent Firefox
    • FireFox + gechodriver + selenium-webdriver
  • Just use Chrome & chromedriver
• Normalization of data (名寄せ)
  • 全角スペース、全角英数 (full-width)
  • Special whitespace characters

Some fact(mystery) of amazon security

• Sleep on key strokes doesn’t help
• Security question could be displayed after some tries
  • 3 ~ 5 times of successful signin in a short time
  • CAPTCHA could be displayed 3 ~ 5 times in a row
    • when you click submit button with code
  • CAPTCHA passes when you press submit button manually
    • Do amazon check mouse movement or scroll or something?

Patterns of Security questions

• Ask characters in image (captcha)
• Ask registered phone number
• Ask registered zip code
• Ask security code through email

Security of credentials

I assume this library is used in private projects/machines.
It doesn’t have strong protection of credentials.

There is a tool called envchain which works with macOS Keychain.
This can be used as an alternative of dotenv.

The app I created

Tsundoku app can

• Fetch/import kindle books list
• Quick search/filter
• Tagging (free words)
  • Read, Hope to read
  • Sample (automatically tagged)
  • Checked etc.

Official kindle books site

Reading status, rating, public notes

Other usage of capybara

• Smoke test
  • Daily diagnostic
• Check sites requiring signin
  • Possible to check GMail
• Captures, Recording
• Data collection for data analysis
• more

General News

• System test in Rails5
• Chrome headless
  • phantomjs will be deprecated
  • capybara-webkit continues

Bonus (kindle highlights)

The site for Kindle notes and highlights is closing
(August 1st, originally July 3rd)

https://kindle.amazon.com/your_highlights

New site for highlights

New site for Kindle notes and highlights

https://read.amazon.co.jp/kp/notebook

Demo app for kindle highlights

I have an app to collect kindle highlights

• Check out kindle_highlight app
• And kindle_manager gem.
  • kindle_manager now supports to get kindle highlights
  • kindle_manager works inside heroku

Use chromedriver in heroku

heroku buildpacks:add https://github.com/heroku/heroku-buildpack-chromedriver
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-xvfb-google-chrome

Any questions

I have some experience of capybara/testing.
Ask me later if you have questions something like

• Debug spec which fails only on CI
  • Take screenshots in closed instances(Travis CI, heroku)
• Keep(save/restore) cookies in different sessions
• vcr (Record api testing of external services)

URL: http://slides.com/kyamaguchi/capybara

How to keep cookies with capybara

session = Capybara::Session.new(:chrome)
# login
session.visit 'https://github.com/login'
session.fill_in 'login_field', with: ''
session.fill_in 'password', with: ''
session.click_on 'Sign in'
# store cookies
data = Marshal.dump session.driver.browser.manage.all_cookies
File.open('all_cookies.txt', 'wb') {|f| f.write(data)}

session.driver.quit

Restore cookies with capybara

session = Capybara::Session.new(:chrome)
# First visit is required before restoring cookies
session.visit 'https://github.com/'
# restore cookies
data = File.read('all_cookies.txt')
Marshal.load(data).each do |d|
  session.driver.browser.manage.add_cookie d
end
session.visit session.current_url

Store cookies into database

• The data with Marshal doesn’t work with postgres text column
• Restored hash data from json column needs some conversion

# Store 'session.driver.browser.manage.all_cookies' into json column
cookies_from_db.each do |d|
  # :name needs to be symbol on 'add_cookie'
  d.symbolize_keys!
  # :expires needs to be Time class
  d[:expires] = Time.parse(d[:expires]) if d[:expires]
  session.driver.browser.manage.add_cookie d
end

Bonus 2 (Orders)

Created a gem to collect amazon orders data

Check out amazon_order

Fetch amazon orders

Load amazon orders