Automated Bot

for Web/API Operations

Backend 讀書會

04/03

15:00~16:30

@Universe Tech

What Can Bots Do?

  • Periodical Operations

  • IoT Integrations

  • Application Integrations

  • Automatic Tests for Web App

  • Web Crawlers

  • Ticket/Product Sniping
  • Academic Research

Inside of Web Requests

How to monitor

your web requests?

Inside of Web Requests

Inside of Web Requests

Inside of Web Requests

Inside of Web Requests

curl 'https://kktix.com/g/events/wdsegy-03/base_info' \
  -H 'accept: */*' \
  -H 'accept-language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7,zh-CN;q=0.6' \
  -H 'cache-control: no-cache' \
  -H 'cookie: locale=zh-TW; kktix_session_token_v2=b50b9089129186074beb3cb4ad19a94c; XSRF-TOKEN=EwZ9Gsxx2Ct86sQlCNhTjl39OGcGmYgubq0%2FNJj77jyRRWQD%2BvWJz3bUKfIp%2BOH6wzl6A4TBQqfFD29y9YirXw%3D%3D' \
  -H 'pragma: no-cache' \
  -H 'referer: https://kktix.com/events/wdsegy-03/registrations/new' \
  -H 'sec-ch-ua: "Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36' \
  -H 'x-csrf-token: EwZ9Gsxx2Ct86sQlCNhTjl39OGcGmYgubq0/NJj77jyRRWQD+vWJz3bUKfIp+OH6wzl6A4TBQqfFD29y9YirXw==' \
  -H 'x-requested-with: XMLHttpRequest'

Inside of Web Requests

:status: 200
content-type: application/json
content-length: 2734
date: Sun, 31 Mar 2024 07:42:27 GMT
server: nginx
cache-control: no-cache
content-encoding: gzip
set-cookie: XSRF-TOKEN=ajQUXVXkqfPSFo2OfPhtoDl7%2Bw4fIoibBlFQI31%2BPvvodw1EY2D4F9goYFld2N%2FUp7%2B5ap16QhKt8wBlEA17mA%3D%3D; Path=/; Secure
vary: Accept-Encoding
x-cache: Miss from cloudfront
via: 1.1 a8e6d13425093fadbdcc8b1d50527eec.cloudfront.net (CloudFront)
x-amz-cf-pop: TPE51-C1
x-amz-cf-id: 4gSoR_FOgPBXSi7M-0ECgxqoM6Y_nKWWWL6PmSt495hpJObK99sisQ==

{"eventData":{"organization": {"contact_url":"http://kktix.com/organizations/chenhong/contact/new","fans_management_enabled":false,"google_analytics4_code":"","name":"辰紘行銷有限公司","public_domain":"chenhong.kktix.cc","show_contact_link":true,"slug":"chenhong"},
 "event":{"accept_atm":true,"accept_creditcard":true,"accept_famiport":false,"accept_famiport_pay_and_pickup":false,"accept_paypal":false,"adult_only":false,"allow_not_continuous_seats":true,"arena":{"id":2898,"img":"https://t.kfs.io/upload_images/arena_images/2898/__02.jpg","name":"臺北市兒童新樂園","sections":{"45570":{"id":45570,"label":"0524全場域","needAllocation":false,"coords":[112,133,113,353,257,353,257,317,324,317,324,354,467,354,468,134,468,133,468,133],"flag":[464,145],"tooltip":[484,174],"pegman":[157,152],"extraText":"","seatStyle":"auto_seating","color":"#333333","situations":[],"areas":{"63841":{"applicable":[702259,702274,702275],"id":63841}},"position":4,"displaySectionName":true,"displayLineNo":false,"displaySeatNo":true},"45571":{"id":45571,"label":"0525全場域","needAllocation":false,"coords":[111,374,112,495,469,495,467,375,467,375],"flag":[443,514],"tooltip":[478,546],"pegman":[158,523],"extraText":"","seatStyle":"auto_seating","color":"#333333","situations":[],"areas":{"63842":{"applicable":[702260,702274,702275],"id":63842}},"position":2,"displaySectionName":true,"displayLineNo":false,"displaySeatNo":true}}},"artist_id":[],"benefits":"","booking_skippable":false,"booking_ttl":600,"booth_enabled":false,"cancelable_by_user":false,"capacity":0,"captcha_type":1,"co_organizer":"","culture_subsidy":null,"end_at":"2024-05-25T13:30:00Z","entry_info":"","event_currency":"TWD","famiport_enabled":true,"famiport_pay_and_pickup_enabled":false,"fun_coupon_type":"N","gcal_url":"https://www.google.com/calendar/event?action=TEMPLATE\u0026text=%E3%80%90%E9%A0%90%E5%94%AE%E7%A5%A8%E3%80%91%E5%8C%97%E6%8D%B730+%E6%98%A5%E5%AD%A3%E9%9F%B3%E6%A8%82PARTY\u0026dates=20240524T110000Z/20240525T133000Z\u0026details=https://chenhong.kktix.cc/events/wdsegy-03\u0026location=%E5%8F%B0%E5%8C%97%E5%B8%82%E5%A3%AB%E6%9E%97%E5%8D%80%E6%89%BF%E5%BE%B7%E8%B7%AF%E4%BA%94%E6%AE%B555%E8%99%9F\u0026trp=true\u0026sprop=https://chenhong.kktix.cc/events/wdsegy-03\u0026sprop=name:KKTIX","geo_lat":25.0981175,"geo_long":121.515366,"has_geo":true,"ibon_center_note_1":"主辦:台北捷運公司、辰紘行銷","ibon_center_note_2":"一人一票、憑票入場,預計18:30開放進場","ibon_center_note_3":"禁帶外食及攝(錄、相)機等入場,如有未盡事宜請詳售票頁面說明","ibon_enabled":false,"ibon_left_note_4":"本節目採級距式退票,請詳閱該節目頁退換票規定,退票期限以退票寄達日為準","id":102569,"invite_user_to_join_org_fan":false,"is_kktix_handle_refund":true,"is_published":true,"kktix_refund_due_before":null,"kktix_refund_fee_percentage":null,"kktix_refund_time_limit_type":"normal_d","location":"臺北市兒童新樂園 ","location_address":"台北市士林區承德路五段55號","mailing_enabled":false,"max_to_buy":4,"max_to_buy_per_user":4,"milli_kkpoint_magnification":null,"more_info":"","name":"【預售票】北捷30 春季音樂PARTY","need_booking":true,"need_mobile_verified":true,"og_image_url":"https://t.kfs.io/upload_images/194725/KKTIX1200x630_v03.jpg","order_data_deadline":null,"org_id":36501,"organizer":"","pickup_types":[{"enable":true,"expires_at":"2024-05-25T13:30:00Z","fee_cent":0,"type":"qrcode"},{"enable":true,"expires_at":"2024-06-01T15:59:59Z","fee_cent":3000,"type":"famiport"}],"plan_company":"WALKIE","policy":"","public_url":"https://chenhong.kktix.cc/events/wdsegy-03","qrcode_enabled":true,"qrcode_expires_at":"2024-05-25T13:30:00Z","register_ttl":600,"registration_theme":"vertical","sales_launch_count_down":true,"short_description":"北捷30 春季音樂PARTY","show_people":false,"show_registrant_serial":false,"slug":"wdsegy-03","start_at":"2024-05-24T11:00:00Z","support_quick_registration":false,"time_zone":"Asia/Taipei","type":null,"visible_level":0},
 "payment_gateways":{"id":12701,"owner_type":"Organization","fee_twd_atm_payment_gateway":"ESUN_BANK_ATM_WALKIE_HOURLY","fee_twd_famiport_payment_gateway":"FAMIPORT_WALKIE","fee_twd_famiport_pay_and_pickup_payment_gateway":"","fee_twd_creditcard_payment_gateway":"NEWEBPAY_3D","fee_twd_paypal_payment_gateway":"","fee_hkd_creditcard_payment_gateway":"","fee_hkd_paypal_payment_gateway":"","fee_jpy_creditcard_payment_gateway":"","fee_jpy_paypal_payment_gateway":"","fee_usd_creditcard_payment_gateway":"","fee_usd_paypal_payment_gateway":"","fee_sgd_creditcard_payment_gateway":"","pickup_twd_booth":8000,"pickup_twd_mailing":8000,"pickup_hkd_mailing":500,"pickup_hkd_booth":1000,"company":"WALKIE","booking_fee":false},
 "order_qualifications":[],
 "tickets":[{"accounting_subject_id":null,"booking_fee_cents":0,"capacity":0,"disability_constraint":0,"end_at":"2024-05-24T10:00:00Z","end_at_for_registration":"2024-05-24T10:00:00Z","for_online_event":false,"id":702259,"limit_per_kkbox_uid":-1,"max_to_buy":4,"milli_kkpoint_magnification":null,"min_to_buy":1,"name":"預售單日票 05/24","need_invitation_code":false,"position":0,"price":{"cents":105000,"currency":"TWD"},"start_at":"2024-04-14T04:00:00Z","tax_free":false,"unit_to_buy":1,"unlimited_capacity":true},{"accounting_subject_id":null,"booking_fee_cents":0,"capacity":0,"disability_constraint":0,"end_at":"2024-05-25T10:00:00Z","end_at_for_registration":"2024-05-25T10:00:00Z","for_online_event":false,"id":702260,"limit_per_kkbox_uid":-1,"max_to_buy":4,"milli_kkpoint_magnification":null,"min_to_buy":1,"name":"預售單日票 05/25","need_invitation_code":false,"position":1,"price":{"cents":105000,"currency":"TWD"},"start_at":"2024-04-14T04:00:00Z","tax_free":false,"unit_to_buy":1,"unlimited_capacity":true}],
 "stop_selling_tickets":[],
 "contact_fields":[{"default_field":"name","description":null,"field_key":"field_text_876703","field_type":"text","i18n":{"name":{"default":"姓名","en":"Name","ja":"名前","zh-TW":"姓名"}},"id":876703,"is_required":true,"name":"姓名","options":null,"position":0},{"default_field":"email","description":null,"field_key":"field_email_876704","field_type":"email","i18n":{"name":{"default":"Email","en":"Email","ja":"Email","zh-TW":"Email"}},"id":876704,"is_required":true,"name":"Email","options":null,"position":1},{"default_field":"mobile","description":null,"field_key":"field_text_876705","field_type":"text","i18n":{"name":{"default":"手機","en":"Mobile","ja":"携帯番号","zh-TW":"手機"}},"id":876705,"is_required":true,"name":"手機","options":null,"position":2}],
 "attendee_fields":[]}}

Preprocess Requests

  • Login

    • Cookie-Session Based

    • Short-Term Token

    • Long-Term Token

    • Oauth2

    • Renew Token

    • Get first token manually?

  • Bypassing Anti-Bot Protection

  • CSRF Token

  • Application Defined Tokens

Preprocess Requests

  • Login Example

    • Init first cookies

    • Request login with cookies

    • Save logined cookies

    • Send other requests with cookies

Preprocess Requests

  • Login Example

protected function setCookiesAndToken()
{
    $this->info('Resetting cookies and auth token...');
    app('cookieJar')->clear();
    $response = $this->client
        ->get(static::LOGIN_URI, ['cookies' => app('cookieJar')])
        ->getBody()
        ->getContents();

    $pattern = '/<input(.*?)name=\"authenticity_token\"(.*)value=\"(.*?)\"/i';
    preg_match($pattern, $response, $matches);

    $this->authToken = end($matches);

    return $this;
}

Preprocess Requests

  • Login Example

protected function setLoginCookie()
{
    $this->info('Log in with username: ' . $this->username . '...');

    $loginResponse = $this->client->post(static::LOGIN_URI, [
        'form_params' => [
            'authenticity_token' => $this->authToken,
            'user' => [
                'login' => $this->username,
                'password' => $this->password,
                'remember_me' => '0',
            ]
        ],
        'cookies' => app('cookieJar')
    ]);

    $token = app('cookieJar')->getCookieByName('user_display_name_v2');
    // ......
    return $this;
}

Preprocess Requests

  • Login Example

protected function setRegisterInfo()
{
    $this->info("Checking status by event id: {$this->eventId}");
    try {
        $response = $this->client
            ->get(static::EVENT_URI . "{$this->eventId}/register_info", [
                'cookies' => app('cookieJar')
            ])
            ->getBody()
            ->getContents();
    } catch (ClientException $e) {
        $this->error("Invalid event id: {$this->eventId}");
        exit;
    }

    $result = json_decode($response, true);
    $status = $result['inventory']['registerStatus'];

    $this->ticketStatus =  $result['inventory']['ticketInventory'];
    $this->captchaContent = $result['ktx_captcha'] ?? null;
}

Act as A Real Human

  • Request Headers

    • Referer

    • More than necessary cookies

    • More than necessary headers

  • Avoid using VPN/Proxy IPs

  • Request Frequency
  • Reasonable delayed requests

Sniffing App Requests

  • Man in the middle

Sniffing App Requests

  • Sniffing Tools

    • Fiddler

    • Charles

    • HTTP Toolkit

    • mitmproxy

    • Wiresharks

Anti Bot Techniques

  • Captcha

  • Invisible Captcha

  • Detect Agents, like Headless Chrome, etc.

  • IP Detecting

  • TLS Handshake Detecting

  • Rate Limit for IP / Regions

  • Payload Encryption / Signature

  • SSL Pinging in Android / iOS

Bypassing Techniques

  • Captcha

    • OCR with Trained AI Model

      • ex: https://github.com/linsamtw/TaiwanTrainVerificationCode2text

    • Voice Recognition by AI

    • Selenium with Image Recognition

      • ex: https://blog.csdn.net/qq_52525445/article/details/122757053

    • Anti Captcha

      • https://anti-captcha.com

Bypassing Techniques

  • Anti Captcha

Bypassing Techniques

  • Native IP Proxy

    • https://www.lumiproxy.com/

    • https://www.lunaproxy.com/

    • https://www.iproyal.net/

    • https://www.webshare.io/residential-proxy

Bypassing Techniques

  • TLS Handshake

Bypassing Techniques

  • TLS Handshake


<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>
        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> inline.app</h2>
      </div><!-- /.header -->

      <div class="cf-section cf-highlight">
        <div class="cf-wrapper">
          <div class="cf-screenshot-container cf-screenshot-full">

              <span class="cf-no-screenshot error"></span>

          </div>
        </div>
      </div><!-- /.captcha-container -->

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>

            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>
          </div>
curl https://inline.app/api/menus?companyId=-LamXb5SAQN7JcJfyRKi%3Ainline-live-2a466&time=2024-03-31T00%3A00%3A00.000Z

Bypassing Techniques

  • curl-impersonate

    • A special build of curl that can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. curl-impersonate is able to perform TLS and HTTP handshakes that are identical to that of a real browser.

      curl-impersonate can be used either as a command line tool, similar to the regular curl, or as a library that can be integrated instead of the regular libcurl.

    •  

    • https://github.com/lwthiker/curl-impersonate

Bypassing Techniques

  • Headless Chrome (heavy cost)
    • https://github.com/chrome-php/chrome
use HeadlessChromium\BrowserFactory;

$browserFactory = new BrowserFactory();

// starts headless Chrome
$browser = $browserFactory->createBrowser();

try {
    // creates a new page and navigate to an URL
    $page = $browser->createPage();
    $page->navigate('http://example.com')->waitForNavigation();

    // get page title
    $pageTitle = $page->evaluate('document.title')->getReturnValue();

    // screenshot - Say "Cheese"! 😄
    $page->screenshot()->saveToFile('/foo/bar.png');

    // pdf
    $page->pdf(['printBackground' => false])->saveToFile('/foo/bar.pdf');
} finally {
    // bye
    $browser->close();
}

Bypassing Techniques

  • Scrapfly

    • https://scrapfly.io/

Discussion

Automated Bot for Web/API Operations

By Albert Chen

Automated Bot for Web/API Operations

  • 71