• Education:
• Ph. D. in Finance, NSYSU, TaiWAN.
• Ph. D. Candidate in MS&E, CSU, China.
• Experience:
• Assistant Professor (P/T),  NKFUST & NKU.
• Managing Director, Chihfeng Financial Advisors Ltd.
  
• Manager, Folion Financial Technology Inc.
• Software Engineer, U-Vision Biotech Inc.
• System Analyzer,  Taiyuan Ltd.
• Programming Skill:
• Basic, COBOL, C++/STL, SAS, Splus,  Matlab, PHP, Mathematica, R, Lingo, EViews, Stata.

• DIY your own  spideR is VERY EASY! 
  (Trust Me, You Can Make It! --> R Spirit)
• R is Easy & Powerful!
• Taiwan R User Group is Great!
• Keeping Yourself out of Trouble
• Legal Issues
• Stealthy Rules

• Encoding Issues and General Solution (In My Opinion)
• OS / Software /  Data /  URL / Server
• Some Other Useful Tips

• Goal:
• Scraping the news from a China website by R (spideR) 
• Tools: R
• RCurl (getURL)
• XML (readHTMLTable / XPath)
• RMessenger (Powered by Wush Wu)
• Platform OS:
•  Windows / Mac / Linux (Ubuntu) 
• Time (Coding)           : 7 Days
• Time (Implement) : 1 Month (In fact, ...)
• # of Query :     1,538,992
  
• # of Obs.      : 25,828,673 

• Legal Issues
• Copyright
• Assume “All Rights Reserved”
• Trespass to Chattels !!!
• Traditional Trespass: unauthorized use of real property (land or real estate) [新聞]
• Trespass to Chattels: prevents or impairs an owner’s use of or access to personal property
• Case: eBay Inc. vs Bidder's Edge Inc.

• Step 1: Analyzing a Form (Reverse Engineering)
• Chrome DevTools
  
• WebbotsSpiderScreenScrapers Form Analyzer (by PHP)
• Step 2: Automating Form Submission & Parsing Query Results
• Form Emulation
• Parsing Query Results (X) 
• Parsing with Basic String Functions 
• Parsing with Regular Expressions 
• Encoding Problemsss

• Fault-Tolerant Mechanism (X)
  
• Managing Large Amounts of Data (X)

• Form Elements
• Data Fields (Text / Select / Radio / Checkbox / ...)
• Method (GET / POST / Multipart Encoding)
• Form Handler
• Event Triger (Submit / onClick / onMouseOut / ...)

(DEMO)

• OS (Windows / Mac / Linux / ...)
• Software (RStudio / Revolution R / ...)
• Data (File / Database)
• URL
• Server

Character Encoding / 字元編碼 / 字符編碼

• OS: Utf-8 / Big5 
• Software: RStudio
• Sys.getlocale()
• [1]  "LC_COLLATE=Chinese (Traditional)_Taiwan.950;
  LC_CTYPE=Chinese (Traditional)_Taiwan.950;
  LC_MONETARY=Chinese (Traditional)_Taiwan.950;
  LC_NUMERIC=C;
  LC_TIME=Chinese (Traditional)_Taiwan.950"
  

• Software: RStudio
• Sys.getlocale()
• [1]  "LC_CTYPE=zh_TW.UTF-8;
  LC_NUMERIC=C;
  LC_TIME=en_SG.UTF-8;
  LC_COLLATE=zh_TW.UTF-8;
  LC_MONETARY=en_SG.UTF-8;
  LC_MESSAGES=zh_TW.UTF-8;
  LC_PAPER=C;
  LC_NAME=C;
  LC_ADDRESS=C;
  LC_TELEPHONE=C;
  LC_MEASUREMENT=en_SG.UTF-8;
  LC_IDENTIFICATION=C"

• ## [開始處] 先轉成 "C"
  Sys.setlocale(category = "LC_ALL", locale = "C")     # Q: C?

• ## 中間依據需求統一轉成特定編碼
  myStrVec = read.table("myFile.csv", sep=",", ... , encoding="UTF-8")
  myStr <- iconv(myStrVec[i], from="UTF-8", to="gb2312")
  myURL <- URLencode(myStr)
  myRes <- getURL(myURL, ... , .encoding='gb2312') 
  myRes <- readHTMLTable(myRes, encoding='gb2312', which=7)
• ## [結尾處] 再轉回 ""
  Sys.setlocale(category = "LC_ALL", locale = "")        # "" 為原本設定

• Sys.info()["sysname"]     # {base}  查詢 OS 的類型
  .Platform$OS.type           # {base}   
  
• Sys.getlocale()                      # {base}  查詢系統的 Encoding
  sessionInfo()                          # {utils} 

• Sys.setlocale()                       # {base}  設定系統的 Encoding
• Sys.setlocale(category = 'LC_ALL', locale = 'C')
• read.table()                             # {utils} 以特定 Encoding 載入資料
• myStrVec <-  read.table(myFile , ... , encoding='UTF-8')
• Encoding()                               # {base} 載入資料檔後轉換 Encoding
• Encoding(myStrVec) <- 'gb2312'
•  iconv()                                       # {base}  載入資料檔後轉換 Encoding
• myURL <- iconv(myStrVec[i] , from='UTF-8', to='gb2312')

• URLencode()                      # {utils}  對 URL 字串進行轉碼  (Query 傳送前)
• myURL <- URLencode(myURL)
• getURL()                               # {RCurl} 承接檢索結果 (Query 傳送後)
• myRes <- getURL(myURL, ... , .encoding='gb2312') 
• readLines()                      # {base} 承接檢索結果 (Query 傳送後)
• myRes <- readLines(myURL, encoding='gb2312')
• readHTMLTable()   # {XML} 解析並取出檢索結果中的特定表格
• myRes <- readHTMLTable(myRes, encoding='gb2312', which=7)

• Code

ChangeEncoding <- function(Flag) {
# Flag: 開頭 0；結尾 1
  if (Flag == 0) {
    switch(Sys.info()[['sysname']],     # .Platform$OS.type
    Windows  = {Sys.setlocale(category = "LC_ALL", locale = "C")},
    Mac      = {Sys.setlocale(category = "LC_ALL", locale = "C")},
    Linux    = {Sys.setlocale(category = "LC_ALL", locale = "C")})  } else if (Flag == 1) { 
    switch(Sys.info()[['sysname']],
      Windows  = {Sys.setlocale(category = "LC_ALL", locale = "")},
      Mac      = {Sys.setlocale(category = "LC_ALL", locale = "")},
      Linux    = {Sys.setlocale(category = "LC_ALL", locale = "")})     } else {
    break
  }
}

• RMessenger (Powered by Wush)
• Demo:
• ## Google Talk 設定區 
• MessFrom     <- "abc@gmail.com"
• MessFromPass <- "123"
• MessTo       <- "efg@gmail.com"
• ##  提取電腦名稱
• SysName <- Sys.info()["nodename"]
• ## 發送 Google Talk 訊息
• MessContent  <- paste("SysName:",SysName,"; NumDate:",d,"; \t",date(),"\n") 
• sendXMPPMessage(jid = MessFrom, password = MessFromPass, to = MessTo, message = MessContent)

• runif()   {stats}    +   Sys.sleep() {base}
• SleepSec = runif(10) * 100;
• Sys.sleep(SleepSec);

• na.omit()   {stats}
  
• myRes <- na.omit(myRes)

• Taiwan R User Group is GREAT! Join Us Now!
• Facebook (Taiwan R User Group)
• Youtube (TW use-R)
• Encoding - A General Solution In My Opinion
• Sys.setlocale(category = 'LC_ALL', locale = 'C')
      ...
  Sys.setlocale(category = 'LC_ALL', locale = '')
• Keep Yourself Out of Trouble 
• Legal Issues
• Stealthy Rules

• Taiwan R User Group
• 20130818 MLDM spideR (Ronny Wang) 
• 20130325 MLDM spideR -1 (c3h3)
  
• 20130325 MLDM spideR -2 (c3h3)

• COS 統計之都
• http://cos.name/cn/topic/17816 (RCurl 引介)
  
• Stack Overflow (很好的技術論壇)
  
• inside-R (比較美觀的 R 文件)
• Others
• http://zh.wikipedia.org/wiki/字符编码 (Wiki 百科)
  
• 网页爬虫-R语言实现，函数库文件 (DataGuru 煉數成金)
  
• 矛與盾大對決 (PHP Conf. Taiwan 2013)
• 金融業菜鳥實習生的 Python Project 初體驗 (Chia-yi Yen)
  

• Taiwan R User Group
• MLDM spideR Talks Series (Ronny Wang / Chia-Chi Chang)
• Chia-Chi Chang (c3h3)
• China News Spider Program (by Python)
• Meiyu Chen
• 中文編碼解析工具 EncAnalyzer132.zip
• Wush Wu
  
• RMessenger
• FreeDigitalPhotos.com