Q: How does a programmer surf the web? A: With a scraper.
There are several sites I visit periodically to track a specific set of data relevant to my interest. This rinse and repeat process quickly becomes boring and prone to error. Now I started thinking as a programmer…if only a piece of software can do this for me, I can spend the free time on drinking coffee and surfing other sites! Apparently this is pretty much what a web crawler does and more specifically “scraping” is the term used to describe targeted searches. Below is a two part video on a couple of scraping examples I’ve built and a basic tutorial on how to build your own.
Part 1: Introduction
Part 2: Implementation
To run the main class in a Linux env:
java -cp “jsoup-cronjob-pubmed.jar:lib/*” com.adfhomebrew.jsoup.cronjob.pubmed.PubMedSearchClient
mysql jdbc driver: http://dev.mysql.com/downloads/connector/j/5.0.html
The live examples and links to tutorial code can be found at http://www.adfhomebrew.com