_                 _              
    | |__   ___  _ __ | |_ __ _ _ __  
    | '_ \ / _ \| '_ \| __/ _` | '_ \ 
    | |_) | (_) | | | | || (_| | | | |
    |_.__/ \___/|_| |_|\__\__,_|_| |_|


Bontan is a simple scraper with specialized behavior for some sites (like
Wikipedia) and smart fallbacks for others. This repository also includes other
scraping utilities.

- **bontan:** Attempt to scrape all useful text content on the page.
- **kinkan:** Like Bontan, but as a summary - just grabs the first paragraph. ("kinkan" is Japanese for "kumquat".)
- **sudachi:** Simple utility to scrape using css selectors. 
- **mikan:** A simple clone of [import.io](http://import.io). Prints each item in the biggest list on the page.

# Examples

First install it: 

    npm install -g bontan

### bontan

For a full text version of the page:

    bontan 'https://en.wikipedia.org/wiki/Pomelo'

Images will have their `src` printed.

### kinkan

Like Bontan, but with less output - typically it will use the page title, the first image, and the first `p` element.

    kinkan 'https://en.wikipedia.org/wiki/Pomelo'

    Pomelo - Wikipedia, the free encyclopedia
    /wikipedia/commons/thumb/1/1c/Citrus_grandis_-_Honey_White.jpg/220px-Citrus_grandis_-_Honey_White.jpg
    Citrus maxima (or Citrus grandis), (Common names: shaddick,[1] pomelo, pummelo, pommelo, pamplemousse, or shaddok) is a natural (non-hybrid) citrus fruit, with the appearance of a big grapefruit, native to South and Southeast Asia.

### sudachi

For grabbing things by css selectors. It uses a virtual dom ([domino](https://github.com/fgnass/domino)), which makes it comparatively fast but unable to handle contents generated by JS after page load. Let's try getting all the `h3` elements:

    sudachi 'https://en.wikipedia.org/wiki/Pomelo' h3

    Possible non-hybrid pomelos[edit]
    Hybrids[edit]
    Personal tools
    Namespaces
    etc.

You can pass `-r` to return `innerHTML` instead of `textContent`.

    sudachi -r 'https://en.wikipedia.org/wiki/Pomelo' h3

    <span class="mw-headline" id="Possible_non-hybrid_pomelos">Possible non-hybrid pomelos</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Pomelo&amp;action=edit&amp;section=5" title="Edit section: Possible non-hybrid pomelos">edit</a><span class="mw-editsection-bracket">]</span></span>
    <span class="mw-headline" id="Hybrids">Hybrids</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Pomelo&amp;action=edit&amp;section=6" title="Edit section: Hybrids">edit</a><span class="mw-editsection-bracket">]</span></span>
    Personal tools
    Namespaces
    etc.

### mikan

This attempts to replicate some of the magic of [import.io](http://import.io) using a simple trick - usually, the most interesting list on a page is the longest one. Here's what happens when you point it at Stack Overflow:
    
    mikan 'http://stackoverflow.com'
       0 votes   1 answer   4 views     Unique index not working  ruby-on-rails unique-constraint database-indexes   answered 1 min ago Thong Kuah 1,960   
       0 votes   0 answers   2 views     fprintf giving me a blank .txt file in MATLAB  matlab   asked 1 min ago physicist82 1   
       0 votes   0 answers   11 views     Node.js / Inheritance of variables and modules  javascript node.js inheritance   modified 1 min ago MiddleWare 138   
       5 votes   2 answers   30 views     Global Events in Angular 2  angular2   modified 1 min ago pixelbits 14.5k   
       0 votes   1 answer   5 views     Running “mvn test site” giving [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project  maven selenium xslt   modified 1 min ago Tunaki 29k   
     etc.

Or Hacker News:

    mikan 'http://news.ycombinator.com'
     1.      The Trouble with the TPP, Day 5: Rights Holders “Shall” vs. Users “May” (michaelgeist.ca)
     46 points by walterbell 3 hours ago  | discuss
    
     2.      Tesla Model S can now park itself (techcrunch.com)
     151 points by prostoalex 6 hours ago  | 89 comments
    
     3.      Nvidia GPUs can break Chrome's incognito mode (charliehorse55.wordpress.com)
     374 points by charliehorse55 11 hours ago  | 120 comments

     etc.

# License

WTFPL, do as you please.

-POLM