# This is a simple recursive crawler that users [PhatomJS](http://phantomjs.org) to create snap shots of your site # and save them as static, SEO friendly HTML files. # # The crawler works on the basis that your app's URLs are # [RESTful](https://www.google.co.uk/search?q=restful+api+design) and compensates for this design by saving the # generated document's HTML in an `index.html` file. This means that if your URL is `/foo/bar`, it can then be accessed # via `/foo/bar/` and `/foo/bar/index.html` in addition to the original schema, fs = require 'fs' page = (require 'webpage').create() system = require 'system' # This crawler can be used directly with PhatomJS in the terminal by running # `phantomjs crawler.coffee http://your-server/ output-folder/ page-timeout ignore-regex`. base = system.args[2] home = system.args[1] ignoreRegex = system.args[4] pageTimeout = system.args[3] || 8000 if ignoreRegex ignoreRegex = new RegExp ignoreRegex # This `indexed` array stores all the URLs that have already been indexed. indexed = [] # And this `queue` array stores the URLs waiting to be indexed. queue = [] # This function returns the HTML document of the indexed page. getHTML = () -> # Since radian uses [RequireJS](http://requirejs.org/), this strips the `