# openwebspider

## 0.3.0
* Support for PostgreSQL
* Update Mode
* Support for HTTP Headers: Last-Modified and ETag
* Indexer Web Service
* Example tools using Indexer Web Service

## 0.2.3
* Support for MongoDB (2.6+)
* Support to UTF-8 pages
* Fixed a bug in pdf.js
* Changed license to MIT

## 0.2.1
* Delete duplicate pages (option)
* Support for <META> ROBOTS tag (<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">)
* Support for <BASE> URL (<base href="http://www.w3schools.com/images/" target="_blank">)
* New plugin: regex
* Saving last URLs (worker + pages map) + last search query
* Minor fixes