============================================================================ # Standalone executable of DEiXTo Perl Executor for Windows # ---------------------------------------------------------------------------- Description: deixto_executor.exe is a standalone executable (command line utility) that executes DEiXTo generated wrapper project files (*.wpf) and produces user specified formatted output (tab delimited, XML, HTML, RSS, CSV, Excel, ODS). It also provides database support via DBI (the Database independent interface for Perl) and a dbconfig file mapping the extracted data fields to database table columns. For HTML output specifically, an HTML template processor is being used in combination with a simple editable "template.tmpl" file (contained in the zip file). Moreover, it provides a flexible and efficient postprocessing mechanism via a config file. DEiXTo is a free yet powerful web data extraction tool. It allows users to create highly accurate extraction rules (wrappers), which describe what pieces of data to scrape from target web pages. DEiXTo extraction rules are based on the W3C DOM (Document Object Model) representation that reflects a page's HTML tag hierarchy. Version: 1.4.0 Release Date: 26/01/2014 Licence: GNU General Public License (GPL) Website: http://deixto.com Contact: kntonas@gmail.com Copyright: 2007-2014, Kostas Ntonas IMPORTANT: Prior to using DEiXTo for your next extraction task, please make sure that you don't violate any access and/or copyright restrictions set by the target site. You should check their copyright statement and the robots.txt file in their root folder. ================ # Installation # ---------------- No installation is needed. The user can just run it from a DOS command line (typically called a Command Prompt window, found in the Accessories program menu). It has been tested successfully on Windows 2000, XP, Vista and 7. ========= # Usage # --------- deixto_executor.exe PROJECT [-out path] [-in path] [-target url(s)] [-append 0|1|2] [-format txt|csv|xls|ods|xml|html|rss|dbi] [-delay time] [-max #hits] [-nice 0|1] [-mimic] [-search keyword] [-proxy http://your.proxy.server:port] [-image url] [-credentials "username password"] [-style xsl] [-depth \#pages] [-charset encoding] [-dbconfig path] [-postprocess path] [-native_url 0|1] [-timestamp] [-pagenc encoding] [-print] [-help] [-usage] [-version] Note that command line parameters override those defined in a wpf. Options: -help|? Displays this message. -usage Usage summary. -version Version number. -charset encoding Operating system's character set -out filepath Output filename. -in filepath Input filename with target URLs. -target url(s) Override target URL(s). -append 0|1|2 Output mode (Overwrite or Append or Prepend). -native_url 0|1 Extract record's native URL. -format filetype Output file format. -max hits Max number of hits. -search keyword Keyword for auto_fill_and_submit_form. -depth pages Max crawling depth when following 'next' links. -delay time Sleep time between agent's http requests. -print Print results to screen. -nice 0|1 Respect robots.txt file on target website's root folder. -mimic Mimim Mozilla browser. -proxy Proxy server to be used. -style xsl XSL Stylesheet to be used (for XML output). -credentials Credentials for HTTP Basic authentication. -image url Image URL used in RSS output files. -dbconfig filepath Dbconfig file. -postprocess filepath Enable the postprocessing mechanism via a config file. -timestamp Add a timestamp to the records found. -pagenc encoding Force use of a certain encoding instead of that specified in the target page source. Sample execution: deixto_executor.exe project.wpf -format xls -out data.xls The command above executes the "project.wpf" wrapper project file and generates an Excel output file (data.xls) containing the extracted data. NOTES: - By default, the executor complies with the robots.txt file of the target website. However, you can override that by setting the '-nice' command line option to 0. Note, though, that you should respect the website's owner's demands and keep away from pages that have access restrictions. - By default, the executor identifies itself as DEiXToBot/1.x.x. However, it can masquerade as a Mozilla browser by entering the -mimic command line parameter. This is not recommended though due to ethical reasons. - Your system's character set is assumed to be iso-8859-1 but this can be easily changed just by using the -charset option. For instance, you can use cp737 (iso-8859-7) if your operating system is Greek Windows. To find out in detail which encodings are supported and their code names, see Encode::Supported (http://search.cpan.org/~jhi/perl-5.8.1/ext/Encode/lib/Encode/Supported.pod). - If you want to prevent a wrapper (with "MultiplePage" mode enabled) following 'Next' links, you can set "depth" equal to 0 from the command line. If you don't want any limit upon the crawling depth, you can just give it a -1 value. - In order to produce RSS output, you should give appropriate labels to the nodes that extract data. Valid RSS labels for the wpf parser are: RssTitle, RssLink, RssDescription, RssAuthor and RssPubDate. - In order to export extracted data to HTML, an HTML template is used (template.tmpl). The encoding of this file should be UTF-8. Moreover, you should place the labels of the extracted fields of your wrapper to the tags of the template. - To insert the extracted data to a database, you will have to provide the database user credentials, the driver (e.g. mysql), the database name, the server (e.g. localhost), the mode (insert or update) and a mapping between the extracted fields of the wrapper and the columns of the corresponding tables. Moreover, you will have to set a field as the global primary key by entering a line "key 1" in the section of the key field at the dbconfig file. In your database schema, you should set the key field as a foreign key in the rest tables and use ON DELETE CASCADE and ON UPDATE CASCADE. Thus, you should provide just one single key for all tables at the dbconfig file. - Your database should have utf8_general_ci collation. So, add the following when creating your schema: "DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci". - To use the postprocesing mechanism you will have to edit a configuartion file like the sample 'config' file contained in the zip file. Operations supported are merges and splits. Moreover, it allows on-the-fly Perl subroutine evaluation on the value of a certain field, thus it lets you perform various transformations to the data extracted, e.g. make a value lower case or substitute a substring. KNOWN ISSUES: - The agent cannot follow 'Next' links or submit forms that use JavaScript. This is due to the fact that WWW::Mechanize lacks support for JavaScript. - Sometimes, a page's DOM tree built by XML::LibXML is not exactly the same with that built by Internet Explorer. Consequently, the DEiXTo generated pattern may not work exactly as expected when executed by the command line executor. Thankfully, most times this is not the case.. ACKNOWLEDGEMENTS: Many thanks to Enrique Brito (http://enrique.brito.es/blog/) for his help, time and support. He has contributed significantly to the executor's testing and proposed very good ideas on its features and functionality. ============================================================================= This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. Please do not hesitate to contact me for any questions, comments, suggestions or bugs. Thank you very much for downloading and using the DEiXTo Perl executor. ;-)