# Ultimately Littlefork - [Requirements](#requirements) - [Simply Littlefork](#simply-littlefork) - [Intermezzo 1: Using files](#intermezzo-1-using-files) - [Plugins](#plugins) - [Intermezzo 2: Examples](#intermezzo-2-examples) - [Search for content from The Guardian](#search-for-content-from-the-guardian) - [Download videos from Youtube](#download-videos-from-youtube) - [Query Twitter and deal with media](#query-twitter-and-deal-with-media) - [Read tweets in Telegram](#read-tweets-in-telegram) - [Search Google for links and images](#search-google-for-links-and-images) - [Advancedly Littlefork](#advancedly-littlefork) - [Use Littlefork on the TOR network](#use-littlefork-on-the-tor-network) - [Best practices](#best-practices) - [Intermezzo 3: Data](#intermezzo-3-data) - [TODO: Persist data with MongoDB](#todo-persist-data-with-mongodb) - [TODO: Littlefork on UNIX](#todo-littlefork-on-unix) - [Look at this, look at that ...](#look-at-this-look-at-that-) ## Requirements To run `littlefork` you have to install [NodeJS](https://nodejs.org/en/). The easiest way to do this, is to use [nvm](https://github.com/creationix/nvm). It is recommended to use at least version 7.5.0. Follow the [installation instructions](https://github.com/creationix/nvm#install-script) and make sure that `node` as well as `npm` are installed: ``` node --version npm --version ``` ## Simply Littlefork The quickest way is to start with the example project. To start a new project from scratch, see the section on *Best Practices* below. ``` wget -qO- https://gitlab.com/littlefork/project-example/repository/archive.tar.gz | tar xzv cd project-example* npm install ``` This installs everything you need to run Littlefork. We will run Littlefork now. We want to make a search on DuckDuckGo for the term `Keith Johnstone` and print the results on the screen. ``` node_modules/.bin/littlefork -Q ddg_search:Keith\ Johnstone -p ddg_search,tap_printf ``` This prints up to 30 search results on the screen. Using the `-Q` option, we specified a query, and the `-p` defines which plugins should be run. The order of the plugins matter. You can't print results to the screen, if you haven't fetched yet any. We run two plugins, first the `ddg_search` plugin, which queries the search engine, and after that we run the `tap_printf` plugin, which prints the results, queries and configuration to the screen. There is not much more to Littlefork than that. Some plugins offer configuration options, that allow to control aspects of their behavior. If you look at the output of `node_modules/.bin/littlefork -h`, you will see the `--tap.limit` argument. We can use it to print less results on the screen. ``` node_modules/.bin/littlefork -Q ddg_search:Keith\ Johnstone \ -p ddg_search,tap_printf \ --tap.limit 1 ``` ## Intermezzo 1: Using files As you start to define more complicated data pipelines, the size of the command will grow. To make life easier, but also to make pipeline runs repeatable, is it possible, to store configuration in text files. The format of those files is JSON. The previous example would look like this: ``` { "plugins": "ddg_search,tap_printf", "tap": { "limit": 1 } } ``` The project example contains this config file in `configs/searches.json`. Use the `-c` argument to specify the file location: ``` node_modules/.bin/littlefork -Q ddg_search:Keith\ Johnstone \ -c configs/searches.json ``` Options provided by the command line will take precedence over the configuration file. This allows it to override options for a one off pipeline run. ``` node_modules/.bin/littlefork -Q ddg_search:Keith\ Johnstone \ -c configs/searches.json \ --tap.limit 10 ``` It is also possible to store queries in files, similar to configuration options. ``` [{ "type": "ddg_search", "term": "Keith Johnstone" }] ``` The project example contains this query file in `queries/ddg.json`. We can use the `-q` argument to tell Littlefork to look for queries in a file. ``` node_modules/.bin/littlefork -q queries/ddg.json -c configs/searches.json ``` It is possible to maintain multiple queries and of different type in one file. Query files like the following are not uncommon: ``` [{ "type": "ddg_search", "term": "Keith Johnstone" }, { "type": "ddg_search", "term": "Machinocene" }, { "type": "guardian_search", "term": "Aleppo" }] ``` ## Plugins The list of plugins form a sort of pipeline, which takes data in at the beginning and returns data at the end of it again. In this pipeline a plugin always receives the output of the previous plugin. We can form chains of transformation like this. Each plugin receives the current version of the data, can add to it or do something with the existing data, and returns data again. Using the CSV plugin, we can convert our data to CSV and write it to a file. Edit your `searches.json` and add another plugin. The file should look like this: ``` { "plugins": "ddg_search,tap_printf,csv_export", "tap": { "limit": 1 } } ``` ``` node_modules/.bin/littlefork -q queries/ddg.json -c configs/searches.json ``` Our pipeline consists now of three plugins. The `csv_export` plugin transforms the search results into CSV and writes it to the `out.csv` file in the same directory, but you can change the file name with the `--csv.filename` config option. ``` node_modules/.bin/littlefork -q queries/ddg.json \ -c configs/searches.json \ --csv.filename data.csv ``` To see other configuration options, run `node_modules/.bin/littlefork -h`. One last example. This pipeline does the following: - Make a reverse image search on Google for an image found in `data`. - Make a complete copy of the websites found in the search using the `wget` tool. The websites are stored in `downloads/url`. - Print the results to the screen. ``` [{ "type": "ddg_search", "term": "Keith Jonstone" }, { "type": "google_search", "term": "Anna Lowenhaupt Tsing" }, { "type": "google_images", "term": "Anna Lowenhaupt Tsing" }, { "type": "glob_pattern", "term": "data/**/*.jpg" }] ``` ``` node_modules/.bin/littlefork -q queries/search_terms.json \ -c configs/reverse_search.json ``` ## Intermezzo 2: Examples ### Search for content from The Guardian The `guardian_content` plugin allows to search the archive of the Guardian. In order to use this plugin, you have to obtain an API key. Find instructions [here](https://gitlab.com/littlefork/littlefork-plugin-guardian). Lets do a quick search: ``` node_modules/.bin/littlefork -Q guardian_search:Aleppo \ -p guardian_content,tap_printf \ --guardian.key ``` ### Download videos from Youtube This plugin requires an API key. TODO: How to get this key? The `youtube_channel` plugin lists the contents of a channel, a list of videos. The `youtube_download` plugin uses [`youtube-dl`](https://rg3.github.io/youtube-dl/) to fetch the videos. Set a custom download target with the `--youtube.download_dir` option. The default download target is in `downloads` of the project directory. ``` node_modules/.bin/littlefork -Q youtube_channel:UC1NpRGow8m-yrWo0Mqp6DOg \ -p youtube_channel,youtube_download \ --youtube.api_key ``` ### Query Twitter and deal with media In order to use the Twitter plugin, you need four different API keys. - A consumer secret - A consumer key - An access token key - An access token secret First create a new Twitter account. Then go to [apps.twitter.com](https://apps.twitter.com) and log in. Create a new app. This pipeline consists of the following steps: - Search on twitter for the term `Jair Bolsonaro`. - Fetch the user timeline of `@jairbolsonaro`. - Download images that are found in any of the tweets fetched before. - Extract the EXIF data from every image found in any of the tweets fetched before. ``` node_modules/.bin/littlefork -Q twitter_user:@jairbolsonaro,twitter_query:Jair\ Bolsonaro \ -p twitter_search,twitter_feed,http_get,media_exif \ --twitter.consumer_key \ --twitter.consumer_secret \ --twitter.access_token_key \ --twitter.access_token_secret ``` Or use a config file: ``` { "plugins": "twitter_search,twitter_feed,http_get,media_exif", "twitter": { "consumer_key": "", "consumer_secret": "", "access_token_key": "", "access_token_secret": "" } } ``` ### Read tweets in Telegram We will need a bot key and channel ID to use the Telegram plugin. See [here](https://gitlab.com/littlefork/littlefork-plugin-telegram) for more information. The Telegram packages has one plugin, which can send units of data to a chat channel. It will construct a string out of the fields listed in `_lf_content_fields`. Using it is quite simple: ``` node_modules/.bin/littlefork -Q twitter_user:@jairbolsonaro \ -p twitter_feed,telegram_send_message \ --twitter.consumer_key \ --twitter.consumer_secret \ --twitter.access_token_key \ --twitter.access_token_secret \ --telegram.bot_key \ --telegram.chat_id ``` ### Search Google for links and images The Google module provides plugins to search on https://google.com, make an image search and reverse image search from a local image. The first two plugins are very straight forward. ``` node_modules/.bin/littlefork -Q google_search:Keith\ Johnstone -p google_search,tap_printf ``` The same goes for a Google image search. ``` node_modules/.bin/littlefork -Q google_search:Keith\ Johnstone -p google_images,tap_printf ``` There is also a plugin that can conduct a reverse google image. It uploads images from your local computer, and return the results by google. The example project contains some images to get started. ``` node_modules/.bin/littlefork -Q glob_pattern:data/**/*.jpg \ -p google_reverse_images_files,tap_printf ``` The `google_reverse_images_files` plugin uses a [glob pattern](https://github.com/isaacs/node-glob#glob-primer) as it's query input. It specifies the path to your files. ## Advancedly Littlefork ### Use Littlefork on the TOR network It is possible to anonymize littlefork using [torsock](https://github.com/dgoulet/torsocks). But since torsocks prevents requests to `localhost`, the MongoDB plugin is likely failing. This has to be worked. There is a `tor_check` plugin, that allows to test if requests are made over TOR. ``` node_modules/.bin/littlefork -p tor_check torsocks node_modules/.bin/littlefork -p tor_check torsocks node_modules/.bin/littlefork -Q guardian_search:Aleppo \ -p tor_check,guardian_content,tap_printf \ --guardian.key torsocks node_modules/.bin/littlefork -Q youtube_channel:UC1NpRGow8m-yrWo0Mqp6DOg \ -p tor_check,youtube_channel,youtube_download \ --youtube.api_key ``` ### Best practices #### Littlefork projects We start by creating a new Littlefork project. This is nothing more than a directory with some files. ``` mkdir exposingtheinvisible cd exposingtheinvisible npm init -y ``` You should end up with a single file in the `exposingtheinvisible` directory named `package.json`. This file is used by to manage versions and dependencies of Littlefork. Next on we install `littlefork` distribution. ``` npm install -S littlefork ``` If all went fine, you should have a folder named `node_modules`, which contains `littlefork` and all it's dependencies. Make sure we have the `littlefork` command as well: ``` node_modules/.bin/littlefork -h ``` #### TODO: Group queries and configs in files #### TODO: Make pipeline runs repeatable with config files ## Intermezzo 3: Data Littlefork organizes data as a set of units. A unit is an atomic piece of data, such as a single tweet or a single article from the guardian. Every plugin can decide what constitutes an unit. Every plugin receives the complete set of data at once. Littlefork cares about the form of data, not the content. This allows to treat Youtube videos and search engine results the same. Every unit of data has a set of keys and values. `tweet` and `tweet_time` would be such keys, and the values are the actual tweet, and the time when the tweet occurred. There is a series of extra fields that are added to every unit by Littlefork. Their keys usually start with `_lf`. These extra fields often convey an additional semantic meaning in a plugin independent manner. It allows plugins to operate on data in an independent manner, e.g. `_lf_links` holds all links, that a data unit might contain. Now any plugin that wants to operate on links (such as `http_get` or `http_wget`) don't need to know anything about the specific plugin that generate the data in the first place. This is an example, how such a unit of data looks like. ``` { "_lf_links" : [{ "type" : "url", "href" : "https://www.theguardian.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "_lf_id_hash" : "3e4ec721a206f46c1b1d30c958be6355e8f3665fe26ec51302dad565c5449096" }, { "type" : "self", "href" : "https://content.guardianapis.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "_lf_id_hash" : "5779a75ec7b430a5ebdf4402324067d1be03f25ade0506453e1082c335a1df01" }], "_lf_media" : [ ], "_lf_pubdates" : { "fetch" : "2017-03-21T23:22:35.426Z", "source" : "2016-08-14T12:00:35Z) }, "_lf_relations" : [{ "type" : "url", "term" : "https://www.theguardian.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "_lf_id_hash" : "ed7a6bc7c6694fa4879e4e1f956db03d61fb327ab1a70a9dbe1c5020f8291245" }, { "type" : "url", "term" : "https://content.guardianapis.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "_lf_id_hash" : "e3895a0e1ce5a3d44f23fc759d34c9da96e859604b30713817c40bb99314d6f1" }], "_lf_downloads" : [{ "type" : "url", "term" : "https://www.theguardian.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "_lf_id_hash" : "172f6d9f96f85f976267f4ceeebd48e9049ee1427ca4b1a7d80026b40ce04211" }, { "type" : "json", "term" : "https://content.guardianapis.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "_lf_id_hash" : "eda10207ffe39ce48c4883c2cd982c98275fafa6bddcdd2aa0fb4e9f37baa254" }], "id" : "us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "type" : "article", "sectionId" : "us-news", "sectionName" : "US news", "webPublicationDate" : "2016-08-14T12:00:35Z", "webTitle" : "Third-party presidential candidates fight for 15% in polls – and a spot in debates", "webUrl" : "https://www.theguardian.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "apiUrl" : "https://content.guardianapis.com/us-news/2016/aug/14/third-party-candidates-johnson-stein-mcmullin-debate-polling-percentage", "isHosted" : false, "_lf_id_fields" : ["id"], "_lf_content_fields" : ["webTitle"], "_lf_id_hash" : "7c4e094182ad5369dec90910463c587a50e2c5125b1685c784488c05b2a3f2c8", "_lf_content_hash" : "1306025b72901b844a134a9bd7eb9f76d4d4aa3909fddccbb7b131260da71367", "_lf_markers" : ["B1QarNJ2x", "S1l0B41nx"] } ``` The `_lf_id_hash` is the unique identifier of that unit of data. It is a SHA256 hash sum of the identifying fields of the unit. If Littlefork fetches a unit of data, it calculates the `_lf_id_hash` and can so determine if a unit is new or already known. Similar the `_lf_content_hash` is a hash sum of the content of a unit of data. This can be the actual tweet, or the contents of an article. If a unit is already know to Littlefork (same `_lf_id_hash`) but the content changed (different `_lf_content_hash`), then we have a revision of a unit of data. Units can form relations. If two separate articles link to the same image, then that image becomes a relation. If two tweets use the same hashtag, then that hashtag becomes a relation between those two tweets. ## TODO: Persist data with MongoDB - `mongodb_store` - `mongodb_fetch_*` - `mongodb_query` ## TODO: Littlefork on UNIX - Use pipes to connect Littlefork to other UNIX tools - Quiet mode to redirect output on stdout - pipe csv into awk - post json data to a web service - Run pipelines using GNU Parallel ## Look at this, look at that ... ``` node_modules/.bin/littlefork -q queries.json -p youtube_channel,twitter_search,twitter_feed,youtube_download,http_get,media_exif,mongodb_store,csv_export,tap_printf \ -c config.json ```