Tutorial
========

Here we explain in greater details how to use sh.js, and to a lesser extent, how it works.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Before you start
----------------

The following tutorial aims people who are at least novice at shell scripting and know their way in node.js. In particular, you should be familiar with:

-   standard streams: standard input, output and error
-   pipes: `ls / | grep etc`
-   redirections: `find . -name "*.js" > results`
-   exit statuses and operators such as `&&` or `||`
-   JavaScript
-   `require`
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

### `require`

Depending on your [installation](./install.html), there are two ways you can `require` sh.js.

#### 1. On npm installs

If you installed with npm, simply require `sh`:

    ///pygments:javascript
    var sh = require('sh');
    sh('echo hello');

#### 2. On git installs

If you installed with git, you need to require a path:

    ///pygments:javascript
    var sh = require('/path/to/shjs/sh');
    sh('echo hello');

### Introduction

Examples in this tutorial may have a file name indicating you can run it in the `examples` directory.

<b>In a terminal, head to the `examples` directory, and run `./00-hello.example.js`</b>

At its core, sh.js launches programs:
    
    ///pygments:javascript
    ///file:examples/01-ls.example.js:5:link

This spawns a new process that executes the `ls` program.

That returns a function that you may call:
    
    ///pygments:javascript
    ///file:examples/02-ls_grep.example.js:5:link

By calling it, you tell sh.js that you want `ls`' output to be used as `grep`'s input. You can pipe like that as many times as you want:

    ///pygments:javascript
    ///file:examples/03-multi_pipe.example.js:5:link

That is the equivalent to the following Bash command:

    ///pygments:bash
    echo hello | cat | cat | cat | cat

Now here's how to redirect standard output to a file:

    ///pygments:javascript
    ///file:examples/04-find_files.example.js:5:link

Bash equivalent:
    
    ///pygments:bash
    find . -size -100c > 100c_files

Alternatively, there is an `.append()` method that appends to a file instead of (over)writting it (like `>>`).

You can also cache the output of a command, and receive it in an callback:

    ///pygments:javascript
    ///file:examples/05-count_find_results.example.js:5:link

Bash equivalent:
    
    ///pygments:bash
    echo found `cat 100c_files | wc -l` file\(s\) smaller than 100 bytes

I hope you can see the pattern here: anywhere piping is syntactically correct, `.file()`, `.append()` and `.result()` are correct too:

    ///pygments:javascript
    sh('echo hello')('cat');
    //              ^^^^^^^ piping is correct
    
    sh('echo hello').file('hello.dump');
    //              ^^^^^^ therefore .file() is correct too
    
    sh('echo hello').append('hello.dump');
    //              ^^^^^^^^ and so is .append()
    
    sh('echo hello').result(function(output) { console.log(output); });
    //              ^^^^^^^^ as well as .result()

Piping stderr is quite similar to stdout, you need to interleave `.err`:

    ///pygments:javascript
    ///file:examples/06-pipe_err_to_null.example.js:5:link

Bash equivalent:
    
    ///pygments:bash
    find /var -type s 2> /dev/null

That will find socket files and discard errors messages.

<b>Note</b>: see the section on advanced piping to redirect both stdout and stderr.

### Quoting and escaping arguments

Typing commands with sh.js is meant to feel *almost* like a standard shell.

You can quote arguments:

    ///pygments:javascript
    ///file:examples/07-quoting.example.js:5:link
    
    // Output:
    // hello        world

You can escape characters:

    ///pygments:javascript
    ///file:examples/08-escaping.example.js:5:link
    
    // Output:
    // hello  world " ' \

In case quoting or escaping are not satisfying, you may pass an array of arguments instead:

    ///pygments:javascript
    sh(['echo', 'hello  world " \' \\']);

Finally, to avoid getting stuck because of a bug, you can access the parser like so:

    ///pygments:javascript
    ///file:examples/09-parser.example.js:3:link
    
    // Output:
    // [ 'echo', 'hello  world', '"', '\'', '\' ]

The `.parse()` method returns an array with the arguments as it interprets them. If you spot a bug, please report it.

### Concurrent vs. sequential commands

*sh.js is non-blocking*. Every call returns immediately after some minor computations. For instance, consider:

    ///pygments:javascript
    ///file:examples/10-concurrent.example.js:5:link
    
    // Output:
    // jeudi 18 novembre 2010, 20:30:54 (UTC+0100)
    // jeudi 18 novembre 2010, 20:30:54 (UTC+0100)

Unlike in Bash, both dates are the same because calls to sh.js do not block. Both `date` commands are run <b>concurrently</b>. This is useful to run tasks in parallel.

Bash equivalent:

    ///pygments:bash
    date &
    sleep 2 &
    date &

If you want to run commands <b>sequentially</b>, use `.then()`:

    ///pygments:javascript
    ///file:examples/11-sequential.example.js:5:link
    
    // Output:
    // jeudi 18 novembre 2010, 20:33:07 (UTC+0100)
    // jeudi 18 novembre 2010, 20:33:09 (UTC+0100)

This does not block the execution of your program, but it <b>schedules</b> the next command only after the previous one exits. Notice dates are not the same, unlike before.

Bash equivalent:

    ///pygments:bash
    date
    sleep 2
    date

Concurrent and sequential commands are not mutually exclusive, you can use both:

    ///pygments:javascript
    ///file:examples/12-concurrent_sequential.example.js:5:link
    
    // Output:
    // jeudi 18 novembre 2010, 20:35:46 (UTC+0100)
    // jeudi 18 novembre 2010, 20:35:46 (UTC+0100)

`sleep 2` commands are run after the `sleep 1` command, so that's sequential. But notice the dates are the same because `sleep 2` commands are run concurrently.

Bash equivalent:

    ///pygments:bash
    sleep 1 && ( ( sleep 2 && date ) & ( sleep 2 && date ) & )

### Reacting to an exit status with `.and()` and `.or()`

Using `.and()` and `.or()` methods, you can handle the exit of the previous process depending on its exit status. Take the following example:

    ///pygments:javascript
    ///file:examples/13-gzip_zeros.example.js:5:link
    
    // Output:
    // 10000+0 enregistrements lus
    // 10000+0 enregistrements écrits
    // 512000000 octets (512 MB) copiés, 4,00177 s, 128 MB/s
    // compression stopped
    // compression succeeded

That copies 500 million null bytes from `/dev/zero` and `gzip`s them to `zeros.gz`.

- Line (5): the `.then()` method prints "compression stopped" as soon as gzip exits.
- Line (6): the `.or()` method prints "compression failed" as soon as gzip exits *and* if gzip returns a status not equal to zero (not the case here).
- Line (7): the `.and()` method prints "compression succeeded" as soon as gzip exits *and* if gzip returns a status of `0`.

Bash equivalent:

    ///pygments:bash
    dd if=/dev/zero count=10000 bs=50K | gzip > zeros.gz
    
    status=$?
    
    echo compression stopped
    test $status || echo compression failed
    test $status && echo compression succeeded

`.and()`, `.or()` and `.then()` all accept functions as arguments, so you can run JavaScript instead of a program:

    ///pygments:javascript
    ///file:examples/14-callback.example.js:5:link

### Setting the environment: `.cd()` and `.define()`

If you want to run commands in a particular directory, call `.cd()` before them in a sequence:

    ///pygments:javascript
    ///file:examples/15-ls_cd_ls.example.js:5:link

Pay attention on how to call `.cd()`. Here we call it as a method of `.and`.

You may also use it as a method of `.or`, `.then` or `sh` like below:

    ///pygments:javascript
    sh('ls -l').or.cd('..').and('ls -l');
    sh('ls -l').then.cd('..').and('ls -l');
    sh.cd('..').and('ls -l');

Now if you want to set an environment variable, here's how:

    ///pygments:javascript
    ///file:examples/16-grep_my_var.example.js:5:link

You may also set several variables in one call:

    ///pygments:javascript
    ///file:examples/17-grep_my_vars.example.js:5:link

Finally, to unset a variable, just set it to `sh.UNSET`:

    ///pygments:javascript
    ///file:examples/18-unset_my_var.example.js:5:link

By the way, once you `cd` to a directory or set environment variable, you can store the shell in a JavaScript variable, and reuse it later:

    ///pygments:javascript
    ///file:examples/19-store_shell.example.js:5:link

Don't forget the `.and` however.

There is a restriction in that `.cd()` or `.define()` cannot be in a pipeline like so:

    ///pygments:javascript
    // this doesn't work for now
    sh.cd('abcd').file('/dev/null').and('ls -l');

I hope that will work soon.

### Advanced piping and redirecting

<b>Note</b>: the main challenge of making sh.js was to find the right syntax. Suggestions are welcome.

On the one hand, a process has three standard streams and you may want to start several processes upon its exit. On the other hand, JavaScript doesn't have lots of idioms to embed such syntax.

We're trying to reproduce the following Bash command (this uses process substitution which is not part of the standard):

    ///pygments:bash
    ls / non_existent_file 
      2> >( sed s/non_existent/NON_EXISTENT/ ) \
      > >( grep etc )

    # Output
    # etc
    # ls: ne peut accéder NON_EXISTENT_file: Aucun fichier ou dossier de ce type

    # Note: everyone is more familiar with the following, but for the purpose of
    # writing shell scripts in JavaScript, the syntax poses the same problems
    # as above:
    
    ls / non_existent_file > out_stream 2> err_stream
    

So I've found four ways that I believe do make sense, three of which are implemented, the other one being removed from the code:

-   chaining
-   variables
-   closures
-   arguments

#### 1. Chaining

This syntax was removed. It looked like:

    ///pygments:javascript
    // This was removed, it won't work
    sh('ls / non_existent_file')
      .pipe('grep etc')
      .err('sed s/non_existent/NON_EXISTENT/');

The reason this was removed is I found it too complicated when there is a lot of recursion. It felt awkward because `.pipe()` and `.err()` break easily if they are not properly ordered.

#### 2. Variables

By declaring a variable, you can make several method calls on the first command:

    ///pygments:javascript
    ///file:examples/20-pipes_with_variables.example.js:5:link

Pros:

-   variables are native to JavaScript

Cons:

-   it quickly gets messy

#### 3. Closures

Passing a closure after the command string will run it and identify the piping:

    ///pygments:javascript
    ///file:examples/21-pipes_in_closures.example.js:5:link
    
Pros:

-   it doesn't clobber the variable namespace like above
-   very nice in [CoffeeScript](http://jashkenas.github.com/coffee-script/):

        ///pygments:coffeescript
        sh 'ls / non_existent_file', ->
          @out 'grep etc'
          @err 'sed s/non_existent/NON_EXISTENT/'

Cons:

-   more syntactic noise
-   it still feels messy

#### 4. Arguments

In this syntax, you use one argument for each stream to tell the command how you want to pipe it

    ///pygments:javascript
    ///file:examples/22-branch_arguments.example.js:5:link

Pros:

-   rather clean syntax
-   CoffeeScript-friendly:

        ///pygments:coffeescript
        sh 'ls / non_existent_file',
          sh.out 'grep etc'
          sh.err 'sed s/non_existent/NON_EXISTENT/'

Cons:

-   compared to closures, you can't run code

### Putting everything together

Of course, the idioms above are not mutally exclusive.

    ///pygments:javascript
    ///file:examples/23-advanced.example.js:5:link

Here we combine:

-   piping standard output to a process line (4)
-   redirecting standard error to `/dev/null` line (7)
-   reacting to the exit status lines (5), (8), (9) and (13)
-   arguments lines (4) and (5)
-   a closure lines (6) througth (10)
-   a variable line (13)

### Random tricks

#### Output Only

If you want to redirect stderr to stdout, like with `&>` or `|&` in traditional shells, pass `sh.OO` after the command:

    ///pygments:javascript
    sh('./configure', sh.OO).file('output+errors');

That's double-capital-O, as in "Output Only".

`sh.OO` must come before a closure if you use one:

    ///pygments:javascript
    sh('./configure', sh.OO, function(c) {
      c.and('echo configure succeeded');
      c.or('echo configure failed');
    }).file('output+errors');

#### Define environment

If you want a clean environment, use `sh.ENV`:

    ///pygments:javascript
    ///file:examples/24-clean-env.example.js:5:link
    
    // Output: the environment is empty except for the variable we just defined
    // MY_VAR=123

### Conclusion

That tutorial covered mostly everything. This is a work in progress. Every piece of feedback will be greatly appreciated (use my email in `git log`).