# @codincod/codemirror-lang-shell [![NPM version](https://img.shields.io/npm/v/@codincod/codemirror-lang-shell.svg)](https://www.npmjs.org/package/@codincod/codemirror-lang-shell)

[ [**CHANGELOG**](https://codeberg.org/reeven/codemirror-lang-shell/src/branch/master/CHANGELOG.md) ]

This package implements shell language support for the
[CodeMirror](https://codemirror.net/) code editor, using a
[Lezer](https://lezer.codemirror.net/) grammar written for this package.

CodeMirror ships a shell mode in `@codemirror/legacy-modes`. It colours
keywords, strings and comments from a word list and a small state machine, and
it produces no tree, so nothing above the token level is available: no folding,
no indentation, no structural selection. It also has to guess, and guesses
wrong in both directions. `echo a=b` comes out as an assignment. The `done` in
`while true; do echo done; done` comes out as a keyword, when it is the word
the loop prints.

Written in part for [CodinCod](https://codincod.com/), a competitive coding
platform, where it colours the editor people solve puzzles in.

This code is released under an
[MIT license](https://codeberg.org/reeven/codemirror-lang-shell/src/branch/master/LICENSE).

## Usage

```javascript
import {EditorView, basicSetup} from "codemirror"
import {shell} from "@codincod/codemirror-lang-shell"

const view = new EditorView({
  parent: document.body,
  doc: `#!/bin/sh
for f in *.txt; do
  echo "$(wc -l < "$f") $f"
done
`,
  extensions: [basicSetup, shell()]
})
```

`shell()` on its own reads the shell command language, which is what `sh`,
`bash`, `zsh` and `ksh` all are. The two shells that are not take a name:

```javascript
shell({shell: "fish"})   // or "csh", or "tcsh"
```

`csh()` and `fish()` are the same thing said shorter, and `shellLanguageFor`
answers with the `LRLanguage` for a name if you are assembling the extensions
yourself.

## Coverage

The shell command language of IEEE 1003.1, with the bash extensions that are
everywhere in practice: `[[ ]]`, `(( ))`, `$(( ))`, arrays and associative
arrays, process substitution, here-strings, `&>`, `+=`, the `${x/a/b}` family
of expansions, and the extended patterns `@(a|b)` that ksh has always had and
bash has under `shopt -s extglob`.

zsh is read by the same grammar rather than by a switch, the way this package's
AWK grammar reads gawk: the expansion flags in `${(kv@)h}`, an expansion whose
name is another expansion, `$#array`, several names in one `for`, a list in
parentheses, the short forms that drop the `do` and the `then`, and the
qualifier in `*.txt(D)`. Somebody reading a shell script does not always know
which shell wrote it, and reading more than one shell writes costs an editor
nothing.

ksh93's compound types are read too. `typeset -T Point_t=( ... )` puts commands
where an array puts words, and which of the two a `=(` opens is decided by
looking for a `;` inside it, since that is a syntax error in an array and the
punctuation between two commands everywhere else. A field reached through a
subscript, `${people[i].name}`, comes with them.

Measured against 19600 scripts published on GitHub, one per repository and
sifted for the Python, Perl and Ruby that a `.sh` extension also names, 95.35%
parse with no error node.

Against the 754 scripts on Rosetta Code the figure is 93.50%, and every
zsh-labelled file among them is clean. Of the 49 that are not, most are not
this shell: 13 are csh and 5 are `rc`. The rest are terminal sessions with the
prompt still in them, program output saved as source, and files the wiki
mangled, where a tab under a `<<-` became spaces or a quote became a
typographic one.

### The C shell

csh is not a dialect of the shell command language and does not share a grammar
with it here. It shares the word layer, the quoting, the pipelines and the
redirections, and replaces everything above them: `if` takes an expression
rather than a list of commands, arithmetic is a command of its own spelled `@`,
a loop ends with `end` rather than `done`, and there are no functions at all.
So it has a grammar of its own in this package, `csh.grammar`, built from the
same three questions and answering to a name of its own.

What it adds is an expression language, and it is the C operator table rather
than anything the shell has: `==` and `!=` compare, `=~` matches a pattern,
`-e` and its relatives ask about a file, and `{ cmd }` stands for the status a
command exits with. The precedence is C's, so `$n % 15 == 0` is the remainder
compared against nothing else.

Two of its questions have no counterpart on the sh side. A number is a number
only inside an expression, because `echo 1` passes a word and csh has no
numbers anywhere else. And `@ n++` ends the name before the `++` where `echo
a++` does not, which cannot be asked of the parser, since the word the operator
would follow has not been reduced yet; it is asked of the text instead, which
says the same thing.

Of the 13 csh files in the corpus, 12 parse with no error node. The one that
does not is an `alias` whose body is a whole csh program inside a raw string,
escaped a character at a time.

### fish

fish gets a grammar of its own as well, and it is the shortest of the three,
because fish threw out most of what makes a shell hard to parse. There is no
word splitting. There are no heredocs. There is no `${}`, no `$(())`, no
backticks and no subshell: a parenthesis is a command whose output stands in
for it, everywhere, and that is the only thing it is. Every block ends with
`end`.

It has one question of its own. A `[` opens a subscript where it touches the
word in front of it and is the name of the test command where it does not:
`$argv[1]` against `[ -f $file ]`. Both readings are open at once, so the
tokenizer settles it and hands the parser one or the other, never both. A
closing parenthesis is the exception it makes, since `(seq 10)[-1]` indexes
what the command printed.

`else if` is one line and an `else` on its own is another, so the line ending
tells a continued conditional from a nested one. Neither of the other two
grammars here can be read that way, and fish can because it has no `then`.

Every fish file in the corpus parses with no error node, though there are only
21 of them; Rosetta Code files fish under two headings and one of them is a
different language with the same name.

## What a tokenizer has to settle

Shell is not a language of expressions with a few strings in it. It is a
language of words, and nearly every hard question here is about where a word
begins and ends. Seven of them are settled by asking the parser what it has
room for.

**A word is a run of pieces that touch.** `foo"bar"$baz` is one argument and
`foo "bar" $baz` is three, so the pieces of a word are parsed with nothing
skipped between them. What closes a word is a token of no width, produced where
the shell would end one: at whitespace, or at one of `; & | ( ) < >`. The
comment rule falls out of that and needs no code of its own. Nothing is skipped
inside a word, so `a#b` is one word and `a #b` is a word and a comment.

**A reserved word is only reserved where a command may start.** `if` opens a
conditional and `echo if` prints two letters, and the difference is nothing but
position. A command cannot start until the one before it has been terminated,
so the parser has room for `done` after a `;` and not after `echo`, which is
the whole rule. The same question settles `{`, `}`, `[[`, `]]` and `!`, each of
which a word would otherwise swallow.

**An assignment is a word in the right place.** `x=1` sets a variable and
`echo x=1` prints one, and again only position tells them apart. The value has
to touch the `=`, because `x= cmd` runs `cmd` with `x` set to nothing.

**A newline ends a command where a command can end,** and is skipped everywhere
else. That one rule covers every place the shell allows a line to be broken
without a marker: after `|`, `&&`, `do`, `then`, `else`, `{`, `(` or a comma in
an array.

**A heredoc's body is nowhere near the `<<` that promised it.** `<<EOF` says the
body starts on the next line, and one line may promise several. Rather than
carry the promise from token to token, the body is read at the newline by
looking back over the line that just ended, and is then skipped like
whitespace.

**A backslash before a newline is space between words and a letter inside one.**
`grep foo \` continued on the next line passes two arguments, and `echo ab\`
continued on the next passes one, and the only difference is the space before
the backslash. A skipped token cannot also be shifted, so the second case is
produced where a word is already under way.

**A parenthesis after a glob narrows it, in zsh.** `*.txt(D)` matches the hidden
files as well. A parenthesis ends a word everywhere else, so this is asked
first, and only where a word has already begun; `(cd /tmp)` at the start of one
is still a subshell.

Two more are settled by counting rather than by asking.

**Three parentheses in a row are ambiguous,** and how they close says which was
meant. `((( n % 2 == 0 )) && echo even)` is a subshell holding an arithmetic
command; `(((x += 2) <= 8))` is arithmetic holding a parenthesised sum. The
shell decides this by parsing the arithmetic and taking the other reading when
that fails, which a parser with no backtracking cannot do. Counting the parens
gets the same answer.

**A closing brace needs no semicolon after a compound command.** `f() { if x;
then y; fi }` is accepted by bash and `f() { echo hi }` is not, because `}` is a
reserved word where a command may start and a plain word where an argument may.
What the line ended with decides it, which is the question bash asks of the
token it read last.

One is settled by parsing the same text twice.

**A backtick substitution is closed by the character that opens it,** so a
grammar reading left to right cannot tell which of the two it has. It is one
token, and the commands inside it are parsed afterwards, by handing the range
between the backticks back to the same parser. Across the corpus that is 197
backticks and 2 that hold something this grammar cannot read, both of them Tcl
filed under the wrong heading.

## What the tree does not say

Three things are deliberately shallower than the language reference.

**A bracket expression is text.** `*` and `?` are marked as the wildcards they
are, and `[a-z]` is left alone, because `[` is also the name of a command and
`[ -f x ]` is a great deal more common than a glob.

**Arithmetic and words are separate languages.** Inside `$(( ))` a bare name is
a variable and `2` is a number. Outside it, `echo 42` passes a word, because the
shell has no numbers anywhere else and colouring one there would be a lie.

**A `${...}` keeps its operator and loses the shape of what follows.**
`${x:-default}` is an expansion with an operator in it; the default itself is
read as the text and expansions it is made of, without being told apart from a
pattern or a replacement.

## What it gives an editor

Syntax highlighting, folding for groups, subshells, loop bodies, `case`
statements and command substitutions, indentation that knows `fi`, `done`,
`esac`, `}` and `;;` close what came before them, and completion for the
reserved words, the shell builtins and the variables the shell sets.

`shellLanguage` is exported for use with `LanguageSupport`. `parser` is the
grammar on its own, without the highlighting props and without the re-entry
into backticks; `shellLanguage.parser` is the one an editor runs.

`cshLanguage`, `cshCompletion` and `cshParser` are the same three things for
the C shell, and `fishLanguage`, `fishCompletion` and `fishParser` for fish.

## Testing it on your own code

```
npm run corpus -- path/to/your/scripts
npm run corpus:csh -- path/to/your/csh/scripts
npm run corpus:fish -- path/to/your/fish/scripts
```

It prints the files that failed, the first line of each that did, and the
proportion that came out clean. The corpus itself is not included; Rosetta Code
is under the GNU Free Documentation License 1.2 and cannot be redistributed
under this one.