.TH CHOICESCRIPT-TESTING 1 "" "Testing ChoiceScript Programms"
.SH Testing ChoiceScript Games Automatically
.PP
Multiple-choice games written in ChoiceScript can be very hard to test,
because each \f[C]*choice\f[] point can create new variations of the
game that need testing.
When testing by hand, it can be hard to be certain whether some path of
choices can cause the game to crash, or whether some important part of
the story can't be reached at all.
.PP
To help with this, we've developed two tools that can make debugging
ChoiceScript games considerably easier.
.IP \[bu] 2
\f[B]Randomtest\f[]: The Randomtest tool automatically plays your game
over and over, making random decisions for every choice, and generates a
“hit count” report, which tells you how many times each line of code was
used in random play.
.IP \[bu] 2
\f[B]Quicktest\f[]: Instead of running the code at random, Quicktest
attempts to test every \f[C]#option\f[] in every \f[C]*choice\f[], and
both sides of every \f[C]*if\f[] statement.
In order to do this quickly, Quicktest “cheats” by skipping lines that
it has already tested.
.PP
Both types of tests are necessary.
Quicktest can find some bugs that Randomtest can't find, and vice versa.
Unfortunately, Quicktest can also find some “fake” bugs \[em] problems
with code style that \f[I]could\f[] cause a problem, but don't actually
do any harm for actual players.
.IP "1." 3
Running the Tests
.IP "2." 3
Interpreting Error Messages
.IP "3." 3
Detecting Dead Code
.IP "4." 3
Fake Bugs in Quicktest
.IP "5." 3
Randomtest Game Log
.IP "6." 3
Randomtest Hit Count
.SS Running the Tests
.PP
On OSX, you'll need to use the Terminal application.
\f[C]cd\f[] to your game directory and type
\f[C]choicescript\ quicktest\f[] or \f[C]choicescript\ randomtest\f[] to
run the tests.
You can also type \f[C]choicescript\ test\f[] to run both.
.PP
\f[B]Quicktest a Single File\f[]: By default, Quicktest attempts to test
every file listed in your \f[C]mygame.js\f[] list, but you can also use
it to test just one file at a time.
(Testing the whole game can be a waste of time when you're only working
on one file.)
 For example, to test a file called “spaceship.txt”, you can run
\f[C]choicescript\ quicktest\ spaceship\f[].
.PP
\f[B]Randomtest Iterations\f[]: By default, Randomtest runs your game
10,000 times.
To run Randomtest only 500 times, type
\f[C]choicescript\ randomtest\ 500\f[].
.PP
\f[I]Note: Randomtest is not truly random.\f[] In fact, Randomtest's
output is completely deterministic; if you run it twice in a row, you'll
get exactly the same results both times.
This catches a lot of people by surprise.
(Specifically, Randomtest is using a pseudo-random number
generator (http://en.wikipedia.org/wiki/Pseudorandom_number_generator)
with a hard-coded “seed” value.)
.PP
Doing it this way has one big advantage: if Randomtest fails with an
error and you run it again, it will fail again with the same error.
If you fix a bug and Randomtest passes, you can be sure that you've
fixed that problem.
.PP
As a result, if you want to run 30,000 random playthroughs with
Randomtest, it won't work to just run Randomtest three times; you'll
just play the same 10,000 variations over again.
Instead, use the command above to specify 30,000 iterations instead of
10,000.
.SS Interpreting Error Messages
.PP
If Quicktest passes, it will say “QUICKTEST PASSED” on the console.
If not, it will print out a confusing error message! (Our apologies; the
error message could really be much better.)
.PP
``
.IP
.nf
\f[C]
js:\ "web/scene.js",\ line\ 285:\ exception\ from\ uncaught\ JavaScript\ throw:\ Error:\ line\ 16:\ Non-existent\ command\ \[aq]oops\[aq]
\ \ \ \ at\ web/scene.js:285\ (runCommand)
\ \ \ \ at\ editor/embeddable-autotester.js:141\ (test_runCommand)
\ \ \ \ at\ web/scene.js:87\ (printLoop)
\ \ \ \ at\ web/scene.js:250\ (execute)
\ \ \ \ at\ editor/embeddable-autotester.js:252\ (autotester)
\ \ \ \ at\ autotest.js:88
\f[]
.fi
.PP
The reason this error message is so bad is because most of it is useless
to ChoiceScript authors but very useful to Choice of Games, the
developers of ChoiceScript.
.PP
The worst thing about this error message is that it reports a problem on
“line 285,” which is somewhat true, but not really very true at all,
because that's line 285 of ChoiceScript itself, the program we provide
to you! In this case, the real problem appears later in the error
message, “line 16.”
.PP
In this example, I added a broken line \f[C]*oops\f[] on line 16 of a
ChoiceScript file, and this is what it printed out.
.PP
If Quicktest (or ChoiceScript) gives you an error message that you can't
understand, feel free to ask us about it on our Google group.
Be sure to post not only the error message, but also the entire file of
code that causes the problem.
If you don't want to share that much code with the group, you can also
just send the code to us at support\@choiceofgames.com; we'll do our
best to help you out.
.SS Detecting Dead Code
.PP
But even if Quicktest passes, it may still report a number of lines
untested, e.g.
.PP
``
.IP
.nf
\f[C]
13\ UNTESTED\ startup
18\ UNTESTED\ startup
19\ UNTESTED\ startup

SOME\ LINES\ UNTESTED
QUICKTEST\ PASSED
\f[]
.fi
.PP
If Quicktest says “SOME LINES UNTESTED,” it means that Quicktest thinks
those lines are completely unreachable.
Either those lines are dead code, or there's a bug in Quicktest!
.PP
For example, here's a way to introduce dead code without noticing it.
We start with code like this:
.PP
``
.IP
.nf
\f[C]
*set\ leadership\ %+\ 20
*goto\ big_speech
\f[]
.fi
.PP
But then later we realize we also want to decrease \f[C]strength\f[].
Foolishly, we just add a line at the end, like this:
.PP
``
.IP
.nf
\f[C]
*set\ leadership\ %+\ 20
*goto\ big_speech
*set\ strength\ %-15
\f[]
.fi
.PP
That \f[C]*set\ strength\ %-15\f[] line is dead code; it can never be
reached, because it comes immediately after a \f[C]*goto\f[].
.PP
When Quicktest finds dead code, you should fix the problem, either by
deleting the code or by fixing the bug that kills the code.
In this example, we'd probably just move the \f[C]*goto\f[] line down to
the end.
.PP
Note that Quicktest isn't guaranteed to find all dead code\&... due to
the way Quicktest “cheats,” it can sometimes reach code that normal
human players can't reach.
Use Randomtest to find some of the other dead code in this system.
.SS Fake Bugs in Quicktest
.PP
\f[B]tl;dr: You may need to turn some of your \f[C]*if\f[] statements
into \f[C]*else\f[] statements to get Quicktest to pass.\f[]
.PP
Quicktest automatically plays through the code as a normal player would,
but when encountering a \f[C]*choice\f[] statement or an \f[C]*if\f[]
statement, Quicktest makes multiple copies of itself and attempts to run
them.
(The copies run one at a time; for example, we test the \f[C]true\f[]
lines of an \f[C]*if\f[] statement before we test the \f[C]*else\f[]
lines.)
.PP
To save time, Quicktest “cheats.” If one copy of Quicktest verifies a
line of code, and then a later copy of Quicktest reaches the same line
of code, the second copy of Quicktest will quit, assuming that the
earlier copy of Quicktest has already done its job.
.PP
But Quicktest “cheats” in another way, too, by testing both sides of
\f[C]*if\f[] statements even if the lines are not actually possible for
a player to reach.
This can cause Quicktest to identify “fake” bugs: bugs that can't
actually happen in real life.
.PP
For example, ChoiceScript attempts to guarantee correctness by requiring
every \f[C]#option\f[] in a \f[C]*choice\f[] statement to end with
\f[C]*goto\f[] or \f[C]*finish\f[].
Quicktest can help you catch bugs like this:
.PP
``
.IP
.nf
\f[C]
Example\ 1\ (Buggy)

*choice
\ \ \ \ #Be\ very\ naughty.
\ \ \ \ \ \ \ \ Santa\ refuses\ to\ give\ you\ a\ present.
\ \ \ \ #Be\ mostly\ nice.
\ \ \ \ \ \ \ \ Santa\ gives\ you\ a\ present\ reluctantly.
\ \ \ \ #Be\ as\ nice\ as\ can\ be.
\ \ \ \ \ \ \ \ Santa\ gives\ you\ a\ present\ enthusiastically.

Inside\ the\ gift\ box\ is\ a\ video\ game!
\f[]
.fi
.PP
If ChoiceScript allowed this code, it would create a hard-to-detect bug;
if you're very naughty, you still get a video game.
Instead, ChoiceScript crashes if you write code like this; Quicktest can
detect the crash automatically, allowing you to catch the bug easily.
.PP
You can fix the code like this:
.PP
``
.IP
.nf
\f[C]
Example\ 2\ (Fixed)

*choice
\ \ \ \ #Be\ very\ naughty.
\ \ \ \ \ \ \ \ Santa\ refuses\ to\ give\ you\ a\ present.
\ \ \ \ \ \ \ \ *finish
\ \ \ \ #Be\ mostly\ nice.
\ \ \ \ \ \ \ \ Santa\ gives\ you\ a\ present\ reluctantly.
\ \ \ \ \ \ \ \ *goto\ present
\ \ \ \ #Be\ as\ nice\ as\ can\ be.
\ \ \ \ \ \ \ \ Santa\ gives\ you\ a\ present\ enthusiastically.

*label\ present
Inside\ the\ gift\ box\ is\ a\ video\ game!
\f[]
.fi
.PP
But now suppose we included an \f[C]*if\f[] statement in the middle of
this \f[C]*choice\f[].
Suppose we have a \f[C]gender\f[] variable, which we set to either
“male” or “female”.
Then we might write code like this:
.PP
``
.IP
.nf
\f[C]
Example\ 3\ (Gender\ Bug)
*choice
\ \ \ \ #Be\ very\ naughty.
\ \ \ \ \ \ \ \ Santa\ refuses\ to\ give\ you\ a\ present.
\ \ \ \ \ \ \ \ *finish
\ \ \ \ #Be\ mostly\ nice.
\ \ \ \ \ \ \ \ *if\ gender\ =\ "male"
\ \ \ \ \ \ \ \ \ \ \ \ *goto\ boy_present
\ \ \ \ \ \ \ \ *if\ gender\ =\ "female"
\ \ \ \ \ \ \ \ \ \ \ \ *goto\ girl_present
\ \ \ \ #Be\ as\ nice\ as\ can\ be.
\ \ \ \ \ \ \ \ Santa\ gives\ you\ a\ video\ game.
\f[]
.fi
.PP
In fact, this code has a bug, but it might never happen in real life:
what if \f[C]gender\f[] is neither “male” nor “female?” For example, in
Choice of the Dragon, your dragon may be “neither” male nor female, or
your gender may be “unknown.”
.PP
If a genderless dragon were playing through Example 3, the game would
crash, with the same error as Example 1; Quicktest automatically detects
that.
.PP
Here's how: at the first \f[C]*if\f[] statement, Quicktest creates a
copy of itself: it starts with one copy where
\f[C]gender\ =\ "male"\f[], and then another copy where
\f[C]gender\ !=\ "male"\f[]; that non-male copy then makes a copy of
itself, one where \f[C]gender\ =\ "female"\f[] and another copy where
\f[C]gender\ !=\ "female"\f[].
.PP
In that final copy, Quicktest tests the case where \f[C]gender\f[] is
neither male nor female; since there is no \f[C]*goto\f[] or
\f[C]*finish\f[] in that case, Quicktest crashes with an error.
.PP
But, in your game, you may not actually have any genderless characters.
(It's understandable; not every game does!) But Quicktest can't know
that for sure, so Quicktest will say that Example 3 is buggy, even if
there's a 0% chance of the bug occurring in real life.
.PP
You can fix Example 3 with an \f[C]*else\f[] statement, which helps
Quicktest to understand that there's only two possibilities in this
case:
.PP
``
.IP
.nf
\f[C]
Example\ 4\ (Gender\ Fixed)
*choice
\ \ \ \ #Be\ very\ naughty.
\ \ \ \ \ \ \ \ Santa\ refuses\ to\ give\ you\ a\ present.
\ \ \ \ \ \ \ \ *finish
\ \ \ \ #Be\ mostly\ nice.
\ \ \ \ \ \ \ \ *if\ gender\ =\ "male"
\ \ \ \ \ \ \ \ \ \ \ \ *goto\ boy_present
\ \ \ \ \ \ \ \ *else
\ \ \ \ \ \ \ \ \ \ \ \ *goto\ girl_present
\ \ \ \ #Be\ as\ nice\ as\ can\ be.
\ \ \ \ \ \ \ \ Santa\ gives\ you\ a\ video\ game.
\f[]
.fi
.PP
Here, Quicktest only makes two copies: one where
\f[C]gender\ =\ "male"\f[] and another where
\f[C]gender\ !=\ "male"\f[].
The \f[C]*else\f[] guarantees no other possibilities.
.SS Randomtest Game Log
.PP
When Randomtest runs, it generates a huge file called
\f[C]randomtest-output.txt\f[].
The file is in two parts: the first part of the file is the “game log”
of all 10,000 random playthroughs.
The second part of the file is a “hit count” report, which tells you how
many times each line of code was used (“hit”) in the course of playing
the game.
.PP
Here's a sample of the first part of the \f[C]randomtest-output.txt\f[]
file, generated from our example game provided with ChoiceScript.
.PP
``
.IP
.nf
\f[C]
\ \ \ \ *****0
\ \ \ \ startup\ 42#1\ (43)
\ \ \ \ startup\ 46#2\ (51)
\ \ \ \ animal\ 19#3\ (26)
\ \ \ \ variables\ 24#1\ (25)
\ \ \ \ variables\ 33#2\ (40)
\ \ \ \ *****1
\ \ \ \ startup\ 42#2\ (59)
\ \ \ \ startup\ 62#2\ (67)
\ \ \ \ animal\ 19#1\ (20)
\ \ \ \ variables\ 24#1\ (25)
\ \ \ \ variables\ 33#1\ (34)
\ \ \ \ *****2
\ \ \ \ startup\ 42#1\ (43)
\ \ \ \ startup\ 46#3\ (55)
\ \ \ \ animal\ 19#2\ (22)
\ \ \ \ variables\ 24#1\ (25)
\ \ \ \ variables\ 33#2\ (40)
\ \ \ \ *****3
\f[]
.fi
.PP
The \f[C]*****\f[] lines tell you when Randomtest finished a playthrough
and started over again on its next playthrough.
The other lines specify which exact choices Randomtest made while
playing through the game.
.PP
For example, the line \f[C]startup\ 42#1\ (43)\f[] says that in the file
\f[C]startup.txt\f[] there was a \f[C]*choice\f[] on line 42.
The rest of the line indicates which option Randomtest chose, and the
line number of that option.
In this example, Randomtest chose option #1 on line 43.
In the second playthrough, we see \f[C]startup\ 42#2\ (59)\f[] which
says that Randomtest instead chose option #2 on line 59.
.PP
Ideally, the last line of randomtest-output.txt says “RANDOMTEST
PASSED.” If not, it contains an error message; see “Interpreting Error
Messages” above for additional details.
But note that you can use the Randomtest log to tell you exactly how to
reproduce the bug by hand: just make the exact same choices that
Randomtest did (“option #1, #2, #3, #1, #2 \&...”) before the error
occurred.
Reproducing Randomtest bugs by hand can make them much easier to
understand and fix.
.SS Randomtest Hit Count
.PP
The game log is usually the longest part of
\f[C]randomtest-output.txt\f[]; the hit count typically appears way down
at the end of the file.
The hit count is a report of how many times Randomtest used (or “hit”)
each line in the game.
.PP
For example, here's a sample from the hit count for the
\f[C]startup.txt\f[] example:
.PP
``
.IP
.nf
\f[C]
startup\ 10000:\ Your\ majesty,\ your\ people\ are\ starving\ in\ the\ streets,\ and\ threaten\ revolution.
startup\ 10000:\ Our\ enemies\ to\ the\ west\ are\ weak,\ but\ they\ threaten\ soon\ to\ invade.\ \ What\ will\ you\ do?
startup\ 10000:
startup\ 10000:\ *choice
startup\ 10000:\ \ \ #Make\ pre-emptive\ war\ on\ the\ western\ lands.
startup\ 3418:\ \ \ \ \ If\ you\ can\ seize\ their\ territory,\ your\ kingdom\ will\ flourish.\ \ But\ your\ army\[aq]s
startup\ 3418:\ \ \ \ \ morale\ is\ low\ and\ the\ kingdom\[aq]s\ armory\ is\ empty.\ \ How\ will\ you\ win\ the\ war?
startup\ 3418:\ \ \ \ \ *choice
startup\ 3418:\ \ \ \ \ \ \ #Drive\ the\ peasants\ like\ slaves;\ if\ we\ work\ hard\ enough,\ we\[aq]ll\ win.
startup\ 1133:\ \ \ \ \ \ \ \ \ Unfortunately,\ morale\ doesn\[aq]t\ work\ like\ that.\ \ Your\ army\ soon\ turns\ against\ you
startup\ 1133:\ \ \ \ \ \ \ \ \ and\ the\ kingdom\ falls\ to\ the\ western\ barbarians.
startup\ 1133:\ \ \ \ \ \ \ \ \ *finish
startup\ 3418:\ \ \ \ \ \ \ #Appoint\ charismatic\ knights\ and\ give\ them\ land,\ peasants,\ and\ resources.
startup\ 1132:\ \ \ \ \ \ \ \ \ Your\ majesty\[aq]s\ people\ are\ eminently\ resourceful.\ \ Your\ knights\ win\ the\ day,
startup\ 1132:\ \ \ \ \ \ \ \ \ but\ take\ care:\ they\ may\ soon\ demand\ a\ convention\ of\ parliament.
startup\ 1132:\ \ \ \ \ \ \ \ \ *finish
startup\ 3418:\ \ \ \ \ \ \ #Steal\ food\ and\ weapons\ from\ the\ enemy\ in\ the\ dead\ of\ night.
startup\ 1153:\ \ \ \ \ \ \ \ \ A\ cunning\ plan.\ \ Soon\ your\ army\ is\ a\ match\ for\ the\ westerners;\ they\ choose
startup\ 1153:\ \ \ \ \ \ \ \ \ not\ to\ invade\ for\ now,\ but\ how\ long\ can\ your\ majesty\ postpone\ the\ inevitable?
startup\ 1153:\ \ \ \ \ \ \ \ \ *finish
startup\ 10000:\ \ \ #Beat\ swords\ to\ plowshares\ and\ trade\ food\ to\ the\ westerners\ for\ protection.
startup\ 3278:\ \ \ \ \ The\ westerners\ have\ you\ at\ the\ point\ of\ a\ sword.\ \ They\ demand\ unfair\ terms
startup\ 3278:\ \ \ \ \ from\ you.
startup\ 3278:\ \ \ \ \ *choice
startup\ 3278:\ \ \ \ \ \ \ #Accept\ the\ terms\ for\ now.
startup\ 1601:\ \ \ \ \ \ \ \ \ Eventually,\ the\ barbarian\ westerners\ conquer\ you\ anyway,\ destroying\ their
startup\ 1601:\ \ \ \ \ \ \ \ \ bread\ basket,\ and\ the\ entire\ region\ starves.
startup\ 1601:\ \ \ \ \ \ \ \ \ *finish
startup\ 3278:\ \ \ \ \ \ \ #Threaten\ to\ salt\ our\ fields\ if\ they\ don\[aq]t\ offer\ better\ terms.
startup\ 1677:\ \ \ \ \ \ \ \ \ They\ blink.\ \ Your\ majesty\ gets\ a\ fair\ price\ for\ wheat.
startup\ 1677:\ \ \ \ \ \ \ \ \ *finish
startup\ 10000:\ \ \ #Abdicate\ the\ throne.\ I\ have\ clearly\ mismanaged\ this\ kingdom!
startup\ 3304:\ \ \ \ \ The\ kingdom\ descends\ into\ chaos,\ but\ you\ manage\ to\ escape\ with\ your\ own\ hide.
startup\ 3304:\ \ \ \ \ Perhaps\ in\ time\ you\ can\ return\ to\ restore\ order\ to\ this\ fair\ land.
startup\ 3304:\ \ \ \ \ *finish
\f[]
.fi
.PP
Randomtest plays 10,000 times by default, so you can see the intro and
the option text was displayed all 10,000 times.
There are three options (“war”, “trade”, and “abdicate”) each of which
was hit approximately one third of the time: Randomtest hit “war” 3,418
times, “trade” 3,278 times, and “abdicate” 3,304 times.
.PP
Under “war” there are three sub-options; those options divided the 3,418
“war” hits approximately into thirds: 1,133, 1,132, and 1,153.
Under “trade” there are only two sub-options; those options divided the
3,279 “trade” hits in half: 1,601 and 1,677.
.PP
If the hit count report tells you that some lines were hit zero times,
that suggests that the lines might be “dead” code \[em] code that can't
be reached no matter what choices the player makes.
However, code with 0 hits doesn't guarantee that the code is dead \[em]
it could just be very difficult to reach.
.PP
If you find code that's hard to reach, you'll have to decide for
yourself whether that indicates you have a bug.
For example, in the traditional “Choose Your Own Adventure” books, it
was very hard to reach a “good” ending; most endings had a bad outcome
(e.g.
death).
On the one hand, that might be a good thing, if it encourages players to
try again; on the other hand, it might be frustrating to keep reaching
bad endings.
.PP
To pick another example, if your choices have “right” and “wrong”
answers (e.g.
if your game has a lot of puzzles in it), Randomtest may tell you that
it's very unlikely to win your game when playing randomly.
But that may be ideal; if you can beat the game just by random choices,
your puzzles may be too easy!
.PP
When interpreting the hit count report, remember that you can divide the
hit count by 10,000 to get the percentage likelihood of hitting a given
line.
If a line is hit less than 100 times, then there's less than 1% chance
of hitting the line at random.
If that's too low in your opinion, consider sculpting your game balance
to allow more players to reach that code.