Annotate stories

It might not take as long as you think it will. Annotation usually starts out slow but speeds up as you work through your stories. Still, if you have thousands of stories, you might need to trim your annotation ambitions. Here are some ideas that could help.

1. Triage your annotation questions. Cut down the list to the bare minimum. Try out each question on ten or twenty stories. Evaluate which questions seem the most useful in small subsets of stories. It's always hard to guess whether you will find patterns in annotation questions, but with a little exploration you can get an intuitive feel for which questions seem the most worth pursuing.
2. Figure out how long it will take to answer your annotation questions for one story, on average. The process will speed up later on, so discount the time by a quarter or a third. Then make an educated guess as to how many stories you will be able to annotate given the time you have. For example, if it will take you three minutes to annotate each story, and you have 300 stories, you will need (3x300)/60 = 15 hours to do your annotations. Can you spend that much time? If not, what do you want to cut down? The number of questions? The number of stories? To draw interesting and useful graphs, and to run statistical tests, it is best to generate data for at least 100 stories. So, how long would it take you to annotate 100 stories? (For the above example, it would be 5 hours.) Can you do that?
3. If you think you will have enough time to annotate some but not all of your stories, annotate a random sample of stories. Below the list of stories on the "Annotate stories" page is a "Random" button. Keep clicking that button and answering questions until you have used up your annotation time.
4. If you can't afford the time to annotate even a subset of your stories, see if you can get some help. Find some friends, family members, or colleagues to help you. Explain how you plan to ensure that your annotation process is rigorous, ethical, and transparent. Then have each person do the same thing you're doing: click the Random button, answer the questions, repeat. (If you use this option, be sure to add a "who annotated this" question to check for inter-rater reliability.) When you have all reached your limit of time or patience, see how many stories you have annotated. Then look at your graphs.

A few. Try these ideas:

• You can use the story table's column headers to sort stories by their annotation answers. This can be a big help when you are consolidating lists of answers. For example, if you need to decide whether you should merge two answers, you can skim through the stories marked with each answer to make your decision.
• If you need to find a story but can't remember its name or where it was in the list of stories, use your browser's text-search function to search the web page for any words you remember from the story. The texts of all of your stories are in the table, so a general text search will help you find the story you want to find.
• If you are working on emergent sets of answers, you may find yourself unsure of what to mark for some of your stories. Should you create a new answer, choose an existing answer, or leave the question blank? To help with such quandaries, you can use annotation questions to keep track of "problem stories." For example, you could create a free-text "Notes" annotation question and use it to leave yourself reminders on issues you need to resolve. The reminders will appear (as a column) in your table of stories, and you can sort the stories by them. Or you could set up a "Resolved?" annotation question, with answers like "firm," "tentative," and "problematical." Creating patterns to graph is only one of the things you can do with annotation questions. You can also use them to improve your annotation process.
• If you are sharing the annotation task with another person, you can divide up the stories using their index numbers in the table. For example, if you and I were annotating 100 stories, you might confine yourself to the first 50 while I handle the last 50. Also, we might agree to only add new answers while we work, then consolidate our answers together.
• It can be annoying to have to keep scrolling up and down the page as you fill in answers and look at your graphs of evolving answers. Your browser might have a key combination that makes this easier. Try out clicking on various parts of the page and hitting the Control (or Command) key, plus the up and down arrows, to see if you can jump up and down more easily.

This is the best way to do it, I think.

1. Write your annotation questions.
2. For a small number of stories, say five or ten, answer the questions by hand, using the "Annotate questions" page. As you do this, think about your questions and make sure you are happy with them.
3. Export your hand-entered annotations to a CSV file.
4. Open the file in a spreadsheet program. Look at its structure.
5. Maintain that structure as you type or paste in the remainder of your annotations. Don't touch the story names and texts. Just add data to the other columns.
6. Check that your annotations will import correctly by clicking Check before importing and using your browser console to look for errors. Look for:
   • rows that don't match any stories
   • rows that match multiple stories
   • header names that don't match question names
   • answers not found in the lists of fixed answers specified in the questions
7. When you find no more errors, import your annotations.
8. Double-check that your annotations have been imported correctly by looking at the graphs on the "Annotate stories" page (below the list of stories). Spot-check a few graph totals against the totals in your spreadsheet.

For each story in your collection, the first two columns in your annotations CSV file must be the story name and story text, exactly as they appear in NarraFirma. Both name and text need to match before any annotations will be applied to a story. This is to prevent accidental assignment when multiple stories have the same names or texts.

For each column after the name and text, enter data that matches the annotation questions you created in NarraFirma. You cannot create annotation questions by importing a CSV file (as you can with story forms). You have to create the questions in NarraFirma itself. Write the short name of each question in the header of the column(s) for that question.

For a boolean or checkbox question, use one column, and write either "true", "false", "yes", or "no."

For a select or radiobuttions question, use one column, and write the answer in the column. It must exactly match one of the fixed answers entered for the question.

For a slider question, use one column, and write the number in the column. It must be a value between 0 and 100, inclusive.

For a checkboxes question, use as many columns as there are fixed answers entered for the question. Write the question short name as the header of each column (yes, over and over). Write each answer, exactly as specified in the fixed list of answers, in its own column. Do not write multiple answers in the same column. For example:

Story name	Story text	Feeling	Feeling	Feeling
The blue dog	Once there was a blue dog.	happy
The red hen	Once there was a red hen.		sad
The yellow spider	Once there was a yellow spider.			bored
The green cat	Once there was a green cat.	happy

New annotations always overwrite older annotations. However, empty cells in the CSV file will not delete existing data. Any answers that are in one import but not in the next will not be removed. So for example, if you say a story has the value "happy" for the question "Feeling," and then you import the same file again, but with "happy" removed, the answer "happy" will still be connected to the story (because it was imported the first time). You can only add data by importing annotations. You cannot remove data in this way. To remove an annotation answer, change it on the "Annotate stories" page.

Yes. Go back to the Collection part of NarraFirma. Export your entire story collection to a CSV file. Remove the annotation data. Then re-import the story data into a new story collection. Your annotation questions will still exist (they are stored in the project), but there will be no annotation data in the new story collection.