\section{Results}
\label{section:results}
In this section results from evaluating the Naive Bayes Classifier against various datasets and using a few different
tokenizing approaches are described.

\subsection{Datasets}
\label{subsection:datasets}
In addition to the required Dickens and Hardy books required for the assignment, some additional datasets
were taken from the UCI Machine Learning Repository \cite{uci}. Specifically, classification datasets stored in an
easy-to-use text format were selected. The datasets used are described in table \ref{table:datasets}. For each dataset,
its ``type" is specified, indicating the structure in which the data is stored. The available dataset types are
inline\footnotemark[1] and guten\footnotemark[2].

\begin{table}
    \begin{tabular}{lll}
        \hline
        \textbf{Dataset} & \textbf{Source} & \textbf{Type} \\ [0.5ex]
        \hline\hline
        SMS & UCI - SMS Spam Collection & inline\footnotemark[1] \\
        Badges & UCI - Badges & inline \\
        Main & Gutenberg - Dickens \& Hardy & guten\footnotemark[2] \\
        \hline
    \end{tabular}
    \caption{Datasets used for this paper}
    \label{table:datasets}
\end{table}

\footnotetext[1]{Dataset is stored as a single file in which each line represents a training point. The first word in
each line is the class/category, while the rest of the line is a list of words used as the training "text blob."}
\footnotetext[2]{Dataset is stored as a list of directories representing classes/categories (e.g. "dickens", "hardy").
Each file within the class directories represent a training point. These files are actually books, but are abstractly
considered to be "text blobs," just like the inline dataset type.}

\subsection{Performance Measurements}
\label{subsection:performanceMeasurements}
Evaluations of the classifier are recorded in terms of ``True Positive" (TP), ``False Negative" (FN), ``False Positive"
(FP), \& ``True Negative" (TN), indicating what was predicted verse what the data's actual class. Then, the results are
presented in terms of some commonly used formulas: ``Precision", ``Recall", ``Specificity", and ``Accuracy"
\cite{measures}. A description of these formulas is shown in table \ref{table:measures}.

\begin{table}
    \begin{tabular}{lll}
        \hline
        \textbf{Measure} & \textbf{Formula} & \textbf{Meaning} \\ [0.5ex]
        \hline\hline
        Precision	& TP / (TP + FP) & \% of correct +'s \\
        Accuracy	& (TP + TN) / (total) & \% correct \\
        Recall 	    & TP / (TP + FN) & \% of +'s predicted as + \\
        Specificity	& TN / (TN + FP) & \% of -'s predicted as + \\
        \hline
    \end{tabular}
    \caption{The measurement terms used in evaluations for this project, with their formulas and intuitive meanings}
    \label{table:measures}
\end{table}

\subsection{Evaluation of the Basic Classifier}
\label{subsection:basicResults}
Here, the basic classifier is evaluated. The basic classifier simply tokenizes text by splitting on spaces. It does
nothing to filter or alter the tokens before training and classifying on them. Here, the results of evaluating SMS,
Badge, and Dickens-vs-Hardy data are discussed.

\subsubsection{SMS Spam Collection}
\label{subsection:smsBasic}
The SMS Spam Collection dataset \cite{sms} contains 5,574 data samples, of which 4,827 (86.6%) are not spam (``ham") and 747 (13.4%)
are spam (``spam"). The precision \ref{fig:smsBasicPrecision} and accuracy \ref{fig:smsBasicAccuracy} of the classifier
are shown for varying ratios of the dataset used to train with 5 random distributions for each ratio. Recall and
specificity are not shown because they were found to always be 100\% and 0\%, respectively. Specific numbers
for a few train:test ratios are shown in table \ref{table:smsBasicResults}.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/sms_basic-precision.png}
    \caption{SMS Precision: Overall, spam-specific, and ham-specific precisions found at varying portions of the dataset used to train}
    \label{fig:smsBasicPrecision}
\end{figure}

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/sms_basic-accuracy.png}
    \caption{SMS Accuracy: Overall, spam-specific, and ham-specific accuracies found at varying portions of the dataset used to train}
    \label{fig:smsBasicAccuracy}
\end{figure}

\begin{table}
    \begin{tabular}{rrrrrrrr}
        \hline
        \textbf{Train} &
        \textbf{\begin{math}P_{All}\end{math}} & \textbf{\begin{math}P_{Ham}\end{math}} & \textbf{\begin{math}P_{Spam}\end{math}} &
        \textbf{\begin{math}A_{All}\end{math}} & \textbf{\begin{math}A_{Ham}\end{math}} & \textbf{\begin{math}A_{Spam}\end{math}} &
        \textbf{t} \\ [0.5ex]
        \hline\hline
        1\%  & 44.7 & 36.3 & 98.7 & 44.7 & 36.3 & 98.7 & 79s \\
        5\%  & 78.0 & 75.1 & 97.1 & 78.0 & 75.1 & 97.1 & 66s \\
        15\% & 88.8 & 87.6 & 96.7 & 88.8 & 87.6 & 96.7 & 68s \\
        25\% & 91.7 & 91.0 & 96.6 & 91.7 & 91.0 & 96.6 & 69s \\
        50\% & 94.4 & 94.1 & 96.5 & 94.4 & 94.1 & 96.5 & 79s \\
        75\% & 95.5 & 96.5 & 95.3 & 95.5 & 96.5 & 95.3 & 115s \\
        \hline
    \end{tabular}
    \caption{Mean \% precision (P) and accuracy (A) found over 1000 iterations at a number of train/test ratios,
    and the time (t) to split and evaluate each}
    \label{table:smsBasicResults}
\end{table}

\subsubsection{Badge Problem}
\label{subsection:badgesBasic}
The Badge Problem dataset \cite{badge} contains 294 data samples, of which 210 (71.4%) are positive (P) and 84 (28.6%)
are negative (N). The dataset contains a list of names, each of which has a class of P or N (originally ``+'' and ``-'',
but it was converted for this project to avoid special characters) This is an interesting dataset because the original
algorithm determining how to assign P and N is unknown. The original intent was to create a challenge for computer
scientists to generically apply machine learning to discover the target function. The precision
\ref{fig:badgeBasicPrecision} and accuracy \ref{fig:badgeBasicAccuracy} of the classifier are shown for varying ratios
of the dataset used to train with 5 random distributions for each ratio. Recall and specificity are not shown because
they were found to always be 100\% and 0\%, respectively. Since the basic approach simply looks at words, it did a
poor job on this dataset. Improved results could be possible by evaluating gender and length of name, or perhaps
simply be tokenizing on characters rather than words.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/badge_basic-precision.png}
    \caption{Badge Precision: Overall, P-specific, and N-specific precisions found at varying portions of the dataset used to train}
    \label{fig:badgeBasicPrecision}
\end{figure}

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/badge_basic-accuracy.png}
    \caption{Badge Accuracy: Overall, P-specific, and N-specific accuracies found at varying portions of the dataset used to train}
    \label{fig:badgeBasicAccuracy}
\end{figure}

\subsubsection{Dickens vs. Hardy Novels}
\label{subsection:novelsBasic}
Finally, a data set of 15 Charles Dickens books \cite{gutenberg:dickens} and 15 Thomas Hardy books \cite{gutenberg:hardy}
was evaluated against the naive bayes classifier. Only slight alterations were made to the original books downloaded
from the Gutenberg Project. First, all preamble preceeding ``Chapter 1" or similar was removed so as not to train the
classifier on the author's name. Additionally all postamble added by Project Gutenberg was removed for the same reasons.

Without applying any special tokenizing, the stringified naive bayes classifier was found to be about 6 MB (6,125,536 B)
after training on 5 of 30 books simply from storing and tracking every word seen in the training set. Obviously the
basic approach is not scalable yet, for the sake of completeness, results are shown here as was done for the SMS Spam
Collection and the Badge Problem. In general, the classifier did much better with Dickens than Hardy. This likely
stems from the fact that the Dickens novels used here were larger than the Hardy books and thus any given word is more
likely to be a Dickens word. Again, the precision \ref{fig:novelBasicPrecision} and accuracy \ref{fig:novelBasicAccuracy}
are shown for varying ratios. A more limited range of ratios was used because the dataset was only of size 30.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/novels_basic-precision.png}
    \caption{Novels Basic Precision: Overall, Dickens-specific, and Hardy-specific precisions found at varying portions of the dataset used to train}
    \label{fig:novelBasicPrecision}
\end{figure}

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/novels_basic-accuracy.png}
    \caption{Novels Basic Accuracy: Overall, Dickens-specific, and Hardy-specific accuracies found at varying portions of the dataset used to train}
    \label{fig:novelBasicAccuracy}
\end{figure}


\subsection{Extending the Classifier}
\label{subsection:advancedResults}
After the poor results from the basic classifier on the Dickens verse Hardy novels, several attempts were taken to
improve the accuracy and precision of the classifier on these large bodies of text.

First, a stemming step was added to the tokenizing process, in hopes of slimming the dataset and removing noise;
however, this had little impact on the effectiveness of either the precision \ref{fig:novelsStemPrecision} or accuracy
\ref{fig:novelsStemAccuracy}.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/novels_stem-precision.png}
    \caption{Novels Stem Precision: Overall, Dickens-specific, and Hardy-specific precisions found at varying portions of the dataset used to train}
    \label{fig:novelsStemPrecision}
\end{figure}

\begin{figure}[ht!]
    \centering
    \includegraphics[width=90mm]{img/novels_stem-accuracy.png}
    \caption{Novels Stem Accuracy: Overall, Dickens-specific, and Hardy-specific accuracies found at varying portions of the dataset used to train}
    \label{fig:novelsStemAccuracy}
\end{figure}
