Accessibility digest

Introduction

This is an explanation of the scoring of the page.

Motivations for scoring

Why score? Specifically, why aggregate many facts about a page into a single number?

One motivation is to simplify comparison and tracking. If you want to compare a page with other pages, or if you want to track changes in a page over time, aggregating many test results for each page into one page score simplifies the task. It becomes possible to say things like Page A is more accessible than page B or Page A is becoming more accessible over time.

Another motivation is to influence behavior. A score arises from decisions about importance, urgency, and other attributes. People may use scores to award benefits, impose costs, or prioritize work. So, scoring can influence who gets and does what.

How to score?

Scoring is subjective. Accessibility testing tools use various scoring procedures, based on various ideas.

The built-in scoring procedures of Testilo are based on the idea that multiple attributes should affect a web-page accessibility score, including:

• conformity to standards
• adherence to best practices
• how many different issues exist
• how many instances of each issue exist
• how many tools report violations of their rules
• how trustworthy each test of each tool is
• how serious each violation of a rule is
• simplicity
• speed of page responses
• testability
• conformity to expectations of browsers

Some ideas found elsewhere in accessibility scoring are rejected by the Testilo procedures:

• density: that an accessibility score should be based on the count of accessibility faults as a fraction of the count of potential accessibility faults (so, the score of a large page with particular faults should be better than the score of a smaller page with exactly those faults)
• applicability: that the score of a page should depend on the types of its content (so, the score of a page with accessible video content should be better than the score of an otherwise identical page without video content)

Scoring method summary

The scoring method is found in the code of the procedure. The file for any procedure tspnn is in the file procs/score/tspnn.js within the Testilo package.

The score computed by this procedure for any page is a non-negative number. The numbers represent accessibility faults, so a higher-scoring page is considered less accessible than a lower-scoring page. The best possible score is 0.

The procedure makes the score of a page the sum of 7 components:

• issue score: how many instances of issues were reported
• solo score: how many violations of rules not yet classified into issues were reported
• tool score: how many violations of their rules did tools report
• element score: how many HTML elements did any tool identify as violating at least one tool rule
• prevention score: how many tools were unable to perform their tests on the page
• log score: how much logging of page errors and other events did the browser do
• latency score: how long did the page take to load and become stable

Scoring method details

The above component descriptions omit various details.

About a thousand tool rules are classified into issues in the tic49.js file used by this procedure. Each of those rules has a quality, ranging from 0 to 1. When the issue score is computed, the count of violations of each rule is multiplied by the quality of that rule. Whichever tool has the largest quality-weighted violation count for an issue, that count is treated as the instance count for the issue. Thus, if 8 tools each report 15 violations within the issue, and their rule qualities are all 1, the instance count is 15, not 120. Moreover, the issue itself has a weight, ranging from 1 to 4, representing its importance. This instance count is multiplied by that weight. That product is further multiplied by the issueCountWeight value, namely 10. That final product is further adjusted if the issue is inherently limited in instance count. For example, if the issue is that the page html element has no lang attribute, the instance count is limited to 1. If there is such a limit, the maxWeight value, namely 30, is divided by the actual instance count and the quotient is added to 1. That product (or 1 if there is no limit) is multiplied by the instance count, and then the result is treated as the contribution of the issue to the issue score.

Each solo (not yet issue-classified) rule violation is multiplied by the sum of 1 and the ordinal severity of the rule, to produce the solo score.

Each rule violation reported by each tool is severity-weighted in the same way as solo rule violations are. Then the sum of those violations is multiplied by the toolWeight value, namely 0.1, to produce the tool score.

The count of elements reported as violators of any rule is multiplied by the elementWeight value, namely 2, to produce the element score.

The count of prevented Testaro rules is multiplied by the testaroRulePreventionWeight quantity, namely 30, and the count of other prevented tools is multiplied by the preventionWeight value, namely 300, to produce the prevention score.

The log score is the sum of several components:

• log count: how many times the browser logged a message
• log size: how many characters large the collection of browser log messages was
• error log count: how many browser log messages reported page errors
• error log size: how many characters large the collection of browser log messages reporting page errors was
• prohibition count: how many times the browser logged a prohibited response status (403)
• visit rejection count: how many times the browser logged an abnormal response status

Each of these components is multiplied by a weight found in the logWeights object.

Finally, the latency score is based on how much longer it took for the page to become loaded and stable than expected. The expected total duration in seconds is the normalLatency value, namely 22 seconds (that is 2 seconds per visit, multiplied by the 11 visits of the 11 tools). This is subtracted from the actual total latency, and that difference is multiplied by the latencyWeight amount, namely 2.