The tables below detail the results of our usability testing. For example, when asked
the first question ("How much money do you owe"), 12 out of 20 participants using the
paper EOB answered correctly with 15 being confident of their answer while 19 out of
20 participants answered correctly using the application with 19 being confident of their
answer. Only the results of the first two questions are recorded and analyzed here because
being of quantitative nature, it is easier to speak of their accuracy compared to the latter
two questions.
Table 2. Overall Study Results
The results of the first question ("How much money do you owe") is analyzed here.
First, an unpaired
t-test was performed on the means of the confidence scores and the
accuracy in order to determine whether or not the document ordering had any effect on the
result. When comparing the paper EOB between documents, the resulting
p-value of the confidence
scores was 0.6210 and the resulting p-value of the accuracy was 0.6051 indicating that the difference
in mean values between the ordering was not statistically significant. When comparing the application
use between documents, the resulting
p-value of the confidence scores was 0.8445 and the resulting
p-value of the accuracy was 0.380 indicating that the difference in mean values between the ordering
was not statistically significant. Therefore, we’ve first established that the ordering in which the
two documents were presented to the participants did not have a statistically significant effect
on the accuracy or confidence results.
Next, a paired
t-test was performed in order to compare the
means of the confidence scores and the accuracy between the use of a paper EOB versus a mobile
application. The resulting
p-value of the accuracy was 0.0047 and the
p-value of the confidence
score was 0.0002, indicating that the difference of mean values are very statistically significant.
The results of the second question ("What percentage of your total bill will you have to pay?") is analyzed
next. As above, an unpaired
t-test was performed on the means of the confidence scores and the accuracy
in order to determine whether or not the document ordering had any effect on the result.
When comparing the paper EOB between documents, the resulting
p-value of the confidence scores
was 0.7690 and the resulting
p-value of the accuracy was 0.6051 indicating that the difference
in mean values between the ordering was not statistically significant. When comparing the application
use between documents, the resulting
p-value of the confidence scores was 0.3193 and the resulting
p-value of the accuracy was 0.8885 indicating that the difference in mean values between the ordering
was not statistically significant. Therefore, we’ve first established that the ordering in which
the two documents were presented to the participants did not have a statistically significant effect
on the accuracy or confidence results.
Next, a paired
t-test was performed in order to compare the means of the confidence scores and
the accuracy between the use of a paper EOB versus a mobile application. The resulting
p-value
of the accuracy was 0.0102 and the
p-value of the confidence score was 0.0001, indicating that
the difference of mean values are very statistically significant. Digging deeper, we wanted to see
whether or not other attributes of the demographics had any effect on the results.
Upon closer inspection, participants who were already familiar with EOBs in some capacity performed
fairly well while using the paper EOB to answer the questions. In contrast, participants who were
unfamiliar with EOBs performed very poorly but their accuracy and confidence increased when using
the mobile application. Table 3 and Table 4 show more detailed results of this finding.
Table 3. Paper EOB Results
Table 4. App EOB Results
Table 5 details the corresponding
p-values. With the exception of the confidence score of the
first question (which is not quite statistically significant), all the other
p-values show
that the differences in the means between the paper EOB and the mobile application are statistically
significant.
Table 5. P-Values of Unfamiliar Participants
Finally, when asked how likely they would be to use this particular application
to understand their EOBs in the future, approximately 80% of the participants said that they would
be highly likely to use it again.