Abstract

Faith Japheth Paul and Yohan Bruce Jean

This study examined the use of a college course evaluation instrument in an effort to better understand
and improve the assessment data derived from the instrument. The purpose of the analysis was to
examine the dimensionality and reliability of the instrument but, more importantly, to understand what
the data really tells us and whether assessment literacy would improve data usability. Analysis of the
results suggests that the instrument tended to produce internally consistent data. However, the
instrument measured predominately one aspect of course quality. In addition, based on results of the
assessment literacy exercise, respondents seem to use very different criterion for rating; the constructs
being measured generally did not match what was intended; and the questions being asked did not
always match the scale being used. As a result the usefulness of any data interpretation and
subsequent decisions was deemed suspect. The results of this study also suggest that simple
assessment literacy interventions by themselves do not seem to drastically change the ability of raters
to score items reliably. A much more comprehensive effort would be needed to produce results that
would be beneficial for long term, data-driven decision making.