The DH & Assessment session Saturday afternoon started with the usual mini-rants about our regional accreditor and reductionist assessment and turned into an “oh, wow, here’s a tool for this” discussion. The core of the discussion revolved around the open-source <emma> assessment tool built for the University of Georgia’s first-year composition class. (Links: <emma> front-end, which will be frustrating because it’s just the sign-in for UGA students, and the website of the Calliope Initiative, the non-profit continuing development and the business end for other institutions.)
At the lunchtime Dork Shorts, Robin Wharton had demonstrated the gist of <emma>: students submit papers in Open Document Format. Then instructors and peers can comment on specific passages and code their comments by area of the comment (e.g., thesis development might be coded as green, something else as yellow, etc.). Students’ revisions are linked to their original documents, they declare when a revision is the final document, etc. So far, this looks like a useful, user-friendly way to comment on student work.
In the afternoon session, it became clear that <emma> was also being used for institutional assessment–it has the ability to look at the comments and the comment categories, a sample can be drawn for assessment by a set of readers, with disagreement on basic judgments by two readers kicked to a third reader or other moderation process, etc. And the system has the capacity to allow conclusions such as shifts in comment categories (i.e., student skill development) across a course or a longer span of time. In other words, institution-level judgments based on the day-to-day evaluative culture within composition instruction.
Those at the session had the obvious questions about the system (expensive? it was developed by one person in the English department who taught himself programming, along with two graduate students) and then we started talking about what would be necessary to develop parallel systems for performances (e.g., faculty-juried music performances at the end of the semester). So we gabbed a bit about Pear Note, Transana, and some other options. And then we discovered that as ODF documents, the base documents students submit can include media. Hmmn…
Bottom line for me: Huge thanks to Rob Balthazor and his team at UGA for showing how digital humanities can put assessment on a much less shaky footing.