Overall, every criteria on which we assessed student essays improves over the course of the core writing classes. Students generally come into our classes having some ability in each of the criteria, so they are not blank slates. The changes in scores from the beginning of 111x to the end of 111x are dramatic and the changes that happen during the 200 level course are less so.
English 111x, 211x, and 213x classes taught UAF in Fall 2014 and Spring 2015 participated in assessment. The assessment consisted of one essay collected from each student at the beginning of English 111x, one essay at the end of 111x, and one essay at the end of English 211x or English 213x. This process was designed to track student progress across the core writing courses. Five classes at each level were randomly chosen to be assessed and the essays were collected electronically via Google Drive. All identifying information was deleted. Five writing teachers were chosen to assess the essays. The Director of the University Writing Program met with the five assessors to calibrate their scales for assessment using a program-created rubric based on the AAC&U rubrics and designed specifically for our program in three categories: Control of Syntax and Mechanics, Self-Assessment, and Transfer. Essays were each assessed by two teachers in these categories. The director and research assistants compiled the results, assessing the essays a third time if there was more than a one point difference between assessor scores. After compiling the results and making our calculations we went to the Math Lab on campus for an opinion on the work we did and for advice on how to improve the reliability and accuracy of our assessment.
We chose to assess the essays for Control of Syntax and Mechanics, Self-Assessment, and Transfer. Measuring the control of syntax and mechanics shows how well students can manipulate language in order to convey meaning to their audience. Self-assessment shows how the students understand their own writing processes and the strengths and weaknesses therein. Transfer allows us to see if the students can take what they’ve learned about writing and apply it to new and different situations. These require critical thinking skills, which we value as a program. The rubric that we used is below.
| Capstone | Milestones | Benchmark | ||
| 4 | 3 | 2 | 1 | |
| Control of Syntax and Mechanics | Uses graceful language that skillfully communicates meaning to readers with clarity and fluency, and is virtually error-free. | Uses straightforward language that generally conveys meaning to readers. The language has few errors. | Uses language that generally conveys meaning to readers with clarity although writing may include some errors. | Uses language that sometimes impedes meaning because of errors in usage. |
| Transfer | Makes explicit references to previous learning and applies in an innovative (new and creative) way that demonstrates comprehension and performance in novel/differing situations. | Makes reference to previous learning and shows evidence of applying that knowledge and those skills to demonstrate comprehension and performance in novel/differing situations. | Makes reference to previous learning and attempts to apply that knowledge and those skills to demonstrate comprehension and performance in novel/differing situations. | Makes vague references to previous learning but does not apply knowledge and skills to demonstrate comprehension and performance in novel/differing situations. |
| Self- Assessment | Envisions a writing process or future self (and possibly makes plans) that build on past experiences that have occurred across multiple and diverse contexts. | Evaluates changes in a writing process in depth, revealing fully clarified meanings or indicating broader perspectives about meaning-making (through audience awareness for example). | Reviews a writing process by articulating strengths or weaknesses with some depth, revealing slightly clarified meanings or indicating a somewhat broader perspective about an aspect of meaning-making to increase effectiveness. | Reviews a writing process at a surface level, without revealing clarified meaning or indicating a broader perspective about meaning-making. May describe own performances with general descriptors of success and failure. |
Findings:
| Change within 111x | Change from 111x End to 200 Level | Overall Change | |
| Control of Syntax and Mechanics | 7.77% | 0.89% | 8.73% |
| Self- Assessment | 37.99% | 14.06% | 57.39% |
| Transfer | 16.72% | -7.16% | 8.37% |
Analysis:
SUMMARY: On average, student writing shows improvement in all categories.
Students begin English 111x with essays that use language that generally conveys meaning to readers with clarity but may include some errors. This improves slightly over time. The beginning scores in control of syntax and mechanics are higher than in the other categories, meaning that their use of the language is better coming in than their ability to reflect on their own writing process or to apply previous learning. Self-assessment shows the greatest improvement. Students generally just reach the benchmark for self-assessment when they enter our program, which means that they are able to review the writing process at a surface level, but don’t reveal a clarified meaning or show a broader perspective about meaning-making. By the time they finish 211x or 213x, they are much closer to the 2 milestone, so they can articulate strengths and weaknesses with some depth, revealing slightly clarified meanings or indicating a somewhat broader perspective about an aspect of meaning-making to increase effectiveness. The picture with transfer is a bit more complicated, as the improvement is not linear, but there is still an 8.37% increase in the average transfer scores by the end of 211/213x.
Analysis:
SUMMARY: The education that students receive before they arrive has prepared them for the English 111x course in terms of control of mechanics and syntax, and they continue to improve in this competency.
When students begin English 111x, nearly 70% reach the first milestone of generally conveying meaning to their audience with clarity but having some errors. By the time they finish 211/213x, fewer than 5% of students score at or below the benchmark, using language that sometimes impedes meaning because of errors in usage. By this stage, in comparison to 111x, more students, over 75%, have reached at least the first milestone. Additionally, some students had reached a score of 3.5, nearly making the capstone for syntax and mechanics, using graceful language that skillfully communicates meaning to readers with clarity and fluency and is virtually error-free.
Analysis:
SUMMARY: Our department’s emphasis on students’ awareness of the choices they make as writers is paying off.
Nearly 70% of students entering English 111x are at the benchmark or lower, meaning that they review their writing process at a surface level without revealing clarified meaning or indicating a broader perspective about meaning-making. This is partially due to the question, which could be interpreted as not asking about one’s self but about good writers in general. Students may not include themselves in this group and therefore might not address their own writing process. However, students make a huge leap in their self-assessment, especially over the course of English 111x. By the time they finish the course, nearly 70% of students score above the benchmark score of 1. This is in line with our program curriculum, which values this kind of self-reflection and awareness of one’s choices as a writer. Additionally, scores continue to rise during 211/213x. By the end of the courses, only twenty percent of students score at the benchmark or below and about twenty percent of students score above the second milestone of evaluating changes in their writing process in depth, revealing fully clarified meanings or indicating broader perspectives about meaning-making.
Analysis:
SUMMARY: There is a slight decrease between the end of 111x and the end of the 200 level courses; however, students still show improvement between the beginning of 111x and the end of the 200 level courses.
This graph shows the percentage of student scores in transfer at each stage of the assessment process. The information that we collected about students’ transfer abilities is more complicated than self-assessment or control. The decrease may not be indicative of actual student learning but of differences between the questions being asked. There is a subtle but important difference between the question asked at the end of 111x and the one at the end of the 200 level classes that could account for the negative change in transfer demonstrated in student responses; the 200 level question isn’t asking directly about previous learning. When they begin 111x, less than 30% of students reach the first milestone of making reference to previous learning and attempting to apply that knowledge to demonstrate comprehension and performance in novel situations. At the end of 211/213x, nearly 40% of students reach that milestone. And it is significant that the 0 scores are virtually eliminated by the end of the 200 level courses. This is a small improvement and one that we hope to continue to work on.
Proposed curricular changes resulting from conclusions drawn above
It is clear that students are improving their writing skills over the course of the core writing classes. The jump that happens during English 111x is remarkable and we would like to see those same results at the 200 level. This suggests that we need to:
- Revise guidelines for teaching analysis
- include more analysis resources on WriteAlaska
- include more detailed training on analysis for incoming teachers
- Improve support for 200 level instructors
- conduct training workshops for 200 level instructors
- Adjust student expectations for 200 level courses
- give 200 level courses new title
- possible inclusion of 214, so the emphasis is on continuing the progress that happens in 111x instead of having a divide between disciplines
All University Writing Program changes will be addressed by the Composition Committee of the English Department.
The assessment committee included the following faculty.
Sarah Stanley, Chair
August Johnson, Write Alaska Research Assistant
Jaclyn Bergamino, Write Alaska Research Assistant
Eileen Harney
Jennifer Tilbury
Cindy Hardy
Victoria Avery
Megan Mericle