Assessment Results
Writing Assessment Results 2009
Assessment of Writing Skills among Beginning and Advanced Students
IU Northwest General Education / Assessment Committee
May 12, 2009
Report Date: July 8, 2009
Participating Committee Members: Mary Ann Fischer (co-Chair), Doug Swartz, Barbara Peat, Latrice Booker, Sheila Marie Trzcinka, Linda Delunas, Karen Evans, Dee Dee Ige, Jim Thomas, Eva Mendieta, Cynthia O'Dell, Taylor Lake
Introduction
During May of 2009, the combined General Education / Assessment Committee of the IU Northwest Faculty Organization completed an assessment of writing skills revealed in a sample of forty six final papers selected from either W131, Elementary Composition, or from upper division courses in each of the six schools and colleges. This report details the procedures used the solicit the papers, to review and evaluate them, and the results as revealed in quantitative judgments and comments of the faculty evaluators who served on the committee. We hoped to establish a baseline of evidence for writing skills and to find evidence of more advanced skills among upper division papers than among W131 papers.
Selection of Writing Samples
The Associate Vice Chancellor of Academic Affairs requested that the Deans of the schools and colleges solicit sample papers from their faculty. The request was for at least two papers along with descriptions of the assignment, from a given instructor, one judged by the instructor as relatively skilled and the other as relatively weak. All identifiers except the course number were stripped from the papers before they were submitted to the AVCAA. The final sample included 23 papers drawn from W131 sections and 23 papers from upper division courses. All schools and colleges submitted at least 2 papers from upper division courses.
Assessment Procedures
The committee adapted a tool designed by Professor Anne Balay, Department of English. It included eight categories, including definitions, of writing skills: research, persuasion, prose and syntax, counter-argument, mechanics, complexity, reader pleasure, and organization, (See attachment). Each evaluator read three or four papers and judged them as excellent (3 points), acceptable (2 points), or unacceptable (1 point) in each category; additional open-ended comments were optional. Each paper was evaluated by only one rater; thus no inter-rater reliability can be reported.
Results
The mean rating for each skill category among W131 and upper division papers is shown in the table below. As expected, in every category upper division papers were given higher ratings than those from W131. Readers knew the course level of the paper, so expectations for better performance on the upper division papers may have influenced ratings. Future assessments should remove the course identifiers. Ratings for the W131 papers were low, that is, the means were below 2, defined as “Acceptable.” Mean ratings for the upper division papers were just above “Acceptable” in all but 2 categories.
| Course Level |
| W131 (N=23) | Upper Division
(N=23) | Combined
sample |
| Mean | Mean | Mean |
Research | 1.83 | 1.96 | 1.89 |
Persuasion | 1.74 | 2.17 | 1.96 |
Prose and syntax | 1.65 | 2.23 | 1.93 |
Counterargument | 1.61 | 1.68 | 1.64 |
Mechanics | 1.61 | 2.13 | 1.87 |
Complexity | 1.65 | 2.13 | 1.89 |
Reader pleasure | 1.73 | 2.09 | 1.91 |
Organization | 1.68 | 2.02 | 1.86 |
Total points | 13.70 | 16.24 | 14.97 |
The comments (see attachment 2) about the papers were much more negative than the mean ratings would have suggested in the case of the upper division papers. It may be that raters hesitated to classify the upper division papers us “Unacceptable” even though they were highly critical. If so, future assessment efforts should use a rating scale with a wider range.
In general, evaluators found the experience of reading student papers valuable and useful. The committee did not discuss the question of what steps should be taken to improve the quality of student writing at that time. However, we agreed to take up this issue at the next General Education / Assessment Committee meeting in the Fall of 2009. Comments and suggestions on the process included the following:
- Our assignment instructions should be clearer to the units to ensure more conformity to the assignment.
- Rubric should be more universal to any assignment; suggested areas of assessment include: focus, development, mechanics, philosophy.
- Sheila Marie T. brought a rubric that we might consider for next time rather than the one we used this time?
- Suggestion that everyone, including faculty, have a Handbook (perhaps purchased by the university) that is used in most w130 or w131 classes to encourage students to keep them and use them in all future classes.
Attachment One Writing Assessment Form
General Education / Assessment Committee Assessment Project
May 12, 2009
Reader __________________ Paper #
This list of desired attributes has been adapted from one designed by Anne G. Balay , although she strongly favors holistic assessment rather than evaluating attributes separately, as we will be doing here. She has argued that after a student has completed one year of a college-level writing course, a final essay should demonstrate most of these attributes. After reading each essay please use the definitions which have been provided to rate each sample paper for each attribute.
| Excellent
3 points | Acceptable
2 points | Not acceptable
1 point |
use of research | | | |
effectiveness of argument or persuasion | | | |
style reflecting prose and syntax | | | |
use of counter-argument without bias or ethos | | | |
attention to mechanics and punctuation | | | |
complexity of ideas | | | |
reader pleasure | | | |
structure reflecting organization and focus. | | | |
Total points in each category | | | |
Total points: ______________
Additional comments:
Definitions of the categories:
Use of research: Students should execute both library and Internet research in gathering sources for their projects. Sources appearing in student papers should be drawn from electronic databases, and contain a combination of academic, public, and primary sources. Students are expected to demonstrate an ability to summarize, analyze, and synthesize sources appropriate to academic writing.
Effectiveness of argument or persuasion: Students are expected to establish a research driven thesis supported by warranted claims that shaped their purpose in writing. Claims are supported by evidence gleaned from research, primarily, as well as from personal experience and common knowledge.
Style reflecting prose and syntax: At the surface level, the writing should be clear. Sentences should reflect a degree of sophistication appropriate to a college writer and should flow in prose that sustains the argument.
Use of counter-argument: The essay should include a counter-argument as reflecting a writer’s sense of ethics and fair-mindedness in working a thesis. Furthermore, a writer should guide the argument without undue bias, acknowledging opposing opinions and points of view.
Attention to mechanics and punctuation: Writing should be clear and free of errors. The writer should demonstrate mechanics that reflect the grammar of Standard American English, free of slang and jargon, and should attend to the impact of punctuation on the reader.
Complexity of ideas: Papers should reflect thoughtful interpretations and applications of research (considered critical thinking). As such, successful papers avoid pedestrian arguments on mundane subjects.
Reader pleasure: A good essay interests its writer as well as its reader, and is fresh in ideas and well written.
Structure reflecting organization and focus: Deep issues of organization and development define this category. A paper’s thesis and claims should guide the reading, and those claims should be developed through extended discussions and use of research as support.
Attachment 2 Comments on papers
Total points
Course level
Comments
8
131
this essay does not follow assignment, does not provide an adequate or clear presentation of the sources pursue a thesis, provide fresh ? Or cite sources. Furthermore, it contains an array of grammatical errors
8
131
a lot of serious errors in sentence structure. I get no sense of her sources, arguments and writer relies too much on personal experience
9
131
mostly writer opinion, lack of research support. Fails to cite sources in text references unacceptable, many writing errors (mechanics)
9
131
works cited page has appropriate references but they are not really used in text as a basis for the "argument" this is simply an "opinion piece" with attempts to make it appear as if the research supports the writers opinion. Completely unsophisticated, no fresh ideas, very repetitive
10
131
10
131
10
131
major claim was not clear enough and sub-claims weren't clear and didn't adequately support and develop major claim. Use of research indicates lack of critical reading skills.
11
131
11
131
no central thesis other than title, just a flow of thoughts with personal example and a few quotes supporting only the authors point of view, problems with grammar
12
131
13
131
Very mixed, when formal scores were low, in middle,l when personal, much better
14
131
14
131
little depth to analysis
15
131
15
131
flow more conversational than formal essay - no central thesis and so supporting TPs don't have connection other than conversational, but sent structure good and mechanics
16
131
one of the sources is not cited. There are many generalizations about minorities and communities that would seem to require documentation
16
131
16
131
met the criteria
17
131
opening and closing ¶ reflect acceptable writing (whereas middle ¶'s seem more disorganized and subject to inappropriate sequence of ideas
18
131
19
131
21
131
Basically good, arguments occasionally personal, bit long
23
131
8
upper division
this was unacceptable for a 400 level class
8
upper division
Student did not follow instruction in assignment and did not seem to even understand the point of the assignment
8
upper division
no citations / references, can't tell what kind of assignment this is. Not required length for a take-home paper. No book identified (except 1984) so not a book review
10
upper division
If student ? Article o a study to answer the questions in the instructions then, according o the ? Noted deficiencies in information the ? Should not have ? This article for review
12
upper division
did not follow instructions of assignment; weak on syntax and grammar which hindered reader understanding - was persuasive in argument that good source for classroom instructor and did connect approp. To the standards
12
upper division
show and tell paper
13
upper division
Frankly, I didn't understand the assignment so evaluated the paper totally on its own. Was research required?
14
upper division
ineffective use of transitions, spelling errors, and sentence structure errors detract from this broad look at schizophrenia and cautionary view of certain medications
15
upper division
the paper is too much driven by summary: it's often not clear why we are being provided with description of people and events from the book being reviewed. There needs to be a better sense of purpose in the writing, a central evolution
15
upper division
incorrect APA format, some misuse of words, assignment calls for figures to be placed at end of paper but writer includes them with text, conclusion is vague / cursory
16
upper division
18
upper division
18
upper division
18
upper division
good grammar syntax
18.5
upper division
19
upper division
Citations in text lack year and ?. Student uses footnotes instead -maybe okay? Problems with confusing outline structure. Intro poorly developed
20
upper division
20
upper division
well written paper
21
upper division
good writer, little or no evidence of field research visits to the church
22
upper division
22
upper division
Good
23
upper division
there weren't a sufficient number of sources, and I wasn't sure how the source provided historical background for the memoir. The citation system is also confusing
23
upper division
well documented, effectively written paper!
