Presentation:

Introduction

The CLA is a performance-based test that measures (1) problem solving, (2) critical thinking, (3), quantitative reasoning , and (4) written communication. The CLA provides an institutional assessment, rather than assessment of individual students.  There are two formats in which to use the CLA:  1) A longitudinal study (costing $28,000) that studies the same cohort of students from freshman year to senior year. Students are tested as incoming freshmen, rising juniors, and graduating seniors.  To provide baseline information, a sample of seniors is given the CLA during the academic year in which the longitudinal study begins.  2) A cross-sectional study in which a cohort of freshmen and seniors are tested and their scores are compared, after controlling for students’ SAT scores. They suggested that the cross sectional method may not be a good measure of value added.

Implementation

Hurd and Beil discussed how the CLA was implemented at their respective institution. At Syracuse University (SU) the decision to implement it was top-down, coming from the chancellor of the University. At that time, SU did not have an office with overall responsibility for institutional research and assessment, and there was neither a mechanism nor criteria for determining what assessment strategies would be most appropriate and beneficial for the institution. Their chancellor, who is committed to assessment and to participating in national research, made the decision to use the CLA. 
At George Washington University (GW), it was a bottom-up decision. The arts and science dean wanted to evaluate the freshmen dean’s seminars, a program in which full-time faculty taught seminars on their areas of research that were for freshmen only. He wanted to see if these seminars had an impact on student learning. Beil was very impressed with the CLA and wanted to use it to measure learning outcomes at GW. They agreed to use the CLA to measure the impact of the dean’s seminars using the CLA. Half the sample included students who were enrolled in a dean’s seminar; half were not enrolled in the dean’s seminars. Both schools participated in a longitudinal study.

Process

The CLA is a three-hour test that students take online. It is not a high-stakes test: Getting students in the room does not guarantee they will be motivated to do their best. The institutions had different experiences implementing the CLA.  GW’s was more positive. To encourage student participation, freshmen and sophomores were given the opportunity to register early for courses, a very desirable incentive. Seniors were given a university blanket. In addition to assessing students’ responses to the questions, the CLA keeps track of the number of keystrokes a student makes and the amount of time spent on each section. Students also are asked to rate their motivation to do well on the test.
At SU, implementation was more difficult. While each of SU’s nine undergraduate schools and colleges must have some kind of first year experience, they are very different. The first-year forum sections in The College of Arts and Sciences were selected both because The College has the largest undergraduate enrollment and because of the structure of the forums. Faculty members are trained in the summer and there is a common curriculum, but there is also flexibility within that. Faculty were asked to give up two of their ten classes for this testing, and many said they could not do it. SU used Amazon gift cards as incentives, plus other incentives if students came back and finished test. But unfortunately they encountered another problem. There is a significant amount of transferring among SU’s schools and colleges, making it difficult to maintain a cohort. It is possible that there will not be sufficient students left in the cohort to complete the study. The study was also extremely resource intensive. So far, the assessment has required 362 staff hours (recruiting students, proctoring, etc.). Direct expenses have exceeded $29,000, not including the cost of participation. Administration of the test also required closing computer labs, which are supposed to be open to students at all times, for 89 hours.

Outcome

The CLA is designed to provide a summative assessment of the value-added components of the school’s instructional programs. At this point in time, the schools have data about their first-year students and rising juniors. Their seniors will take the test in the spring.
The CLA asserts that the test is one of multiple assessment tools that colleges and universities should use to assess the value-added component of their educational programs. Both presenters concluded that the findings are not helpful in identifying areas needed for improvement. Hurd noted that only two or three people at her institution reviewed the report. Because the data were not useful, additional research was not done, and the report was not disseminated more widely.
Beil’s team’s review and analysis looked at students’ performance by major, by which courses they took, their grades, and by whether a majority of the courses were taught by full-time or part-time faculty. For those seniors who took the test as a baseline measure, they looked at scores in terms of students’ GPA, whether or not they graduated with honors, major, and time spent taking the test. Only time spent on the test had an impact on students’ scores: Students who spent less time taking the test earned lower scores than those who spent more time on it.

Discussion:

Determining What CLA is Measuring

One issue that arose was the accuracy of students’ scores on the writing components. Only 20% of the exams are scored by people; most are scored by computers. Concern was raised about the level of sophistication of the programs using artificial intelligence.

Comparison Groups

Another issue was the importance of the comparison groups. Someone from the U. of Texas system noted that the “value added” scores at UT-Austin were much smaller than those at other Texas universities, which had a negative impact on how UT-Austin was viewed by the state education officials. Texas administrators did not take into consideration that the freshmen scores at UW-Austin were much higher than those at other schools, making it more difficult for UT-Austin students to advance as seniors.
Both Beil and Hurd thought that the scores were only as useful as the group of comparison institutions. If none of one’s comparison schools is included in the testing, it is difficult to know what the scores mean. They suggested that if institutions use CLA, they should make sure to have a few similar institutions with which to compare their results.

Importance of Faculty Assessments for Students

Another important point raised is that students are being assessed constantly. The form of assessment they care most about is that which is from the instructor, but somehow an instructor’s assessment gets eliminated from institutional assessment. Institutions could spend more time training faculty to assess their students. Assessment designed within the institution may be a more accurate form of assessment.

Recommendations:

• Schools should be aware of CLA’s limitations before relying on it for institutional assessment and priority-setting.

References/Resources:

Websites

1. Collegiate Learning Assessment (Council for Aid to Education’s website): http://www.cae.org/content/pro_collegiate.htm.