Outcomes Assessment Report, March 20, 2006

Outcomes

Assessment

Reports

a newsletter and Web log

by Frank Vozzo

Director of Student Learning Outcomes Assessment

March 20, 2006

In this report:

Program Coordinators, course sequencing, and the learning continuum.
Validity and reliability.
Assessment of online learning.

The Sage Undergraduate Program Coordinators, the curriculum leaders within departments (see CampusCruiser/Campus Life/Committees/TSC Faculty Staff and Administrators/Shared Files/Academic Information/Job Description - Coordinators & Directors - Responsibilities 3 09-03.doc), are expected to be familiar with the ways in which the courses of their majors build on one another. But across departments, and even within many departments, it is difficult to obtain useful information about courses you do not teach.

Meaningful assessment of student learning outcomes in a program should include faculty satisfaction with course sequencing. Even in programs where the courses are not structured in a definite hierarchy (100-level courses for first year students, 200-level for second year, etc.) or where learning is not expected to be linear and sequential, there are expected course sequences (made necessary by the scarcity of course offerings, unfortunately); so as an individual instructor, you can anticipate the courses that a student takes immediately before and after your own course. Unless you sit down and have a free and open discussion with the instructors in these other courses, you can’t know how well or poorly your course fits into the continuum of a student’s experiences. The sharing of “war stories” with your faculty colleagues, followed by the adjustment of sequencing*, content or emphasis in two or three connected courses, might make a big difference in the overall student learning of your program and of general education. The discussion and collective action has to happen with the student in mind; sometimes it becomes necessary to put aside differences in style and teaching philosophy to find common ground. You also have to get past anecdotal information and find ways to measure the deficiencies that you perceive.

Faculty members need assistance in how to collaborate toward a more effective curriculum. At your next department meeting or retreat, reserve some time to discuss this. I would be happy to assist with this exercise.

*Don’t forget to assess the results of course prerequisite changes. Many such changes were made last year to facilitate online course registration. Were any of these changes unwise? How can you find out?

I found some good definitions for the terms validity and reliability in a pedagogical paper by the International Baccalaureate Organization:

“According to the standard definition, the validity of an assessment is the extent to which it actually measures what it is stated to measure. The term reliability is used to define the accuracy of measurement resulting from an assessment, and how likely it is that the same result would be produced in slightly different circumstances. An assessment is reliable if a student would gain the same result were he/she to repeat the assessment on different occasions, and also give the same result if the assessment were marked by different markers. Validity and reliability are widely regarded as essential characteristics of any assessment system, particularly a high-stakes one where the outcome is of great importance to the student or the teacher.”

Grading can be considered as assessment, provided that the grading is reliable (can be arrived at in an objective way, such as through Primary Trait Analysis). A standardized test can be considered as a valid assessment instrument, provided that the answers it accumulates are those which the experts would agree are indicative of deep learning having taken place (as opposed to just memorization and regurgitation).

Rather than assigning letter grades to particular point spreads in an arbitrary way, consider assigning grades to your tests with reliability, according to a rubric which defines your expectations clearly to yourself and your students. Consider constructing some of your quizzes and tests such that they contain questions that assess deep learning in a valid way, and collect that assessment information separately from your list of grades. This will help to inform your future teaching on those particular course items.

According to the Fall 2005 Newsletter of the Middle States Association, “Approximately 18% of the 3,103,128 students in the Middle States region are enrolled in distance learning courses”. Many undergraduate courses at Sage already use online course materials, and many more will use them in the future (in adjusting to the new block schedule, I plan to use Blackboard to build more homework into all of my courses in the coming academic year). Thus far relatively few courses use online delivery to partially or totally replace “seat time.” The number of online courses will increase as programs develop for Sage After Work. For the next four sessions at SAW (summer I and II, fall and spring), there are 25 identified courses that will be delivered partially or totally online. Sage has appointed a faculty member, Carol DiMambro, to oversee “Sage Online.”

Sage is undergoing a distance learning Institutional Capability Review with the New York State Education Department. One important NYSED criterion for online courses and programs is that they “produce the same learning outcomes as comparable classroom-based programs”.

Middle States Association, working within a group of eight regional accrediting bodies, published this set of guidelines for programs that rely partially or totally on online learning:

As with all curriculum development and review, the institution assures that each program of study results in collegiate level learning outcomes appropriate to the rigor and breadth of the degree or certificate awarded by the institution, that the electronically offered degree or certificate program is coherent and complete, and that such programs leading to undergraduate degrees include general education requirements.
Academically qualified persons participate fully in the decisions concerning program curricula and program oversight. It is recognized that traditional faculty roles may be unbundled and/or supplemented as electronically offered programs are developed and presented, but the substance of the program, including its presentation, management, and assessment are the responsibility of people with appropriate academic qualifications.
The importance of appropriate interaction (synchronous or asynchronous) between instructor and students and among students is reflected in the design of the program and its courses, and in the technical facilities and services provided.
As a component of the institution’s overall assessment activities, documented assessment of student achievement is conducted in each course and at the completion of the program, by comparing student performance to the intended learning outcomes.
When examinations are employed (paper, online, demonstrations of competency, etc.), they take place in circumstances that include firm student identification. The institution otherwise seeks to assure the integrity of student work.
Documented procedures assure that security of personal information is protected in the conduct of assessments and evaluations and in the dissemination of results.
Overall program effectiveness is determined by such measures as:

The extent to which student learning matches intended outcomes, including for degree programs both the goals of general education and the objectives of the major.
The extent to which student intent is met.
Student retention rates, including variations over time.
Student satisfaction, as measured by regular surveys.
Faculty satisfaction, as measured by regular surveys and by formal and informal peer review processes.
The extent to which access is provided to students not previously served.
Measures of the extent to which library and learning resources are used appropriately by the program’s students.
Measures of student competence in fundamental skills such as communication, comprehension, and analysis.
Cost effectiveness of the program to its students, as compared to campus-based alternatives.

In addition to these considerations, Middle States requires that distance learning students have equal access to services and involvement in the academic community of the institution.

One of the best survey-based instruments I have come across was produced as part of an online learning research project at the University of British Columbia. The questions allow for side-by-side comparison of course delivery formats.

In next week's OARs: tracking student usage of Blackboard course materials.

Home...