Final Assessment Report for the 1997-99 Cycle

Department: Psychology

Assessment Coordinator: Linda Reinhardt, UW-Rock County

1) Course Information:

Course Title and Number: PSY 201/202, Introductory Psychology

Credits: PSY 201 – 4 credits; PSY 202 – 3 credits

Total number of instructors teaching the course: Almost all instructors in the department teach this course each semester

Total number of sections taught: approximately 25 per year

Number of sections assessed:

97-98: 3 98-99: 9 (11)*

Number of faculty/academic staff assessing this course:

97-98: 3 faculty 0 academic staff

98-99: 6(8)* faculty 0 academic staff

Total number of students participating in the assessment:

97-98: 100

98-99: 373*

  1. Breadth of knowledge outcomes:

1) Understand the basic concepts of experimental and correlational methods.

2) Understand the basic procedures for classical and operant conditioning.

3) Develop basic understanding of the major theoretical perspectives in psychology (cognitive, behavioral, biological, psychodynamic, and humanistic) and how these perspectives approach personality, psychological dysfunction, and psychotherapy

Assessment methods for breadth of knowledge outcomes:

1997-98

Department members wished to devise their own methods for assessing breadth of knowledge and proficiencies. Three faculty members submitted the results of their assessment to the DAC.

Two members developed multiple choice questions on unit exams and/or the final exam. Each instructor used several items (4-20) to assess each outcome. The instructors calculated the percentage of students who answered each item correctly, and then calculated an average of these percentages across the several items assessing each content outcome.

The DAC has combined those results by calculating a weighted grand mean for accuracy (weighted by size of instructor’s class, 24 and 54 students) for each content outcome, averaged across the several multiple choice items for each outcome.

A third department member developed essay questions on the final exam to assess students’ knowledge of the scientific methods used in psychology (Content 1), and understanding of major theoretical approaches (Content 3).

1998-99

Department members decided to use a standard set of 10 multiple choice questions they would administer at the end of the course (see item analysis attachment).

Results:

The weighted mean accuracy of students on the multiple choice items that assessed the content outcomes were:

1) Understand the basic concepts of experimental and correlational methods: 97-98: 65% 98-99: 64%

2) Understand the basic procedures for classical and operant conditioning: 97-98: 60% 98-99: 73%

3) Develop basic understanding of the major theoretical perspectives:

97-98: 72.0% 98-99: 64%

In interpreting these results it is important to note some of the differences in assessment between the two academic years that makes it difficult to make direct comparisons between the two sets of numbers:
    1. in 1997-98 instructors were using questions that they had designed for their courses, while in 1998-99 instructors were using a standard set of questions;
    2. in 1998-99 some of the instructors incorporated the questions into their final exam while others gave the questions to students unannounced on the last day of classes;

c) many more faculty and students participated in assessment in the 1998-99 year

In 1997-98, the average score on the 5-point essay questions were:

1) Research methods: 3.21 (2.17 the previous semester)

3) Theoretical approaches: 2.75 (2.57 the previous semester)

In 1998-99, students in four course sections (163 students) answered a set of True-False items in the last week of classes. The number of students who answered each of the items correctly was averaged 65% across the 15 items. As a point of comparison, students in two of these sections (87 students) had previously responded to the items in the first week of class. The average accuracy on the questions was 44% correct in the pretest. On eight of the 15 items, at least 20% more students answered the item correctly in the posttest.
  1. Proficiency outcomes:

Clear and logical thinking:

1) Analyze, synthesize, evaluate and interpret information and ideas.

Effective communication:

2) Read and listen with comprehension and critical perception.

3) Recognize fallacies and inconsistencies.

4) Develop a large and varied vocabulary.

Assessment methods for proficiency outcomes:

1997-98

Department members wished to devise their own methods for assessing breadth of knowledge and proficiencies. Three faculty members submitted the results of their assessment to the DAC. Several of the methods designed to assess content outcomes cross-cut assessment of proficiencies, but instructors also identified exercises or exam questions specifically designed to assess proficiencies. Not all of the instructors assessed all of the proficiencies explicitly.

Two of the instructors used essay questions on unit exams and/or the final exam to assess these proficiencies. One instructor also assessed the proficiencies with comprehension monitoring assignments based on functional measurement techniques of information integration theory, and some multiple choice items. The third instructor used an "extra credit" critical thinking exercise in which students looked for flaws in brief descriptions of research protocols.

1998-99

The set of 10 multiple choice questions covered three of the four proficiency outcomes (1, 2, and 4). As an additional option, some department members decided to use a standard set of true- false items that they would either administer at the end of the course, or use as a pretest-posttest for the course to assess proficiency outcome 3 (see attached item analysis).

Results:
  1. Analyze, synthesize, evaluate and interpret information and ideas

1997-98

a) Average accuracy on essay questions was: 76.5%

b) Average score on 5-point essay to apply concepts: 3.69 (2.61 the previous semester
    1. Critical thinking exercise on flaws in research methods: Twenty-four students (44% of class) submitted responses to optional assignment. Scores ranged from 1 to 10 points, with mean of 3.4

1998-99

All the multiple choice items used this year require this proficiency. The average number of students who answered each item correctly was 67%.
  1. Read and listen with comprehension and critical perception.

1997-98
    1. Average performance on comprehension monitoring assignments: 89%
    2. Average score on 5-point critical thinking essay

(also Proficiency 3): 3.85 (3.24 the previous semester)

1998-99

All the multiple choice items used this year require this proficiency. The average number of students who answered each item correctly was 67%.
  1. Recognize fallacies and inconsistencies

    2. 1997-98

    Average performance on multiple choice items: 66%

    1998-99

    Being able to answer the True-False questions accurately requires this proficiency. The number of students who answered each of the items correctly was averaged 65% across the 15 items.

  2. Develop a large and varied vocabulary.

1997-98

Average performance on matching and/or multiple choice items: 84%

1998-99

All the multiple choice items used this year require this proficiency. The average number of students who answered each item correctly was 67%

4) Department specific proficiency outcomes (if any): None

5) Changes enacted during the 1997-99 assessment cycle to improve student learning:

The following are some of the changes that department members have made to improve student learning of psychology:

  1. Made a concerted effort to formalize assessment of all of the breadth of knowledge and proficiency outcomes that the department has identified for Introductory Psychology. The department decided to use a standard set of assessment items in the second year of the cycle.
  2. Shared with department colleagues proven and/or promising exercises and supplemental instructional materials.
  3. Developed and shared additional classroom exercises and examples that specifically target those content areas and proficiencies where are students appear to be having difficulty.
  1. Evidence of changes in student learning based on improvements:

    2. The instructors have reported that they have gained considerable insight into their students’ learning by participating in the assessment process. The aggregate data necessary for this report does not capture the detail that individual instructors gleaned about the strengths and weaknesses of their students’ knowledge of psychology.

    The department changed our method of assessment from the first to the second year of this cycle to have more standardization across instructors in the assessment instruments we use. While we believe this standardization will make our assessment more informative, it makes it difficult to compare the extent to which student learning has improved from the first to the second years of this cycle.

  2. Further changes to be made to improve student learning based on 1998-99 data:

The department has not yet had an opportunity to review the result of the 1998-99 data as the data was not assembled until the end of the 1999 academic year. However, at our Fall 1999 meeting, we will discuss the assessment results and how to use them to improve student learning.

Among the topics we will discuss will be:

a) How to create better measures of our students’ breadth of knowledge and proficiency outcomes. Although we believe it was wise to move toward a standard set of assessment items, we are not satisfied that they measure students’ knowledge and proficiencies very well. We feel we need to put particular emphasis on assessing proficiencies.

b) How to conduct frequent informal classroom assessments (formative evaluation) prior to examinations (summative evaluation), so instructors can adjust their teaching to meet student needs, and provide students with feedback from the assessments that can help them develop during the course of the semester.

c) How to take better advantage of instructional technology (e.g., e-mail, electronic forums, Web pages) for the opportunities it provides to do some of the assessment and presentation of supplemental instruction and materials outside of the classroom.

8) Additional contributions to assessment such as publications, presentations, qualitative classroom innovations, other items relating to assessment that the department wishes to note: None

 

* As the DAC tabulated the data for the 1998-99 academic year, she did not have all the data that department members submitted during the year. A loss of email messages stored on the DAC’s computer required that she ask department colleagues to resend their reports. It was not possible to recover all the data—eight faculty members submitted data, but the data from two faculty members could not be recovered for this report.