Table of Contents

1.0  Introduction

1.1      Course structure  p. 1

            1.2   Course objectives      p. 1
            1.3   Learning objectives      p. 2
            1.4   Assessment tools             p. 2

2.0  Comparison of entrance/exit tests

2.1      Methods  p. 3

2.2      Results  p. 3
Table 1. Results of the entrance and exit tests, broken down by topic p. 4
Figure 1. Entrance and exit test results, by topic p. 5
Figure 2. Frequency histogram of the changes from entrance to exit test p. 6

2.3      Discussion  p. 7

3.0    Student evaluations

3.1    Methods .  p. 7

3.2    Results .  p. 8
Table 2. Relative rankings of course components.. p. 9

3.3    Discussion  p. 9
Table 3. Average student course evaluation scores, Clare and Bio. 101 p. 10

4.0    Conclusions and recommendations  p. 11

       Acknowledgements ...  p. 12....................... ....................... ....................... .....................

Inquiry in the Natural World

Spring 2000

Course Assessment

1.0  Introduction

            This report presents an assessment of the classroom component of Clare 102, Inquiry in the Natural World, as taught in Spring semester of 2000.  The laboratory component will be assessed separately.  The course was team-taught by the following individuals:

                        * Lecture coordinator                                     ** Lab coordinator

1.1   Course structure:

1.11  Class (lecture) component:  3 credit hours.  The class met three times each week, 50 minutes per meeting.  There were 13 topics, each covered in one week.  The first class meeting for each topic occurred as a single, large lecture for all 5 sections.  Various instructors presented these lectures (see list of topics and presenters, Appendix 1).  The second class meeting used a variety of formats, including cooperative group exercises, to review the material covered in the large lecture and the assigned textbook reading.  The third class was dedicated primarily to student discussion of assigned readings.  There were 4 tests given during the semester; all sections of the course used identical tests.

1.12  Laboratory component:  1 credit hour.  Laboratory groups meet once a week for 110 minutes.  Laboratory exercises were different than other laboratory courses because the laboratory procedures were rarely be specified.  Instead, groups of either two or four students were given a problem, and expected to figure out how to solve it.  The laboratory experience was designed to be one of discovery rather than recapitulating worn out "follow the instructions" labs.  Students were asked to proceed in the same way that scientists make new discoveries, by making observations, forming, testing, and evaluating hypotheses.  Grades were based on in-lab evaluations (65%), a written paper based on a student-initiated investigation (25%) and class participation (10%).

Syllabi for the class and laboratory parts of the course are in Appendix 2.

1.2  Course objectives

            The objectives approved by the Faculty Senate for Clare 102 (numbered for clarity) are:

1. to introduce the mode of inquiry of the natural sciences
   
   
2. to enable students to understand and apply basic investigatory skills in a problem solving context
3. to examine a sample of fundamental discoveries of the natural sciences

The laboratory component of the course has its own objectives:

1.      Students will understand that science is an objective process that allows them to distinguish between adequate and inadequate explanations of natural phenomena and that scientific inquiry can be applied to everyday phenomena that are often taken for granted.

2.      Students will be able to perform a complete scientific investigation of an appropriate, testable, and measurable phenomenon of interest.

3.      Students will be able to apply the appropriate statistical test to a data set and correctly interpret the result.

1.3    Student learning objectives

            Specific learning objectives for each week’s topics were given to the students.  Class exercises and tests were based directly on these objectives.  The learning objectives for Spring 2000 are listed in Appendix 3.  When analyzed from the perspective of the three course objectives, 53% of the learning objectives related most closely to the first objective (to introduce the mode of inquiry of the natural sciences), 1% to the second (to enable students to understand and apply basic investigatory skills in a problem solving context), and 46% to the third (to examine a sample of fundamental discoveries of the natural sciences). 

            How well did the tests reflect the course objectives?  Fifty-two% of the points on the semester’s tests related most closely to the first objective, 4% to the second, and 44 to the third.  The distribution of test questions closely matched the learning objectives, and supported objectives 1 and 3 to approximately equal degrees.  Objective 2 was not well represented in learning objectives or test questions, but this is because it was intended to be satisfied primarily by means of the laboratory component of the course.

1.4   Assessment tools

            Two formal assessment tools were used:

(1)   Matched entrance/exit tests were administered to assess both the level of understanding of students beginning the course and the extent to which student understanding was improved by the course.  The questions were designed by the instructors to reflect general knowledge of science as a process and as a body of historical results.  A definite attempt was made not to tailor test questions to the specific topics and ideas which are covered in the course, but rather to test fundamental scientific knowledge that we might hope any educated person would possess.  Copies of the entrance and exit tests are in Appendix 4.

(2)   A detailed student evaluation instrument was used at the end of the semester to assess levels of student satisfaction and to what extend students believed that the course objectives had been met.  The instrument is shown in Appendix 4.

In addition to these tools, both the class and laboratory teaching teams met weekly during the semester to evaluate the success of the previous week’s exercises and to plan for the following week.  These team meetings gave instructors a chance to discuss student reactions to the course and helped give us a sense of the effectiveness of each course component.

2.0  Comparison of entrance/exit tests

2.1   Methods:

            The entrance test was given during the first week of classes and students were told that their scores would not affect their class grades.  Perhaps as a result, quite a few students left answers blank.  The exit test used identical questions and was administered for credit as part of the final examination.  Students were not told that that the questions were from the entrance test or that they would be used for course assessment as well as for grading students.  Two of the 25 questions from the entrance test (Multiple choice, # 8 and 12) were not used on the exit test because they were judged by the teaching team to include material that hadn’t been covered in the course.  The tests did not cover the 13 topics in the course at all evenly:  three topics (#6, 11 and 12) were not tested at all and three others (#4, 8 and 9) were tested by only one question.

            A few students missed either the entrance or exit test (<10 in each case); their tests were not included in this analysis.  Results of all the other tests (n=128) were entered into a spreadsheet, with correct answers coded as “1”, wrong answers as “0” and answers left blank by a blank.  It’s not clear how to handle answers left blank.  It seems reasonable to assume that students left blank the answers they were least sure of, so that blanks should be treated as wrong answers.  This approach would probably underestimate student understanding at the beginning of the course and exaggerate the degree of improvement caused by the course.  Blanks were treated both ways in this analysis and the results (presented below) indicate that the difference was small.

2.2  Results:

            Average scores for each question and for each topic are shown in Table 1.  The same results are shown graphically in Figure 1.  Student performance on the entry test was quite poor.  When blanks were counted as wrong, the overall score was 35.3%, ranging from a low of 7% on the questions about biology and evolution (topics #8 and 9) to a high of 56.0% for the scientific method (topic 1).  Not counting blanks increased the entry scores to 40.7%, still an amazingly low figure for such general science knowledge.  Scores on the exit test improved dramatically, to 66.6% when blanks were counted and the same when they weren’t.  The degree of improvement was somewhat higher (31.3%) when blanks were counted than when they weren’t (26.3%).

            A frequency histogram of improvement by individual students is shown in Figure 2.  As can be seen, most students improved by a substantial amount; only 8 of 128 showed an improvement of less than 10%.  It is equally true that few students (15) showed very high improvements (> 50%), presumably because the better students scored so well on the entrance test that high levels of improvement were impossible.  The overall improvement averaged 31.3%, with a standard deviation of 15.54.  Improvement was highly significant statistically:  the chance that the level of improvement observed could happen by chance, without a significant effect from taking the course, is 1.0 x 10^-46 (one-tailed t-test, df=127).

Table 1.  Results of the entrance and exit tests, broken down by topic.  TF = true/false, MC = multiple choice questions.  Refer to Appendix 4 for the actual questions.

Figure 1.  Entrance and exit test results, by topic.  Topics 6, 11 and 12 were not tested.  Topics 4, 8 and 9 were tested with only one question, Topics 1-3, 7 with 2 questions, Topic 13 with 3 questions, Topic 5 with 4 questions, and Topic 10 with 5 questions.

Figure 2.  Frequency histogram of the changes from entrance to exit test for individual students.    The columns are coded as follows:

                        Column 1  = Change       < -10%       Column   6  = Change  30 to 40%    

                        Column 2  = Change  -10 to 0%        Column   7  = Change  40 to 50%    

                        Column 3  = Change     0 to 10%      Column   8  = Change  50 to 60%    

                        Column 4  = Change   10 to 20%      Column   9  = Change  60 to 70%    

                        Column 5  = Change   20 to 30%      Column 10  = Change  70 to 80%    

2.3  Discussion:

            Although the test results show a high level of improvement, there are some limitations that should be noted:

(1)    The assessment tests are not independent of the course.  To some extent they test the topics and even the specific facts covered in Clare 102.  This possible bias was worsened by the decision of the teaching team to eliminate two questions from the exit test that addressed material that hadn’t been covered in class.  A more valid assessment could be attained if Clare 102 students were given standardized entrance/exit tests from some external source such as a national testing agency.  This approach would also allow us to compare our students’ exit performance with external benchmarks. 

(2)    The test questions do not adequately test the first two course objectives.  Of the 25 questions used in the entrance test, only two focus primarily on the process of inquiry in the sciences; two others test the history of science, and none test students’ ability to apply investigatory skills.  It is simply easier to write content-oriented test questions.  Also, the results for topics #1, 8 and 9 are based on only one question and cannot be considered adequate assessments of how much students learned about these topics.  In general, more questions are needed for each topic.  Also, the use of essays should be considered.  Most of the methodological or historical concepts in the course were tested during the semester with essays rather than objective questions.   However, grading essays would add substantially to the effort required for course assessment.

(3)    The exit results, while showing definite improvement, could be higher.  It would be useful to compare them to results from other general science courses or even other Clare courses.   The teaching team is continuing to discuss which topics should be included in the course and the degree of detail in each topic. 

3.0  Student course evaluations

3.1   Methods:

            A course evaluation was given to students during the final exam period.  The evaluation uses the standard system of ratings on a scale of 1-5, with 1 = strongly disagree, 3 = neutral and 5 = strongly agree.  The evaluation instrument (copy in Appendix 5) is modeled closely after the one used in the Biology Department.  We did add three items related to Clare 102’s objectives:

            #  3.  I have a better understanding of the scientific process after taking this course.
            #12.  This course helped me understand how scientists answer questions about the natural
                        world.

            #13.  This course introduced me to many of the fundamental discoveries of the natural
            sciences.

We were concerned that students may not respond carefully to each item in course evaluations, tending instead to give high scores to all items if pleased with the course and low scores if not.  To offset this possibility, we added a section (“Aids to Learning”) that asked students to rank the various components of the course rather than rate each component separately (as for items 6-11 under the Course Evaluation section).  This way, we reasoned, we would receive a relative evaluation of the parts of the course, whether a given student liked or disliked the overall course.  The scheme worked reasonably well but despite careful directions, about one-third of the students rated the items in items #23-25 rather than ranking them.  These responses were not included in the statistical analysis.

            Each course instructor tallied their own evaluations and turned in means and sample numbers for the Course Evaluation and Aids to Learning items (#1-14 and #23-25, respectively).  Responses to the Instructor Evaluation items (#15-22) and written comments (#26-30) were kept by individual instructors and not tabulated.  Grand averages for each item were calculated by multiplying the average for each section by the number of evaluations, totaling these products, and dividing the total by the number of evaluations for all sections combined.

3.2  Results:

            The average evaluation scores for each section, and the grand averages for all sections combined, are given in Appendix 7.  Overall ratings of the course fell in the mid-twos:  students rated their personal interest in the course material (item 2) at 2.58, the fairness of the grading (item 5) at 2.83, and whether they would recommend the course to other students (item 14) at a low 2.12.  In terms of how they believed Clare 102 met its course objectives, students were somewhat in agreement (ave. = 3.72) with the statement (#1) that the topics covered in class fit the course description and about neutral as to whether they have a better understanding of the scientific process after taking the course (item #3, ave. = 3.19) and whether the course helped them understand how scientists answer questions about the natural world (item #12, ave. = 3.16).  Students rated their coverage of the content of science somewhat higher:  they gave a rating of 3.41 to the statement (#14) that the course introduced them to many of the fundamental discoveries of the natural sciences. 

            Some clear differences emerged in terms of which components of the course students believed to have helped them learn.  Items #6-11 in the Course Evaluation section asked students to rate the large lectures, class discussions, active learning exercises, textbook, assigned readings, and laboratories as aids to understanding the course material.  Items #23-25 asked them to rank the large lectures, textbook, assigned readings, classroom presentations/discussions, active learning exercises, and review sessions in terms of their ability to grasp basic scientific facts (item #23), understand science as a human activity (#24), and prepare for tests (#25).  The two sets of questions overlap except for the items on laboratories (#12) and review sessions (#23-25).  Accordingly, I have ranked student evaluations of the remaining 5 course components in Table 2.  Students were remarkably consistent in ranking class discussions as the most helpful course component (after review sessions) in both sections of the evaluation form.  Active learning exercises were consistently ranked next highest.  This result is in considerable contrast to the opinions of the course instructors, who generally found these parts of the course the least effective.

Table 2.  Relative rankings of course components, based on student course evaluations.       1 = lowest rank, 6 = highest.

            The results for large lecture, textbook, and the assigned readings showed more variability (Table 2).  Overall rankings, based on items #6, 9 and 10, placed the textbook next highest, the readings next, and the large lectures the least useful.  Students showed some discrimination, however, when responding to items #23-25.  Item # 23 asked students which components were most helpful for learning the basic scientific facts:  here they ranked the large lectures above the textbook, and the readings last.  When asked about which helped them understand science as a human activity, students ranked the large lecture and readings the same, and place the textbook last.  Finally, for help in preparing for tests, students ranked the textbook the highest of the three, the readings lower, and the large lecture last.

            By far the strongest fact which emerged from the student evaluations was the high value awarded to review sessions as aids to learning.  This is interesting, as the review sessions were voluntary activities carried out by the instructors but not part of the planned structure of the course.  It may reflect a regrettable focus of students on last-minute “cramming” rather than diligent completion of weekly assignments.

3.3  Discussion:

            Students show moderate levels of agreement (3.16 to 3.41) with statements that involve the fulfillment of course objectives.  They agree overall (3.72) that the topics covered in the course meet the course description.  Active modes of learning (class discussions and active learning exercises) are ranked about pedagogies such as large lectures and assigned readings that are more traditional in science courses.  These findings validate, to some extent, the decision to emphasize active student learning in the course.  The modes of learning that the students found most helpful are, however, viewed by the instructors as the least effective means of communicating the course material.  The evaluations reflect student attitudes but do not necessarily tell us which components of the course actually produced positive learning.  The discrepancy between student and faculty assessment of pedagogy needs to be explored and understood if learning outcomes in the course are to be improved.

            Overall, students did not like the course.  This observation is based not only on formal evaluations but on many conversations with students during the semester.  The evaluation instrument used doesn’t shed much light on why the course was disliked:  none of the 13 items under Course Evaluation had as low a rating as #14 (“I would recommend this course to other students”).   In the future Clare 102 instructors should pursue more “nuanced” ways of assessing student reactions to the course.

            Two considerations may balance the low overall student evaluation of the course.  First, the course was dramatically successful at improving students’ grasp of science.  It worked!   Second, non-science majors generally do not enjoy science courses.  To put the Clare 102 evaluations in perspective, they are compared in Table 3 with comparable questions from student evaluations of selected sections of Biology 101 here at St. Bonaventure over the past 10 years.  Responses to the first two items, which involve suitability of topics and tests, are not statistically different for the two courses (2-tailed t-test, P = 0.20 and 0.32).   Students rated Clare 102 significantly lower on the next two items, however:  fairness of grading (2-tailed t-test, P = 0.010) and usefulness of readings (2-tailed t-test, P = 0.011).  Finally, the overall rating of Clare 102 (“I would recommend this course to other students”) was very much lower than for Biology 101 (2-tailed t-test, P = 0.001). 

Table 3.  Average student course evaluation scores for the 5 sections of Clare 102, Spring 2000 compared with 5 selected sections of Biology 101.  1 = strongly disagree, 3 = neutral, 5 = strongly agree.

                                                         ------  Clare 102 ------    difference   --------  Bio. 101 -------

 Significance of the observed difference, as the probability of their occurring when the null hypothesis of no difference between the two courses is true:  n.s. not significant (P > 0.05), * P < 0.05, ** P < 0.01.  Based on t-tests, unequal variances.

4.0    Conclusions and recommendations

            The following conclusions and recommendations are made by me, without necessarily being supported by the other Clare 102 instructors:

1. Clare 102 produced successful learning outcomes for the vast majority of the students, particularly with regard to its third objective, “to examine a sample of fundamental discoveries of the natural sciences”.   The average score on the exit test was nearly double that of the entrance test.  Nearly 94% of students who completed the course improved by at least 10% in test performance, and over three-quarters (76.6%) improved by more than 20%.
   
   
2. Fulfillment of the first course objective, “to introduce the mode of inquiry of the natural sciences”, was not as fully assessed, but the available evidence suggests that it was successfully accomplished.  Anecdotal observations by the instructors and student essays during the semester indicate that many students gained a sense of the historical and philosophical complexities of scientific investigations, along with a heightened appreciation of the flaws and foibles of some of the most prominent scientists of the past few centuries.
   
   
3. The second objective, “to enable students to understand and apply basic investigatory skills in a problem solving context”, was primarily addressed in the laboratory component of the course and was not examined by the assessment tools reported here.
   
   
4. It is clear that the assessment tools used this past semester can be improved.  In particular, I recommend that future entrance/exit tests be focused more on science as a mode of inquiry, and perhaps use essay as well as objective questions.  It may also be useful to administer a standardized, external test to a subsample of Clare 102 students, especially if a test can be found that addresses the three course objectives equally.
   
   
5. While the course seems to have increased student understanding of science, it did not necessarily improve their appreciation of science, either as a valuable human activity in its own right or as a crucial component of contemporary and future societies.  While these attitudes of students are not an explicit part of the course objectives adopted by the Faculty Senate, it still may be worthwhile for Clare 102 instructors to think about how the course might be changed to improve student attitudes toward the natural sciences.

Acknowledgments

            I am very grateful to Patsy O’Brien, the building secretary for DeLaRoche Hall, for doing the tedious work of coding all the test results.   Thanks to the other Clare 102 class instructors for providing tabulated results of student evaluations for their sections.   This report benefited from the comments and suggestions of others, but the analyses and opinions are ultimately my responsibility.