Center for Assessment & Improvement of Learning

User Experiences

Sam Houston State University

Quality Enhancement Plan

In 2009, a team of faculty at Sam Houston State University (SHSU) developed a new course called Foundations of Science as part of its Quality Enhancement Plan (QEP) for reaccreditation by the Southern Association of Colleges and Schools. The primary goal of the course was to improve the scientific literacy and critical thinking of our students. We defined critical thinking as the process of drawing reasonable, fair-minded conclusions based on evidence and logic. This definition matches the process of scientific reasoning, wherein empirical evidence is evaluated in an objective manner.

As part of the reaccreditation process, we were required to evaluate our students’ gains in critical thinking resulting from having taken the course. We considered several possible instruments designed to assess critical thinking and chose the Critical Thinking Assessment Test (CAT) because the nature of the questions on the CAT fit perfectly with our working definition of critical thinking. Specifically, the test requires students to examine data and information, develop hypotheses, propose a means of testing those hypotheses, and draw reasonable conclusions. It also tests students’ ability to recognize one of the most common fallacies that scientists (and people in general) must avoid; namely, the False Cause fallacy. One of the many strong points of the instrument is that it does not require specific knowledge of a scientific discipline or set of facts; rather, it assesses the scientific/logical reasoning process of students; consequently, it can be used by students in any discipline.

We have administered the CAT almost every semester since 2009, both as a pre-test and a post-test, in order to gage student gains in critical thinking skills. We hold a formal scoring session at the end of each semester and invite faculty members from across campus to participate in scoring the tests. Our results in all semesters have shown a statistically significant increase in critical thinking.

The CAT instrument has also served to highlight the importance of reading and written communication skills. Indeed, scorers frequently comment on the weaknesses in these areas which are highlighted in the responses to the test. Moreover, many faculty members have left the scoring sessions with a desire to incorporate CAT-like questions into their coursework in order to promote critical thinking.

The use of the CAT has been so successful at SHSU that we plan to continue using it for the Foundations of Science course, and also to assess the gains in critical thinking of students as a result of their overall education at SHSU. Specifically, it will be given to students in their junior/senior year and the results will be compared to those from other universities. The ability to compare results with those from other universities across the U.S. is one of the benefits of using the CAT instrument.

Marcus Gillespie, Associate Dean, GEO_BMG@SHSU.EDU

Sam Houston State University

Assessment of Learning Outcomes

We are using the CAT as an end-of-experience measure, in order to capture data on how well are students are doing with critical thinking and problem solving as they prepare to graduate. To this end, we are administering the CAT in 3000- and 4000-level courses from each of our academic colleges over a three-year cycle. Each year, we are anticipate collecting approximately 500 completed CAT tests for scoring by a cross-discipline group of faculty scorers. These data will be used to inform our core assessment efforts; however, results will also be provided to the participating colleges and departments for use in programmatic assessment. Over time, it is hoped that these data will help inform curricular and pedagogical changes to help improve critical thinking for our students. We are particularly excited to dive deeper into the data we get back from y’all about our student performance. Although the various institutional and college averages have their role, we really want to start looking at that data by various student demographic characteristics, such as gender, ethnicity, socio-economic class, etc. Over time, we are also looking to align the data we get from the CAT with other critical thinking measures we have in place on campus to better inform both our CAT results, and the results from those locally developed measures.

Jeff Roberts, Director of Assessment, jlr022@SHSU.EDU

Florida State University

Florida State University has adopted the Critical Thinking Assessment Test as the assessment instrument for its Quality Enhancement Plant Think FSU. The decision was based on several factors including proven validity and reliability and widespread usage at similar type institutions. Our choice has proven to a solid one. In addition to the characteristics mentioned above, we have been extremely pleased with the broad support we receive from the CAT team.

Early in our implementation, we visited with the team to discuss the test and to sketch out an assessment plan. This experience was typical of our interactions with the CAT group. They worked to understand our objectives and, based upon that information, provided advice on how to make the most of our assessments. These conversations included recommendations ranging from sampling strategies to nuancing the script in order to get authentic student participation. This individualized attention is not limited to FSU. At other training events, colleagues from around the country have commented on the outstanding level of support they receive from the CAT group.

An added feature of our involvement with the CAT group has been the attention paid to methods for enhancing critical thinking skills among our students. Assessment is important, but the ultimate goal is improved skills. As part of our faculty development efforts, we brought the CAT folks to FSU for a session on constructing assignments aimed at boosting critical thinking. Participants continue to talk about the lessons learned as part of the sessions. There is little doubt that these sessions resulted in changes to the classroom experience around campus.

Because of the statistical strength of the CAT and the comprehensive support offered, our decision to utilize the CAT has proven to be one of the smartest we have made to date.

Lynn Hogan, Director, Critical Thinking Initiatives, lhogan@fsu.edu

Laurie Molina, Associate Director, Critical Thinking Initiatives, lmolina@fsu.edu

Keene State College

Keene State College uses the Critical thinking Assessment Test to assess critical thinking as one of our college-wide learning outcomes. We chose CAT over other options because we have local control over how we use the test. We chose to administer it as a pre- and posttest in selected first-year and upper-level courses across the curriculum. We also chose CAT because scoring the tests locally provides important faculty development opportunities. We believe that over time using CAT will improve the teaching of critical thinking skills on our campus. The follow-up rescoring and data analysis by Tennessee Tech ensures that our scores are accurate and comparable to other institutions.

Cathryn Turrentine, Director of Institutional Research and Assessment, cturrentine@keene.edu

Successful Projects

City University of New York (CUNY)

NSF TUES Project #0942790. CREATE Cornerstone--Inspiring undergraduates to persist and succeed in the Biology major. PI: Dr. Sally Hoskins (shoskins@ccny.cuny.edu) Related Website: www.teachcreate.org

The CREATE Cornerstone course adapted the CREATE approach for first-year students with the goal of building transferable science learning skills and deepening students' understanding of the nature of science. Students gained in critical thinking and experimental design abilities, and also showed maturation of epistemological beliefs about science (Gottesman and Hoskins, 2013, http://www.lifescied.org/content/12/1/59.full).

Clemson University

NSF TUES (CCLI) Project #0837540. Development of an Inquiry-Based Cell Biology Laboratory with Emphasis on Scientific Communication Skills. PI: Dr. Lesly Temesvari (ltemesv@clemson.edu) or Dr. Terri Bruce (terri@clemson.edu).

This project involved the development of a new cell biology laboratory course that emphasized critical thinking, effective writing and communication, and ethical reasoning. The new course used an inquiry-based pedagogic strategy allowing students to design and perform experiments in the context of mini research projects. Students also gained experience in communicating their findings through poster/oral presentations and through the writing of manuscripts in standard journal format. As a part of the scientific inquiry and communication processes, students also engaged in the discussion of the ethics of scientific communication.

Duquesne University

NSF TUES (CCLI) Project #717685. A Model for Incorporating Application-Based Service Learning in the Undergraduate Science Curriculum. Dr. Nancy Trun (PI) (trun@duq.edu), Dr. Lisa Ludvico & Dr. Becky Morrow (Co-PIs). http://www.scienceresearch.duq.edu/bio/biofac/ntrun/ABSL/index.html

Application Based Service Learning (ABSL) is a pedagogy that we are developing to address the need for novel approaches to Science, Technology, Engineering and Math (STEM) education at the undergraduate level. ABSL combines traditional service learning with novel undergraduate research on a community problem. For the service-learning portion of the class, students spend a set number of hours throughout the semester in a specific community environment so that they learn about and understand the community problem. In class, the students conduct novel research, using the scientific method, on various parts of the community problem and investigate solutions to the problem.

Jamestown Community College

NSF ATEP Project #1204209. HURI SURI Helping Future Biotechnologists in Rural Appalachia HURI-up with Undergraduate Research. Jackie Crisman (PI) (Jackie.Crisman@mail.sunyjcc.edu).

HURI SURI at Jamestown Community College prepares a cohort of high school biotechnology instructors with authentic research experiences. These instructors teach a new interdisciplinary first semester sub-cellular biology course at their high schools which is infused with these research experiences. High school students taking this course also have the opportunity to do summer research at JCC. The project also provides JCC biotech undergraduates with a greater array of biotechnology research experiences on site, thus expanding the scope and depth of their authentic learning. In addition to on site research, JCC biotechnology students also participate in summer research internships at regional universities.

Northeastern Illinois University

NSF TUES (CCLI) Project #1140126. A Hybrid Course Model of Peer-Led Learning for the Social Sciences. Maureen Erber (PI) (maureen.erber@gmail.com). Linda Rueckert & Christopher Merchant (Co-PIs).

Peer-Led (PLTL) is a collaborative model in which a more experienced student (i.e., Peer Leader) guides a group of students through critical discussions of course concepts and facilitates problem-solving activities. Over the course of a semester, students participate in roughly 10 PLTL sessions and are exposed to a wide array of problems (e.g.,“Think like a shrink”) in which they engage in conversations on controversial topics or are asked to solve statistical problem sets or to detect problems in specific instances of media misuse of statistics. This community of learners engages in critical thinking activities in a welcoming and comfortable environment.

Purdue University

NSF TUES (CCLI) Project# 941921. An Adaptation of a Research-Based Laboratory Model to Life Sciences. Dr. Dennis Minchella (PI), Dr. Gabriela Weaver, Dr. Jason Curtis, and Dr. Stephanie Gardner (Co-PI's) (sgardne@purdue.edu).

The goal of this project is to introduce freshmen biology majors to the culture of scientific discovery by engaging them in authentic research projects as part of their introductory biology curriculum. This is an alternative to the cookbook/practical skills introductory laboratory because it not only provides the students with needed basic laboratory skills, but also enhances their scientific understanding and communication skills. Emphasis was placed on the following objectives: (1) Students will demonstrate an understanding of the nature of science, (2) Students will be able to critically evaluate information and design experiments to test hypotheses, (3) Students will analyze and interpret their results and make inferences, and (4) Students will develop and improve their scientific communication skills in both written and oral formats.

Rose-Hulman Institute of Technology

NSF TUES Project # 000260456. The Development and Assessment of a New Test and Product Engineering Certificate Program. Dr. Tina Hudson (PI) (hudson@rose-hulman.edu) , Dr. Chris Miller (Co-PI).

As part of this grant, RHIT teamed with industry to create a Mixed-Signal Test and Product Engineering course. In this course, students are required to use critical thinking skills to design tests that isolate the effect of multiple variables on the measurements of mixed-circuit integrated circuits and evaluate their measurements for sufficient accuracy while minimizing test time. The instructor models critical thinking skills early in the course and provides many opportunities for the students to practice them with problems of increasing difficulty on a system that provides immediate feedback.

Sam Houston State University

A multidisciplinary general education course, Foundations of Science, was developed to improve students'·critical thinking·and scientific literacy. The course is taught collaboratively by faculty in Geography/Geology and Biology. Dr. Marcus Gillespie (GEO_BMG@SHSU.EDU) and Dr. Matthew Rowe (MPR002@SHSU.EDU).

In the course, students critically evaluate a diversity of extraordinary and engaging claims (sometimes controversial), ranging from astrology to alternative medicines to the lost continent of Atlantis to help them understand the relevance of science in their daily lives. Students work in groups to discuss various Case Studies (many designed specifically for this course). The course emphasizes the way scientists·think critically about information and ideas more than the facts of science. The course also incorporates discussion of common logical fallacies, and other types of reasoning/perceptual biases that can mislead us. Students are introduced to the importance of sample size, double-blind clinical studies, and the placebo effect during our discussions of alternative medicines and alleged paranormal phenomena. Throughout the course, we try to help students understand that they can use what they learn about science and critical thinking to help them make better decisions for themselves, for their families, and for society.

University of Central Florida

NSF-TUES CCLI Project #0941980. Learning Environment and Academic Research Network (LEARN): A Model for Retention in the STEM Disciplines. Dr. Kimberly Schneider (PI) (krs@ucf.edu) & Dr. Alison Morrison-Shetlar (CoPI). Related Website: www.our.ucf.edu/learn

LEARN™ was established in 2011, creating a small research community for 28 first-year STEM students. Learning Environment and Academic Research Network (LEARN) is a formal living-learning community with numerous benefits to the students: student and faculty mentors, 12-week research apprenticeships, specialized courses in students' resident halls, personalized academic advising, and scholarships. Studenst have demonstrated increases in critical thinking abilities. LEARN graduates have continued into research programs including, but not limited to: Stanford, National Institutes of Health (NIH), Massachusetts Institute of Technology (MIT), and Berkeley.

University of Wisconsin - Madison

NSF-TUES (CCLI) Project #0737352. Teaching nature of science·and scientific inquiry in the context of scientific paradigms:· Assessing student understanding. Dr. Basil Tikoff (PI) (basil@geology.wisc.edu)& Dr. Nancy Ruggeri.

Excerpt from Course Description: This course is about how scientists figure things out. To put it more formally, it is about how people make sense of the natural world in the past, understand the present, and make predictions for the future. An integral (and interesting) part of this process is the different methods scientists use, the ways they work as a community, and how that community interacts and reflects the larger society... We will ask you to reflect upon your own learning process, and we will try to foster this through a variety of short assignments throughout the semester. Our hope is for you to reveal to us- and to yourself- what you learn over the course of the semester.