Download the original document in its entirety.
Pilot of First-Year Design Course 1994-95 "Introduction to Engineering"
July 14, 1995
Leon Shohet and Denice Denton
and the Faculty:
The LEAD Center
LEAD Center Project Team:
The first-year design course evaluated in this report is a key element of an Advanced Research Projects Agency-Technology Reinvestment Program (ARPA-TRP) grant to the Engineering Research Center for Plasma-Aided Manufacturing (ERC for PAM). The principal investigator of this TRP ARPA grant is J. Leon Shohet, ECE Professor and Director of the ERC for PAM. The grant manager is Denice Denton, ECE Professor and Leader of the Education Thrust Area of the ERC for PAM. The Learning through Evaluation, Adaptation, and Dissemination (LEAD) Center is conducting a third-party evaluation of the course. This report, a product of that evaluation, focuses on student learning experiences as well as instructor course implementation processes during the pilot year, 1994-95. The ARPA-TRP grant and funding by Procter and Gamble financed summer 1994 faculty salaries for the course planning, five pentium-based personal computers and peripherals, stipends for the senior assistants, supplies for students' projects, and other costs associated with the course. The ARPA-TRP grant, matched by a Graduate School project assistantship, LEAD start-up support from the Chancellor, and a portion of the IBM Quality Award, supported the LEAD Center evaluation. Some of the professors were granted release time from their regular teaching schedule by their departments to teach this new course. Others simply participated on an overload basis. The departments of the seven faculty and the administration have been supportive in terms of providing space for lectures and laboratories and photo-copy services. Departments encouraged others including machinists and shop managers to be consultants to students. In summary, departments and administrative offices have supported the instructional effort. When resources to develop a first-year design course were made available through the ARPA-TRP grant, seven faculty who, through the College's pilot Teaching Improvement Program, had participated in a year-long study of effective teaching and learning strategies volunteered to develop and pilot this course. First presented as an elective titled, "Introduction to Engineering," eighty-two first-year students (out of a total pre-engineering class of 783) completed the pilot course during the 1994-1995 academic year. Students registered for the two-credit course under one of three titles: ECE 379, ME 602, or NEEP 602. Sixty-seven completed the course in the fall, while fifteen completed it during the spring. Seven labs or sections of up to 12 students each were taught in the fall and two labs of seven and eight students each in the spring. Students were both male and female. Females represented 30% of the fall class, a higher proportion than the 22% females in the fall 1994 entering pre-engineering cohort overall. Only one female participated in the spring. Other minorities represented less than five percent both semesters. The low enrollment during spring was likely caused by three factors: the course was difficult to find in the Course Timetable, was not on a list of courses that fit the engineering curriculum, and did not benefit directly from promotion during Summer Orientation and Registration (SOAR). The first year retention rates for students in the course were 96 and 67% respectively for the two semesters. These rates compare to a lower average first year retention rate for entering pre-engineering cohorts of approximately 75%. Of note, the retention rate for women who enrolled in the design course was high: 18 of the total 20 women finished the course and were still enrolled in the College as of summer 1995. Goals and Methods: An Emerging Philosophy and Approach Based on input from business, industry, colleagues, and students, the faculty articulated the objectives and goals first during their summer 1994 planning sessions. They designed the course around these goals and presented them to students in the fall semester's course notes. Based on their fall semester experience, faculty recognized the need to revise the goals during spring, 1995. After much discussion among themselves and with their colleagues, and following a number of formative feedback discussions with the third party evaluators from the LEAD Center, they articulated a philosophical framework for the course. This philosophy describes specific concepts that the faculty want students to learn. The concepts are grouped into "process" and "product" categories. The process concepts include design methodology, team interaction, communication, and confidence-building. The product concepts focus on engineering science topics and tools. A copy of the "Philosophy of `Introduction to Engineering'" appears in Tab 8. A comparison of the goals listed for first semester and second semester demonstrates how the faculty modified the course goals and objectives. Fall semester goals for the students as stated by the faculty in the student handbook are as follow:
By the end of the semester, one member of the faculty team expressed the goals of the course in an email message to other College of Engineering faculty this way: Major goals of the class include introducing students to the scope and breadth of engineering, developing team skills, building student confidence and abilities, and creating a framework within which students' college curriculum makes sense. Spring semester goals for the students as stated by the faculty in their "Philosophy of "Introduction to Engineering'" follow:
The faculty anticipate that the goals and philosophy that emerged during 1994-95 will continue to evolve as they strive to meet the need of the students and various stakeholders. As the following representative interview excerpt makes clear, the faculty believe that although they laid a strong foundation for the course during the pilot year, additional improvements will be made. R: I think [our discussion] has made all of us more curious about assessment of student learning, and provided us with a lot of questions. We're very much aware that where we are is kind of neat, but we're probably going to end up in a different place--[although] not substantially different, philosophically different. What we've got now is a good start, but it's not the finished product yet. The section on instructor experiences elaborates on faculty expectations for further course revisions, based on dialogue and interaction among the students, faculty and others. For a comprehensive look at the design project, the learning environment, course organization, and facilities, please refer to the student handbooks for both semesters. For a closer look at the process by which the course was designed and the manner in which the course was conducted, refer to the paper which the faculty presented at June 1995 annual meeting of the American Society for Engineering Education. The Students: A Learning Community of First-year Students During both semesters students worked in small laboratory groups of seven to twelve students. Within each lab, students worked in at least two smaller teams throughout the semester. During the first half of the semester, the team created a proposed solution to the "real-world problem." During the last half of the semester, the entire lab was once again a larger team, but students created new teams to accomplish the task of designing and testing their proposed solution. The student teams became small learning communities in which students made connections among themselves by working together to design and test solutions to real-world problems. The value of the common experience that the learning community provided them became apparent through interviews and focus group discussions with the students. For example, during a focus group discussion, two male students, one from each semester, described the value of the small group learning community this way. S1: Definitely [making friends in small design teams] makes a difference. You come there and you don't know anyone pretty much and you walk around and you see all these new people. And you take these classes and you get to know people [in general], and then you are just walking around campus you see these [friends from the design class] and you say, "Hi" just as you walk by. And it's really nice to feel like you fit [in engineering], that you know people. You're not a stranger. It just feels nice to have someone to say "Hi" to you! S2: Once in awhile it's like scary. Like if you are going down University Avenue or something and you see 3 people in a row that you know from other classes. It's like "Hi!" "Hey!". I only stay in touch with one person that I know from my high school... pretty much everybody that I know down here, I met down here. It's just weird. I: Do you feel it was easier to recognize the people that you were with in your design course last semester vs. some other class? Does the group size make a difference? S2; Yeah, because you know them better. I'll recognize a lot of people, but it's like, "Hey, I know you!" ...So everything becomes more familiar, but it's easier when you actually know the person and you might know both their first and last name and some stuff about them. You've actually worked with them. I: Let's talk more about that: you've actually worked with them. What role does that have to play here and in making you, to use your words, feel like you fit here in engineering? S1: Well, you've had a common experience. You've got something you can talk about--remember when you did this or that. Moreover, the students made connections among the various components of the course. As the following representative excerpt indicates, they became aware during the course of how these different components worked together in support of the learning process. S2: It just seemed like an on-going process even though you only met twice a week. It would have been almost nicer if we had two labs a week because things tended to get rushed. The email even brought you closer. Like I said, you had it twice a week and lab just once a week, but you felt the whole connection going on all week pretty much. And we had to do those Friday reports, which was good. I: So you're saying that there was a connection all week because of the email? S2: Yeah, and like in a physics course, you go to your lab, you have your discussion, and you have your lecture, and whenever you're in class that's kind of when you are concerned with it, besides studying outside of class... But in the design course, it's "I have to get this ready for next week." And if it's something important, everybody is going to be depending on you, and you are depending on them for other things. Whereas in a [regular] course you have maybe a quiz every once in awhile and then you have your midterms and it's like segmented in these big time frames. For more details on the student experience, see Tab 2. The Instructors: A Learning Community of Faculty and Senior Assistants Faculty represented four of the nine engineering disciplines taught at UW-Madison:
Mike Corradini, Professor, Nuclear Engineering and Engineering Physics and
Mechanical Engineering The course was additionally supported by seven senior engineering students each semester, known as "senior assistants" (SAs) who assisted faculty and students in the laboratory sections.
Lauren Barker, Civil and Environmental Engineering, first semester Both faculty and senior assistant teams functioned as learning communities as well. They learned from one another and recognized that students were observing them as they too grappled with the challenges of teamwork. This faculty comment illustrates what all the faculty members said in one way or another: Because of the kind of class, the faculty team was very helpful. The other issue though is faculty working as a team. I have to say it is difficult if not somewhat hypocritical for faculty in a class like this to preach teamwork and really grade students on teamwork and not demonstrate or learn or experience it themselves...demonstrate their own teamworking abilities. I wouldn't want to say that everybody should be graded for their goodness or badness as a team member, but I think that it is a very different thing to have experience first hand being a member of a team. rather than just the academic issue of "oh teams are good, so theoretically you should all like being in teams. So go be a theoretical team." So in that sense, I think that the experience of working in teams is pretty important for faculty if they want to have any sense of what their students are experiencing. For more details on the faculty experience, see Tab 3. The Content and Teaching/Learning Methods This section describes the course components as they would appear to an observer. Lectures: Students attended one 50-minute lecture at 8:30 a.m. each Tuesday. Lectures took place in rooms in Engineering Hall (EH) that featured blackboards as their only education technology. Final presentations took place in room EH 1610 (first semester) and EH 3609 (second semester), which featured more advanced educational technologies. Labs: Students attended one of seven, three-hour labs each week during first semester and one of two evening labs during second semester (Tuesday and Wednesday). First semester, two labs were in the evenings (Tuesday and Wednesday) and the other four were held during the day (either Tuesday, Wednesday, or Thursday). Lab size varied from seven to twelve each semester. Faculty and students received an outline of lab activities for each week. Students attended labs regularly; seldom did students miss. Project: A "real" engineering project with "real" customers is the cornerstone of the course around which the design process revolves. The concept of the "real" customer sets this first-year design course apart from similar courses across the nation, as one faculty person learned at the 1995 ASEE Annual Conference. The fall design project on which the students focused was access for wheel chair users for historic buildings at Old World Wisconsin. Wheel chair access to an additional building, T23 on the University of Wisconsin-Madison engineering campus, also was involved. The spring project was a counter system to determine the number of people who enter the Kurt F. Wendt Library and Olbrich Gardens. Students in small design teams evaluated customer requirements, brainstormed solutions, selected the best solution, designed the product, and constructed and tested it. Homework: Students completed several homework assignments early in the semester. Topics included computer software for word processing and drawing, stress analysis, and a library search at the Kurt F. Wendt Engineering Library. Journals: Students kept a personal design journal to track both the design process and class learning throughout the semester. Faculty reviewed the journals periodically and incorporated them into the course grade. Notes: For fall semester, a handbook that the faculty had prepared during summer 1994 constituted the textbook. Additional notes were often distributed during lectures and labs throughout the semester. For spring, the handbook was more extensive since handouts in the previous semester were incorporated into the handbook. Presentations: Students prepared and delivered two presentations during the semester. The first was within their own lab when each team of three or four students presented their team design. The second was presented to all students, all instructors, some real customers, COE faculty, and COE dean. Each lab group of up to twelve students presented their lab's design using the multimedia facilities of a high tech classroom. Assessments: Students participated in peer and self assessments after both presentations. Faculty used the results to help assess student learning in terms of team process and project. Some faculty shared the assessment results after the first presentation; most did not. A new assessment second semester was an end-of-semester essay that students wrote to reflect their engineering experience. This essay accounted for a portion of the grade. Office hours, Email Dialogue, and Weekly Updates: Both faculty and senior assistants held regular office hours and encouraged open communication by electronic mail throughout the semester. A large number of email messages were exchanged, with the weekly number received ranging from two to ten. Often faculty shared these replies by forwarding them to all students and instructors because the answer to one students' individual question would benefit everyone involved in the course. (A log of email dialog among faculty and students is available in the LEAD Center.) We have insufficient information to report how frequently students attended office hours. During second semester, students submitted weekly individual progress reports by email. This replaced the team reports by email that were part of the first semester course. Faculty responded to each student's weekly update on a regular basis. Research Questions and Preliminary Answers The purpose of the qualitative research reported here is to yield contextualized understanding of a curriculum innovation aimed at enhancing and modifying the teaching and learning experience in an engineering college. Further, the research centers around a case study of a curriculum innovation. The curriculum innovation is the new freshman design course. We list here the research questions investigated by the evaluation project, as articulated in the evaluation proposal. This section distinguishes between research questions regarding student learning experiences and questions regarding instructor teaching and learning experiences. The methodology and procedures for analyzing and reporting both student and instructor experiences are described in Tab 9.
Preliminary answers to these questions are provided in the remainder of this Summary. Research Question 1.
What is the impact on students of a curriculum innovation aimed at enhancing and modifying the teaching and learning experience in an engineering college? The qualitative evaluation, supported by the limited available quantitative data, indicates that the first-year design course has had a positive effect on the students. We are confident that effects are present and have important consequence for student learning and undergraduate curriculum in the College of Engineering. Effects based on the quantitative data are listed first (a - c, below); effects based on the qualitative data help answer the question "Why is the course having the effects?" and are listed next (d - k). Key patterns emerging for the students include:
Research Question 2.
Can qualitative evaluation methods provide insightful answers to the question of how, what, and how well are students learning in this course? Yes, qualitative evaluation methods provide insightful answers to the question of how students learned in the first-year design course. Virtually all students mentioned how the group work which was intrinsic to teamwork helped them learn. They frequently mentioned in one-on-one interviews, focus group discussions, and on the open-ended question survey that hands-on experience with a real project and real customers helped them learn. Although few students mentioned their journals as a tool to help them learn, faculty who reviewed the journals are confident that students' reflective writing in the journals helped students learn. Data from observations of lectures and labs support the conclusion that student teams, hands-on experiences, and journals are effective tools to effective learning. Although the observation method revealed that students were learning specific skills like teamwork, presentation, and design skills, only the interviews and survey methods were able to confirm from the students' perspectives that teamwork is instrumental in how they learned. Conversely, data from across all sources show that the homework and notes were the least effective as tools that fostered student learning. Yes, qualitative evaluation methods provide insightful answers to the question of what students are learning. Student responses during one-on-one interviews, focus groups, and on surveys provided open-ended opportunities to identify what they were learning. Using structured interview protocols and referring to the goals for the course, student respondents gave more insightful answers than traditional methods may be able to capture. Yes, but to a limited degree, qualitative evaluation methods provide insightful answers to the question of how well students are learning. Although the student interviews could capture some of the enthusiasm that students had about their learning, the qualitative methods are not designed to determine how well students are learning. It is necessary to make careful observations over time to assess whether the course successfully improves students' abilities to form, interact, and present information as teams. This type of intensive observation data was not collected by the third-party evaluators. Faculty alone are in a position to determine how well students are learning specific information and skills. See data points and learning outcomes in Tab 2 for information that would support these conclusions and Tab 9 for a description of the evaluation research methodology. Research Question 3.
Do student assessment procedures (journals, homework, observations) adequately measure what students are learning and foster the learning process? Faculty used a variety of assessment procedures to measure what students learned in the course, including continual observations, peer and self assessments, and journal writing. Yes, the student assessment procedures foster the learning process. Minor adjustments may improve the assessment procedures so that they more adequately measure what students are learning. Researcher interviews with students indicated that students perceive these procedures as measuring what they are learning and that they do, in fact, help students in their learning process. Virtually all students commented about how the non-threatening, non-competitive teaching environment with the journal writing, instructor observations, and self-and peer assessments helped students' learning process. Virtually all students also commented on how frequent reviews and comments from faculty helped them as well. According to interviews with faculty, the assessment procedures measured what students learned. Given the goals and objectives for the course, almost all the instructors were comfortable using their own observations, student journal entries, and peer and self-assessments to assess what student goals were accomplished. The question of what students were actually learning in the first-year course was debated throughout second semester among the faculty. Faculty recognize the need to continue to experiment with assessment procedures to help them determine what their students are learning and what they as faculty can do to improve the learning. (See data points and learning outcomes in Tab 3.) Research Question 4.
How well is the course achieving the faculty's goals for student learning? Data from across all sources support the conclusion that the first-year design course is achieving the faculty's goals for student learning. Interviews with students help document what and how the students learned. Their UW-Madison classifications at the end of their freshmen year help document that most students who completed the design course are still interested in an engineering career. Interviews with faculty help document faculty perceptions as to how well the course is achieving the goals for student learning. Virtually all the faculty agree that the course they designed is helping students achieve the intended goals. Virtually all the faculty also agreed that goals related to process were more successful than goals related to product; that is, they recognized that the weakest parts of the course were helping students develop skills for engineering tools and engineering science principles and business, and integrating industrial ties into the course. Worthy of note is that faculty debated the course goals throughout the year, especially following the first semester. Although a few faculty were concerned that they never reached consensus, virtually all faculty recognize the goals set forth in the "Philosophy of 'Introduction to Engineering'" as the goals for the course (see Tab 8). These goals, while somewhat different from those stated in the 1994-95 student handbook, are the result of the faculty's debate over appropriate goals and how to measure them. Also worthy of note is that goals stated by the principal investigators in one-on-one interviews are aligned with the goals stated in the philosophy (see Tab 4). Both faculty and principal investigators describe the goals in terms of giving first-year students a series of successful experiences as they explore real engineering problems. (See data points 3, 4 of Tab 2.) Research Questions Based on Instructor Objectives for the Course Research Question 5.
What is the impact on instructors (faculty and senior assistants) of a curriculum innovation aimed at enhancing and modifying the teaching and learning experience in an engineering college? The qualitative evaluation, supported by the limited quantitative data, indicates that the first-year design course has had a positive effect on the faculty, although some negative effects became apparent. Key effects emerging for the faculty are listed under instructor learning outcomes (LO) as described in Tab 3. LO1. Achievement of goals for first-year students and SAs is an outcome by which faculty gauge the value of the course.
LO2. Faculty develop their own learning community, but find it may be more difficult for faculty to function as members of a team than for first-year students.
LO3. Faculty develop as individuals.
LO4. Departments/administrators affect and are affected by the course.
LO5. Faculty become aware of the value and challenges of shifting teaching into a collaborative mode in a public arena.
LO6. Faculty express excitement, pride, and anticipation of an even better course in future years.
Research Question 6.
What do the faculty who are team-teaching this course learn from each other while investigating the key question, "What and how are the students learning?" Faculty who team-taught the first-year design course learned a variety of things from each other in a variety of ways. Again, the key patterns emerging for the faculty are listed under instructor learning outcomes as described in Tab 3. LO1. Achievement of goals for first-year students and SAs is an outcome by which members of the faculty team gauge the value of the course.
LO2. Faculty develop their own learning community, but find it may be more difficult for faculty to function as members of a team than for first-year students.
LO4. Departments/administrators affect and are affected by the course.
LO5. Faculty become aware of the value and challenges of shifting teaching into a collaborative mode in a public arena.
LO6. Faculty express excitement, pride, and anticipation of an even better course in future years. Upon subsequent analysis, additional answers to these research questions may become apparent.
Analytic Generalizations: From the Researchers' Point of View Analytic generalizations (AG) from the researchers point of view about the teaching and learning processes are divided into two groups: those pertaining to students and those pertaining to faculty. Analytic Generalizations for Student Processes (Tab 2) AG1. Approach to problem-solving: Systematic, teamwork engenders creative, multiple approaches to problem-solving. AG2. Attitude toward learning: Virtually all students demonstrate a positive attitude toward learning and pleasure in connecting the engineering design process to real-world problems. AG3. Sources of affirmation: Interaction with peers and "more-experienced others" helps students create and build their own design and feel comfortable in both a social and professional engineering environment. AG4. An "island" of connected understanding and real learning: The course needs to "count" as a standard part of the College of Engineering curriculum. AG5. A family away from home: The instructor team provides the environment and curriculum to help students make informed decisions about engineering as a career. Analytic Generalizations for Faculty Processes (Tab 3) AG1. Approach to teaching and learning: Collaborative teamwork engenders creative, cooperative approaches to teaching and learning. AG2. Attitude toward teaching and learning: Faculty demonstrate positive attitudes toward teaching and learning and pleasure in connecting their TIP study of cooperative, collaborative teaching to a freshman engineering design course where they are both teachers of student teams and learners in a faculty team. AG3. Sources of affirmation: Interaction with colleagues helps faculty develop their self-confidence in designing a curriculum that meets the needs of students and other stakeholders. AG4. An "island" of connected understanding and cross-disciplinary teaching and learning: The course needs to gain buy-in from other faculty for acceptance within the College of Engineering. AG5. Early connections with high schools: Faculty come to believe that the "before" connection with high schools is as important as the "after" connection with business and industry. AG6. Assessment of student learning and evaluation of the learning process: Faculty are addressing these significant issues. AG7. Project with real customers: This feature is essential to the "spirit" of the course. Technical Information The following information will help the reader understand the presentation of qualitative data presented in this report. 1. Use of Verbal Quantifiers in Reporting Qualitative Data Specific verbal quantifiers are used to denote the relative size of a group of participants who presented particular perspectives or described particular experiences in interviews. It is important to note that due to the nature of qualitative interviews, the size of a group that discussed a particular type of experience does not indicate the size of the group who had this type of experience. Although the same interview protocol was used in each interview, respondents' answers often prompted discussion on a particular area that may not have emerged in other interviews. The verbal quantifiers used in this report are
2. Presentation of Transcribed Materials Dialogue from interviews is presented inside boxes. Students are denoted with "S:", instructors and principal investigators with "R:", and the interviewer with "I:". If more than one respondent is involved in a conversation as is the situation with focus group discussions, responses are denoted with "R1:" and "R2:" and so on. Ellipses (...) in quoted material indicate deleted dialogue occurring within the reproduced material. Deletions are made so that readers can appreciate the speakers' views on a particular topic without having to sort through the divergent twists and turns of the raw dialogue. The quoted material is presented as faithfully as possible to the speaker's intent. If additional text is necessary to understand the quote in context to the rest of the discussion, it is added in brackets.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||