INTRODUCTION
Internet-based and online teaching and learning is becoming popular and was needed during COVID-19 pandemic. The relatively recent advent of Learning Management Systems (LMS), such as blackboard, Desire2Learn (D2L), eCollege, Moodle, and WebCT, and systems for offering lectures via MS Teams, Zoom and other platforms in the undergraduate setting in educational institutions has made it easy to provide online education, that is, web-based enhancement to traditional (face-to-face) classroom instruction (Rutter & Matthews, 2002). This online, hybrid or other mixed delivery approach allows instructors to combine the advantages of online class learning with the benefits of face-to-face interaction with relatively limited technological requirements on their part (Edling, 2000). The addition of a hybrid/online approach to the existing hybrid or face-to-face lecture-centric environmental engineering course is not supposed to reduce the quality of teaching and learning and would be welcomed and well received by students (Karim, 2016; Black, 2002). Preliminary reports suggest that the hybrid approach provides significant benefits for students and instructors, regardless of their level of technological expertise (Black, 2002; Brakels et al., 2002) and regardless of whether the classroom is hard-wired for live Internet access (Bento & Bento, 2000). Despite frequent use of an LMS for course administration purposes (content and lecture delivery), the instructors do not appear to be harnessing the full pedagogical potential of web-based augmentation via LMSs. The potential of LMS tools along with other online and mobile technology platforms to increase course administration/lecture delivery efficiency and maximize or enhance learning in traditional settings is an important educational issue that must be fully authenticated from both instructor and student perspectives (Papadakis et al., 2017; Papadakis et al., 2018; Woods et al., 2004). However, combining multiple modalities for online content delivery with a pot pouri of hybrid learning exercises that appeal to several learning styles may precipitate higher overall learning outcomes (McCray, 2000).
Theoretical Background
Traditional perception of teaching and learning in all levels is usually a face-to-face approach. Due to advancement of technology and internet connectivity teaching has evolved. In order to keep up with the technological advancement and new generations’ mindset traditional teaching moved away and web-based teaching found its way in the past decades. Although course delivery using either a hybrid or online approach may increase time demands and, in some cases, result in a loss of control, many instructors enjoy this approach because it allows for significant flexibility and benefits in instructions. Due to COVID-19 in March 2020 course delivery for many institutions worldwide had to change to online synchronous and/or asynchronous formats, and exams had to be administered online and/or use an alternative assessment process.
This study was designed to answer two questions: (1) What are the students’ perceptions and attitudes about the online synchronous course delivery along with online exam-taking options and online platforms as a learning environment? (2) Is there any effect of course delivery approach changes due to COVID-19 pandemic on students’ performance levels?
To answer the above questions, two objectives of this study were formulated. The first objective was to understand the students’ perceptions and attitudes about online synchronous course delivery along with online exam-taking options and online as a preferable learning environment for future environmental engineering courses. The second objective was to see the effect of COVID-19 change in the students’ performance levels. The first objective was accomplished via an anonymous online survey and the second objective was accomplished with statistical analysis of final exam scores, weighted average GPA, and the overall course grades. The goal of this study was to understand the overall effect of COVID-19 pandemic on students’ perceptions and attitude about an online synchronous course delivery and to assess the performance level, as well as to compare the performance levels between hybrid (Pre-COVID-19) and online synchronous (Post-COVID-19) course delivery options. The following sections outline the intervention, study methodology, instruments used, data collection and analysis, results and discussions, and finally conclusions and recommendations.
Intervention
Intervention simply means purposeful actions by a human agent to create and implement change. As we all know that end of 2019 and early in 2020, a pandemic of coronavirus (COVID-19) broke out in China and then spread globally. In the USA, spring semester starts in January and ends in May. Due to public health advisory and presidential Corona virus taskforce guidance, the education institutions in the USA had to make several changes in the course delivery in order to limit the spread of COVID-19. Within two weeks of this advisory, the educational institutions had to come up with an approach that would meet the guidance (6-ft social distance, washing hands, and face covering) without interruption of education. Several options were thought out such as converting all the courses to 100% online, either asynchronous or synchronous. Asynchronous delivery calls for video recording of lecture sessions and posting them in LMS. Since all faculty were not trained to be online instructors, synchronous option was chosen, with some training sessions for the faculty how to use MS Teams, Zoom, or Blackboard Collaborate for online synchronous delivery platform. That is how our university ended up delivering all courses online synchronous since mid-March 2020. The parameters for optimum group forming strategy, content design, effectiveness measurement, meta-analyses, impact of technology on PBL, implementation framework, alternative assessment in PBL, and other procedures for optimum learning that are acceptable to students and instructors were elaborated from the literature (Albanese & Mitchell, 1993; Berkson, 1993; Blumenfeld et al., 1991; Diehl et al., 1999; Frank et al., 2003; Karim, 2015; Krajcik et al., 1994; Moursund, 1999; Mills & Treagust, 2003; Ravitz & Blazevski, 2014; Strobel & Van Barneveld, 2009; Thomas et al., 1999; Vernon & Blake, 1993; Watson & Fang, 2012; Boubouka & Papanikolaou, 2013).
METHODOLOGY
The course, ‘Introduction to Environmental Engineering’, was developed as an online course for Civil and Environmental Engineering program students but taught as a hybrid (50% time in-class lecture and 50% time online self-paced) before COVID-19 (Pre-COVID-19) pandemic and online synchronous during COVID-19 (Post-COVID-19) pandemic for several semesters, to test the concept. In the hybrid course set up, all the quizzes and homework were online while the midterm and final exams were in-class. The spring 2020 final exam was online and during summer 2020 both midterm and final exams were online due to COVID-19 adjustment. The following two subsections outline the type of instruments used and the type of data collected and analyzed to accomplish the objectives of this study.
Instrument
To understand the effect of COVID-19 on the perceptions and attitudes of students learning (the first research question), an online anonymous survey via D2L was conducted at the end of the semester with five questions. The survey questions are presented in Figure 1. The first two questions were meant to investigate the students’ perceptions and attitudes about the course content and alignment delivered with online synchronous approach although no changes were made in the course content and other alignment. The third question probed an alternative assessment process using technologies and investigated students’ perceptions and attitudes about the challenges of conducting online closed book exams using a lockdown browser and webcam, as most of the traditional students were not familiar with these technologies. The fourth and fifth questions were to understand the students’ perceptions and attitudes about several test taking options (alternative assessments) and levels of learning. The second instrument used investigated the final exam scores, weighted average GPA, and the overall course grades to assess the students’ performance level and to compare the students’ learning environment between hybrid (Pre-COVID-19) and online synchronous (Post-COVOD-19) delivery (the second research question).
Data Collection and Analysis
The data collected through the online survey was analyzed to understand students’ perceptions and attitudes about the course content and alignment, online exams using technologies, the exam-taking alternatives, and the degree of learning. The data that was collected and used to assess the performance levels was the final exam scores (maximum minimum, and average) and the overall course grades (Letter grades and weighted average GPA considering A = 4.0, B = 3.0, C = 2.0, and D = 1.0). F-grade was not included in the assessment as the students only receive an F-grade when they stop coming to the class or drop out after the deadline. The data was collected for the spring and summer 2020 semesters as online synchronous delivery and compared with data from the summer and fall 2019 semesters as hybrid delivery. There was a total of 34 students enrolled in summer 2019, 35 in fall 2019, 48 in spring 2020 (2 sections), and 33 in Summer 2020 semesters. Overall, 76 (about 51%) students participated in the survey for all 4 semesters. Nine students (about 26%) participated in the survey for summer 2019, 15 (about 43%) in fall 2019, 29 (about 60%) in spring 2020, and 23 (about 70%) in summer 2020. Seventy-four (74) students (about 49%) did not take the survey because the survey was not mandatory, and no incentive/grade points was given for taking the survey. Final exam scores, weighted average GPA, and the overall course grades were statistically analyzed and compared for differences between Pre-COVID-19 and Post-COVID-19 delivery. The analysis of data was performed with simple statistical tools and excel using goodness-of-fit tests such as ANOVA, χ2-tests, student t-tests, and F-tests, as necessary. The results of the data analysis are illustrated in the following section and in Figure 2 through Figure 6. Some of the responses to questions/options/choices, as seen in the figures, do not sum up to 100% as a few students did not respond to all questions or select all options or choices.
RESULTS AND DISCUSSIONS
Course Content and Alignment
For Q.1, overall, about 92% agreed that tests materials reflected what was covered in the class (Figure 2), both before and during the COVID-19 situation. The highest score was for the summer 2020 class (96%), followed by spring 2020 (93%), summer 2019 (89%), and fall 2019 (87%).
The distributions of Q.2 responses are presented in Figure 3. It can be seen that, overall, about 93% of the students, participating in the survey, agreed that is there is a good agreement between the course outline and the course content, with the highest score (100%) for summer 2019, followed by fall 2019 and spring 2020 (93%), and summer 2020 (91%). It is apparent that students’ perceptions and attitudes about the course content and the alignment were consistent and similar for the Pre-COVID-19 and Post-COVID-19 situation.
Online Exam Using Technologies
The weighted average response to Q.3, as to how the participants liked to take online midterm and final exams through D2L using Respondus Lockdown Browser and Webcam, was 2.83, which is close to the middle, with only a minority (25%) of students being positive and declining from spring to summer 2020. (see Figure 4; there were of course no values for summer 2019 and fall 2019). About 7% of the participants did not answer this question. It is clear that online examination with lockdown browser and webcam is not popular, presumably because it can be cumbersome to make the lockdown browser and webcam work, depending on the computer and the individual knowledge of computer operations.
Alternative Evaluation and Learning
Regarding Q.4 on preferences for the midterm and final exams, 17% of the participants chose option 1: “Get the questions from D2L, print it, take it, scan and submit it in submission folder without proctoring”, 37% chose option 2: “Take-home exam for a day or two”, 13% chose option 3: “100% online with Multiple Choice Questions like a quiz”, and 25% chose option 4: “100% online that is get the questions in D2L like a quiz, take the exam like quiz and do the detail work in papers proctored using webcam, scan the papers in pdf and submit the papers in submission folder” (see Figure 5). About 8% participants did not answer this question. Option 2 (take-home exam) has the highest score, whereas students’ preference for online quizzes seem to decline from spring to summer.
This could indicate that the students prefer to complete the test in their own time with the possibility of external help, as it has to be open book; and that this creates less anxieties than being watched by someone or a webcam. However, this cannot be confirmed until a take-home exam is conducted and evaluated.
To check the face-value outcomes above, a chi-square goodness-of-fit test was performed to validate or reject the null hypothesis “no differences from semester to semester and among four exam-taking options”. The chi-square test data are shown in Table 1. From the chi-square test, a p-value of 0.2913 was obtained, which is greater than both 0.05 (α = 5%) and 0.01 (α = 1%). A χ2-value of 3.7371 was also obtained. For a degree of freedom of 3, the critical values for χ2 are 7.81 (for α = 5%) and 11.3 (for α = 1%). The chi-square (χ2) value is less than the critical values of both the significance levels. So, the null hypothesis cannot be rejected, and it cannot be concluded that the differences from semester to semester and among the four exam taking options are statistically meaningful.
|
Table 1. Chi-square goodness-of-fit test for Q.4 data
|
Semester
|
Observed Values
|
Expected Values
|
|
Option 1
|
Option 2
|
Option 3
|
Option 4
|
Total
|
Option 1
|
Option 2
|
Option 3
|
Option 4
|
Total
|
|
Spring 2020
|
4
|
10
|
6
|
9
|
29
|
5.438
|
11.479
|
4.229
|
7.854
|
29
|
|
Summer 2020
|
5
|
9
|
1
|
4
|
19
|
3.563
|
7.521
|
2.771
|
5.146
|
19
|
|
Total
|
9
|
19
|
7
|
13
|
48
|
9
|
19
|
7
|
13
|
48
|
p-value = 0.2913; χ2-value = 3.7371; DF = 3, χ2-critical = 7.81 (for α = 5%) and 11.3 (for α = 1%)
|
To verify this, a single factor ANOVA was performed, and the data is presented in Table 2. Since F < Fcritical (in this case, 2.1538 < 6.5913), the null hypothesis indeed cannot be rejected.
|
Table 2. ANOVA for Q.4 data
|
Group
|
Sum
|
Count
|
Average
|
Variance
|
Source
|
SS
|
DF
|
MS
|
F
|
p-value
|
F-crit
|
|
Option 1
|
9
|
2
|
4.5
|
0.5
|
Between group
|
42
|
3
|
14
|
2.154
|
0.2362
|
6.591
|
|
Option 2
|
19
|
2
|
9.5
|
0.5
|
Within group
|
26
|
4
|
6.5
|
---
|
---
|
---
|
|
Option 3
|
7
|
2
|
3.5
|
12.5
|
Total
|
68
|
7
|
---
|
---
|
---
|
---
|
|
Option 4
|
13
|
2
|
6.5
|
12.5
|
|
|
|
|
|
|
|
|
Regarding Q.5 on learning, 29% of the participants chose option 1: “Learned same as hybrid/face-to-face”, 15% chose option 2: “Learned more online than hybrid/face-to-face”, and 48% chose option 3: “Learned less online than hybrid/face-to-face”. About 8% participants did not answer this question. The distributions of Q.5 responses are presented in Figure 6 and it seems obvious that students think they learned less online than in the hybrid or face-to-face conditions.
To see the variations of the three learning options for spring 2020 and summer 2020 (summer 2019 and fall 2019 were not included as these semesters were not a part of the remote offerings), a chi-square goodness-of-fit test was performed to validate or reject the null hypothesis “no differences from semester to semester and among three learning options”. The chi-square test data are shown in Table 3. The p-value is too high and the χ2-value is too low to reject the null hypothesis, so the differences between the semesters and among the three learning options are not statistically significant.
|
Table 3. Chi-square goodness-of-fit test for Q.5 data
|
Semester
|
Observed Values
|
Expected Values
|
|
Option 1
|
Option 2
|
Option 3
|
Total
|
Option 1
|
Option 2
|
Option 3
|
Total
|
|
Spring 2020
|
11
|
5
|
13
|
29
|
9.063
|
4.833
|
15.104
|
29
|
|
Summer 2020
|
4
|
3
|
12
|
19
|
5.938
|
3.167
|
9.896
|
19
|
|
Total
|
15
|
8
|
25
|
48
|
15
|
8
|
25
|
48
|
p-value = 0.4063; χ2-value = 1.8015; DF = 2, χ2-critical = 5.99 (for α = 5%) and 9.21(for α = 1%)
|
To verify this, a single factor ANOVA was performed, and the data are presented in Table 4. Since F < Fcritical (in this is the case, 4.0555 < 9.5521), it was confirmed that the null hypothesis cannot be rejected.
|
Table 4. ANOVA for Q.5 data
|
Group
|
Sum
|
Count
|
Average
|
Variance
|
Source
|
SS
|
DF
|
MS
|
F
|
p-value
|
F-crit
|
|
Option 1
|
15
|
2
|
7.5
|
24.5
|
Between group
|
73
|
2
|
36.5
|
4.055
|
0.1403
|
9.552
|
|
Option 2
|
8
|
2
|
4.0
|
2.0
|
Within group
|
27
|
3
|
9.0
|
---
|
---
|
---
|
|
Option 3
|
25
|
2
|
12.5
|
0.5
|
Total
|
100
|
5
|
---
|
---
|
---
|
---
|
|
Assessment
The analysis of assessment outcomes was based on the final grades for hybrid (summer 2019 and fall 2019) and online (spring 2020 and summer 2020) exam options. The data are presented in Table 5. From the chi-square test, a p-value of 0.0401 was obtained, which is less than 0.05 (α = 5%) but greater than 0.01 (α = 1%). A χ2-value of 17.5994 was also obtained. For a degree of freedom (DF) of 9, the critical values for χ2 are 16.9 (for α = 5%) and 21.7 (for α = 1%). Therefore, with α set at 1%, the null hypothesis cannot be rejected but when α is set at 5%, it can. So, with some uncertainty, it can be concluded that the differences in the final grades from semester to semester and between hybrid and online exam options are significant.
|
Table 5. Assessment based on final grades using Chi-square Goodness-of-fit test
|
Exam Option
|
Semester
|
Observed Grades
|
Expected Grades
|
|
A
|
B
|
C
|
D
|
Total
|
A
|
B
|
C
|
D
|
Total
|
|
Hybrid
(Pre-COVID-19)
|
Summer 2019
|
10
|
14
|
7
|
3
|
34
|
8.04
|
14.01
|
9.65
|
2.30
|
34
|
|
Fall 2019
|
9
|
18
|
5
|
1
|
33
|
7.80
|
13.60
|
9.36
|
2.23
|
33
|
|
Online
(Post-COVID-19)
|
Spring 2020
|
9
|
13
|
22
|
4
|
48
|
11.35
|
19.78
|
13.62
|
3.24
|
48
|
|
Summer 2020
|
7
|
16
|
8
|
2
|
33
|
7.80
|
13.60
|
9.36
|
2.23
|
33
|
|
|
Total
|
35
|
61
|
42
|
10
|
148
|
35
|
61
|
42
|
10
|
148
|
p-value = 0.0401; χ2-value = 17.5994; DF = 9, χ2-critical = 16.9 (for α = 5%) and 21.7 (for α = 1%)
|
A second analysis of assessment was based on the weighted average GPA for hybrid and online exam semesters. The data is presented in Table 6. From the chi-square test, a p-value of 0.9974 was obtained which is greater than both 0.05 (α = 5%) and 0.01 (α = 1%). A χ2-value of 0.0459 was also obtained. For a degree of freedom of 3, the critical values for χ2 are 11.1 (for α = 5%) and 15.1 (for α = 1%). The chi-square (χ2) value is less than the critical values of both 7.81 (α = 5%) and 11.3 (α = 1%). Therefore, the null hypothesis cannot be rejected so the differences in GPA from semester to semester and between hybrid and online exam options are not statistically significant. A t-Test and an F-Test performed for this parameter confirmed this.
|
Table 6. Assessment based on weighted average GPA using Chi-square Goodness-of-fit test
|
Exam Option
|
Semester
|
Observed GPAs
|
Expected GPAs
|
|
|
Hybrid (Pre-COVID-19)
|
Summer 2019
|
2.9118
|
2.8458
|
p-value = 0.9974
c2 value = 0.0459
|
|
Fall 2019
|
3.0606
|
2.8458
|
|
Online (Post-COVID-19)
|
Spring 2020
|
2.5625
|
2.8458
|
|
Summer 2020
|
2.8485
|
2.8458
|
|
|
Total
|
11.3834
|
11.3834
|
|
The third analysis of assessment was based on the final exam Minimum, Average, and Maximum scores obtained by students in the hybrid and online delivery semesters. The analysis is presented in Table 7. From the chi-square test, a p-value of 0.9959 was obtained which is greater than both 0.05 (α = 5%) and 0.01 (α = 1%). A χ2-value of 1.3995 was also obtained. The critical values for χ2 are 12.6 (for α = 5%) and 16.8 (for α = 1%) for a degree of freedom of 6. The chi-square (χ2) value is less than the critical values of both significance levels. Therefore, the null hypothesis cannot be rejected, and it cannot be concluded that differences between exam scores from the semester to semester and between hybrid and online exam options are significant.
|
Table 7. Assessment based on the final exam scores using Chi-square Goodness-of-fit test
|
Exam Option
|
Semester
|
Observed Values
|
Expected Values
|
|
Min
|
Avg
|
Max
|
Total
|
Min
|
Avg
|
Max
|
Total
|
|
Hybrid
(Pre-COVID-19)
|
Summer 2019
|
30
|
58
|
88
|
176
|
30.05
|
56.98
|
88.96
|
176
|
|
Fall 2019
|
30
|
67
|
100
|
197
|
33.64
|
63.78
|
99.58
|
197
|
|
Online
(Post-COVID-19)
|
Spring 2020
|
35
|
55
|
95
|
185
|
31.59
|
59.90
|
93.51
|
185
|
|
Summer 2020
|
30
|
57
|
87
|
174
|
29.71
|
56.34
|
87.95
|
174
|
|
|
Total
|
125
|
237
|
370
|
732
|
125
|
237
|
370
|
732
|
p-value = 0.9659; χ2-value = 1.3995; DF = 6, χ2-critical = 12.6 (for α = 5%) and 16.8 (for α = 1%)
|
The summary of the goodness-of-fit test analyses is listed in Table 8 for four different types of data. Based on the goodness-of-fit test and ANOVA it was apparent that students’ perceptions and attitudes about the four different exam-taking options and three different learning options did not differ significantly, although face-value analysis of the responses suggested otherwise. The analyses of the final exam scores (maximum, minimum, and average), weighted average GPA, and the overall final grades confirmed that the observed differences were not big enough to say that outcomes differed significantly. The only value that proved significant (and only at the 5% level) was the difference between the final grades for the hybrid (2019) and the online (2020) exams.
|
Table 8. Summary of Goodness-of-fit test analyses
|
Data Type: c2-Test
|
p-value
|
c2-value
|
DF
|
Critical Value
|
c2-Test Comment
|
|
0.05
|
0.01
|
|
Students’ choices for four exam options (Survey – Q.4)
|
0.2913
|
3.7371
|
3
|
7.81
|
11.3
|
p-values are greater than both 0.05 (a = 5%) and 0.01 (a = 1%) and c2-values are less than the corresponding critical values. The differences are not significant.
|
|
Students’ choices for three learning options (Survey – Q.5)
|
0.4063
|
1.8015
|
2
|
5.99
|
9.21
|
|
Weighted average GPA
|
0.9974
|
0.0459
|
6
|
12.6
|
16.8
|
p-values are greater than both 0.05 (a = 5%) and 0.01 (a = 1%) and c2-values are less than the corresponding critical values. The differences are not significant.
|
|
Final exam scores (Min., Avg, Max.)
|
0.9659
|
1.3995
|
6
|
12.6
|
16.8
|
|
Final Grades
|
0.0401
|
17.5994
|
9
|
16.9
|
21.7
|
p-value is less than both 0.05 (a = 5%) and greater than 0.01 (a = 1%) and c2-value is greater than 16.9 (a = 5%) and p-value is greater than 0.01 (a = 1%) and c2-value is less than 21.7 (a = 1%). Differences are significant at the 5% uncertainty level.
|
|
Data Type: t-Test
|
p-value
|
t-value
|
DF
|
tcritical
(two tail)
|
t-Test Comment
|
|
Weighted average GPA
|
0.2237
|
1.7413
|
2
|
4.3026
|
Since t-value is within -tcritical and +tcritical, the null hypothesis cannot be rejected. The observed values are not big enough to conclude that differences are significant.
|
|
Data Type: F-Test
|
p-value
|
F-value
|
DF
|
Fcritical
(one tail)
|
F-Test Comment
|
|
Final Grades
|
0.3055
|
0.2708
|
1
|
0.0069
|
Since F-value > Fcritical, the null hypothesis is rejected. Therefore, variances of the two populations, hybrid and online exam options, are NOT equal.
|
|
Data Type: ANOVA
|
p-value
|
F-value
|
DF
|
Fcritical
|
ANOVA Comment
|
|
Students’ choices for four exam options (Survey – Q.4)
|
0.2362
|
2.1538
|
3
|
6.5914
|
Since F < Fcritical (in this is the case, 2.1538 < 6.5914), therefore, the null hypothesis cannot be rejected. The four exam taking options are statistically equal.
|
|
Students’ choices for three learning options (Survey – Q.5)
|
0.1403
|
4.0555
|
2
|
9.5521
|
Since F < Fcritical (in this is the case, 4.0555 < 9.5521), therefore, the null hypothesis cannot be rejected. The three learning options are statistically equal.
|
|
CONCLUSIONS AND RECOMMENDATIONS
From the data of this study, it can be concluded that a project-based or problem-based learning (PBL) course with alternative assessment options without physical contact, such as oral assessment or take-home exam can be introduced and implemented during COVID-19 without harm to the students’ learning and performance. In this study, an effort was made to understand the students’ perceptions and attitudes with regard to the learning environment as well as their performance levels in environmental engineering for the changes in the course offerings due to COVID-19 in the middle of spring 2020. Students agreed that the course content in the online format was aligned with the content of assessment, but their perceptions and attitudes about learning in online environment and taking online exams using technologies (respondus lockdown browser and webcam) appeared to be not favorable. Students’ choice was the take-home exam. Fortunately, it could not be proved by statistical analysis that the online synchronous approach significantly degraded the level of students’ performance, although face value analysis suggested that online synchronous delivery approach does not maintain the same level of students’ performance. It is the author’s opinion that PBL delivery with take-home exam, as preferred by the students in this study, along with other alternative evaluation processes (cf. Boubouka & Papanikolaou, 2013) can be adopted to maintain the students’ learning and performance at pre-COVID-19 levels.
STUDY LIMITATIONS
A source of bias for this study could be the fact that the author was the only person who designed this study, conducted the survey, collected the semester end data, analyzed the data and had, as a teacher, an interest in a positive outcome. Another limitation is that all students were in the same engineering program, making it unclear whether the outcomes would be the same for other STEM students. The last limitation is the number of students involved to test the hypotheses. Perhaps, with a larger sample, face value differences do become significant.