Revelations from an online diagnostic arithmetic and algebra quiz for incoming students

FUNDING: Eskom; National Research Foundation: South African Agency for Science and Technology Advancement A review of relevant literature revealed that research indicates an alarming number of secondary school students who enter tertiary institutions display a serious lack of mathematical basics required for the study of university mathematics. This so-called change from the secondary school to tertiary study of mathematics has become an important field of mathematical learning research. In this context, at the University of KwaZulu-Natal, online diagnostics for pre-calculus knowledge and skills were made available to first-year mathematics students. We report here on some findings from an analysis of the provision of diagnostic questions for basic arithmetic and algebra. The results show that the diagnostic questions which were in the form of an online quiz assisted students to cope with their mathematics module in the sense that it positively impacted on the final results of students who attempted the quiz.


Introduction
The increase in the number of students entering university has revealed a greater inconsistency in the background levels of those students enrolled for first-year mathematics modules. 1 This transition from secondary to tertiary education in mathematics creates a complex phenomenon covering a huge range of problems and issues. 2lthough it is believed that there is evidence of similar transitional issues in other disciplines in science, it seems that the transition in mathematics is by far the most serious and the most problematic. 3,4A possible reason for this discrepancy is the many changes that take place in the transition from secondary to tertiary education.These include changes in teaching and learning styles, the type of mathematics that is taught, conceptual understanding, procedural knowledge that is required to advance through and the changes in the level of advanced mathematical thinking needed. 5News24, a local online newspaper, reported that, according to Umalusi (a South African council for quality assurance in general and further education and training), matric (Grade 12) mathematics results were worse in 2014 than in the previous years. 6This news report supports the perception that there is a very serious decline in core mathematical skills.Because a percentage of those matric students enrol for tertiary studies in mathematics in subsequent years, this issue should be of concern to those who lecture to such students, especially in first-year mathematics modules.
Jennings 2 is of the opinion that it has now become even more important to understand a student's prior knowledge of mathematics before they study tertiary mathematics.The main reasons for this need are: (1) to find out whether a student would benefit from additional support on basic arithmetic and algebra, offered in an online format and (2) to produce suitable first-year students who are going to be competent throughout their course of study.These are not the only reasons, but we have chosen in this paper to focus on these two given reasons.For students to be competent throughout their course of study, necessary steps need be taken to improve their performance, which implies that all possible assistance should be available to students to reach the ultimate goal of succeeding in their mathematics course.For this reason, at the University of KwaZulu-Natal (UKZN), the provision of online diagnostic questions for basic mathematical knowledge and skills for incoming students was rolled out at the beginning of February 2015.Keeping in mind the fact that not all the students attempted the online quiz, for this paper the final performance of students who completed the quiz was considered for the analysis and compared with the performance of those students who did not take the quiz.

Research question
The main research question was: What does an investigation into completion of the online diagnostic basic arithmetic and algebra quiz by incoming university students reveal?To answer this question the following subquestions were formulated for the provision that was in place for 2015: (1) What was the participation level of students with regard to the online diagnostic quiz?(2) What was the performance level of students with regard to the online diagnostic quiz?(3) How did students' participation in the quiz affect their performance in the final examination?(4) What are the views of students on the impact of this online diagnostic quiz?

Literature review
There has been a huge concern about the inflow of students to tertiary level institutions who are lacking the fundamentals in mathematics.It is perceived that the majority of students have difficulties with mathematical basics in arithmetic and algebra.Breiteig and Grevholm 7 noted that the abstractness of algebra is one reason for students' problems.This problem impacts on the study of calculus because algebra is a pre-requisite for such studies.This lack of student preparedness has consequences for student and staff satisfaction, student self-esteem, retention and progression of students, and financial costs to universities and to students arising through failure to progress or complete a course. 3According to Kaput 8 , algebra is a route to higher mathematics, but at the same time a barrier for many students which forces them to take another educational direction.On the transition from arithmetic to algebra, Moseley and Brenner 9 noted that ' …moving from arithmetic problems to working with more symbolic representations of relationships with variables in algebra is an area of persistent difficulty for many' (p.4).The provision of diagnostic questions for basic arithmetic and algebra focused on in this paper was intended to assist the students to develop the ability to see variables as representations of unknown quantities that, despite their abstract nature, can still be altered by the arithmetic operations. 9Looking at the challenges that students have concerning mathematics and also considering the importance of mathematics in our modern society, it is not surprising that learning support in mathematics is becoming ever more important. 10 address the under-preparedness of incoming students to study mathematics, many universities offer mathematical support services such as tutors, online resources and individual consultations in order to improve the results of the students. 11The majority of institutions in Ireland and internationally now make use of diagnostic testing for incoming firstyear students. 11This kind of support gives students an opportunity to revise some basic concepts in mathematics.It is believed that, generally, diagnostic testing has helped to indicate the level of preparedness of students who are entering tertiary level and has also contributed to some level of improvement in their final results. 11Thus it bridges the gap between the level of preparedness expected or required upon entry level and the mathematical capabilities acquired by new students.At UKZN, such online support was put in place.Students attempt the quizzes online to improve their knowledge or to practise a particular topic.Online quizzes can be attempted an unlimited number of times and there is no time limit to complete the quiz questions.Thus, a student has the opportunity to work on the quiz questions and improve on their basic knowledge and skills considered relevant for the study of calculus.The article by Maharaj and Wagh 12 provides detailed information on how the quiz questions were formulated based on the identified expected learning outcomes for basic arithmetic and algebra.Those identified outcomes were used to frame suitable online diagnostic questions.In this paper, we focus on the provision of those diagnostics to students.

Conceptual framework
Our conceptual framework was informed by the literature review and was guided by the following: 1.The study of mathematics is hierarchical in nature. 13 To study university calculus, students need to have adequate knowledge and skills relating to basic arithmetic and algebra.4 3. Tprovision of suitable online diagnostic questions on basic arithmetic and algebra could improve the level of student competencies with regard to these.11,12 4. Theres a need to research the impact of the provision of online diagnostics for pre-calculus as offered at UKZN before a decision could be arrived at as to whether the quizzes should be made compulsory for all incoming students who want to study firstyear calculus.

Methodology and participants
The UKZN learning MOODLE site for Diagnostics for Pre-calculus was used for data collection.On this site, students had access to five precalculus quizzes that were designed to focus on relevant knowledge and skills for the study of calculus.The students had to first complete an online consent letter, as required by the research office at UKZN for ethical clearance (protocol reference number HSS/1058/014CA).
The Basic Arithmetic and Algebra quiz was used to gather data on the participants who completed the quiz.The quiz consisted of 24 questions which involved purely basic mathematical knowledge in arithmetic and algebra.Each question carried a maximum of 2 marks.The system awarded 2 marks if a student submitted the correct answer for their first attempt.If the first attempt was incorrect, the system awarded 1 mark for a correct second or third attempt.A final mark of zero was awarded for a question if the first three or more attempts were incorrect.For the study, only those students who completed all the quiz questions were regarded as having taken the quiz.Students who submitted their responses to only some of the questions were considered as having not attempted the quiz.
The quiz was attempted by students from different first-semester mathematics modules offered at UKZN: Introduction to Calculus (Math130), Mathematics for Natural Sciences (Math150), Mathematics 1A Engineering (Math131) and Quantitative Methods (Math134).In total, there were 1888 attempts for the quiz, which indicated the overall number of attempts.That number was reduced to 997 based on the criteria that only students who completed the quiz were considered for the study.
For the purpose of this study, it was decided to investigate the impact of the quiz on the final results of students enrolled for the Math130 and Math150 modules.The reason for this choice was that Math130 is a core module for students who want to continue with further studies in mathematics, while Math150 is a compulsory module for other students who want to study science.
A total of 250 students enrolled for Math130, and out of those, 95 students attempted the quiz, so 155 students did not attempt the quiz.A total of 872 students enrolled for Math150, and out of those, 392 students attempted the quiz, so 480 did not attempt the quiz.For the students who attempted the quiz more than once, only their highestgraded complete attempt was considered.Students' responses for the quiz were used to gather information to answer the research question.
On the system, each question was allocated a maximum of 2 points.As indicated earlier, if the first attempt submitted was incorrect then there was a penalty for subsequent attempts.After studying the students' performances, it was decided to use the following interval mark ranges for each question: 1.70 and below; 1.70 to 1.89; 1.90 and above.
A questionnaire was also formulated to obtain students' views on the diagnostic quiz and how it could be improved.Participants were chosen based on the criteria that they used the learning site to attempt the quiz questions.An email was sent to such students indicating that research was being conducted to gauge the effectiveness of the provision of the online diagnostics in which they participated and that they were invited to complete the online questionnaire.Also, the final examination results for all the students registered for the two modules were obtained from the UKZN Student Management System website.Those final results were used to determine if there were differences in the final scores between the students who attempted the quiz and those who did not.
The final examination results lists were obtained for Math130 and Math150.Each list was used to separate the students into two groups: students who completed the quiz and students who did not complete the quiz.Because we were comparing the two population means within a large data set, the two-sample z-statistical test was used, which follows a normal distribution and the population samples are independent. 14The Z-test formula used was: where Z calc is a calculated Z value; and x, s, µ and n represent the sample mean, standard deviation, population mean and the number of students who attempted the quiz, respectively, for both groups.Setting up the hypothesis test, we wanted to determine if there is a difference in the final examination results of the students who attempted the quiz compared to those who did not.Thus our null hypothesis was: attempting the quiz had no effect on final examination results.That is: Against the alternative hypothesis: According to the null hypothesis, our population mean difference is zero.
http://www.sajs.co.zaThe data for such students were excluded from the study.For the testing of this hypothesis, only student attempts for Math130 and Math150 were considered.As indicated before, the former is a mainstream module undertaken by students who want to continue with studies in mathematics, while the latter is a compulsory service module offered to science students.

Entry requirements for the mathematics modules
The following entry requirements were obtained from the UKZN College of Agriculture, Engineering and Science handbook. 15For Math130 (Introduction to Calculus), the pre-requisite requirement was: Higher Grade D or Standard Grade A for matric Mathematics, or NSC Level 5 Maths.For Math150 (Mathematics for Natural Sciences), the prerequisite requirement was: Higher Grade E or Standard Grade B for matric or NSC Level 4 Maths.The minimum pre-requisite requirements for each of those two modules are regarded as being equivalent by the College of Agriculture, Engineering and Science.For each of those two modules, the students who were enrolled satisfied the minimum requirements.In a case in which a student did not meet the minimum requirements, the student was required to enrol for a foundation mathematics module, called Math199.After a student passed Math199 with a specified minimum percentage, then they qualified to enrol for Math130 or Math150.The purpose of Math199 is to equip students with mathematical tools that are needed for their chosen mathematics module.Table 1 indicates the percentage ranges for the different levels and their meaning, which were in operation from 2008 onwards for the National Senior Certificate examinations (NSC).Because some participants completed their Grade 12 examinations before the year 2008, during which period symbols were used, the necessary conversions were done to match the old Grade 12 symbols with the new levels.

Participants' Grade 12 mathematics and English proficiency levels
In this section we focus on the mathematics and English proficiencies of the Math130 and Math150 students as determined by the NSC examinations.For each module, bar graphs represent visually the spread of the NSC levels and the frequency for each of those levels.
Figure 1 indicates that the majority of students in both the groups had proficiency levels from 5 to 7, with regard to their senior certificate mathematics and English results.Also note that from the Math130 entry requirements it was required that students who wanted to be enrolled for Math130 had to have obtained at least a level 5 in their senior certificate mathematics.There was also a small number of students who achieved less than level 5.Those students were the ones who did foundation mathematics (Math199), and after passing with at least 60% were enrolled for Math130.
Figure 2 indicates that the majority of students attained levels from 4 to 6 with regard to their results for senior certificate mathematics and English.Note that for Math150, the entry requirement was a pass at at least a level 4 for mathematics.There was also a small number of students who achieved less than level 4. Those students were the ones who did foundation mathematics (Math199), and after passing with at least 50% were enrolled for Math150.
With the above focus on the Grade 12 entry requirements and their frequencies, it was revealed that for each module the two student groups considered were not of different ability levels with regard to their school achievement for mathematics and English.For mathematics, they could be grouped into one category because of the entry requirements for each module.The implication here is that in testing the hypothesis for each module, the participants who completed and those who did not complete the quiz could be regarded as being of similar ability levels with regard to their mathematics and English proficiency when they were enrolled for their university mathematics modules.

Analysis, findings and discussion
The analyses and findings are presented under the following subheadings: Average student scores per question; student participation and performance levels; and student views on impact of quiz.

Average student scores per question
These results are for all the students across the different modules who attempted and completed the quiz on Basic Arithmetic and Algebra.
Looking at Figure 3, which gives the average student scores for each question, we observe that the questions could be placed into one of three categories, for which the students: (1) scored an average of 1.70 and below, (2) between 1.70 to 1.89 and (3) an average of 1.90 and above.A high average shows that most students scored full marks on the particular question, that is, they correctly answered the question.Likewise, when the average is low it shows that on that question students did not perform well.The finding is that questions on the application of basic properties for logarithms were not adequately understood by students.The implication is that first-year mathematics lecturers should be aware of and plan to address this shortcoming.If one accepts the principle of moving from the known to the unknown, then it is important to determine and address this shortcoming before the students are exposed to the natural logarithm and its notation, that is lnx.
The finding here is that, although students performed better, these types of questions could become stumbling blocks to their success when faced with calculus-related questions that require knowledge and skills based on the following: (1) addition, subtraction and division of algebraic fractions, (2) adding like terms, (3) applying the raising a power to a power law for exponents and (4) solving a linear equation in which a constant is represented by a letter.The implication is that measures need to be put in place to improve the proficiency of students with regard to these fundamentals.

Scores of 1.90 and above
It is evident from Figure 3 that average student performance was the highest for Questions 1-5, 8, 10, 11, 13, 14 and 24.This gives the impression that these questions were simple and easy to answer for the participants.The following are the types of questions for which students scored in this interval range: x 2 x x =... ; =... ; 1 1 x x x =... ; 2x 3 x 3x 5 =... ; 2x 3 -3x 3 =... ; 2x 3 ÷ (3x 5 ) =... ; if 2x + 5≥3, The finding here is that students displayed adequate proficiency for knowledge and skills based on: (1) addition, subtraction, multiplication and division of arithmetic fractions, (2) multiplication of simple algebraic fractions of the type indicated, (3) applying the multiplication and division laws of exponents for which base is the same, (4) subtracting like terms and ( 5) solving simple linear inequalities.The average student scores suggest that students were more proficient in subtracting like terms than they were at adding like terms, which seems strange; the question on the subtraction of like terms seems to be more difficult because it requires the calculation 2 -3 compared with the easier calculation of 2 + 3.

Student participation and performance levels
These results are presented under the following sub-headings: Students enrolled for Math130 and students enrolled for Math150.
Table 2 indicates that of 250 students who wrote the Math130 examinations, 95 students completed the quiz and 155 did not.For Math150, 872 students wrote the exam, 392 completed the quiz and 480 did not.We note that average examination marks for both modules were higher for students who completed the quiz compared with students who did not.For both modules, there was a large number of students who did not attempt the quiz, because participation was voluntary.

Table 2:
Final examination mean percentages of students who attempted the quiz and those who did not

Students enrolled for Math130
Here we present our findings on the impact of the quiz on the average final examination marks of students who completed the quiz (n=95) and those who did not (n=155).Figure 4 shows the overall performance in the Math130 final examinations for students who completed the quiz and those who did not.
From Figure 4 and Table 2, we observe that students who completed the quiz scored an average examination mark of 63.94% and those who did not scored an average mark of 58.01%.So the students who completed the quiz scored on average about 6% higher in their final examination compared with students who did not take the quiz.The implication here seems to be that completing the quiz impacted positively on the final examination results of the students.However, before any conclusions can be reached, we need to determine if the difference is statistically significant.A 99% level of significance (α=0.01) was used.We tested the null hypothesis: Versus the alternative hypothesis: The test statistic Z calc =3.128 and H 0 is rejected if Z α < Z calc where Z α was obtained from a Normal Z table.As Z α =Z 0.01 and Z calc =3.128, it follows that Z α < Z calc so we conclude that H 0 is rejected.Thus, the results verify that the difference is statistically significant at a 99% confidence level, that is, that the students who completed the quiz performed better in their examinations than those who did not.

Students enrolled for Math150
We present our findings on the impact of the quiz on the average final examination marks of students who completed the quiz (n=392) and those who did not (n=480).Figure 5 illustrates the overall average performance of Math150 students who completed the quiz and those who did not.From Figure 5 and Table 2, we observe that students who completed the quiz scored an average final exam mark of 67.68% and those who did not scored 63.73%.So students who took the quiz scored on average about 4% higher in their final examination than students who did not take the quiz.The implication seems to be that attempting the quiz improved the final examination results of the students.However, once again, before any conclusions can be reached, we need to determine whether the difference is statistically significant.
A 99% level of significance (α=0.01) was used.We tested the null hypothesis: Versus the alternative hypothesis: The test statistic arrived at was Z calc =4.142.Z α was obtained from a Normal Z table.As Z α =Z 0.01 =2.58 and Z calc =4.142, the implication is that Z α < Z calc so we conclude that H 0 should be rejected.Thus the results verify that the difference is statistically significant at a 99% confidence level, that is, that the students who completed the quiz performed on average better in their final examinations than those who did not.

Student views on impact of quiz
A questionnaire consisting of 10 multiple-choice and open-ended questions was completed by 60 students from both Math130 and Math150.The questions and student responses are indicated in Table 3.
Table 3 indicates that the majority of the respondents felt that the online diagnostic quiz was helpful to them, it was easy to access the diagnostic quiz online and that they were likely to continue attempting diagnostic quizzes online.Furthermore, students felt that the online diagnostic quiz related well to the course topics and that the online diagnostic quiz on Basic Arithmetic and Algebra helped them to understand key concepts in the relevant university mathematics module for which they were registered.This response is quite significant because the diagnostic quiz focused on identified knowledge and skills for basic arithmetic and algebra which are prerequisites for the modules for which the respondents were enrolled.The response to Question 6 indicates that the majority of the respondents were satisfied with the online diagnostic quiz.Note that at least 15% indicated that they were extremely satisfied.This response is worth noting as extremely satisfied was not indicated as a possible option in the framing of the question.The finding from the data was that the majority of the respondents rated the importance of attempting the online quiz towards the higher end of the scale, that is 7 or more.The responses for Questions 5 to 9 indicate that the majority of the students rated highly that taking the quiz was of benefit to them.The second and third responses that are indicated as common responses to Question 10 could be used to improve on the online diagnostic quiz.It is clear that some students want more challenging quiz items.Also, for students who do not answer correctly, there is a need to give more feedback that could help them.Perhaps the idea of building in pop-ups to further assist such students, as was suggested by a previous study 16 , should be considered.

Conclusions and recommendations
Although participation in the quiz was voluntary, it was found that a satisfactory number of students registered for each of the modules on which the study focused completed the quiz.For the Introduction to Calculus (Math130) module, the participation from registered students was 38% while for the Mathematics for the Natural Sciences (Math150) module it was 45%.Quiz items that were based on the application of basic laws for logarithms gave the students the most difficulty.Further, it was found that the students who completed the quiz performed better in their examinations than those who did not attempt or complete the quiz.This was determined at a 99% confidence level for each of the two modules, Math130 and Math150.For those modules it was found that the majority of students in both groups (those who completed and those who did not complete the quiz) had similar proficiency levels with regard to their senior certificate mathematics and English results.Thus the quiz should be made compulsory for incoming students who want to study calculus, because the research shows that there was a positive correlation between completing the quiz and final examination results.This positive correlation was for both the modules.Also from the questionnaire responses it was noted that the majority of those students who responded felt that the online diagnostic quiz benefitted them and contributed to their understanding of key concepts in the mathematics module for which they were enrolled.These responses suggest that those students appreciated that the basic concepts and skills for arithmetic and algebra formed the foundation for their university calculus modules.
We recommend that the quiz should be made compulsory because the research shows that it had a positive impact on students' final examination results.It is also recommended that the lecturing staff should analyse the results of quiz items to determine which pre-calculus knowledge and skills require attention and address these areas so that subsequent teaching of the relevant calculus concepts is likely to be beneficial to their students.

Figure 1 :
Figure 1: NQF Mathematics and English proficiency levels for Math130 students who (a) completed the quiz and (b) did not complete the quiz.

Figure 3 :
Figure 3: Average student scores for each question (n=997).Scores of 1.70 and below Looking at Figure 3, student performance in Questions 17 to 22 was below the average of 1.70.This score indicates that students had difficulty in answering these questions.The following are the types of questions for which students scored 1.70 and below: log 2 2=... ; log a 2+log a 5=log a ... ; log a 2-log a 5=log a ... ; 7log 2 a=log 2 ... ; log 2+log 5= ... ; log 16 ÷ log 8 = ...

Figure 2 :
Figure 2: NQF Mathematics and English proficiency levels for Math150 students who (a) completed the quiz and (b) did not complete the quiz.

Figure 4 :
Figure 4: Average exam percentages for Math130 students who attempted and completed the quiz and those who did not.

Figure 5 :
Figure 5: Average final exam percentages for Math150 students who attempted and completed the quiz and those who did not.

6 South
African Journal of Science http://www.sajs.co.za Volume 114 | Number 5/6 May/June 2018 Not all students wrote the final examinations.Possible reasons for not writing the final examination include that students dropped out of the module or did not obtain Duly Performed (DP) Certificates.

Table 1 :
Grade 12proficiency levels and their meanings