Exploring the relationship between entry requirements and throughput rates for honours students

FUNDING: None In order for a student to enrol in an honours programme at the University of KwaZulu-Natal (UKZN), a weighted average mark for their final year of undergraduate study must exceed a particular threshold value. Students are then ranked according to this weighted average mark, with entry into the honours programme offered on a top-down basis, within the constraints of teaching resources and space. A proposal has been made at UKZN to remove existing barriers for entry into an honours programme, i.e. to allow entry to any student who has completed a 3-year undergraduate degree with a major in that discipline. The impact of such a decision was investigated. By lowering the requirement for entry into an honours programme, one is expected to predict how a new cohort of students will perform. Apart from obviously having a lower weighted average mark for their final year of undergraduate study, these new students may also differ in other unobservable ways which need to be accounted for. In a regression modelling context, one is asked to predict outside the range of a collected data set. A Heckman selection model was used to account for a possible self-selection bias that may arise because the subpopulation for which a prediction is required (namely those new students who will now be able to enter an honours programme), may be significantly different from the population of UKZN undergraduate students who are currently permitted entry to an honours programme.


Introduction
In order to prepare students for entry into a job market that is rapidly evolving, 3-year undergraduate degrees that were once common in North America and parts of East Asia and South America have been replaced with 4-year undergraduate degrees.In Australia, the University of Sydney recently announced a radical reorganisation of their course offerings with an emphasis on the restructuring of their 3-year degrees into 4-year degrees.Noting that only a 'relatively small fraction' of students left their university after completing a 3-year degree, by embedding an honours degree within a 4-year undergraduate degree not only would the university be able to better prepare students for postgraduate study but they would also be able to substantially reduce the number of degrees that they have to offer.In South Africa, a similar debate is taking place at the University of KwaZulu-Natal (UKZN).In some disciplines, such as Engineering, a 4-year qualification has for many years been the norm.In the College of Management Sciences, students have the option to complete a 3-year undergraduate degree (which is then followed by an additional year of honours study) or a 4-year degree.When faced with this choice, students in that college seem to prefer enrolling for a 4-year degree.This paper is concerned primarily with the identification of the impact of the decision to lower the requirement for entry into honours on the throughput rate within that programme.The positive (or negative) consequences of allowing these (possibly weaker) students to enrol for an honours degree need to be weighed against the 'massification' of undergraduate programmes that is taking place, in which an increasing number of undergraduate students are wanting to improve their qualifications by enrolling in an honours programme.
Given that there is a financial cost incurred in training these students, by relaxing the entry requirement one may simply be increasing the number of students who are being set up to fail.A White Paper outlining a National Plan for Higher Education 1 made mention of this problem, highlighting at an undergraduate level the need for a planned expansion of the system to be 'responsibly managed and balanced in terms of the demand for access, the need for redress and diversification, the human resource requirements...and the limits of affordability and sustainability'.The document further states 1 : While it is possible to achieve rapid enrolment growth without extra expenditure, the penalties for doing so are harsh.Experience…shows that expansion without new investment results in overcrowded facilities, low morale of academic staff, poor quality programmes, a fall in research output and quality, and…devalued products of higher education.
More recently 2 the debate around curriculum change has grown; the Council on Higher Education released a document in 2013 proposing an overhaul of the undergraduate curriculum in South Africa.However, the focus of this paper is on the consequences of streamlining the pipeline for entry into honours.
Undeniably, being able to increase the number and knowledge base of students in our higher education institutions has many benefits.Research into academic performance at an undergraduate level has traditionally identified prior performance as being an important predictor variable.However, more recent studies [3][4][5][6][7][8][9] have found that academic success also depends significantly on the perception of a student regarding their learning environment and how http://www.sajs.co.za

Research Article
Entry requirements and throughput rates for honours Page 2 of 6 successful they have been in coping with difficult situations they have encountered at university [10][11][12] .Economic, social and cultural factors also have an important role to play in determining academic performance at an undergraduate level.However, the socio-economic factors that affected one's performance at an undergraduate level (in the above studies [3][4][5][6][7][8][9][10][11][12] ) may not be as strongly pronounced for an honours degree.Nevertheless, including these factors as covariates in our prediction model will be important for the analysis that follows.
Currently, entry into an honours programme at UKZN is not automatic.A weighted average mark in the final year of undergraduate study must exceed a threshold value of 55% before a student is allowed to enrol for an honours degree.4][15][16] One can then look for a difference in performance between these two cohorts -controlling, where necessary, for background variables that may also impact on their performance in their honours year of study.In this study, however, it was not possible to observe an honours-based performance for students who, under the current criteria, do not qualify for entry into honours.Consequently, the focus in this paper is different: to determine how this new cohort of students would perform if they had been given the opportunity to enrol for an honours degree at UKZN.A Heckman model needs to be used to adjust for a possible self-selection bias that may arise because the outcome of interest (in our case a suitably chosen measure of performance in the honours programme) can be observed for only a subset of students who were previously eligible for entry into honours.
8][19][20][21] In an educational setting, however, its application has been restricted mainly to identifying what sort of causal effect a particular level of education has on a given socioeconomic response variable (such as earnings).In fact, a detailed review of the literature 22 has found that only 14% of 386 articles discussing the problem of selection bias have done so in an educational context with a Heckman selection model then fitted to the collected data.The application of a Heckman model to our problem -for which 'entry into honours' replaces 'level of education' as the treatment variable and a weighted average mark in honours is used as a response variablecould well be novel, but is most certainly well supported by a number of other applications in the literature.

An analysis based on a weighted average mark in honours
This study focuses on a very specific cohort of students: those who completed an undergraduate degree at UKZN during the period 2011-2013.This cohort comprises 9398 students who obtained a weighted average mark in their final year of study that exceeded a threshold value of 50%.Under the prevailing set of rules allowing for a progression into honours, 3233 students from this sample were allowed to enrol for an honours degree because their weighted average mark exceeded a threshold value of 55%.An additional 2021 students from other universities were also allowed to enrol.Apart from noting very briefly that these other students generally did not perform as well in the honours programme as their counterparts who completed undergraduate studies at UKZN, the focus of this paper will rest entirely on comparing the observed performance (in honours) of the 3233 students who completed undergraduate studies at UKZN and were able to enter honours, with the performance that one would expect from a hypothetical cohort of 9398 UKZN graduates if the other 6165 students who achieved a weighted average mark in their final year of study of 50-55% joined the 3233 students currently in the honours programme.
To measure their performance in honours, each course was assigned an appropriate credit point value; an honours degree is considered to be completed when a certain number of credit points have been acquired.Using these credit point values as appropriate weights, a weighted average mark for all the compulsory courses was used as response variable Y i .For example, if eight courses must be passed in order to complete an honours programme and each course has been given a credit weighting of 16 points, then 16 * Mark for course 1 + ... + 16 * Mark for course 8 16 + ... + 16 will be the observed weighted average mark in honours for that student.If a course had to be repeated then both marks together with their appropriate weights were included in the response variable calculation for that student.Permission to use this data set was given by the Teaching and Learning Office at UKZN.
The results in Table 1 suggest that students who completed an undergraduate degree at UKZN performed significantly better in the UKZN honours programme than those who completed undergraduate studies at another university (t=12.185).Perhaps a significant number of the students who transferred from another university were weaker students who were refused entry into an honours programme at that university.There may also be other reasons for such an anomaly.For example, the demographic breakdown shown in Table 2 indicates that the students from other universities were older than those who entered from a UKZN undergraduate programme.Because the goal was to assess the impact of lowering the entry requirement on the throughput rate in the honours programme, the actual performance of the 3233 UKZN graduates enrolled in honours was compared with the expected performance of all 9398 graduates who would have been allowed to enrol if the entry requirement had been a weighted average mark of 50% or more.
Table 3 shows the results of regressing the weighted average mark in honours of each of the 3233 students against a set of appropriately chosen predictor variables.Ordinary least squares methods were used to produce the parameter estimates that appear in Table 3.For the interpretation of Table 3, 'female' refers to a 0/1 indicator variable that is set equal to 1 for a female student.'Black' refers to a 0/1 indicator variable that is set equal to 1 for a black student.Similarly, 'honours funding' refers to a 0/1 indicator variable that is set equal to 1 for a student who has managed to obtain some form of external funding for honours study.Because UKZN is composed of four colleges (College of Agriculture, Engineering and Science, College of Health Sciences, College of Humanities and College of Law and Management Studies), indicator variables for the first three colleges were included as covariates in Table 3 with an effect for the College of Law and Management Studies forming part of the intercept term in the above model structure.These covariates were chosen based on previous research [23][24][25][26] at UKZN that has identified them as being important predictor variables for determining performance at UKZN.
As one would expect, the weighted average mark that a UKZN undergraduate student obtains in their final year of undergraduate study correlates strongly and positively with the weighted average mark that they record for their honours year of study.Older students tend to not perform as well as their younger counterparts and female students perform better than their male counterparts.Having some form of honours funding also has a positive effect on performance in honours.
Significant college effects are also present, with students from the College of Health Sciences scoring 3.139% lower than that of a baseline student from the College of Law and Management Studies.Students in the College of Agriculture, Engineering and Science score on average 0.649% more than a baseline student from the College of Law and Management Studies.The most significant effect, however, appears to be associated with race, with black students scoring on average 4.733% lower than their counterparts from other race groups.
In the above regression model setting, a set of parameter estimates {β 0 , β 1 , ... ,β p } is identified that optimally links a set of demographic variables {X 1i , ... ,X pi } to the weighted average mark Y i that a student i obtains in their honours year with e i forming an error term accounting for a lack of fit in the following model structure: The results given in Table 3, however, relate to a very specific data setnamely those 3233 students with an undergraduate degree from UKZN who were allowed to enrol for an honours degree under the current set of rules which the university is now wanting to relax.With these results in hand, a student with a demographic profile of, for example: Because these results are estimated using a sample of 3233 UKZN graduates who were able to enrol in an honours programme, one cannot simply assume that the same sort of effects would occur if UKZN were to lower their entry requirement to include UKZN graduates with a weighted average value for their final year of undergraduate study of 50-55%.Apart from obtaining a lower weighted average score, these students may also differ in other important respects from the 3233 UKZN graduates who are currently eligible for entry into honours.To correct for a possible (sample selection) bias that may arise if we were to simply assume that the same set of results that appear in Table 3 would apply to this new cohort, a Heckman selection model was fitted to a now extended set of data comprising the 6165 UKZN graduates with a weighted average score for their final year of undergraduate study of 50-55% and the 3233 UKZN graduates currently in the honours programme for which we have a weighted average score for honours.
A Heckman selection model attempts to solve the problem associated with a potential sample selection bias by linking the actual selection process for entry into an honours programme to a set of appropriately chosen predictor variables {Z 1i , ... , Z ki }.Let R i denote a 0/1 indicator variable that models the selection process that is actually taking place.More specifically, assume entry into an honours programme (i.e.setting R i =1) is determined by the following decision rule: where u i ~N(0,1) denotes a random error term used to reflect the fact that once the threshold value for admission into an honours programme has been passed, admission into an honours programme is not necessarily automatic.For example, UKZN applicants are ranked according to the http://www.sajs.co.za To complete the model formulation, u i and e i are assumed to have a bivariate normal distribution with In the context of this paper, it is important to note that one is only observing an honours performance based outcome for someone who has actually been selected for entry into that programme.For the self-selected subpopulation of random variables Y i that we are able to observe, Heckman has shown that 5,7,8 : where denotes an inverse Mills ratio, ϕ a standard normal density and Φ a standard normal cumulative distribution function.
One could estimate the unknown model parameters in Equations 2 and 3 simultaneously using a maximum likelihood method based on the joint normality assumption of u i and e i .As an alternative, however, if λ i were observable, then ordinary least squares could be used on with w i denoting a zero mean error term with variance σ 2 (1-p 2 δ i ) that is now distributed independently of {X i , Z i }.As λ i is unknown, Heckman proposed the following two-step procedure to estimate β in the outcome Equation 3: • Step 1: Use all 9398 observations on Z i that are available, namely those associated with students who were eligible for entry into honours (i.e.those with R i = 1 and also those who were refused entry into honours (i.e.those with R i = 0 to generate a maximum like lihood estimator δ for δ based on the selection Equation 2. This estimate can then be used to compute the following inverse Mills ratio term for each observation; viz. Step 2: Now use the self-selected sample only (i.e.all 3233 observations for which we have R i = 1) to run an ordinary least squares regression on the following equation that is implied by the formulation given by Equation 4: an appropriate correction needs to be made to the variance covariance matrix that is associated with (β, β λ ) .
Tables 4 and 5 present the parameter estimates that result from fitting a Heckman model to the UKZN undergraduate based data set that is available.Table 6 contains a parameter estimate for p with a likelihood ratio based Wald test for p=0 rejected at all of the major levels of significance (chi-square=23.55,p≈0).This result confirms that the results in Table 3 do in fact need to be corrected for a sample selection bias because unobservable factors making up the error term u i in the selection equation are correlating positively with the error term e i that forms part of the response model determining the weighted average mark for honours.One cannot simply extrapolate the results shown in Table 3 to the new cohort of students and assume that these exact same results will be true for this new cohort of students.Replacing the '3-year plus honours' degree structure with a single 4-year structure is synonymous with lowering the honours entry requirement for UKZN undergraduates to 50% and excluding all foreign (non-UKZN applicants) who want to do an honours degree at UKZN.For example, for a 25-year-old, black, female UKZN undergraduate (with 43 matric points) who obtained a weighted average mark of 50% for her final year of undergraduate study and wants to enrol for an honours degree in the Faculty of Humanities, for which she did not receive any extra funding, the parameter estimates that appear in One cannot therefore use a regression model to naively extrapolate beyond the range of the data and expect to obtain appropriate results when attempting to answer the question posed in this paper.Furthermore, significant college based effects are recorded in Table 4 which suggest that any decision to relax the entry requirement for a specific college should not be applied as a blanket rule for all colleges.For example, if a weighted average mark of 50% is all that is required to complete an honours year of study, then the results in Table 4 suggest that students in the College of Health Sciences, black students, older students and male students do not perform as well as their counterparts in their honours year of study.Having access to some form of funding seems to improve results, but it is important to note that we are dealing with an associative rather than causative effect in this analysis.For example, funding may be associated with better results because higher achievers are more likely to receive funding; thus they perform better because they are higher achievers rather than because of the funding they received.
In conclusion, the purpose of this paper was twofold.Firstly, to contribute to a debate on the restructuring of undergraduate degrees in which there is a danger associated with naively extrapolating a regression model beyond the range of the data.Secondly, to provide a modelling technique that accounts for a possible sample selection bias that may arise because the profile of the proposed students may be very different from the profile of the students in the sample from which these inferences were drawn.To answer more specifically the question posed: lowering the requirement for entry into honours for UKZN graduates will result in this new cohort not performing as well in honours as their counterparts who currently are allowed into honours.However, the results in Table 1 indicate that students from other universities who currently are allowed into honours do not perform as well as their UKZN graduate counterparts.Therefore, replacing some of these students with a new cohort of UKZN graduate applicants may in fact improve throughput rates in some of the Colleges at UKZN.

Possible limitations
Because this study is primarily exploratory in nature, there are several issues or limitations.Because the focus was on UKZN data only, results from this study cannot necessarily be extrapolated to other universities in South Africa.However, showing how a self-selection bias can be corrected for using a Heckman model can easily be extended to data from another university, once the need to correct for a possible selfselection bias has been fully appreciated.
Because UKZN wants to relax the entry requirement for honours study from a weighted average mark of at least 55% to that of 50% for the final year of undergraduate study, this study focused on a very specific cohort of new students, namely those in the 50-55% band who would then be eligible to join students in the 55+% bracket who currently are allowed to enrol in an honours programme.If UKZN were to further relax the requirements, for example, to include students in the 45-55% band, then a very different set of results may arise.
Given that the aim was primarily to relate the weighted average mark recorded in honours to the mark obtained in the third year of undergraduate study, the inclusion of additional factors related to race, gender and matric point score may be viewed as complicating the analysis unnecessarily.
The estimates that appear in Tables 3 and 4, however, will be appropriate only if the error term e i that is referenced in Equations 1 and 3 is no longer correlated with any of the variables {X1 i … X pi } that are included as predictors in these respective regression model structures.By including enough of these variables in the model building process, the hope is to reduce the omitted variable bias problem that might arise if these confounders in the analysis are not controlled for.

Table 1 :
Weighted average marks (%) achieved by students enrolled in a honours course at the University of KwaZulu-Natal (UKZN) over the period 2011-2013

Table 3 :
Parameter estimates associated with a regression model fitted to their weighted average mark Y i in honours Law and Management is the baseline category for the Colleges.
Volume 113 | Number 9/10 September/October 2017 weighted average mark that they record in their final year of undergraduate study with 'top-slicing' then occurring based on other factors relating to classroom size and staff supervisory constraints.Parameter estimates for Equation 2 are derived from the extended set of data that include the 6165 UKZN graduates with a weighted average score for their final year of undergraduate study of 50-55% and the 3233 UKZN based undergraduate students who were allowed to enrol in honours.

Table 4 :
Parameter estimates associated with fitting the Heckman model given in Equation3Law and Management is the baseline category for the Colleges.

Table 5 :
Parameter estimates associated with the selection model given in Equation 2 http://www.sajs.co.za

Table 6 :
Parameter estimate associated with the correlation structure between e i and u i