Does the DHET research output subsidy model penalise high-citation publication ? A case study

HOW TO CITE: Harley YX, Huysamen E, Hlungwani C, Douglas T. Does the DHET research output subsidy model penalise high-citation publication? A case study. S Afr J Sci. 2016;112(5/6), Art. #2015-0352, 3 pages. http://dx.doi.org/10.17159/ sajs.2016/20150352 South African universities are awarded annual subsidy from the Department of Higher Education and Training (DHET) based on their research publication output. Journal article subsidy is based on the number of research publications in DHET-approved journals as well as the proportional contribution of authors from the university. Co-authorship with other institutions reduces the subsidy received by a university, which may be a disincentive to collaboration. Inter-institutional collaboration may affect the scientific impact of resulting publications, as indicated by the number of citations received. We analysed 812 journal articles published in 2011 by authors from the University of Cape Town’s Faculty of Health Sciences to determine if there was a significant relationship between subsidy units received and (1) citation count and (2) field-weighted citation impact. We found that subsidy units had a significant inverse relationship with both citation count (r= -0.247; CI = -0.311 – -0.182; p<0.0001) and field-weighted citation impact (r= -0.192; CI= -0.258 – -0.125; p<0.0001). These findings suggest that the annual subsidy awarded to universities for research output may inadvertently penalise high-citation publication. Revision of the funding model to address this possibility would better align DHET funding allocation with the strategic plans of the South African Department of Science and Technology, the National Research Foundation and the South African Medical Research Council, and may better support publication of greater impact research.


Introduction
South African universities are awarded annual subsidy from the Department of Higher Education and Training (DHET) based on research publication output -a significant proportion of which is composed of journal article publications.The journal article subsidy is based on the number of journal output units generated by the university, calculated from the number of research publications in DHET-approved journals and the proportional contribution of authors from the university.This subsidy provides financial incentive to increase research output.
Given that the DHET subsidy rewards and intends to stimulate research for the benefit of the country, the manner of awarding subsidy should align with strategies to maintain or improve the impact of South Africa's research.Ideally, university researchers would publish high-quality research that makes an impact in the scientific field, and, where appropriate, work collaboratively with other groups to add value to studies and aid further development and translation of the research.
The goals of South Africa's National Research Foundation (NRF) Strategic Plan 1 incorporate not only research output, but also 'citation intensity', emphasising the importance of the impact of the country's research (not only the volume).The NRF system of rating researchers is also based primarily on the quality and impact of their outputs. 2The South African Medical Research Council (SAMRC) Strategic Plan 2014/15-2018/19 3 highlights the need to publish in highimpact journals, and includes the number of articles published in the 'top four' journals (New Engl J Med, Lancet, Science and Nature) as an indicator towards meeting its objectives.The SAMRC also encourages its scientists to work collaboratively, as 'no single group can respond alone to the priorities' 3 .This sentiment is echoed by the NRF Strategic Plan 1 , which advocates for 'promoting and enhancing international networks and partnerships', as well as by the South African Department of Science and Technology's (DST's) Ten-Year Innovation Plan 4 , which states that greater networking and collaboration (domestic and international) is needed for the country's biotechnology industry to grow.However, the current subsidy model does not factor in research quality or impact (other than specifying that journals must be DHET accredited).In addition, the greater the co-authorship with other institutions (domestic or international), the lower the subsidy received by a university.This consequence may result in a disincentive to collaboration.It has been argued that the current system may lead to 'non-virtuous practices in research' 5 , such as writing short 'salami-sliced' papers, targeting low-tier journals with high acceptance rates, and avoiding collaboration 5,6 to increase subsidy.Cautioning that the drive to increase research volume had come at the expense of the pursuit of excellence, the 2014 Report of the Ministerial Committee for the Review of the Funding of Universities argued it was time to change the funding framework. 5Yet, the new Research Outputs Policy published in March 2015 7 did not make any changes to the journal article subsidy formula to address these concerns.
Inter-institutional collaboration may affect the scope and quality of research as well as the impact of the resulting publications in the scientific field, as indicated by the number of citations received.Indeed, it has been shown that research that is more collaborative is associated with higher citation rates. 8,9The inverse relationship between DHET subsidy units received by an institution for a paper and the proportion of authors from outside that university may therefore lead to greater subsidy being awarded to articles of lower citation impact than those of higher citation impact.We hypothesised that greater subsidy (as a result of fewer 'outside' authors) would in fact be associated with lower citation impact.We analysed a set of journal articles published in 2011 to determine if there was a significant relationship between subsidy units received and (1) the number of citations received (citation count) and ( 2) the field-weighted citation impact.The latter measure is the ratio of citations received by a publication and the average number of citations received by all other similar publications 10 (i.e. with the same publication year, publication type and discipline), and so takes into account differences in citation patterns across disciplines or publication types.

Materials and methods
We analysed journal articles published in 2011 by authors from the University of Cape Town's (UCT's) Faculty of Health Sciences that were audited and approved for subsidy by DHET.For each article, we identified the subsidy units assigned by DHET to UCT and the proportion of non-UCT authors.For each article, we extracted (on 15 December 2015) the citation count (total number of citations received) and the field-weighted citation impact for the period 2011-2015 using data drawn from Scopus 11 accessed via SciVal 12 .
Articles not listed on the Scopus database were excluded from the analysis.
The relationship between subsidy units assigned to articles and their citation count, as well as between subsidy units and the articles' fieldweighted citation impact, were examined using a Pearson correlation, with a two-tailed p-value and a 95% confidence interval.

Results
Following exclusion of 38 articles not listed on the Scopus database, 812 articles were included in the analysis, with a mean subsidy unit assignment of 0.53±0.35units (Table 1).Of these, 589 (72.5%) articles had non-UCT co-authorship.
Both citation count and field-weighted citation impact were negatively correlated with subsidy units (Figure 1).While the shared variance was small, the correlation was significant in both cases.

Discussion
DHET subsidy units assigned to UCT for journal articles published by the Faculty of Health Sciences in 2011 had a significant inverse relationship with both citation count and field-weighted citation impact.This finding implies that subsidy allocation is smaller for articles receiving a greater number of citations than it is for those receiving a lower number of citations, whether in absolute terms (citation count) or when compared with the average number of citations received by similar publications 10 (i.e. with the same publication year, publication type and discipline).
Greater collaboration is associated with greater citation rates and, author affiliation aside, citation count tends to increase with the number of authors (with self-citation likely to play only a minor role). 8,9,13,14It is therefore not unexpected that lower-subsidy-earning publications, which will have been more collaborative, are more highly cited.The analysis has confirmed our hypothesis.By directly relating subsidy to the proportional contribution of authors from a university, and therefore penalising universities for collaborative research, the annual subsidy awarded by DHET to South African universities for research publication output may also be inadvertently penalising high-citation, 'high-impact' publication.
Given the financial benefit of subsidy unit assignment, the existing model discourages inter-institutional collaboration.This situation seems particularly punitive in the case of international collaboration.Universities only receive subsidy for the proportional contribution of authors from that university, whether the external authors are based at other South African institutions or international ones.The annual research publication output subsidy distributed by DHET is currently valued at about ZAR1.6 billion. 15,16Such large-scale funding should align with the strategic goals of the government, i.e. should incentivise collaborative research that may be associated with high-impact science.
Collaborative publishing has increased in recent years in African institutions, and Pouris and Ho 17 suggest that the large proportion of interinstitutional articles from South African universities indicates that factors encouraging collaboration outweigh the adverse impact of the funding model.Indeed, we found 72.5% of the publications in our analysis had non-UCT co-authorship.That noted, South African universities have different methods of internally allocating subsidy received, and this internal funding distribution could influence their researchers' publishing behaviour.We also note that the nature of research output in the university sector is differentiated with respect to volume, journals, level of collaboration, citation rates and scientific field.Our findings are representative of health sciences articles from a research-intensive university that does not directly allocate publication subsidy to researchers.
Our findings support the recommendation of the 2014 Report from the Review of the Funding of Universities 5 that the funding framework be revised.The Report proposed that subsidy units be divided only among South African authors of articles, so the model no longer actively discourages international collaboration.This revision would better align the DHET model with the DST's Innovation Plan 4 , the SAMRC's Strategic Plan 3 and the NRF Strategic Plan 1 , which encourage collaboration.
DHET research output subsidy is a means of distributing government funding in a way that factors productivity.Research output has been steadily increasing in South Africa over the last decade and the subsidy system is thought to have contributed to this increase. 5,18However, quantity should not be emphasised at the expense of quality. 19It is perhaps notable that while South Africa's medical publication output increased during 1996-2010, the number of citations per document declined. 20The 2014 Report from the Review of the Funding of Universities 5 also recommends that quality and scientific impact of publications be directly factored in the model; preferential weighting of journals with higher impact factors was suggested.The notion of a journal's impact factor being a measure of the quality of papers published in it has been contested, and deficiencies of this measure as a tool for research assessment have been highlighted. 21he impact factor may have some utility in the funding framework as an indicator of the quality of the journals in which South African universities publish, especially given research output is aggregated at an institutional level.This is in line with the original intention of the impact factor, namely as a measure of journal (rather than article) quality. 22Revising the subsidy model to include a weighting for research impact would potentially better align it with the NRF Strategic Plan 1 and SAMRC Plan 3 , which advocate for targeting impact.As bibliometric measures of quality and impact evolve in an attempt to minimise inherent flaws (e.g. through the addition of weighting by field), their utility in directing funding allocation should be evaluated on a regular basis.
In summary, the annual subsidy awarded by DHET to South African univer sities for research publication output may be inadvertently pena lising high-citation publication.Revision of the funding model to address this effect would better align DHET funding allocation with government strategic plans and may better support publication of greater impact research.

Figure 1 :
Figure 1: Correlation of the subsidy units assigned to the University of Cape Town for journal articles published by authors from the Faculty of Health Sciences in 2011 with (a) the citation count and (b) the field-weighted citation impact, for the period 2011-2015 (accessed 15 December 2015).

Table 1 :
Descriptive data for journal articles (n=812) published by authors from the Faculty of Health Sciences at the University of Cape Town (UCT) in 2011 that were approved for subsidy by the Department of Higher Education and Training (DHET) http://www.sajs.co.za Volume 112 | Number 5/6 May/June 2016