The REF and UK Academic Macroeconomics: Friends or Foes?

Research evaluation exercises and metrics are an inherent part of academic life. But how do they affect individual research decisions, as well as a discipline’s body of knowledge?

In our new working paper we evaluate how the Research Excellence Framework (REF) affects academic macroeconomists in the UK, and what it means for the future of research in the field of macroeconomics. Despite substantial research on the REF in the sociology of academia, including its general impacts to academics (Martin and Whitley, 2010; Arnold et al., 2018), to date there is little work that has examined how the REF impacts upon sub-disciplines in economics.

Our Rebuilding Macroeconomics study group aimed at understanding how the REF can affect macroeconomics, both the field and individual macroeconomists, as well as its potential connections to a persistent monoculture in the sub-discipline. Noteworthy in our results was the evident dissonance between what macroeconomists (and economists in general) report about the REF affecting their research, and what REF sub-panels expect to provide with their peer-review exercise, and we now share them in more detail.

Our key findings are:

• macroeconomists report considerable concerns that the REF is impinging on their work directly or indirectly via different channels. In their views, the REF intensifies a “publication culture” that exists in macroeconomics and leads to an obsessive focus on journal articles. This undermines interdisciplinary research, shifts research efforts away from debates that researchers would like to contribute to, and reduces the diversity of ideas and methods in the field.

• These impacts may change depending on institutional affiliation and seniority level: early-career scholars and academics based in less prestigious universities report an “unfair playing field” when it comes to the UK’s research assessment system, and claim they need to be more strategic when it comes to research topics or outlets, even if this means a tick-box exercise to “get stars”. The role of research committees in universities is also important: many economists and economics departments report the use of journal lists ubiquitously across a wide set of activities, particularly in relation to determining what work is submitted to the REF (what universities commonly refer to as “mock REFs”), hiring and recruitment, appraisal, workload planning and promotion processes.

• The majority of scholars considered that individual research evaluation measures, and particularly journal lists, were more damaging than the REF. However, generally many scholars blur the boundaries between the REF and other evaluation tools. Indeed, there was considerable confusion as to the distinctions between them. While REF plays an important and distinct role unpicking the damage individuals consider the REF is doing to them, due to the interrelated interaction between individual and national metrics, a broader approach mitigating the negative impacts of the full set of research tools is necessary.

But what do these results mean for macroeconomics and for the ability of the REF to affect the field’s research?

REF exacerbating the epistemic cultures and structures of macroeconomics

First, that the REF is not a neutral tool, and has the ability to exacerbate the epistemic cultures that exist in macroeconomics – defined by its own participants as being hierarchical, elitist, competitive, and with a strong publication culture that values top journals. This also affects the balance of power in the sub-discipline: when research evaluation exercises such as the REF become mandatory, academic elites (identified by our participants as faculty members of top economics departments and those who sit in editorial boards of top journals) can have their prestige and power increased as their judgement is a proxy for quality. Our interviewees stressed how important is the judgement of editors, mock REF panels and REF sub-panels when deciding future research projects, choosing research outlets, collaborations and other strategies.

REF pushing for immediatism and distorting incentives

Second, the vast majority of our interviewees stressed the importance of having publications as a proxy for success suggests that the REF contributed to reinforce two main behaviours in the field of macroeconomics. First, it bolsters the epistemic culture of macroeconomics of rewarding the publication of journal articles (“only publications count”, and “you really have to publish” were mantras repeated frequently by our participants). Second, it helps to promote distorted incentives or repetitive research in the field, rather than pushed by quality and or novelty. Immediatism and quick turnover on journal publications is a shared goal by many interviewees. Further, the fact that publishing in journals outside economics is not acknowledged by the REF, our participants saw interdisciplinarity as “difficult”, “costly”, and a “research practice that does not pay off”, even though many acknowledged it is an important practice. Distorted research incentives also affect macroeconomists individually.

Thus, being strategic increasing one’s “REFability” is the norm for many, particularly those who are at earlier stages in their careers or are not affiliated with elite universities. Even though the majority of participants reported not making drastic changes in their research agenda, being successful in macroeconomics requires “adaptation” and “preparation” for the REF, which is linked to career progression prospects in many cases. Starting from the best journals possible when submitting articles and targeting well-ranked journals are common practices, despite some mentioning this does not have a direct connection with the REF.

REF affecting diversity of ideas and methods in macroeconomics

Several participants linked the lack of methodological and theoretical diversity in macroeconomics with journals, or how current editorial policies employed by top journals in the field can limit potential change: some mentioning of “openness to topics, but less so to methods” was also pointed out in our results. Further, a few interviewees mentioned how REF sub-panels (particularly Economics and Econometrics) are seen as closed and could better account for diversities in the economics profession: institutional, gender, theoretical and methodological. We claim this interchange between a diversity of ideas and diversity of individuals (gender and ECR) aligns strongly to substantive literature in economics looking at incentives.

A (potential) failure of communication between the REF and universities?

Finally, a contrasting discourse appears when it comes to assessing “research excellence” by universities (who select research outputs via “mock REFs”) and REF sub-panels. Uncertainty, lack of clarity and costly processes were common impressions shared by our interviewees when describing how mock REFs work, besides an insistence on using journal lists and, to a lesser extent, citation metrics. However, a document analysis of the final reports made by the Economics and Econometrics and Business and Management REF sub-panels in 2014, as well as their guidance for REF 2021 reveals some discrepancies in comparison to what was reported by our sample of macroeconomists about mock REFs/university practices.

The panels’ discourse broadly signals inclusivity and openness to diverse, novel and interdisciplinary forms of research more than often reported by our participants, opening up many questions regarding what higher education institutions follow and filter when it comes to succeeding on the REF, given they are the ones who will select research outputs and employ the required resources as per UKRI’s guidelines. academics reported a rather conservative approach from universities when trying to predict how sub-panels will behave. These discursive differences point out to unclear communication and lack of regulation from the UKRI to verify how universities (and departments) are implementing the required REF norms and practices. Also, it shows how REF sub-panels could signal their message more clearly, such as emphasising their acceptance of different forms of research outputs, and more diversity of research topics and methods.