Research Excellence Framework: outcomes of consultation and next steps

To:

Heads of HEFCE-funded higher education institutions, Heads of universities in Northern Ireland

Dear Vice-Chancellor or Principal

1.   From November 2007 to February 2008, we consulted on the future research assessment and funding framework - the Research Excellence Framework (REF) - that will be the successor to the 2008 Research Assessment Exercise (RAE).

2.   We received 274 responses to the HEFCE consultation paper (338 including responses sent to the other funding bodies). Respondents included all the relevant major national representative and stakeholder bodies, most higher education institutions (HEIs) and many subject bodies. We held three consultation events for HEIs in England, attended by a total of 111 institutions, and a fourth event was held by the Scottish Funding Council in Edinburgh. We have also had informal discussions with a range of national representative, research and subject interest bodies and some HEIs.

3.   We welcome the constructive debate about the future of research assessment and funding, and would like to thank institutions for their responses. A full summary of the responses is now available on our web-site. A brief summary of the key points raised is at Annex A of this letter.

4.   In the light of responses to the consultation, our Board and the Secretary of State for Innovation, Universities and Skills have agreed that we will proceed to develop the new framework with two modifications to the proposals that we had set out in the consultation paper:

1. Responding to widespread concerns about what would be the likely consequences of two discrete assessment regimes (one for the 'science-based' disciplines and one for other subjects), we will now develop a unified framework to cover all disciplines. This integrated system will combine bibliometrics and other quantitative indicators with light-touch peer review within a variable geometry of assessment. Bibliometric indicators of research quality will be a key element in quality assessment wherever this is appropriate, with light-touch review of research outputs operating where it is not. In all cases we will use other indicators of quality and impact appropriate to the disciplinary area and determined after advice from expert panels.
2. To allow time for the development of this more flexible approach and to enable the new bibliometric indicator to be thoroughly piloted, the timetable for designing the new framework will be extended by 12 months.

Key features of a unified framework

5.   We have now established the shape of a unified framework for quality assessment and funding across all disciplines. In each subject, quality will be assessed using some or all of:

• bibliometric indicators of quality or expert review of outputs (and the possibility of some combination of these)
• other quantitative indicators
• supplementary qualitative information.

6.   The precise combination used for each subject group will vary, with regard to advice from the expert panels and to further consultation as appropriate. We envisage that bibliometrics and other quantitative indicators will be used to the extent that they are applicable to each subject. They will be supplemented or replaced by peer review of outputs and supplemented by other qualitative elements where this is necessary to produce a robust assessment that fully reflects the range of research approaches and outputs characteristic of each subject group. We envisage that this will result in a spectrum of assessment approaches along a continuum, as follows:

1. Some subjects may be assessed through bibliometrics in combination with other quantitative indicators only (where this approach is sufficiently robust to be used without the need for qualitative elements or expert review of outputs).
2. For subjects where bibliometrics and other quantitative indicators are partially informative, they would be used in combination with qualitative elements and possibly some expert review of outputs.
3. For subjects where bibliometric indicators are not sufficiently mature to be informative, expert review of outputs would be used in combination with other applicable indicators and qualitative information.

7.   Quality assessment in all subjects will be overseen by expert panels. Within an operational framework to be determined, the panels will advise on:

• selecting indicators appropriate to the characteristics and diversity of research in their subject field
• interpreting and combining these indicators (together with peer review judgements as appropriate) to produce an overall quality profile that can be used to inform the allocation of the Council's grant for research
• ensuring that quality assessment is conducted for all subjects with equal rigour and applying common quality standards having regard to international benchmarks.

8.   Assessment across all subjects will take place during calendar year 2013 to drive quality-related research funding from academic year 2014-15. For subjects where bibliometric indicators play a leading role in quality assessment, these will start to influence HEFCE funding phased in from 2011-12. A full exercise to produce bibliometric indicators of research quality will therefore take place, for appropriate subjects, in 2010.

Next steps

9.   We will now undertake a programme of work to develop the operational detail of the new framework, as follows:

1. We will continue as planned to pilot the bibliometric indicator in 2008. Our chosen approaches to data collection and analysis will be decided during 2009, with sufficient notice provided to institutions to prepare for the full bibliometrics exercise mentioned in paragraph 8 in 2010. Some details of what we hope the pilots will achieve and how we plan to conduct them are set out at Annex B. We have already set in hand the process of establishing the pilot group by writing to institutions that have offered to take part.
2. In parallel to the bibliometrics pilot, we will develop proposals for light-touch approaches to expert review of research outputs, to be applied in subjects where bibliometric indices do not yet provide a sufficiently robust quality measure.
3. Also in parallel, we will develop our approach to choosing and using other quantitative and qualitative indicators, including approaches to capturing the significance and impact of research activity where this cannot be reliably established through bibliometric analysis; and for other aspects of the framework including the subject structure (recognising that the structure proposed in our recent consultations requires further thought) and the frequency of assessment.
4. We will maintain regular discussions with HEIs, national bodies and subject groups on these issues. We will work closely with the other higher education funding bodies and a range of stakeholders, including other organisations that fund research.
5. We will undertake a full sector impact assessment of the REF, informed by the bibliometrics pilot, to understand the potential accountability burden, behavioural incentives, equal opportunities and sustainability implications.

10.   Following the pilot and development of proposals for other aspects of the framework, we will consult the sector formally from spring 2009, leading to decisions on the way forward by summer 2009. This approach brings the timetable for developing the different aspects of the framework across all disciplines closer together.

11.   Our overall timetable is now as follows:

April 2008 - spring 2009	Undertake pilot of bibliometric indicators. Develop proposals for other aspects of the REF across all subjects. Assess the impact of our proposals in terms of equality and diversity, sustainability and regulatory burden.
December 2008	Results of 2008 RAE published.
Spring - summer 2009	Consult on all main features of the REF including operational details of the bibliometrics process, use of other quantitative indicators, subject boundaries and procedures for light-touch peer review.
Late summer 2009	Decide on the main operational features of the framework, including what statistical and bibliometric datasets will be used.
Calendar year 2010	Establish expert panels for all subjects. Undertake bibliometrics exercise in appropriate subjects. Consult on choice and use of assessment components for each subject group.
2011-12	Metrics begin to inform an element of HEFCE funding in appropriate subjects.
Calendar year 2012	Submissions to 2013 light-touch peer review process
2013	Undertake full assessment process for all subjects including light-touch peer review.
2014-15	HEFCE research funding for all subjects driven by REF from this time onwards.

Further information

12.   Information about the bibliometrics pilot and the development of proposals in other areas will be posted regularly on our web-site. We have also set up a REF-NEWS jiscmail mailing list for anyone specifically interested in updates on REF developments (for example, to be alerted to any new materials posted on the web-site or forthcoming events). To subscribe to REF-NEWS, follow the link on the REF page of the HEFCE web-site as above.

Yours sincerely

Professor David Eastwood
Chief Executive

Share this:

• 
• 
• 
• 
• 
• 
• 

---