30/12/2020

Green University Rankings: Research Note 30 December 2020

Background

During the last decades, attempts to measure the performance of universities have multiplied. These performance scores, rankings, or league tables as they are called in the UK, are used by universities for their brand and reputation management, by students to determine their university choice, by academics to decide their career moves, and by several other stakeholders for different purposes.

Lately, there has been criticism of university rankings, mainly for not being well governed or transparent, and lacking rigor, as well as external validity (Gadd, 2021). We fully endorse their warning against the uncritical use of university rankings and research metrics in the higher education sector. The authors, however, based their arguments on deficiencies in the handling of data on research output, not in the lack of measurement of universities teaching performance or other activities. In order to address this, we argue that considerable additional efforts must be made to harmonize teaching effectiveness and outcome measures. 

Lately, universities have been trying to justify their contribution to the public good and global challenges. In this note, we wish to focus on a few specialized rankings focussing on environmental and social performance of universities. 

In 2010, the Integrated Laboratory and Research Center (ILRC) at the University of Indonesia created the Greenmetric ranking. Here we briefly evaluated the Greenmetric ranking and examined whether it is correlated with any other “green” or social impact rankings on the basis of the information publicly available on the website.

GreenMetric Logo


Universities’ Social & Environmental Impact

Since 2019 Times Higher Education (THE) published a university impact ranking (THE-IR) using universities’ self-declared activities around the United Nations’ Sustainable Development Goals (SDGs) as a ranking criterion. As part of the THE-IR, a separate “climate action” score and ranking are reported. It is, therefore, possible to compare the Greenmetric ranking with THE impact or climate action ranking, expecting a degree over overlap or correlation between them.

As we can see in the table below a total of 57 universities in the top-200 Greenmetric ranking are included in any of THE rankings. Moreover, there is very little overlap between THE main ranking and THE-IR: only 7 universities are included in THE main ranking. Similarly, only 10 universities are included in THE social impact ranking, and only 8 in the climate action dimension. In all cases, there is insufficient overlap to calculate the significance of the correlations due to the small sample size.

Overlap between Green Rankings

We evaluated the Greenmetric ranking using the four dimensions proposed by the INORMS working group: good governance, transparency, rigor, and measure-what-matters, or validity. As to good governance, of all the reputable rankings only Times Higher Education performs an independent audit of all their methods and data. Greenmetrics does not disclose any conflict of interest when doing consulting for its client universities.

As to continuous improvement, the 2020 web page refers to the first version of 2010: “While we have put every effort into the design and implementation of the questionnaire, we realize that this first version is bound to have shortcomings. Therefore, we will be reviewing the criteria and the weightings continuously to reflect input from participants and state of the art developments in the field. We welcome your comments and suggestions.” Have no changes or improvements been made since 2010?

Greenmetric explains its methodology used to produce its scores and rankings, although not always to the extent that it can be replicated. GreenMetric “ranks universities worldwide on sustainability issues, based on their self assessed input for the criteria set and extensive independent research and survey responses conducted for the UI GreenMetric World University Ranking.” (FAQ Greenmetric, 2020). It is unclear, however, whether a hard or a soft concept of sustainability is being applied. In the first case, no natural capital should be depleted, while in the second case loss of one type could be compensated by another. Possibly, more information is available in the book on the Greenmetric ranking development from 2010 to 2017, but what is published on the web page is hardly satisfactory (Sari, et al. 2019).

Greenmetric’s major flaws however regard its rigor and validity. It is unclear whether any methods are used to test the statements made by the universities. On assurance or validation processes its statement is evasive and based on a promise for the future: “We have held of on this for the first version at least. We felt that we had to balance scientific respectability with the pragmatic realities of the extra effort involved. However, we are looking at ways of working towards some sort of validation process for future versions.” (FAQ Greenmetric, 2020). Instead of vague promises, Greenmetric could hire a reputable accounting firm to go over all its methods and data handling like Times Higher Education has done since 2016.

Moreover, the inclusion of specific questions is not theoretically justified publicly, nor are statistical tests performed to see whether particular indicators are redundant. It is unclear how the error of spurious precision has been avoided. The process to establish specific weights is not explained: “Each of the criteria will be categorized in a general class of information and when we process the results, the raw scores will be weighted to give a final calculation.” (FAQ Greenmetric, 2020). In order for the Greenmetric to be credible at least summary explanations on how errors of composition, weighing, and spurious precision are avoided (Soh, 2017).

Conclusion

In conclusion, much care needs to be taken to take Greenmetric results at face value. Many universities, should be a bit more careful in celebrating their high positions in Greenmetric ranking, and rather make a greater effort to participate in more robust ranking exercises. Meanwhile, Greenmetric would do well to address the issues regarding rigor and transparency mentioned here, which are not difficult to fix. It would be sad if it would lose interest from a group of over 1,000 universities that have participated in its surveys during the last 10 years.

References



Posted by at

No comments:

Post a Comment

Note: only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)