Methodology: How and Why We Rank by Influence
High-Stakes College Rankings
College and university rankings carry enormous weight. A school’s reputation, enrollment numbers, hiring, firing, and revenue stream are all at stake.
A school’s rise in a leading ranking is experienced as recognition and praise, and can translate into greater prestige for alumni diplomas, more grant money for researchers, and a larger pool of top students from which admissions creates the next Class of 20XX.
Conversely, a school’s decline in a leading ranking is experienced as a slight and even humiliation, translating into powerful negatives for the school. As educational journalist Jeffrey Selingo writes in his book Who Gets In and Why, “When a top school drops in the rankings, it’s forced to admit more students in subsequent years and is more generous with financial aid to make up for falling yield rates, according to a study for the National Bureau of Economic Research.”
All this drama leads to perverse incentives. Schools are motivated to “game” the rankings, introducing superficial or cosmetic changes that elevate their ranking but do little to provide a better education for students or a more fruitful work environment for faculty.
Annual announcements of college rankings come with so much fanfare that they make the news. Unfortunately, students often go missing amid all this excitement, gaining few clear insights into which schools or programs are indeed best for them.
This is why we created AcademicInfluence.com. Our rankings are objective and non-gameable, giving students the reliable information they need to draw insightful comparisons among schools. The stakes are too high to settle for anything less.
The Need for InfluenceRanking™
Everyone is drowning in data. That’s especially true of students struggling to make sense of the many conflicting college and university rankings proliferating across the Web. In response, a team of data scientists and academics decided to rethink college and university rankings entirely from the ground up. Taking the view that higher education is ultimately about ideas and the people who promote them, we sought to answer these questions:
- Which factor speaks the most truth about the stature of an idea or of its promoter?
- How can we measure that factor while avoiding “gamed” or biased sources?
- How can we form rankings from such a measure that people can understand and use to answer their questions?
In biology, genetic factors that confer the greatest advantage to their possessors express themselves more widely. Similarly, on the internet, viral memes that resonate with people express themselves more widely.
In all such propagation phenomena—and thus at the core of all ideas and information everywhere—the factor that determines whether something persists into the future is influence, the level of attention and penetration that ideas and entities achieve in the world.
For higher education and the world of knowledge, influence always starts with specific disciplines and specific contributions to those disciplines. Einstein did not start by being influential in the world of learning as such. He became influential through his specific contributions to physics.
From influence in specific disciplines, it is possible to extend influence to wider spheres of research and wider professional impact. Faculty and alumni with influential ideas thus become the foundation of influential degree programs, and a critical mass of such influential degree programs make for influential colleges and universities.
Such is the influence we seek to measure through InfluenceRanking, the influence-based ranking engine developed by our AcademicInfluence.com team of data scientists and academics.
What InfluenceRanking™ Measures and How
The more data points, the more data sources for those data points, and the greater variety of those data sources, the less prone the data is to staleness, bias, and bad actors manipulating it.
This is why InfluenceRanking consults the largest repositories of open source data on the Web and entrusts their analysis to machine-learning technology developed with funding from the Defense Advanced Research Projects Agency (DARPA). These databases contain billions of continuously updated data points, which by their nature resist biased manipulation and by their scope contain built-in protections against wholesale gaming.
Our technology thus uses artificial intelligence (AI) to
- aggregate scholarly and academic citations from databases such as Wikipedia and Crossref,
- evaluate links and other semantic information contained in these databases, and
- weigh their merits against other information sources such as periodicals, journals, and global media outlets.
Importantly, our machine-learning technology doesn’t simply scrape the web for mentions. Influence at AcademicInfluence.com is therefore not a popularity measure. Rather, the AI underlying our InfluenceRanking engine drills down deeply to
- identify people and institutions, locating mentions of them across the Web in 15 different languages,
- associate those mentions with records of achievement in specific fields of study,
- map the people that make up an institution, keeping track of independent work done by those people as well as their affiliations with other institutions, and
- aggregate the influential output of people at an institition, thereby gauging the institution’s influence by disciplinary programs and overall.
Our machine-learning algorithms improve over time at measuring influence of people and institutions. As independent corroboration that our technology is indeed getting better at gauging influence, we employ web traffic analysis from third parties (including Semrush.com and Ahrefs.com) to track the organic search traffic leading to web domains, each domain’s keyword footprints, the number of referring domains, and the authority of those referring domains.
As a further part of this corroboration effort, we also pull in additional secondary sources such as the Integrated Postsecondary Education Data System (IPEDS) data. These data allow for specialized rankings, including those that track costs, test scores, and acceptance rates. Such specialized rankings are often of interest in their own right and may then be correlated with our influence-based rankings.
Finally, because the InfluenceRanking AI technology is database driven, its results can be updated as quickly as the databases on which it depends. InfluenceRanking is thus capable of generating hourly result updates. This rate of updates turns out to be excessive. To ensure results can be accurately compared and cited, AcademicInfluence.com limits itself to quarterly updates. Quarterly updates incentivize the AcademicInfluence.com team to get the InfluenceRanking engine, along with the databases on which it depends, into the best shape possible for the next update.
For more technical detail about this ranking technology, see ”The InfluenceRanking™ Engine: The Nuts and Bolts of Our Ranking Technology.”
What Constitutes “Best”?
What is the best school or degree program? Answering this question depends on the meaning assigned to “best.”
For higher education rankings, “best” may mean having the most knowledgeable departmental faculty, the most accomplished professors, or the most noteworthy alumni. For students passionately interested in a specific discipline, the best school is likely to be one widely recognized for excellence in that discipline. Other students may put a premium on economic, geographic, and academic imperatives, such as educational outcomes, career prospects, and lifetime earning potential. Many dimensions of institutional excellence can factor into “best,” and competing ranking systems prioritize these dimensions according to their own internal—and often impenetrable—criteria and weighting schemes.
Our extensive experience with and research into academic rankings, however, suggests that influence, measured properly, gets at the heart of what is truly best in education while also offering a portal into countless ways of viewing and analyzing the different dimensions of educational value. This is why AcademicInfluence.com allows users to customize and personalize their own rankings based on the factors that matter most in their search for the “best school” (see our Custom College Rankings), while maintaining influence as the master key (especially in the form of Concentrated Influence).
Influence Isn’t Always Pretty
Because our ranking methodology is not just objective but also driven by algorithms and dababases, its results emerge from a computational process and therefore not by direct human intervention. This is a strength of our approach, removing the hand-waving and gameability that are so common in higher-education ranking approaches. But this absence of direct human intervention can also lead to counterintuitive results. When an influential person or institution ranks highly at AcademicInfluence.com, it’s because our InfluenceRanking engine has picked up on some signal indicating influence. But sometimes it’s not immediately clear what that signal is. Fortunately, we are often able to “look under the hood” of our InfluenceRanking engine and see why it delivered the results that it did.
Our machine-learning approach to academic rankings is necessarily morally neutral. In the case of influential individuals, some exhibit significant impact in a field despite an apparent lack of proper education or socialization or achievement in it. But even in such cases, our InfluenceRanking engine is detecting a real signal (it’s not just noise). For more on this important issue, see our article ”Influence, Infamy, and the Case of Osama bin Laden.”
With our InfluenceRanking engine, the onus is on all to use their common sense and practical wisdom in interpreting its results. Think of the results from our InfluenceRanking engine as a starting point and not as an end point for inquiry. An otherwise influential school may still leave you with an unhappy educational experience. And a widely influential thought leader may engage in thinking that is now archaic. In the end, the users of our InfluenceRanking engine remain the arbiters of the persons and institutions whose influence they must gauge in light of their own system of values. Simply put, influence as presented at AcademicInfluence.com is objective, but its value and relevance to you is not.
AcademicInfluence.com’s InfluenceRanking engine opens up exciting, new means for students, teachers, and researchers to find answers to what constitutes true value in education.
As our AcademicInfluence.com website has matured and we have continued to publish more rankings of persons and institutions, influential academics across the range of disciplines tell us that they find our rankings remarkable for their accuracy, insights, and serendipitous turns. Some have even reached out to connect with younger thought leaders, whose influence in their field is growing but, without our technology, remained unsuspected.
Such connections thrill us because our mission is to connect learners to leaders, and that inludes leaders (who are always also learners) to other leaders. But that mission begins with ensuring that our method for making these connections is sophisticated, scientific, and sound. Our InfluenceRanking engine is by now a mature technology with years of research and effort behind it. It has proven itself. Whatever your educational odyssey, your discoveries at AcademicInfluence.com will enhance it, helping you to realize your academic goals and aspirations.