Jesper Grimstrup and Jarl Sidelmann have an interesting new paper up on the arXiv, entitled Competition and survival in modern academia: A bibliometric case study of theoretical high-energy physics. It uses bibliometric data to study career paths in hep-th, especially how many people who start out in the field are still in it at various later times.
If you think that things are going fine in hep-th, this kind of study is of limited interest. If you think the field is in trouble, it’s of interest as pointing to one source of the trouble. The problem with this kind of thing though is that on the whole the people making decisions about what to do are the “survivors”, for whom the current system has worked out just fine. They’re the least likely people to think there’s a crisis or to see any reason to do anything about it. As for the job situation (which has been terrible since 1970), I can report that when one doesn’t have a permanent job this seems to be an important and serious problem, but once one does have a permanent job all of a sudden it seems much less important.
What has struck me most in recent decades about hep-th is not the bad job environment, but the monotone-decreasing number of interesting new ideas, now so small that I don’t think “intellectual collapse” is an unfair characterization of what’s happened. I started carefully following the latest preprints in the field more than 40 years ago, pre-arXiv, when they were collected physically at a “preprint library” in one’s institution. Most preprints in hep-th have always been minor advances, not of much interest unless you’re working on much the same problem, but in the past there were always a significant number with something really new and significant to report. The arrival in the preprint library of something new from Witten or any number of other well-known figures in the field was an event, and there also was a steady stream of new ideas coming from people not so well-known. In recent years the situation has been very different, with something worth reading appearing in the arXiv hep-th section less and less often, to the point where it’s a rare occurrence.
This slow death of the field I believe is a very real phenomenon, although I’m not sure how one could quantify it. There are multiple reasons for why it has happened, some of which are just facts of life (the SM is too good, no unexpected experimental results). I do think though that one reason is the one the authors here are trying to get at: decades and decades of a difficult job situation where the only viable way to win the game of survivor is to publish lots of papers in a dwindling number of accepted research programs. This is one problem that the field actually could do something about, but chances of that happening seem remote.
Peter,
The abstract says “early career collaboration and productivity rates have emerged as reliable predictors of author survivability…” Do you think too much early career collaboration can also contribute to the lack of new ideas by rejecting novel ideas that might not receive the approval, and hence collaboration, of a group?
Sina,
Early career collaboration is definitely good for ones career, making contacts that will be valuable and getting more papers written. But if you are someone who wants to pursue a new idea that is significantly different than what other people are working on, finding collaborators is going to be much more difficult.
Fig. 6 of https://direct.mit.edu/qss/article/2/1/225/99129/Gender-issues-in-fundamental-physics-A shows the same phenomenon, and that it’s not limited to theory: a similar situation is found in experiment and astro/cosmo: decades ago a larger fraction of PhD-level authors found permanent positions.
About the death of the field: I have bibliometric indicators that should be able to notice and quantify it. Unfortunately, every year these indicators say it’s too early to write a dedicated bibliometric paper.
“This is one problem that the field actually could do something about, but chances of that happening seem remote.”
Remote indeed. But, even if it happens, so what? We would get some flourishing of some alternative ideas… but after some time, they will also hit the same issue that you mention before: the SM is too good, no unexpected experimental results. The facts of life are that this just seems like a bad time for being a theoretical physicist in the style of the early 20th century to the 1960s. It’s over… for at least a couple of decades if not more.
Maybe one can go to mathematical physics and try to explore more the mathematical basis of our current theories to gain deeper insights. But this also has already been done to quite some depth in some cases. The remaining problems are just the very, very hard ones, difficult to build a career on that if you are a young researcher.
I’m pretty sure there have been periods like this in the past. After all, newtonian physics was the dominant paradigm for how long, 200 years? Sure, some of the clues that something was not covererd by it were already known in the early 19th century, but it still took quite a while.
Maybe we thought because of our super technological and interconnected world of today, with so many more people dedicated to the problems will help avoid the depression… but the thing is that the problems and theories also are quite more complex and difficult than those before.
Who knows, maybe it’s time for AI to take the torch (not the current one, certainly, but maybe the one from the next decades). So, yeah, I’m extremely pessimistic.
The descriptions “intellectual collapse” and “slow death of the field” seem appropriate.
One high-level way to quantify this is by looking at Nobel Prize data: https://blog.jakobschwichtenberg.com/p/is-scientific-progress-slowing-down
There are definitely many contributing factors.
An increasingly competitive job market and funding environment means more and more people play it safe to have any realistic career prospects. An incremental project will earn you almost certainly a solid number of citations. Plus the project can be finished in a predictable time frame. Looking for interesting new ideas is far more risky and strongly discouraged by most supervisors. Hence fresh ideas from young researchers are increasingly rare.
The steady decline of single author papers is another factor. Co-authoring papers is a smart strategy to pad your publication list. But it’s often not the best approach to develop unique ideas.
I think mental constructs like “the field ” are not really useful here. No one is in charge. You have hundreds of self interested actors optimizing over what you pointed out (SM too good, little experimental data). The current situation is a local optimization at some level of resolution. If you aren’t able to produce results then highly complex untestable theories curated by mutually back scratching elites is a sensible small scale survival strategy. The only thing that will change it is new experimental data or a real theory innovation from outside the mainstream. Don’t count on the mainstream to actively fund its own demise.
Jakob,
Nice link and study. It is in line with many other studies . I am noting that all these studies try to avoid our sacro-saint mathematic. But mathematic suffer the same decline. I remember one of a few study stating there was around 800 PHD in mathematic at Einstein time , now (it was around year 2000) approximately 100,000. We have an exponential growth in the number of mathematicians, and the number of papers published, but progress growth is linear, with not so major fundamental breaktrough and revolutions. We can think of an incredible number of fondational problems (like the 6 remaining Millenium problems) that are not resolved, many are more than a century old.
As a private sector (MSc Quantum Physics) guy, looking in from the outside, the solution is simple. The university system is socialism. The solution is actually Trumpian.
Fire everyone who is a bullshitter, and demand non-bullshit results. With brave critics, such as Peter W. and perhaps even Sabine H, put in charge of refereeing all this, based on their known scepticism of the failed status quo.
Of course, nearly all academics will recoil in horror at this, because such a capitalist approach directly threatens their job security, and, at the very least, forces them to stop supporting failed research programs: to a large extent, anyway.
Of course, my solution will be dismissed as crazy. But, from my perspective, academia is increasingly dysfunctional and failing to do its job.
The comment from Jim Eadon is typical of many that have come in in recent years, all of which I’ve moderated out. I’m letting this one through today because among the news I’ve seen that gets past my algorithm (stop reading at the words “Trump” or “Musk”), it looks like what’s coming is open warfare on federal funding of scientific research at universities. This is beginning with biomedical research with a supposed huge cut to the NIH ICR rate.
I have no idea what’s going to actually happen, except that it’s not going to be redirecting spending to higher quality research. The theoretical research I have been critical of is not something that involves a lot of money, it will survive with a smaller fraction of DOE/NSF funds flowing to the universities involved. What will get hit is the much more expensive experimental research that keeps the theorists honest.
Closing comments now, since I don’t want to host a discussion here of this at this time, and you can find better places for such a discussion. In any case, events are in the process of overtaking any sensible discussion of how to improve the current system. The US is now under the control of oligarchs and those who believe the way forward is to just start smashing anything with an aspect that fits in their long list of resentments. I don’t see how this ends well, hope that I’m wrong.