Reading this Nautilus article about Julian Barbour led me recently to something I don’t think I’ve ever read before, Dirac’s 1963 Scientific American article The Evolution of the Physicist’s Picture of Nature. There is a very famous quote from this article that I’ve often seen:
It is more important to have beauty in one’s equations than to have them fit experiment
but I was unaware of the context of that quote, in which the famous part is prefaced by “I think there is a moral to this story, namely that…” The story that Dirac had in mind was that of the discovery of the Schrödinger equation. Famously, Schrödinger first wrote down a relativistic wave equation (now known as the Klein-Gordon equation). This equation is what one quickly gets if one follows de Broglie’s idea that matter is described by waves, and uses the relativistic energy-momentum relation. Here’s the full story, as told by Dirac, giving his famous quote in context:
I might tell you the story I heard from Schrödinger of how, when he first got the idea for this equation, he immediately applied it to the behavior of the electron in the hydrogen atom, and then he got results that did not agree with experiment. The disagreement arose because at that time it was not known that the electron has a spin. That, of course, was a great disappointment to Schrödinger, and it caused him to abandon the work for some months. Then he noticed that if he applied the theory in a more approximate way, not taking into ac count the refinements required by relativity, to this rough approximation his work was in agreement with observation. He published his first paper with only this rough approximation, and in that way Schrödinger’s wave equation was presented to the world. Afterward, of course, when people found out how to take into account correctly the spin of the electron, the discrepancy between the results of applying Schrodinger’s relativistic equation and the experiments was completely cleared up.
I think there is a moral to this story, namely that it is more important to have beauty in one’s equations than to have them fit experiment. If Schrodinger had been more confident of his work, he could have published it some months earlier, and he could have published a more accurate equation. That equation is now known as the Klein-Gordon equation, although it was really discovered by Schrödinger, and in fact was discovered by Schrödinger before he discovered his nonrelativistic treatment of the hydrogen atom. It seems that if one is working from the point of view of getting beauty in one’s equations, and if one has really a sound insight, one is on a sure line of progress. If there is not complete agreement between the results of one’s work and experiment, one should not allow oneself to be too discouraged, because the discrepancy may well be due to minor features that are not properly taken into account and that will get cleared up with further developments of the theory.
There’s another remarkable aspect of this Scientific American article, something about it that would be completely inconceivable today: they write down three equations, including both the famous non-relativistic Schrödinger equation for the Coulomb potential, as well as the relativistic Klein-Gordon version.
Dirac notes that Schrödinger found his formulation of quantum mechanics in a very different way than Heisenberg found his:
Heisenberg worked keeping close to the experimental evidence about spectra that was being amassed at that time, and he found out how the experimental information could be fitted into a scheme that is now known as matrix mechanics. All the experimental data of spectroscopy fitted beautifully into the scheme of matrix mechanics, and this led to quite a different picture of the atomic world.
whereas
Schrödinger worked from a more mathematical point of view, trying to find a beautiful theory for describing atomic events, and was helped by De Broglie’s ideas of waves associated with particles. He was able to extend De Broglie’s ideas and to get a very beautiful equation, known as Schrödinger’s wave equation, for describing atomic processes. Schrodinger got this equation by pure thought, looking for some beautiful generalization of De Broglie’s ideas, and not by keeping close to the experimental development of the subject in the way Heisenberg did.
At the end of the article, Dirac makes the case that progress in fundamental physics may not come from a theorist like Heisenberg finding a scheme to match experimental results, but from a theorist like Schrödinger pursuing mathematical beauty:
It seems to be one of the fundamental features of nature that fundamental physical laws are described in terms of a mathematical theory of great beauty and power, needing quite a high standard of mathematics for one to understand it. You may wonder: Why is nature constructed along these lines? One can only answer that our present knowledge seems to show that nature is so constructed. We simply have to accept it. One could perhaps describe the situation by saying that God is a mathematician of a very high order, and He used very advanced mathematics in constructing the universe. Our feeble attempts at mathematics enable us to understand a bit of the universe, and as we proceed to develop higher and higher mathematics we can hope to understand the universe better.
This view provides us with another way in which we can hope to make advances in our theories. Just by studying mathematics we can hope to make a guess at the kind of mathematics that will come into the physics of the future. A good many people are working on the mathematical basis of quantum theory, trying to understand the theory better and to make it more powerful and more beautiful. If someone can hit on the right lines along which to make this development, it may lead to a future advance in which people will first discover the equations and then, after examining them, gradually learn how to apply them. To some extent that corresponds with the line of development that occurred with Schrodinger’s discovery of his wave equation. Schrödinger discovered the equation simply by .looking for an equation with mathematical beauty. When the equation was first discovered, people saw that it fitted in certain ways, but the general principles according to which one should apply it were worked out only some two or three years later. It may well be that the next advance in physics will come about along these lines: people first discovering the equations and then needing a few years of development in order to find the physical ideas behind the equations. My own belief is that this is a more likely line of progress than trying to guess at physical pictures.
The context for his famous quote is thus an argument for pursuing fundamental physics by looking for mathematical beauty, not giving up on a beautiful equation just because it doesn’t seem to fit experiment. As in Schrödinger’s case, more effort may be needed to understand the actual relationship of the equation to reality.
Besides this argument, which I’ve always been well aware of and sympathetic to (despite not knowing the context in which Dirac was making it), there’s something else I found very striking about the 1963 article. Dirac begins by explaining that the four-dimensional Lorentz symmetry of relativity is in a sense broken by the choice of a way of describing the state of the world:
What appears to our consciousness is really a three-dimensional section of the four-dimensional picture. We must take a three-dimensional section to give us what appears to our consciousness at one time; at a later time we shall have a different three-dimensional section. The task of the physicist consists largely of relating events in one of these sections to events in another section referring to a later time. Thus the picture with four dimensional symmetry does not give us the whole situation. This becomes particularly important when one takes into account the developments that have been brought about by quantum theory. Quantum theory has taught us that we have to take the process of observation into account, and observations usually require us to bring in the three-dimensional sections of the four-dimensional picture of the universe.
The special theory of relativity, which Einstein introduced, requires us to put all the laws of physics into a form that displays four-dimensional symmetry. But when we use these laws to get results about observations, we have to bring in something additional to the four-dimensional symmetry, namely the three-dimensional sections that describe our consciousness of the universe at a certain time.
Dirac also refers to work on canonical formulations of general relativity, aimed at quantizing gravity:
… if one insists on preserving four-dimensional symmetry in the equations, one cannot adapt the theory of gravitation to a discussion of measurements in the way quantum theory requires without being forced to a more complicated description than is needed bv the physical situation. This result has led me to doubt how fundamental the four-dimensional requirement in physics is. A few decades ago it seemed quite certain that one had to express the whole of physics in four-dimensional form. But now it seems that four-dimensional symmetry is not of such overriding importance, since the description of nature sometimes gets simplified when one departs from it.
Thinking about twistor unification has led me to some similar thoughts: a Euclidean formulation of quantum theory requires picking a choice of imaginary time direction and breaking SO(4) symmetry in order to define states. Dirac thinks of our consciousness as giving us access to the state of the universe defined on a 3d slice, but the twistor point of view is even more directly related to our conscious experience. A point in space time is defined by the sphere of light rays through the point, and it is this sphere that our vision gives us direct access to, with 3d space something we make up out of these spheres.
A common argument against Dirac’s point of view is that it’s engaging in mysticism. For another recent article that touches in a different way on the mystical nature of a discovery about fundamental physics, see this interview with Frank Wilczek, where he tells this story:
The mystical moment came while I was visiting Brookhaven National Laboratory, on Long Island. Somehow—I don’t remember how, exactly—I wound up alone, standing on a jerry-rigged observation platform above a haphazard mess of magnets, cables, and panels. This was a staging area for assembling detectors and renovating pieces of the main accelerator there, the Alternating Gradient Synchotron (AGS). I must have gotten separated from my host for a few minutes. In any case, there I was, alone inside an aircraft-hangar-sized metallic box, staring down at the kind of equipment that people use to explore the fundamentals of Nature experimentally.
And then it happened. It came to me, viscerally, that the intricate calculations I’d done using pen and paper (and wastebasket) might somehow describe this entirely different realm of existence—namely, a physical world of particles, tracks, and electronic signals, created by the kind of machinery I was looking at. There was no need to choose, as philosophers often struggled to do, between mind or matter. It was mind and matter. How could that be? Why should it be? Yet I somehow, I suddenly knew that it could be so, and should be so.
That was my mystical experience. I warned you that it was ineffable.
