More recent attempts to estimate the energy of empty space have only exacerbated the problem, producing values that are much, much higher—so high, in fact, that they cannot be considered anything but absurd. For instance, physicists first assumed naïvely that they could sum up the zero-point energies up to the scale where our theory of gravity breaks down. That is, to that point at which the universe was so small that one needs to have a quantum theory of gravity (a theory that does not exist currently). In other words, the hypothesis was that the cosmological constant should correspond to the cosmic density when the universe was only a tiny fraction of a second old, even before the masses of the subatomic particles were imprinted. However, when particle physicists carried out that estimate, it resulted in a value that was about 123 orders of magnitude (1 followed by 123 zeros) greater than the combined cosmic energy density in matter and radiation. The ludicrous discrepancy prompted physics Nobel laureate Steven Weinberg to dub it “the worst failure of an order-of-magnitude estimate in the history of science.” Obviously, if the energy density of empty space were truly that high, not only would galaxies and stars not have existed but also the enormous repulsion would have instantly torn apart even atoms and nuclei. In a desperate attempt to correct the guesstimate, physicists used symmetry principles to conjecture that adding up the zero-point energies should be cut off at some lower energy. Dismally, even though the revised estimate resulted in a considerably lower value, the energy was still some 53 orders of magnitude too high.

Faced with this crisis, some physicists resorted to believing that a yet-undiscovered mechanism somehow completely cancels out all the different contributions to the energy of the vacuum, to produce a value of exactly zero for the cosmological constant. You’ll recognize that mathematically speaking, this is precisely equivalent to Einstein’s simple removal of the cosmological constant from his equations. Assuming that the cosmological constant vanishes means that the repulsive term need not be included in the equation. The reasoning, however, was completely different. Hubble’s discovery of the cosmic expansion quickly subverted Einstein’s original motivation for introducing the cosmological constant. Even so, many physicists regarded as unjustified the assignment of the specific value of zero to lambda for the mere sake of brevity or as remedy to a “bad conscience.” In its modern guise as the energy of empty space, on the other hand, the cosmological constant appears to be obligatory from the perspective of quantum mechanics, unless all the different quantum fluctuations somehow conspire to add up to zero. This inconclusive, frustrating situation lasted until 1998, when new astronomical observations turned the entire subject into what is arguably the most challenging problem facing physics today.

Since Hubble’s observations in the late 1920s, we knew that we live in an expanding universe. Einstein’s theory of general relativity provided the natural interpretation of Hubble’s findings: The expansion is a stretching of the fabric of space-time itself. The distance between any two galaxies increases just as the distance between any two paper chads glued to the surface of a spherical balloon would increase if the balloon were inflated. However, in the same way that the Earth’s gravity slows down the motion of any object thrown upward, one would anticipate that the cosmic expansion should be slowing, due to the mutual gravitational attraction of all the matter and energy within the universe. But in 1998 two teams of astronomers, working independently, discovered that the cosmic expansion is not slowing down; in fact, over the past six billion years, it has been speeding up! One team, the Supernova Cosmology Project, was led by Saul Perlmutter of the Lawrence Berkeley National Laboratory, and the other, the High-Z Supernova Search Team, was led by Brian Schmidt of Mount Stromlo and Siding Spring Observatory and Adam Riess of the Space Telescope Science Institute and the Johns Hopkins University.

The discovery of accelerating expansion came as a shock initially, since it implied that some form of repulsive force—of the type expected from the cosmological constant—propels the universe’s expansion to speed up. To reach their surprising conclusion, the astronomers relied on observations of very bright stellar explosions known as Type 1a supernovae. These exploding stars are so luminous (at maximum light, they may outshine their entire host galaxies) that they can be detected (and the evolution of their brightness followed) more than halfway across the observable universe. In addition, what makes Type 1a supernovae particularly suitable for this type of study is the fact that they are excellent standard candles: Their intrinsic luminosities at peak light are nearly the same, and the small deviations from uniformity that exist can be calibrated empirically. Since the observed brightness of a light source is inversely proportional to the square of its distance—an object that is three times farther than another is nine times dimmer—knowledge of the intrinsic luminosity combined with measurement of the apparent one allows for a reliable determination of the source’s distance.

Type 1a supernovae are very rare, occurring roughly only once per century in a given galaxy. Consequently, each team had to examine thousands of galaxies to collect a sample of a few dozen supernovae. The astronomers determined the distances to these supernovae and their host galaxies, and the recession velocities of the latter. With these data at hand, they compared their results with the predictions of a linear Hubble’s law. If the expansion of the universe were indeed slowing, as everyone expected, they should have found that galaxies that are, say, two billion light-years away, appear brighter than anticipated, since they would be somewhat closer than where uniform expansion would predict. Instead, Riess, Schmidt, Perlmutter, and their colleagues found that the distant galaxies appeared dimmer than expected, indicating that they had reached a larger distance. A precise analysis showed that the results imply a cosmic acceleration for the past six billion years or so. Perlmutter, Schmidt, and Riess shared the 2011 Nobel Prize in physics for their dramatic discovery.