Robertson’s "uncertainty" vs Callen’s "disorder"

Harry Robertson may take Herb Callen’s place as our main man of statistical mechanics. Robertson is the author of Statistical Thermophysics (1993). I don’t see this textbook cited very often, but I remember when I ran across it on the “New Books” shelf in the Physics-Math-Astronomy library at UT-Austin. I liked it enough to make some photocopies from it right away, but I never bought a copy of the book myself until a few weeks ago. Based on what I’ve studied in it so far, I think it’s the best graduate-level statistical mechanics text I've seen.  And maybe it’s also one of the most overlooked, although Mark Loewe mentions it in his 1995 notes I included in my previous post (see last page of notes, last footnote, about the “surprise” function--more about that later). One reason instructors might not use Robertson's book as a text is that complete solutions to the problems are given in the book. Instructors would have to assign different problems if they wanted to grade students on problem solving.

Callen uses the idea of the disorder in a probability distribution {p_i} – he labels it {f_i} but I’m using Robertson’s notation—whereas Robertson uses the idea of the uncertainty in information content of such a distribution.  They both write the entropy (a lá Claude Shannon) as

S = –k∑ p_i ln p_i .

For the special case (the subject of this post) of having N possible outcomes, the maximum entropy occurs when each outcome has a probability 1/N. For Callen, this is the maximum disorder for the N outcomes. My problem with Callen’s “disorder” interpretation of entropy is that I don’t see why this equiprobability of outcomes should be called the maximum disorder.  Callen’s discussion of this, at the beginning of Chapter 17, doesn’t agree with my intuition of what disorder is. Equiprobability seems very ordered!

Robertson’s discussion of how the {p_i} should be interpreted in the N outcomes case is worth quoting from the beginning of his section on Information Theory, on page 3:

 

It has long been realized that the assignment of probabilities to a set of events represents information, in a loose sense, and that some probability sets represent more information than do others. For example, if we know that one of the probabilities, say p₂, is unity, and therefore all the others are zero, then we know that the outcome of the experiment to determine y_i will give y₂. Thus we have complete information. On the other hand, if we have no basis whatever for believing that any event y_i is more or less likely than any other, then we obviously have the least possible information about the outcome of the experiment.

 

Having “no basis whatever for believing” that any one event is more, or less, probable than any other means assigning all the events the same probability, and this equiprobability of events is something I can understand intuitively as giving the least amount of information. And this is the “information-theoretic maximized uncertainty” that Robertson uses in place of Callen’s maximum disorder.

Robertson subscribes to what is often called the subjective assignment of probabilities, while Callen sticks with the frequency-of-occurrence assignment of probabilities. Both men use Claude Shannon’s 1948 information theory formulation to define entropy (see above equation), but their interpretations of what the {p_i} represent are very different.

Callen wants to use only frequencies of occurrence as a measure of probabilities, as in, for example (my example), the objectively calculable and measurable frequencies of various sums-of-dots appearing on the upward faces of many dice tossed simultaneously many times. Robertson, on the other hand, is a follower of Edwin Jaynes’ 1957 re-interpretation of Shannon’s information theory as a subjective-probabilities theory.  Lots of controversy is involved in that interpretation. (The dice-throwing example probabilities are not uncertain enough to even need a subjective-probabilities approach.)

In spite of not agreeing with the subjective interpretation, Callen gives a great discussion (p. 380) of the subjectiveness in the general meaning of the word “disorder” before he introduces Shannon’s solution to the problem of the meaning of disorder. As one example of subjective disorder, Callen says a pile of bricks appears to be very disordered, but it may be “the prized creation of a modern artist,” and thus may not be any more disordered than a brick wall once the artist’s intention is understood.

But Callen then says this sort of apparent subjectiveness in the idea of disorder is removed by Shannon’s definition of the type of disorder used in information theory.  “The problem solved by Shannon,” Callen claims, “is the definition of a quantitative measure of the disorder associated with a given distribution {p_i}.” By “quantitative measure” he means the entropy expression above, and there’s no controversy over that. The controversy is about how the set {p_i} can legitimately be determined.

That’s as far as I’ll go on the subject at the moment. I only wanted to say how much better the idea of minimal information (and thus maximal uncertainty in information) is than the idea of maximal disorder when the case of equiprobability of outcomes is being described.

P.S. (15 October 2021)  Maybe it occurred to you that there are two different ideas of "maximum" being discussed here? After a week of pondering these and related concepts--like uncertainty and probability in quantum theory compared with uncertainty and probability as discussed above in relation to the expression for entropy--it's finally occurred to me that we have a particular case of known equiprobable outcomes as a case of maximum entropy, and we also have the general case that involves the entropy function with unknown or arbitrary p_i 's and we want to find it's maximum and in the process find the p_i themselves for this case, which would usually be a thermodynamic equilibrium case.  

Robertson distinguishes the particular case from the general cases at the end of his Information Theory section, prior to his introducing the idea of Maximum Uncertainty in the next section: "For a given number of possible events n, it is easily shown that S is a maxima when the events are all equally probable ... . The next problem to be examined is that of assigning the p_i's on the basis of our knowledge. This problem leads to the conceptual foundation of the approach to statistical mechanics used in the present development." The "present development" means Robertson's textbook.

The particular case is the subject of the first Problem at the end of Robertson's first chapter. In this problem, he gives the constraint of the probabilities summing to unity as the only constraint to use in maximizing the entropy expression, meaning one Lagrange unknown multiplier rather than the usual two--try it! You only need the above entropy expression and the constraint

∑ p_i = 1,

where the sum goes from 1 to n. (Or 1 to N, or however you want to label the number of possible outcomes.)

Callen's and Robertson's textbooks are actually very complementary in the different subjects, and levels of subjects, and interpretation of the subject itself, that they cover. Callen's book is an undergrad text, the best one in my opinion, and it covers more thermodynamics than Robertson's book, which is intended to be a graduate-level text (the best one in my opinion) with statistical mechanics as its primary subject rather than macroscopic thermodynamics. The difference in their interpretations has been the subject of this little essay, thank you.

Fundamental relations and equations of state

A fundamental relation contains all the information about a system--all the thermodynamic information about a thermodynamic system--and involves independent extensive variables. For instance, entropy S expressed as a function of internal energy U, system volume V, and conserved particle number N (same type of particles)  is a fundamental relation, expressed symbolically as S(U,V,N).  Or, equivalently, the internal energy U can be expressed as a function of entropy, volume, and particle number, U(S,V,N), and this is also a fundamental relation.

In contrast, the more commonly used equations of state are not fundamental relations, and involve functions of state that are not independent of each other, such as temperature T,  pressure P, and chemical potential μ, that are partial derivatives of fundamental relations. A complete set of equations of state, however, is informationally equivalent to a fundamental relation. The three equations of state 

T(S,V,N) = ∂U/∂S,

 -P(S,V,N) = ∂U/∂V,

and μ(S,V,N) = ∂U/∂N,

for example, contain all the information of the energetic fundamental relation, U(S,V,N). For an interesting case where N isn't involved (because it's not conserved) see my July 22 post on the Stefan-Boltzmann law as one of the equations of state for black-body radiation.

(This discussion should make you think of similar descriptions used in quantum theory, such as "the wave function contains all the information about the system," the need for "a complete set of commuting observables," and the choice of  energy or entropy fundamental relations being somewhat like the choice of  Schrödinger or Heisenberg 'pictures'.)

The connections between fundamental relations and equations of state are discussed in Callen's book in the first three sections of Chapter 3, "Some Formal Relationships and Sample Systems." In case you don't have access to that, I'm posting (below) some meticulous notes on this subject put together for a Statistical Physics class by Professor Mark Loewe in the mid-1990s, when he was teaching at Southwest Texas State University in San Marcos (now Texas State University-San Marcos). He used Callen's book, but he also handed out notes such as these that he wrote himself.  He is concise and thorough, as you can see. (Well, yeh, sorry the notes are hard to see. Slide 'em over to your desktop, maybe.) There's more info in the notes than you would likely ever want or need, but the accompanying descriptions are worth reading. One little thing Mark wasn't thorough about: in the corrections and clarifications, he doesn't say which problem Callen gives an incorrect answer to. Maybe I have it in my class notes, and will post it if I find it. 

Just how fundamental relations and equations of state are related to the first law of thermodynamics in its most basic form,

,

is something worth thinking about and looking up. Does the first law contain all the thermodynamic (macroscopic) information about a system? Oh yeah, and we have to talk about isolated systems versus systems attached to a heat reservoir. The 1st law equation above is for the change in internal energy for Q (energy as heat) supplied to the system, and W the work done by the system on its surroundings. It's a good starting point for imagining either an isolated system that has energy added to it (temporarily connected to, then disconnected from, some kind of reservoir) or a system connected to a heat reservoir and not yet in thermal equilibrium with the reservoir.

Just to keep things in perspective before going on to specific examples in later posts, I'll quote Callen from the top of page 26, "The single, all-encompassing problem of thermodynamics is the determination of the equilibrium state that eventually results after the removal of internal constraints in a closed, composite system." 

Mark Loewe's notes:

Herb Callen problems 3.6-1, 3.6-2, and 3.6-3

(Revised 5 Aug 2021.  See my previous post for notes and comments on Callen's Section 3.6.)

3.6-1   The universe is considered by cosmologists to be an expanding electromagnetic cavity containing radiation that now is at a temperature of 2.7 K. What will be the temperature of the radiation when the volume of the universe is twice its current value?  Assume the expansion to be isentropic (this being a nonobvious prediction of cosmological model calculations).

So the entropy will be the same when the volume V₁ becomes V₂  and when  T₁ becomes T₂ :

S(U, V)   =  constant =  (4/3) (bV₂)^1/4 (U₂ )^3/4   =   (4/3) (bV₁ )^1/4 (U₁ )^3/4 , 

where  U₂ = bV₂ (T₂ ) ⁴   and  U₁ = b V₁ (T₁ ) ⁴ .  

Canceling the 4/3 and b factors on each side of the above entropy equation and putting in temperature T in place of the energy U gives (exponent on V's  becomes 1/4 + 3/4 = 1, and exponent of T's becomes 4 x 3/4 = 3)

V₂ (T₂ )³  =   V₁ (T₁ )³  

(T₂ /T₁ ) ³  =  V₁/V₂  =

(T₂ /T₁ ) ³  =  1/2    

T₂  =  (1/2)^1/3 T₁    =     0.79 (2.7 K)  =   2.1 K .

Question for further research: in how many years will this doubling in size occur? The accepted answer to this question is not the same now as it was in 1985 when Callen’s book was published, due to the discovery in 1998 of the acceleration of the expansion (dark energy). 

 

3.6-2 Assuming the electromagnetic radiation filling the universe to be in equilibrium at  T = 2.7 K,  what is the pressure associated with this radiation?  Express the answer in both pascals and atmospheres.

From Stefan-Boltz law, we have U/V = bT⁴ , and the pressure equation for isotropic radiation is P_R = U/3V.  So P_R = (b/3)T⁴

= (7.56x10^-16 Joule · meter^-3 · Kelvin^-4))(2.7 K)⁴/3 

= 134 x 10^-16  Newton/meter²

P_R =  1.34 x 10^-14 Pa

P_R =  1.32 x 10^-19 atm.

 

3.6-3  The density of matter (primarily hydrogen atoms) in intergalactic space is such that its contribution to the pressure is of the order of 10^-23 Pa. 

(a) What is the approximate density of matter (in atoms/meter³) in intergalactic space?

(b) What is the ratio of the kinetic energy of matter to the energy of radiation in intergalactic space? (Recall Problems 3.6-1 and 3.6-2.)  

(c)  What is the ratio of the total matter energy (i.e., the sum of the kinetic energy and the relativistic energy mc²) to the energy of radiation in intergalactic space?

 

(a)  We want to calculate N/V from P_M, and, assuming (maybe incorrectly) the hydrogen gas in intergalactic space is in equilibrium with the CMB radiation, we use the ideal gas law, P_MV = NkT, where T = 2.7 K.  Rearranging gives

 N/V  =  P_M/kT  =  10^-23 Pa/[(1.38 x 10^-23 J/K) · 2.7K]

=  (1/3.7) atoms/meter³ 

=  0.27 atoms/meter³ 

^{The answer is actually about one per cubic centimeter, or one million per cubic meter, alas. That means the temp of the very diffuse hydrogen gas is about 4 million degrees Kelvin.  Here's the first paragraph from a good discussion of the subject:}

There are some missing details that are often skipped over in popular science articles/documentaries, which can make this confusing. First: the temperature of space, 2.7 K or -270 C, refers to the remaining radiation from the Big Bang, not to the temperature of any matter. If you were in space and somehow prevented any matter from touching you, you would still absorb microwave radiation equivalent to being surrounded by matter with a temperature of 2.7 K (since all matter with temperatures above 0K emits radiation). Of course, with all the stars and such, you also receive a lot of other radiation, so you wouldn't necessarily cool down to 2.7 K. That temperature refers specifically, and only, to the background microwave radiation left over from the early history of the universe.

^--------

   (b)  For our intergalactic atomic hydrogen gas, the only energy is kinetic energy (well, ignoring the spin flipping the electron can undergo relative to the spin of the proton, resulting in the famous 21 cm spectral line).  (Also ignoring, until part (c), the relativistic mass-energy.)  So we have (KE)_H = (3/2) NkT = (3/2)P_MV for the kinetic energy of matter, which we want to divide by the U_rad =  bVT⁴  = 3P_RV of radiation:

 

(KE)_H / U_rad  =  (3/2) P_MV / 3P_RV

=  P_M / 2P_R

=  10^-23 Pa / 2(1.34 x 10^-14 Pa)

=   3.7 x 10^-10 

 

(c) The relativistic energy is U_rel = Mc², where M is the mass of all the hydrogen atoms in intergalactic space. We could do this calculation by estimating the size of the universe which we’d multiply by the density of hydrogen atoms. But volume cancels out in a ratio of energy density calculations, and energy density is the determining factor for the total energy of radiation, of hydrogen atoms’ KE, and of relativistic energy. And we've calculated number density of H atoms, so we should use it. We divide by total volume, V, of the universe to get density:

U_rel/V = (Mc² /V) =  m(N/V)c²

= (1.67 x 10^-27 kg for one H atom) (0.27 H atoms per meter³)(3 x 10⁸ m/s)²

=  9 x 10¹⁶ · 0.45 x 10^-27 joule/meter³

=  4 x 10^-11 joule/meter³

Now I need to divide this by U_rad/V for radiation, and add the result to the answer to (b). I didn’t do the U_rad/V calculation separately for radiation energy density in part (b)—but we know U/3V = P for the radiation pressure, so we take 3P as our radiation energy density

U_rad/V = 3P = 3 x 1.34 x 10^-14 Pa = 4 x 10^-14 joule/meter³

so

U_rel/U_rad  =  1000.

The KE/U_rad  ratio of 3.7 x 10^-10 is negligible in comparison with the factor-of-1000 ratio of relativistic-to-radiation energy. No point in adding them! But we’d expect the relativistic energy to be much greater than the total KE of atoms in a gas at temperature 2.7 K.

Questions for further research:

At what temperature does the KE of the H atoms equal their mass-energy?

At what temp does the radiation energy equal the H atoms mass-energy?

Does the mass-energy of all the loose H atoms in the universe have any significance beyond the fact that it has to be included when we account for all the energy released (created) in the Big Bang?

The Stefan-Boltz law as a thermo equation of state

In this discussion, the Stefan-Boltzmann law will be written in terms of the energy contained in a "trapped" electromagnetic field, rather than being written in terms of the power of an "escaping" electromagnetic field (as I discussed for sunlight in a recent post).  An energy-versus-temperature equation is one example of a thermodynamic equation of state. The well-known energy-versus-temp relation for a monatomic ideal gas, for instance, is U=3NkT/2. I'll look at many more examples of equations of state, and at how they are related to the "fundamental relations" of thermodynamics in a future post.

Our main man Herb Callen has a good discussion (Section 3-6 in his 2nd edition text) on the thermodynamics of radiation in an empty box: 

If the walls of any 'empty' vessel are maintained at a temperature T, it is found that the vessel is, in fact, a repository of electromagnetic energy. The quantum theorist might consider the vessel as containing photons, the engineer might view the vessel as a resonant cavity supporting electromagnetic modes, whereas the classical thermodynamicist might eschew any such mechanistic models.

The empty vessel could be almost anything. A metal breadbox, a wooden breadbox, a closed cardboard box, a spherical Christmas tree ornament, or Santa Claus ornament, as long as it's hollow. Yes, it's generally going to have air in it, which is a point that doesn't get discussed much. That's not to say it will be inflated with air like a tennis ball, football or basketball.  But it could be, and there's an electromagnetic field in there with that compressed air. We can just let these "vessels" be at room temperature. Otherwise we have the problem of keeping the walls uniformly at the same high or low temperature. 

Even if a box has walls that are at different temperatures, there will be some sort of radiation spectrum in the box.  Let's say one wall is heated so it's emitting most of the radiation in the cavity, presumably with a different emission spectrum near its surface than the spectra at the surfaces of the other walls.  But is this even possible? Can the spectrum of the radiation in the box be different on different sides of the box?

Well, what we would have is the usual linear superposition (addition) of the amplitudes of the all the waves/photons in the box.,* resulting in constructive and destructive interference, which results in an equilibrium energy-versus-frequency spectrum.  If we have a rectangular or square box, we can ask what the role of the radiation is in changing the temperature distributions in the five walls of our box that are not directly heated. This changing of the temps is a temporary, or transient, behavior. An equilibrium temperature distribution will come into being after the transient behavior is over, due to heat conduction in the walls and radiation from the heated wall of the box to the other walls of the box.

But the  radiation in the box won't have a black-body spectrum unless the walls of the empty vessel are maintained, as Callen says, at the same temperature, which gives equality of absorption and emission rates for every part of the walls. At least this is the desired black-body cavity experimental set-up. The other thing we need for the usual, real, set-up is a little peek-a-boo hole in the box to observe the spectrum of the radiation. 

But we aren't going to try to observe it here. We just want to ask, for a given temperature how much electromagnetic energy is in the vessel? The answer is given by the Stefan-Boltzmann law in yet another form, different from the three ways I wrote it in my February post. Instead of flux (power per unit area) or intensity, the total energy of the radiation is given as proportional to Kelvin-temperature-to-the-fourth-power. Callen writes this as

U = bVT⁴ 

where V is the volume of the empty vessel and b = 4σ/c = 7.56x10^-16 J/m³K⁴, where σ is the Stefan-Boltzmann constant and c is the speed of light. (See my 12 November 2020 post, where the integral of the Planck spectral energy density function at temperature T is shown symbolically and the factor of  c/4 is used.  Here, we're getting rid of that factor. If you divide both sides of the above equation by V, the result is energy density.) Sooooooo, the above equation really comes from integrating the Planck spectral energy density formula over all frequencies while holding temperature constant.  

What, then, is the difference between the views of the quantum theorist, the engineer, and the thermodynamicist in Callen's description above? The photons of the quantum theorist are after all normal modes of the electromagnetic field (see page 97 of these notes on the EM field), and a normal mode is a standing wave that has one of the resonant frequencies of the box. The best brief reference on this subject is Rodney Loudon's two-page discussion, "Introduction: The Photon," at the beginning of his book The Quantum Theory of Light, 3rd ed., and the five references therein.

The quantum theorist is not so much interested anymore in cavity radiation by itself as he or she is in cavity quantum electrodynamics, when there are a controlled number of atoms in a very small cavity (and no air, unless the "atoms" in the cavity are a few oxygen and nitrogen molecules) . An electrical engineer who wants to use the "vessel" as a resonant chamber for electromagnetic radiation is just like an acoustical engineer who wants to use an acoustic chamber to enhance certain frequencies and suppress others. (Here we can think about the usual "box of gas" as having acoustical modes of vibration.) Microwave ovens possess electromagnetic resonant cavities to enhance the microwave cooking frequencies, while musical instruments, for example, enhance the desired harmonics of sound waves.

But I'm getting off the subject here. The subject, which I haven't fully explained yet, is the equations of state of blackbody radiation, and how they can be used to find the entropy of blackbody radiation in terms of U and V. This is a "fundamental relation" in thermodynamics, and would be something the thermodynamicist is interested in, particularly the fact that N--whether he or she thinks of it as the number of photons in the box or number of normal modes--isn't an independent variable along with U and V. (See the quote from Callen below, and note he uses an exclamation mark.)

Besides the above equation of state for the total energy of trapped blackbody radiation in terms of temperature, there's a well-known relation between radiation pressure P, total energy U, and volume V,

P = U/3V

which you can copy and paste into an internet search box to find out more about. "It will be noted," says Callen, "that these empirical equations of state are functions of U and V, but not of N. This observation calls our attention to the fact that in the 'empty' cavity there exist no conserved particles to be counted by a parameter N. The electromagnetic radiation within the cavity is governed by a fundamental equation of the form S = S(U, V) in which there are only two rather than three independent extensive parameters!"

The relevant equation for finding the entropic fundamental relation is 

ST  =  U  +  PV ,

or 

S  =  U / T  +  PV / T.

From the two equations of state, we have 

1/T   = b^1/4 (V/U)^1/4

and 

P/T   = (1/3) b^1/4 (V/U)^1/4

so that the entropy fundamental equation is 

S(U, V)   = (4/3) (bV)^1/4 U^3/4

One of the things I'm interested in is how this expression for the entropy of black-body radiation in terms of U and V compares with the logarithmic expression found by Planck, but I'll come back to that later. First, I'll solve the three problems in this section of Callen's book (Section 3.6), then I'll post some detailed typed handouts from a Statistical Physics class I took in 1995 at Texas State University that used Callen's book as the text (thank you,  Mark Loewe!).

----------------

*Professor Hazeltine said in a junior-level electrodynamics class I took at the UofTexas at Austin 30 years ago that "photons don't interact." I raised my hand and said, "What about interference and diffraction?" Hazeltine explained that "interacting" has to do with things that exert forces on each other, and photons don't exert forces on each other. I asked, "Could we still say they interact linearly then?" I don't recall his response, but he stuck with the "photons don't interact" paradigm. It's not a good paradigm, because photons, being bosons, are gregarious and will indeed interact in a certain way (a sort of attraction) that fermions don't (they sort of repulse each other). We don't call mutual boson interactions and mutual fermion interactions by the name "forces," but maybe we should, or maybe we should modify our idea of forces.