Robertson’s "uncertainty" vs Callen’s "disorder"

Harry Robertson may take Herb Callen’s place as our main man of statistical mechanics. Robertson is the author of Statistical Thermophysics (1993). I don’t see this textbook cited very often, but I remember when I ran across it on the “New Books” shelf in the Physics-Math-Astronomy library at UT-Austin. I liked it enough to make some photocopies from it right away, but I never bought a copy of the book myself until a few weeks ago. Based on what I’ve studied in it so far, I think it’s the best graduate-level statistical mechanics text I've seen.  And maybe it’s also one of the most overlooked, although Mark Loewe mentions it in his 1995 notes I included in my previous post (see last page of notes, last footnote, about the “surprise” function--more about that later). One reason instructors might not use Robertson's book as a text is that complete solutions to the problems are given in the book. Instructors would have to assign different problems if they wanted to grade students on problem solving.

Callen uses the idea of the disorder in a probability distribution {p_i} – he labels it {f_i} but I’m using Robertson’s notation—whereas Robertson uses the idea of the uncertainty in information content of such a distribution.  They both write the entropy (a lá Claude Shannon) as

S = –k∑ p_i ln p_i .

For the special case (the subject of this post) of having N possible outcomes, the maximum entropy occurs when each outcome has a probability 1/N. For Callen, this is the maximum disorder for the N outcomes. My problem with Callen’s “disorder” interpretation of entropy is that I don’t see why this equiprobability of outcomes should be called the maximum disorder.  Callen’s discussion of this, at the beginning of Chapter 17, doesn’t agree with my intuition of what disorder is. Equiprobability seems very ordered!

Robertson’s discussion of how the {p_i} should be interpreted in the N outcomes case is worth quoting from the beginning of his section on Information Theory, on page 3:

 

It has long been realized that the assignment of probabilities to a set of events represents information, in a loose sense, and that some probability sets represent more information than do others. For example, if we know that one of the probabilities, say p₂, is unity, and therefore all the others are zero, then we know that the outcome of the experiment to determine y_i will give y₂. Thus we have complete information. On the other hand, if we have no basis whatever for believing that any event y_i is more or less likely than any other, then we obviously have the least possible information about the outcome of the experiment.

 

Having “no basis whatever for believing” that any one event is more, or less, probable than any other means assigning all the events the same probability, and this equiprobability of events is something I can understand intuitively as giving the least amount of information. And this is the “information-theoretic maximized uncertainty” that Robertson uses in place of Callen’s maximum disorder.

Robertson subscribes to what is often called the subjective assignment of probabilities, while Callen sticks with the frequency-of-occurrence assignment of probabilities. Both men use Claude Shannon’s 1948 information theory formulation to define entropy (see above equation), but their interpretations of what the {p_i} represent are very different.

Callen wants to use only frequencies of occurrence as a measure of probabilities, as in, for example (my example), the objectively calculable and measurable frequencies of various sums-of-dots appearing on the upward faces of many dice tossed simultaneously many times. Robertson, on the other hand, is a follower of Edwin Jaynes’ 1957 re-interpretation of Shannon’s information theory as a subjective-probabilities theory.  Lots of controversy is involved in that interpretation. (The dice-throwing example probabilities are not uncertain enough to even need a subjective-probabilities approach.)

In spite of not agreeing with the subjective interpretation, Callen gives a great discussion (p. 380) of the subjectiveness in the general meaning of the word “disorder” before he introduces Shannon’s solution to the problem of the meaning of disorder. As one example of subjective disorder, Callen says a pile of bricks appears to be very disordered, but it may be “the prized creation of a modern artist,” and thus may not be any more disordered than a brick wall once the artist’s intention is understood.

But Callen then says this sort of apparent subjectiveness in the idea of disorder is removed by Shannon’s definition of the type of disorder used in information theory.  “The problem solved by Shannon,” Callen claims, “is the definition of a quantitative measure of the disorder associated with a given distribution {p_i}.” By “quantitative measure” he means the entropy expression above, and there’s no controversy over that. The controversy is about how the set {p_i} can legitimately be determined.

That’s as far as I’ll go on the subject at the moment. I only wanted to say how much better the idea of minimal information (and thus maximal uncertainty in information) is than the idea of maximal disorder when the case of equiprobability of outcomes is being described.

P.S. (15 October 2021)  Maybe it occurred to you that there are two different ideas of "maximum" being discussed here? After a week of pondering these and related concepts--like uncertainty and probability in quantum theory compared with uncertainty and probability as discussed above in relation to the expression for entropy--it's finally occurred to me that we have a particular case of known equiprobable outcomes as a case of maximum entropy, and we also have the general case that involves the entropy function with unknown or arbitrary p_i 's and we want to find it's maximum and in the process find the p_i themselves for this case, which would usually be a thermodynamic equilibrium case.  

Robertson distinguishes the particular case from the general cases at the end of his Information Theory section, prior to his introducing the idea of Maximum Uncertainty in the next section: "For a given number of possible events n, it is easily shown that S is a maxima when the events are all equally probable ... . The next problem to be examined is that of assigning the p_i's on the basis of our knowledge. This problem leads to the conceptual foundation of the approach to statistical mechanics used in the present development." The "present development" means Robertson's textbook.

The particular case is the subject of the first Problem at the end of Robertson's first chapter. In this problem, he gives the constraint of the probabilities summing to unity as the only constraint to use in maximizing the entropy expression, meaning one Lagrange unknown multiplier rather than the usual two--try it! You only need the above entropy expression and the constraint

∑ p_i = 1,

where the sum goes from 1 to n. (Or 1 to N, or however you want to label the number of possible outcomes.)

Callen's and Robertson's textbooks are actually very complementary in the different subjects, and levels of subjects, and interpretation of the subject itself, that they cover. Callen's book is an undergrad text, the best one in my opinion, and it covers more thermodynamics than Robertson's book, which is intended to be a graduate-level text (the best one in my opinion) with statistical mechanics as its primary subject rather than macroscopic thermodynamics. The difference in their interpretations has been the subject of this little essay, thank you.