Harry Robertson may take Herb Callen’s place as our main man of statistical mechanics. Robertson is the author of Statistical Thermophysics (1993). I don’t see this textbook cited very often, but I remember when I ran across it on the “New Books” shelf in the Physics-Math-Astronomy library at UT-Austin. I liked it enough to make some photocopies from it right away, but I never bought a copy of the book myself until a few weeks ago. Based on what I’ve studied in it so far, I think it’s the best graduate-level statistical mechanics text I've seen. And maybe it’s also one of the most overlooked, although Mark Loewe mentions it in his 1995 notes I included in my previous post (see last page of notes, last footnote, about the “surprise” function--more about that later). One reason instructors might not use Robertson's book as a text is that complete solutions to the problems are given in the book. Instructors would have to assign different problems if they wanted to grade students on problem solving.
Callen uses the idea of the disorder in
a probability distribution {pi} – he labels it {fi}
but I’m using Robertson’s notation—whereas Robertson uses the idea of the uncertainty
in information content of such a distribution. They both write the entropy (a lá Claude Shannon) as
S = –k∑ pi
ln pi .
For the special case (the subject of this post) of having N possible outcomes, the maximum entropy occurs when each outcome has
a probability 1/N. For Callen, this is the maximum disorder for the N outcomes. My problem
with Callen’s “disorder” interpretation of entropy is that I don’t see why this
equiprobability of outcomes should be called the maximum disorder. Callen’s discussion of this, at the beginning
of Chapter 17, doesn’t agree with my intuition of what disorder is. Equiprobability
seems very ordered!
Robertson’s discussion of how the {pi}
should be interpreted in the N outcomes case is worth quoting from the beginning of his section on
Information Theory, on page 3:
It has long been realized that the assignment of probabilities to
a set of events represents information, in a loose sense, and that some
probability sets represent more information than do others. For example, if we
know that one of the probabilities, say p2, is unity, and
therefore all the others are zero, then we know that the outcome of the
experiment to determine yi will give y2.
Thus we have complete information. On the other hand, if we have no basis
whatever for believing that any event yi is more or less
likely than any other, then we obviously have the least possible information
about the outcome of the experiment.
Having “no basis whatever for believing” that
any one event is more, or less, probable than any other means assigning all the
events the same probability, and this equiprobability of events is something I
can understand intuitively as giving the least amount of information. And this
is the “information-theoretic maximized uncertainty” that Robertson uses in
place of Callen’s maximum disorder.
Robertson subscribes to what is often called
the subjective assignment of probabilities, while Callen sticks with the frequency-of-occurrence
assignment of probabilities. Both men use Claude Shannon’s 1948 information
theory formulation to define entropy (see above equation), but their
interpretations of what the {pi} represent are very
different.
Callen wants to use only frequencies of
occurrence as a measure of probabilities, as in, for example (my example), the
objectively calculable and measurable frequencies of various sums-of-dots
appearing on the upward faces of many dice tossed simultaneously many times. Robertson,
on the other hand, is a follower of Edwin Jaynes’ 1957 re-interpretation of
Shannon’s information theory as a subjective-probabilities theory. Lots of controversy is involved in that
interpretation. (The dice-throwing example probabilities are not uncertain
enough to even need a subjective-probabilities approach.)
In spite of not agreeing with the subjective
interpretation, Callen gives a great discussion (p. 380) of the subjectiveness
in the general meaning of the word “disorder” before he introduces Shannon’s
solution to the problem of the meaning of disorder. As one example of subjective disorder, Callen says a pile of bricks
appears to be very disordered, but it may be “the prized creation of a modern artist,”
and thus may not be any more disordered than a brick wall once the artist’s
intention is understood.
But Callen then says this sort of apparent subjectiveness
in the idea of disorder is removed by Shannon’s definition of the type of disorder
used in information theory. “The problem
solved by Shannon,” Callen claims, “is the definition of a quantitative measure
of the disorder associated with a given distribution {pi}.” By
“quantitative measure” he means the entropy expression above, and there’s no
controversy over that. The controversy is about how the set {pi}
can legitimately be determined.
That’s as far as I’ll go on the subject at the
moment. I only wanted to say how much better the idea of minimal information
(and thus maximal uncertainty in information) is than the idea of maximal
disorder when the case of equiprobability of outcomes is being described.
P.S. (15 October 2021) Maybe it occurred to you that there are two different ideas of "maximum" being discussed here? After a week of pondering these and related concepts--like uncertainty and probability in quantum theory compared with uncertainty and probability as discussed above in relation to the expression for entropy--it's finally occurred to me that we have a particular case of known equiprobable outcomes as a case of maximum entropy, and we also have the general case that involves the entropy function with unknown or arbitrary pi 's and we want to find it's maximum and in the process find the pi themselves for this case, which would usually be a thermodynamic equilibrium case.
Robertson distinguishes the particular case from the general cases at the end of his Information Theory section, prior to his introducing the idea of Maximum Uncertainty in the next section: "For a given number of possible events n, it is easily shown that S is a maxima when the events are all equally probable ... . The next problem to be examined is that of assigning the pi's on the basis of our knowledge. This problem leads to the conceptual foundation of the approach to statistical mechanics used in the present development." The "present development" means Robertson's textbook.
The particular case is the subject of the first Problem at the end of Robertson's first chapter. In this problem, he gives the constraint of the probabilities summing to unity as the only constraint to use in maximizing the entropy expression, meaning one Lagrange unknown multiplier rather than the usual two--try it! You only need the above entropy expression and the constraint
∑ pi = 1,
where the sum goes from 1 to n. (Or 1 to N, or however you want to label the number of possible outcomes.)
Callen's and Robertson's textbooks are actually very complementary in the different subjects, and levels of subjects, and interpretation of the subject itself, that they cover. Callen's book is an undergrad text, the best one in my opinion, and it covers more thermodynamics than Robertson's book, which is intended to be a graduate-level text (the best one in my opinion) with statistical mechanics as its primary subject rather than macroscopic thermodynamics. The difference in their interpretations has been the subject of this little essay, thank you.
