LifeUniverse&Everything – Page 3

The Isomorphism is not the Terrain

—Draft—

We do not often think of mathematical models as tools so much as the embodiment of a theory. We take formulae such as f=ma to be the theory, not a tool for understanding the theory. Yet mathematical formalisms are chosen for their user-friendliness —their tractability and solvability— just as much as for their correctness, which suggests that theories which rely on them may be calculable yet less than true.

Shannon's Information Theory, for instance, provided a tractable solution to the problem of what is the very best rate of message transfer we can achieve over a wire, and has been so widely re-used one might think that all the world's an abstract communication channel.

By contrast, Sundman's 1912 infinite series solution to the three-body problem languishes in obscurity not because it is wrong, but because it is incalculable. Instead, dozens and hundreds of models of special cases of the 3-body problem have been published since then, each having the virtue of user-friendliness: you can actually use them to calculate something.

With a theory such as Shannon's, usability leads to enthusiastic adoption and enthusiastic adoption leads to a sense of well-being and belief we can solve all the things with this one simple trick.

What it does not lead to, is a careful enumeration of assumptions made in the adoption of a model.

This is fine for the engineer who just wants to calculate and get a result. It is less fine when the scientist enthusiastically declares that their mathematical model solves all the philosophical problems too.

For the philosophical problems are rarely captured in the mathematical model. They are more usually either captured or carefully avoided in some well-worded definitions and assumptions in the preamble and the prerequisites of the model.

For instance, Shannon defines a measure of information: “If the number of messages in the set [of all possible message you might want to send] is finite then this number or any monotonic function of this number can be regarded as a measure of the information produced …”

The monotonic function chosen is the logarithm and leads to a definition of Shannon Information as the negative log probability of choosing that message out of all possible messages. This definition has been extremely usable because it is tractable: you can compute with it.

Sadly it has only a tangential relationship to the meaning of “information”, as Shannon noted: “Frequently messages have meaning … These semantic aspects of communication are irrelevant to the engineering problem.”

Imagine for instance, you drive from Durham to Barnard's Castle and lost at a crossroads, ask a farmer for directions. He draws his fair 6-sided die from a pocket, tosses it, and reports, “It's a 6.”

Per the mathematical model, the farmer is rather informative. He has conveyed 2.6 shannons of information. Had he instead said, “Ah yes, for Barnard's Castle you want to turn left here” that would have conveyed barely 2 shannons of information (less, if “do a U-turn” was not on the table) and so been less informative.

The model is not reality. Information in the model is not information in the real world. Rather, information in the model is something that is isomorphic to something that is quite close to information in the real world.

The model definition of information — negative log likelihood — is certainly good and very usable for a definition of the maximum possible information rate achievable over a communication channel. It has also turned out to be very … usable … in related applications. Because the idea of an abstract communication channel can be used for a wide range of natural phenomena, and so a theory of them is widely useful.

But usability of a mathematical model leads to enthusiasm and enthusiasm may lead to completely ignoring niceties such as the relationship between the model and reality.

We could go with the flow and allow the model definition of information to override our previous understanding, and declare that information is negative log likelihood of selecting a particular message from a finite set of messages. It's tractable so it's true. After all, no-one likes to be stuck with intractable problems. Sundman's result was big at the time, but no-one remembers him now.

A mathematical model is a structure. A particular structure is picked out because it is user-friendly and because an isomorphism exists which maps onto it something that is close to something we are interested in. Shannon information is tangentially close to real information: there's a vanishingly small place where it's an exact match, a fuzzier area where it's very close and lots of places, such as en route to Barnard's Castle, where it's just wrong.

It seems an elementary mistake to say, “this structure lets us calculate stuff and therefore reality is exactly like this” yet it happens all the time. As people learn a result calculated using information theory, or Turing computability, or Bayes theorem, they go into the world convinced they're seen the light.

They've seen the light of a structure that is isomorphic to something that is close to something that interests us. They may never notice the assumptions and definitions required for the structure to count as a map of reality.

The isomorphism is not the terrain.

The Many Worlds Interpretation of Quantum Mechanics

If Everett's Many Worlds interpretation of quantum mechanics is true, I will not only eat my hat, I will eat yours too.

In fact, if Many Worlds is true, you will buy a hat specifically for me to eat.

A Supernaturalist and a Physicalist Swap Accounts of the Universe

A supernaturalist S and a physicalist P swap accounts of the existence of the world.

The supernatural account (S) of “something exists and behaves reliably enough for us to theorise about it” might go like this:
S: Something exists that is eternal and able to create a reliable universe which we can theorise about.
P: “How can that be?”
S: I don't know, it's supernatural.
P: Well that's not very satisfactory! Here's my account.

The physicalist account (P) of “something exists and behaves reliably enough for us to theorise about it” might go like this:
P: Something exists that is …[insert P's detail here]…
S: “How can that be?”
P: As a physicalist, I have a proper explanation. Look, here's my explanation, E, showing how it follows mathematically from certain equations and conservation laws.
S: “Nice. 2 questions though. (1) May I read your explanation E? and (2) does it include an explanation of how come these conservation laws and equations hold?”
P: Regarding (1) no you can't see explanation E because no-one has written it yet. And (2) it probably won't include that, we usually just accept them as a given.
S: Ok. Let's pass over the fact that no-one has written explanation E yet. When someone does write E, the account of “these conservation laws and equations hold” will be supernatural. Look, here's my account:

S2: Something exists that is eternal and able to create a reliable universe which instantiates these conservation laws you and conforms to these equations you mentioned.
P: “How can that be?”
S2: I don't know, it's supernatural.
P: Well that's not very satisfactory! Here's my account.

P2: Something exists and it instantiates these conservation laws and conforms to these equations we mentioned.
S: “How can that be?”
P2: It just does. It's a brute fact. That's the theory.
S: What makes this account physicalist rather than supernatural?
P2: Because I don't invoke anything supernatural.
S: What's a brute fact if not something supernatural? You explicitly use it to mean those phenomena for which you offer no physical account. Otherwise we wouldn't be calling it brute fact. So it's super-physical. Or, as we say in everyday English, supernatural.
P2: That is not what we usually mean by supernatural.
S: Hmmm. I thought “not explained by physical laws” was what we meant by supernatural?
P2: You can't expect the physical laws themselves to be explained in terms of physical laws.
S: You're right, I don't expect that. What I expect is, to recognise that saying physical laws can't be explained in terms of physical laws means they are inexplicable (in physicalist terms at least), and hence anything that a moment ago you explained in terms of them is, also, inexplicable in physicalist terms. There is no physicalist account of the world we live in, only supernatural ones.

Error-freeness per kilowatt-hour : a Proposed Metric for Machine Learning

Abstract

Accuracy on well-known problems is widely used as a measure of the state of the art in machine learning. Accuracy is a good metric for algorithms in a world where energy has negligible cost. We do not live in such a world.

I propose an alternative metric, error-freeness per kilowatt-hour, which improves on accuracy by trading-off accuracy against energy efficiency in a useful way. It has the desirable properties of approximate linearity in the relevant ranges (1 error per 1000 is 10 times better than 1 error per 100) and of weighting energy use in a way that accounts for cost of training as a realistic fraction of a delivered service. Error-freeness per kWh is calculated as e = 1/(1 + g - Accuracy)/(h + (training time in hours * (GPU+CPU Wattage)/1000)). The granularity g is the point of diminishing returns for improving accuracy. The overhead h is the energy cost of delivering a software service with no ML training. The parameters may be tuned to circumstance, but as a general-purpose metric, I propose g=1/100,000 and h=100kWh are good human scale, commercially relevant parameter values.

Detail

Where training is very expensive, it is not helpful to score machine learning algorithms on accuracy alone, with no account taken of the resources consumed to train to the level reported. In a competitive setting, it biases to the richest player; in the global, or society-wide, or customer-focussed setting, it ignores a real cost. This leads at best to sub-optimal choices and at worst to a growing harm.

I suggest that a useful, general purpose, metric has the following
characteristics:

For gross errors, it is linear in the error rate. Halving the error doubles
the score.
For very small errors, improving the error rate even to perfection adds only incremental value. Perfection is only notionally better than an error rate so small that one error during the application's lifetime is unlikely.
For extremely large energy consumption, the financial cost of the delivered service becomes proportional to the energy cost. We are concerned that the total human cost of energy use is very much dis-proportionate, in that emissions from increased energy consumption is an existential threat to the human race. For much less extreme energy consumption we might nonetheless accept the linear financial cost of energy as a proxy for the real cost.
For small energy consumption, the energy cost of training becomes an insignificant fraction of the whole cost of delivering a software service. The parameter h represents the energy cost of a service that uses no training.

Error-freeness per kWh can now be formulated as:

e = 
    1 / (1 + g - Accuracy)
      / (h + (training time in hours * GPU+CPU training wattage)/1000)

Setting the Parameters g and h

The granularity g

For general purpose human scale and commercial purposes I suggest a granularity of g=1/100,000 is a level at which halving the error rate grants only incremental extra value. It is about the level at which human perception of error takes real effort. Consider a 1m x 10m jigsaw of 100x1,000 pieces in which 1 piece is missing. An observer standing back to see the entire 1m x 10m work will not see the error. They would have to spend effort searching the 10 meter length of jigsaw.

Changing g by an order of magnitude either way makes little different to scores, until accuracy approaches 99.999%. So an alternative way to think about g is:

“If you can tell the difference between accuracy of 99% versus 99.9%, but cannot tell the difference between accuracy of 99.99% and 99.999%, then your granularity g is smaller than 1/1,000 but not smaller than 1/100,000.”

The software overhead energy cost, h

We estimate the energy cost of an algorithm-centric software service as follows:

A typical single-core of cloud compute requires 135W ¹ , for an energy cost of 1.0 MWh per year per server.
The software parts of a service in a fast moving sector (and, “being a candidate for using ML” currently all but defines fast-moving sectors) have a typical lifespan of about 1 year. (The whole service may last longer, but as with the ship of Theseus, the parts do not).
A typical size of service that uses a single algorithm is 4 cores plus 4 more for development and test. (Larger services will use more algorithms. We want the cost of a service of a size that uses only one algorithm).
8 such cores running 24/7 for 1 year is 8MWh.

We should set h to some fraction of 8MWh. There is little gain in attempting a more accurate baseline for general-purpose use. See the supplementary discussion below. We set that fraction based on 2 considerations.

Those 8 cores are often shared by other services, both in cloud-compute and self-hosting deployments. The large majority of the world's systems—anything outside the global top 10,000 websites—have minimal overnight traffic; office-hours is more realistic. Anything from 1% to 99% of a CPU-year might be a realistic percentage, the lower figure for virtual cloud-computing and the highest for dedicated hardware.

It is pragmatic to measure training train as the time for a single training run, rather than imagining developers keep careful record of every full or partial training run during development. We can more properly account for the total energy cost of all training time by dividing 8MWh by a typical number of training runs. If the final net takes 100 hours to train, it may have taken 10 or 10,000 training runs to settle on that net, in development, hyper-parameter tuning, multiple runs for statistical analysis, comparison with alternatives and so on. For a researcher, 1000 training runs may be too little, where for a commercial team doing only hyper-parameter training and testing, 50 runs might be more than enough. It is the widespread commercial usage that concerns our metric.

Combining the shared CPU usage with a typical number of training runs, one might argue for any fraction of 8MWh as typical, from 1/20th to 1/1000th. I propose a broad-brush rule of thumb setting h= 1/80th of 8MWh, or 100kWh. For large projects, it is simple to set g and h to values based on an actual business case and costs.

Proposed general purpose parameter values

This gives us standard parameters for error-freeness per kWh of

g=1/100,000
h=100kWh

Examples

A net is trained for 100 hours on a grid of ten 135W servers, each with a 400W GPU (i.e. 0.535 kW per server), and achieves an accuracy of 99%:

e = 1/(1+ g - 0.99)/(h + 100*10*.535) = 0.16.
It reaches accuracy=99.5% by quadrupling the number of servers: e =0.09.
A different algorithm for the same task achieves 99.4% on the original 10 servers: e=0.26.

On a different task, a net required only 10 hours on just a single server and GPU to reach 99% accuracy:

e = 1/(1+ g - 0.99)/(h + 10 * .535) = 0.95.
It reaches accuracy=99.5% by quadrupling training time to 40hours. e=1.64.
A different algorithm achieves 99.4% in the original 10 hours. e=1.58.

In the first case, training costs ½ a megawatt-hour per run (around £4,000 for 40 training runs at UK 2020 energy prices) and energy cost is well-reflected in the score. We may consider the doubling of accuracy not worth the quadrupling of cost. Where the training cost is a small fraction (around £40 for 40 training runs) of the cost of a delivered service, even a small gain in accuracy outweighs a quadrupling of energy cost.

Conclusion

When you measure people's performance, “what you measure is what you get”. People who are striving for excellence will measure their success by the measure you use. By promoting a metric that takes explicit account of energy usage, we create a culture of caring about energy usage.

The question this metric aims to answer is, “Given algorithms and training times that can achieve differing accuracy levels for different energy usage, which ought we to choose?” The point is to focus our attention on this question, in preference to letting us linger on the increasingly counter-productive question “what accuracy score can I reach if I ignore resource costs.”

Because the parameters are calibrated for real world general purposes, this metric represents a useful insight into the value vs energy cost of deploying one algorithm versus another.

Supplementary Discussion [Work In Progress] – the energy cost of a software service

To ask for the energy cost of a deployed software service is like asking for the length of a piece of string. In the absence of a survey of systems using ML, the calculation given is anecdotal on 3 points: How big a service does a typical single ML algorithm serve; what is the lifespan of such a service; for what fraction of that lifespan is the service consuming power?

In 1968, typical software application lifespan was estimated at 6-7 years ², but a single service is a fraction of such an application, and the churn of software services has increased with the ease of the development and replacement. I propose 1 year or less is a realistic lifespan for an algorithmic service in a competitive commercial environment.

The figure of 4 cores for a service arises from considering that although the deployed algorithm may only use a single core (or one low-power GPU), a service is typically deployed as part of an application with a user interface and some persistence mechanism. A whole service might then use 2 cores (for a monolithic deployment with redundancy) or 6 or more (for a multi-tier service with redundancy). Anything beyond that is likely already looking at parts of a larger application, unconnected to the work of machine learning. We can reasonably set the boundary for “that part of the system which we are only shipping because we have an algorithm to power it” at no bigger than that.

The figure of 135W for a single socket server might, in the context of efficiency-driven cloud computing, be discounted even 99% or more for low-usage services sharing hardware and consuming zero energy when not in use. Setting h=1/80 rather than, say h=1/500, probably represents very heavy usage.

Category: LifeUniverse&Everything