Inductio Ex Machina

Fisher Information and the Hessian of Log Likelihood

2012-04-04T00:00:00+10:00

I’ve been taking some tentative steps into information geometry lately which, like all good mathematics, involves sitting alone in a room being confused almost all the time.

I was not off to a very good start when a seemingly key relationship between Fisher information and the second derivative of the log likelihood eluded me, despite being described as “obvious” or “simple” in several books. I finally figured out the main trick and thought I’d share it here in case someone else has trouble with it (e.g., me in six months).

Fisher Information

Fisher information is a quantity associated with parametric families of probability distributions. Let be a set of outcomes and for each parameter in some set let be the distribution over associated with . The Fisher information for the family is the matrix valued function where the entry¹ at the th row and th column is

where the expectation is over the random variable drawn from the distribution , and denotes the partial derivative . The Fisher information is always symmetric and positive semi-definite and can be seen as measuring the “sensitivity” of the log likelihood on the outcomes in a neighbourhood of .

… and the Hessian of log likelihood

The result that had me puzzled for some time was the “obvious” fact that

where denotes the second-order partial derivative . What this says is that the Fisher information is closely related to the curvature of the log likelihood function, as measured by its Hessian – that is, the matrix of its second derivatives .

After much head-scratching, I realised that the “trick” I was missing was the observation that (under some mild conditions) the second derivatives and integrals can be switched so

since each is a distribution.

With the above identity in hand, establishing the relationship between Fisher information and the Hessian of log likelihood is just an application of the chain and product rules and noting that . Thus,

Taking expectations and using the aforementioned trick gives the result since .

Everything is obvious in hindsight!

I’m going to ignore issues such as convergence, existence, etc. Just assume things are “well-behaved” where necessary.
↩

Prediction with Expert Advice as Online Convex Optimisation

2011-09-15T00:00:00+10:00

I have been working with Bob Williamson and Tim Van Erven recently to better understand the notion of mixability in what is known as the Prediction With Expert Advice (PWEA) setting for online learning. I was curious as to how this setting relates to another one that is commonly studied in learning theory: online convex optimisation (OCO).

It is already known that PWEA is a special case of OCO (see, for example, Peter Bartlett’s summer school course or Kalai and Vempala’s JCSS paper) but I wanted to work out the correspondence explicitly for myself. Since there is one of those obvious-in-hindsight tricks involved I thought it would be worth writing up and sharing it.

Introduction

Prediction With Expert Advice is typically posed as a game where in each round a learner receives advice in the form of predictions from a set of experts about a future outcome and then merges these expert opinions to form its own prediction. The outcome is then revealed and the learner and all of the experts receive a penalty depending on how well their prediction fits with the revealed outcome. This penalty is determined by a fixed loss function that is known to the learner. The aim of the learner in this game is to incur an aggregate penalty over many rounds that is not much worse than the best expert.

You can easily imagine playing such a game yourself: each day you check a dozen different weather forecasts then make up your own mind about the chance of rain tomorrow, e.g., you predict a 75% chance of rain. The next day it will either rain or not and imagine that you and the experts lose points depending on how bad your predictions were: predicting a 75% chance when it is sunny loses you more points than if you predicted a 20% chance of rain. The function that determines exactly how many points you lose for predicting p% chance of rain when the outcome is sunny is called the loss function.

Mixability is a property of a loss function that characterises when learning can occur efficiently in a PWEA game. That is, when it is possible to make the difference between the learner and the best expert—the regret—decrease rapidly (specifically like after rounds).

In our recent COLT paper we were able to characterise mixability in terms of the curvature of the loss for a natural class of losses known as proper losses. These losses are “sensible” in that if the true probability of an outcome is then the expected loss is minimised by predicting . This seemingly innocent requirement actually gives rise to a lot of geometric structure that has been well studied in the economics literature, and that we exploit in our paper.

Online Convex Optimisation is a similar type of game to PWEA in that both are competitive online prediction games: a learner repeatedly makes predictions and receives a penalty based on that prediction and its performance is compared to a class of simple alternatives. The main differences between PWEA and OCO are that: the learner does not have access to expert predictions and their penalties; the regret of the learner is relative to a possibly uncountable set of alternatives; and the loss functions involved are assumed to be convex.

Despite these differences, it is possible to present Prediction With Expert Advice as a very special case of Online Convex Optimisation. After formalising the two games, I’ll present the “trick” for turning the former into the latter.

Prediction with Expert Advice

In the general Prediction with Expert Advice (PWEA) game a learner competes against experts in a game consisting of rounds. Each round, the each expert reveals a prediction from , the set of probabilities over outcomes. The learner observes and combines these to form its own prediction from . The world then reveals one of outcomes and the experts’ and learner’s predictions are assessed via a loss function so that a prediction incurs a penalty when outcome occurs.

Expressed in a kind of pseudo-code, the game is:

For :

Experts make predictions

Learner predicts based on expert predictions

World reveals outcome

Experts incur penalties and the learner incurs

The aim of the learner in this game is to minimise its total loss relative to the smallest expert loss . The difference is called the regret.

Online Convex Optimisation

Online Convex Optimisation (OCO) is similar in that a sequential game is played over rounds where a learner makes prediction from some convex set . However, as mentioned above, the OCO game is simpler in that there are no (explicit) experts and more general in that the finite number of outcomes that the world can reveal is replaced by an arbitrary set of convex functions . The function chosen by the world and used to assign a penalty to the learner.

Expressed in pseudo-code, OCO is the following game:

For :

Learner predicts

World reveals a convex function

Learner incurs penalty

The learner’s aim here is to minimise the regret relative to the best single prediction in hindsight. That is, the learner wants to minimise the difference between and . Once again, the difference is called the regret.

PWEA is a special case of OCO

We can show that PWEA is a special case of OCO by defining an OCO game that mimics the PWEA game.

The main trick is to define the set of functions for OCO so that step 1 in the PWEA game (where the experts reveal their predictions) can be simulated. Specifically, if for each in the PWEA game expert makes prediction and the outcome is , we define a OCO game via a sequence of linear (and thus convex) functions . These are defined so that where the vertices of and are linearly extended to all mixtures of experts, denoted , by defining .

This construction means that the learner in the OCO game can always mimic the performance of a single, fixed expert in the PWEA game by constantly playing . In some sense, this is how step 1 of the PWEA game is recovered in the OCO game.

Now consider what happens when we minimise the total loss for this OCO game. This involves finding a mixture such that is minimised. Since we see that where is the total loss for expert in the PWEA game. This weighted sum is clearly minimised by choosing the mixture that puts all its mass on the single expert corresponding to the smallest term. Furthermore, for that choice of we have and so .

The above argument shows that any PWEA game can be presented as an OCO game and that the best single expert in the PWEA game corresponds to the best single prediction in the corresponding OCO game.

Regret Bounds

Since the minimal total loss in the PWEA and OCO games are equivalent, we can look at the regrets for both games by just considering the total loss for each. If a learner playing the OCO game predicts at round the loss it incurs is . If all of the partial losses are convex then we see that predicting in the PWEA game will incur a penalty in that round.

Therefore, any regret bound that holds for OCO will also hold for an OCO-simulated PWEA game with convex losses since the OCO regret dominates the PWEA regret achieved by just playing convex combinations of the expert predictions. For a recent summary of lower and upper bounds for various types of online optimisation games, I point the reader to the COLT 2008 paper by Jake Abernethy and co-authors.

What happens if the PWEA losses are not convex? The same reduction argument can be run only if for every mixture there exists prediction such that for all outcomes we have . This is similar condition required of the substitution function needed in the Weak Aggregating Algorithm so I suspect this condition is related to mixability but will leave the details for another time.

NIPS 2011 Workshop on Relations Between Machine Learning Problems

2011-09-02T00:00:00+10:00

There is now a bewildering array of inference problems that techniques from machine learning can address: classification, regression, density estimation, clustering, ranking, recommendation, feature selection, hypothesis testing, and more. Multiply this by some of the modes in which learning can occur (batch vs. online, active vs. passive, partial vs. full feedback, inductive vs. transductive, semi-/un-/supervised) and there is a veritable zoo of problems out there.

As part of a continuing effort to make sense of it all, Bob Williamson, John Langford, Ulrike von Luxburg, Jennifer Wortman Vaughan, and I are running a workshop at NIPS this year titled Relations Between Machine Learning Problems–an approach to unify the field:

This workshop will focus on relations between machine learning problems. The idea is that by better understanding how different machine learning problems relate to each other, we will be able to better understand the field as a whole.

At this stage we plan to have Peter Grünwald and Amos Storkey give invited talks and a panel discussion on “How to build a map of all of machine learning”.

Submissions are due by the end of September.

If you need any further encouragement to join us, I just note that the NIPS workshops are in Spain’s Sierra Nevada this year.

Hope to see you there!

Artificial AI v2.0

2011-08-12T00:00:00+10:00

The first Artificial AI that my co-developer and I started almost three years ago — project Ada — is coming along nicely. Her (non-computer) vision, (non-machine) learning, NLP, and planning systems are coming along nicely.

Here’s a recent screenshot¹.

: Ada (AAI v1.0)

Encouraged by the success of the first version of the project we decided to take the plunge and start on version 2.

We call this version “Edith Valerie Reid”.

: Edith (AAI v2.0)

Her initial release on the 26th of July and so far everything is going really well. As with the first version, there are some issues to resolve such as the sleep-synchronisation routines but these are very minor and, from what I’ve heard, common to this type of project.

All in all, I’m declaring the version two release a resounding success.

Both photos are courtesy of Bob Williamson.
↩

ML Discuss for ICML 2011

2011-06-20T00:00:00+10:00

ICML 2011 is just around the corner so, with the help of some of the conference organisers, I recently updated the ML Discuss site I set up last year¹ to allow everyone to comment on accepted papers at ICML 2011.

If you are an author of a paper in ICML 2011, I urge you to find your papers and subscribe to their comment feeds via email or RSS. Instructions can be found on the main page. Of course, you can also subscribe to comments for papers you are interested in even if you are not an author.

Last year at ICML 2010 most of the online discussion happened during the conference itself. I expect that will also be the case this year but I would encourage people—even those, like me, who are not attending ICML 2011—to start commenting now. If you are after somewhere to start I would recommend checking out the Best and Distinguished Papers:

Looking ahead, I spent some time making sure the current version of ML Discuss can handle multiple conferences and separately track the top and recent comments for each. If you are a conference organiser for another machine learning-related conference and would like to have it discussed at ML Discuss, drop me an email and let me know. All that I require is a text file containing all the accepted papers, along with their authors and abstracts.

Happy commenting!

I’ve been running ICML discussion sites since 2008 but built a custom system for ICML 2010. I hope to port the older discussion sites to the new system when I find some time.
↩