Inductio Ex Machina

ICML and COLT Highlights

2009-07-07T00:00:00+10:00

Now that I have had a much needed holiday I am feeling refreshed enough to write up some of my notes on the recent ICML and COLT conferences held in Montréal this year.

I did not get to see as much of either conference as I would have liked to as I was nursing an awful cold. Fortunately, I was able to fight it off long enough to finish and give my ICML and COLT presentations. The ICML one almost didn’t happen as I had completely lost my voice the day before.

ICML

One theory paper I really liked at ICML was PAC-Bayesian Learning of Linear Classifiers by Pascal Germain, Alexandre Lacasse, François Laviolette and Mario Marchand. They give a general statement of the PAC-Bayes bound in terms of a convex function measuring the divergence between the true and empirical risks and, most importantly, give a remarkably simple proof using only Markov and Jensen’s inequalities.

Although I didn’t really follow the active learning aspects of Learning from Measurements in Exponential Families by Percy Liang, Dan Klein and Michael Jordan, I did think their introduction of measurements as a way of combining labels and constraints was elegant. Rather than assuming instances X have observed labels Y, they assume the labels are hidden and their values are only available implicitly through a set of aggregated measurements M(X,Y) over the whole data set. Various choices of the function M result in many existing learning problems.

COLT

Of the presentations I managed to attend at COLT, there were two that I found thought-provoking.

Learnability and Stability in the General Learning Setting by Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro and Karthik Sridharan examined the “general learning setting” proposed by Vapnik where, instead of trying to minimising the expected cost of a function that predicts labels for instances we just want to find a point in a hypothesis space that minimises . The relationship between hypotheses and instances in the latter case is much more arbitrary than in the usual supervised setting.

Interestingly, the usual correspondence between uniform convergence of empirical risks to the expected risk and learnability which holds for supervised learning with empirical risk minimisation (ERM) breaks down in the general setting. In particular, it turns out uniform convergence is sufficient for learning with ERM but not necessary. The authors therefore explore generalised notions of ERM and show that learnability and stability are equivalent for these more general versions.

Finally, I really liked the connections made between online learning and traditional statistical learning theory in A Stochastic View of Optimal Regret through Minimax Duality by Jacob Abernethy, Alekh Agarwal, Peter Bartlett and Alexander Rakhlin. In this paper the authors use the von Neumann’s minimax theorem to relate the minimax regret that is normally used in online convex optimisation (OCO) games to a supremum over a set of distributions of the expectations of a loss. Through a neat geometrical interpretation of this loss they are able to establish upper and lower bounds on the optimal regret for various online learning problems.

More Highlights

If you are after more highlights, I recommend having a look at John and Hal’s overviews of the conferences.

ICML Discussion

2009-06-11T00:00:00+10:00

Another quick note, this time about the ICML Discussion site I have updated for ICML 2009 in Montréal, Canada.

Thanks to this year’s organisers, I was given a list of authors and papers for ICML 2009 and was able to add them to the base I set up last year for ICML 2008. I’ve made separate indices for this year’s and last year’s papers but have bundled all the authors across the two years in together.

The statistics I have kept for the site since I started it show that there has been a boost of interest since I’ve added the new papers. I’m hoping that will result in a flurry of comments during the conference as people attend talks and posters. Keep in mind that if you are commenting on a paper you can use LaTeX to get your point across.

Unfortunately, there is no notification system in place for the discussion site at the moment so I encourage authors with papers accepted to this year’s conference to check the front page of the site regularly to see if anyone has asked a question about their paper.

If there is any questions or suggestions about the site you can leave me some feedback either here or in the feedback section of the site itself.

Enjoy the conference and the discussion site. See you in Montréal!

Generalised Pinsker Inequalities and Surrogate Regret Bounds

2009-05-23T00:00:00+10:00

Just a quick note to say that Bob Williamson and I have improved a few of the results we had previously written up in our technical report and even managed to get some of them accepted to ICML and COLT this year.

The ICML paper is Surrogate Regret Bounds for Proper Losses. In it we extend earlier results on surrogate bounds for 0-1 misclassification loss to cost-weighted misclassification losses and proper losses. We also investigate conditions for convexity of composite proper losses.

As the abstract puts it:

We present tight surrogate regret bounds for the class of proper (i.e., Fisher consistent) losses. The bounds generalise the margin-based bounds due to Bartlett et al. (2006). The proof uses Taylor’s theorem and leads to new representations for loss and regret and a simple proof of the integral representation of proper losses. We also present a different formulation of a duality result of Bregman divergences which leads to a demonstration of the convexity of composite losses using canonical link functions.

What I particularly like about this paper is that these results and several earlier ones regarding proper losses are arrived at using little more than a Taylor series expansion.

The COLT paper, Generalised Pinsker Inequalities, is (unsurprisingly) a bit more technical. As the name suggests, we were able to generalise the classical Pinsker inequality¹ that provides a lower bound on the KL divergence between the distributions and in terms of their variational divergence :

Not only were we able to tighten this bound but we provide a template for constructing similar (and tight) bounds for any f-divergence in terms of variational divergence. Furthermore, we also generalise the bounds to make use of one or more non-symmetric variational divergences.

Here’s the abstract:

We generalise the classical Pinsker inequality which relates variational divergence to Kullback-Liebler divergence in two ways: we consider arbitrary f -divergences in place of KL divergence, and we assume knowledge of a sequence of values of generalised variational divergences. We then develop a best possible inequality for this doubly generalised situation. Specialising our result to the classical case provides a new and tight explicit bound relating KL to variational divergence (solving a problem posed by Vajda some 40 years ago). The solution relies on exploiting a connection between divergences and the Bayes risk of a learning problem via an integral representation.

Although the proof requires some careful, technical detail, the approach is very simple and geometric. It makes use of the fact that any f-divergence can be written as a weighted integral of variational-like divergences and the fact that f-divergences are related to Bayes risk curves for proper losses.

In a very loose sense, these papers are duals of each other: the bounds in both share a similar structure – what can be said about a “complex” divergence given information about some “simple” divergences vs. what can be said about a “simple” loss given information about a “complex” one – and both are driven by the integral representation of losses and divergences which provides an interpretation of “simple” and “complex”.

I’ll be presenting both these papers in Montréal in June so if you are also attending ICML and COLT and are interested in these results please let me know.

The statement of Pinsker’s inequality on the Wikipedia page uses a version of variational divergence that differs from ours by a factor of 2.
↩

Irving “Jack” Good (1916–2009)

2009-05-04T00:00:00+10:00

The Language Log recently noted the passing of the eminent statistician, probability theorist, WWII code-breaker, colleague of Alan Turing and populariser of Go, Irving John “Jack” Good who died on April the 5^th at the ripe old age of 92.

As I have mentioned earlier on this blog, Good’s 1965 monograph, ”The Estimation of Probabilities: An Essay on Modern Bayesian Methods” was especially influential for my research. In this slim volume was a beautifully written treatise on how to estimate probabilities from “effectively small samples” – exactly the problem I was facing in my thesis. When samples are scarce, evaluating models becomes difficult because many models can share exactly the same misclassification rates.

It was by chance that I read his discussion of Dirichlet priors for multinomial samples and realised that they were the perfect tool for smoothing estimations of misclassification counts in contingency tables. By learning these priors on one task and transferring them to new, similar tasks I was able to break ties between models in a principled way by using the performance of similar models on earlier tasks.

The other time I encountered Good’s work was when reading about population estimation. Good-Turing estimation is a clever statistical technique developed by Good and Turing when they worked together as code-breakers at Bletchley Park. When randomly sampling items from some unknown number of classes, the Good-Turing estimator takes into account the probability that the next randomly selected item comes from a previously unseen class. This is very useful in areas such as linguistics where vocabularies are only incompletely known through some relatively small corpus of text.

Essentially, their technique estimates then smoothes the frequencies of the frequencies of the observed classes – the number of times a class with one item is observed, the number of times a class with two items is observed, and so on. By fitting a linear model to a log-log plot of these smoothed frequencies of frequencies estimates of low frequency classes can be revised upwards, possibly from 0, and an estimate of number of unseen class can be made.

In general, Good’s work could be described as that of a practical subjective Bayesian and I think he summed up this position very elegantly in a footnote in his monograph:

In our theories, we rightly search for unification, but real life is both complicated and short, and we make no mockery of honest ad hockery.

Priors and the Argument By Design

2009-04-21T00:00:00+10:00

Although I’m not completely sold on the book as a whole, Ken Binmore’s ”Rational Decisions” is a great collection of very sharp, terse insights into many paradoxes and counter-intuitive results regarding probability and decision theory.

One such digression is his five sentence analysis of the “Argument By Design” that is central to the Intelligent Design movement. With minor modification to his notation, it goes like this:

Let F be the event that something appears to have been organized. Let G be the event that there is an organizer. Everybody agrees that , but the argument by design needs to deduce that if God’s existence is to be more likely than not. Applying Bayes’ rule, we find that

If , we can deduce the required conclusion that , but we are otherwise left in doubt.

From this snappy argument he concludes

Bayesianism therefore has an explanation of why religious folk are more ready to accept the argument by design than skeptics!

As an atheist with Bayesian tendencies, I was very ready to accept this argument against the argument by design, but upon reflection I think this conclusion is arrived at too quickly.

Since we are computing probabilities anyway, why stop at checking if one is just larger than the other when we can compute odds?

Rather than just ask whether we can compute the odds and see how far it is from 1. Specifically,

since the denominator in Binmore’s calculation is just – the probability that “something appears to have been organized” and is assumed to be the same for believer and skeptic alike.

Now we see that the argument by design increases one’s belief in God given that organisation is observed (since we have assumed ) but will only “convert” a non-believer to a believer if the likelihood ratio for God is larger than the non-believer’s initial skepticism, as measured by his or her subjective odds against God’s existence.¹ That is, if

The strength of the argument by design is a function of the likelihood ratio of observing organised things with and without assuming a God.

However, contrary to Binmore’s conclusion, I believe the real reason ID advocates and skeptics disagree as to the strength of the argument by design is due to differences in this likelihood ratio.

It is fair to assume, as Binmore does, that the the likelihood ratio for an ID advocate and the likelihood ratio of a skeptic are both greater than 1 since there are many man-made artefacts such as watches that clearly do have an “organizer”. However, I would additionally argue that . Why? Well, presumably a skeptic who belies in evolution sees the natural world as a whole lot of things that were organised without an organiser whereas the ID advocate does not.

An ID advocate with a large will believe that even if you were very skeptical (small ), the product of and would be much larger than 1 and you would therefore be convinced to believe in God. From the point of view of a skeptic though would not be larger than and so the argument by design is not convincing.

This type of updating is what Richard Jeffrey evocatively called “probability kinematics” and is otherwise known as Jeffrey Conditioning.
↩