Bayesian Updating as Regularised Optimisation

I recently attended a workshop at Tsinghua University in Beijing on social networking and machine learning. One of the more machine learning focused talks by Jun Zhu caught my attention with a simple but surprising generalisation of Bayesian updating which he and his co-authors call “Regularized Bayesian Inference” or RegBayes.

The core idea is very simple: express classical Bayesian updating as an optimisation problem (see below) and then add constraints and regularisers to the posterior distribution. The advantage to this approach is that it affords an extra way to encode domain knowledge about problems.

The closest thing I’d seen to something like output regularisation was in a 2007 JMLR paper, Value Regularization and Fenchel Duality by Rifkin & Lippert.¹ However, RegBayes specifically starts with Bayesian updating and regularises distributions instead of just values.

I really like seeing new takes on old ideas, so I thought I’d work through the representation of updating as optimisation. The derivation below is an expansion of what is worked through in Section 3.1 of a recent paper by Zhu, Chen, and Xing titled Bayesian Inference with Posterior Regularization and Infinite Latent SVMs. Interested readers can also find an extended discussion, applications and connections with earlier work there.

Bayesian Updating as Divergence Optimisation

We’ll start by letting \(\mathcal{M}\) be a set of models and \(\Delta_\mathcal{M}\) the set of distributions over \(\mathcal{M}\). If \(\pi(M)\) denotes the prior probability of a model \(M \in \mathcal{M}\) and \(p(D|M)\) denote the data likelihood for \(D\) given \(M\) then, as everyone knows, Bayes’ rule states that the posterior \(p(M|D)\) can be computed via \[ p(M|D) = \frac{\pi(M) p(D|M)}{p(D)} \] where \(p(D)\) is the marginal distribution over the data.

Now, it is well known that the KL divergence from a distribution \(p\) to a distribution \(q\) — denoted \(KL(q\|p)\) — is minimised and equal to zero when, and only when, \(q = p\) (almost everywhere). This means that if we had some arbitrary distribution \(q(M)\) over models and wanted to ensure it was equal to the posterior distribution \(p(M|D)\) obtained by updating a prior \(\pi\) with data \(D\) we could do so in a round-about sort of way by solving the following optimisation problem: \[ \begin{equation} \inf_{q(M)}\ KL(q(M)\|p(M|D))\quad \text{s.t.}\quad q \in \Delta_\mathcal{M}.\label{eq:kl} \end{equation} \] Although this is not a particularly interesting optimisation problem, things get a little more interesting if we expand the KL divergence term. Letting \(p_L = p(D|M)\) denote the data likelihood and \(p_D = p(D)\) the data marginal we see that \[ KL\left(q\|\frac{\pi p_L}{p_D}\right) = \mathbb{E}_q\left[\ln\frac{q}{\pi p_L / p_D}\right] = \mathbb{E}_q\left[\ln \frac{q}{\pi} - \ln p_L + \ln p_D\right] \] and so the \(KL\) term in the optimisation problem above can be written as \[ \begin{align*} KL(q(M)\|p(M|D)) &= KL(q(M)\|\pi(M) p(D|M) / p(D)) \\ &= KL(q(M)\|\pi(M)) - \mathbb{E}_{M\sim q}\left[ \ln p(D|M) \right] + \ln p(D) \end{align*} \] since \(p(D)\) does not depend on \(M\). The \(\ln p(D)\) term is also irrelevant for the optimisation in \eqref{eq:kl} so we can now express that problem as \[ \inf_{q(M)}\ \underbrace{KL(q(M)\|\pi(M))}_{\text{closeness to prior}} + \underbrace{\mathbb{E}_{M\sim q}\left[ -\ln p(D|M) \right]}_{\text{fit to data}} \quad \text{s.t.}\quad q \in \Delta_\mathcal{M}. \]

The nice thing about expressing the posterior as the solution of the above problem is that we can see the Bayesian updating as a trade-off between staying close to the prior and fitting the data. The KL term keeps the posterior close to the prior while the other term measures the expected log loss of the model’s ability to predict the data.

Adding in Regularisation

The interesting insight Jun and his co-authors had was that the above optimisation problem can be tweaked slightly by adding either extra constraints to where \(q\) can lie in \(\Delta_\mathcal{M}\) or by adding an additional regularisation term \(U(q)\) which may or may not depend on the data. In its most general setting, updating a prior \(\pi\) given data \(D\) can be written as \[ \inf_{q(M)}\ KL(q(M)\|\pi(M)) + \mathbb{E}_q\left[-\ln p(D|M)\right] + U(q(M)) \quad \text{s.t.}\quad q\in\mathcal{Q}\] where \(U\) is a regulariser and \(\mathcal{Q}\subseteq \Delta_\mathcal{M}\) are the allowable distributions. The distribution \(q(M)\) solving the above problem is called the “post-data distribution” to distinguish it from the traditional posterior distribution.

Questions

One question that sprung to mind when I first saw this was whether this setting is strictly more general than normal Bayesian updating. The regularisation and constraints control which post-data distribution is selected but could such a post-data distribution be achieved as the posterior of normal Bayesian updating for a different prior?

It turns out the answer is “no”. Since the regulariser \(U\) can depend on the data, it is possible to create updating schemes that are not doing strict Bayesian updating. I’m not sure whether the same holds if \(U\) and \(\mathcal{Q}\) are not allowed to depend on the data but I suspect RegBayes would still be more general.

Some other questions that came up in my discussions with Jun included:

• What sort of convergence and consistency guarantees can be given for this scheme?
• Could there be connections between RegBayes and PAC-Bayesian theory?
• Could KL divergence and log loss be replaced by other loss/divergence pairs?
• Do some choices of regulariser yield updates for non-standard uncertainty calculi?

I haven’t yet spent much time thinking about these but I find this approach a fascinating way to couch the problem of managing uncertainty.

Machine Learning Research Jobs at NICTA

This is just a short note to say that the machine learning group that I am a part of at NICTA is looking to hire several new machine learning researchers. The full details of the positions can be found here and the links therein.

To summarise:

• Several full-time research positions in machine learning at NICTA are available.
• Positions can be taken up in either the Sydney or Canberra labs.
• Appointments are up to 3 years with the possibility of extension.
• Includes adjunct status at either the Australian National University or the University of Sydney, with the ability to supervise PhD students.
• Salaries from $85,000 to $100,000 (AUD) per annum.
• Application closing date: May 26th, 2013.

On a personal note, I’ve been an ANU-contributed researcher at NICTA for the last four years and find it an excellent place to work. We’ve got a great collection of people with diverse interests across machine learning, lots of challenging theoretical and applied problems, fantastic students, and a solid publishing record at NIPS, ICML, COLT, and JMLR.

I’m happy to answer any questions about the research environment here. If you want more details about the position and how to apply, please read the advertisement and/or email Bob Williamson (ML group leader) at .

I hope you can join us!

Switching from Jekyll to Hakyll

This is a non-machine learning-related post about a recent change to how I run this blog and the rest of my site. As you will notice, the design and content has largely stayed the same. However, I’ve made several changes under the hood that will make it easier for me to continue to develop my site.

The biggest change in my set up was shifting from using Jekyll to generate my site to using Hakyll. I thought I would write up my experiences in case any one else in interested in making this switch.

Leaving Jekyll

I’ll get this out of the way up front: Jekyll is fantastic and I would heartily recommend it to anyone wanting to set up a new statically generated site. It is easy to set up (especially if you use Octopress or Jekyll Bootstrap), has an activity community of developers led by the impressive Tom Preston-Werner, and you can easily host your site using Github Pages.

So why make the switch to Hakyll? The main reason is that I am using Pandoc as my main tool for rendering Markdown and wanted my site to do the same. Less crucially: I wanted an excuse to get back into using Haskell.

Yes, there is a Jekyll plugin for pandoc but I wasn’t keen on bringing another dependency into my already hard to maintain ruby environment. Also, after a bit of preliminary playing around with Hakyll I noticed it was substantially faster at generating my site than my Jekyll set up.

Using Hakyll

As the name suggests, Hakyll is inspired by Jekyll but written in Haskell. Both systems are built around the idea of taking a folder full of blog posts written in markdown (or some other human-friendly markup language) and generating HTML by rendering their content and wrapping templates around them.

Jekyll follows the general ruby philosophy of “convention of configuration” and, out of the box, will look for specially named files (e.g., *.md) and folders (e.g., _posts and _layouts) to find the content and templates needed to build you site.

In constrast, Hakyll requires you to write a small Haskell program to describe how you want pages to be rendered using a very flexible “Rules” language. For example, in my site.hs file I use the following code to set up my CSS files and render and wrap my blog posts.

main = hakyllWith siteConfig $ do
  -- CSS
  match "css/**" $ do
    route   $ idRoute
    compile $ compressCssCompiler

  -- Posts
  match "blog/posts/*" $ do
    route   $ gsubRoute "posts" (const "") `composeRoutes` rmDateRoute
    compile $ blogPostCompiler True
  
-- Blog post compiler
blogPostCompiler snapshot = mathJaxRenderer
  >>= loadAndApplyTemplate "_templates/post/full.html" postCtx
  >>= (if snapshot then (saveSnapshot "content") else return . id)
  >>= loadAndApplyTemplate "_templates/nav/blog.html" postCtx
  >>= loadAndApplyTemplate "_templates/default.html" postCtx
  >>= relativizeUrls

-- Blog context
postCtx =
  dateField "date" "%B %e, %Y" `mappend`
  dateField "shortdate" "%e %b %y" `mappend` 
  constField "top" "Inductio Ex Machina ← Mark Reid" `mappend`
  constField "section" "Blog" `mappend`
  defaultContext

-- Take out the post/YYYY-MM-DD part from the post URL
rmDateRoute = 
  gsubRoute "/[0-9]{4}-[0-9]{2}-[0-9]{2}-" (const "/")
  `composeRoutes` setExtension "html"

The route rule describes how the pattern in the match argument is converted into a URL. The compile rule describes how the items matching the pattern should be processed to build resource on your site.

In the case of the CSS files, each one is one through a (Hakyll provided) function that compresses the CSS by stripping whitespace.

Each of my blog posts starts life as a text file in my blog/posts/ directory and has a name of the form YYYY-MM-DD-title.md. The YYYY-MM-DD part specifies the post’s creation date and the title is used to form the URL. The route that handles blog posts first strips out the posts part from the file’s path and removes the data part of the the filename. The blogPostCompiler then runs its contents through a Pandoc renderer with MathJax output enabled then runs it through a pipeline that wraps various bits of HTML around it (div blocks, navigation elements, and finally head and body elements).

The blog template I use looks like this:

<div id="page">

<h1 class="emphnext">$title$</h1>

$body$

<address class="signature">
  <a class="author" href="http://mark.reid.name">Mark Reid</a> 
  <span class="date">$date$</span>
  <span class="location">$location$</span>
</address>

</div>

The $keywords$ in the template specify per-post values that are substituted in. In Hakyll, these are set using “Context” functions. In my case, the postCtx function sets the $date$ keyword in a “Month Day, Year” format and grabs the title and other values from the metadata at the top of each post using Hakyll’s built in functions.

As you can see from the above, configuring a Hakyll site boils down to explicitly writing code for how you want things processed. It is certainly more complicated than Jekyll. Fortunately, what Hakyll provides is a great “mini-language” of sorts that takes care of many of the fiddly bits (e.g., reading files, managing dependencies, rendering markdown). Anything that isn’t provided by Hakyll you can write yourself using Haskell.

It’s the difference between getting a ready-made meal versus buying good ingredients and making it yourself. In the latter case, you can easily stray from a recipe and tweak things to your own taste.

Housekeeping: Redirects and Comments

Once I had Hakyll generating my site I had to make a couple of final changes to finish the transition.

The first, and easiest, was remapping some URLs. Because I merged my Inductio ex Machina and Structure & Process blogs, which were previously at /iem/ and /sap/ respectively, into a single blog at /blog/ I had to ensure posts from the old URLs were sent to the new one.

I did this using RedirectMatch directives in my .htaccess file:

RedirectMatch 301 ^/iem/(.*)$ /blog/$1
RedirectMatch 301 ^/sap/(.*)$ /blog/$1

These simply match any post location from the old URLs and map them to the corresponding place under the new scheme. For example, my Behold! Jensen’s Inequality post used to be at /iem/behold-jensens-inequality.hml but is now at /blog/behold-jensens-inequality.hml. The 301 in the commands above tell any browsers that the posts have “Moved Permanently”.

The other main job I had was migrating my comments over to the new URL scheme. I use Disqus to host the comments for this site. The way Disqus works means the comments for each post are associated with the URL of that post. Changing the post URLs meant all the comments for the posts were no longer appearing.

Fortunately, Disqus provides an incredibly simple migration tool to solve this problem. Because I had set up the 301 redirects, all I had to do was click a button in Disqus’ admin panel for my site to tell its crawlers to go over my site, find the posts which had been 301ed, and update its database accordingly.

It worked like a charm.

Well… except for the Atom/RSS feed I accidentally regenerated with all my posts in it, thereby filling up many people’s feed readers with my old posts. Sorry about that.

Pros and Cons

So was the switch worth it?

For the purpose of regaining some of my Haskell-fu I can definitely say “yes”. I struggled quite a bit with remembering various idioms after a 12 year hiatus. Now I feel much more comfortable coding in Haskell and am probably more proficent than when I last did it.

I really like working in Pandoc’s style of markdown. It does (almost) everything Maruku, the previous renderer I used, did but it is much, much faster and has significantly better error reporting. I do miss Maruku’s syntax for div block, but apart from that I’m much happier with Pandoc.

Writing configuration code in Hakyll is much more time consuming, especially at first. However, given the fairly bespoke nature of my site and the even more arcane changes I have planned, I was already bending Jekyll a lot (through configuration, templates, and hacks) to get it to do what I wanted. Being able to drop down to Haskell and write code to specify how I want pages built is much easier in Hakyll.

That said, the templating in Hakyll is very basic and I had to put a lot more thought into how I wrote them in order to get the results I wanted.

Overall though, I am happy with this new set up and am looking forward to putting more content up on my site again.

Gun Deaths vs. Gun Ownership

In the wake of the tragic shooting at Sandy Hook Elementary School there was a flurry of discussion about gun control in my Twitter and Facebook feeds. One statistic from an April 2012 New Yorker article caught my attention:

The United States is the country with the highest rate of civilian gun ownership in the world. (The second highest is Yemen, where the rate is nevertheless only half that of the U.S.) No civilian population is more powerfully armed.

I posted this quote on Facebook and was asked whether there was any correlation between rates of gun ownership and gun-related deaths. After 20 minutes of grabbing some data on worldwide gun ownership and gun deaths from Wikipedia and running it through R I produced the following graph.

Gun-related deaths per capita vs. gun ownership per capita in OECD countries.

I posted this graph on Twitter late on Sunday night and woke up this morning to find it had been retweeted over 200 times. In addition, there were a large number of questions and requests for different graphs.

Out of curiosity, I grabbed gun death and gun ownership data for OECD countries from Wikipedia and ran it through R: dl.dropbox.com/u/38668/deaths…

— Mark Reid (@mdreid) December 16, 2012

So that others can analyse the data I used themselves, I thought I would write up how I created the graph.

The Data

The first step was finding data on gun ownership and gun-related deaths and getting it into R.

A quick search revealed Wikipedia articles on countries by firearm-related death rate and Number of guns per capita by country. I cut and pasted the tables in these articles into a text editor and saved them as CSV files. I also grabbed a list of OECD countries from Wikipedia and converted it. You can grab the resulting files here:

• deaths.csv (as of 17 Dec. 2012)
• guns.csv (as of 17 Dec. 2012)
• oecd.csv (as of 17 Dec. 2012)

The Code

The R code I used to create the plot is given below. It simply reads in the data from the guns and deaths CSV files, merges them, adds a column denoting whether a country is in the OECD, and then plots the results.

guns <- read.table("data/guns.csv", sep="\t", header=TRUE)
deaths <- read.table("data/deaths.csv", sep="\t", header=TRUE)
oecd <- read.table("data/oecd.csv", sep="\t", header=TRUE)

data <- merge(guns, deaths, by="Country")
data$OECD <- data$Country %in% oecd$Country

with(subset(data, data$OECD == TRUE), {
	plot(Deaths ~ Guns,
		main="Gun Deaths vs Gun Ownership",
		xlab="Guns per 100 people",
		ylab="Gun deaths per 100k people",
		xlim=c(0,100)
	);
	text(Guns, Deaths, Country, pos=1)
})

You can get all the code I used to generate the figures in this post here:

• guns.R

Other Graphs

I was asked a number of good questions about the graph I posted on Twitter. In particular, several people were surprised at the number of gun-related deaths for Switzerland. As the following graph of gun homicides vs. gun ownership shows, most of those deaths are actually suicides.

Gun homicides per capita vs. gun ownership per capita in OECD countries.

Detail of gun homicides per capita vs. gun ownership per capita in OECD countries.

Finally, here is a plot of homicide vs. ownership across all countries — not just those in the OECD. The mass of countries in the lower left corner is unreadble but you can see those countries with either high homicide rates or high gun ownership clearly.

Gun homicides per capita vs. gun ownership per capita in all countries.

Just to be clear, I am not making any claims about the effect of gun ownership on shootings. I just wanted to show how easy it is to use freely available data to help answer reasonable questions about heated political issues like gun control.

I’ll leave you to draw your own conclusions.

Update (19th Dec. 2012)

The response and attention to these plots — both positive and negative — on Twitter, in the comments below, and via email has been a little overwhelming. So much so that my site went down yesterday due to me hitting the (usually sufficient) bandwidth cap my host provides.

Data Updates

Several people noted that the data I plotted for Switzerland was inconsistent with what was on Wikipedia. If you look at the edit history for the firearm-related death rate page on Wikipedia you will see that someone updated the Switzerland data late on the 16th of December. Taking into account that Canberra, Australia is GMT+11 this update occurred after I scraped the page for the data I used here.

I have redone the OECD plots with data scraped from the same pages as of 3:30pm on the 19th. You can see the results and get the new data below.

Gun deaths (left) and homicides (right) per capita vs. gun ownership per capita in OECD countries (click to view larger images).

The new data is here:

• guns–20121219.csv
• deaths–20121219.csv

Other Analyses

A number of people pointed me to analyses they had done of similar data, criticisms of this post, and alternative sources of data. Here is a summary of some of those other contributions:

• Diego Basch redid some of the above plots with overall homicide data (not just gun homicide).

• aphyr redid one of my plots on a log scale and also compared gun homicide to income and wealth.

• Dominikus Baur created a more interactive version of the above plots using d3.

• Ryan Rosario pointed me towards another analysis of deaths vs. ownership.

• finem respice has a very detailed response to this post (and other reporting) that explains why the data I used for Switzerland in my initial graphs was incorrect.

• Walter Hickey at Business Insider summarised my post. There is a vigorous discussion in the comments there.

• Lev Reyzin tweeted some plots of homicides vs. percentage of gun ownership, included one broken down by US states.

• Tim Raupach pointed me towards Nate Silver’s analysis of US gun ownership statistics.

Disclaimer

Finally, in light of some of the more vocal criticism I’ve received, I feel I should reiterate a few points:

• At no point have I ever claimed that this is even close to a rigorous analysis. In my original tweet and here I have made it clear that I have used data from Wikipedia. Since I thought it was common knowledge that Wikipedia should never be used as a primary source for any kind of real research, I didn’t expect people to confuse this post with a serious study.

• I have not attempted to use these plots to draw any kind of conclusion about the relationship between gun ownership and gun deaths. Indeed, I do not think there exists a simple explanation. All I intended to do was attempt to shed some light on a question I was asked about a possible correlation.

• I will not run any kind of statistical test on the above data (e.g., rank correlation coefficients). For any such quantity to be meaningful I would need to assume that the data I collected was much more carefully controlled for. Since I just grabbed it from Wikipedia, this is clearly not a valid assumption.

• I do not intend to create more graphs showing other data or other statistics. Other people (see the links above) have already done so for a variety of alternative data. Also, I realise their are better sources of data out there than Wikipedia. However, as I’ve said before, my intention was not to do a definitive study but rather show how a readily accessible source (i.e., Wikipedia) can be used to quickly get an overview of a topic.

I will probably not the last person to use Wikipedia for this sort of amateur analysis. Therefore, if you believe the data I have used is incorrect or out of date, I suggest you solve the problem at the source and update (with references) the relevant entries on Wikipedia.

If you have a question about other possible correlations or relationships, feel free to apply the code above to what ever sources of data you think might help answer it. I’d appreciate it if you link to what you find in the comments below.

"Who is ML Hipster?"

To avoid keeping you in suspense, I’ll get this out of the way right up front: ML Hipster is me!

Who are you talking about?

I started the ML Hipster account on a whim and found it fun to mash up clichés about hipsters and their attitude with jargon from machine learning and statistics.

Is this Bayesian? You know I’m a strict Bayesian, right?

— ML Hipster (@ML_Hipster) July 13, 2012

I guess I was partially inspired by the rash of other technical novelty accounts out there such as PLT Borat and Hipster Hacker.

There was also the added enjoyment of obliquely poking fun at what some “old school” machine learning people might think of the recent, “big data”-driven incursion onto our (ahem) their turf.

Everyone is all big data this and online that. My methods are small batch: they only handle a few instances but really look at them, y’know?

— ML Hipster (@ML_Hipster) August 16, 2012

Ultimately though, it was really just an excuse to make bad puns that only a select few would understand, let alone find amusing.

“Oh sure, going in that direction will totally minimize the objective function” —Sarcastic Gradient Descent.

— ML Hipster (@ML_Hipster) July 20, 2012

Why take off the mask?

To be honest, I had no idea the account would get so popular. So much so that shortly after starting it I promised myself that if it ever got more followers than my main twitter account I’d spill the beans.

Well, 103 days, 61 tweets, and 1,651 followers later, here I am presiding over scattered legumes.

What now?

I have no intention of abandoning ML Hipster and will likely post there whenever I have yet another terrible, technical pun to inflict which I don’t want friends and family to hear.

I prefer Chervonenkis’ solo work.

— ML Hipster (@ML_Hipster) July 14, 2012

I’m also not going to reveal who ML Hipster is via that account. I figure there are people following ML Hipster (but not this blog or my other account) who couldn’t care less about who it actually is and I don’t want to spoil their fun.

Or mine.