Structure & Process

Online Learning in Clojure

2009-07-14T00:00:00+10:00

Online Learning is a relatively old branch of machine learning that has recently regained favour for two reasons. Firstly, online learning algorithms such as Stochastic Gradient Descent work extremely well on very large data sets which have become increasingly prevalent (and increasingly large!). Secondly, there has been a lot of important theoretical steps made recently in understand the convergence behaviour of these algorithms and their relationship to traditional Empirical Risk Minimisation (ERM) algorithms such as Support Vector Machines (SVMs).

In order to understand these algorithms better, I implemented a recent one (Pegasos, described below) in Clojure. This had the added advantage of seeing how well Clojure’s performance held up when doing some serious number-crunching.

Online Learning

One very appealing property of online learning algorithms is that they are extremely simple. Here’s what a general supervised online learning algorithm looks like. Given a loss function and a stream of examples of the form , do the following:

Initialise a starting model w
While there are more examples in S
    Get the next feature vector x
    Predict the label y' for x using the model w
    Get the true label y for x and incur a penaly L(y,y')
    Update the model w if y ≠ y'

Models are usually represented as vectors of weights for the features used to represent the examples. For binary classification problems predictions involve looking at the sign of the innner product and the update step in line 2.3 modifies the current model by moving in the direction that most reduces the loss of the incorrect prediction: that is, in the direction given by the negative gradient of the loss.

Pegasos

One recent online algorithm (and the one I’ve chosen to implement) is Pegasos: Primal Estimated sub-GrAdient SOlver for SVM (PDF) introduced by Shai Shalev-Shwartz, Yoram Singer and Nathan Srebro at ICML 2007 ¹. As this is my programming blog (not my research blog) I’ll just give enough of the detail of Pegasos so you can follow the implementation.

Pegasos solves the same optimisation problem as support vector machines. That is, it minimises the empirical hinge loss with regularisation:

where is a set of training examples, the norm, the regularisation constant and is the hinge loss.

The neat observation that allows optimisation problems like this to be cast as online learning problems is that the above loss can be computed using example-by-example updates rather than as a large sum. With a little care about how these updates are made fast convergence guarantees can be established.

In the case of Pegasos, if is the model after having seen examples and is an incorrectly predicted example, the (unnormalised) updated model is:

If the new model is outside a ball of radius it is projected back onto this ball.

Implementing it in Clojure

Once I understood what it was doing, Pegasos struck me as a very simple algorithm so I was itching to implement it. As mentioned earler, I was also curious as to Clojure’s performance on number-crunching tasks like this, especially when the canonical data set for online learning has over 700,000 examples and over 45,000 features.

Represented as 45k entry feature vectors, the examples and models would quickly become unwieldy so the first order of business was to implement some sparse vector operations. Here I chose to represent vectors as hash maps where non-zero elements of a vector are stored with their index as a key and the value of the entry as value.

(defn add
	"Returns the sparse sum of two sparse vectors x y"
	[x y] (merge-with + x y))

(defn inner
	"Computes the inner product of the sparse vectors (hashes) x and y"
	[x y] (reduce + (map #(* (get x % 0) (get y % 0)) (keys y))))

(defn norm
	"Returns the l_2 norm of the (sparse) vector v"
	[v] (Math/sqrt (inner v v)))

(defn scale
	"Returns the scalar product of the sparse vector v by the scalar a"
	[a v] (zipmap (keys v) (map * (vals v) (repeat a))))

(defn project
	"Returns the projection of a parameter vector w onto the ball of radius r"
	[w r] (scale (min (/ r (norm w)) 1) w))

The only slightly tricky thing here is the use of zipmap to scale a sparse vector by mapping all the keys in the original vector to their values times a scalar multiple a.

The other bit of framework code I required was to parse the training data. The format is a simple version of that used by SVMlight. Each line of the text file containing the training data is of the form:

y k_1:v_1 k_2:v_2 ... k_n:v_n

where y is the label (either 1 or -1), each k_i is an integer key representing a feature index, and each v_i is a floating point value.

The Clojure code to parse this format is a pretty straight-forward application of regular expressions:

(defn parse-feature 
	[string] 
	(let [ [_ key val] (re-matches #"(\d+):(.*)" string)]
		[(Integer/parseInt key) (Float/parseFloat val)]))

(defn parse-features
	[string]
	(into {} (map parse-feature (re-seq #"[^\s]+" string))))

(defn parse
	"Returns a map {:y label, :x sparse-feature-vector} parsed from given line"
	[line]
	(let [ [_ label features] (re-matches #"^(-?\d+)(.*)$" line) ]
		{:y (Float/parseFloat label), :x (parse-features features)}))

The main parsing function parse takes a whole line in this format as input and returns a hash map with key :y giving the label of the example and :x giving a hash map representing the feature vector.

Finally, the code to perform a single update step for a model given an example is built using some helper functions. The loss is computed by hinge-loss, the function correct performs a single gradient descent step, and report is just for diagnostics and prints some simple statistics about the model and its performance.

(defn hinge-loss
	"Returns the hinge loss of the weight vector w on the given example"
	[w example] (max 0 (- 1 (* (:y example) (inner w (:x example))))))
	
(defn correct
	"Returns a corrected version of the weight vector w"
	[w example t lambda]
	(let [x   (:x example)
		  y   (:y example)
		  w1  (scale (- 1 (/ 1 t)) w)
		  eta (/ 1 (* lambda t))
		  r   (/ 1 (Math/sqrt lambda))]
		(project (add w1 (scale (* eta y) x)) r)))

(defn report
	"Prints some statistics about the given model at the specified interval"
	[model interval]
	(if (zero? (mod (:step model) interval))
		(let [t      (:step model)
		      size   (count (keys (:w model)))
		      errors (:errors model) ]
			(println "Step:" t 
				 "\t Features in w =" size 
				 "\t Errors =" errors 
				 "\t Accuracy =" (/ (float errors) t)))))

(defn update
	"Returns an updated model by taking the last model, the next training 
	 and applying the Pegasos update step"
	[model example]
	(let [lambda (:lambda model)
		  t      (:step   model)
		  w      (:w      model)
		  errors (:errors model)
		  error  (> (hinge-loss w example) 0)]
		(do 
			(report model 100)
			{ :w      (if error (correct w example t lambda) w), 
			  :lambda lambda, 
			  :step   (inc t), 
			  :errors (if error (inc errors) errors)} )))

As you can see, this function returns a new, updated model as a hash that contains the feature weights :w as well as several other useful bits of information including the culmulative number of errors (in :errors) and the total number of update steps (in :steps). The parameter which controls the amount of regularisation is also passed along in the model (in :lambda) for convenience.

A brief aside: If I have one criticism of Clojure as a language it’s that implementing numerical procedures is a real pain. Prefix notation (while neatly side-stepping problems of operator precedence) is just a lot harder to read than the infix notation that many non-Lisp languages use.

Now the update step is implemented, training a model online from a sequence of examples is a simple application of reduce. The following code repeated calls (update model example) where each example is taken from the sequence examples and the model output by the last call to update is used as input for the next.

(defn train
	"Returns a model trained from the initial model on the given examples"
	[initial examples]
	(reduce update initial examples))

All that’s needed now is a main method to read examples from the standard input, parse them into vectors and train a model from some starting point:

(defn main
	"Trains a model from the examples and prints out its weights"
	[]
	(let [start 	{:lambda 0.0001, :step 1, :w {}, :errors 0} 
		  examples 	(map parse (-> *in* BufferedReader. line-seq)) 
		  model     (train start examples)]
		(println (map #(str (key %) ":" (val %)) (:w model)))))

When finished the main function prints the weights for the final trained model to the command line in a format similar to the input data. The training starts with an empty model and a regularisation constant of 0.0001 (as was used in the paper describing Pegasos).

The full version of the code is available at GitHub.

Running It

To see whether the algorithms (or at least my implementation of it) performs as advertised I ran it on the aforementioned RCV1 data set.

This is a big data set.

The gzipped version of the full data set weighs in at 423Mb. Understandably, I’m not going to host a file that size so to get the full data set it you will have to follow the instructions at Léon Bottou’s SGD page and make it yourself. However, for the purposes of this blog post I’ve created a 2,000 example version called train2000.dat.gz that is checked into the repository.

With the training data in hand I ran my implementation of Pegasos (in the file sgd.clj) like so:

$ zless train2000.dat.gz | clj sgd.clj > output.txt
Step: 100 	 Features in w = 2145 	 Errors = 64 	 Accuracy = 0.64
Step: 200 	 Features in w = 3333 	 Errors = 123 	 Accuracy = 0.615
Step: 300 	 Features in w = 4051 	 Errors = 175 	 Accuracy = 0.5833333
Step: 400 	 Features in w = 4755 	 Errors = 229 	 Accuracy = 0.5725
Step: 500 	 Features in w = 5236 	 Errors = 276 	 Accuracy = 0.552
Step: 600 	 Features in w = 5576 	 Errors = 318 	 Accuracy = 0.53
Step: 700 	 Features in w = 5870 	 Errors = 356 	 Accuracy = 0.50857145
Step: 800 	 Features in w = 6050 	 Errors = 388 	 Accuracy = 0.485
Step: 900 	 Features in w = 6325 	 Errors = 418 	 Accuracy = 0.46444446
Step: 1000 	 Features in w = 6578 	 Errors = 444 	 Accuracy = 0.444
Step: 1100 	 Features in w = 6747 	 Errors = 471 	 Accuracy = 0.42818183
Step: 1200 	 Features in w = 6934 	 Errors = 502 	 Accuracy = 0.41833332
Step: 1300 	 Features in w = 7109 	 Errors = 526 	 Accuracy = 0.40461537
Step: 1400 	 Features in w = 7300 	 Errors = 555 	 Accuracy = 0.39642859
Step: 1500 	 Features in w = 7515 	 Errors = 592 	 Accuracy = 0.39466667
Step: 1600 	 Features in w = 7655 	 Errors = 615 	 Accuracy = 0.384375
Step: 1700 	 Features in w = 7836 	 Errors = 644 	 Accuracy = 0.37882352
Step: 1800 	 Features in w = 8040 	 Errors = 672 	 Accuracy = 0.37333333
Step: 1900 	 Features in w = 8239 	 Errors = 697 	 Accuracy = 0.3668421
Step: 2000 	 Features in w = 8425 	 Errors = 718 	 Accuracy = 0.359

As you can see the algorithm slowly adds more and more features to the weight vector and, as a result, slowly improves the accuracy.

The reported accuracy is simply the cumulative total number of errors divided by the number of steps. This is a fairly pessimistic take on how the later models are performing. In the last 100 examples the models made a combined total of only 19 mistakes so the final model accuracy is probably closer to 20% than 35%.

Performance

My biggest issue with my implementation of online learning in Clojure is that it is too slow. The 2,000 example test described above took about 40 seconds to complete.

These algorithms are meant to be ridiculously fast. Léon Bottou reports training times for his C++ stochastic gradient descent algorithm on the full 780k example RCV1 data set of 1.4 seconds!

Granted I was timing both the parsing and training of the data but on the other hand I’m using less than 0.3% of the data. Indeed, a quick test shows that just parsing the 2,000 examples takes less than 2 seconds so just training on the 2,000 examples takes 35 seconds or more.

Firing up the JVisualVM to see where my code is spending most of its time reveals that a lot of time is spent getting variable values and looking up values in maps.

The performance culprit then is very likely my hastily thrown together sparse vector “library” built from hash maps. Although hash maps are fast there is still a lot of overhead in packing float and integer values in and out of Java Objects and I suspect this is where most of the time is wasted.

If I have time to write a next version, I’ll make use of the sparse vector data structures in the Java Parallel Colt library.

Conclusions

Despite its lack of speed, I was impressed with how easy it was to implement an online algorithm in Clojure. Minus the comments, the whole thing – vector operations, reporting, data parsing and training – weighs in at less than 100 lines of code.

Given Clojure’s ability to call high-performance Java libraries such as Parallel Colt, I’m optimistic that I can keep the terseness and transparency of the code and get performance comparable to the C++ implementations. I would also like to experiment with exploiting Clojure’s concurrency features to chunk and parallelise the main training algorithm. I suspect that this will be relatively straight-forward and, with a bit of tuning I should get good performance on a multi-core machine.

There’s a good discussion of Pegasos over at the LingPipe blog.
↩

Minilight in Clojure: Triangles

2009-04-11T00:00:00+10:00

Previously on “Porting Minilight to Clojure”, our intrepid programmer (a.k.a, yours truly) braved lists, cross products and testing suites to bring you a namespace full of vector operations. In this, the nail-biting sequel, I use these to build up some simple geometry in the form of triangles.

House-keeping…

Thanks to some feedback from the Clojure mailing list, I’ve made a couple of small changes to the way I’ve set up this project:

I now use of fully qualified names. That is, instead of a package called vec, I will use the longer, Java-style mreid.minilight.vec to avoid potential name collisions.
The code now lives under a src directory in folders corresponding to their fully-qualified package name. This meant adding a .clojure file with the text src in it so that my clj script can find the code.
The tests now define a class and are executed via a main method. This is so that IDE users won’t have the tests run every time the test.clj file is compiled.

Tests are now run from a simple shell script runtests.sh which has the following contents:

#!/bin/bash
JARS=$HOME/Library/Clojure/lib
CP=.:./src:$JARS/clojure.jar:$JARS/clojure-contrib.jar
TEST="src/mreid/minilight/test/all.clj"
MAIN="(mreid.minilight.test.all/-main)"

java -cp $CP clojure.main -i $TEST -e "$MAIN"

…and a Bug-Fix

In the process of writing some new code I found a bug in my cross product implementation. After a bit of head-scratching, I realised that the form I use to extract a particular component from a vector assumes that the underlying data structure is a Clojure vector. I was using this (v 0) to get at the x-coordinate of the vector v which works fine for ([1 2 3] 1) but fails spectacularly when the vector is computed as a lazy sequence. For example:

Clojure
user=> ((map - [1 2 3] [1 1 1]) 1)
java.lang.ClassCastException: clojure.lang.LazySeq cannot be cast to
clojure.lang.IFn (NO_SOURCE_FILE:0)

I used map in several places to compute addition, subtraction and scaling of vectors so when these were passed into cross they blew up.

I’ve added some new tests to test/vec.clj that test for this situation explicitly:

;; --- src/mreid/minilight/test/vec.clj ---
(def dyn100 (sub [1 2 3] [0 2 3]))
(def dyn010 (sub [1 2 3] [1 1 3]))

(deftest test-dynamic-add
  (is (= [1 1 0]  (add dyn100 dyn010))))

(deftest test-dynamic-sub
  (is (= [1 -1 0] (sub dyn100 dyn010))))

(deftest test-dynamic-scale
  (is (= [2 0 0]    (scale 2 dyn100))))

(deftest test-dynamic-cross
  (is (= [0 0 1]   (cross dyn100 dyn010))))

and changed the definition of cross to use the sequence friendly nth instead:

;; --- src/mreid/minilight/vec.clj ---
(defn cross 
    "Returns the cross product vector for the 3D vectors v1 and v2."
    [v1 v2] 
    [ (- (* (nth v1 1) (nth v2 2)) (* (nth v1 2) (nth v2 1)))
      (- (* (nth v1 2) (nth v2 0)) (* (nth v1 0) (nth v2 2)))
      (- (* (nth v1 0) (nth v2 1)) (* (nth v1 1) (nth v2 0))) ])

I guess this is one of those instances where static typing would have helped me realise my mistake earlier.

Triangles

Triangles are the simplest possible polygon. They are defined by three vectors, one for each vertex. As we will be using triangles to model surfaces in a ray-tracer, I will also need two other vectors to define how light is reflected and emitted from a surface. In Clojure, this is neatly captured using a struct:

;; --- src/mreid/minilight/triangle.clj ---
;; A structure and functions for defining and querying triangles.
(ns mreid.minilight.triangle
  (:use mreid.minilight.vec))
 
(defstruct triangle
  :vertices ; Collection of 3 vectors
  :reflect  ; Vector with all values in [0,1)
  :emit     ; Vector with positive values
)

To interrogate the structure and calculate related quantities I will also need a number of simple functions:

;; --- src/mreid/minilight/triangle.clj ---
(defn vertex
  "Returns (the zero-indexed) vertex i in the triangle t"
  [t i] (nth (:vertices t) i))

(defn edge
  "Returns the edge in the triangle t from vertex i to vertex j"
  [t i j] (sub (vertex t j) (vertex t i)))

(defn tangent
  "Returns a unit vector tangent to the given triangle t"
  [t] (normalise (edge t 0 1)))

(defn normal
  "Returns a vector normal to the given triangle t (edge01 x edge12)"
  [t] (cross (edge t 0 1) (edge t 1 2)))

(defn unit-normal
  "Returns a unit vector normal to the given triangle t"
  [t] (normalise (normal t)))

(defn area
  "Returns the are of the given triangle t"
  [t] (/ (norm (normal t)) 2))

This is all very straight-forward and, once again, Clojure just “gets out of the way” and make it very easy to express each of these simple functions on a single line and with a minimum of fuss.

Unit tests for these can be found in the file test/triangle.clj. This consists of a few simple triangle structures and then a number of tests on them:

;; --- src/mreid/minilight/test/triangle.clj ---
;; Tests for triangle.clj using the test-is library.
(ns mreid.minilight.test.triangle
  (:use mreid.minilight.triangle)
  (:use clojure.contrib.test-is))

(def xytriangle
  (struct triangle 
     [ [0 0 0] [1 0 0] [0 1 0] ] ; Triangle in xy-plane
     [0.5 0.5 0.5]               ; Reflectivity
     [1 1 1]                     ; Emitivity 
  ))

(def y2ztriangle
  (struct triangle 
     [ [0 0 0] [0 2 0] [0 0 1] ] ; Triangle in xy-plane
     [0.5 0.5 0.5]               ; Reflectivity
     [1 1 1]                     ; Emitivity 
  ))

(def zxtriangle
  (struct triangle 
     [ [-10 5 -10] [-9 5 -10] [-10 5 -9] ] ; Triangle parallel to zx-plane
     [0.5 0.5 0.5]                         ; Reflectivity
     [1 1 1]                               ; Emitivity 
  ))

(deftest test-vertex
  (is (= [0 0 0] (vertex xytriangle 0)))
  (is (= [1 0 0] (vertex xytriangle 1)))
  (is (= [0 1 0] (vertex xytriangle 2)))

  (is (= [0 0 0] (vertex y2ztriangle 0)))
  (is (= [0 2 0] (vertex y2ztriangle 1)))
  (is (= [0 0 1] (vertex y2ztriangle 2))))

(deftest test-edge
  (is (= [1 0 0] (edge xytriangle 0 1)))
  (is (= [0 1 0] (edge xytriangle 0 2)))
  (is (= [-1 1 0] (edge xytriangle 1 2)))
  (is (= [1 -1 0] (edge xytriangle 2 1)))

  (is (= [0 2 0] (edge y2ztriangle 0 1)))
  (is (= [0 0 1] (edge y2ztriangle 0 2)))
  (is (= [0 -2 1] (edge y2ztriangle 1 2)))
  (is (= [0 2 -1] (edge y2ztriangle 2 1))))

(deftest test-tangent
  (is (= [1 0 0] (tangent xytriangle)))
  (is (= [0 1 0] (tangent y2ztriangle))))

(deftest test-normal
  (is (= [0 0 1] (normal xytriangle)))
  (is (= [2 0 0] (normal y2ztriangle))))

(deftest test-unit-normal
  (is (= [0 0 1] (unit-normal xytriangle)))
  (is (= [1 0 0] (unit-normal y2ztriangle))))

(deftest test-area
  (is (= 0.5 (area xytriangle)))
  (is (= 1   (area y2ztriangle))))

There are three main functions left to port: one that computes a bounding-box for a triangle, one that tests whether a vector intersects a triangle, and one that randomly samples a point from a given triangle.

Bounding Boxes

The following function creates the smallest box containing all the points belonging to a triangle and then slightly expands it. It is pretty straight-forward, the only slightly tricky part is the higher-order function tweak. This takes in either a + or a - and will return a function that expands or shrinks its input by the specified TOLERANCE. This is used by bounding-box to pull the point of a triangle closest to the origin about 0.1% closer and push the furthest point away by 0.1%.

The reason for this function is that without it I would have have two copies of the code inside the function returned by tweak with only a sign change. I figured this was a more DRY was of writing it.

;; --- src/mreid/minilight/triangle.clj ---
(def TOLERANCE (/ 1.0 1024.0))
(defn tweak
  "Returns a function that adds or subtracts a small amount"
  [add-or-sub]
  (fn [x] (add-or-sub x (* (+ (Math/abs x) 1.0) TOLERANCE))))

(defn bounding-box
  "Computes the bounding box for a triangle t, returning the result as
  a list of two vectors [lower-corner upper-corner]." 
  [t]
  (let [vs (:vertices t)]
    [ (map (tweak -) (apply map min vs))
      (map (tweak +) (apply map max vs)) ]))

The above code is definitely not the most efficient way of computing and expanding a bounding-box. Indeed, the Ruby code that I’m cribbing this from explicitly optimises this using a single loop through each of a triangle’s vectors. In the interests of simplicity, I’ve opted for more two sweeps through each vector—once for the min/max and once for the tweak.

When testing these functions, I’ve opted for hand-building the correct values as constants defined in terms of the tolerance and then testing the output of tweak and bounding-box against these:

;; --- src/mreid/minilight/test/triangle.clj ---
(def tweak+0 (* 1 TOLERANCE))
(def tweak-0 (* -1 TOLERANCE))
(def tweak+1 (+ 1 (* 2 TOLERANCE)))
(def tweak-1 (- 1 (* 2 TOLERANCE)))
(def tweak+2 (+ 2 (* 3 TOLERANCE)))
(def tweak-2 (- 2 (* 3 TOLERANCE)))

(deftest test-tweak
  (is (= tweak+0 ((tweak +) 0)))
  (is (= tweak-0 ((tweak -) 0)))
  (is (= tweak+1 ((tweak +) 1)))
  (is (= tweak-1 ((tweak -) 1)))
  (is (= tweak+2 ((tweak +) 2)))
  (is (= tweak-2 ((tweak -) 2))))

(deftest test-bounding-box
  (is (= [[tweak-0 tweak-0 tweak-0] [tweak+0 tweak+2 tweak+1]] 
         (bounding-box y2ztriangle))))

Apply Liberally

One discovery I made while attempting to write this was the apply method. Its (very terse) documentation says

clojure.core/apply
([f args* argseq])
  Applies fn f to the argument list formed by prepending args to argseq.

In English, what it really does is treat a sequence of items as though they were passed in as arguments to the given function. In my case, I wanted to compute the point-wise minimum of three vectors. If these were, say, [1 2 3], [0 5 5] and [1 1 1] I would simply map min onto them as arguments:

Clojure
user=> (map min [1 2 3] [0 5 5] [1 1 1])
(0 1 1)

This works because min can take an arbitrary number of arguments (e.g., min 1 0 1 gives 0) and map, when given several sequences, will pull items from each in parallel and pass them to the function it is mapping.

However, the problem I faced here was that the three vectors were inside a sequence and so naïvely trying the following gives the wrong result:

user=> (map min [ [1 2 3] [0 5 5] [1 1 1] ])
([1 2 3] [0 5 5] [1 1 1])

This is where apply comes in. It strips away the container and presents its contents to the function being applied as arguments, along with any extra arguments before the sequence:

user=> (apply map min [ [1 2 3] [0 5 5] [1 1 1] ])
(0 1 1)

Intersection

Much of the main loop of a ray-tracing algorithm involves checking whether a ray will hit a surface. Minilight is no exception, so we have the following method for testing whether a ray, represented as a starting point and a direction vector, will intersect with a given triangle:

;; --- src/mreid/minilight/triangle.clj ---
(defn intersect
  "Finds the intersection with the triangle t of the ray starting at r0 in 
  direction rd. The returned value is a positive number a such that r0 + a.rd 
  is contained within t, or nil if there is no such a."
  [t r0 rd]
  (let [ e01    (edge t 0 1)
         e02    (edge t 0 2)
         p      (cross rd e02)
         invdet (invdet e01 rd e02) ]
    (if (number? invdet)
      (let [ v0 (vertex t 0)
             tv (sub r0 v0)
             u  (* (dot tv p) invdet) ]
        (if (and (>= u 0) (<= u 1))
          (let [ q (cross tv e01)
                 v (* (dot rd q) invdet) ]
            (if (and (>= v 0) (<= (+ u v) 1))
              (let [a (* (dot e02 q) invdet)]
                (if (>= a 0) a)))))))))

I’m least happy with the style of this code. The repeatedly nested let/if blocks seem ugly but I cannot see a more elegant way to do this sort of thing. Partly, it’s because the series of conditional tests are essential to what is being computed, but I can’t help but feel there is a lot of what Brook’s calls ”accidental complexity” in there as well.

Suggestions on how to improve this function are very welcome.

We can, at least, test that it is working though. Here I check whether hand-picked rays intersect on of the test triangles or not:

;; --- src/mreid/minilight/test/triangle.clj ---
(deftest test-intersect
  (is (= 1 (intersect xytriangle [0 0 1] [0 0 -1])))
  (is (= 2 (intersect xytriangle [0 0 2] [0 0 -1])))
  (is (= 1 (intersect xytriangle [0.9 0 1] [0 0 -1])))
  (is (= 1 (intersect xytriangle [0.1 0.1 -1] [0 0 1]))))

(deftest test-no-intersect
  (is (nil? (intersect xytriangle [0 0 1] [0 0 1]))) ; Dir. is opposite
  (is (nil? (intersect xytriangle [0 0 1] [1 0 0]))) ; Dir. is parallel
  (is (nil? (intersect xytriangle [0 0 2] [0 1 -1])))) ; Goes wide

Sampling

Finally, the renderer needs a way of choosing a random point uniformly from a given triangle. I’ve lifted the formula for choosing the point at random directly from the Ruby code and translated as directly as possible into Clojure.

;; --- src/mreid/minilight/triangle.clj ---
(def rnd (java.util.Random.))
(defn sample-point
  "Returns a random point as a vector from inside the given triangle t"
  [t]
  (let [ sqr1 (Math/sqrt (.nextFloat rnd))
         r2   (.nextFloat rnd)
         a    (- 1 sqr1)
         b    (* (- 1 r2) sqr1) ]
    (add (vertex t 0)
        (add (scale a (edge t 0 1)) 
             (scale b (edge t 0 2))))))

By its very nature, sample-point is non-deterministic. This makes testing a little unusual. Rather than testing whether a specific function call returns a specific value, I’ve opted for testing an invariant. In particular, if I sample a point from a triangle and translate it one unit along its normal then a ray starting at that point which points back along the normal should intersect the original triangle:

;; --- src/mreid/minilight/test/triangle.clj ---
(defn random-ray
  "Returns [r0 rd] where rd is the unit normal of the triangle t and r0 is a 
   random point on t translated by -rd" 
  [t]
  (let [ray-direction (unit-normal t)]
    [ (sub (sample-point t) ray-direction) 
      ray-direction ]))

(deftest test-sample-point
  (let [xy-random-ray (random-ray xytriangle)
        zx-random-ray (random-ray zxtriangle)]
    (is (= 1 (apply intersect xytriangle xy-random-ray)))
    (is (= 1 (apply intersect zxtriangle zx-random-ray)))))

Running this test on different occasions will check whether the invariant holds for different points. A down-side is that this test is not strictly repeatable and may fail very rarely. This may make tracking down a failing test difficult (even though the failure output will offer clues). The upside, however, is that the coverage for the intersect function is increased.

Summary

I’m definitely feeling more confident with Clojure even though I cannot instinctively count parentheses yet. Structures in Clojure feel very natural and are easy to work with and many of the functions I needed can still be expressed concisely. That said, the more mathematically involved functions didn’t lend themselves to Clojure’s prefix notation.

I suspect I’ll find these easier to read as time goes on but it is one case where simplicity of S-expressions falls short. Perhaps there are common ways of breaking up or otherwise formatting formulae that makes them easier to work with but they are far from obvious to me.

As usual, the code is on GitHub, this time tagged with sap-2.1. See you there!

Updates

15 April 2009: Fixed a bug spotted by HXA7241 in sample-point as well as a related bug in intersect. Updated the relevant tests to reproduce the bug.

Minilight in Clojure: Vectors

2009-04-05T00:00:00+11:00

Now that I am happy with my Clojure set up, it is time to dive in and try to implement something using it. As it happens, I recently discovered the minilight project which aims to implement a “minimal global illumination renderer” in several languages including Ruby, Python, Java, Scala, and C++. I’ve dabbled with 3D graphics in the past and since there is currently no implementation¹ in Clojure I thought porting minilight to Clojure would be a great way to start.

In this first part, I’ll describe how I ported the Vector3fc class from the Ruby version of minilight. I am keeping all my code for this project over at GitHub. The code I’ll be talking about here is tagged ’sap-1’.

I’m going to assume you have a basic familiarity with Clojure. Skimming over this overview of the Clojure syntax would be a good place to start.

100 Functions on One Data Structure

Vector manipulation is central to any 3D rendering code and Minilight is no exception. In object-oriented languages like Java, Ruby, Python and C++, the general practice is to create a new class for vectors and define methods for their manipulation.

It is possible to do OO programming in LISP-like languages like Clojure but it is not idiomatic. As Alan Perlis put it:

It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.

So I’ve instead opted for defining a namespace vec along with a number of operations that treat lists of numbers as vectors and all live in the file vec.clj.

;; --- vec.clj ---
;; A simple vector package that defines functions for working with geometrical 
;; vectors.
(ns vec)

; Constants
(def origin [0 0 0])    ; Zero vector in 3D
(def epsilon 0.000001)  ; Tolerance for equality

; Helper functions
(defn approx0 
    "Returns true iff the value x is within epsilon of 0"
    [x] (< (Math/abs x) epsilon))

(defn clamp
    "Constrains all elements in v to be between vmin and vmax"
    [vmin vmax v] (map (fn [x] (max vmin (min vmax x))) v))

; Binary operators
(defn dot 
    "Returns the value of dot product of the vectors v1 and v2"
    [v1 v2] (reduce + (map * v1 v2)))
    
(defn add 
    "Returns a vector that is the sum of the vectors v1 and v2"
    [v1 v2] (map + v1 v2))
    
(defn sub 
    "Returns a vector that when added to v1 gives v2"
    [v1 v2] (map - v1 v2))

(defn cross 
    "Returns the cross product vector for the 3D vectors v1 and v2."
    [v1 v2] 
    [ (- (* (v1 1) (v2 2)) (* (v1 2) (v2 1)))
      (- (* (v1 2) (v2 0)) (* (v1 0) (v2 2)))
      (- (* (v1 0) (v2 1)) (* (v1 1) (v2 0))) ])

; Scalar operators
(defn scale 
    "Returns the vector that is m times the vector v"
    [m v] (map #(* m %) v))

; Unary operators
(defn norm 
    "Returns the (Euclidean) length of the vector v"
    [v] (Math/sqrt (dot v v)))

(defn normalise 
    "Returns a vector of unit length in the same direction as v"
    [v] (scale (/ 1 (norm v)) v))

; Determinants
(defn det 
    "Returns the determinant of vectors v1 v2 and v3, i.e. v1.(v2 x v3)"    
    [v1 v2 v3] (dot v1 (cross v2 v3)))

(defn invdet
    "Returns the inverse det. of v1, v2 and v3 or nil if it is too close to 0"
    [v1 v2 v3]
    (let [d (det v1 v2 v3)]
        (if (approx0 d)
            nil
            (/ 1.0 d))))

The string that appears after a function’s name in a defn form documents the function. You can view this documentation for any function using the doc function. For example, we can see the documentation for add like so:

Clojure
user=> (use 'vec)
user=> (doc add)
-------------------------
vec/add
([v1 v2])
  Returns a vector that is the sum of the vectors v1 and v2

Here are some other observations about how I implemented these functions and some of the Clojure-related tricks I learnt along the way.

Apart from the function cross, all of these definitions are very concise thanks to the use of the higher-order functions map, reduce and every? which apply functions to sequences. Writing the functions this will almost certainly be slower than dealing with arrays of floats but I can always aim to improve efficiency later and thus heed Don Knuth’s warning that “premature optimization is the root of all evil (or at least most of it) in programming.”
The function Math/abs denotes a call to the static method abs() in the class Math that comes standard with Java.
The form #(* m %) in the definition of scale is shorthand for (fn [x] (* m x))—that is, the function that multiplies its input by m. The shorthand is known as a reader macro. Clojure expands forms like these into their equivalent long version when it first encounters it.
The let form is Clojure’s way of assigning local bindings. The first vector² [d (det v1 v2 v3)] binds d to the determinant of v1, v2 and v3. In general, a let can take many pairs of symbol/value bindings like so: let [a 1 b 2 c3] (...). The form after the binding is evaluated in the context of the bindings.

Testing, testing

Writing tests for your code is almost always a good idea, especially if most of the code being tested does some kind of number-crunching. Since I plan to rewrite and optimise my vector functions later it is definitely a good idea.

Clojure comes with some built-in support for testing via its special form :test which allows tests to sit right next to the code, much like documentation. However, I prefer separating tests into their own files so I’ve been using Stuart Sierra’s test-is library that comes with the clojure-contrib package. It is a basic unit testing library that allows me to write simple tests for the above vector methods.

Here is a small suite of tests for the vector functions. These appear in the file test/vec.clj which means the appropriate namespace is test.vec. Note the :use form after the definition which imports all the functions in vec and the test-is library.

;; --- test/vec.clj ---
;; Tests for vec.clj using the test-is library.
(ns test.vec
    (:use vec clojure.contrib.test-is))

(deftest test-approx0
    (is (approx0 -0.000000001))
    (is (not (approx0 0.001))))

(deftest test-clamp
    (is (= [1 0 0]       (clamp 0 1 [2 -1 -1])))
    (is (= [0.5 0.5 0.5] (clamp 0 1 [0.5 0.5 0.5]))))

(deftest test-dot
    (is (= 0 (dot [1 1 1] [-1 1 0])))
    (is (= 2 (dot [1 2 3] [3 -2 1]))))

(deftest test-add
    (is (= [1 1 0]  (add [1 1 0] origin)))
    (is (= [-2 3 4] (add origin [-2 3 4])))
    (is (= [1 2 3]  (add [1 -1 2] [0 3 1]))))

(deftest test-sub
    (is (= [1 2 3]  (sub [1 2 3] origin)))
    (is (= [0 0 0]  (sub [1 2 3] [1 2 3])))
    (is (= [-2 0 2] (sub [1 2 3] [3 2 1]))))
    
(deftest test-scale
    (is (= [0 0 0]    (scale 0 [1 2 3])))
    (is (= [-1 -2 -3] (scale -1 [1 2 3])))
    (is (= [2 4 6]    (scale 2 [1 2 3]))))

(deftest test-cross
    (is (= [-3 6 -3] (cross [1 2 3] [4 5 6])))
    (is (= [0 0 1]   (cross [1 0 0] [0 1 0])))
    (is (= [0 0 0]   (cross [1 0 0] [1 0 0]))))

(deftest test-norm
    (is (= 0 (norm origin )))
    (is (= (Math/sqrt 3) (norm [1 1 1])))
    (is (= (Math/sqrt 2) (norm [1 0 -1]))))

(deftest test-normalise
    (is (= [(/ 2.0 7.0) (/ 3.0 7.0) (/ 6.0 7.0)] (normalise [2 3 6]))))

(deftest test-det
    (is (= 1 (det [1 0 0] [0 1 0] [0 0 1])))
    (is (= 3 (det [1 1 0] [1 2 3] [4 5 6])))
    (is (= 0 (det [1 1 1] [1 2 3] [4 5 6]))))
    
(deftest test-invdet
    (is (= 1 (invdet [1 0 0] [0 1 0] [0 0 1])))
    (is (= (/ 1 3) (invdet [1 1 0] [1 2 3] [4 5 6])))
    (is (nil? (invdet [1 1 1] [1 2 3] [4 5 6]))))

As you can see, these are fairly basic tests but they do exercise all of the important functions in vec.clj.

The tests can be run from the interactive session of Clojure like so:

Clojure
user=> (use 'test.vec 'clojure.contrib.test-is)
user=> (run-tests 'test.vec)                   

Testing test.vec

Ran 11 tests containing 28 assertions.
0 failures, 0 errors.

No failures. Excellent!

To avoid having to run these tests manually, I’ve set up a master test file in test/test.clj that looks like this:

(ns test 
	(:use test.vec)
	(:use clojure.contrib.test-is))

(run-tests 'test.vec)

This will be make it easier to add new test suites as I write port more of Minilight and can be run as a script from the Terminal like so:

$ clj test/test.clj 

Testing test.vec

Ran 11 tests containing 28 assertions.
0 failures, 0 errors.

In Closing

Well, that’s covered the vector functions and their tests. In the next part I’ll port the Triangle class which defines the geometry and properties of a triangle.

I’d like to emphasise that I’m very new to Clojure so if you see that I’m doing something unidiomatic or just plain stupid in the above please let me know in the comments.

Shortly after starting my port of minilight I discovered that someone else had the same idea and is at about the same stage. It will be interesting to compare our respective approaches as time goes on, and I cannot promise that I won’t borrow an idea or two.
↩
This is slightly confusing since Clojure sequences defined using square brackets (e.g., [1 2 3]) are called “vectors” after the Java class of the same name.
↩

Setting Up Clojure for Mac OS X Leopard

2009-03-29T00:00:00+11:00

Clojure is a fairly new Lisp-like, functional language that is built on top of the JVM. It features great Java interoperability and is built from the ground up with concurrency in mind.

Below is a brief description of how to get Clojure up an running on Mac OS X Leopard. I also describe a small shell script that invokes Clojure and sets its classpath using a simple, project-specific configuration file.

The first step is to create a Clojure directory in your Library folder that contains the subfolder lib. Do this at your Terminal:

$ mkdir -p ~/Library/Clojure/lib
$ cd ~/Library/Clojure

Getting Clojure

The latest stable version of Clojure can be found here. At the time of writing the latest stable version is the zip file for Clojure release 20090320.

Once you’ve downloaded it, copy clojure.jar to the ~/Library/Clojure directory:

$ cp ~/Downloads/clojure/clojure.jar ~/Library/Clojure/lib/

To make Clojure’s interactive mode easier to you, you should grab the JLine library and install it as well.

First download jline-0.9.94.zip from the jline project site and then:

$ cp ~/Downloads/jline-0.9.94/jline-0.9.94.jar ~/Library/Clojure/lib/jline.jar

Startup Script

I’ve created a bash script called clj that I put in ~/Library/Clojure and symbolically link to from somewhere in my path.

This script sets up the Clojure classpath and, if it is present, also adds the contents of a .clojure file to the classpath before executing Clojure. It also adds the current directory to the classpath for ease of use.

You can download the script from my GitHub repository. Once you’ve got it, copy it to the Clojure directory and make it executable:

$ cp ~/Downloads/clj ~/Library/Clojure/
$ chmod u+x ~/Library/Clojure/clj

Now you will want to symbolically link to it from a directory in your $PATH Type echo $PATH at the Terminal to see a list of options. I have a directory ~/bin/ where I keep such things.

To make the link use, for example:

$ ln -s ~/Library/Clojure/clj ~/bin/clj

Now I can type clj from any directory and see:

$ clj
Clojure
user=> (= (* 6 8) 42)
false
user=>

I can also run Clojure programs. For example, suppose I have the following file called test.clj in the current directory:

(println "Hello, world!")

Then, when I run the following, I see:

$ clj test.clj
Hello, world!

Extending the Classpath

If you need any project-specific jar files added to the classpath when running Clojure, this can be done by putting a .clojure file in the same directory you will be running the project from.

For example, suppose I have a program ~/code/cafe/macchiato.clj that requires class from the Java libraries grinder.jar and frother.jar in the ~/code/cafe/lib/ directory.

I can easily create a file .clojure that specifies where Clojure can find these extra libraries:

$ cd ~/code/cafe
$ echo "lib/grinder.jar:lib/frother.jar" > .clojure

Then when I run:

$ clj macchiato.clj

from the ~/code/cafe directory, the earlier clj script will pick up the extra jar files from the .clojure file and add them to the classpath before invoking Clojure.

Installing and Using clojure-contrib

This step is optional. You only need to do it if you wish to use some of the extra, community-contributed Clojure libraries.

This step also requires git which is not a standard part of OS X Leopard but a simple OS X installer is available.

Once you have git installed, you can get the current version of the contributed libraries like so:

$ git clone git://github.com/kevinoneill/clojure-contrib.git

Now build the jar file using ant and copy it to the Clojure directory:

$ cd clojure-contrib
$ ant -Dclojure.jar=$HOME/Library/Clojure/lib/clojure.jar
$ cp clojure-contrib.jar ~/Library/Clojure/lib/

Now you should be able to use things like Stuart Sierra’s library. For example, suppose I was writing a simple vector library called vec.clj and wanted to put the following tests in test.clj in the same directory:

(ns test 
	(:require vec)
	(:use clojure.contrib.test-is))
	
(deftest test-cross-product
	(is (= [-3 6 -3] (vec/cross [1 2 3] [4 5 6])))
	(is (= [0 0 1]   (vec/cross [1 0 0] [0 1 0]))))
	
(run-tests)

Then, because the clojure-contrib.jar is on the classpath, I can run and see the following:

$ clj test.clj

Testing test

Ran 1 tests containing 2 assertions.
0 failures, 0 errors.

Success!

Editing

I use TextMate as my Clojure editor along with the Clojure bundle created by nullstyle.

It’s probably overkill for what I need since that bundle includes a working installation of Clojure (i.e., it doesn’t call the clj I discuss here). I edit in TextMate and then run sessions and scripts from the Terminal. All I’m really taking advantage of is the syntax-highlighting, auto-formatting and the online help (^H).

There are other options around such as Clojure modes for Vim and Emacs but I haven’t tried them.

In Closing

I wrote these notes mainly to document the sometimes frustrating processes of getting a flexible, easy-to-use Clojure environment set up. The ”Getting Started” page at the main Clojure site are great for getting a REPL up and running but didn’t help me at all when it came to using other jars and clojure-contrib. Of course, if I’m doing something here that is patently stupid, please let me know in the comments.

I keep a shorter version of these notes with more concise step-by-step instructions at the GitHub repository for this set up.

Hopefully, this short introduction will make it easier for others to get up and running with this great new language.

Updates

1 April 2009: Added notes about the editor I use and link to my Github repository.
4 August 2009: Fixed some incorrect paths in the instructions.