ITPoogle: An ITP-Specific Search Engine

25 August, 2010


This entry documents the first of my projects for this summer's ITP 4-in-4. 4-in-4s are occasional events entailing the execution of one complete project each day for four consecutive days. They're a great opportunity to finally do a daydream project you've been talking about for ages or to play with a new technology or technique you'd be meaning to try out.

Unsurprisingly, the ITP community has a lot of places it posts information online. There's the homeship, itp.nyu.edu, with all its offshoots, such as the pcomp site and the people directory. There's hundreds of student blogs, dozens of class sites, even our own wiki: ITPedia.

This is totally to be expected in a contemporary, technically literate, information-generating organization. The problem is: how to navigate it all? When you're looking for a specific piece of information — that tutorial about motors your classmate mentioned or a list of which alumni have previously worked with animatronics — it can become quite laborious to browse one-by-one through each of the many ITP-related sites. And a general Google search will tend to bury the ITP needle in a large stack of non-ITP hay.

Considering this, it occurred to me that what we needed was an ITP-specific search engine. Such a tool would let students explore and hunt across all the program's varied online resources from one familiar interface.

The problem is that, having done it before, I know from personal experience that building a search engine is no quick hack. However, I realized that, using the AJAX API for Google site search, I could easily put together a prototype of an ITP-specific search engine that would do at least part of the job — enough, hopefully, to discover if such a thing would be useful enough to consider building for real.

Hence, I would like to present: ITPoogle.com, the ITP-specific search engine. Enter a query and ITPoogle presents results in three columns: one from ITPedia, one from itp.nyu.edu (which includes all student blogs and class websites hosted on that domain), and a third from a growing list of class syllabi and other resources past and present on other domains.

I'll spend the rest of this post briefly discussing how I built ITPoogle since I think it's an interesting case study of using AJAX to explore interface and tool design quickly without having to setup any server components whatsoever.

I'd built a similar site in this style once before: rtumblr, another Google search mashup, this one designed to help Tumblr users find people who were reblogging them, a service Tumblr itself inexplicably fails to provide. I wrote up a full post about rtumblr at the time and if you read it and follow the links, you'll see that I'd built myself a few interesting tools for creating dynamic single-page sites in javascript: a library for handling templating, one for routing URLs, etc.

In the year and a half since I built rtumblr, I met the great Aaron Quint, and discovered that he'd also been thinking about dynamic, single-page javascript-driven sites. And AQ had taken the idea much further than I had, developing the full-featured and quite cool Sammy.js, a framework for solving exactly the common problems of these types of sites: routing, events, templating.

So, I decided I'd use Sammy.js as the basis for ITPoogle. I started the project by writing a simple HTML page with a search form, installing jQuery and Sammy.js on it, and getting Sammy configured to handle search URLs. The idea was to be able to provide permalinks for searches so that people can share results pages (like this: http://itpoogle.com/#/search/4-in-4) even though the site is pure javascript. Sammy's routing API makes stuff like that super easy; the basic outline of the code looks like this:

var app = $.sammy(function() {
this.get('#/search/:q', function() {
// do stuff here with params['q']
};
}
$(function() {
app.run();
});

And then I simply used jQuery to take over the submit event on my search form to change the URL, re-triggering Sammy's routing ("#q" here is the id of the input where the user enters their search query):

$("#search").submit(function(){
window.location.hash = "#/search/" + encodeURI($("#q").val());
return false;
});

Once I had the basic structure of the page in place, the next step was to start running the Google searches. I'd already written the start of a wrapper around the Google AJAX Search API when creating rtumblr, so I decided to extract that out into a library I called Giggle. Giggle is a relatively straightforward aid to using the Google Search API. It handles things like composing queries, iterating through pages of responses, and packaging the results into something convenient.

Initially, I built Giggle naively as a singleton object, but I quickly realized that doing so meant the danger of having results get intermingled when conducting multiple different searches simultaneously. So, I spent some time getting the scope right within Giggle so that I could create multiple instances of it, each of which could fire off their own searches without polluting the results coming from the others.

The advanced features of Javascript scope like this always trip me up when I come back to it after a break but it's not too complicated once you remember how it works.

In the end the Giggle API ends up looking like this:

var search = new Giggle();
search.q("ITP", {title: "ITP"}, function(results, locals){
console.log(locals.title);
console.log(results);
})