This post explains how to control Firefox from the command line with Telnet and Ruby. After presenting some context to explain why I think this hack represents an important area of concern in contemporary web application development, I'll show how to execute it with actual install directions and code samples.
This week, trying to solve exactly these types of problems, I discovered a tantalizing avenue towards addressing some of these questions: browser automation from the command line and from scripting languages. Here was my situation.
The easiest way to install JSSH is to download the JSSH.xpi and open it with Firefox which will offer to install the extension (if you're interested in compiling Firefox with it from scratch or installing an existing binary, you should read these instructions).
Start Firefox with JSSH
Once you've got a copy of Firefox with JSSH installed, you'll need to run it. You can do this by providing the correct options when launching Firefox from the command line. On Mac OS X, that looks like this:
/Applications/Firefox.app/Contents/MacOS/firefox -jssh &
The "&" at the end of that line will background your command so it doesn't take over your terminal session.
Once Firefox is running, we can use telnet to log into JSSH like so:
$ telnet localhost 9997
telnet: connect to address ::1: Connection refused
Connected to localhost.
Escape character is '^]'.
Load a URL from JSSH
Now that we're in, we can tell Firefox to load pages for us, thusly:
var w0 = getWindows()
var browser = w0.getBrowser()
Now, all we've got left to do is make it so that we can trigger this process from our application code. So, we'll...
Automate the Process with Ruby
Like any good scripting language, Ruby has a telnet library, which means that once we've got an instance of Firefox running with JSSH enabled, we can talk to it from Ruby whenever we want. Here's an example script that logs into the telnet shell and loads a series of URLs one at a time:
my_urls = ["http://urbanhonking.com/ideasfordozens", "http://atduskmusic.com", "http://grabb.it", "http://pdxpopnow.com"]
puts "starting telnet session"
firefox = Net::Telnet::new("Host" => "localhost", "Port" => 9997)
firefox.cmd "var w0 = getWindows()"
firefox.cmd "var browser = w0.getBrowser()"
# load each page
my_urls.each do |url|
sleep 10 # so that the browser has time to load even if the page is slow
I haven't really tackled this problem with JSSH, but I do have some leads. For example, here's how you get the html of the document:
[object XPCNativeWrapper [object HTMLDocument]]
If you want to explore this avenue further, one of the best places to look is Firewatir, a project to add Firefox support to the WATIR browser testing framework. They do lots of click-by-click automation and checking for results, so I'm sure they've figured out approaches for a lot of what you'd confront when screen scraping. The JSSH documentation itself is useful and clear but not the most in depth.
Happy automating! Let me know what you discover...