Accessing Subversion via SSH on a non-standard port

This may be common knowledge, but it took me a few minutes to figure out and I didn't want to have to spend a few minutes again in the future when I forget how to do this months down the line :).

There are two, somewhat easy ways to configure different options to SSH for SVN. The first option lets you make the options specific to SVN access; in other words, the options you use to access the SSH server will only affect SVN access. The second lets you make those options standard across any SSH access you make to that target server.

Let's say I have a server called "reposerver" that has SSH listening on port 12345. The SVN repository there is under "/var/svn/repos" and contains a project called "test-project". To check out that project via SVN+SSH, here's what you need to know:

Option one:

Basically, in your home directory (on a Linux system), there's a file .subversion/config. In that config file, there's a section called [tunnels]. Edit that section so that it has an entry like this:

[tunnels]
reposerver = $SVN_SSH ssh -p 12345

The "reposerver" part could be anything; I just named it the name of the server so it'd be easy to remember when I have to use it in a later step. What the rest of that entry does is call your SSH command with any of the normal SSH options, so if there are other options you'd normally use, you can also put them there.

Finally, when you're done with that file, save it. Then attempt to check out a project from your SVN server. Normally, you'd type "svn+ssh" as the protocol portion of the URL. But this time, you'll use the name of the "tunnel" you defined in your SVN config file:

svn co svn+reposerver://reposerver/var/svn/repos/test-project

It would definitely be nice if SVN supported passing ports along in the normal way URLs do (ie. http://server:port), but at least this gives you some level of configurability with different options in accessing the repository. In fact, you could, for example, use a different SSH implementation, or RSH, or whatever you wanted to do.


Option two:

You can edit the .ssh/config file in your home directory. Basically, you'd add a "Host" entry, and beneath that put any options you want to use whenever you access that host via SSH. For our example, the entry might look like this:

Host reposerver
     Port 12345

Once you add the options you want to your .ssh/config file (check out the SSH2 client config man page for other options and more information), when you attempt to access SVN on the SSH server, those options will be passed on, too. So our checkout command would look like this:

svn co svn+ssh://reposerver/var/svn/repos/test-project

Note, this time we're using the normal "svn+ssh" as the protocol; you don't need to worry about any "tunnel" names or anything like that, and that's the advantage to this approach: you don't need to remember anything out of the ordinary. However, if somehow you want different options for SVN SSH access and regular SSH access, then option one above will probably be better for you.

Using Groovy to connect to GMail

Lately, I've been working on a project where I want to allow someone (namely, my wife) to email to-do items to my GMail account, which an automated process would then read, find the relevant items, then add them to this really useful GTD app I've been using, Tracks. The language I've been using over the last year or so to automate things like this is Groovy, and Groovy actually makes a task like this fairly easy

GMail also has made it easier if you want to automate some of your mail handling like I do: they provide access via regular POP3 and IMAP. IMAP is particularly useful because, if you have created labels in GMail (and I only know one person who hasn't ;)), GMail IMAP will treat those labels as IMAP folders. So there are basically two parts to this task: actually connecting to the GMail IMAP server, then interacting with the labels/folders in your account.

Connecting to GMail

GMail requires you to use SSL to connect to their servers, so the first thing I had to do was get GMail's certificate into a Java keystore. Based in part on the instructions in this blog post by Vincent Liu, here's what I did:

First, I connected to GMail's IMAP service with openssl:

openssl s_client -connect imap.gmail.com:993 -showcert

From the output of that session, I copied the lines from "-----BEGIN CERTIFICATE-----" to "-----END CERTIFICATE-----" (inclusive), to a text editor and saved it as the file gmail.pem. To make the certificate usable by Groovy, I then used openssl again to convert the PEM file to a DER format that I could import via Java's keytool utility.

openssl x509 -in gmail.pem -inform PEM -out gmail.der -outform DER

Just to make sure things were on the up-and-up, I viewed the resulting certificate:

keytool -printcert -file gmail.der

Then I imported the certificate into my keystore:

keytool -import -alias gmail.imap -file gmail.der

Interacting with GMail

Once the certificate was taken care of, I could get down to actually connecting to GMail with Groovy. One of the best things about Groovy is that it can make use of all the Java libraries that already exist for doing things like connecting to mail servers. I used the JavaMail libraries to do the real work of connecting to GMail, and there is some good documentation out there on how to use it. Here are some of the basics, though:

First off, you need to import the JavaMail classes (as well as java.util.Properties for initiating a JavaMail connection):

import javax.mail.*
import java.util.Properties

and then set some properties for your mail Session:

String  gmailServer = "imap.gmail.com"
int gmailPort = 993

Properties props = new Properties();
props.setProperty("mail.store.protocol", "imaps");
props.setProperty("mail.imaps.host", gmailServer);
props.setProperty("mail.imaps.port", gmailPort.toString());

Note that the "mail.store.protocol" property is "imaps" NOT plain "imap". JavaMail added SSL support to all protocols in version 1.3.2.

Once you have those properties set, you can create a Session and connect to the GMail IMAP server (the Store, in JavaMail terms) via IMAPS:

def session = Session.getDefaultInstance(props,null)
def store = session.getStore("imaps")
store.connect(server, user, password)

With the connection to the Store, the next thing you probably want to do is see what's there. As I mentioned, GMail treats all your labels as IMAP folders. You can list the folders that are available, and then interact with the messages in whatever folder you're interested in:

store.defaultFolder.list().each { folder ->
    println folder.name
}

def inboxFolder = "INBOX"
def folder = store.getFolder(inboxFolder)
folder.open(Folder.READ_ONLY)
folder.messages.each { msg ->
            println "Subject: ${msg.subject}"
            println "Sent on: ${msg.sentDate}"
            println "From: ${msg.from}"
            println "Content: ${msg.content}"
            println "----------------------\n"
}

Once you're connected like this, you can interact with GMail as if it were any regular IMAP server, so check out the JavaMail API to see all the ways you can interact with a mail server.

Adding HTTP Basic Authorization to GroovyHTTP

A while back, I discovered a great little Groovy utility by Tony Landis called GroovyHTTP which allows you to generate web requests in Groovy. I've used it with great success on a number of projects.

Lately, I've been playing with a neat GTD web app named Tracks, which has a nice RESTful API for adding tasks, etc. To interact with the application via the API, though, you need to be able to authenticate via HTTP Basic authentication, and GroovyHTTP doesn't seem to support that. Since I really want to interact with Tracks on my internal network with Groovy, I've taken a stab at adding HTTP Basic authentication to GroovyHTTP. Of course, I realize that HTTP Basic is not the most secure authentication method (that's an understatement!), but when you need it, you need it.

It really boils down to adding one method to set the user and login on the GroovyHTTP object, then adding a Base64-encoded authorization string to the HTTP request headers. The extra header line ends up looking like this (for the user/password "Aladdin/open sesame"):

Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

To use the modified GroovyHTTP class to access a URL protected by Basic authentication, you can do something like this:

        def h = new GroovyHTTP(protectedUrl)
        h.setMethod('POST')
        h.setAuthorization(login, password)
        h.setParam('p1', p1)
        h.setParam('p2', p2)

Since I can't find any contact information for Tony Landis on his site, I figured I'd post the Groovy code here since he's released it under the BSD license. The Groovy class is available here: GroovyHTTP with HTTP Basic Authentication. NOTE: It looks like Google Docs has messed up the formatting in the Groovy class. When I get some time, I'll try to neaten it up.

Rendering complex collections as JSON in Grails

So I have a set of somewhat complex objects I need to render as JSON in Grails for some AJAX calls. Let's say I have an array of "Foo" objects, each of which contains a name and an array of "Bar" objects, each of which have a name and a location.

I want the JSON to look something like this (granted, there are a number of ways I could probably use simpler data structures with more frequent AJAX calls, but I'm trying to make as few trips to the server as possible) :

{"foos":
   [
       {
           "fooName":"fooOne",
           "bars":[
                   {"barName":"barOne","location":"here"},
                   {"barName":"barTwo","location":"there"},
                   {"barName":"barThree","location":"everywhere"}
                  ]
       },
       {
           "fooName":"fooTwo",
           "bars":[
                   {"barName":"barFour","location":"Istanbul"},
                   {"barName":"barFive","location":"Constantiople"},
                   {"barName":"barSix","location":"Nobody's Business"}
                  ]
       }
   ]
}

I tried for a while to get this to render using Grails JSONBuilder, but to no avail: the secondary arrays kept being built outside the Foo objects. Eventually, I came across this really helpful post: Mini-Guide to rendering JSON with Grails that pretty much described the things I had tried and the problems I encountered. The post suggested trying the JSON converter that's bundled with Grails and gave a fairly simple example of how to use it.

The JSON I wanted to build was a bit more complex than the example, though, so I went searching for more examples of how to use the JSON converter. In the examples I found, most of the structures didn't go as deeply as the one I wanted; most of them just included a list at the top level of the structure, using the Grails "list" method (e.g. Books.list(params), which would get you the listing of books with the param IDs). I needed to go one step deeper, so I thought I'd see if Grails would automatically recurse the Lists in my objects. No such luck: when I tried rendering like this "render foos as JSON", the first-level items rendered correctly, but the array of Bars was empty.

Finally, I just decided to try constructing the data structures myself since maybe the JSON converter had trouble navigating object graphs but not basic collections. Since Javascript doesn't really differentiate between Objects and Maps (which is basically how JSON works), I created a Map for each Bar, put them into an array which then went into a Foo Map containing the attributes for the Foo. Finally, I created an array of the Foo Maps, which I then rendered using the JSON converter.

That actually worked! The downside, though, is that this is pretty fragile: if the data I want to return from the Foo or Bar objects changes, I have to change how the containers are rendered.

I don't know how much of what I encountered in JSONBuilder or the JSON converter is by design (maybe trying to force less complex data structures), but it certainly would be helpful to be able to create these kinds of arbitrarily complex structures based on the objects themselves rather than having to create our own containers. I'll have to see if anyone is already working on that.

Here's the final code I ended up with (it's probably not idiomatic Groovy, but I'm more familiar with Java, so it was quick):

       import grails.converters.JSON
           . . .
       def getJson = {
           def fooList = new ArrayList()
           data.foos.each { foo ->
              
               def fooMap = new HashMap()
               fooMap.put("fooName", foo.fooName)
              
               // see what bars this foo has
               def bars = new ArrayList()
               foo.bars.each { bar ->
                   def barMap = new HashMap()
                   barMap.put("barName", bar.barName)
                   barMap.put("location", bar.location)
                   bars.add(barMap)
               }
               fooMap.put("bars", bars)

               fooList.add(fooMap)
           }

           def output = [
               foos: fooList
           ]

           render output as JSON
       }

Rich Web Experience 2008: the Nate Schutta Sessions

Thinking back on the RWE 2008 last week, I realized there were a number of sessions that really stood out as being well presented and extremely informative. In particular, the sessions by Nate Schutta (check out his blog at http://ntschutta/jat) were among some of the best I've seen. They were all clear, well-thought out, in-depth, and engaging. A number of his presentations were introductions to various popular Javascript libraries like Prototype, YUI, and Dojo. These were particularly good because he showed not only why the libraries were useful but also how to use them. He didn't leave it at theory or sales pitch, though; in each of his script library presentations, he coded a number of examples using the libraries in question. And here's the best part: in each presentation, he built the same type of applications (a simple PIM app, showing how to use a date picker, etc) but using the library under discussion. It made for a good comparison of the strengths and APIs of each library. Also, he showed in his examples how the different libraries could actually work together, building on the strengths of each. All in all, the content of his presentations was excellent.

In addition to the excellent content, though, his method of presentation was right up my alley. He did what I consider the best thing to do in using slide presentations: he used the slides to initiate discussion rather than regurgitating the points from the slides. Some presenters tend to read their slides and that's their presentation. That gets old really quickly. As well, I remember one time when I was in the Army and giving a briefing to a warrant officer being told that people can either read your slides or listen to you, not do both at the same time. Nate kept his slides clean and simple, usually just a few words or phrases per slide, which allowed him to emphasize his points and guide his discussion of the topic without the listeners getting distracted by a lot of visual noise on the screen. He actually did something really interesting, too, for his final presentation of the weekend (A Software Engineer's Guide to Usability): he had two different slide decks. The first contained his simple slides described above, and the other had more content grouped in bullet points and images. I guess people had complained that the slides didn't make good take-aways (which is probably true, unless you're an extremely thorough note taker) since they were more for guiding the discussion than posting information. I think it was a good tactic. in fact, I actually found myself referring to his "note slides" earlier today when writing up some notes for some of my colleagues.

I know some people don't like this form of presentation. In environments like this weekend, I tend to be a more audio learner, so Nate's presentation style really worked for me. But for people who are more visually oriented and who want to get their information mainly from the screen, that "simple" form of presentation might cause problems. But overall, I'd say his presentations, from my point of view, were among the best there, and actually were among the best I've seen in attending other NFJS events, too.

Tracking visitor statistics on a Blogger blog

So I wanted to track various statistics on who's visiting my blog, and it didn't appear that there was any sort of tool available via Blogger.  So what I ended up doing is hooking my blog into Google Analytics.

First, I had to sign up for a free Google Analytics account.  When it asked for the URL of my site, I put in the URL to this blog (http://agileice.blogspot.com) and gave the new account a name.  After putting in my contact information and agreeing to the terms of service, Google gave me some tracking code that I needed to add to my blog site.

To add that code to my blog, I logged into my blog and clicked the "Customize" link at the top of the page.  From the "Layout" tag, I clicked the "Edit HTML" link which brings up the actual HTML template for the blog page.  I scrolled down to the bottom of the HTML, and inserted the code from the Analytics page right above the closing </body> tag.  That's really all there was to it.  NOTE: the code *has* to go at the bottom of your page.

Once the code was installed, I logged in to my Google Analytics account.  In the stats table for my blog, the cell at the far right was a different color and said "Tracking Unknown": Google Analytics hadn't yet communicated with the site.  I clicked the "Check Status" link and the cell changed to read "Tracking Installed".  That's pretty much all there is to it.

Now I can easily check the visitor statistics for my blog just by logging in to my Google Analytics account.  I can get reports on how many visitors I've had, where they're from, how many hits I've gotten over time, and all sorts of other data views.  There are even tools that I can install on the blog that will show the stats updated hourly (though I think I'd rather just keep it on the Analytics site).

Registering a domain using Google Apps, but hosting "www" yourself

So I registered a new domain today: agileice.com. In the past, I've registered a few domains (eriksontech.net and eriksonweb.com) with a couple other companies (one with a hosting package that expired a couple years ago and one with Yahoo!). They weren't bad experiences, but I decided to go with Google's partners this time, partly because of Google Apps. I like the idea of not having to mess with email at the domain and being able to use GMail's familiar interface for the mail that goes to that address. Initially, though, they registered the "www" alias back to the "start" pages they set up for me, which meant that the address in the address bar of the browser went from the simple "http://www.agileice.com" through what appeared to be a couple forwards to end up as "http://partnerpage.google.com/agileice.com". If I do end up doing business on this domain, I don't want potential customers going to an address like that. I also wanted to be able to control a number of other services and things behind www.agileice.com (SSH, in particular).

Here's what I ended up doing: Once you log into the domain you registered with Google Apps (by going to https://www.google.com/a/[domain name here]), you can go to the "Domain Settings" then the "Domain Names" section. Once you're there, you can get to the actual domain registrar's site (Google partners with a couple other companies to register the domain name; it doesn't register the name itself) by clicking the "Advanced DNS Settings" link. In my case, I registered the address with eNom, who has a fairly straightforward interface for managing the various parts of the DNS entries. In my case, I needed to edit the "Host Records" section where wanted to forward the "www" host to my actual IP address.

In the "Host Records" section, there are actually a few types of entries; the two types that really mattered in this case are "A" and "CNAME". An "A" entry stands for an actual IP address; a "CNAME" entry is more of an alias, forwarding the host to the "named" address in the entry. When I registered with Google Apps, eNom automatically filled in a number of entries for me, setting a few "A" entries to what I assume are Google's IP addresses, and then aliasing a number of other services (Mail, Calendar, etc) to the Google services address "ghs.google.com". All I ended up doing was changing the "www" entry (which Google had set as a "CNAME" entry) to an "A" entry pointing at the static IP address I have through my ISP.

It took a little while for the changes to propagate through the net, but now when someone goes to www.agileice.com, they'll actually get the pages I'm hosting here rather than being forwarded to the Google pages (or they will once I actually start hosting some pages here ;)). And things like Google Mail for agileice.com still work.

Thoughts from RWE 2008 final keynote

Just got finished up with a Rich Web Experience conference today, where the final *scheduled* keynote speaker--Jared Richardson--made some very good points about getting your ideas out there both for earning some name recognition and for being able to get to them yourself some time in the future when you may need them. He suggested blogging (and I wonder just how many blogs are getting started up right now :)) as a great way to start pushing your information out there.

One of the things he suggested is blogging anything that's taken you more than 10 or 15 minutes to figure out. The idea is that if it took you some time to figure out, it'll take someone else some time to figure out, and by posting it on your blog, you'll have access in to it in the future and, if people find it useful, then you'll be getting some exposure. Enough exposure, and it might lead to other things.

He also suggested getting involved in open source projects, not just in starting them up or becoming a leader or main contributor from the get go, but starting small. Pick a bug and fix it. Or, even better, do something no one else wants to do, like write documentation. It's a good way to get involved and to start getting your name out there.

The other interesting thing he mentioned is starting a personal Wiki. Basically, it becomes a repository of ideas that you have, that eventually you can go back and flesh out. When you get an idea for an article (or anything else), you can basically just create a post with a title in it. Then you can go back and fill in the content as you get the time and inclination.

A few other really good points he made:

• if a goal doesn't effect your behavior today, it's not a good goal (my interpretation: do something today to move yourself forward.)
• if you want to write a book, a book is made of chapters, a chapter is made of sections, a section is pretty much an article, and what's an article but a glorified blog post?
• get out there and teach/speak to really learn something (and get more exposure)
• write and post short tutorials as you learn to do things. Those short tutorials also become good entry points for people learning to do something and get you some more exposure.

All in all, it was a great talk, probably the best of the keynote speeches, and worth sticking around after lunch for :).