Berkeley Bits

I spent today at the Sakai hackathon hosted by CARET.  In addition to enjoying the sights and avoiding the pouring rain in Cambridge, I spent some time looking at the new gadgets+sdata techniques that the cambridge folks are using to build their dynamic (and flat!) portal.  It’s great to see that the ajax “revolution” has made its way into sakai.  Here are my notes on sdata, and how DWR may be an appropriate substitute for this tool in certain situations.

Sakai’s sdata tool is a webapp that can be configured to dispatch URLs to specific handlers.  The documentation, as far as I can tell, is here.  The web.xml file contains the configuration that maps URLs and query parameters (methods) to the Java handlers.

My first gadget experiment involved calling /sdata/mcp (which stands for my course and project sites, I believe) to begin implementing part of the UX improvement mockups.  The call to /sdata/mcp returns a list of sites for which I’m currently a member, including interesting tidbits like the number of members in the site.  It’s clear that this could be a useful and reusable “Javascript Service” for a number of UI gadgets.

However, once I started working on this particular UI, I realized that I needed data from a tagging service, and this isn’t provided via /sdata/mcp (or any other /sdata URLs at this time).  I could imagine getting a set of requirements that required merging gradebook data with, say forum posts, along with site memberships, for some kind of forum grading overview widget.  Who knows what those UI designers will come up with

The problem this poses is that there will be a constant push on the sdata tool to either incorporate new URL handlers as gadget designers and implementors think of new and exciting gadgets, or there will be a proliferation of ajax calls from clients trying to join data from a variety of services.

In these cases, I propose that we forego the use of sdata entirely, and utilize the DWR library instead.  DWR provides some compelling advantages over sdata, along with some added costs.  Here’s how I think DWR stacks up against sdata for sakai gadget development

Pros

1. Data can be shaped on the server side however the client wants it shaped, so a single XHR can provide everything the gadget needs to render itself.
2. Gadget developers can do all of their development in their gadget without need for modifying sdata, and dealing with the merge headaches that will inevitably come from a centralized service.
3. DWR is a popular library for exposing java classes as ajax services, so the pool of developers with DWR expertise will likely be larger than those with sdata expertise.

Cons

1. Each gadget (or collection of gadgets sharing a webapp) wishing to use DWR requires a dwr.xml configuration file.  My gadget’s dwr.xml file is 9 lines, so I don’t think this is too onerous.
2. Gadgets wishing to reuse an existing ajax service either need to copy the service into their own dwr-enabled webapp, or they need to assume that the other webapp will be deployed at runtime.  For this reason, it may make sense to centralize often-used ajax services in sdata.  It’s all a matter of tradeoffs between reuse and flexibility.
3. Performance.  DWR generates Javascript files on the fly via a servlet.  This will presumably add more load to an application server than serving up a static .js file, but my guess is that this will be minimal.  DWR 3, along with JAWR, provides a mechanism for serving cached and compressed .js files.

I’ve uploaded a zip file containing a preliminary sites gadget.  It is nowhere near feature-complete, but it demonstrates both the sdata method (via /devwidgets/sites/sites.html) and the dwr method (…/sites_dwr.html).  Please have a look and let me know (via email, IM, or via sakai-dev) what you think.

I think I’ve come up with an interesting approach to building a UI, and I’d like some honest feedback. I’m in that geeky “I just built the greatest thing in the world” state, and I’d benefit from a reality check.

Here’s how it works:

1) On the server side, you make your service calls and marshall the objects that the UI needs into JSON strings. Stick those strings into the request, and consume them in the JSP/templating language, or whatever. Nothing new here, other than the fact that you’re sending JSON strings rather than java objects.
2) In the JSP, you consume those JSON strings *once*, at the top of the section of the DOM. Here’s an example:

$(document).ready(function(){
var eventAsJson = (function () {
return eval(’${eventAsJson}’);
})();
…

Note that ${eventAsJson} is the only JSP-ism on the page, and this could be easily replaced with a real JSON string (provided by a developer). See where I’m going with this yet?

3) Build the dynamic parts of the page in JavaScript using jQuery. For example:

var eventTitle=eventAsJson.title;
$(’#title’).html(eventTitle);
…

<h3 id=”title”>The Event Title</h3>

4) What you have now is a 99% generic html template, and with a tiny bit of effort, you can do all of the UI development you like without ever touching the server side.

What do folks think? Feasible?

I’ve been looking at the available ECL license compatible XMPP server options out there, and the pickings are unfortunately slim. A quick perusal of this comparison shows that there are just two non-GPL XMPP servers that we can use: jabberd14 and openim.

Upon checking out openim, I noticed that the svn revision number was 10. There haven’t been any commits since May 7, so this project is either incredibly mature and in no need of bug fixing or new features, or the project is dead. My guess is the latter.

jabberd14, unfortunately, isn’t a java project. It’s implemented in C/C++, which means that it’ll be tricky to add to Sakai’s build and to the component manager.

Onward…

Update: openim pegs my CPU without having any work to do… lovely.

Sakai’s CM service is very chatty. In particular, there are two places that make many small calls to CM. The first is Sakai’s CM-based GroupProvider. The second is a data loader to populate CM with a sample university SIS.

In both the GroupProvider and the data loader, I’ve been noticing some pretty poor performance. After reading a lot of spring source code and hacking in some extra logging, I realized why. If I call the CM service from outside of its transaction boundary, I get a new hibernate session for each CM call. When each method call uses a unique hibernate sessions, we get no L1 caching at all. And without a properly configured second level cache in Sakai, this means that there’s no object caching at all, and CM — due to the shape of the api — is just going to perform horribly.

Fortunately, there’s a way around performance bottleneck. If I make the CM calls from within one of the global transaction manager’s transactions, all of the hibernate activity will share the same session and will thus take advantage of hibernate’s L1 object caching.

Well, this worked pretty well for the GroupProvider. I noticed much less DB activity after I wrapped the GroupProvider impl in a transaction proxy. But what about the data loader? Well, the data loader does its thing during startup, with an init() called by spring after the bean has been instantiated and its dependencies set. That means that the data load occurs becore the current thread has a transaction bound to it. So even though the loader is wrapped in a transaction proxy, the init() isn’t transactional.

Of course, this didn’t deter me. I just need another spring bean to do the initialization, so it’s init() calls the dataloader’s init() through the transaction proxy! Brilliant, no?

No.

I believe the reason why this didn’t work is because the transaction proxies are set up with bean post processors, and my init() was run before the post-processing. Oh well, I guess having a slow sample data load (which is only for demo purposes) is better than having slow runtime performance.

When spring’s DefaultListableBeanFactory (a superclass of a ClassPathApplicationContext, which is at the heart of Sakai’s ComponentManager implementation) fails to pre-instantiate all of it’s singleton beans, it immediately destroys all of them to ensure that they aren’t hanging around in a bad state. This makes sense, but in 1.2.8 the cleanup is implemented in a very quiet way. No logging, no stacktraces, nothing.

In order to debug why my component manager was failing to start up on my machine, I edited Spring’s DefaultListableBeanFactory so it would be a little noisier about *why* it was immediately destroying all of its singletons. If you find yourself in the uncomfortable position of debugging a component manager that is quietly failing, this will help you out.

The modified java file is available, as is the modified jar for those with better things to do than modify and build third-party jars.

For the Fall 2006 bSpace deployment, we’re running a modified Sakai 2.1.2 along with an unreleased, bleeding-edge, post 2.2.0 version of the gradebook that’s been backported to work in 2.1.2. There were a lot of small changes to make to get this version of the gradebook to work in 2.1.2, including reverting to hibernate 2, updating the gradebook’s dependencies, and changing config files and import statements to point to the new (old) package names. But in the end, it was no big deal.

However, the name of the external gradebook service changed between Sakai 2.1.2 and 2.2. What was sakai-gradebook-service became sakai-gradebook-service-api. Notice the “api” appended to the end of the name.

Remember, other projects rely on the gradebook service to add assessments and scores into the gradebook.
So the name of the API changed… big deal. Well, it turns out that when a client of the gradebook service can’t locate its dependency locally in the current source tree, we get a build failure. However, that didn’t happen. What we did get was happy developer deployments to our local machines and happy deployments (mostly) to the integration box. But once it was time to deploy to QA, we got compilation errors.

That’s right, compilation errors. Not your typical NoClassDefFound classpath errors, but “symbol not found” errors.

After doing some digging, I realized that the sakai 2.1.2 jars had been published to the sakai maven repository. So even though there were gradebook service clients that couldn’t find the “sakai-gradebook-service” project locally, those clients could find the dependency out on the web. If the client code used a method that didn’t exist in the old jar, there would be a compilation failure. And this is what we saw on happen on QA.

So, I did two things to fix this. First, I updated the gradebook service clients in bspace to use the new API name so their build dependencies are satisfied locally. Second, I changed our jar version numbers to match our official bspace version, ‘bSpace-2.1.2_1_RC1′. This way, none of the bspace code will try to compile against published jars. They will always use the local code.

The moral of the story? If you’re making modifications to services, always use a custom version number for your jars.

It turns out that the jboss-cache config file used in jforum must have a bind_addr set to the (non-loopback) IP address on the NIC. Because of our proxy / load balancing setup, we have routable IP addresses on the loopback, which confuses jboss-cache. It simply doesn’t know which IP address to bind with.

So, we now need to customize each jforum-cache-cluster.xml file with its own bind_addr setting.

It also turns out that the ‘multi’ in multicast is no joke. I was picking up multicast packets on my laptop in Dwinelle Hall that originated from the app servers across campus in the data center. According to Cisco’s documentation:

“The range of addresses from 239.0.0.0 through 239.255.255.255 contains limited scope addresses or administratively scoped addresses. These are defined by RFC 2365 to be constrained to a local group or organization. Routers are typically configured with filters to prevent multicast traffic in this address range from flowing outside an autonomous system (AS) or any user-defined domain.”

So, we need to ensure that we use these “limited scope addresses” to separate the cache traffic on the dev, qa, and prod clusters… and from the rest of the world. We don’t want a script kiddies sending jgroups messages via multicast to our apps.

We’re setting up a nightly instance so we can play with different applications and/or versions of applications within Sakai.  We’re starting off with a vanilla trunk deploy, but with the gradebook branch that Louis is working on.  This will add a file upload/import capability to the gradebook.

To create a nightly instance, all we did was:

• establish svn+ssh://svn.berkeley.edu/svn/bspace/nightly/v1/
• create our custom .externals file
• svn propset svn:externals v1 -F ./v1/.externals
• svn commit

I’m in the process of writing a utility to explode BB 5.5 content packages into a meaningful filesystem, which can be uploaded to Sakai via webdav. The problems I’m running into so far are:

1. Do we use the BB title for the Sakai filename (which is what’s displayed in the BB UI)? [Update: No… use the original filenames, since they are unique.]
2. How do we clean up the titles? Some are long, have invalid characters, etc. How much monkeying with filenames will users tolerate? [Update: Windows and Mac have their own illegal characters, as does Sakai’s webdav. We’re cleaning them all up so the full process works for every file.]
3. This is a slow process. Unzipping files and extracting them in java takes forever. [Update: Using Buffered input/output streams helps tremendously. It had been a long time since I’d worked with these APIs.]

[Update: The project is available in Sakai’s contrib]