Berkeley Bits

With the deadline for a grant proposal fast approaching, I’ve been placed in charge of writing about the development techniques and technologies we’ll be using in the project.  Apparently, we don’t have to include promises to use specific products, but we should reference the options we’re considering and what standards we’re planning on using.  In order to write a somewhat well thought through proposal, I’ve been researching:

1) Containers: JEE App Servers, OSGI frameworks (Felix and Equinox), and Spring DM
2) Web services frameworks: Axis2, CXF, and JAX-RS implementations (CXF and Jersey)
3) Messaging and Service Buses (Servicemix, Mule, and ActiveMQ)

From my brief experiments with these products, I’ve come up with a pretty trim proof-of-concept that might serve our needs in Matterhorn.  It works like this:

A main class boots an instance of Apache Felix, which is configured to auto-start two OSGI bundles (there are others, but only two are of particular interest): CXF and a Jackrabbit JCR server.  Other OSGI bundles can be added, removed, or refreshed at anytime via Felix’s web interface (I don’t have it up and running yet in this environment, though I have tested the management webapp that comes with Felix, and it’s pretty easy to use).

I’ve added a sample component that uses the JCR server to get or set content (currently a string… it’s just a proof-of-concept afterall ) at a particular path in the repository, and this functionality is exposed as a webservice. I was pleasantly surprised that it took only a few minutes to publish a web service.  The implementation of the component is under 100 lines of code, too.

What this setup buys us is a very lightweight way to manage components in a running system, so we should be able to hot-swap services as we make changes during development without stopping/restarting the entire system.

In addition to the cost of migrating code from existing projects (such as REPLAY) into OSGI bundles, the strict classloading rules of the OSGI environment take some time to learn, and will most likely slow developers down (not that everyone will need to work in this area of the system).  Still, with the number of projects moving toward OSGI, it seems like this is becoming the de facto Java component system, and it will behoove us all to become adept in this environment.  That, and we don’t have to get into the business of providing our own container, or trying to use one that doesn’t meet our needs for component isolation.

I just ran across Apache CXF and at first glance, I’m impressed.  If it runs in an OSGI environment without too much trouble, it seems like it’ll offer a simple mechanism to add rest/soap/jms remoting while allowing for dynamic service redeployments during development.

I’m starting to work on my tasklist for Matterhorn (Opencast v1.0).  One of the research tasks I have is answering how a number components spread across different JVMs (or not running in a JVM at all) and different servers can efficiently access a common content repository.  There may not be an efficient mechanism for non JVM components, but there must be some kind of network access.

I’ve put together some experiments to help me understand JCR in general and Jackrabbit in particular, with the hope that we’ll be able to leverage this JCR implementation in Matterhorn.  So far, I’ve had luck reading and writing to JCR from multiple JVMs.  The writes from JVM1 show up in JVM2, so this looks like a good start.  The code is at https://opencastproject.svn.sourceforge.net/svnroot/opencastproject/jcr/

Quick progress update…

I’ve managed to get the REPLAY manager webapp to run individual jobs, return the ID of the job, and then return the status of the job.  This proof-of-concept (based on replay v0.4) just executes an indexing job on a pre-defined media bundle.  More work will obviously be needed to make this fully functional.

This is no big deal, really, but it does mean that REPLAY can fulfill the “media processing manager” component, as described in the opencast draft architecture document.

I spent a lot of time last week trying to get replay to encode, OCR, and index some video files that were captured and encoded for webcast.berkeley.edu.  It was productive to go through the source code, looking for the reason I couldn’t get anything to index.  After a couple of hours help from Tobias, we tracked down the problem: there is an undocumented dependency on the Perian toolkit on OS X.  Now that perian is installed, mp4s, dv files, and the like are properly encoded, OCR’d, and indexed.

The Opencast project’s first deliverable will be a version 0.1 integration of UCB’s Webcast system and ETH Zurich’s REPLAY (possibly also including Bjoern’s ‘channel syndication’ work).  The primary barrier to the integration is the mismatch between REPLAY’s statically defined workflows (defined as ant scripts) and Webcast’s task based finite state machine (stored in the DB and loaded into memory at startup).

Tobias and I have thrown some ideas around for overcoming this mismatch, including:

1. Colocate Webcast and REPLAY in the same JVM (or package the custom ant tasks and scripts in a jar, deployed to both systems), where the commands executed on state transitions call the ant tasks directly
2. Call REPLAY’s internal SOAP interfaces
3. Iterate through the “success” transitions in the state machine to generate a “directives” file for REPLAY to execute
4. Generate a REPLAY directives file from the front-end, avoiding the state machine for everything but status reporting
5. Colocate Webcast and REPLAY in the same JVM and call the REPLAY Java APIs from Webcast’s transition commands
6. Integrate with Webcast’s ical based task reporting.  This would involve:

• Adding discrete, atomic workflows to REPLAY
• Deploy an adapter to REPLAY’s manager component to consume Webcast’s ical tasks and execute the relevant workflow
• Update Webcast’s ical feed to include all relevant recording metadata so the adapter can re-create the bundle if necessary

I’d like to go with option #6, but I need to do some more digging in REPLAY to see if it’s possible to either reuse existing media bundles from past workflows or less desirably, whether we can create new bundles by reusing an existing media file.

I spent today at the Sakai hackathon hosted by CARET.  In addition to enjoying the sights and avoiding the pouring rain in Cambridge, I spent some time looking at the new gadgets+sdata techniques that the cambridge folks are using to build their dynamic (and flat!) portal.  It’s great to see that the ajax “revolution” has made its way into sakai.  Here are my notes on sdata, and how DWR may be an appropriate substitute for this tool in certain situations.

Sakai’s sdata tool is a webapp that can be configured to dispatch URLs to specific handlers.  The documentation, as far as I can tell, is here.  The web.xml file contains the configuration that maps URLs and query parameters (methods) to the Java handlers.

My first gadget experiment involved calling /sdata/mcp (which stands for my course and project sites, I believe) to begin implementing part of the UX improvement mockups.  The call to /sdata/mcp returns a list of sites for which I’m currently a member, including interesting tidbits like the number of members in the site.  It’s clear that this could be a useful and reusable “Javascript Service” for a number of UI gadgets.

However, once I started working on this particular UI, I realized that I needed data from a tagging service, and this isn’t provided via /sdata/mcp (or any other /sdata URLs at this time).  I could imagine getting a set of requirements that required merging gradebook data with, say forum posts, along with site memberships, for some kind of forum grading overview widget.  Who knows what those UI designers will come up with

The problem this poses is that there will be a constant push on the sdata tool to either incorporate new URL handlers as gadget designers and implementors think of new and exciting gadgets, or there will be a proliferation of ajax calls from clients trying to join data from a variety of services.

In these cases, I propose that we forego the use of sdata entirely, and utilize the DWR library instead.  DWR provides some compelling advantages over sdata, along with some added costs.  Here’s how I think DWR stacks up against sdata for sakai gadget development

Pros

1. Data can be shaped on the server side however the client wants it shaped, so a single XHR can provide everything the gadget needs to render itself.
2. Gadget developers can do all of their development in their gadget without need for modifying sdata, and dealing with the merge headaches that will inevitably come from a centralized service.
3. DWR is a popular library for exposing java classes as ajax services, so the pool of developers with DWR expertise will likely be larger than those with sdata expertise.

Cons

1. Each gadget (or collection of gadgets sharing a webapp) wishing to use DWR requires a dwr.xml configuration file.  My gadget’s dwr.xml file is 9 lines, so I don’t think this is too onerous.
2. Gadgets wishing to reuse an existing ajax service either need to copy the service into their own dwr-enabled webapp, or they need to assume that the other webapp will be deployed at runtime.  For this reason, it may make sense to centralize often-used ajax services in sdata.  It’s all a matter of tradeoffs between reuse and flexibility.
3. Performance.  DWR generates Javascript files on the fly via a servlet.  This will presumably add more load to an application server than serving up a static .js file, but my guess is that this will be minimal.  DWR 3, along with JAWR, provides a mechanism for serving cached and compressed .js files.

I’ve uploaded a zip file containing a preliminary sites gadget.  It is nowhere near feature-complete, but it demonstrates both the sdata method (via /devwidgets/sites/sites.html) and the dwr method (…/sites_dwr.html).  Please have a look and let me know (via email, IM, or via sakai-dev) what you think.

I think I’ve come up with an interesting approach to building a UI, and I’d like some honest feedback. I’m in that geeky “I just built the greatest thing in the world” state, and I’d benefit from a reality check.

Here’s how it works:

1) On the server side, you make your service calls and marshall the objects that the UI needs into JSON strings. Stick those strings into the request, and consume them in the JSP/templating language, or whatever. Nothing new here, other than the fact that you’re sending JSON strings rather than java objects.
2) In the JSP, you consume those JSON strings *once*, at the top of the section of the DOM. Here’s an example:

$(document).ready(function(){
var eventAsJson = (function () {
return eval(’${eventAsJson}’);
})();
…

Note that ${eventAsJson} is the only JSP-ism on the page, and this could be easily replaced with a real JSON string (provided by a developer). See where I’m going with this yet?

3) Build the dynamic parts of the page in JavaScript using jQuery. For example:

var eventTitle=eventAsJson.title;
$(’#title’).html(eventTitle);
…

<h3 id=”title”>The Event Title</h3>

4) What you have now is a 99% generic html template, and with a tiny bit of effort, you can do all of the UI development you like without ever touching the server side.

What do folks think? Feasible?

I’ve been looking at the available ECL license compatible XMPP server options out there, and the pickings are unfortunately slim. A quick perusal of this comparison shows that there are just two non-GPL XMPP servers that we can use: jabberd14 and openim.

Upon checking out openim, I noticed that the svn revision number was 10. There haven’t been any commits since May 7, so this project is either incredibly mature and in no need of bug fixing or new features, or the project is dead. My guess is the latter.

jabberd14, unfortunately, isn’t a java project. It’s implemented in C/C++, which means that it’ll be tricky to add to Sakai’s build and to the component manager.

Onward…

Update: openim pegs my CPU without having any work to do… lovely.