In the OSGi context, clients would no longer connect directly to service implementations, even when they are deployed in the local container.  They must always connect to service proxies, which route the service call out of the JVM, through the ESB, and back in through the service endpoint (REST or WSDL).  The service proxy endpoints could be configured via the configuration admin service, or via context properties (e.g. config.properties).

The advantages include centralization of cross-cutting concerns, but all service requests would require marshalling/unmarshalling, or in other words would sacrifice runtime efficiency for centralized configuration / customization.

1. WSDLs are not easily usable outside DOSGI
2. There is no obvious way to handle cross-cutting concerns

It may be possible to fix my WSDL gripe by modifying the CXF DOSGi implementation to use JAXB and JAX-WS annotations.  This would make the WSDLs more usable, but there was quite a bit of disagreement on the CXF mailing list about the wisdom of annotating the service interfaces.  Still, I might spend some time investigating whether this is a workable solution.

To handle  cross-cutting concerns, I’m about to take a look at Equinox Aspects, but I’m hesitant to go down that path for two reasons.  First, it seems to be designed specifically for the equinox container.  And second, it is yet another pre-1.0 project, and I’m hesitant to keep building upon such bleeding-edge technologies.

But we can’t stop there.  The software itself needs to add value, and I think flexibility is where we should focus.  We know that we can not make any assumptions about the deployment topologies of Matterhorn capture devices or application servers.  Depending on their needs, institutions will choose the number of capture devices per venue, the number of venues, the number of encoding nodes, and the number of media analysis nodes needed to handle the quantity and types of media captured and processed.  It seems that the Matterhorn partners are in agreement about the need for this type of flexibility.

What we haven’t talked much about yet is flexibility of storage.  Storage presents a fundamental budget and IT challenge for any institution intending to produce and manage many terabytes of content per semester.  Institutions should be able to utilize a SAN or a content repository if they choose, but neither of these should be a requirement.  Considering the ever expanding cloud computing options, a distributed file system such as Hadoop’s HDFS might be an attractive choice as well.

The implications of this kind of infrastructure flexibility will necessarily impact the software design.  Direct access to java.io.File will simply not be an option.  It remains to be seen whether the performance penalty of streaming multi-gigabyte files between capture clients, repositories, and application servers will be acceptable.

I particularly liked this quote: “What you want to do is create your number one goal, to say ‘what do we have to do to elevate the quality, the desirability, of the end product?’  And when you worry about costs, you hurt that.”

In our case, there are usually so many different metrics of quality (user interaction, scalability & performance, ease of installation, ease of customization, enterprise integration, etc) that answering this question alone becomes a challenging exercise.  Throw in cost, ROI, and time-to-delivery, and that challenge just grows.

RESTEasy pulled the following jars into my testing webapp (I chose the gradebook, since I’m familiar with its structure):

javassist-3.6.0.GA.jar
jaxrs-api-1.0.0.GA.jar
jsr250-api-1.0.jar
resteasy-jaxrs-1.0.0.GA.jar
scannotation-1.0.2.jar
slf4j-api-1.5.2.jar
slf4j-simple-1.5.2.jar

Is that too heavy?  Well, since two of these are part of the logging facade, two are java standards libraries, and one is the RESTEasy implementation itself, I’d say that’s not too bad.

Once I was able to resolve URLs to my annotated Java classes, I decided to try something real, like annotating the method on GradebookService that finds a student’s score for a particular assignment.  Here’s where a pretty severe RESTEasy limitation hit me: the spring integration does not work with manually proxied beans.  Or put another way, you can’t resolve the JAX-RS annotations on most services in Sakai.

This got me thinking… is annotating the service APIs really what we want to be doing?  Is there any reason to think that a Java service API necessarily models the granularity needed for a REST-ful web service?  Assuming that the granularity of local APIs will meet the needs of remote APIs has killed a lot of technologies in the past.

I think in the most common cases, RESTful APIs will be different than Java APIs.  A grading widget, for instance, would most likely need to show a paged set of scores, either one student’s scores for a set of assignments or the scores for many students on one assignment, while the API might use a more fine- or course-grained call (possibly further relying on lazily loaded collections, etc).

I think we need two very different kinds of REST-ful web services:

* Services that performs CRUD operations on entities.  These should follow the RoR URL microformat.  JAX-RS isn’t really designed to handle these kinds of boilerplate operations against entities.

* Services that provide the server-side support for “tool views”.  These services would return exactly what particular UIs demand, possibly including data from a variety of Java services, using URL schemes defined by the needs of the tool itself.   As a bonus, these services wouldn’t need any manual proxies, so they would get around the problems in RESTEasy’s spring integration (if the tool developer wanted to use spring).

I haven’t tried RESTEasy with any other IOC containers yet (I’m testing in Sakai 2.5.x, afterall), but I’m sure we can figure out a way to inject back-end / data services into these front-end REST services.

KEW

License: ECL

Dependencies: KEW is part of the larger “Rice” system, and depends on a properly configured “Kuali Service Bus”.  If KEW is embedded in another Java application, a standalone Rice server is required.

Intended use: KEW seems to be focused on the routing of xml documents for approval.  Routing decisions can be based on the content of the xml document.  KEW includes features for searching within documents and auditing documents throughout their lifecycle.

First impressions: Some of KEW’s features — its content-based routing, its user and workgroup services, its handling of notes and attachments — seem to align well with Matterhorn’s workflow needs.  Others, such as runtime editing of routing rules, don’t seem particularly useful.  Between the overhead of deploying a standalone Rice server (which seems pretty high) and KEW’s catering to a small niche market, the benefits of KEW don’t seem to be great enough to justify the costs.

jBPM

License: LGPL

Dependencies: Hibernate (+ transitive dependencies)

Intended use: Generic workflow engine for BPM and service orchestration.  Workflow definitions are defined in jPDL or BPEL, either via XML editing or graphically in eclipse.

First impressions: Seems relatively simple to embed, though the particulars of the classloading environment could make the use of hibernate problematic.  Good documentation and a quick and easy installation to jboss.  Includes a UI for instantiating new process instances and operating on existing instances.  Also includes a (beta?) GWT version of the UI.  Several major products, including Nuxeo and Alfresco, delegate their workflow needs to jBPM.

os-workflow

License: Modified Apache License

Dependencies: OSCore and PropertySet (both also from opensymphony) and JCL.  Beanshell is optional but useful for scripting workflows.  Quartz and XFire (which is now CXF and has been for some time) add job scheduling and webservice capabilities, respectively.

Intended use: Generic workflow engine.  Defining workflows involves editing xml and scripting processes.

First impressions: Documentation is either out of date or the project itself is fairly stagnant.  Workflow definitions must be either on the classpath or you need to write an extension.  User / group functionality is provided through OSCore.  Proven to work in a range of classloading environments (JEE servers, OSGI, etc).  os-workflow is bundled in ServiceMix 4.

ODE

License: ASL 2.0

Dependencies: The usual suspects… various apache libraries, xml parsers, axis, etc.

Intended use: Executes business processes written following the WS-BPEL standard.

First impressions: Although this is apparently considered a workflow engine, it seems to handle service orchestration but not routing processes between automated tasks and human users.  Doesn’t seem to meet our needs.

Intalio

License: Mozilla Public License / Community (not open source, but still free) / Commercial.

Dependencies: Unclear.  I signed up on their site (which is required to download the Intalio server), but never received the promised confirmation email containing further instructions.  After trying again with a different email address, I quit trying.

Intended use: Workflow engine and service orchestration.  Handles BPEL4PEOPLE extension.

First impressions: Intalio’s “Open Source Edition” runs on Apache Geronimo.  Process definitions can be created and edited in eclipse, though Intalio’s eclipse plugins force you to log in to the Intalio site.  Although technically open source, Intalio the business keeps injecting itself into Intalio the product.  Unless they commit to the spirit of open source, I would not recommend using their product in Matterhorn.

Update (01/16/2009): I chatted with some very helpful people from Intalio, who provided me with the skinny on their licensing, which places limits on what you can and can not do with the community edition of the software.  In particular, it states that you can “use the Intalio Software only with Apache Geronimo with a process data persistence provided by the Derby java database or by the MySQL database, and with authentication services provided by the built-in file-based authentication or Apache DS” [emphasis mine].  After first stating that “you have the freedom to do whatever you want with the community version,” I learned that in fact “you cannot use it with tomcat or oracle… However, you can deploy it on any number of CPUs, in production and development environment, for a unlimited use.  You cannot distribute the community license. However, you [are] free to let your clients download your community version, which is covered in the terms and agreement.”  So there you go.

tester.beginAt(”/portal”);
tester.setTextField(”eid”, userEid);
tester.setTextField(”pw”, pw);
tester.submit();

you can log a user in, and then

tester.gotoPage(”/portal/site/” + siteId);
tester.clickLinkWithText(”Resources”);

navigates the users through the portal and tools.  One issue I had with JWebUnit and Sakai is the opaque names and IDs that Sakai’s portal generates for the tool iFrames.  In Sakai, tester.gotoFrame(frameNameOrId) doesn’t work without a slight modification.  Adding some code to this method to find the frame by its order on the page rather than its name or id, solves this problem.  Calling tester.gotoFrame(”0″) put me into the first tool’s iframe, enabling navigation through tools themselves.

Although this kind of stress testing shouldn’t typically involve a Java developer, the code is easy enough to follow that reasonably tech-savvy QA staff should be able to put together new test scenarios.  The only “tricky” part is the multi-threading required to generate concurrent users, but that’s sufficiently separate from the “user behavior” part of the code that it shouldn’t be a problem.

1) Containers: JEE App Servers, OSGI frameworks (Felix and Equinox), and Spring DM
2) Web services frameworks: Axis2, CXF, and JAX-RS implementations (CXF and Jersey)
3) Messaging and Service Buses (Servicemix, Mule, and ActiveMQ)

From my brief experiments with these products, I’ve come up with a pretty trim proof-of-concept that might serve our needs in Matterhorn.  It works like this:

A main class boots an instance of Apache Felix, which is configured to auto-start two OSGI bundles (there are others, but only two are of particular interest): CXF and a Jackrabbit JCR server.  Other OSGI bundles can be added, removed, or refreshed at anytime via Felix’s web interface (I don’t have it up and running yet in this environment, though I have tested the management webapp that comes with Felix, and it’s pretty easy to use).

I’ve added a sample component that uses the JCR server to get or set content (currently a string… it’s just a proof-of-concept afterall ) at a particular path in the repository, and this functionality is exposed as a webservice. I was pleasantly surprised that it took only a few minutes to publish a web service.  The implementation of the component is under 100 lines of code, too.

What this setup buys us is a very lightweight way to manage components in a running system, so we should be able to hot-swap services as we make changes during development without stopping/restarting the entire system.

In addition to the cost of migrating code from existing projects (such as REPLAY) into OSGI bundles, the strict classloading rules of the OSGI environment take some time to learn, and will most likely slow developers down (not that everyone will need to work in this area of the system).  Still, with the number of projects moving toward OSGI, it seems like this is becoming the de facto Java component system, and it will behoove us all to become adept in this environment.  That, and we don’t have to get into the business of providing our own container, or trying to use one that doesn’t meet our needs for component isolation.

I’ve put together some experiments to help me understand JCR in general and Jackrabbit in particular, with the hope that we’ll be able to leverage this JCR implementation in Matterhorn.  So far, I’ve had luck reading and writing to JCR from multiple JVMs.  The writes from JVM1 show up in JVM2, so this looks like a good start.  The code is at https://opencastproject.svn.sourceforge.net/svnroot/opencastproject/jcr/