I’ve been doing distributed computing in various forms for many years. Some at the higher level, some at the lower level and can tell you that with all the research advances and toolkits out there, it’s getting easier, but it’s still not as straight forward as it should be. Similar issues exist in multi-[process|thread] implementations. Although abstraction toolkits exist and they definitely make it easier to perform such actions without knowing much about implementing distributed algorithms, they are still leaky abstractions that in most non-trivial cases lead to having to have knowledge of the memory model, synchronization semantics, mutability, network topologies, etc… I’m not saying this is bad, I’m just saying that we haven’t yet reached the point where distributed or multi-[process|thread] computing is a cross cutting application concern. We have to actively bake it into our applications. I’m not arguing that abstractions should make developers ignorant of the underlying mechanisms, it’s just that they should be introduced at different levels of abstraction. It’s good to know what makes things tick (i.e. algorithms, data structures, etc…). Just look at the ORM world. The promise of not having to know SQL and just programming using OO took the OO world by storm. The naive thought that if you didn’t know the latest/greatest in ORM, you weren’t worthy. Years later, it turned out to be just a fad. Most have now turned back to SQL or some abstraction that’s flexible enough to allow you to work as low level or as high level as needed. In some cases, people are turning away from SQL data stores completely, but that’s another story.

A Little History

About 3 years ago, I was in the process of starting a company with my partners and we had a dilemma. We needed to process large amounts of data in near real-time. We also needed to be able to scale this horizontally in a “near” linear fashion. Data throughput was not temporally predictable, if predictable at all. After doing some searching and trying to fight the urge to implement things from scratch, we came upon the tuple-space programming model. It’s similar to the blackboard system for those that have an AI background. This is exactly what we needed, some shared distributed memory model that we didn’t have to think about (it presented itself as a single unified tuple space), and a programming API that allowed us to distribute jobs to work on the data stored in that model. Javaspaces is the java specification for Tuple Space Model. At the time, the only company that implemented this spec was GigaSpaces. We took their toolkit for a spin and it worked. The model was pleasant to program to and things were great. That’s until they didn’t work. Debugging was difficult, it leaked distributed abstraction. Deployment was also not very straightforward. None of that was the limitation of the Tuple Space Model, rather it was the implementation. I’m not saying GigaSpaces didn’t have a good implementation. I actually think it was rather nice at the time and am sure it’s way better now. At one point, we wrote an intermediary home-brewed rewrite of the system, so that we didn’t have to rush with the main implementation and can flush it out without harsh time constraints. In a few months, we ended up folding the plans to use GigaSpaces not because of the software, but rather because the company [GigaSpaces] had financial difficulties and their software, not being open source, was in flux in our opinion and we didn’t want to bet the success of our company on a commercial product of a company that looked like they were going to fold. Years later, they are still in business, great for them, but I don’t particularly regret our decision, especially looking at today’s landscape.

Most of our backend software is written in Java, Scala, and Python. Our web CRUD front end is written in PHP. The front end has a model that reflects our business domain, though it already encapsulates all of the business rules for our relational backend data store. We have a calculation process that utilizes these business rules to perform a bunch of operations and in the process reads/writes to the database. This process is very involved as it crunches hundreds of thousands of data points and will go up to millions in the next few months. It was written in PHP. We want to rewrite it using the model of distributing the data and computation using data affinity (collocating data and computations together). We’ve done it before and it works. So we’re happy to do it again, but we want to do this right and that might take some time. In the meantime, we wanted to take an intermediary step of distributing the workload amongst multiple servers (brought up and down as needed). I’ve been looking at numerous distributed toolkits for a while, from Hadoop to Akka to Gridgain. One that always stood out in the crowd has been Gridgain and in the last 6 months I’ve tried to find some place where it would pay its dividends. This project was it. We had a distributed scheduling service running on ec2 within a week, not bad, being that I had to learn the APIs and various ec2 deployment ins and outs.

Implementation

Our implementation has a scheduler that decides what computations need to be performed. We then schedule these computations by pushing a job to a distributed worker node. Because our job is run as a shell script (invoking PHP) and outputs statistics after it successfully runs, we run the job using java’s ProcessBuilder class. We then run the process, and capture its output (in json). The output is then returned to the scheduling node, evaluated, and logged. The scheduler then knows that this job can run again (we have a need to ensure the job is a singleton when it comes to running in a grid).

Our implementation is in Scala. Gridgain has a rather nice Scala DSL. We used it as much as we could, but in some cases resorted to java API for reasons I’ll explain later.

First, here is our simple task that scheduler (once it figures out a job needs to run), pushes to remote nodes…

import org.gridgain.grid.GridClosureCallMode._
import org.gridgain.scalar.scalar
import scalar._
import java.io.File
import io.Source
import net.liftweb.json._
import net.liftweb.json.JsonDSL._
import java.util.{Date, Collections}
import java.util.concurrent.{Callable, TimeUnit, Executors, ConcurrentHashMap}


class GridTask(scriptCommand:String, scriptEnv:String, scriptDir:String) extends Callable[JValue] {

  def call() = {
    val proc = new ProcessBuilder(scriptCommand.split(" "): _*)
    proc.command()
    proc.directory(new File(scriptDir))
    proc.environment().put("APPLICATION_ENV", scriptEnv);
    val p = proc.start()
    p.waitFor();
    val retString = Source.fromInputStream(p.getInputStream).getLines()
    val errString = Source.fromInputStream(p.getErrorStream).getLines()
    if (p.exitValue == 0 && !retString.isEmpty) {
      parse(retString.mkString("\n"))
    } else {
      ("event_id" -> 53) ~
        ("start_time" -> new Date().getTime) ~
        ("error" -> errString)
    }
  }
}

The above is pretty self explanatory. I kept of bunch of irrelevant things around, like inferring the return of the process and parsing/returning json.

Our scheduler is more complex, so I won’t show it, but it’s all custom business logic, nothing that has to do with scheduling a job on the grid. To scheduler the job, all you have to do is…

val retVal = grid.remoteProjection().call(
      UNICAST, 
      new GridTask(scriptCommand, scriptEnv, scriptDir))

retVal is now the JValue instance returned from the remote job. If you get rid of the custom business logic in the GridTask implementation, the whole thing is a few lines of code. Talk about an “abstraction”! Also, one mention is that don’t let simplicity in their examples fool you. Their API is full blown and gives you the level of granularity you need, jus ask, and ye shall have. For example, grid.remoteProjection() returns a projection of all remote nodes (not including the current node, which is local). This is important for us because we didn’t want the local node (scheduler) doing any computations as it’s running on a box not able to support it.

Deployment

One great thing about Gridgain, is that it works the same on a single node as it does on multiple nodes (same physical box), as it does on multiple physical nodes. You can also start multiple nodes within a single JVM. When I first heard this, I thought to myself, sounds great, but why? Nikita mentioned debugging and then a light came on. I remembered debugging GigaSpaces stuff and what worked on a single node, sometimes didn’t on multiple nodes. Mind, it was almost always my mistake, but debugging it was not very easy.

Our infrastructure runs on EC2. Gridgain provides EC2 images, but besides the fact that they run CentOS I believe, which I’ve grown to dislike, I’m also a control freak when it comes to my servers. I want them clean and mean:-). I prefer debian/ubuntu boxes, though I opted to create my own AMI. Installing Gridgain was easy, configuring is also a 2 minute task. It took me a few hours to figure it out and with the help of the forum, the configuration was a few lines of XML. We’re using the community edition, which comes with rudimentary IP discovery SPI. They have much more robust discovery SPIs available in their enterprise edition. One which I think makes the most sense on EC2 is S3 discovery. Basically, it uses S3 to write node information, and all nodes communicate using a S3 bucket. Makes sense. We weren’t ready to dish any money out for enterprise version yet, so I had to settle for IP discovery. In our case, it wasn’t hard. Basically, the scheduler in a single node that runs behind an elastic IP address that never changes. That means that the other boxes only have to know the IP address of the scheduler to make the initial communication. Once it can connect to one node, it joins the full grid. Because we have a single scheduler, if the scheduler goes down, the workers are no longer a part of the grid until the scheduler comes back up. This is OK for us, since due to some domain details, we can only have a single scheduler at this time and we’re OK with that single point of failure, especially being that we can bring it back up in no time and the worker nodes patiently retry the join operation at an interval and then rejoin the grid once the scheduler is back up. This is out topology. Gridgain supports pretty much what ever you want, including P2P no single point of failure topology. Below are the relevant configurations for our stuff…

Scheduler

<property name="discoverySpi">
    <bean class="org.gridgain.grid.spi.discovery.tcp.GridTcpDiscoverySpi">
        <property name="localAddress" value="10.1.1.1"/>
        <property name="ipFinder">
            <bean class="org.gridgain.grid.spi.discovery.tcp.ipfinder.vm.GridTcpDiscoveryVmIpFinder">
                <property name="shared" value="true"/>
                <property name="addresses">
                    <list>
                        <value>10.1.1.1:47500</value>
                    </list>
                </property>
            </bean>
        </property>
        <property name="heartbeatFrequency" value="2000"/>
    </bean>
</property>

The 10.1.1.1 is the local IP address of the scheduler box. The ip is repeated in the “addresses” section, telling gridgain that this can be the sole server and it doesn’t have to join a grid topology before it goes live. Also, shared=”true” is important, as it tells gridgain to share configurations amongst the boxes in the grid. Without it, you’ll have an “order of operations” issue, where a master must be started first before the worker. With it, that issue is moot and you can start/stop things as you please. I wish they would make this the default.

Right now, Gridgain cannot bind to a wildcard, though you have to specify the private IP address. If it changes (box reboots), you have to change it too. They promised a solution in their next release, which will allow to listen on private IP and communicate over public IP. This will help in other NAT topologies. Being able to listen to a wildcard will also help in having a config you never have to change. But even with this caveat, this is quite a breeze.

The worker config is similar, except it only needs to know about the scheduler and does not need to operate until it has joined a topology…

<property name="discoverySpi">
    <bean class="org.gridgain.grid.spi.discovery.tcp.GridTcpDiscoverySpi">
        <property name="localAddress" value="10.1.1.2"/>
        <property name="ipFinder">
            <bean class="org.gridgain.grid.spi.discovery.tcp.ipfinder.vm.GridTcpDiscoveryVmIpFinder">
                <property name="shared" value="true"/>
                <property name="addresses">
                    <list>
                        <value>scheduler.node.com:47500</value>
                    </list>
                </property>
            </bean>
        </property>
        <property name="heartbeatFrequency" value="2000"/>
    </bean>
</property>

Caveats

There are small caveats I found, none of which created much of a hurdle.

First is serialization. In my case, I’m was using logback for logging, gridgain uses log4j. We both use slf4j, which takes the first in the classpath. If you’re going to distributed a job that references something that clashes with classpaths, you have to do some classpath mangling. Removing log4j from gridgain’s lib directory would fix the issue, but I didn’t want to customize the install. I was originally using a Scala closure as a job unit, which had no references to the log object. In theory, if that’s the unit that gets serialized and sent over the wire, the other end should not have to worry about any logback references, since they aren’t a part of the serialized closure references. In my case, that didn’t work. Somehow the serialization decided to serialize logback related stuff, because the top level class where the closure was being created used the logger. I’m not sure if this is a problem with serialization or a leaky abstraction of the JVM and the fact that functions aren’t first class citizens. I think the lowest level of serialization is at the class level, though I had to extract it to a class and implement a GridTask instead. Because GridTask class extension didn’t reference any logger object, it was serialized as needed and sent over without causing a classpath conflict. I haven’t had the time yet to figure out whether it’s the fault of Gridgain’s optimized serializer or whether this is a side effect of the JVM (as I mentioned above). I’ll try to find some time to test this later.

Second, Gridgain Community Edition has discovery that works great in a homogeneous topology, but for EC2 (NAT, ephemeral private IP addresses, etc…), configuration is transient in terms of if any of the things I listed changes, the config must change. This can be remedied by startup scripts, but Nikita said they’ll add better support for it in the next version (public IP communication would be a good first step, binding to wildcard interfaces would be a great second).

Conclusion

Overall we had an awesome experience with Gridgain. The grid application ran flawlessly during our busiest time of the year (Thanksgiving weekend). It ran so flawlessly, that this morning, I forgot which boxes it was physically running on.

I plan on using Gridgain in the future and hopefully utilize their data grid to rewrite our computation system to utilize in memory data/compute collocation (data affinity).

Nikita, thanks for all the help getting things sorted out in the first few days.

If you are running a java app on ec2 and want to remotely connect to it using jconsole or jvisualvm, you need to start your java app with a few options. Here is my configuration. Also, note that disabling authentication, opens this up for everyone. Not good, so don’t do this in production or on a box that matters. Also, this doesn’t work with a restrictive firewall. Since RMI port is chosen randomly, you must have a rather loose firewall policy. There are ways around it, with ssh tunneling I believe, but this post won’t cover it as this point, I might do it at some point later.

First you need a policy file. Again this can be fine tuned, the example below shows a dangerously loose one…

grant {
  permission java.security.AllPermission;
};

Place this file in some directory. In my example it’s sitting in my home dir and is named .java.policy

java -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9001 \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Djava.security.policy=.java.policy \
  -Dcom.sun.management.jmxremote.local.only=false \
  -Djava.rmi.server.hostname=your.public.hostname.com \
  -jar test.jar Runner

This starts the app and an jndi service listening on port 9001.

In jvisualvm, you now can connect to your.public.hostname.com:9001. You can tune your parameters as needed, but two are crucial in my experience: com.sun.management.jmxremote.local.only and java.rmi.server.hostname. I had to specify these in order to make things work. Your mileage may vary.

I don’t have access to the exact code at this time, but below is a simple example that demonstrated similar issue in Java.

import java.util.*;

public class ChainDemo {
  
  static interface Command {
    void execute(Map<String, Object> ctx);
  }

  static class Chain {
    private List<Command> commands = new ArrayList<Command>();
    
    public void addCommand(Command cmd) {
      commands.add(cmd);
    }                   
    
    public void execute() {
      Map<String, Object> ctx = new HashMap<String, Object>();
      
      for (Command cmd : commands) {
        cmd.execute(ctx);
      }
    }
    
  }  
  
  public static void main(String[] args) {
    Chain chain = new Chain();
    chain.addCommand(new Command() {
      public void execute(Map<String, Object> ctx) {
        // Perform some calc here
        ctx.put("result1", 1234);
      }
    });
    chain.addCommand(new Command() {
      public void execute(Map<String, Object> ctx) {
        int res = (Integer) ctx.get("result1");
        ctx.put("result2", res * 2);
      }
    });
    chain.addCommand(new Command() {
      public void execute(Map<String, Object> ctx) {
        System.err.println("Results: ");
        for (String key : ctx.keySet()) {
          System.err.println("\t" + key + ": " + ctx.get(key));
        }
      }
    });
    chain.execute();
  }

}             

Running the above yields:

Results: 
    result1: 1234
    result2: 2468

Besides the mandatory java ceremony, it’s also not apparent to me that this can be accomplished any better without the use of reflection, which of course would add yet more boilerplate.

Macros to the rescue. If any of you aren’t familiar with what makes lisp (besides its simple syntax allure) so powerful, you should familiarize yourself with macros. The example I give below doesn’t even make a dent into the possibilities of macros.

With a simple macro

(defmacro make-cmd [args & body]
  `(fn [{:keys [~@args] :as ctx#}]
     (doseq [ar# [~@(map keyword args)]]
       (if (not (contains? ctx# ar#))
         (throw (Exception. (str "'" (name ar#) "' argument is required!")))))
     ~@body
     ))

(defn- exec-cmd [cmd context]
  (merge context (cmd context)))

(defn exec-chain [& cmds]
    (loop [chain cmds
           ctx {}]
      (if (first chain)
        (recur (rest chain) (exec-cmd (first chain) ctx)))))

The above can now be utilized with the following api…

(exec-chain 
  (make-cmd [] (println "Executing first") {:first "first arg"})
  (make-cmd [first] (println "Executing second:" first) {:second "second arg" :random "random arg"})
  (make-cmd [second random] (println "Executing third:" second random) {:third "third arg"}))

Yielding (in SLIME REPL):

Executing first
Executing second: first arg
Executing third: second arg random-arg
nil

Running a similar script with an added parameter in the third command that doesn’t exist in the chain

(exec-chain 
    (make-cmd [] (println "Executing first") {:first "first arg"})
    (make-cmd [first] (println "Executing second:" first) {:second "second arg" :random "random arg"})
    (make-cmd [second random does-not-exist] (println "Executing third:" second random) {:third "third arg"}))

Throws an exception (in SLIME REPL):

'does-not-exist' argument is required!
  [Thrown class java.lang.Exception]

Of course the above can be much improved, one off the top of my head improvement is to allow default values for arguments that don’t exist and possibly be able to specify arguments who’s lack of should throw and exception an in other cases either bind a nil or default value. But the concise example demonstrates similar abilities of my java program but with added runtime argument name matching. I’d love to see if the same can be accomplished with a statically typed language with type compile time vs. runtime argument checking. Going to investigate this with Haskell and Scala this week.

It might be that Apple wants Oracle to take the charge in maintaining the OS X port and deprecating support might be a way to negotiate this without face-to-face negotiations. I’m fine with that, frankly I could care less who provides the JVM, as long as one is provided and is relatively actively supported. Until that announcement happens, this is yet another bump in the future of the JVM. First the Oracle purchase of Sun, then the lawsuit, and now the decision by Apple, definitely creates unneeded distractions for this platform’s developers all over the globe.

So I started this not to gripe any more about Apple’s decision, I’m sure there are enough posts out there flooding your RSS streams to keep you busy, rather I wanted to question the future of languages/industry in regards to the language/platform of the future.

Last week I attended the StrangeLoop conference in St. Louis. It was a gem of a conference, definitely the best I’ve been too in a long time. Alex, besides seeming like an overall awesome guy, has some extraordinary “brilliant people herding” abilities. How he managed to bring together a group of brilliant speakers and then convince another group of awesome developers to attend, is beyond me. The conference had some great talks and panels about the latest/greatest and bleeding edge tech stuff. One of the best panels was about the future of programming languages. The panel consisted of (Guy Steele, Alex Payne, Josh Bloch, Bruce Tate, and Douglas Crockford), all whom I have great respect for. One prevailing factor in most discussions in this panel as well as throughout the conference, has been concurrency. In the mutli-core/cpu world, what language/platform will allow for this paradigm transition to happen seamlessly. The fact is that although there are some awesome innovations/implementations going on in this area, STM, Actors, fork/join, and various others, none have yet abstracted the concurrency model away from the developer, as seamless as memory management and garbage collection is done in today’s runtime environments. But this is an exciting time to be in, many ideas are flowing around and something will appear on the horizon sooner or later. This something, as Guy Steele pointed out, will most likely be a model that will allow for divide/conquer, map/reduce operations to happen through language idioms and possible seamless abstractions. Accumulators are evil

There are many languages/platforms out there today, but none have been as predominant and as overall polished as Java and the JVM. From the language perspective Java’s getting stagnant and to some, boring, but the fact that it has an ecosystem of wonderful libraries and products is hard to ignore. The fact that all of these are bytecode compatible is even more to rant about, as with the advent of numerous great languages built on top of the JVM, it makes the transition to a different language and programming paradigm, much easier. It is truly hard to think of any current platform/VM that’s more prevalent and better suited for large scale enterprise development than the JVM. .NET comes to mind, but I doubt anyone from the non-Microsoft camp will be switching . There are other platforms, most notably Python and Ruby, but although both are credible, the presence of GIL on both, make the choice of using them in a concurrency model very difficult. You can architect and deploy your system as multi-process vs. multi-thread and arguably that model has its benefits, mostly by getting rid of the shared state model concurrency issues, but we (at least I do) like to have a choice. This decision shouldn’t be shoved down our throats because the language development camp doesn’t want, doesn’t think one is necessary, or [add your own excuse here] to produce a thread-safe non-GIL thread model.

The other issue with most of these languages/platforms, as well as the other ones I like, is the deployment options. They suck! From providing modular builds to deploying production applications, they just aren’t as polished and in most cases as stable/supported as the JVM ones. Common Lisp, one of my favorites as of late, for example, is an awesome language with numerous compilers/interpreters. Lisp doesn’t have a good packaging, dependency resolution, and build story, but even if you can get past that with some of the available half-baked solutions, then when it’s time to build/compile/deploy your app, you’re fucked, unless you want to build one yourself. I enjoy such challenges on Friday/Saturday nights, but not when time is limited and milestones are due (which is most of the time).

Ruby and Python for example, have a decent package managers gem and easy_install/pip respectively, but two problems lurk. First, lots of modules are written in C and in many cases, in my experience, are a big pain in the ass to compile, especially with today’s heterogeneous architectures i386, x86_64, etc… Lots of incompatibilities arise, forcing more time away from doing what I should be doing. Somehow my milestones never include the 2+ day derailments due to such issues. Maybe that’s what’s left of my optimism. The second problem only applies if you’re writing a web app and if you are, then you know the issues. Where are those stable/supported app servers? WSGI and rack should provide answers soon, for now, there are many options and none are without major issues as well. Some are a pain to install/deploy, some aren’t actively maintained. I mean, am I just being anal and asking for way too much or am I eternally spoiled by the JVM. Is it too much to ask to bundle the application into some archive or directory structure and just drop it in somewhere or point your server config towards it. Either way, even if they ease the pain of deploying webapps, the fact that [Python/Ruby] are not suitable in multi-core/cpu environments where threads are needed, is a show stopper for lots of apps I write. I know I can architect around the issues, but again, why should I have to program to the platform vs. the other way around. Give me the choices and trust me to make the best decision.

The next things is native GUI development. It is true that lots of interesting apps today are developed and deployed as web apps, but that doesn’t discount the fact that there is still a need for a native GUI in lots of use cases. Swing provides a good and in some instances really good, cross platform GUI library which allows to deploy your GUI across most popular platforms with 95% or more cross platform consistency. That sounds pretty good to me.

There are other toolkits, wxWidgets, QT, etc…, which also have bindings to python and ruby, but again, with today’s multicore, it would be a shame to not be able to utilize these cores simultaneously due to GIL. The bindings in languages that due provide a better concurrency story, work great, but these languages still suffer from the other pain points I mentioned before (i.e. deployment, build, package management, etc…). It’s a Catch-22.

So maybe I’m missing something here, but I think the JVM is the best option we have at this time that allows for multiple platforms, languages, paradigms, and comes with a great success story in the enterprise (build tools, deployment/modularity, enterprise grade servers, etc…). Languages implemented on top of the JVM benefit from this quite successful ecosystem. Ah, and might I mention that great libraries exist for about anything you’re trying to do. This is also true of Python, but I can’t say the same for Ruby. Ruby has numerous gems for most tasks, but they all seem half-baked at most. There are frameworks like Rails and Sinatra, which are great and fully supported with active communities, though as long as you don’t venture too far off the traditional path.

JVM has it’s own set of issue, the fact that it was written with static languages in mind and lacks support for dynamic bindings, tail call optimizations, and other things that make writing languages on top of it more difficult. It’s future is now also in question due to the new Oracle stewardship and the legal obstacles it chose to pursue rather than spend that time and money on the platform. Nevertheless, the ecosystem is still flourishing, kept afloat but tons of great developers and supporting companies who care about the platform and greatly benefit from it. JVM allows us to program in different languages while being concentrated on the task at hand, not peripheral issues like compiling for different architectures, battling the deployment inadequacies, not being able to utilize cores efficiently, and a variety of other issues. JVM ecosystem might not have the most ideal solutions to these problems, but they are far better than anything out there right now. If people that spend their time bashing the JVM platform would spend as much time making their platform better, maybe we’d have other choices.

I’d love to hear other’s thoughts on this topic. What do you think about the JVM and what’s your language/platform of choice. How do you build, deploy, distribute your applications? What concurrency options are available on that platform and how they compare to others? I’m familiar with most JVM options, especially Clojure and Scala, so I’m mostly asking for anything outside of the JVM ecosystem. I hope to someday compile a list of these and present them in an objective manner, for now, all I have is my empirical opinions.

The application bootstrapping options are also super flexible, allowing one to configure how routes are resolved and nest routes for cascading capabilities. I ran into a small caveat when I tried to configure extension based content negotiation. Basically, the idea of extension based content negotiation, is that instead of using “Accept” headers, one can append a mime extension to their request URI to request a particular return format. Say, we have a http://localhost/api/resource uri, one can request xml or json formats by simply doing http://localhost/api/resource.xml or http://localhost/api/resource.json. Of course your resource has to support these formats. The documentation on this type of content negotiation is non-existent. I had to scour a bunch of users group messages and javadocs before I figured it out. I figured I’ll shared if someone else is interested.

My applications is written in Scala, so examples will be provided as such. I’m sure any experienced developer can easily discern the java equivalent.

First, in your application bootstrapping, you must turn on the extensionsTunnel option. Here is my code, which also demonstrates nested routes. Then, in your resource you must conditionally infer the MediaType provided and emit the representation of this resource based on it.

import org.restlet.{Restlet, Application => RestletApplication}
import scala.xml._
//... other imports excluded

class TestApplication extends RestletApplication {
  override def createInboundRoot: Restlet = {

    val apiRouter = new Router(getContext)
    apiRouter.attach("/test", classOf[TestResource])

    val rootRouter = new Router(getContext)
    rootRouter.attach("/api/v1", apiRouter).getTemplate.setMatchingMode(Template.MODE_STARTS_WITH)

    getTunnelService.setExtensionsTunnel(true)

    return rootRouter
  }
}

class TestResource extends ServerResource {

  @Get("xml|json")
  def represent(v:Variant):String = {
    return v.getMediaType match {
          case MediaType.TEXT_XML | MediaType.APPLICATION_XML => <response><message>Hello from Restlet!</message></response>.toString
          case MediaType.APPLICATION_JSON => "{\"message\": \"Hello from Restlet\"}"
        }
  }
}

First, the root router’s matching mode must be set to Template.MODE_STARTS_WITH, otherwise it will try to match based on full absolute uri path and not find any nested resources. So the matching mode is very important in the case where you’re working with nested resources.

Second, you set the extensions tunnel property to true: getTunnelService.setExtensionsTunnel(true). This will turn on the extension tunneling service and perform content negotiation based on the URI’s extension. Note: if an extension is not provided, it will resort to first available representation supported by the resource. It can get more complicated I believe based on other configurations, but this is what happens in the most simple scenario.

Now, with content negotiation on, the resource has to conditionally infer the proper MediaType requested and provide its representation for the MediaType. In Scala this is very elegantly done using the super flexible match/case construct. This construct can be used as Java’s switch statement, but it is way more powerful and allows for advanced pattern matching. As you can see, I check for both xml and json media types and provide the proper representation. The supported media types are handled through @Get annotation. For more info, see Restlet’s annotations and Resource documentation.

Now, accessing the resources yields the following results:

  $ curl http://localhost:8080/api/v1/test.xml
  Hello from Restlet

  $ curl http://localhost:8080/api/v1/test.json
  {"message": "Hello from Restlet"}

The bottom line is: DO NOT USE NULLs unless you are utilizing java libraries and have no choice. Use Option instead, with Some() or None().

The problem is best described with code…

  def checkNullOrEmpty(v:Seq[Any]):Boolean = {
    println("Class:"+v.getClass)
    return (v != null) && !v.isEmpty
  }

  case class Race(val event:String, val protocol:String) {
    println("Event:"+event+", protocol:"+protocol)
    assert(checkNullOrEmpty(event))
    assert(checkNullOrEmpty(protocol))
  }

  val t = new Race(null, null)

Pasting the above into REPL yields the following result…

Event:null, protocol:null
Class:class scala.collection.immutable.WrappedString
java.lang.NullPointerException
    at scala.Proxy$class.toString(Proxy.scala:29)
    at scala.collection.immutable.WrappedString.toString(WrappedString.scala:22)
    at scala.collection.immutable.StringLike$class.length(StringLike.scala:48)
    at scala.collection.immutable.WrappedString.length(WrappedString.scala:22)
    at scala.collection.IndexedSeqLike$class.isEmpty(IndexedSeqLike.scala:81)
    at scala.collection.immutable.WrappedString.isEmpty(WrappedString.scala:22)
    at .checkNullOrEmpty(<console>:6)
    .....

So why is NPE thrown at this breakpoint return (v != null) && !v.isEmpty?

So let’s look further into the output. When a Race instance is created, the constructor values are initialized to null. Inside the constructor we print this out and verify that values are in fact null. When We get to checkNullOrEmpty method call, the class of v is WrappedString and though the object is no longer null. In Java the call to getClass would fail, as v would be null, in Scala it’s casted (converted) to WrappedString.

This happens through Scala’s implicit conversions. The checkNullOrempty method expects a Seq. Although Seq is not a superclass or interface of String. So when we create the Race instance and specify both event and protocol as null, they are still String types with a null reference. Using say event or protocol as an argument to checkNullOrEmpty yields an implicit conversion. Why? Well, in Java the compilation would fail, since the Seq trait is not a part of the inheritance hierarchy of String, but in Scala, it succeeds, as Scala finds an implicit conversion method to convert String to WrappedString. This method is defined in Predef object. We know the Predef is imported by default into all Scala classes. Predef extends LowPriorityImplicits class, which defines this implicit conversion implicit def wrapString(s: String): WrappedString = new WrappedString(s). So basically Scala decides that the best way to convert the String type into Seq[Any] is by using this implicit conversion. So it wraps the null value with the WrappedString.

This causes two issues… First, the not null check no longer works, as the object is not null, due to the fact that it’s an instance of WrappedString, so (v != null) is true. Since that passes, it then executes the RHS of && operator and then tries to infer on !v.isEmpty, which throws the NPE, as the underlying String value wrapped is null.

I’m not necessarily sure whether this is a bug, feature, or maybe there is no real consensus on how null should be handled, but as you see this causes issues and should either be avoided through avoiding null and using Option instead. If you are using java libs that return nulls, wrapping the return value in Option might be a good idea, before proceeding any further.

I’ve also looked at other concurrent programming models, like STM/MVCC. STM is the basis of concurrent programming in Clojure and it’s a different paradigm than Actors. STM is a simpler model if you’re used to programming the old imperative threading, as they abstract you from concurrency primitives by forcing state modifications to occur in a transactional context. When this occurs, the STM system takes care of ensuring the state modifications occur atomically and in isolation. In my opinion this system suites the multi-core paradigm very well and allows smoother transition, the problem with it, at least in the context of MVCC, is that for each transaction and data structure being modified, a copy is made for the purposes of isolation (implementation of copying is system dependent, some might be more efficient than others), but you can already see an issue. For a system that has to handle numerous concurrent transactions involving many objects, this can become a bottleneck and the creation of copies can overburden system’s memory and performance. There are some debates about that in the STM world, mostly involving finding the sweet spot for such systems, where the cost of MVCC is less relevant the the cost of constant synchronization through locking.

Actors model is different, it works in terms of isolated objects (actors), all working in isolation by message passing. None can modify or query the state of another, short of requesting such an operation by sending a message to that particular object (actor). In Scala, you can break that model, as you can send around mutable objects, but if you are to really benefit from the Actor model, one should probably avoid doing that. Actors lend themselves better to concurrent applications, that not only span multiple-cores, but also can easily be scaled to multiple physical nodes. Because messages being passed are immutable data structures that can be easily synchronized and shared, the underlying Actor system can share these message across physically dispersed actors just as it can for the actors within the same physical memory space.

So the world of concurrency is getting more exciting with these awesome paradigms. One thing to remember is that there is no one size fits all concurrency model and I don’t see any one of the above becoming the de-facto standard any time soon. There is a sweet spot for each, so one should learn the ins and outs of each model.

Now that I got the concurrency out of the way, let’s get back to the actual syntax of Scala. Scala is very powerful (at least compared to Java). This power comes with responsibility. You can use Scala to write beautiful/concise programs, or you can use it to write obscure/illegible programs that no one, including the original author, will be able to comprehend. Personally, I prefer and can responsible handle this responsibility. I’m a long time Perl programmer (way before I started programming Java), and I’ve seen (and even written at times), programs that Larry Wall himself wouldn’t be able to comprehend.

Scala comes with operator overloading, but when not judiciously used, that power alone can be responsible for ineligibility of any system. This is one of the major reasons why languages like Java decided to not include it. Personally, I think operator overloading can be a beautiful addition to any API. It can make writing DSLs easier and using them more natural. Again, this power is great in the use of experienced and responsible programmers.

After having experience great power (Perl) and great restraint (Java), I’m leaning more towards power (who wouldn’t . One one hand, it’s nice to be able to read and comprehend anyone’s Java program, even when it’s not nicely written, on the other hand, it’s a pain trying to write a program and jumping through all the hoops and limitations because of the various constraints. In a perfect AI world, the compiler would infer the capabilities of the programmer and restrict its facilities based on those, in some way as to not offend anyone:-) So if a novice is inferred, ah, there goes the operator overloading and implicit conversions, etc… But for now, I’d rather have a powerful tool to use when I write software and Scala seems to push the right buttons for me at this point.

I’m going to start of a list of posts, starting with this one, about my experiences with Scala.

Here is a little something I came up with a few hours ago. Our software has some limited interoperability with a SQL database and requires a light abstraction. We chose not to use any 3rd party ORM or SQL abstraction, mostly due to the fact that the dependency on these abstractions don’t really provide any benefit for our limited use of SQL. So I developed a simple SQL variant abstraction layer, which allows us to execute SQL queries which are defined in the SQLVariant implementation. Moving from one database to another, just requires one to implement a SQLVariant interface to provide the proper abstraction. I initially wrote this in java and although it was decent, it required quite a bit more code and didn’t look as concise as I wanted it. One issue was PreparedStatement and it’s interface for placeholder bindings. How would one bind java’s primitive and wrapper types as placeholders and how would the SQLVariant know which PreparedStatement.bind* method to call? I resorted to using an enumeration which defines these operations and reflection for the purpose of invoking these operations. I’m basically sidestepping static typing in a place I’m not sure I really want or have to. Here is the java implementation.

I got rid of a few methods, specifically dealing with resultset, statement, and connection cleanup, as they don’t really emphasize my point here.

  import java.lang.reflect.Method;
  import java.sql.*;
  import java.util.ArrayList;
  import java.util.Collections;
  import java.util.List;

  public abstract class SqlVariant {

    public abstract SqlSelectStatement getResultsNotYetNotifiedForStatement(NotificationType... types);

    public abstract SqlSelectStatement getResultsNotYetNotifiedForStatement(int limit, NotificationType... types);

    public abstract SqlUpdateStatement getUpdateWithNotificationsForStatement(Result result);

    private abstract static class SqlStatement<T> {

      protected String sql;
      protected List<BindParameter> bindParams = new ArrayList<BindParameter>();
      protected PreparedStatement stmt;

      public SqlStatement(String sql) {
        this.sql = sql;
      }

      public SqlStatement addBindParam(BindParameter param) {
        bindParams.add(param);
        return this;
      }

      public String getSql() {
        return sql;
      }

      public List<BindParameter> getBindParams() {
        return Collections.unmodifiableList(bindParams);
      }

      protected PreparedStatement prepareStatement(Connection conn) throws SQLException {
        PreparedStatement stmt = conn.prepareStatement(sql);
        for (int bindIdx = 0; bindIdx < bindParams.size(); bindIdx++) {
          BindParameter p = bindParams.get(bindIdx);
          try {
            Method m = stmt.getClass().getMethod(p.type.method, Integer.TYPE, p.type.clazz);
            m.invoke(stmt, bindIdx + 1, p.value);
          }
          catch (Exception e) {
            throw new RuntimeException("Couldn't execute method: " + p.type.method + " on " + stmt.getClass(), e);
          }
        }
        return stmt;
      }

      public abstract T execute(Connection conn) throws SQLException;
    }

    public static final class SqlSelectStatement extends SqlStatement<ResultSet> {

      public SqlSelectStatement(String sql) {
        super(sql);
      }

      @Override
      public ResultSet execute(Connection conn) throws SQLException {
        return prepareStatement(conn).executeQuery();
      }
    }

    public static final class SqlUpdateStatement extends SqlStatement<Boolean> {
      public SqlUpdateStatement(String sql) {
        super(sql);
      }

      @Override
      public Boolean execute(Connection conn) throws SQLException {
        stmt = prepareStatement(conn);
        return stmt.execute();
      }
    }


    public static final class BindParameter<T> {
      private final BindParameterType type;
      private final T value;

      public BindParameter(Class<T> type, T value) {
        this.type = BindParameterType.getTypeFor(type);
        this.value = value;
      }

      public BindParameter(BindParameterType type, T value) {
        this.type = type;
        this.value = value;
      }
    }

    private static enum BindParameterType {
      STRING(String.class, "setString"),
      INT(Integer.TYPE, "setInt"),
      LONG(Long.TYPE, "setLong");

      private Class clazz;
      private String method;

      private BindParameterType(Class clazz, String method) {
        this.clazz = clazz;
        this.method = method;
      }

      private static BindParameterType getTypeFor(Class clazz) {
        for (BindParameterType t : BindParameterType.values()) {
          if (t.clazz.equals(clazz)) {
            return t;
          }
        }
        throw new IllegalArgumentException("Type: " + clazz.getClass() + " is not defined as a BindParameterType enum.");
      }
    }
  }

Now, here is how one would implement the SQLVariant interface. The below implementation is in groovy. I choose groovy when I have to do lots of string interpolation, which somehow java and scala refuse to support. The code was shortened to just demonstrate the bare minimum.

  class MySqlVariant extends SqlVariant {

    @Override
    public SqlVariant.SqlSelectStatement getResultsNotYetNotifiedForStatement(int limit, NotificationType[] types) {
      SqlVariant.SqlSelectStatement stmt = new SqlVariant.SqlSelectStatement("SELECT ...")
      for (NotificationType t : types)
        stmt.addBindParam(new SqlVariant.BindParameter(String.class, t.name().toUpperCase()))
      return stmt;
    }

    @Override
    public SqlVariant.SqlUpdateStatement getUpdateWithNotificationsForStatement(Result result) {
      SqlVariant.SqlUpdateStatement stmt = new SqlVariant.SqlUpdateStatement("INSERT INTO ....")
      result.notifications?.each { Notification n ->
        stmt.addBindParam(new SqlVariant.BindParameter(SqlVariant.BindParameterType.LONG, n.id))
        stmt.addBindParam(new SqlVariant.BindParameter(SqlVariant.BindParameterType.LONG, result.intervalId))
      }
      return stmt
    }

    ......
  }

I started reimplementing the above in Scala and I ran across a very powerful and beautiful Scala implicit conversion feature. This allowed me to truly abstract the SQLVariant implementations from any bindings specific knowledge, through an implicit casting facility that normally only dynamically typed languages provide. Scala gives us this ability, but also ensures static type safety of implicit conversions during compilation.

Another wonderful feature, is lazy vals, which allows us to cleanly implement lazy evaluation that we (java programmers) are so used to doing by instantiating a member field as null and then checking it before initializing on the initial accessor call. If you’ve seen code similar to below a lot, you’ll rejoice to find out that you no longer have to do this in Scala.

public class SomeClass {
  private SomeType type;

  public SomeType getSomeType() {
    if (type == null) type = new SomeType(); // Often more complex than that
    return type;
  }
}

The above, besides not being ideal, is also error prone if say a type is used anywhere else in SomeClass and you don’t use the accessor method to retrieve it. You must ensure the use of accessor through convention or deal with the fact that it could be non-instantiated. This is no longer the case in Scala as its runtime handles lazy instantiation for you. See below code.

Note: I still allow the client data access abstractions to work with a raw jdbc ResultSet returned from the SQLVariant. I don’t see this as an issue at this point, first since these abstractions are SQL specific and also because ResultSet is a standard interface for any JDBC SQL interaction. Here is my concise Scala implementation. I’m still learning, so this might change as I get more familiar with Scala idioms and start writing more idiomatic Scala code.

  import javax.sql.DataSource
  import java.sql.{ResultSet, Connection, PreparedStatement}
  import com.bazusports.chipreader.sql.SqlVariant.{SqlSelectStatement, BindingValue}

  abstract class SqlVariant(private val ds: DataSource) {

    def retrieveConfigurationStatementFor(eventTag: String): SqlSelectStatement;

    protected final def connection: Connection = ds.getConnection
  }

  object SqlVariant {

    trait BindingValue {def >>(stmt: PreparedStatement, idx: Int): Unit}

    // This is how implicit bindings happen.  This is beauty, we can now
    // bind standard types and have the compiler perform implicit conversions
    implicit final def bindingIntWrapper(v: Int) = new BindingValue {
      def >>(stmt: PreparedStatement, idx: Int) = {stmt.setInt(idx, v)}
    }

    implicit final def bindingLongWrapper(v: Long) = new BindingValue {
      def >>(stmt: PreparedStatement, idx: Int) {stmt.setLong(idx, v)}
    }

    implicit final def bindingStringWrapper(v: String) = new BindingValue {
      def >>(stmt: PreparedStatement, idx: Int) {stmt.setString(idx, v)}
    }

    abstract class SqlStatement[T](conn: Connection, sql: String, params: BindingValue*) {

      // Ah, another beautiful feature, lazy vals.  Basically, it's
      // evaluated on initial call.  This is great for the
      // so common lazy memoization technique, of checking for null.
      protected lazy val statement: PreparedStatement = {
        val stmt:PreparedStatement = conn.prepareStatement(sql)
        params.foreach((v) => v >> (stmt, 1))
        stmt
      }

      def execute(): T
    }

    class SqlUpdateStatement(conn: Connection, sql: String, params: BindingValue*)
            extends SqlStatement[Boolean](conn, sql, params: _*) {
      def execute() = statement.execute()
    }

    class SqlSelectStatement(conn: Connection, sql: String, params: BindingValue*)
            extends SqlStatement[ResultSet](conn, sql, params: _*) {
      def execute() = statement.executeQuery()
    }
  }

  /* Implementation of the SQLVariant */

  class MySqlVariant(private val dataSource:DataSource) extends SqlVariant(dataSource) {

    def retrieveConfigurationStatementFor(eventTag: String) =
      new SqlSelectStatement(connection,  "SELECT reader_config FROM event WHERE tag = ?", eventTag)

  }

And the obligatory unit test using the o’ so awesome Scala Specs framework.

  object MySqlVariantSpec extends Specification {
    val ds = getDataSource();

    "Requesting a configuration statement for a specific event" should {
      "return a SqlSelectStatement with properly bound parameters" in {
        val sqlVariant:SqlVariant = new MySqlVariant(ds)
        val stmt:SqlSelectStatement = sqlVariant.retrieveConfigurationStatementFor("abc")
        stmt must notBeNull
        // .... Other assertions go here
      }
    }
  }

Although I barely scraped the tip of the iceberg, I hope this helps you see some of what Scala has to offer. More to come as I progress.

So here is the code, which you can copy and paste into your project, instructions on how to use it are below…

import org.apache.commons.httpclient.protocol.Protocol;
import org.apache.commons.httpclient.protocol.ProtocolSocketFactory;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.net.SocketAddress;
import java.net.UnknownHostException;

import javax.net.SocketFactory;

import org.apache.commons.httpclient.ConnectTimeoutException;
import org.apache.commons.httpclient.HttpClientError;
import org.apache.commons.httpclient.params.HttpConnectionParams;
import org.apache.commons.httpclient.protocol.SecureProtocolSocketFactory;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public static class TrustAllSSLProtocolSocketFactory implements ProtocolSocketFactory {

    public static void initialize() {
        Protocol.registerProtocol("https", new Protocol("https", new TrustAllSSLProtocolSocketFactory(), 443));
    }

    private SSLContext sslcontext = null;

    private static TrustManager trustAllCerts =
            new X509TrustManager() {
                public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; }
                public void checkClientTrusted( java.security.cert.X509Certificate[] certs, String authType) {}
                public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {}
            };

    /**
     * Constructor for TrustAllSSLProtocolSocketFactory.
     */
    private TrustAllSSLProtocolSocketFactory() {
        super();
    }

    private static SSLContext createSSLContext() {
        try {
            SSLContext context = SSLContext.getInstance("SSL");
            context.init(null, new TrustManager[]{trustAllCerts}, null);
            return context;
        } catch (Exception e) {
            throw new HttpClientError(e.toString());
        }
    }

    private SSLContext getSSLContext() {
        if (this.sslcontext == null) {
            this.sslcontext = createSSLContext();
        }
        return this.sslcontext;
    }

    public Socket createSocket(String host, int port, InetAddress clientHost, int clientPort)
            throws IOException, UnknownHostException {
        return getSSLContext().getSocketFactory().createSocket(host, port, clientHost, clientPort);
    }


    public Socket createSocket(final String host, final int port, final InetAddress localAddress,
                               final int localPort, final HttpConnectionParams params
    ) throws IOException, UnknownHostException, ConnectTimeoutException {
        if (params == null) throw new IllegalArgumentException("Parameters may not be null");
        int timeout = params.getConnectionTimeout();
        SocketFactory socketfactory = getSSLContext().getSocketFactory();
        if (timeout == 0) return socketfactory.createSocket(host, port, localAddress, localPort);
        else {
            Socket socket = socketfactory.createSocket();
            SocketAddress localaddr = new InetSocketAddress(localAddress, localPort);
            SocketAddress remoteaddr = new InetSocketAddress(host, port);
            socket.bind(localaddr);
            socket.connect(remoteaddr, timeout);
            return socket;
        }
    }

    public Socket createSocket(String host, int port) throws IOException, UnknownHostException {
        return getSSLContext().getSocketFactory().createSocket(host, port);
    }

    public Socket createSocket(Socket socket, String host, int port, boolean autoClose)
            throws IOException, UnknownHostException {
        return getSSLContext().getSocketFactory().createSocket(socket, host, port, autoClose);
    }

    public boolean equals(Object obj) {
        return ((obj != null) && obj.getClass().equals(TrustAllSSLProtocolSocketFactory.class));
    }

    public int hashCode() {
        return TrustAllSSLProtocolSocketFactory.class.hashCode();
    }
}

Now all you have to do is call TrustAllSSLProtocolSocketFactory.initialize() anywhere in your application initialization code or right before you access any https resources, either through the URL class or through any other library, like HttpClient.

Hope this helps, though it’s still a pretty ugly hack IMO.

Everything seemed like it was working using Drools defined facts using declare. I was able to dynamically create and query the fact types using reflection. This was great, as we can inject different rules and deploy functionality on the fly. I then created a groovy class, parsed it and injected into the drools package builder as well as its working memory.

Code I used to test the rule inference on the fact type.

def domainClass = '''
  package com.buycentives.types
  import com.buycentives.incengine.*
  class Input {
    String name
    int age
    boolean valid
  }
'''
Class inputClass = new GroovyClassLoader().parseClass(domainClass)
def input = inputClass.newInstance()
input.name = "Ilya"
input.age = 25

/// Here we inject the fact and fire the rules, see the following java code snippet

// This code was originally failing
assertEquals input.valid, true

Here is the code I used to assemble the working memory. This code is more modularized in my code and I tried to compile it into a snippet, though I might of missed something and some object instantiations are not shown.

ClassLoader loader = new GroovyClassLoader()
PackageBuilderConfiguration config = new PackageBuilderConfiguration();
config.setClassLoader(loader);

KnowledgeBuilder kBldr = KnowledgeBuilderFactory.newKnowledgeBuilder(config);
kBldr.add(ruleResource, ResourceType.DRL);

if (kBldr.hasErrors())
  throw new RuntimeException("Unable to compile rules. " + kBldr.getErrors().toString());

KnowledgeBase kb = KnowledgeBaseFactory.newKnowledgeBase(new RuleBaseConfiguration(loader));
kb.addKnowledgePackages(kBldr.getKnowledgePackages());
StatelessKnowledgeSession kSession = kBase.newStatelessKnowledgeSession();
kSession.execute(factObject);

The rule that was being asserted

package com.buycentives.incengine.rules

import java.util.Date
import com.buycentives.types.Input

rule "Assert for age"
  when
    $input: Input( age >= 20, age < 30 )
  then
    $input.setValid(true);
end

To make the story short, the rule never fired, although I verified that the Input object instance was injected into the working memory, though never changing the fact’s valid property. I beat my head against the wall over and over and over and finally I realized what was going on. It’s java’s class loader caveat. It’s not really a bug, rather a feature I guess. It’s not really documented in the javadocs, but I’ve seen references to this caveat before.

When you load the same class using two different class loaders, inside the jvm, they are considered two different classes, although logically they are the same, therefore failing the Class.isAssignableFrom and instanceof conditional test.

So the following test would fail…

def domain = "class Test {}"

GroovyClassLoader loader1 = new GroovyClassLoader()
Class clazz1 = loader1.parseClass(domain)

GroovyClassLoader loader2 = new GroovyClassLoader()
Class clazz2 = loader2.parseClass(domain)

assertTrue clazz1.isAssignableFrom(clazz2)

So the lesson here, beware of injecting class loaders, especially when you’re comparing classes that were loaded with a different class loader than the class loader of the thread that’s inferring equality.

Also, beware of GroovyClassLoader.parseClass method. It parses, defines, and links the class, but if you then use it again on the same class, the resulting classes are different.

The following test fails…

GroovyClassLoader loader = new GroovyClassLoader()
Class clazz1 = loader.parseClass(domain)
Class clazz2 = loader.parseClass(domain)

assertTrue clazz1.isAssignableFrom(clazz2)

Also, this link from “Java Specialist Newsletter” is a good source of some interesting class loader information.

I think this handles a large portion of data bindings in web applications, but there wasn’t (at least in frameworks I worked with), a straightforward/reusable way of binding multiple html fields to one data type. I’ve always either had to roll my own custom solution and/or rely on a command (transfer) objects to have a one to one representation of form fields, which I would then bind to the domain object inside a controller action. Some use cases can really benefit from command objects and its decoupling from the domain, but I try to bind directly to domain objects as much as I can, reducing the amount of noise that transfer objects generate. Another solution which I’ve personally never implemented because I thought it was too ugly and leaked too much domain binding implementation into the view layer, was to use a hidden field and then format the multi-field controls into a single text value representation during a submit event, though allowing you to still have a one to one binding between the hidden control and the domain object field.

Today I discovered Grails StructuredPropertyEditor, and after figuring out how to implement it, I was on my way to reducing a bunch of binding boilerplate. This solution is not documented, as many other grails related intricacies, though it does seem like a stable/supported API. After digging through the source code, I figured it out and wanted to share it. So here it is…

I have a form which collects date and time information. The date uses a custom datepicker control and the time uses a custom time control. So two controls, that I need to marshal into one DateTime field in my domain class.

We start off by implementing a custom property editor, which extends PropertyEditorSupport and implements a grails specific StructuredPropertyEditor interface. I decided to use an excellent DateTime implementation class which comes with Joda Time library to store the date/time information. You can easily use any other type of your liking, including the java.lang.Date class.

  public class DateTimePropertyEditor extends PropertyEditorSupport implements StructuredPropertyEditor {

    DateTimeFormatter dateTimeFormat;

    public DateTimePropertyEditor(String format) {
      this.dateTimeFormat = DateTimeFormat.forPattern(format);
    }

    public List getRequiredFields() {
      List requiredFields = new ArrayList();
      requiredFields.add("date");
      return requiredFields;
    }

    public List getOptionalFields() {
      List optionalFields = new ArrayList();
      optionalFields.add("time");
      return optionalFields;
    }

    public Object assemble(Class type, Map fieldValues) throws IllegalArgumentException {
      if (!fieldValues.containsKey("date")) {
        throw new IllegalArgumentException("Can't populate a date without a year");
      }

      String date = (String) fieldValues.get("date");

      try {
        if (StringUtils.isBlank(date)) {
          throw new IllegalArgumentException("Can't populate date/time without a date");
        }
        String time = (String) fieldValues.get("time");
        if (StringUtils.isBlank(time)) time = "00:00 AM";
        String dateTime = date + " " + time;

        return dateTimeFormat.parseDateTime(dateTime);
      }
      catch (Exception nfe) {
        throw new IllegalArgumentException("Unable to parse structured DateTime from request for date.");
      }
    }

  }

Now that we have the property editor defined, we should create a registrar to register it with the application.

  public class CustomPropertyEditorRegistrar implements PropertyEditorRegistrar {

      public void registerCustomEditors(PropertyEditorRegistry registry) {
        registry.registerCustomEditor(DateTime.class, new DateTimePropertyEditor("MM/dd/yyyy hh:mm a"));
      }
  }

Now that we have both the property editor and its registrar, we have to declare the registrar bean in the spring beans resources file /grails-app/conf/spring/resources.groovy

  beans = {
    propertyEditorRegistrar(CustomPropertyEditorRegistrar)

    // .... other beans ....
  }

That’s it, we’re done. The only thing left to do is actually create the html controls. There are a few conventions that must be followed in order for the StructuredPropertyEditor to grab the right fields.

1. You must have a hidden field that’s there pretty much to declare the name of the field you’re binding to and also to state that it’s a ‘struct’ field, which means it’s a structured control made up of various other controls.
2. The second rule, is that the names of the sub-fields that you are using in StructuredPropertyEditor, in our case ‘date’ and ‘time’, have to be prepended with the field name. So if the field name is ‘myField’, the sub-fields would be prepended with ‘myField_’. In our case, the html inputs should be named ‘myField_date’ and ‘myField_time’.

Here is a GSP snippet…

  <g:hiddenField name="startTime" value="struct" />
  <g:textField name="startTime_date"/>
  <g:textField name="startTime_time"/>

Notice how the field we’re binding is ‘startTime’ and the sub-fields that make up the whole, are ‘startTime_date’ and ‘startTime_time’ linked to ‘date’ and ‘time’ fields used in the assembly of the final field.

Now, all you have to do in the controller, is bind as you usually would, in my case…

bindData(domainObj, params)

That’s it, quick and simple and you have reusable marshaling of multi-control data types.