28 Nov 2007

Scala and db4o Native Queries

Recently, I’ve been playing with Scala, a multiple paradigm programming language that runs within the Java virtual machine. While searching for on-line resources describing practical applications of the language, how-tos and best practices, I ran across a blog entry by N. Chime showcasing the object database db40. I immediately felt a rush of excitement. I have been keeping an eye on db4o for years but have not had the opportunity to use it beyond a few short-lived pet projects or explorations of new-to-me technology (e.g. Wicket). The technology has always interested me, and I’ve been hoping to use it in a real application.

Using Chime’s blog as a road map, I started creating a contact management app with Scala and db4o. Yes, this is yet another exploratory project, but I hope to evolve it into something real. Anyhow, everything was coming together fine until I tried to implement db4o’s native queries. No matter what I did in the filter method of the Predicate class, I was unable to get the expected results. As it turns out, there is a bit of a Catch-22 when using this feature.

Here you can see an example of a native query taken directly from Chime’s blog entry. You can trust that the Pilot class and listResult(…) function work as expected. I will provide fully functioning code later, but I want to show the obvious/intuitive implementation and how it fails.

def retrieveComplexNQ(db : ObjectContainer) = {
  val result = db.query(new Predicate() {
    def `match`(point : Any) : boolean = {
      val p = point.asInstanceOf[Pilot];
      return (p.getPoints > 99) &&
             (p.getPoints < 199) &&
             (p.getName.equals("Rubens Barrichello"))
    }
  });
  listResult(result);
}

Note that I have changed the match method’s input parameter from All, as shown in Chime’s entry, to Any, the Scala equivalent to java.lang.Object. Without doing this, the Scala compiler complains because the Predicate class defines the abstract match method with a generically typed parameter. Scala does not support Java generics, so more restrictive typing is lost in translation. That is, Java has generics and Scala has generics, but they don’t work together yet. As a result, the Scala compiler will complain if you try to implement the match method with any type other than Any. I suppose more restrictive type checking of the compiler was introduced through the evolution of the Scala language, which is why the All type (includes null) no longer works.

My attempts at using this implementation failed. I started using the db4o-6.3-java5.jar, and found that my native queries failed at run time with a java.lang.IllegalArgumentException due to invalid predicate. With a little digging on-line, I found this error is generated when the Predicate does not have a properly defined filter method: the boolean returning method named “match” that accepts a single parameter. The above Predicate implementation does contain a “match” method that returns a boolean and accepts a single parameter, so I considered that Scala’s missing support of Java generics could be throwing a wrench into things. I switched the db4o implementation to db4o-6.3-java1.2.jar, which is intended for Java 1.2 through 1.4, to remove generics from the equation. This eliminated the exception; however, the queries failed to return correct results. No matter what I did to the “match” method, even hard coding boolean return values, the database behaved as if the method always returned the same value. Implementing the same queries in Java yielded the expected query results, so this issue was clearly a problem with the Scala code.

And the hero of the day is… open source! I finally got around to looking at the db4o source code to find the root cause of my issue, and it didn’t take long to find the problem within the Predicate.getFilterMethod() method. The code explicitly ignores any match method that accepts a single java.lang.Object parameter. While the db4o documentation clearly states that the match method must take one parameter, it fails to mention that this parameter cannot have the general java.lang.Object type.

So the Scala compiler requires that the match method accepts a parameter typed as Any (effectively an alias for java.lang.Object) while db4o explicitly ignores such a parameter. The workaround is to provide both.

def retrieveComplexNQ(db : ObjectContainer) = {
  val result = db.query(new Predicate() {
    def `match`(point : Any) : boolean = 
        throw new Exception("This should never be called!")
    def `match`(p : Pilot) : boolean = {
      return (p.getPoints > 99) &&
             (p.getPoints < 199) &&
             (p.getName.equals("Rubens Barrichello"))
    }
  });
  listResult(result);
}

The above implementation will generate the expected results because:

The Scala compiler is satisfied by the stub method accepting the Any type
The db4o reflection logic can find a match method with a more constrained parameter type

Certainly this code can better conform to the DRY principle by pulling the stub method into a custom abstract class. For your viewing and testing pleasure, here is the complete code for my working native queries test of the db4o object database with Scala. While I have improved DRY for the stub match method, I’m sure more could be done for converting the query returns into native Scala Iterators (Read: It’s late and I want to get this post up before going to bed).

package db4osc.chapter1;

import com.db4o._;
import com.db4o.query._;
import scala.collection.jcl.MutableIterator

object NQExample extends Application with Util {

val db = Db4o.openFile("chapter1.db");

  storePilots(db);
  retrieveComplexSODA(db);
  retrieveComplexNQ(db);
  retrieveArbitraryCodeNQ(db);
  clearDatabase(db);

  db.close();

  def storePilots(db : ObjectContainer) = {
    db.set(new Pilot("Michael Schumacher",100));
    db.set(new Pilot("Rubens Barrichello",99));
  }

  def retrieveComplexSODA(db : ObjectContainer) = {
    val query : Query = db.query();
    query.constrain(classOf[Pilot]);

    val pointQuery : Query = query.descend("points");
    query.descend("name").constrain("Rubens Barrichello")
      .or(pointQuery.constrain(99).greater()
        .and(pointQuery.constrain(199).smaller()));
    val result = query.execute();
    listResult(SIterator(result));
  }

  def retrieveComplexNQ(db : ObjectContainer) = {
    val result = db.query(new Filter() {
      def `match`(p : Pilot) : boolean = {
        return (p.points > 99) &&
               (p.points < 199) &&
               (p.name.equals("Rubens Barrichello"))
      }
    });
    listResult(SIterator(result));
  }

  def retrieveArbitraryCodeNQ(db : ObjectContainer) = {
    val points = Array(1,100);
    val result = db.query(new Filter() {
      def `match`(p : Pilot) : boolean = {
        return (p.points == 1) || (p.points == 100)
      }
    });
    listResult(SIterator(result));
  }

  def clearDatabase(db : ObjectContainer) = {
    val result : ObjectSet = db.get(classOf[Pilot]);
    while(result.hasNext()) {
      db.delete(result.next());
    }
  }
}

case class SIterator[A](underlying : ObjectSet) extends CountedIterator[A] {
  def hasNext = underlying.hasNext
  def next = underlying.next.asInstanceOf[A]
  def remove = underlying.remove
  def count = underlying.size
}


abstract class Filter extends Predicate() {
  def `match`(dummy:Any) : boolean = throw new Exception("Not supported") 
}


trait Util {
  def listResult(result : Iterator[Any]) : Unit = {
    println(result.counted.count);
    result.foreach(x => println(x))
  }
}


class Pilot(val name : String, var points : Int) {
  def points_+(apoints : Int) : Int = {
    points = points + apoints;
    return points;
  }
  override def toString() : String = return name+"/"+points;
}

Executing this should result in the following output. The integer indicates the first row of output for each result set, and corresponds to the number of hits from each query.

2
Michael Schumacher/100
Rubens Barrichello/99
0
1
Michael Schumacher/100

I suspect the evolution of the Scala programming language led to the original code that Chime implemented becoming invalid code. Hopefully, when Scala gains support for Java generics, the need for this workaround will go away, and a more intuitive solution will work as expected. This was successfully implemented with:

Eclipse 3.3.1.1
Scala plugin for Eclipse with compiler 2.6.9RC312812,
JRE 1.6.0.03
db4o-6.3-java5.jar

EDIT November 29, 2007 – I got to thinking that the above solution makes perfect sense for the Java 5 version of the db4o library, but makes no sense for the version that does not support generics. So, I revisited my implementation with db4o-6.3-java1.2.jar, and found that this will behave correctly when implemented in the obvious way. I was running into trouble because my match method was still implemented with the Any type. When I changed the code to use a more constrained type, it just worked. For example:

def retrieveComplexNQ(db : ObjectContainer) = {
  val result = db.query(new Predicate() {
    def `match`(p : Pilot) : boolean = {
      return (p.points > 99) &&
             (p.points < 199) &&
             (p.name.equals("Rubens Barrichello"))
    }
  });
  listResult(SIterator(result));
}

In this case, my trouble was due to the undocumented (at least in the API) requirement that the java.util.Object is an invalid parameter type for the match method. Still, I’m surprised I did not get an exception. If I get some time and motivation, maybe I’ll dig into it more.

matlik.net

Watching the paint peel

28 Nov 2007