Fun with Dependencies

Announcing longjohn: long stack traces for node.js

Mon, 02 Jul 2012 20:38:00 -0700

Alright, so I’m deploying some new code at PageLever to move our Facebook sync code to node.js. I’ve just slaved over it for a few weeks, creating a queue/worker system (to be publicly released soon) that keeps track of lots of stats and more importantly errors. Very very important to see our errors. Is the problem with us or Facebook? Have we hit our API limits again? Did I fat-finger yet another FQL statement? You know, the vital stuff.

So now I’ve pushed it out there and started up a bunch of workers and I’m monitoring all of them. I start getting a few errors trickling in. No problem, this happens. Let’s check out what they say.

Error: socket hang up
    at Object. (http.js:1124:15)
    at CleartextStream. (http.js:1173:23)
    at CleartextStream.emit (events.js:64:17)
    at Array. (tls.js:792:22)
    at EventEmitter._tickCallback (node.js:190:38)

So that’s really not very helpful. Like not in any way, shape or form. So what to do, what to do… A little googling and I found the awesome long-stack-traces module by Tom Robinson. It’s a very cool bit of code that wraps all async callbacks (even within the core libraries!) with a bit of code to save the current stack trace. I stuck a simple require('long-stack-traces') into an initializer in my code and voila! Now my stack traces looked more like this.

Error: socket hang up
    at Object. (http.js:1124:15)
    at CleartextStream. (http.js:1173:23)
    at CleartextStream.emit (events.js:64:17)
    at Array. (tls.js:792:22)
    at EventEmitter._tickCallback (node.js:190:38)
---------------------------------------------
    at Facebook._executeRequest (/app/node_modules/facebook.node/node_modules/rest.node/index.js:109:7)
    at Facebook.request (/app/node_modules/facebook.node/node_modules/rest.node/index.js:154:8)
    at Facebook.get (/app/node_modules/facebook.node/node_modules/rest.node/index.js:160:8)
    at Facebook.fql (/app/node_modules/facebook.node/index.js:67:8)
    at PostFetcher.fetch_posts (/app/lib/post_fetcher.coffee:31:21)
    at Object. (/app/lib/register_worker_commands.coffee:58:25)
    at Object. (/app/lib/queue_worker_child.coffee:35:14)
    at EventEmitter. (/app/lib/queue_worker_child.coffee:47:12)
    at QueueWorker.error (/app/lib/queue_worker.coffee:177:15)
    at ChildProcess. (/app/lib/queue_worker.coffee:149:24)
    at ChildProcess. (events.js:156:14)
    at ChildProcess.emit (events.js:88:20)
    at Pipe.onread (child_process.js:102:16)
---------------------------------------------
    at Object. (/app/lib/queue_processor.coffee:62:10)
    at Object. (/app/lib/queue_processor.coffee:73:4)
    at Module._compile (module.js:446:26)
    at Object..coffee (/app/node_modules/caboose/node_modules/coffee-script/lib/coffee-script/coffee-script.js:22:21)
    at Module.load (module.js:353:31)
    at Function._load (module.js:311:12)
    at Module.require (module.js:359:17)
    at Object. (module.js:375:17)

Ooooooooo!!!! Guess what! It was an FQL call that hung up on me. No more blaming MongoDB. OK, good.

The Rub

Now we’re golden, right? Well, not exactly. There seems to be an issue with this implementation of long stack traces. My http server wouldn’t start up. I found this issue that echoed my problem, and hacked in the solution. Yay once again!

Not so fast! Now when I run my workers, all of my EventEmitter callbacks get called multiple times. Two times. Three times. Four? Five? WTF is going on? OK, more research. Turns out, long-stack-traces doesn’t implement removeListener. Ahhhhhh, I see now. Hmm, the project hasn’t been updated in a year. I could have sent a message to tlrobinson, but you know what, I need this to work today. I’ll just write it.

Introducing longjohn

So I wrote it. It worked. A few days later, I made it a bit neater, exposed some options, published it to npm and posted it on github. Maybe it’ll be useful to you. Maybe not.

Just install longjohn with

$ npm install longjohn

Then somewhere in the initialization of your node app (or just at the top of your script) add

require('longjohn');

Now your stack traces will be nice and long. You say you want to customize something. Want some options? We’ve got options.

Options

The method used to track async callbacks adds some memory overhead if your code looks something like this:

function foo() {
  // do something
  setTimeout(foo, 1000);
}
foo();

You can see that this will wrap every single callback for eternity, keeping around stack traces that node would normally forget about. This is BAAAAD! To solve this, longjohn will automatically prune the traces kept to the last 10 async callbacks. In order to change this limit, set async_trace_limit.

var longjohn = require('longjohn');
longjohn.async_trace_limit = 5;  // defaults to 10
longjohn.async_trace_limit = -1; //unlimited

Another option is to customize the output for an async callback boundary. By default longjohn will print out a bunch of dashes to represent a callback boundary. Change this by setting empty_frame.

var longjohn = require('longjohn');
longjohn.empty_frame = 'ASYNC CALLBACK';

Nice side-effect

Due to the fact that I’m using the same exact output as long-stack-traces, felixge’s node-stack-trace module can parse the output if you’d like your stack traces in object form. Here’s how to do it.

require('stack-trace').parse(err);

Enjoy!

Build a Car. Don't Make a Faster Horse.

Wed, 12 Oct 2011 18:44:00 -0700

“If I had asked people what they wanted, they would have said faster horses.”

- Henry Ford

Henry Ford’s famous line is often used as a guideline for innovation, but I think that the majority of tech start-ups misinterpret the quote. By and large I’m not talking about “cool” products like games or social networks (before they become useful), but websites and applications that serve a direct business need. When you understand what users want, you can examine why they want it, and then figure out how to give them what they need. Too many start-ups fall into the trap of giving users what they want without really examining why they want it, completely ignoring what they need.

Many start-ups ask users for input in the form of surveys like, “which feature do you want us to build next?” or push-poll style questions like, “when you look for a deal, do you look in the newspaper or online?” They are building faster horses, not cars. They are asking questions that serve their own purpose, not their users’.

The earlier versions of the Windows phone were faster horses. People wanted to use Windows on their phone, so Microsoft gave them a start menu and applications without taking into account why users wanted those features and how Microsoft could address the underlying need in a way that fit the platform more intuitively. Microsoft overlooked the fact that the way people use phones and the things they use them for are wildly different than how they use computers. The iPhone, on the other hand, is a car. It addresses what people need—to have their whole life available in two or three taps. There are no hidden menus to navigate because everything is easily accessible on the screen.

When people wanted to send each other files that were too big for email, sites like Yousendit, Sendspace, and drop.io became popular. Sharing files with other people was part of the problem, but not the core issue. These sites made sense and gave people what they wanted, but failed to understand the root problem, which was the need to seamlessly manage files across various computers, devices, and locations. I don’t know how many times I emailed myself sendspace links to download later. When dropbox came along, it was so easy to use that my mother figured it out on the first try. Dropbox effortlessly won over users because emailing someone a link was no longer necessary. Moving files from place to place was easy as it could be. The underlying need, not the want it created, was addressed.

Flight reservation websites are another example of an industry full of faster horses. The vast majority of them provide nothing more than a long list of prices and flights. Yes I know they find better deals, and while I want that, what I really need is something to make choosing the best flight from that list easier. Picking a flight takes so long when I need to scroll through pages, read all the prices, compare times, layovers, and so on. I end up creating a little list of flights and comparing them myself, and that’s a pain. Hipmunk.com gave me what I needed. All the pertinent information is represented visually in a way that makes the right one for me obvious. Hipmunk also has tabs, and while tabs exist in all modern browsers, integration into the product itself shows a great understanding of what people need when they’re looking for a flight. Price charts, fancy when-to-buy graphics and whatever clutter other sites throw on the screen don’t help me do what I need to do. I need to make an easy decision, not accidentally book a car with my flight when I’m trying to pay for it.

My interpretation of Henry Ford’s iconic quote is that it’s up to us to listen to what our users want and then give them what they need. Next time you’re considering that next product or feature, take some time to think about why your users want it, and whether there is a more intuitive way to give them what they need. And if you’re thinking about your next business or start-up idea, remember that true disruption comes from a deep understanding of what people need, not what they want.

MongoNYC 2011: My Talk on Tracking and Analytics at Signpost

Tue, 14 Jun 2011 00:30:00 -0700

Last week I gave a talk about the way we do analytics and tracking at signpost.com. The content was a follow-on to my post about how we moved the tracking system from MySQL to MongoDB. I hope the talk was informative and invite any questions that people may have.

Here are the slides:

Video to come…

Signpost Tracking & Analytics: MySQL -> MongoDB

Tue, 31 May 2011 13:11:00 -0700

Here at Signpost, we believe in track everything. We keep every request, every event, every log, every exception. We use this data for user tracking, analytics and business intelligence, and code health / bug triage.

Original System

We started out with MySQL. Requests were logged into a request table, except we handled the 1-Many relationship of request parameters in a single serialized column. Events were tracked in two tables: event_tracking and event_tracking_property, where there are many properties per event. Logs and exceptions were sent to log files, where they were rotated and hardly looked at for eternity. We ran with this setup in production for the first few months of Signpost’s lifetime and it worked out just fine.

Here’s the basic idea:

create table request (
  request_tracking_id int not null auto_increment,
  visitor_key varchar(64) not null,
  session_key varchar(36) not null,
  ip varchar(64) null,
  user_agent varchar(128) not null,
  uri varchar(255) not null,
  referer varchar(255) null,
  params text,
  email_user_id int null,
  user_id int null,
  response_code int not null,
  response_time int not null,
  timestamp timestamp not null,
  primary key (request_tracking_id)
);

create table event_tracking (
  event_tracking_id int not null auto_increment,
  visitor_key varchar(64) null,
  session_key varchar(36) null,
  event_name varchar(32) not null,
  timestamp timestamp NOT NULL DEFAULT '0000-00-00 00:00:00'
  primary key (event_tracking_id)
);

create table event_tracking_property (
  event_tracking_property_id int not null auto_increment,
  event_tracking_id int not null,
  event_key varchar(32) not null,
  event_value text,
  primary key (event_tracking_property_id),
);

Problems

Capturing and tracking everything is great. With some trouble we can find which pages are working and which are not, where users are coming from (we use Google Analytics as well for this), what they are doing on the site, when they run into problems, and when our system fails to operate as expected. The only problem with our approach, was the crazy amount of work needed to access and analyze this data for simple things. For example:

How many clicks did deal 212001 get from a session starting with an email click?

select count(distinct et.event_tracking_id)
  from event_tracking et, event_tracking_property etp1, event_tracking_property etp2
 where et.event_tracking_id = etp1.event_tracking_id
   and etp1.event_tracking_id = etp2.event_tracking_id
   and et.event_name = 'deal-click'
   and etp1.event_key = 'deal_at_location_id'
   and etp1.event_value = '212001'
   and etp2.event_key = 'session_ca'
   and etp2.event_value = 'eml';

In our new system, this would end up as:

db.event.count({
  event_name: 'deal-click',
  'event_properties.deal_at_location_id': 212001,
  'event_properties.session_ca': 'eml'
});

This is a simple case, but we can already see the thought that needs to go into the SQL vs the MongoDB query. The speed of both queries are comparable with the correct indexes. The request tracking design also made analysis on request parameters difficult. The parameters of a request were serialized into a query string format:

raw=false&feedId=dealActivity&user-opts=username,avatar&count=100&no-cache=1306867013431&feedType=comment&userId=&targetUserId=&deal-opts=id,location-name,title,category&activity-opts=id,user,type,rawtime,commentcontent&types=commenter&dealId=243844

This made it painfully slow to find requests with a certain parameter. Often we would just write some code in Java to parse the parameters and do the analysis. Not very efficient.

Desires

We wanted to ask more interesting questions in the future. Here are some of our desires:

Development wanted a way to see the session and requests that resulted in an exception
Community Management wanted a better way to help users when they had issues on the site (seeing their requests and pages they visited)
Business wanted both real-time and cumulative statistics, like the top 5 deals for today, or the deal with the most clicks in the past month
Business wanted a way to see if users were gaming the referral bonuses

This is just a short list, but it is indicative of the desires of most sites I can think of.

Considerations

I wanted the system to be fast, but also flexible. I knew that we would have new questions all the time and I didn’t want to re-compile some Java code or write long SQL queries every time. I also wanted a fast way to see aggregated data that I could graph over long periods of time. I had worked with MongoDB before and had built some Java tooling to help create applications quickly (GuiceyMongo). I knew that MongoDB was fast, and could handle a ton of writes per second (we’re averaging 4,000+ per second on a single EC2 large instance also running a Tomcat server). I also knew that I could execute random Javascript as part of my queries, which gives me that flexibility I wanted. Sounds like a plan!

New System

The system that I came up with contains three collections: request, event, event.rollup. The simplest is request.

{
  "_id" : ObjectId("4de53548b09a6541874eb641"),
  "method" : "GET",
  "response_code" : 200,
  "parameters" : {
    "raw" : "false",
    "feedId" : "dealActivity",
    "user-opts" : "username,avatar",
    "count" : "100",
    "no-cache" : "1306867013431",
    "feedType" : "comment",
    "userId" : "",
    "targetUserId" : "",
    "deal-opts" : "id,location-name,title,category",
    "activity-opts" : "id,user,type,rawtime,commentcontent",
    "types" : "commenter",
    "dealId" : "243844"
  },
  "user_agent" : "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17 (.NET CLR 3.5.30729)",
  "referer" : "http://www.signpost.com/deals/Boston-MA/Cafe-Gigu/-5-for-10-Worth-of-Food-at-Cafe-Gigu-/243844",
  "visitor_key" : "6f6d7e229f2a06e6a0dcdac42b09b26d17dbfac2318be53bea72a996204e3731",
  "uri" : "/api/recent-activity",
  "user_id" : null,
  "session_key" : "008F75D5B7515B202DC1A610F9D86539",
  "ip" : "123.28.156.46",
  "response_time" : 4,
  "timestamp" : ISODate("2011-05-31T18:36:56.177Z")
}

This is practically the same as the MySQL table, but with an embedded object for the query parameters. I have indexes on fields that we search often, like visitor_id, session_id, and uri. An example of a common query is to find all the visitor_id and session_ids for a user, along with the time range those ids were active. This allows me to see when and how long a user is on the site, and then dig deeper into their requests.

The queries for this data are pretty simple. For example, the first image comes from a map-reduce:

db.request.mapReduce(function() {
  emit(this.visitor_key, {start: this.timestamp, end: this.timestamp});
}, function(key, values) {
  return {start: values[0].start, end: values[values.length - 1].end};
}, {
  query: {user_id: 12345},
  sort: {timestamp: 1}
});

Now let’s get into the events. I have two collections here, the event and event.rollup collections. The event collection is once again, very similar to the MySQL table that I started with, except with embedded objects and time window information for grouping the events.

{
  "_id" : ObjectId("4de53c85c7d8327f56284f3a"),
  "visitor_key" : "50f46d60e00efe0a17c42c7fffdc68bda530572d1f83972a83d6c61d684ba3ce",
  "event_name" : "deal-click",
  "event_properties" : {
    "ca" : "dl",
    "deal_at_location_id" : 98303,
    "session_ca" : "dl",
    "ct" : "?",
    "cr" : "?",
    "node" : "web2",
    "bot" : "false",
    "user_id" : 163181,
    "guest" : "true",
    "session_cr" : "?",
    "session_ct" : "?"
  },
  "session_key" : "1C9FA142A59465B745ECE2C1ACB15E62",
  "timestamp" : ISODate("2011-05-31T19:07:48.533Z"),
  "window" : {
    "minute" : ISODate("2011-05-31T19:07:00Z"),
    "hour" : ISODate("2011-05-31T19:00:00Z"),
    "day" : ISODate("2011-05-31T00:00:00Z"),
    "week" : ISODate("2011-05-30T00:00:00Z"),
    "month" : ISODate("2011-05-01T00:00:00Z"),
    "year" : ISODate("2011-01-01T00:00:00Z")
  }
}

This format is significantly easier to query, and we have pulled a whole lot of ad-hoc data just from querying these events like the query from the beginning:

db.event.count({
  event_name: 'deal-click',
  'event_properties.deal_at_location_id': 212001,
  'event_properties.session_ca': 'eml'
});

I’ve written email tracking, exception tracking, email and session clickthrough tracking, and a bunch of other little close-to-realtime reports from this collection that we host on an internal Business Intelligence server. Iterating has been very quick, and I can usually answer most questions in a few minutes, if not much less. A quick sidenote, we also have a report table that contains the Javascript code to run all of our reports. To add a new report, write some Javascript and store it in that collection. Our internal BI website knows how to read that collection, present the relevant options to the user, and run the reports, resulting in pretty graphs, like this:

So what is this and how is it done? This is where the event.rollup collection comes into play. It gives us the ability to do long-term analysis across a large aggregate of events, very very quickly. The graph above is actually the click traffic of the top 5 deals for May, summed by hour. The query took less than 2 seconds, and the graph took another 3 to render. That’s pretty fast considering the top line is actually over 10,000 clicks (each an individual event). Here’s how the rollup events look:

{
  "_id" : ObjectId("4de54041c7d8327f56285563"),
  "c" : 1,
  "e" : "deal-click",
  "k" : [{
    "k" : "deal_at_location_id",
    "v" : 245931
  }],
  "s" : "month",
  "w" : ISODate("2011-05-01T00:00:00Z")
}

This is a month rollup started May 1, 2011 for the event named “deal-click” with a key/value pair of deal_at_location_id = 245931 and a count of 1. There are over 12 million of these objects in the database currently, and they get updated every minute without raising the cpu usage of mongod past 5%. Pretty neat! We can filter, sort, and group these by any number of key/value pairs, since MongoDB can index on objects within arrays. This allows powerful queries like the graph above. I’ll show you the basic code behind that query:

function sum_top_n_by_range(from, to, event, property, count) {
  /* create collection that looks like
  {
    _id: property_value,
    value: count
  }
  */
  var r = db.event.rollup.mapReduce(function () {
    emit(this.k[0].v, this.c);
  }, function (k, v) {
    var s = 0;
    for (var i in v) {
      s += v[i];
    }
    return s;
  }, {
    query: {
      e: event,
      s: 'day',
      w: {$gte: from, $lte: to},
      k: {$size: 1},
      'k.k': property
    }
  });

  /* take the top count property_values */
  var keys = r.find().sort({value: -1}).limit(count).map(function (o) {return o._id;});
  r.drop();

  var cursor = db.event.rollup.find({
    e: event,
    s: 'hour',
    w: {$gte:from, $lte:to},
    k: {$size: 1},
    'k.k': property,
    'k.v': {$in: keys}
  }).sort({w: 1});

  var result = {};
  cursor.forEach(function(dataPoint) {
    /* organize data points in the result object as needed by your output */
  });
  
  return result;
}

It’s a lot of code, but not all that crazy, right?

Thoughts

I’m very pleased with our decision to move this system to MongoDB. The nature of event and request tracking does not require pre-conceived columns and hard data constraints, which saves a lot of development and planning time. The ability to embed objects inside documents has made querying complex data and conceptualizing the records more straightforward. A large win was the ability to compute rollups in near real-time because of the upsert speed of MongoDB. I can create far more rollups than events if we choose to track more complex data and not worry about bogging down the database with writes. Currently we’re clocking between 4,000 and 7,000 updates per second on a full rebuild, and around 500 per second during peak website usage. If this becomes a pain point, sharding out the rollups would be an easy solution to increase our write throughput.

MongoNYC 2011

I’ll be speaking about this and showing more of the way Signpost uses MongoDB at the MongoNYC 2011 conference next week. If you don’t have a ticket yet, why not get one?

Slides & Video

Slides: http://bit.ly/jvwev4

MongoDB Blog Contest - I Won!

Thu, 29 Jul 2010 11:04:00 -0700

I entered my post on Why (and How) I Replaced Amazon SQS with MongoDB into the MongoDB Blog Contest and won!

As part of the grand prize, I went out to OSCON in Portland, OR and had a great time! Kristina Chodorow’s new book on MongoDB was on display and I spoke to lots of people who are interested in what MongoDB is and what it can do for them. I also went to a lot of talks, the most interesting of which (in my opinion) were about node.js and the analytics done at Twitter using Hadoop and Pig.

If you haven’t checked out node.js yet, you should. It’s a really cool technology that provides an event based I/O system on top of Google’s V8 javascript engine. Now you can take all of that javascript hacking that you’ve been doing and write shell scripts using javascript (sort of like how your C# knowledge helps with Powershell) or even write full-fledged web servers! Node.js is incredibly fast and manages to stay fast at large quantities of connections by following an I/O strategy like NGINX or even a windows application. Rather than having a thread pool to post work to or creating a new thread per connection, it uses a single thread and a message queue of operations (I’m not sure if there are multiple threads with their own queues, but you get the idea). Also, node.js is designed such that any I/O operation at all requires a callback. This can make for some messy looking nested code, but ensures that you think about how to parallelize your application as much as possible. It’s really easy to get started and has a great community behind it. My favorite starting point was Blog rolling with mongoDB, express and Node.js.

I haven’t personally used Pig and I’m not currently using Hadoop, but if I were to start using Hadoop, Pig would be the first thing I would install. Hadoop allows you to store and process large amounts of data in a distributed fashion. In order to process the data, you write map and reduce methods in Java and then let Hadoop take over. The problem with this comes when you’re rewriting large chunks of Java code and compiling over and over again to figure out how you want to view your data and how to process it. At Twitter, with over 75 Billion tweets, a simple map-reduce can take a long time! Then you need to debug your code, compile, and run. Then again, and again. Now if you’re trying to do a distributed join, the pain really starts. Kevin Weil from Twitter showed the Java code, which is over 200 lines and needs to be written custom to each map-reduce you perform. That’s slow and painful! In comes Pig. Pig is a scripting language that rides on top of Hadoop and allows you to quickly perform operations on your data without having to worry about writing the Java code and compiling. It’s much more terse and readable and takes away a lot of the debugging headaches in writing your own joins or other complexities. It seems like a great addition to the map-reduce world!

Why (and How) I Replaced Amazon SQS with MongoDB

Tue, 25 May 2010 12:09:00 -0700

What is Amazon SQS?

Amazon SQS (Simple Queue Service) is a reliable message queuing service hosted in the Amazon cloud. This service is ideal for sending messages between servers that need to acknowledge that processing has been completed. When a message is popped from the queue, it is not deleted, but marked with the client who has made the request. The client is then responsible for telling SQS to delete the message from the queue. If the client does not delete a message it has popped within a certain time frame, the client loses ownership of the message and it is made available for other clients.

How am I using it?

One of the systems I am using SQS for is a distributed email delivery service (using SMTP). Since there is not an asynchronous SMTP client for Java (that I know of), I am using JavaMail to deliver messages. Sending messages with JavaMail is pretty slow and can take a number of seconds per message, with a thread being consumed for each message sent. In order to send many many messages in parallel I decided to queue up the outoging messages and spin up many instances of the SMTP application. This approach is dead simple and scales wonderfully without needing to implement an asynchronous SMTP client of my own.

So what’s wrong with Amazon SQS?

The main problem with using SQS in the above scenario is that I can’t push an entire email message onto the SQS queue since each SQS message is limited to 8K of data. To get around this, I store the message in MongoDB and then queue the message ID on SQS. Each client then needs to pull a message from the queue and then look up the email in Mongo. There’s nothing really wrong with this approach, but it can be done better, faster, easier, and cheaper. Amazon charges me a fraction of a cent for every operation I perform on SQS. This doesn’t seem like much, but if I have 10 SMTP applications polling SQS 4 times a second all day every day regardless of whether there are new messages to send, this can add up. Plus, I have diagnostic applications watching the queue size to see if I need to spin up more instances or take down instances. Even if this adds up to $10/day, that’s still $3,650/year just to send out email. That’s too much for a startup with no financial backing!

The approach

I have been using MongoDB for a while now and am enamored with what it can do. I know that it can store lots of schema-less data in 4MB chunks (a document is limited to 4MB) and can store larger files through the use of GridFS. I know that it’s lightning fast (almost memcached speed) for indexed lookups and can handle thousands of operations per second without spiking the CPU over 10% even. I know that I’m paying for the CPU and hard drive space on Amazon EC2 already and thoroughly enjoy minimizing my monthly, weekly, and even daily costs. Blah. Blah. Blah. I want to implement this in Mongo!

With the introduction of server-side javascript and the findAndModify command, using MongoDB for a queue that can be accessed by any client language (of which there are a ton!) is just easy. Below is the code that I am using on my own projects.

The Code

sqs.js

function sqsQueueExists(name) {
	return db.queue[name].count() != 0;
};

function sqsQueueMessageCount(name) {
	return db.queue[name].count({
		alive: true,
		expires: {$lt: new Date}
	});
};

function sqsDeleteQueue(name) {
	db.queue[name].drop();
};

function sqsListQueues(prefix) {
	var regex;
	if (prefix)
		regex = new RegExp('^[^.]+\.queue\.' + prefix + '[^$]*$');
	else
		regex = /^[^.]+\.queue\.[^$]+$/;

	return db.system.namespaces.find({
		name: regex
	}).map(function (x) {
		return x.name.substring(x.name.indexOf('.') + 7);
	});
};

function sqsPushMessage(queue, message) {
	var _push = function(queue, message) {
		db.queue[queue].save({
			alive: true,
			expires: new Date(0),
			owner: new ObjectId('000000000000000000000000'),
			body: message
		});
	};

	if (message instanceof Array) {
		message.forEach(function(m) {
			_push(queue, m);
		});
	} else {
		_push(queue, message);
	}
};

function sqsPopMessage(queue, owner, count) {
	var now = new Date;
	// 10 second expiration, change this to what you want
	var expires = new Date(now.getTime() + 10000);
	
	if (!count) {
		count = 1;
	}
	var result = [];
	for (var i = 0; i < count; ++i) {
		var item = db.queue[queue].findAndModify({
			query: {
				alive: true,
				expires: {$lt: now}
			},
			update: {
				$set: {
					expires: expires,
					owner: owner
				}
			},
			new: true
		});
		if (friendlyEqual({}, item))
			break;
		result.push(item);
	}
	return result;
};

function sqsDeleteMessage(queue, owner, item_ids) {
	if (item_ids instanceof ObjectId)
		item_ids = [item_ids];
	db.queue[queue].update({
		alive: true,
		expires: {$gte: new Date},
		owner: owner,
		_id: {$in: item_ids}
	}, {
		$set: {
			alive: false
		}
	},
	false,
	true);
};

If you copy the code into a sqs.js file, you can then run the script below to create the stored procedures on the database of your choice. The alternative is to write the code in the MongoDB driver of your choice.

bash

$ for function in sqsQueueExists sqsQueueMessageCount sqsDeleteQueue sqsListQueues sqsPushMessage
	sqsPopMessage sqsDeleteMessage; do
	echo "db.system.js.save({_id: '$function', value: $function})" | 
		mongo [db name] --quiet --shell sqs.js
done

MongoDB Shell

> use [db name];
> load('sqs.js');
> [sqsQueueExists, sqsQueueMessageCount, sqsDeleteQueue, sqsListQueues, sqsPushMessage,
	sqsPopMessage, sqsDeleteMessage].forEach(function (x) {
	var name = x.toString().match(/^function\s(\w+)/)[1];
	db.system.js.save({_id: name, value: x});
});

There are pieces of the SQS API that I have left out of this implementation for simplicity. These are based around the options provided, like changing the visibility timeout of a queue or individual message. This would be pretty trivial to add if necessary. However, if you are not using these features, leaving them out of the code will only make the code faster. To add options to a queue, just add another collection called queue that holds all queue names (in the _id field) and the applicable options. Then just do an query on the queue collection when the options are needed (the _id field is automatically indexed, so this will be fast).

I hope this helps you out!

MongoDB Java DAO Generator - GuiceyData

Thu, 20 May 2010 15:29:00 -0700

One of the best things about MongoDB is the lack of an enforced schema for collections. This flexibility gives developers a lot of power in how they work with their data. Embedding records and arrays inside other records allows both a complexity and simplicity of data organization that RDBMSs can only dream of! All of that being said, working with these records in a language like Java and on large diverse teams of people who don’t want to open the database and inspect the records to see what values and sub-records are available, means that you will always spend time wrapping these records in a strong-typed class. Wrapping up loose data into classes that can both access and create that data sounds just like another project I’ve used recently. If you haven’t heard of Google’s Protocol Buffers, you might want to acquaint yourself.

Since I’ve enjoyed working with Protocol Buffers so much, I thought I could mimic their functionality and ease of use with MongoDB. This would also integrate beautifully with the GuiceyMongo project that I released a month or two ago.

What is the GuiceyData Generator?

The GuiceyData Generator is a quick and easy way to specify strongly typed data structures to be stored in a MongoDB database and mapped to wrappers and builders in Java. The resulting classes are strictly data and have (by design) very limited functionality. Their purpose is to make reading and storing data in MongoDB a breeze from Java.

How does it work?

It’s pretty straightforward. You create a data definition file and then run the generator. This will create wrappers and builders for all of the types you define. Here’s a very simple example:

simple.data

data Person {
  string name;
  set<string> alias;
  blob picture;
}

Then you can run the generator:

$ java -cp guiceymongogenerator-0.1.0.jar 
com.lowereast.guiceymongo.data.generator.GuiceyDataGenerator -p test.data -s src

This will create the directory structure and files below:

+ src
  + test
    + data
      - Person.java

Wrappers

Once you have generated the code, you can do something like this:

Person p = Person.wrap(personCollection.findOne());
if (p.hasName())
  log.trace(p.getName());
if (p.getAliasCount() > 0)
  log.trace(StringUtil.join(p.getAliasSet(), ", "));
if (p.hasPicture()) {
  Image picture = ImageIO.read(p.getPictureInputStream());
  // do something with the picture
}

Please note that in the example above, blobs will only work if you are using GuiceyMongo to configure your databases, collections, and buckets. Using GuiceyMongo, we could also just do this:

@Inject
void loadPerson(@GuiceyMongoCollection("person")
  GuiceyCollection personCollection) {
  Person p = personCollection.findOne();
  // ...
}

Builders

What would be the use of being able to read data if we couldn’t create it as well? For this, there are builders:

Person.Builder p = Person.newBuilder()
                         .setName("Matt Insler")
                         .addAlias("Matt")
                         .addAlias("Guice & Mongo Guru")
                         .setPictureBucket("pictures");
ImageIO.write(picture, format, p.getPictureOutputStream());
personCollection.save(p.build());

And once again with GuiceyMongo:

@Inject
void loadPerson(@GuiceyMongoCollection("person")
  GuiceyCollection personCollection) {
  Person p = Person.newBuilder() // ...
  personCollection.save(p);
}

It’s really just that easy!

Builder Prototypes

You can create builders based on another object. This can be a wrapper or a builder itself. This is useful when you would like to use a prototype object or just copy an object that you’ve just read.

Person p = Person.wrap(collection.findOne());
Person newPerson = Person.newBuilder(p);

More Complex

Here’s an example of a more complex data definition file that shows you more of what the generator can handle:

Contact.data

data Contact {
  data Address {
    string street_1;
    string street_2;
    string city;
    string state;
    int zip_code;
  }
  
  [identity]
  string identity;
  
  string first_name;
  string last_name;
  map<string, Address> address;
  map<string, string> phone_number;
  map<string, string> email_address;
  map<string, InstantMessenger> instant_messenger;
  set<string> tag;
  blob picture;
}

data InstantMessenger {
  enum Application {
    AIM,
    ICQ,
    Jabber,
    MSN,
    Yahoo
  }

  string screen_name;
  string alias;
  IMApplication application;
}

Integration with GuiceyMongo Stored Procedure Proxies

Now that you have generated code to access and build your data, what if you want to return that data from a stored procedure? This is pretty easy!

public interface ContactQuery {
  Contact findContactByName(String name);
  List findContactsByIMAlias(String alias);
}

@Inject
void exercise(ContactQuery query) {
  query.findContactByName("Matt Insler");
}

Not too bad, right?

Supported Data Types

Basic Java Types

Guicey Type                       Java Type

double                      =>    double
float                       =>    float
int32                       =>    int32
int64                       =>    int64
bool                        =>    boolean
string                      =>    String
date                        =>    java.util.Date

MongoDB Types

Guicey Type                       Java Type

object_id                   =>    ObjectId
db_object                   =>    DBObject
db_timestamp                =>    DBTimestamp

Collection Types

Guicey Type                       Java Type

list<[type]>                =>    List<[type]>
set<[type]>                 =>    Set<[type]>
map<string, [type]>         =>    Map<String, [type]>
map<[enum type], [type]>    =>    Map<[enum type], [type]>

where type is:
- Basic Type
- MongoDB Type
- User Data Type
- User Enum Type

User Data Types

data [name] {
  [type] [property name];
  ...
  OtherData.Embedded [property name];
  data [embedded type name] {
    [type] [property name];
    ...
  }
}

User Enum Types

enum [name] {
  [value],
  ...
  [value]
}

A note on enum types: User defined enum types will be stored in MongoDB as [enum value].name(). Conversions will be made when reading and storing these values. Be very very careful when changing enum values. They will need to be changed in the database as well.

Options

Currently there is only one available option. This is the identity option. Options are specified directly above the property they should apply to and are enclosed in square braces.

data Foo {
  [identity]
  string identity;
}
data Bar {
  [identity]
  object_id identity;
}

The identity option specifies that this property should be read and stored as an ObjectId and used as the _id value. The generated code will perform automatic conversions between ObjectId and String if the property’s type is string.

Why not do what others are doing?

In two words, personal preference. While I think that other available libraries have their place, I have certain preferences that have lead me to develop GuiceyData the way that I did.

GuiceyData vs. Morphia

Morphia is a JPA style library that allows you to annotate POJOs and then convert them to and from DBObjects. It’s rather nice and supports a bunch of features like strong types queries, DAO/Datastore generation, embedded and referenced data objects, and more. It’s definitely worth checking out and could be the right library for you.

The main positive about morphia is the ability to persist POJOs that you’ve already created and been using. Just annotate the fields and go! What I don’t like is that I don’t like dealing with full object conversions. The code isn’t as fast or efficient as it could be because it has to use reflection and must process all fields of an object regardless of what fields are actually being used in your code. Plus, with a schema-less database like mongo, you can’t assure that all of the values in the POJO will be filled in when you convert the object, so you can’t use primitive types since they are not nullable. Another problem I have is that I really really really don’t like creating my own custom data stores and the lack of outside control on what database and collections are being used. With GuiceyMongo I can generate object wrappers that will lazily access the data with code written specifically for the object I’m dealing with. No reflection and lazy loading and processing == really fast! GuiceyMongo also separates the data spec from java code. This is very useful for when you want to share objects across an organization that uses different languages to access the database. All languages can be generated from the same specification. In my opinion (realize that I am the creator of GuiceyMongo), GuiceyMongo achieves more than morphia in a cleaner way and is faster.

GuiceyData vs. Sculptor

Sculptor is a code generator that has been adapted to create MongoDB conversion code. It follows the idea of Rails by scaffolding lots of methods and classes to help you deal with your data classes. It’s a bit daunting to understand exactly what’s going on, but you can figure it all out once you play with it a bit. Just like morphia, Sculptor will convert your data objects to and from DBObjects. Unlike morphia, Sculptor generates this conversion code which makes it faster since there’s no reflection. What I really like about Sculptor is query language that it provides. This is a much easier way to query your generated objects than constructing DBObjects by hand or even using QueryBuilder in the com.mongodb package. This functionality has caused me to start creating my own query DSL that I will hopefully release in the next month. Check it out!

Now for the criticisms. My main and largest criticism of Sculptor is that it’s so flexible that it becomes complex. I’m a large proponent of your data objects being just that… Data objects. I don’t see a very compelling use case for scaffolding classes that inherit from your data classes and stubbing out extra functionality. If you want that functionaliy, write it, but leave it out of your data definition. The same goes for repositories. They really aren’t buying you that much. Why do you need to generate the common methods for each and every type of repository? Isn’t this what generics are for? All of the base repository classes will be identical, except for the methods that you add to them. Scaffolding is nice, but not necessary. I also think that generating the conversions is a bit short-sighted. If you’re generating code, why not generate lazy loading in your data objects? I understand that people like POJOs, but they’re very slow. We’re dealing with databases here. Your bottleneck should be the queries and data processing, not constructing the objects around the data that you’ve received or are storing. My last gripe is something that I chose not to support in GuiceyMongo because of the fact that MongoDB is schema-less. Sculptor supports object inheritance. This is a nice feature, but I’m not sure that it’s absolutely necessary. An object in MongoDB can be many different types all rolled into one. I can create an email object and then overlay a task object on top of it. I don’t want an inheritance chain for this. I want a way to convert one object type to another. Wrapping raw DBObjects with accessors is a very clean and fast way to do this. In GuiceyData all you need to do is Email e = Email.wrap(…); Task t = Task.convertFrom(e); The data types can then be stored in the database in the same object or in different ones, or even different collections.

Enterprise Usage

Both GuiceyMongo and GuiceyData were created with enterprise usage in mind. GuiceyMongo allows you to create multiple configuration environments, like Test, QA, Prod, etc. where you can map databases, collections, and buckets (GridFS) to any server, database, or collection that you want. Just by changing the configuration, your data can be loaded from or saved to somewhere else. If this configuration is abstracted into a config file or configuration library, this can be shared across the enterprise and overridden locally if necessary. As for the data generation, the definition files only talk about data. That’s all they ever will talk about. This means that code can be generated in many different languages with ease. If you are writing a data access library that will be distributed to other teams, you can use Google Guice to provide an easy way to configure your library, and package the generated data classes. There is no need to make your users generate the code, just use your jar or so or dll.

The integration of GuiceyMongo and GuiceyData will make it dead easy to get started with MongoDB from Java. In the future I’ll write a quickstart post to s how everyone how quick and easy it can really be!

Try it!

Just head over to the github download page and grab the 0.1.0 jars. To generate your classes run the generator like above. To use GuiceyMongo, add a reference to it and it’s dependencies and look at my other posts on the subject.

Dependencies

guiceydatagenerator-[version].jar - None
guiceymongo-[version].jar - aopalliance.jar (included with Google Guice), guice-2.0.jar, mongo-1.4.jar (or later)

As always, any issues you encounter, please contact me!

MongoNYC Conference May 21st

Fri, 30 Apr 2010 11:07:00 -0700

MongoDB is coming to NYC in May. Andre deCavaignac and I will be speaking at the conference about MongoDB in C#/.Net. We have a cool demo to show everyone and I’ll be posting some of the source here after the conference.

Click here to register for the conference

MongoDB is a document-oriented database that I am currently enamored with and use in all of my data-driven projects. It’s created for and supported on most environments and has drivers for most of the programming languages you could want to use. Just imagine all of your data stored as JSON objects, accessible by a fairly powerful (and getting better every month) query language, along with lightning fast documents updates, inserts and queries.

Guicey Tips - Google Guice Provider Modules

Tue, 13 Apr 2010 05:38:00 -0700

Following up on my post about Google Guice Module De-Duplication I want to show a good use of singleton modules. Motivation for the Provider Module pattern came from writing the GuiceyMongo library. Every time a database or collection is configured, I need to install a provider for that database key or that collection key. Since there can be a Production, Test, QA, etc. configuration for each database or collection key I would get an exception when binding the provider twice for the same key. I needed a way to only install each provider once. This is where Provider Modules come into play.

Rather than binding the database provider like this:

bind(DB.class).toProvider(new DBProvider("dbName"));

We would bind it like this:

install(new DBProviderModule("dbName"));

The real difference is between DBProvider and DBProviderModule. Here is a simple example of these classes:

public class DBProvider implements Provider {
  private final String _database;
  public DBProvider(String database) {
    _database = database;
  }
  public DB get() {
    try {
      return new Mongo().getDB(_database);
    } catch (Exception e) {
      return null;
    }
  }
}

public class DBProviderModule extends SingletonModule<Key> implements Provider {
  private final String _database;
  public DBProviderModule(String database) {
    super(Key.get(DB.class, Names.named(database)));
    _database = database;
  }
  public void configure(Binder binder) {
    binder.skipSources(DBProviderModule.class).bind(DB.class).toProvider(this);
  }
  public DB get() {
    try {
      return new Mongo().getDB(_database);
    } catch (Exception e) {
      return null;
    }
  }
}

In this example, there is one provider created for each DB, @Named combination. You can create your own meaningful annotations rather than using the @Named annotation that comes with Guice. In GuiceyMongo I use @GuiceyMongoDatabase for this purpose.

The beauty of the Provider Module approach is that the same module can be installed multiple times without causing any binding errors. This allows your modules to be reused without worrying that the providers will conflict with each other. They can be installed multiple times, but will only be configured once.

I hope this helps!

Guicey Tips - Google Guice Module De-Duplication

Mon, 05 Apr 2010 16:01:00 -0700

There is no current way in the Google Guice API to run a Module’s configure method only once. This might not seem like a big deal until you try writing something like a logging configuration module or any other module that could be added or augmented in multiple places in your code. For instance, in my GuiceyMongo library, I wanted to allow users to add configurations from multiple modules. So the CalendarModule can add the calendar collection to the configuration and the StopwatchModule can add the stopwatch collection to the configuration. This is similar to the effect that the Guice Multibindings extension accomplishes.

So, how do you do this in your own code? Elite Guice 2: Binding de-duplication over at publicobject tells us how. Just override the hashCode and equals. It would look something like this:

public class RunOnceModule implements Module {
  public void configure(Binder binder) {
    // ...
  }

  @Override
  public boolean equals(Object obj) {
    return RunOnceModule.class.equals(obj.getClass());
  }

  @Override
  public int hashCode() {
    return RunOnceModule.class.hashCode();
  }
}

In my GuiceyMongo library I have abstracted this idea into a reusable class that I call SingletonModule. It looks something like this:

public abstract class SingletonModule implements Module {
  protected T key;

  public SingletonModule(T key) {
    this.key = key;
  }

  @Override
  public boolean equals(Object obj) {
    return obj instanceof SingletonModule
      && ((SingletonModule)obj).key.equals(key);
  }

  @Override
  public int hashCode() {
    return key.hashCode();
  }

  @Override
  public String toString() {
    return getClass().getName() + "(key=" + key.toString() + ")";
  }
}

This allows me to use any type of key object, such as the Class object of the module, or a Keyobject if necessary.

This concept becomes very useful in a pattern I like to call Provider Modules, where a singleton module is used to configure a provider of some type only once. I’ll explain this a bit more in a future post.

When using this pattern, your module’s configure method will be called at one of two points (whichever is first):

Binder.install(…)
Guice.createInjector(…)

Knowing this, you cannot assume anything about when your module will be executed. You cannot keep around information to bind when configure is run. Instead try to bind right away and use a singleton module to configure things like shared providers.

In a future post, I’ll discuss how Guice Multibindings and GuiceyMongo use singleton modules to achieve a kind of distributed binding augmentation.

I hope this helps!

Google Guice & MongoDB - GuiceyMongo Configuration

Wed, 17 Mar 2010 13:00:00 -0700

After using Google Guice for a while, I have come to love it. I feel the same way about MongoDB, especially with all the new features they’ve been adding since 1.0. Following the shiny object, like I do sometimes, I wanted a good way to combine the two. In my own projects I have done this a few different ways, but kept finding the need for something just slightly different. Eventually I ended up with a few features I would want in this kind of library:

annotation-based selection of the database or collection to inject
easy to understand configuration (via Guice or other configuration sources)
ability to map a database key to the actual database name (e.g. always refer to the “User” database, but map it to “prod_user” or “test_user” in the configuration)
ability to have configurations that map database/collection keys to different names for different environments, like Test, QA, and Production

To explain the approach I have gone with, I won’t waste any more time and just show some code examples.

Injector injector = Guice.createInjector(
  GuiceyMongo.configure("TEST")
    .mapDatabase("MAIN").to("test")
    .mapCollection("USER").to("user").inDatabase("MAIN"),

  GuiceyMongo.configure("PROD")
    .mapDatabase("MAIN").to("prod")
    .mapCollection("USER").to("user").inDatabase("MAIN"),

  GuiceyMongo.chooseConfiguration("TEST")
);

Here I create two configurations, TEST and PROD and set up a MAIN database and a USER collection in each. I can map any number of databases and collections within a configuration. Above, the TEST configuration for database key MAIN is mapped to actual database “test” and the PROD configuration for the same key is mapped to actual database “prod”. As for collection mappings, both map the USER collection key to actual collection “user” within their respective databases. Collection mappings must reference a mapped database key.

The last line is very important and should only exist once in your application. It activates one of the configurations that you have set up. Only the mappings within the chosen configuration will be active. This could be passed in from a command line argument and then set in the injector creation for more flexibility.

Now that we have defined our configurations and created an injector, how do we request an injection of a specific database or collection? Using annotations of course! The @GuiceyMongoDatabase annotation is used with a database key to inject a DB object, and the @GuiceyMongoCollection annotation is used with a collection key to inject a DBCollection object.

public class Foo {
  @Inject
  Foo(@GuiceyMongoDatabase("MAIN") DB database,
    @GuiceyMongoCollection("USER") DBCollection collection) {
    ...
  }
}

When working on a large application, it’s probably a good idea to create classes that encapsulate the required databases or collections for the individual module and then create your mappings from a central location that installs all the modules.

public class UserModule extends AbstractModule {
  public static final class Collections {
    private Collections() {}
    public static final String User = "USER";
  }

  @Override
  protected void configure() {
    ...
  }
}

public class ApplicationModule extends AbstractModule {
  public static final class Configurations {
    private Configurations() {}
    public static final String Production = "PROD";
    public static final String Test = "TEST";
  }
  public static final class Databases {
    private Databases() {}
    public static final String Main = "MAIN";
  }

  private final String _configuration;
  public ApplicationModule(String configuration) {
    _configuration = configuration;
  }

  @Override
  protected void configure() {
    install(new UserModule());
    install(GuiceyMongo.configure(Configurations.Test)
      .mapDatabase(Databases.Main).to("test")
      .mapCollection(UserModule.Collections.User).to("user").inDatabase(Databases.Main)
    ),
    install(GuiceyMongo.chooseConfiguration(_configuration));
  }
}

Hopefully this quick walkthrough piques your interest in GuiceyMongo and gives you an idea of how easy the setup actually is. There are more features implemented and I have more planned. I’ll be blogging about other features in the future, so check back periodically!

If you would like to use GuiceyMongo, it is available to the public:

I plan on maintaining and enhancing this library regularly and would love to hear your input! If you use my library, please leave me a comment so I know that people are finding it useful. Thanks!

Google Guice & MongoDB - GuiceyMongo Stored Procedure Proxies

Tue, 16 Mar 2010 12:30:00 -0700

Writing and maintaining stored procedures in a SQL database was both painful and annoying to manage, test, and easily encapsulate in code. I usually ended up with an interface defining all of the stored procedures and a lengthy class implementation with JDBC templates or lines of setup and teardown code for each procedure to run. If the procedure changed on the server, I needed to change the interface and a bit of code in the implementation. As much as I hated maintaining this, I know that stored procedures are necessary.

MongoDB has a version of stored procedures. This is javascript code that you can store in your database and call using the eval command. It’s a very simple system to use, although not completely intuitive to SQL people. Just write your javascript code and then save it into the system.js collection within the database you want to call it from.

> db.system.js.save({
  _id: 'hello_world',
  value: function() {
    print('hello world!')
  }
})

When using the MongoDB shell, you have to realize that these functions will not be available to call directly. You must use an eval. So to call our hello_world function, we must do this:

> db.eval('hello_world()')

This will print out “null” in the shell window (since we don’t return a value) and “hello world!” in the server log (or server output if you ran it from a command shell). You can return any value that you want except cursors. So returning the number 5 or {value: 5} is allowed.

Now, onto the Java code. Let’s assume that you have two methods saved in your test database defined as follows:

> db.system.js.save({
  _id: 'getCount',
  value: function() {
    return db.data.count()
  }
})
> db.system.js.save({
  _id: 'getData',
  value: function(count) {
    return db.data.find().limit(count).toArray()
  }
})

Your code might look something like this:

public class Data {
  private static Logger _logger = Logger.getLogger(Data.class);
  private final DB _database;

  public Data(DB database) {
    _database = database;
  }

  void doSomething() {
    Integer count = (Integer)_database.eval("return getCount()");
    if (count == null) {
      _logger.error("Could not execute getCount");
      return;
    }

    List data = (List)_database.eval(
        "function(c){return getData(c)}",
        Math.min(20, count)
    );
    if (data == null)
      return;

    if (data != null) {
      for (DBObject obj : data) {
        System.out.println(obj);
      }
    }
  }

  public static void main(String[] args) {
    Mongo mongo;

    try {
      mongo = new Mongo();
    } catch (MongoException e) {
      // ...
    } catch (UnknownHostException e) {
      // ...
      return;
    }

    DB database = mongo.getDB("test_db");
    Data data = new Data(database);
    data.doSomething();
  }
}

With GuiceyMongo your code could look like this:

public interface DataQuery {
  List getData(int count);
  int getCount();
}

public class Data {
  private static Logger _logger = Logger.getLogger(Data.class);
  private final DataQuery _query;

  @Inject
  Data(DataQuery query) {
    _query = query;
  }

  void doSomething() {
    try {
      int count = _query.getCount();
      List data = _query.getData(Math.min(20, count));
      if (data != null) {
        for (DBObject obj : data) {
          System.out.println(obj);
        }
      }
    // throwing from a server-side method will be caught in a
    // GuiceyMongoEvalException rather than a MongoEvalException
    } catch (GuiceyMongoEvalException e) {
      _logger.error(e);
    }
  }

  public static void main(String[] args) {
    Injector injector = Guice.createInjector(
        GuiceyMongo.configure("Test")
          .mapDatabase("Main").to("test_db"),

        GuiceyMongo.javascriptProxy(DataQuery.class, "Main"),

        GuiceyMongo.chooseConfiguration("Test")
    );
    Data data = injector.getInstance(Data.class);
    data.doSomething();
  }
}

For more on configuring your databases and collections, check out GuiceyMongo – Injecting databases and collections.

Changes in the server method only need to be reflected in the interface definition. All implementation code is taken care of for you.

Using a query interface and Google Guice injection also makes mocking and writing tests much easier. This approach as well as some other tricks I’ll post in the future have saved me a whole lot of time.

Enjoy!

If you would like to use GuiceyMongo, it is available to the public:

I plan on maintaining and enhancing this library regularly and would love to hear your input! If you use my library, please leave me a comment so I know that people are finding it useful. Thanks!