Fun with Dependencies

Writing and maintaining stored procedures in a SQL database was both painful and annoying to manage, test, and easily encapsulate in code. I usually ended up with an interface defining all of the stored procedures and a lengthy class implementation with JDBC templates or lines of setup and teardown code for each procedure to run. If the procedure changed on the server, I needed to change the interface and a bit of code in the implementation. As much as I hated maintaining this, I know that stored procedures are necessary.

MongoDB has a version of stored procedures. This is javascript code that you can store in your database and call using the eval command. It’s a very simple system to use, although not completely intuitive to SQL people. Just write your javascript code and then save it into the system.js collection within the database you want to call it from.

Read More

After using Google Guice for a while, I have come to love it.  I feel the same way about MongoDB, especially with all the new features they’ve been adding since 1.0.  Following the shiny object, like I do sometimes, I wanted a good way to combine the two.  In my own projects I have done this a few different ways, but kept finding the need for something just slightly different.  Eventually I ended up with a few features I would want in this kind of library:

• annotation-based selection of the database or collection to inject
• easy to understand configuration (via Guice or other configuration sources)
• ability to map a database key to the actual database name (e.g. always refer to the “User” database, but map it to “prod_user” or “test_user” in the configuration)
• ability to have configurations that map database/collection keys to different names for different environments, like Test, QA, and Production

To explain the approach I have gone with, I won’t waste any more time and just show some code examples.

Read More

MongoDB is coming to NYC in May. Andre deCavaignac and I will be speaking at the conference about MongoDB in C#/.Net. We have a cool demo to show everyone and I’ll be posting some of the source here after the conference.

Click here to register for the conference

Read More

One of the best things about MongoDB is the lack of an enforced schema for collections. This flexibility gives developers a lot of power in how they work with their data. Embedding records and arrays inside other records allows both a complexity and simplicity of data organization that RDBMSs can only dream of! All of that being said, working with these records in a language like Java and on large diverse teams of people who don’t want to open the database and inspect the records to see what values and sub-records are available, means that you will always spend time wrapping these records in a strong-typed class. Wrapping up loose data into classes that can both access and create that data sounds just like another project I’ve used recently. If you haven’t heard of Google’s Protocol Buffers, you might want to acquaint yourself.

Since I’ve enjoyed working with Protocol Buffers so much, I thought I could mimic their functionality and ease of use with MongoDB. This would also integrate beautifully with the GuiceyMongo project that I released a month or two ago.

Read More

What is Amazon SQS?

Amazon SQS (Simple Queue Service) is a reliable message queuing service hosted in the Amazon cloud. This service is ideal for sending messages between servers that need to acknowledge that processing has been completed. When a message is popped from the queue, it is not deleted, but marked with the client who has made the request. The client is then responsible for telling SQS to delete the message from the queue. If the client does not delete a message it has popped within a certain time frame, the client loses ownership of the message and it is made available for other clients.

How am I using it?

One of the systems I am using SQS for is a distributed email delivery service (using SMTP). Since there is not an asynchronous SMTP client for Java (that I know of), I am using JavaMail to deliver messages. Sending messages with JavaMail is pretty slow and can take a number of seconds per message, with a thread being consumed for each message sent. In order to send many many messages in parallel I decided to queue up the outoging messages and spin up many instances of the SMTP application. This approach is dead simple and scales wonderfully without needing to implement an asynchronous SMTP client of my own.

Read More

I entered my post on Why (and How) I Replaced Amazon SQS with MongoDB into the MongoDB Blog Contest and won!

As part of the grand prize, I went out to OSCON in Portland, OR and had a great time! Kristina Chodorow’s new book on MongoDB was on display and I spoke to lots of people who are interested in what MongoDB is and what it can do for them. I also went to a lot of talks, the most interesting of which (in my opinion) were about node.js and the analytics done at Twitter using Hadoop and Pig.

Read More

Here at Signpost, we believe in track everything. We keep every request, every event, every log, every exception. We use this data for user tracking, analytics and business intelligence, and code health / bug triage.

Original System

We started out with MySQL. Requests were logged into a request table, except we handled the 1-Many relationship of request parameters in a single serialized column. Events were tracked in two tables: event_tracking and event_tracking_property, where there are many properties per event. Logs and exceptions were sent to log files, where they were rotated and hardly looked at for eternity. We ran with this setup in production for the first few months of Signpost’s lifetime and it worked out just fine.

Read More

Last week I gave a talk about the way we do analytics and tracking at signpost.com. The content was a follow-on to my post about how we moved the tracking system from MySQL to MongoDB. I hope the talk was informative and invite any questions that people may have.

Here are the slides:

Read More