Build a Car. Don’t Make a Faster Horse.
“If I had asked people what they wanted, they would have said faster horses.”
― Henry Ford
Henry Ford’s famous line is often used as a guideline for innovation, but I think that the majority of tech start-ups misinterpret the quote. By and large I’m not talking about “cool” products like games or social networks (before they become useful), but websites and applications that serve a direct business need. When you understand what users want, you can examine why they want it, and then figure out how to give them what they need. Too many start-ups fall into the trap of giving users what they want without really examining why they want it, completely ignoring what they need.
MongoNYC 2011: My Talk on Tracking and Analytics at Signpost
Last week I gave a talk about the way we do analytics and tracking at signpost.com. The content was a follow-on to my post about how we moved the tracking system from MySQL to MongoDB. I hope the talk was informative and invite any questions that people may have.
Here are the slides:
Signpost Tracking & Analytics: MySQL -> MongoDB
Here at Signpost, we believe in track everything. We keep every request, every event, every log, every exception. We use this data for user tracking, analytics and business intelligence, and code health / bug triage.
Original System
We started out with MySQL. Requests were logged into a request table, except we handled the 1-Many relationship of request parameters in a single serialized column. Events were tracked in two tables: event_tracking and event_tracking_property, where there are many properties per event. Logs and exceptions were sent to log files, where they were rotated and hardly looked at for eternity. We ran with this setup in production for the first few months of Signpost’s lifetime and it worked out just fine.
MongoDB Blog Contest – I Won!
I entered my post on Why (and How) I Replaced Amazon SQS with MongoDB into the MongoDB Blog Contest and won!
As part of the grand prize, I went out to OSCON in Portland, OR and had a great time! Kristina Chodorow‘s new book on MongoDB was on display and I spoke to lots of people who are interested in what MongoDB is and what it can do for them. I also went to a lot of talks, the most interesting of which (in my opinion) were about node.js and the analytics done at Twitter using Hadoop and Pig.
Why (and How) I Replaced Amazon SQS with MongoDB
What is Amazon SQS?
Amazon SQS (Simple Queue Service) is a reliable message queuing service hosted in the Amazon cloud. This service is ideal for sending messages between servers that need to acknowledge that processing has been completed. When a message is popped from the queue, it is not deleted, but marked with the client who has made the request. The client is then responsible for telling SQS to delete the message from the queue. If the client does not delete a message it has popped within a certain time frame, the client loses ownership of the message and it is made available for other clients.
How am I using it?
One of the systems I am using SQS for is a distributed email delivery service (using SMTP). Since there is not an asynchronous SMTP client for Java (that I know of), I am using JavaMail to deliver messages. Sending messages with JavaMail is pretty slow and can take a number of seconds per message, with a thread being consumed for each message sent. In order to send many many messages in parallel I decided to queue up the outoging messages and spin up many instances of the SMTP application. This approach is dead simple and scales wonderfully without needing to implement an asynchronous SMTP client of my own.