Candy-gram for Mongo
- Bart: I better go check out this Mongo character.
- [Bart reaches for his gun]
- Jim: Oh no, don't do that, don't do that. If you shoot him, you'll just make him mad.
So I've got this Rails application that uses a MongoDB database. There have been occasional problems in the past, and a few times I've gotten frustrated with Mongo's general bone-headedness, but never anything serious. Then today, I had a face-off that firmly cemented my negative opinions of MongoDB.
It all started when something went wrong with the application. Suddenly, Passenger reports that Rails is unable to connect to Mongo. If it were any other Unix service, a simple service mongod start might do the trick. So we start there:
# service mongod start
Starting mongod: all output going to: /var/log/mongo/mongod.log
forked process: 19583
[ OK ]Peachy, right? Well, when mongod says "OK", what it really means is "I may have started fine, or I may have exploded. I'm not saying." When you check, you might be surprised:
# service mongod status mongod dead but subsys locked
That's no good. It seems to be in some kind of limbo-state, not started, but not really stopped either. So before doing further debugging, let's stop the service:
# service mongod stop Stopping mongod: [FAILED]
What? It failed to stop too? Now what? So tell us Mongo, what's your status now that you've pretented to successfully start, then failed to successfully stop?
# service mongod status mongod is stopped
At this point, it's hardly a surprise. Well OK, so we check the log file and it's complaining that it needs to be repaired or something. So let's try this:
# mongod --repair MongoDB starting : pid=4507 port=27017 dbpath=/data/db/ 64-bit .... exception in initAndListen std::exception: dbpath (/data/db/) does not exist, terminating
That doesn't seem right. It's looking for databases where they don't belong. My databases are in /var/lib/mongo, not in /data/db:
# ls -l /var/lib/mongo drwxr-xr-x 2 mongod mongod 4096 Jan 31 15:50 db_development -rw------- 1 mongod mongod 67108864 Jan 31 15:50 db_development.0 -rw------- 1 mongod mongod 134217728 Jan 31 15:50 db_development.1 -rw------- 1 mongod mongod 16777216 Jan 31 15:50 db_development.ns
And I'm pretty sure that's already configured:
# grep dbpath /etc/mongod.conf dbpath=/var/lib/mongo
Yup. Fine, so I guess mongod doesn't know where to find its own configuration file. Let's tell it, shall we?
# mongod --config /etc/mongod.conf --repair all output going to: /var/log/mongo/mongod.log
Alrighty, so everything should be fine now, right?
# service mongod start
Starting mongod: all output going to: /var/log/mongo/mongod.log
forked process: 27310
[ OK ]But we know better than to trust this particular "OK" by now, right? Sure enough:
# service mongod status mongod dead but subsys locked
What now? Oh yeah, the log says:
couldn't open /var/lib/mongo/db_development.ns errno:13 Permission denied
This makes no sense. The file was owned by mongod, right? A simple --repair shouldn't change that, right? I mean, if --repair instead broke things that were already working, it should probably be called --break, right?
# ls -l /var/lib/mongo drwxr-xr-x 2 mongod mongod 4096 Jan 31 15:50 db_development -rw------- 1 root root 67108864 Jan 31 15:50 db_development.0 -rw------- 1 root root 134217728 Jan 31 15:50 db_development.1 -rw------- 1 root root 16777216 Jan 31 15:50 db_development.ns
Sigh. So I gotta manually fix the stuff that --repair broke:
# chown -R mongod:mongod /var/lib/mongo
And then stop, start, and once again verify the status of mongod, to discover that it's finally working properly now.
What's wrong with this picture? The mongod service flagrantly violates several of the unspoken (and some of the spoken) rules about software, not least of which is the fact that errors should not pass silently. Sure, they go to the log file, but if your service failed to start, then it shouldn't be saying "OK" when you go to start it. Similarly, it shouldn't mislead you by saying "FAILED" when in fact there was some measure of success. What about not being able to find your own configuration file? The mongod service itself seems happy to look in /etc/mongod.conf, but when you run mongod from the command-line, somehow it forgets where to find it. The fact that the --repair command can silently muck up your permissions is fairly disturbing as well; there's no good reason that these files should have their ownership changed.
Maybe it's just an old version (this is CentOS, after all); maybe these problems are fixed in newer versions of MongoDB. I'm not holding my breath, though. Sorry Mongo--I don't want you in my town anymore. Consider this to be your exploding candy-gram.
