Candy-gram for Mongo

  • Bart: I better go check out this Mongo character.
  • [Bart reaches for his gun]
  • Jim: Oh no, don't do that, don't do that. If you shoot him, you'll just make him mad.

So I've got this Rails application that uses a MongoDB database. There have been occasional problems in the past, and a few times I've gotten frustrated with Mongo's general bone-headedness, but never anything serious. Then today, I had a face-off that firmly cemented my negative opinions of MongoDB.

It all started when something went wrong with the application. Suddenly, Passenger reports that Rails is unable to connect to Mongo. If it were any other Unix service, a simple service mongod start might do the trick. So we start there:

# service mongod start
Starting mongod: all output going to: /var/log/mongo/mongod.log
forked process: 19583
        [  OK  ]

Peachy, right? Well, when mongod says "OK", what it really means is "I may have started fine, or I may have exploded. I'm not saying." When you check, you might be surprised:

# service mongod status
mongod dead but subsys locked

That's no good. It seems to be in some kind of limbo-state, not started, but not really stopped either. So before doing further debugging, let's stop the service:

# service mongod stop
Stopping mongod:      [FAILED]

What? It failed to stop too? Now what? So tell us Mongo, what's your status now that you've pretented to successfully start, then failed to successfully stop?

# service mongod status
mongod is stopped

At this point, it's hardly a surprise. Well OK, so we check the log file and it's complaining that it needs to be repaired or something. So let's try this:

# mongod --repair
MongoDB starting : pid=4507 port=27017 dbpath=/data/db/ 64-bit
....
exception in initAndListen std::exception: dbpath (/data/db/) does not exist, terminating

That doesn't seem right. It's looking for databases where they don't belong. My databases are in /var/lib/mongo, not in /data/db:

# ls -l /var/lib/mongo
drwxr-xr-x 2 mongod mongod      4096 Jan 31 15:50 db_development
-rw------- 1 mongod mongod  67108864 Jan 31 15:50 db_development.0
-rw------- 1 mongod mongod 134217728 Jan 31 15:50 db_development.1
-rw------- 1 mongod mongod  16777216 Jan 31 15:50 db_development.ns

And I'm pretty sure that's already configured:

# grep dbpath /etc/mongod.conf
dbpath=/var/lib/mongo

Yup. Fine, so I guess mongod doesn't know where to find its own configuration file. Let's tell it, shall we?

# mongod --config /etc/mongod.conf --repair
all output going to: /var/log/mongo/mongod.log

Alrighty, so everything should be fine now, right?

# service mongod start
Starting mongod: all output going to: /var/log/mongo/mongod.log
forked process: 27310
        [  OK  ]

But we know better than to trust this particular "OK" by now, right? Sure enough:

# service mongod status
mongod dead but subsys locked

What now? Oh yeah, the log says:

couldn't open /var/lib/mongo/db_development.ns errno:13 Permission denied

This makes no sense. The file was owned by mongod, right? A simple --repair shouldn't change that, right? I mean, if --repair instead broke things that were already working, it should probably be called --break, right?

# ls -l /var/lib/mongo
drwxr-xr-x 2 mongod mongod       4096 Jan 31 15:50 db_development
-rw------- 1 root   root     67108864 Jan 31 15:50 db_development.0
-rw------- 1 root   root    134217728 Jan 31 15:50 db_development.1
-rw------- 1 root   root     16777216 Jan 31 15:50 db_development.ns

Sigh. So I gotta manually fix the stuff that --repair broke:

# chown -R mongod:mongod /var/lib/mongo

And then stop, start, and once again verify the status of mongod, to discover that it's finally working properly now.

What's wrong with this picture? The mongod service flagrantly violates several of the unspoken (and some of the spoken) rules about software, not least of which is the fact that errors should not pass silently. Sure, they go to the log file, but if your service failed to start, then it shouldn't be saying "OK" when you go to start it. Similarly, it shouldn't mislead you by saying "FAILED" when in fact there was some measure of success. What about not being able to find your own configuration file? The mongod service itself seems happy to look in /etc/mongod.conf, but when you run mongod from the command-line, somehow it forgets where to find it. The fact that the --repair command can silently muck up your permissions is fairly disturbing as well; there's no good reason that these files should have their ownership changed.

Maybe it's just an old version (this is CentOS, after all); maybe these problems are fixed in newer versions of MongoDB. I'm not holding my breath, though. Sorry Mongo--I don't want you in my town anymore. Consider this to be your exploding candy-gram.

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Each email address will be obfuscated in a human readable fashion or (if JavaScript is enabled) replaced with a spamproof clickable link.

More information about formatting options

CAPTCHA
This question tests whether you are a human visitor, to prevent automated spam submissions.
Question text provided by textcaptcha.com