NoSQL Zone is brought to you in partnership with:

Kristina Chodorow is a core contributor to MongoDB. She has written several O'Reilly books (MongoDB: The Definitive Guide, Scaling MongoDB, and 50 Tips and Tricks for MongoDB Developers) and has given talks at conferences around the world, including OSCON, FOSDEM, Latinoware, TEK·X, and YAPC. Her Twitter handle is @kchodorow. Kristina is a DZone MVB and is not an employee of DZone and has posted 52 posts at DZone. You can read more from them at their website. View Full User Profile

You Can Use Mongobridge to Simulate Network Partitions

  • submit to reddit

Note: mongobridge does not come with the MongoDB binaries, but you can build it by getting the source code and running scons mongobridge.

Let’s say we have a replica set with members (M1, M2, and M3) and we want to see what happens when M1 and M3 cannot reach each other (or any other sort of network partition). How do we do this?

We can’t shut down M3 because that would test something different. If we block incoming connections to M3, we end up blocking M2′s connections, too, which we don’t want.

We want M3 to only block M1′s connection, not M2′s. This is where mongobridge comes in.

A replica set with three members.

mongobridge allows you to accept connections on one port and send them on to a MongoDB process listening on another port. So, for the M1↔M3 connection, we can set up a bridge. This bridge, like most bridges, needs to go two directions (M1→M3 and M3→M1), so we actually need two instances of mongobridge.

So, we’ll make the “onramps” for our bridge M1::10013 (for the M1→M3 connection) and M3:10031 (for the M3→M1 connection). To set this up, we run:

$ ssh M1
M1$ mongobridge --port 10013 --dest M3:27017
M1$ exit
$ ssh M3
M3$ mongobridge --port 10031 --dest M1:27017
M3$ exit

 his means, “Take all traffic heading to M1:10013 and send it to M3:27017. Take all traffic heading to M3:10031 and send it to M1:27017.

However, M1:27017 doesn’t want to send its traffic to M1:10013, its configuration says to send M3-bound traffic to M3:27017. Same with M3:27017, it doesn’t want to send its traffic to M3:10031, it wants to send it to M1:27017. And we can’t reconfigure the set so that different members have different configurations… officially.

Unofficially, we can change each config to anything we want.

Warning: never, ever do what I’m about to do in production, this is just for trying out funky networking tricks.

Shut down M1 and M3, and start each back up without the --replSet option on a different port. This way it won’t connect to the rest of the replica set and you can edit the local.system.replset configurations to point at the bridges, instead of the other members.

> // supposing we started them on port 27018 instead of 27017...
> db = (new Mongo("M1:27018")).getDB("local")
> db.system.replset.update({}, {$set : {"" : "M1:10013"}})
> db = (new Mongo("M3:27018")).getDB("local")
> db.system.replset.update({}, {$set : {"" : "M3:10031"}})

 Now, restart M1 and M3 with their proper ports and --replSet. They will connect to each other through mongobridge and our replica set will be ready to use.

To simulate a network outage, kill the mongobridges between two servers.

Suppose M1 is the primary and you have knocked out the network between M1 and M3. If you do a write on M1, M2 will sync to M1 and then M3 will sync to M2, so you can see that the write will still make it to M3.

And that’s the dramatic payoff.


Published at DZone with permission of Kristina Chodorow, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Goel Yatendra replied on Thu, 2012/03/15 - 3:51pm

does mongobridge do something ssh port-forwarding doesn't, or would that be an option as well?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.