NoSQL Zone is brought to you in partnership with:

Eric is the Editorial Manager at DZone, Inc. Feel free to contact him at egenesky@dzone.com Eric has posted 804 posts at DZone. You can read more from them at their website. View Full User Profile

Scaling the Hyperbola of Evolution with MongoDB

05.14.2013
| 1126 views |
  • submit to reddit

Curator's Note: The content of this article was written by T. Dampier over at the MongoLab blog. 
Palace Hotel lobby, c. 1930

Palace Hotel, c. 1930

You know, I attend a fair number of MongoDB events, and frankly I keep expecting them to get stale. But after being at MongoSF this past Friday, I’m happy to say it hasn’t happened yet. The growth and vigor of the Mongo ecosystem was everywhere apparent, and it has never been more encouraging. Our sincere thanks go out to the 10gen team for putting together another fabulous and informative event.

If you were there and managed to stop by MongoLab’s table in the exhibit hall of the super-elegant Palace Hotel, then thanks! It was nice to meet you and/or see you again! Hope you got as much out of the day as we did. If you didn’t — or if you’d just like my personal take on the whole thing — well, please read on.

Ecosystem predicts viability

Setting aside any of the relative merits of MongoDB as a database for just a moment, I have to say my top takeaway continues to be amazement at the size and enthusiasm of the community around MongoDB.

[I]f an organism or aggregate of organisms sets to work with a focus on its own survival and thinks that that is the way to select its adaptive moves, its “progress” ends up with a destroyed environment. If the organism ends up destroying its environment, it has in fact destroyed itself. … The unit of survival is a flexible organism-in-its-environment.” [1]

History is littered with the scarcely recognizable fossils of good ideas, clever inventions, and even superior products that might have flourished save for one thing: adoption. The modern proving ground for technological species looks less and less like the traditional “marketplace” with pockets of asymmetric information and discrete “deals.”  Today, the landscape has evolved to include open-source transparency, synergies of technologies and ideas, and a globally interconnected (and often, informed) fabric of opinion. A vigorous ongoing conversation (and overlap!) among diversified populations of users and developers is now the surest predictor, I believe, of long-term survival.

So, more than the database technology (which is impressive) or the well-capitalized company devoted to developing it (which is formidable), it is the people and the strength of this community that inspire my confidence that MongoDB will continue to thrive, improving and growing in popularity as a viable or even preëminent database for an ever increasing number of applications.

MongoDB: the Next Generation

Eliot Horowitz, 10gen CTO & Co-founder, kicked things off on a strong note, clearly articulating his focus for the immediate future of MongoDB. In my opinion, these are exactly the right priorities for taking the platform to the next level:

  • Maturity
  • Innovation
  • Operations

If you peer into its internals today, you’ll see the evolutionary legacy of MongoDB: steadily improving and expanding functionality, accreted around a core of pragmatic and sometimes downright scrappy engineering — just what you might expect from a small, clever team with a product rapidly establishing itself in the marketplace. But many of the expedients that accelerate a large piece of software in the short term can eventually bog down development and become obstacles to its further progress. You want a larger team to be able to add and maintain a growing number of features, without commensurate increases in code complexity. At some point, once experience has shown where the grain boundaries lie, there comes a time to refactor (not reinvent!) the core, teasing out clear and minimal abstraction contracts that the new implementations of existing and future features can target.

This engineering story arc is not lost on Eliot. Cleaner factoring, he explains, will be a a key enabler to efficiently deliver capabilities that MongoDB has needed for a long time, to make it a more “mature,” fully-featured general purpose database. It will also form the groundwork for innovating and building on the strengths of MongoDB as a data substrate for modern applications. Specific examples Eliot mentioned included:

  • non-constant query constraints — e.g., find all documents where the values of fields “a” and “b” are equal.
  • inline aggregation operations — e.g., update each document to set its “total” field to the sum of the “dollarAmt” field of each element of its “lineItems” array.
  • index intersections — e.g., optimize a query like {a: 3, b: 6} by dynamically combining an index on “a” with an index on “b” to yield performance comparable to what today would require an explicit compound index comprising both fields.

So that’s the broad story around Maturity and Innovation — right on. What about the third item: Operations? This of course refers to the realities of keeping a database running and available behind a production system of any kind. Happily, there is another three-item list here:

  • Monitoring
  • Backups
  • Management

Eliot spoke to 10gen’s efforts on each of these facets: MMS, which became available some 18 months ago; the remote backup service, which is in Limited Release now; and a suite of management tools to be announced later this year. Without a solid story around each of those items, any database deploymentcannot be considered production grade.

Of course, the topic of production-class Operations is near to our hearts: seamlessly handling these three facets for our customers is what MongoLab is all about!

You got your lagerstätten in my Burgess Shale!

Opabinia

Opabinia, c. 505,000,000 BC

Max Schireson, the 10gen CEO who claims to have beenborn the same year as the relational database, followed up with a pointedly evolutionary perspective on database technologies. He compared today’s landscape to the early part of the Cambrian Explosion, in which biodiversity increased by orders of magnitude in a small fraction of the total history of life on earth up to that point. Of course, the unstated implication was that hitherto more “established” databases (Oracle, MySQL) were the long-dominant single-celled organisms in this analogy, whereas MongoDB would be perhaps more like a sighted predator of some kind.

Schireson quoted some consumption figures from the top of the food chain (e.g., 3 of the top 10 global investment bank use MongoDB) and noted some recent shifts in environmental pressures (e.g., developer-driven decision making).  He also cited an amusing factoid: prior to this year’s report, the lastGartner Research update on databse technology came out in 2003. That’s right: a full ten years ago. Something new must be going on. (Can you guess what?)

In short, Schireson made it sound like a pretty exciting time to be in databases, with MongoDB figuring prominently on the changing landscape.

Okay, now back to your niche…

After this inspiring keynote, of course, there followed a full day of stimulating talks and sessions at all levels of the mongo-guru ladder — oceans of fresh, insightful, useful stuff.

My personal favorite was probably the session led by Charity Majors, who is responsible for the MongoDB servers at the heart of Parse.com. If you were lucky enough to catch her outstanding talk on the care and feeding of a grown-up mongo deployment, you’ll know that there’s a whole host of operational issues that you’d just rather not worry about — or at the very least, you’d very much like an experienced hand at the helm when you do. Why do I say her talk was outstanding? Because that stuff is our bread and butter. It’s what we do all day every day here at MongoLab: hook you up with the database of tomorrow, so you can use more of your energy to dominate YOUR product’s ecological niche today, and still get a good night’s sleep (assuming your species isn’t nocturnal).

There’s never been a richer ecosystem, or a better time to be a database consumer. And there are more reasons than ever today for your consumption preferences to be of the MongoDB phyla. Yummy! Why not try one right now?

T. Dampier, 2013-05-11

Notes

[1] Source: Gregory Bateson, “Form, Substance and Difference”, 19th Annual Alfred Korzybski Memorial Lecture, 9 Januiary 1970, Oceanic Institute, Hawaii. From the book Ecology and Consciousness, edited by Richard Grossinger, North Atlantic Books, 1978. p. 32.

Published at DZone with permission of its author, Eric Genesky. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)