Performance Zone is brought to you in partnership with:

Ayende Rahien is working for Hibernating Rhinos LTD, a Israeli based company producing developer productivity tools for OLTP applications such as NHibernate Profiler (nhprof.com), Linq to SQL Profiler(l2sprof.com), Entity Framework Profiler (efprof.com) and more. Ayende is a DZone MVB and is not an employee of DZone and has posted 434 posts at DZone. You can read more from them at their website. View Full User Profile

Optimizations gone wild, O(N!) memory leaks

12.18.2012
| 1024 views |
  • submit to reddit

So, after doing so much work on the indexing optimization, it turned out that we had a bug. I assume that you remember this optimization, right?

image

In which we were able to pre fetch data from the disk and not have to wait for data at all. This all worked beautifully when running on data sets that included simple indexes. But the moment we had map/reduce indexes, something bad happened. That something bad was that we kept missing the batch that we loaded (this relates to how we load & find the appropriate batches).

We do all the lookups by etag, and map/reduce add gaps in the etags. Which meant that we kept missing the etag, and had to start loading things up again. And because whenever we load something we also start loading the next batch…

Here is what the memory looked like:

image

Yup, for every batch we loaded the next 5 batches, for a total of O(N!) items in memory for everything.

Now, we had some cleanup routines, but we did NOT expect to have that much, so we would recover, eventually, but usually not before we consumed all the memory.

Oops!

Published at DZone with permission of Ayende Rahien, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)