DevOps Zone is brought to you in partnership with:

Reimagining the way work is done through big data, analytics, and event processing, Chris is the cofounder of Successful Workplace. He believes there’s no end to what we can change and improve. Chris is a marketing executive and flew for the US Navy before finding a home in technology 17 years ago. An avid outdoorsman, Chris is also passionate about technology and innovation and speaks frequently about creating great business outcomes at industry events. As well as being a contributor for The TIBCO Blog, Chris contributes to the Harvard Business Review, Venture Beat, Forbes, and the PEX Network. Christopher is a DZone MVB and is not an employee of DZone and has posted 294 posts at DZone. You can read more from them at their website. View Full User Profile

Webscale Means Time-Based Architectures

04.01.2013
| 2920 views |
  • submit to reddit

In an interesting piece yesterday, GigaOM reported that Netflix has an architecture built around timelines. This struck home for a guy who spends a great deal of time talking to people skeptical about the need for zero latency, real-time systems. The truth is that some things have to go at the highest speeds and others don’t. The problem is in creating systems that use resources wisely to get it right in each case.

Immediate response

If it was only about the hype, you’d think Hadoop is the answer to processing web data, but it isn’t that simple. Hadoop is still (despite even newer hype) a batch processing concept that allows data to “get stale and applications probably don’t include the newest user input.” Before that sounds too negative, there is data that can afford to be stale without a business penalty and can be processed and moved in an architecture that is offline and more traditional.

For real-time, Netflix needs to use the absolute latest inputs and has a solution:

Netflix uses online processing for receiving information from users in real time and serving up responses right away, such as looking at a new rating or some other customer action to change the set of movies shown to the customer. Real-time processing works best when algorithms are relatively simple and when data is on the smaller side. The data feeding in to computations must also be available right away.

This takes very high levels of integration and applications reading in-memory data at lightning speed. This means a very particular, state-of-the-art architecture that Gartner’s Massimo Pezzini calls, “The Next Generation Architecture: In-memory computing.

Middle ground

Between batch and real-time exists a middle ground that Netflix calls ‘nearline’. This is also a reality for business today and involves NoSQL and SQL databases that are more complex and less time sensitive than real-time needs.

The Netflix story is available on their blog and is a great example of how companies that are at the front of the ‘time wars’ are managing their requirements in time-based ways rather than monolithic, expensive and risky models. In a world of many shades of gray, this just makes sense.

Netflix architecture

Photo credit: http://ericaannsipes.blogspot.com/2012/08/balancing-tortoise-with-hare-while.html

Published at DZone with permission of Christopher Taylor, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)