Big Data/Analytics Zone is brought to you in partnership with:

John Cook is an applied mathematician working in Houston, Texas. His career has been a blend of research, software development, consulting, and management. John is a DZone MVB and is not an employee of DZone and has posted 171 posts at DZone. You can read more from them at their website. View Full User Profile

The Lindy Effect and Technology Survival Patterns

12.19.2012
| 3472 views |
  • submit to reddit

The longer a technology has been around, the longer it’s likely to stay around. This is a consequence of the Lindy effect. Nassim Taleb describes this effect in Antifragile but doesn’t provide much mathematical detail. Here I’ll fill in some detail.

Taleb, following Mandelbrot, says that the lifetimes of intellectual artifacts follow a power law distribution. So assume the survival time of a particular technology is a random variable X with a Pareto distribution. That is, X has a probability density of the form

f(t) = c/tc+1

for t ≥ 1 and for some c > 0. This is called a power law because the density is proportional to a power of t.

If c > 1, the expected value of X exists and equals c/(c-1). The conditional expectation of Xgiven that X has survived for at least time k is ck/(c-1). This says that the expected additional life X is ck/(c-1) – k = k/(c-1), and so the expected additional life of X is proportional to the amount of life seen so far. The proportionality constant 1/(c-1) depends on the power c that controls the thickness of the tails. The closer c is to 1, the longer the tail and the larger the proportionality constant. If c = 2, the proportionality constant is 1. That is, the expected additional life equals the life seen so far.

Note that this derivation computed E( X | X > k ), i.e. it only conditions on knowing that Xk. If you have additional information, such as evidence that a technology is in decline, then you need to condition on that information. But if all you know is that a technology has survived a certain amount of time, you can estimate that it will survive about that much longer.

This says that technologies have different survival patterns than people or atoms. The older a person is, the fewer expected years he has left. That is because human lifetimes follow thin-tailed distributions. Atomic decay follows a medium-tailed exponential distribution. The expected additional time to decay is independent of how long an atom has been around. But for technologies follow a thick-tailed distribution.

Another way to look at this is to say that human survival times have an increasing hazard function and atoms have a constant hazard function. The hazard function for a Pareto distribution is c/and so decreases with time.

Published at DZone with permission of John Cook, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Willie Wheeler replied on Thu, 2012/12/20 - 12:56pm

What does Taleb have to say about the underlying mechanism here (i.e., the one producing the power law distribution)?

Lon Miller replied on Thu, 2012/12/20 - 1:19pm

I'm sure the standalone MP3 player, the VCR, the land line and Hostess Twinkies will be relieved to read this article.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.