How Clay came to discover the theory of Disruptive Innovation.
More than I ever imagined, the term “disruptive innovation” is invoked by tens of thousands of people around the world, including many people who do not yet understand what the theory of disruptive innovation actually means. They use the phrase to mean radical, different, new, breakthrough, and much more – all in the hope that the words themselves will justify to management or investors whatever they wanted to do in the first place. The misuse of “disruption” isn’t the first time that words have been given specific meanings by scholars, only to be broadly and badly used by people who don’t quite understand them. Think of the appropriation of the term “evolution” in the nineteenth century by Herbert Spencer and the other acolytes of social Darwinism.
Whatever the misinterpretations, the theory of disruption is widely recognized today and has been used as a guide to action by many companies, both the disruptors and the disrupted. It has also been tested for its accuracy and predictability, and has passed these tests with flying colors.[i] It has also been applied and used insightfully outside the world of business—in healthcare and education, for instance. We therefore thought it could serve as a useful case study in theory building.
In this entry and the following one, we have two objectives. We hope, first, that it can show what disruption is and what it isn’t – to put in the public eye what we have learned about the use and misuse of the theory. And second, we hope that people might learn something from our successes and failures in developing the theory. We want to create a model that generalizes from our experience, a simple model that others can follow as they work to develop their own theories.
I was not a typical business school doctoral student when I embarked on this research. I was 38. My wife, Christine, and I had four children—a fifth would follow later—but we had little else. After a five-year stint with the Boston Consulting Group, I had founded a high-tech materials company with several MIT professors in 1984 that today is called CPS Technologies. The professors were the technology brains, and I was the business guy. We raised the needed money. But the financial markets collapsed in October, 1987, three months after we went public. And by 1989 it was clear that creating new markets for these new materials was going to take a lot more time than we imagined. So the board decided that we probably would not be able to raise more money, and that the company needed to cut its fixed costs dramatically.[ii] They fired me on Thursday, September 14, 1989.
Fortunately I had sensed that this had to happen. I had been thinking of becoming a teacher and had conversations with several people at the Harvard Business School to see if that would be a possible path for me. They said it was plausible, but that I should get a doctorate first. Over that weekend, Professors Jay Light, Kim Clark, and Joe Bower admitted me into their DBA program, without asking me to take the GMAT or any other test and without even asking me to fill out an application. They changed my life in a profound way. That was the good news. The bad news was that the doctoral seminars were already in their second week of the semester. So Christine and I made a quick decision. That Monday, instead of driving to Boston’s western suburbs in my business suit, I wore casual student clothing and biked east to Harvard.
I had a general question as I came into the program: why do companies that are highly regarded and generally well managed fail to beat back new competitors? This question came directly from our launch of CPS Technologies. We were a venture-backed entrant in a massive materials industry. The established providers of sophisticated materials included Alcoa, Cabot, Hoechst, General Electric, DuPont, Dow, Corning, and many more. And yet our little company became the most successful in the class of products known as high-tech ceramics. Collectively, our competitors spent several billions of dollars trying to succeed in this market. We spent a tiny fraction of that. Moreover, I knew most of the corporate executives who were trying to grow a new business in high-tech ceramics. They were experienced, capable men and women – especially when compared to us. So why did we win, instead of them?
The first two years of Harvard’s DBA program involved seminars on technology, operations, product development, marketing, business history, organizational design, teams, and many other topics. Each required an end-of-semester paper. Because of our financial constraints—I couldn’t afford a long, drawn-out period of writing a dissertation—Christine and I decided that I ought to use these fields like a lens, and to write a paper about my general question from the perspective of each seminar’s academic literature. Hence, after two years, I had examined my problem from every perspective imaginable. Each seminar honed my question, and this set of seminar papers essentially constituted my thesis proposal.
When I sought Kim Clark’s approval for my thesis topic, he responded (as best I remember), “Rebecca Henderson solved this question. She just finished her thesis – and you should read it.” I did. And I was impressed. Henderson, who subsequently joined the Harvard Business School faculty, had studied the semiconductor alignment equipment industry….She found….[JC1] So I reframed my question as this: “Does the mechanism that Henderson found for why well-run companies stumble over new technologies—and why they often fail in the semiconductor alignment equipment industry—also work in motor controls?” Motor controls was an industry I knew quite well because I had studied it at the Boston Consulting Group. I sensed that established competitors making electromechanical motor controls had been decimated by entrants offering electronic controls.
Fortunately, I had met Professor Richard Rosenbloom, who was an historian of technology. After I took his seminar on the history of technological change, I told him that I was considering a thesis topic about why established companies have a hard time adopting new technologies. AndI proposed studying the motor controls industry. Rosenbloom replied, “You can’t learn much from studying one event. You need to study an industry where there were waves of new technologies that killed the prior generations, like Rebecca’s. I don’t know much about it. But you should look at the disk drive industry.”
I knew nothing about the disk drive industry, except that I had heard there was a disk drive inside of my computer, functionally located behind the “C>” prompt on my DOS operating system. So I dived into the stacks of Baker Library at HBS. I read literally every article about the disk drive industry, and about every company and every technology in the industry, in every monthly issue of Electronic Business magazine between 1976 and 1992. In a sense, I dived into the dumpster—the messy world of people, companies, technology, and so on that made up the disk drive business at that time. I then built a data set from all of these articles about all of the companies in the disk drives. And I set out to see whether Henderson’s thesis applied to this industry.
To be sure, I didn’t stop with Electronic Business. I had noticed that every article that included data referred to a publication called Disk/Trend Report, in Mountain View, California. Disk/Trend Report seemed to be quite comprehensive, so I sent a letter to the publisher, a man named Jim Porter. I asked if I might come to his office to study his back issues. The very next week, instead of responding directly to my letter, he sent me copies of each yearly report from the first issue in 1976 all the way to the most recent report, for 1990.
My computer-savvy children and I then created a database on Excel from Disk/Trend Report. The result was not a sample. It was a complete census of the disk drive industry – every company that was organized to design and build disk drives; every product that was announced, whether or not any were actually sold; and every component used in every product. I am very grateful to Jim Porter for forcing my hand by giving me all of the data on paper and not in electronic format. I had to lift and weigh every piece of it. It forced me to understand the technology, past, present, and future. It forced me to know the phenomena that the data represented.
Next, I categorized this data in as many ways as I could, looking for interesting and unusual patterns. There were 116 disk drive companies in the database. I categorized them by source of capital, because some of them were launched by large companies like Control Data while others were founded by venture capitalists or others. I arrayed their products in five technologically defined generations. I also categorized them by type of disk and head technology, type of interface with the computer, on and on. Finally, I categorized them based upon outcome. Many never developed a viable product. Others became highly successful. Some were successful only in one generation; others were successful in one generation, failed in the next, but rebounded as a leading firm in the following generation.
For each category, I used regression equations to match the characteristics of the categories against the outcomes of interest – various versions of revenues, market share by category, and ultimately by success. I was then able to express the relationships between the categories’ characteristics and the outcomes of interest in probabilistic terms. Most of the results of my early papers were expressed as probabilities: for instance, there was a 35% likelihood that companies following what I would later call a disruptive strategy would become successful, compared to a 6% probability that companies following a sustaining strategy would do so.
My efforts to categorize the companies helped me see what critical events had occurred in the industry and amongst its companies over time. My reading of all the articles about disk drive companies in Electronic Business magazine helped me see which managers and technological leaders played critical roles in their companies. I chose six companies to study in depth, selected because they represented different categories of companies. I then interviewed 54 people who had worked in 20 companies. With this abundance of data I wrote a history of the disk drive industry as carefully and as thoroughly as I could. The history was my semester paper for the doctoral seminar in business history.[iii]
Nobody told me to create a database that was a census of the industry. But for whatever reason, I did it. As an aside, and in retrospect, simply reading what the journalists wrote for Electronic Business was akin to dumpster-diving. A lot of stuff had happened in the disk drive industry – so much that I could not judge what was and was not important. The historical dumpster was filled with heads, disks and drives, along with the people who made them and were made by them – brilliant engineers, salespeople who were fired, investors who got rich through no merit of their own, others who wound up poor through no fault of their own; CEOs who should never had become CEOs; people who were great CEOs; and so on.
Somehow I came to see that data isn’t made in heaven. Every piece of it has been created and recorded in some way by a human. So it is necessarily imperfect. The phenomena that the data represents are almost always too complicated to be wholly captured by data, so the data at best can tell only a portion of the reality. On the other hand, data offers something to a researcher that the phenomena themselves cannot. For example, we can’t usually do experiments with historical phenomena. History is what it is. But if we have data about the past, we can construct experiments by changing the data to represent a different set of historical actions – to see whether and how the outcome would have changed as a result.[iv] The data that wwas recorded, and the data that was not recorded, are both important to us as we try to understand the present and to see clearly what the future has in store.
The hard work of organizing the dumpster’s phenomena into data provided me a ladder of sorts. It enabled me to crawl out of the muck and build models of the industry. VisiCalc, Lotus 1-2-3, and ultimately Excel allowed me to generalize what had happened in the disk drive industry. In retrospect, I can see that had I stayed in the dumpster with all of the data I could not have comprehended the whole. The engineers and managers I interviewed, themselves completely surrounded by thickets of phenomena, could not understand the whole either. Without a model or a theory, both data and individual experiences can take you only so far.
I am grateful that I spent so much time building simple statistical models of the disk drive industry, because it helped me frame the next set of questions for the people I would meet in the next dive into the dumpster.
It happened in August 1991. I had spent the day in Scotts Valley, south of Silicon Valley, where Seagate was located (the address: One Disk Drive). I had lunch with a former sales manager for Seagate. He had recently been fired. (I was learning what every business journalist knows, that former employees are fountains of insight compared with employees who are still at a company.) I asked the man why he had been let go. He responded, “We had announced our first 3.5-inch drive with 10 megabyte capacity. I wanted to sell it, but I was the only one. Our customer (IBM) needed a 40 megabyte drive that would fit into a five-and-a-quarter-inch slot. Our drive was just too small on both counts. Naturally, our salespeople didn’t want to sell it. So they fired me because I couldn’t get my people to do what they didn’t want to do. But I was right – and the upper-ups just don’t get it. If Moses was sales manager of Seagate, he might be able to lead us through the Red Sea on dry land. But he couldn’t do any better than I could in selling 3.5-inch drives. Don’t worry about me, though. I already have a job at Conner Peripherals.”
That evening I was crossing First Street in San Jose. To save money I was spending my nights in Motel 6, and I had just enjoyed dinner at McDonald’s across the street. Right there, in the middle of First Street, it became perfectly clear. There is one trajectory of improvement in disk drives that companies blaze as they try every day to make and sell better products. And there is another trajectory of improvement that customers can use. The first trajectory is steeper than the second – and this means that Conner Peripherals’ “too small” drive, a 3.5-inch drive with 10 megabytes of storage, will in a few years be exactly what IBM needs. It was the beginning of a theory, and the theory enabled me to see the future of the industry with perfect clarity.
These trajectories are what we call “constructs,” a term we’ll discuss in later entries. In my statistical history, the data about disk drives amounted to a static view. It compared one category of products or makers with another at a point in time. If one tries to depict how things happened over time, a statistical history can only compare different points in history. The trajectories, however, conveyed the dynamics in the industry – the capacity of drives provided and the capacity of computers to use storage. The trajectories were not visible objects, but it was as if they were. They helped me visualize how technological progress and market needs interacted.
I observed that established companies competing on the sustaining trajectory of this model are quite capable of succeeding in innovations that are ahead of them on that trajectory. When I mapped my data to the diagrams, I saw that it was rare for the leading firms on that trajectory to be topped by entrant companies. Almost always, existing winners seem to win battles of sustaining innovations.
But I also observed that when a different kind of innovation came into a market below the leaders, defining a new trajectory, the incumbents on the sustaining trajectory found it nearly impossible to succeed, even when the innovation was relatively simple. In other words, it seemed easy to work upmarket and hard to work down. I called this kind of innovation disruptive innovation.
Why could these firms not respond forcefully to disruption? The answer is that it is more profitable for incumbents to go upmarket than to go down. To increase their profits, these firms typically need innovations that they can sell at higher prices to their best customers. But their customers cannot use the disruptive innovation, because it is less than what they need. Although the leaders could go down-market, they have no incentive to do so. That is why entrants producing disruptive products can prosper and grow when they start from a foothold at the bottom of a market, with simple customers who have simple needs. The incumbents are motivated to flee rather than to fight.
As I wrote in the introduction to The Innovator’s Dilemma, “Precisely because these firms listened to their customers, invested aggressively in new technologies that would provide their customers more and better products of the sort that they wanted … they lost their positions of leadership.” [v] I showed, in retrospect, that this same disruptive effect has toppled many of the most powerful incumbents in industries as diverse as mechanical excavation, steel, retaining, computers, motor switches, and insulin for diabetes. Although it took an entire book to convey the theory to those who were interested, this was the theory as it existed in 1997.
Subsequently, of course, we have learned much more about the theory, and about the disruptive effect in general. We will show how this came about in the following entry. To set the stage, we need to address a central question for researchers in general.
i. For instance, Thomas Thurston, formerly of Intel and currently CEO of Growth Science, has been studying disruption’s real-world accuracy. He has found from his sample of 3,400 predictions that the theories of disruption were right about firm successes nearly two out of three times—more than twice as good as the average prediction—and that predictions of failures were 88% right (both at p ≤o.01 values). Thurston drives this point home thus: "Put into perspective, the [disruptive innovation] models have now made more predictions than all U.S. venture capital deals over the past five years combined, with a predictive accuracy more than 2.5X greater than the venture capital industry as a whole." http://techcrunch.com/2014/06/30/christensen-vs-lepore-a-matter-of-fact/
ii. The company, CPS Technologies, ultimately became quite successful under the leadership of Grant Bennett, whom I hired into the company. He turned out to be a much better manager than I am.
iii. In 1993, my paper "The Rigid Disk Drive Industry, 1956-90: A History of Commercial and Technological Turbulence,” published in Business History Review, won The Newcomen Award in Business History from Harvard, given to the faculty member who publishes what the judges deem to be the best paper of that year.
In recent years the counterfactual approach to understanding causality has gained adherents—though there are some detractors (Dawid, 2000). In essence, a counterfactual construction—if our presumed cause had not occurred, the effect would not have occurred—simulates experimental designs where different candidate causes can be tested for their causal impact. See Rubin, D. B., “Estimating causal effects of treatments in randomised and nonrandomised studies.” Journal of Educational Psychology. 1974;66:688–701; Bennett, J. (1987). “Event Causation: the Counterfactual Analysis”, Philosophical Perspectives, 1: 367–86; Collins, J., Hall, E., and Paul, L., 2004; Causation and Counterfactuals, Cambridge, Mass: MIT Press; Halpern, J. and Pearl, J., 2001. “Causes and Explanations: A Structural-model Approach — Part I: Causes”, Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, San Francisco: Morgan Kaufman, pp. 194–202.
iv. The Innovator’s Dilemma, p. ix.