Verified by 
Creativity
Innovative Leaps and Loops in Idea Space
The breakthrough power of bridging from novelty to the known.
Posted April 4, 2021 Reviewed by Jessica Schrader
How do we move within and across the spaces of ideas to discover new possibilities? Should we leap into unexplored territory, or loop within and around what we know well?
This is the classic choice in idea search between exploration and exploitation. Exploratory search is characterized by searching in novel and unfamiliar idea spaces, on the outer frontiers of what is known. In contrast, exploitative search involves seeking within and around relatively familiar ideas, delving deeper into what's already apparently known or understood.
Exploration is characterized as decidedly risky, but it can lead to radical breakthrough innovations and revolutionary discoveries. Exploitation, instead, leads to what seem to be more incremental (step-by-step) discoveries that are less impactful, and do not so sharply upturn or up-heave the idea/making landscape of a field or industry.
But is this depiction of how breakthrough innovations come about fully accurate? Or does it simplify a process that is more nuanced, iterative, and unfolding?
Re-evaluating the exploration vs. exploitation dichotomy
To address this question, two researchers at Harvard Business School turned to a very large database of possible innovations—more than 1.5 million U.S. patent applications, spanning some 30 years and more than 2,500 companies. From within this dataset, they focused especially on those patents that were applied for in the year 2005. To differentiate the patents that involved "breakthrough" inventions, they used a measure of how often the 2005 patent was itself cited (referenced) in later patents within the same technological class. The 4,743 patents that had a high level of forward citations (the top 5% of forward citations) were classified as "breakthroughs" and were compared with 69,499 non-breakthrough patents.
Then, to gain a fuller and richer picture of the process leading up to successful innovations, the researchers specifically looked at all patent applications, not only those that were successful or granted patents. This provided information about what the firm was attempting to do, the technological information/know-how that the firms were using at that time, and how the firms believed their approach built upon and was different than prior patents (known as "prior-art citations") in the same technology class.
The researchers combined these data with other indices of the firm's patents and related patents to create a new measure of how close the sought-for patent was to already-known technology. This new measure, which they called "technological focal proximity" would have a value near one when the invention was very close to the previous theoretical knowledge of the firm, but would approach zero as the content of the patent application diverged very far away from the firm's existing knowledge.
So what did they discover?
The process leading up to all of the patent applications, whether breakthrough or non-breakthrough, initially started at a point when the firms had comparatively less knowledge or expertise (average value of the "technological focal proximity" slightly below .20). This suggests that innovation starts with a period of exploration, in a knowledge or idea space that is quite far away from the firm's prior knowledge.
The breakthrough vs. non-breakthrough inventions also showed different knowledge trajectories. Charting the firms' "technological focal proximity" to the patent applications over the 30 years prior to the application showed breakthrough and non-breakthrough inventions followed different trajectories. Their trajectories were also non-overlapping. This meshes nicely with the well-accepted notion that the search processes behind breakthrough versus non-breakthrough inventions are significantly different.
Crucially, however, especially in the 10 years immediately prior to the application for the patent, the breakthrough patents were closer to the firm's technological competence than were the non-breakthrough patents. Surprisingly, and contrary to the conventional exploration-exploitation dichotomy, breakthrough patents were not farther away from the firm's knowledge or competencies at any point across the 30 years.
Indeed, breakthrough inventions were especially likely to emerge in those firms that, in the five to 10 years leading up to the patent application, concentrated their research and search efforts in the technological and knowledge neighborhood nearby to that of the invention. As ideas and know-how within and surrounding the "promising find" are more deeply delved into and connected, the ideas/processes/materials that were once novel and unfamiliar become increasingly understood and familiar.
Stated simply: The story behind breakthrough innovations, then, is not only one of exploration, or only exploitation, but of both. Although the learning and searching process for both breakthrough and non-breakthrough discoveries started out as exploratory, firms that transitioned to an increasingly concentrated exploitation search in once-unfamiliar idea territory were significantly more likely to produce breakthrough inventions.
To think about
We're remarkably adept at the mental act of categorizing things. It is both a unique strength—and an often-encountered downfall—of the human mind.
The strength of such categorizations, in dealing with one another and with our world, comes from how they allow us to notice and name what otherwise we might have missed. Categorization can change our ways of interacting, responding, and forming effective working models of the world in our heads.
The downfall of such categorizations is that we start to take these lines that we have drawn in the mental sands of our minds, as lines that are really out there, as sharp demarcations and solid boundaries that exist in the world outside our head. We take (mis-take) conceptually created and mentally postulated lines for lines that are real.
Perhaps there are parallels here to another distinction often made with regard to the process of generating creative ideas: that between flexibility (when we move across and between different domains or perspectives) and persistence (dwelling, staying with one domain or perspective to deeply mine and intermesh ideas). Both flexibility and persistence are necessary. Neither alone is sufficient. For breakthrough inventions—or for everyday creatively adaptive problem solving—we need both flexibility and persistence, both exploration and exploitation. And transitions between each.
To deeply and meaningfully innovate, we need both leaps, and loops, in our idea spaces.
References
March, J. (1991). Exploration and exploitation in organizational learning. Organization Science, 2, 71–87.
Nijstad, B. A., De Dreu, C. K. W., Rietzschel, E. F., & Baas, M. (2010). The dual pathway to creativity model: Creative ideation as a function of flexibility and persistence. European Review of Social Psychology, 21, 34–77.
Sarnecka, D. K., & Pisano, G. P. (2020). The evolutionary nature of breakthrough innovation: Reevaluating the exploration vs. exploitation dichotomy. Harvard Business School, Working Paper 21-071.
Wu, Y., & Koutstaal, W. (2020). Charting the contributions of cognitive flexibility to creativity: Self-guided transitions as a process-based index of creativity-related adaptivity. PLOS ONE, 15(6): e0234473.
