Exploring and Exploiting

All organisms balance their energy expenditures between exploring and exploiting.  Last night I watched a National Geographic documentary about the forest elephants of Gabon so I will use that as an example.  These elephants roam hundreds of kilometers from the inland rainforest to the coast.  They forage across numerous ecosystems and in some cases even reform the environment to suit their needs.  The opening portion of the documentary focused on a clearing that the elephants had created by trampling the vegetation over many generations.  The pools in that clearing now collect highly concentrated minerals necessary for the elephants survival and also support plants that can’t be found in the elephants’ “natural” habitat.

In a sense the jungle elephants engage in an instinctual (naturally selected) form of agriculture.  But of course, this limited form of environmental engineering is not sufficient to meet the nutritional needs of an animals that can consume hundreds of kilograms of vegetation a day.  As noted above, the elephants cover hundreds of kilometers in search of a broad variety of food sources.  At every step along this journey they must choose between exploring and exploiting.  They can stay in a given area and pull every last plant from the ground or they can move on to a new location with no guarantee that they will find food.

This sounds like a simple decision but in fact these turn out to be very difficult mathematical problems if you attempt to formalize them, particularly when there are so many unknowns.  Fortunately for the elephants, their decision algorithms are genetically programmed.  Each individual within a given population will have certain innate tendencies.  Evolution has even had to account for higher order explore/exploit dilemmas.  Within a given population, how many individuals should stick strictly to the proven strategy and how many should diverge from the norm and be compelled to experiment with new strategies.

Everyday Optimization

We face these same dilemmas daily.  Most people are rarely aware of them, but once you grasp the concept you see explore/exploit decisions everywhere.

• Should you watch the same tired sitcom every Thursday night or should you experiment with something new?
• Should you introduce that newfangled collaborative tool into your work group or should you stick with what works?
• Should you search for the fastest commute home, risking even worse traffic, or should you take the usual route so you can zone out?
• Should you bring up that thing that’s been bothering you with a particular family member, or is it better to simply smile and nod to keep the peace?

Most people have a tendency towards one or the other.  They like things (or processes) that are well understood or they get anxious for that opportunity to discover something new.  However, the explore/exploit concept is not quite so trite as saying that some people like to try new things while others prefer that which is comfortably familiar.

Here in the San Francisco bay area the foodie culture is something of a metropolitan pastime.  Is the foodie an explorer?  Perhaps. Perhaps not.  The foodie is demonstrating some relatively simple first-order exploration by iterating a formulaic process repeatedly.  So as not to sound condescending, I will throw one of my hobbies, Crossfit, into the mix.  One of the tenets of crossfit is “constantly varied”.  Yet, about 95% of the time that amounts to variety of the same magnitude experience by the foodie.  Only about one in twenty workouts is a completely unexpected curveball that catches everyone off guard.

Higher order exploration patterns are surprisingly difficult to consistently adhere to and seriously limit your exploitation ability.  Crossfit is a case in point.  Nearly everyone follows a workout schedule programmed (explored) by a coach or trainer so that they can focus on the workout itself without equivocation.  In other words, they welcome a set of external constraints that allows them to focus on exploitation.  Attempting to do both, to any significant degree, mixes performance with meta-cognition too thoroughly.

Finding Balance

I noticed this same pattern while reviewing my blogging activity over the past year.  I have reasonably high expectations for what I post here.  I don’t expect my writing to be particularly poetic but I do expect my ideas to be well thought out and coherently presented.  In other words, I try to exploit my ideas at least somewhat comprehensively beforehand, rather than posting stream-of-consciousness speculations.  That has proved a significant hurdle, particularly for the newer material that is most motivating but least refined.

Around midyear I revived my old posterous blog, renamed it the idea incubator, and began using that space to present shorter and more spontaneous musings.  In looking back over those posts I am finding that quite a few of them are among my favorites from the past year.  Here is a sampling:

1. Big Thinking in Dense Packages
2. Moving Beyond the Dialectical Socioeconomic Debate
3. Peter Thiel Needs a Session With Clay Shirky
4. Project Oriented Start-ups and the Entrepreneurial Personality
5. On Working Backwards
6. Fear of Illegibility

I am reminded now of an excellent youtube video I came across on twitter just the other day featuring Ira Glass talking about the creative process:

I have always preferred exploring to exploiting, but I am also painfully aware that that preference has handicapped my exploitation muscles.  As Ira Glass conveys so concisely, even when the work is missing that special something the only option is to try again.  My takeaway from reviewing the exploratory posts linked above, is not to let weaknesses get in the way of strengths.  For me, the refinement process is often a struggle but the stuff that flows spontaneously isn’t that bad.

The broader takeaway is that we are all making explore/exploit choices either consciously or unconsciously.  It is impossible to identify with any certainty the optimal balance between two.  We can however, sensitize our intuition to alert us when we have drifted away towards a definitively non-optimal balance.

(Thanks to Neil LaChapelle for persistently reiterating the explore/exploit theme in our conversations.)

photo courtesy of Eliel Freitas Jr