How to experiment: Guidelines from Stewart Friedman's "Be a Better Leader, Have a Richer Life"

[cross-posted to Quantified Self]

1. Curiosity: An emotion related to natural inquisitive behavior such as exploration, investigation, and learning.
2. Exploration: To travel for the purpose of discovery.
3. Discovery: A productive insight.

I've been thinking of this triumvirate as essential characteristics of scientific inquiry - get curious about something, try out some different things to dig into it, see what you learn, and repeat. My personal interest in this, in addition to the tools and sites we share here on QS, is to figure out how specifically we navigate the process of curiosity, exploration, and discovery.

Taking a cue from Alex's summary of How To Measure Anything, Even Intangibles, I want to share an impressive work called "Be a Better Leader, Have a Richer Life" by Stewart Friedman (see below [1] for where to find it). I'll focus on the experimental aspects of his work and pull out some highlights related to process.

Friedman describes a four-step process, where each step relates to his four general domains of life - work, home, community, and self:

1. Reflect
2. Brainstorm possibilities
3. Choose experiments
4. Measure progress

The first step, reflect, is where you think about your priorities in each of those four domains and compare them to how you actually allocate your time and energy. This will identify conflicts that should guide your choices of where to start experimenting. (I think of this kind of goal-driven approach as "top down" experimenting, as distinguished from "bottom up" where you start from an observation that catches your attention, such as when I noticed I am moodier after drinking alcohol, and start self-experimenting from there.)

In step two you brainstorm possible experiments that will close the gaps identified in step one and bring you more satisfaction in life. The author stresses the importance of putting together a long list of small experiments. The author notes that keeping them small helps minimize risk and gets results quickly. I especially like his guideline that the most useful experiments feel like a bit of a stretch: not too easy and not too intimidating.

The third step is to choose which of the candidate experiments to perform, i.e., which are most promising and will improve your fulfillment and performance in his four dimensions of life. I liked his suggestion that the experiments be ones that would have a high cost of regret and missed opportunities if you didn't do them. He goes on to say that it's not practical to try out more than three experiments at once. Not only do experiments take effort, but in Friedman's experience two turn out to be relatively successful and one "goes haywire."

The final step is to measure progress. He has you develop a scorecard for each chosen experiment where you specify its life dimension, your goals for it, and how you'll measure success. Metrics may be objective or subjective, qualitative or quantitative, reported by you or others, and frequently or intermittently observed. He gives sample ones like cost savings from reduced travel, number of e-mail misunderstandings averted, degree of satisfaction with family time, and hours spent volunteering at a teen center. Friedman stresses the common wisdom that, like a scientist, the only way to fail with an experiment is to fail to learn from it, and metrics help ensure that doesn't happen. They give you hard data to analyze, and can teach you how to make better ones in the future.

Here's a sample experiment, courtesy of the BNET article below:

Exercise three mornings a week with spouse.

Friedman gives plenty of advice beyond the four steps. In particular I like his description of the overall experimental approach:

"...systematically designing and implementing carefully crafted experiments - doing something new for a short period to see how it affects all four domains. If an experiment doesn't work out, you stop or adjust, and little is lost. If it does work out, it's a small win; over time these add up so that your overall efforts are focused increasingly on what and who matter most. Either way, you learn more about how to lead in all parts of your life ."

Finally, I love how he describes the value of the experimental mindset. One example is how framing an experiment as a trial can open doors that would otherwise be closed. Saying "Let's just try this. If it doesn't work, we'll go back to the old way or try something different" lowers resistance because the change seems less threatening. This is valuable because it's our nature to fear change. In fact my wife and I regularly use this with each other, such as when, during a kitchen remodeling when she got me to accept trying a vintage sink that initially, well, made me a little queasy. She pointed out that "It's just a little experiment" and that it was relatively reversible (standard plumbing placement meant a different one could be easily installed). The result: It worked out fine. In my case I suggested we experiment with a couch in the kitchen, an idea she despised but came to love. Give it a try!

Overall, I highly recommend Friedman's work. His book is my next read.

Resources

• [1] You can find a preview of the original April 2008 Harvard Business Review essay at Be a Better Leader, Have a Richer Life, with a short summary on BNET. He has expanded the work into a book, "Total Leadership: Be a Better Leader, Have a Richer Life" (home page, Google Books, Amazon).

"Productive stupidity means being ignorant by choice."

Productive stupidity means being ignorant by choice. Focusing on important questions puts us in the awkward position of being ignorant. One of the beautiful things about science is that it allows us to bumble along, getting it wrong time after time, and feel perfectly fine as long as we learn something each time. No doubt, this can be difficult for students who are accustomed to getting the answers right. No doubt, reasonable levels of confidence and emotional resilience help, but I think scientific education might do more to ease what is a very big transition: from learning what other people once discovered to making your own discoveries. The more comfortable we become with being stupid, the deeper we will wade into the unknown and the more likely we are to make big discoveries.

From The importance of stupidity in scientific research

Designing good experiments: Some mistakes and lessons

[cross-posted from Quantified Self]

Like you I'm an avid self-experimenter, and I'm always on the lookout for things to change that will either a) improve me, or b) help me understand myself better so I can do a). I was comparing notes recently with Seth Roberts (his QS posts are here) about what experiments we've done, what processes we've used to do them, and what lessons we've learned from them. I thought I'd share some of my take-aways with you and ask what you've learned from your own self-experimentation.

Keep experiments specific and simple

A mistake I've commonly made in the past made is trying to track too many things at once. For example, a year ago I was terribly fatigued and decided to improve sleep quality. I tried a bunch of things [1] but I wasn't careful about keeping them separate, or stopping one before starting the next. The lesson is that the changes you make ("treatments") and the things you measure ("variables") should be simple and few. The general goal is to maximize the amount of information you get using the least amount of effort. This should tell you where to go next.

([1] I made changes like going to bed when I first felt tired, implementing a calming and regular nightly routine, eliminating caffeine, stopping using the computer from 9pm on, cutting out bright lights before bedtime, not eating or exercising right before, and taking drugs like Ambien and Xanax. Results: The first technique was, and continues to be, helpful, but time's erasure of the stress from a family emergency a year ago made the biggest difference.)

Know the type of your design

Though I've been experimenting on myself for many years, it was only recently that I understood the basic approach of testing things on myself. I've learned that most kinds of self-experiments are a type of back and forth process called "Reversal or ABA designs." From the Wikipedia article:

The reversal design is the most powerful of the single-subject research designs showing a strong reversal from baseline ("A") to treatment ("B") and back again. If the variable returns to baseline measure without a treatment then resumes its effects when reapplied, the researcher can have greater confidence in the efficacy of that treatment.

The idea as I understand it is straightforward, but it helped me to lay it out:

1. Define the question you're trying to answer (e.g., "Is grinding during the day causing my tooth pain?"),
2. Decide one thing that you're going to change (e.g., wear a night guard during the day),
3. Decide at least one corresponding measurement you'll make (e.g., pain on a scale of zero to two),
4. Start taking measurements for a while (you're in the first "A"),
5. Implement the change and keep measuring (now you're in "AB"),
6. Then cut out the change and continuing measuring until you're done ("ABA").

What you'll look for is whether your variable changes during the "AB" and "BA" transitions. If it does, you probably found something. If not, try something new.

For example, as hinted at above, one experiment I'm doing is working on reducing pain I have in a certain tooth. I've tested a number of things (including cutting out ice cream and acidic foods) and now I'm investigating the contribution to the problem my grinding might be making. In this case "A" is wearing a mouth guard at night (my baseline), and "B" is wearing it during the day too. I just finished the second "A," and my results were surprising: not much difference! (I'm now investigating what appears to be a diurnal cycle.)

A second, odder type of experiment is one that takes advantage of the subject's symmetry by testing two treatments in parallel. (I'm told that in statistics this is called "blocking," but I haven't found a good reference yet.) I recently used this to test different cold weather clothing for mountain biking by wearing different footwear on the left and right sides during the same ride. One result was that the order of sock/bootie/neoprene layers did not matter; left and right sides were not appreciably warmer. Another left/right example is what a friend did when she got poison ivy. She didn't know which over-the-counter treatment to use, so she tested one on each side - brilliant! (In this case, my friend found out something that no one could have told her - how well the treatments work for her. This highlights a fundamental truth that underlies much of our work: What matters most to me is not whether it works for everyone, but whether it works for me.) A final experiment is one I started last year when we painted half of our house with latex paint and the other half with an oil-based one. It's only been a year, but there is zero difference so far.

I am very curious to hear if you know of other types of designs besides these two.

Allow time for understanding to grow

A frustration I face with complex subjects (like human bodies, and our behavior and relationships) is that it can take time to figure out which variables are relevant to the problem, and how long they take to demonstrate their effect. For example, I've struggled with a mood disorder for years, and, like my insomnia experiments, I've tested out different remedies such as meditation and medications. However, it's not so straightforward when you factor in life events, stressors, and biochemical "weather." (I found the post QS Measuring Mood gave a great overview of the complexities of measuring mood, by the way.) Another example is changing diet. Maybe you're different, but it took months before I noticed certain results of a vegan diet. Or take business networking - how do you determine results when effects are indirect, especially with social investments like meeting people?

What I took away is the need to be patient by giving the results time to emerge, and to be flexible, say when you notice something useful and decide to change to a new line of experimentation. Or, alternatively, the situation where you decide to drop an experiment when it doesn't seem to be producing results.

Know that you're doing one

This might be obvious to you, but there were times when I've caught myself doing "stealth" experiments that, had I thought of them that way, would have resulted in my doing something very different. As an extreme example, I started dabbling with Twitter in 2007 and eventually got sucked into spending up to an hour/day on it. Fast forward two years and it hit me during a late night Twitter-fugue that I didn't actually know why I was using the bloody thing! What I should have done was to clarify what my goal was, what I'd be testing, and for how long I'd let it go. Again, because the results can be indirect, you might have to get clever in creating measures. For example, if you're trying to create business by forming relationships on Twitter, one thing to try is to simply ask prospective clients how they found out about you. Again, ideally you'd do an ABA design.

Doing something, even if it's imperfect, is better than nothing

This lesson was important to me because a major result of my analysis of over 100 experiments I've done in the last five years is that many were non-quantified. Not having numbers limited some forms of analysis and learning, but at the same time doing them helped me make lots of improvements in my life. For me what's crucial is the experimental mindset itself - looking at life with curiosity, and bringing mainly questions (rather than answers) to how we go about improving ourselves. Though it's a cliche, I think it's true that the only failed experiment is one that you didn't learn from.

2011-01-22: They did WHAT?

Quick links from the past week of experiments in the World Wide Lab

In I haven't used soap or shampoo in a year, and it's awesome: personal experiment update, Boing Boing's Sean Bonner says "I stopped using soap a year ago. It was easily one of the best moves I've ever made in my entire flippin' life." Now that's a strong experiment!

World chemistry year targets biggest-ever experiment:

Budding scientists at primary and secondary school level will be asked to carry out four tests on water as part of the International Year of Chemistry, due to be launched in Paris on January 27.

Excellent! Sesame Street's Interactive YouTube Experiment: I'm really excited about "Sink or Float," an interactive YouTube experiment that "lets kids play a part in the science fun." It's hosted by Cookie Monster and a human friend of his, and is designed to get kids curious about science--specifically, the scientific method. The experiment centers around a fish tank full of water and several objects, and the goal is to determine which objects sink and which objects float. It uses a simple interactive branching to allow you to say what you'll think will happen. Along the way they give a straightforward description of the scientific method. Brilliant!

Ellen Tarlin at Slate is running a Clean Plate series - "Outlandish experiments in sensible eating." A recent example is Eating Experiment No. 2: Preliminary Findings where her goal was to spend as little as possible on food while still getting all my nutrients. For each experiment she lists: her goal, what she loved, what she hated, what she learned, and her conclusion. Great, detailed series.

Oak Cliff residents accept car-free challenge: Sixteen families in Oak Cliff, Dallas, TX are going car-free for 52 weeks, challenging residents to walk, bike or rely on public transportation. Of course not all communities can do this, given sprawl and big box stores, but at least some folks are trying to make a difference.

How Musical Are You?:

BBC Radio 3 and BBC Lab UK are launching a scientific experiment that asks the public to help discover how musical Britain is. By taking the How Musical Are You? test, participants will reveal their own musical profile, while at the same time helping scientists to define what it really means to be "musical".

Take the test at BBC - Lab UK - Experiments - How musical are you?

Cosmic Log - Psychic proof? Skeptics strike back:

Daryl Bem, a psychology professor emeritus at Cornell University, summed up nine experiments he has conducted over the years into precognition - the idea that human behavior at a given moment can be influenced by information they're given at a later time. Bem's long-running experiments suggested that there was indeed a slight influence. ... The critics cite the same rule of skepticism promulgated by the late astronomer Carl Sagan: "Extraordinary claims require extraordinary evidence."

Makes me a bit embarrassed of the Cornell brand! More at NYTimes.com. (A plug: If you want to support a bunch that promotes skeptical thinking, hop on over to CSI - The Committee for Skeptical Inquiry.)

A hearty congratulations to my Ruzuku friends on their release!

I'm very pleased to give a big pat on the back to friends and clients Abe Crystal and Rick Cecil for their release of Ruzuku. Abe describes it as a new tool, designed for simplicity and ease of use, that allows anyone to quickly set up online courses and learning communities. They have two case studies, one from a photographer/educator and another from a writer/speaker.

From my one-man start-up and entrepreneur perspective I have a ton of respect for the risks, challenges, and determination it takes to create something new, especially one that revolves around building a community. Great work, guys!