Friday, October 8, 2010

Susan Athey on Online Experimentation

Susan Athey is a professor at Harvard and Microsoft's Chief Economist. She just did a short radio commentary about online experiments. Here's how it starts:

Did you know that every time you do a search on Google or Bing, you are improving the quality of the search engine? The more people click on a search advertisement from a clothing company or on a link on an online news story, the more prominently it is displayed for the next consumer. And the firms constantly experiment to get things right. They watch what consumers do and adapt their products in response to the results of their experiments.

You can listen to the rest via the Marketplace website.


  1. I agree with Susan's call to action, that schools need to start focusing MORE attention and resources on training students to do behavioral predictive modeling.

    However, I think that her outlook is a little fatalistic when she says that schools "haven't figured out how to train students to do this kind of modeling and prediction."

    Modeling and prediction may not be part of the core, or even high on students' lists of things to learn before they leave business school, but the offering certainly exists.

    Case in point, is the following class which I took last semester with Professor Braun: 15.840: "Who Are Your Customers and What Will They Do Next? Applied Probability Models in Marketing".

    Our very own Dean Schmittlein received his PhD in probability modeling. We read several of his papers throughout the course. Here is one which is pretty close to the topic Professor Athey's commentary:

    Testing New Direct Marketing Offerings: The Interplay of Management Judgment and
    Statistical Models
    Author(s): Vicki G. Morwitz and David C. Schmittlein
    Source: Management Science, Vol. 44, No. 5 (May, 1998), pp. 610-628
    Published by: INFORMS
    Stable URL:


  2. I'm in complete agreement with Paul. Schools, in general, need to teach students to understand the wealth of data that's available that what you can do with it.

    "Unfortunately, our universities and business schools haven't figured out how to train students to do this kind of modeling and prediction."

    To that point, I feel that MIT Sloan (not biased at all =) has done an excellent job responding to this data driven economy. With classes like Paul mentioned (Who Are Your Customers and What Will They Do Next? Applied Probability Models in Marketing), this class (Economics of Information), and even the MIT Sloan core required class (Data, Models, and Decisions), we are exposed to a large set of tools needed to use this data to understand business challenges.

  3. The difficult part is training people to think in terabyte/petabyte scales. The maximum number of rows you can have in an excel spreadsheet is on the order of 30K, I think. Most of the classes I have taken at Sloan are implicitly working on that scale.

    Lots of the statistics that we've been taught aren't really necessary anymore - they're designed to gather the most possible information from the smallest possible sample sizes.

    The problems we face now are the exact opposite - we're deluged with data, but our frameworks only allow us to use a small portion of the information in a timely fashion.

  4. I agree with what has been said above.

    During our E&I trek to Silicon Valley I had the opportunity to meet several amazing startups who utilize predictive behavioral analytics. This is especially essential in this Web 2.0 environment, such as for search and targeting purposes. I can see applications in entertainment, media and gaming as well. MIT is well-positioned to become a leader in this field if the school decides to pursue this.

  5. Leo,

    Great point! You are absolutely correct. The classes I have taken so far simply scratch the surface of what can be done with the massive amounts of data available.

    Prior to Sloan I worked at ZS Associates (a boutique consulting firm) where I had the luxury (insert sarcasm) of working with what seemed to me at the time massive data sets, with millions of rows and hundreds of columns of data. We used a Unix based SAS program to analyze the data. I am now very grateful for the opportunities I had at ZS because I was introduced to the demands of Web 2.0.