Tuesday, November 30, 2010

Experiment, Evaluate, & then Experiment again

From pricing to finalizing a user interface, experimentation can play an important role in helping a firm navigate a new market or offer a new product.

This engadget report followed Netflix as they experimented with the right pricing formula for its new streaming-only service.

Netflix's experimentation was not only limited to pricing, the firm also experimented with several UIs for the PS3.


  1. I'm on the $7.99 plan and love it. But I now understand why Netlix has been investing so much into algorithms that improve movie recommendations. It's excruciatingly difficult to pick a movie to watch from all the endless options that they offer. The bliss and pain of choice. I now probably spend 15 minutes just picking a movie, as much as I used to spend looking for a movie in a Blockbuster store. In fact, last night I actually switched out of Netflix and used other sites (such imdb.com) to find the movie to watch and then went back on Netflix and watched it there -- a use case that should be a concern for Netflix.

  2. Interesting point, Erdin. If a user is leaving your web property to do research then coming back to it to complete a transaction, it shows an opportunity for the research site to encroach on the transactional site.

    Incredibly, this is a small part of Reed Hastings' master plan to move Netflix completely away from DVD to streaming. The level of calculation and understanding of disruptive technology that Reed and Co have is impressive. They have a great plan around transitioning and are executing on it.

    Also, to underscore their focus on testing. Here's a great answer about A/B testing by Neil Hunt, Chief Product Officer at Netflix. Enjoy:


  3. I find the last sentence of the engadget article on the PS3 UI testing: "we'd still prefer some choice in the matter instead of rolling the dice every time we load the service."

    This begs the question, at what point does A/B testing negatively affect the user experience?

    I would be curious to know if companies such as Google or Netflix consider this, or if the author of the article is just hyper-sensitive.

  4. As a pontential customer, I have to admit that I really like the competition between Netflix and other online video-streaming sites, like Hulu. It's reported that after Netflix downgraded its fee structure, Huhu, following, reset its online subscription price at $7.99, from $9.99 as well. I wonder the impact of "cloud" on these companies.

    Waiting to see more changes on Netflix website to improve customer experience.

  5. I think Netflix has to be careful not to be too quick to shift long-time customers to streaming with their price-hike for physical discs. For one, the quality of movies available for streaming is not the same as on dics. Two, Netflix not only has competition from Hulu but also Redbox where you can rent DVDs for just $1 (a lot cheaper than Netflix). With Redboxs popping up on every corner, the can steal some of the every customers Netflix is looking to convert to streaming. I don't think we're at the point yet where people want to hook-up the streaming service to their large tv to watch a blockbuster movie. Streaming seems more success for TV shows (like on Hulu), dics/blue ray for movies as quality is also an issue. Anyway, streaming still has challenges but it's most probably the way movies will be viewed in the future.