2 A/B Testing Lessons Learned from Amazon Video
It is no secret that Amazon is a data driven organization. The culture of testing is ingrained into pretty much everything Amazon does. However, a company like Amazon also makes mistake as I learned from my recent experience with Amazon Videos. I have two Amazon Fire TV Sticks, one attached to each TV that I have. Both are tied to the same Amazon account. Using my Amazon Video account, I started watching a movie on one TV, turned it off halfway through and then a day later tried to watch it on the other TV. Guess what? I could not find that movie in any of the obvious menu options. I was expecting it to show up in my stream or at least in the same place where I found it last time. Nope. Amazon changed the order of movies on me. It’s not like it added new movies to the lineup and this movie got pushed down. They just reordered the existing movie selection. It appeared to me that some kind of movie display sorting experiments was going on. However, that experiment,...