By running thought experiments before making a site change, you increase the chances that when you roll out the change it will be successful.
When you run a thought experiment, you attempt to prove, disprove or refine an idea with existing evidence. You can iterate on ideas very quickly, thought experiments are extremely cheap to run and, since they are not audience facing, they are also very low risk.
The evidence you dig into can be from a variety of sources, e.g. site analytics, focus groups, surveys, etc. The goal is to understand why a change would or would not work.
To walk through an example… Suppose the company you work for sells artwork that ranges from cheap prints to fairly expensive reproductions. Most of the orders are for cheap prints, so the company focuses on these low value, high volume sales. You have been instructed to increase revenue by promoting the cheap prints more effectively.
As a thought experiment, you decide to test whether the low value sales are what the site should primarily promote.
When you look at the order numbers, you find that most purchases are in fact low value. But when you look at the revenue, there are two peaks. One peak corresponds to the low value sales. The other peak corresponds to the high value sales. While there are far fewer high value sales, the margin is so much higher they bring in a lot of revenue.
This suggests that instead of one primary audience behavior there are two – one behavior pattern is to purchase low value products and the other is to purchase high value reproductions.
To further refine your thought experiment, you pull out information on who these high value purchasers are. You discover that they are concentrated in urban areas, specifically Los Angeles, San Francisco and New York. The types of artwork that they most frequently purchase are large, abstract paintings. The paintings are purchased by individuals, not businesses. It appears likely that people are purchasing these paintings to decorate their apartments or homes.
By following a thought experiment you’ve found out a lot about a profitable audience segment that hadn’t been identified. You also have some new ideas for how to increase revenue, e.g. make high value artwork more visible on the site, expand the types of artwork offered, start a weekly feature about how to decorate your urban apartment with artwork.
Thought experiments can help you find patterns and audience behaviors that have been overlooked. By running thought experiments before making site changes, your site changes are more likely to be successful because they will be informed by evidence rather than assumptions.