...

What is the question and what can we test to answer it?

All of our tests begin with a question we’d like to answer. Some of the earliest tests we ran focused on pretty basic questions, but when they all came together they formed a backlog with two parts: What is the question and what can we test to answer it?

Question What can we test?
What’s the best way to design a country input dropdown? A dropdown with different list orderings, fancy dropdowns with autocomplete, and different page locations for dropdowns.
Are multi-page forms better or worse than single page forms? Test the same form with different page breakups: all one page, multiple pages, and a page for every element.

It’s important to ensure that the question can actually be answered through this methodology. If you’re trying to answer a question regarding how users feel about a certain change or UI implementation, this probably isn’t the right methodology for you. However, questions concerning usability and user experience on a broader, more general scale are appropriate. These questions can apply to specific user interface elements like dropdowns, or an entire site-flow like a registration flow.