About Evaluation

So at this point in the iterative design process, you probably have a prototype or at the very least an idea that you have worked out to test, and you are wondering, “how can I do this so that we get invaluable insights?” or “how much data do I need in order to know what problems are actually real problems and not personal preferences?”.

To begin with, let me explain something; arranging a user test with actual users can be costly in terms of money and time, hence, we cannot and should not necessarily test every small change with real users. Sometimes, one or two design experts can identify problems that would take a lot of user testing to come to the surface. For example, when we have a user interface for our game, there are rules of thumb that help us create good and clear interfaces, and a good designer can go through these rules of thumb and make sure that they are followed before a larger test with the users is performed.

So, it is important to gauge the type of test that we ought to perform. Similarly, if we are to test the prototype with actual users, we do not always need to have 30 or more users testing it to gain useful insights. In fact, in the majority of cases, 5-6 testers will help you capture 80%-90% of the problems that your prototype has- after the 6th tester, most problems that are identified are a repetition of what the previous testers have identified. With that said, a smaller evaluation with 5-6 users is quite suitable for a lo-fi or mid-fi prototype. Once you are stepping into the hi-fi territory, you can start planning for a larger evaluation phase with a larger number of users.

The tests that you want to perform are oftentimes of a higher quality if you combine a qualitative type of test with a quantitative one. For example, when you are performing a user test in the form of a survey/questionnaire -which usually gives you mostly quantitative data - it would be best to combine that with interviews to gain some qualitative data as well. Combining questionnaires with interviews gives you a very solid foundation for your design decisions in the next iteration. With that said, it really depends on where you are in the process as performing a full-fledged questionnaire + interview user testing can be a bit of an overkill if you are in the early stages of the product design/development.

In the following chapters, we will go through a few methods that we can use to evaluate our games. It is important to mention that the evaluation phase most often merges with the next iteration’s research phase as we use the data gathered in this phase for analysis to formulate a good question in the beginning of the next iteration.