UX Methods and Lessons Learned: A Reflection After One Year in the Field

After a year in the UX research world, I’ve gained a lot of new knowledge and experiences, but I’ve also realized that I have so much more to learn. At first, UX research might seem to be a basic task of asking participants a variety of questions, but I’ve quickly learned that it’s a much more nuanced and skillful undertaking. Researchers must be quick on their feet, ask thoughtful and discussion-provoking questions, and keep participants on topic and within the allotted timeframe — which is much harder than you might expect! Researchers must stay vigilant and curious, constantly probing to uncover participants’ inner thoughts, reactions, and impressions, while also noticing their demeanor, tone, and body language. It’s a careful balancing act and an exciting investigative endeavor.

I’ve realized conducting insightful research is an art. Establishing and gaining participants’ trust while encouraging them to share their inner thoughts and feelings — often in a novel and strange setting no less — isn’t easy! After working with product owners and designers to establish the main research goals, you have to carefully translate these goals into interview questions that spark meaningful conversation without leading or biasing. Most often, I’ve found that you need to ask a question without actually asking it. In other words, you need to ask a series of questions about a topic and cleverly piece together participants’ responses to determine what they really think about it. Thus research is quite creative and challenging. It becomes a game of taboo, where you try to discern how people think and feel about something without actually asking them outright.

In addition to sussing out how people feel about something, you must focus on what people do, not just what they say. Because participants are social creatures, they want to make you happy, and they may skew their answers based on what they think you want to hear. If you ask if they like a feature, they will almost always answer “yes!”, even if they don’t actually love it nor find any practical value in it. But you are paying them a pretty penny, and they want to please you. And on a less cynical note, some participants may not knowingly mislead you. Some may not fully understand your question, or they may inadvertently misinterpret a question or key term. In order to combat this, be sure to break complex questions into their component parts to allow for more granular and streamlined responses and be sure to avoid jargon or ambiguous terms.

You also should address these power and social dynamics at the beginning of each research session. Personally, I ask participants to give me honest and candid feedback, letting them know that I didn’t design the sketches/wireframes/prototypes, so they can’t hurt my feelings. I assure them that negative feedback is just as, if not more, helpful than positive feedback. It allows us to identify issues and fix them, which makes it a much better experience for everyone. I’ve found that initially setting the stage can make or break a research session. Don’t hurry through it and impassively rattle off the instructions. Slow down and have a genuine conversation with your participant, making eye contact and being very sincere. This will lay a solid foundation for the next hour you will spend together, and you will get much more genuine and thoughtful feedback.

I’ve found that probing is the most difficult skill when conducting research. You need to figure out when and how best to probe. Remember your overall research goals, and during the interview selectively ask questions to dive deeper into relevant topics to uncover insights. Some researchers suggest asking “why” five times in order to get to participants’ core reasoning. By asking “why” so often, it forces participants to think about and clearly articulate their thinking and prevents you from (unknowingly) making assumptions. You might think you understand why they prefer the layout of design A over the layout of design B based on their previous comments, but ask them why they have this preference, and you might be surprised by their answer.

One of the most important parts of a researcher’s job is to uncover users’ thoughts, feelings, and reactions. Have them explain themselves and their thinking clearly so there is no room for ambiguity or disagreement when you discuss and present your findings later on. Have your users clearly and explicitly explain their reasoning, so you can take their exact words and rationale instead of having to fill in the blanks or make up your own, which introduces your own bias and interpretation into the equation. Probing takes practice, but over time you will improve. Listening to recordings of your interviews allows you to critique and reflect on how you can improve. And if you have the opportunity, have coworkers observe and give you feedback.

As far as methods for expert probing, repeating back participants’ words and asking them to elaborate is a great way to dig deeper without biasing them by introducing new words or unknowingly interjecting your own values and opinions. For example, “What do you mean when you say it’s ‘interesting’?”. Additionally, pausing for six seconds to give participants ample time to think, or beginning a question and trailing off are both quite effective, as participants will often respond to fill the silence.

I’ve also found that research methods vary depending on the type and stage of research. I’ve conducted research at all three stages of the development cycle — exploratory, concept testing, and usability testing — including in-person interviews, remote interviews, onsite visits, focus groups, and surveys.

Personally I find exploratory research the most abstract and challenging type of research, but also the most interesting. Exploratory research occurs early in the product development cycle, and it can strategically impact product roadmaps by identifying key wants, needs, and opportunities. During exploratory research, it’s very important to keep your research goals in mind, as it is all too easy to get off track and collect endless amounts of data. In order to gather and present the most focused and useful findings, researchers must keep participants on topic and uncover their underlying values, opinions, and needs.

The true job of a researcher is to bring the users’ voice into the conversation and tirelessly advocate for them. Although it might seem best to keep product and development limitations in mind, you need to avoid getting bogged down by “limitations”, as some may be more perceived than real. If users have an urgent need, don’t listen to product owners and developers who say it isn’t possible. Instead you must convince your colleagues that it is their job to make the product address this need and encourage them to think creatively and approach the problem in a novel way. Researchers are responsible for fully understanding the problems and making others aware of them, while product owners, designers, and developers are responsible for finding the solutions to these problems.

When conducting exploratory research, there are two theories of thought when it comes to preparation and background knowledge. Some think it is important for researchers to conduct extensive background research and understand the context in order to ask perceptive questions, while others think researchers should go in naïve and let the participants fully educate them. The first camp of thought argues researchers need background information in order to structure their questions and emerge with well-rounded data, while the second camp of thought argues researchers bring less of their own bias into the interview and analysis if they enter naïve.

Initially I strongly agreed with the first camp, feeling ill-prepared and uncomfortable without background knowledge, but now I’m starting to see the value of the second camp. The participants’ reality is the only thing that matters, even if it differs from “objective reality”. As a researcher, I need to convey the user’s reality. If I come in with preconceptions and outside knowledge, I might struggle to convey their reality to stakeholders, inadvertently imposing my own expectations, assumptions, and understanding. If I come in naïve on the other hand, I simply need to repeat back what I’ve heard, without reinterpreting it or comparing it to “reality”. It might feel uncomfortable without any background knowledge, but I’ve found that if you’re up front about this and let your participant know that they’re the expert and you’re here to learn from them, things tend to work out.

For example, if HR Practitioners use a payroll product and don’t know that they can run a pre-payroll report of all hours worked, then this pre-payroll reporting feature is not very useful. The practitioners are not at fault — it is the payroll product provider who must produce an intuitive and user friendly experience. If your product requires extensive training and ongoing implementation support in order for users to discover basic functionality, it is not scalable. Instead you must strive to provide clear, simple, and intuitive design.

Exploratory research is very open-ended, so researchers must stay focused and goal-oriented. It can also be challenging to translate the findings into actionable and useful results, making proper planning and goal-setting very important. With time, practice, and experience, researchers learn to strategically probe and get the most out of each session. They learn to avoid going down rabbit holes, keep participants on topic, and focus in on key areas that will answer the carefully formulated research questions.

Concept testing is mid-stage research and builds upon exploratory work. Exploratory research beginnings narrowing the focus and direction, and concept testing allows for additional iteration and refinement. There are two stages to concept testing — refining an idea and refining a design. Refining an idea is earlier, where you bring some rough ideas to discuss with participants and get their feedback on. Based on their responses, your team will be able to refine your understanding and begin to create more detailed designs. Refining a design comes next, and you have a higher fidelity design or prototype to show users and get feedback on. You can give users a variety of tasks to complete and observe if they understand the designs/prototypes, and determine if you need to iteratively revise the design and continue concept testing.

Usability testing is the final stage of research. It might seem more straightforward, however there’s much more to it than simply checking off if users do or don’t understand a feature or product. In usability testing, as in concept testing, it’s really important to observe participants instead of taking them at their word. For example, I recently had one participant who really struggled with scrolling and navigating a mobile onboarding flow, however at the end she said it was “easy to use”. When I asked her why it was “easy to use”, she said because the navigation was simple. Maybe she had forgotten her previous struggles or was simply trying to find a reason to justify her statement, but I knew I should trust my observation of her behavior (struggling to scroll) instead of her telling me that she found the navigation intuitive.

Focus groups also provide unique challenges. I recently conducted a focus group on painful and time-consuming Time and Payroll processes, and I learned that it is paramount to keep participants on-task, engaged, and within the time limit. We introduced ourselves to participants and shared our research goals — identifying long, tedious, and repetitive Time and Payroll processes that could potentially be automated. Then we gave them 10–15 minutes to individually write down painful processes, why they were painful, and why they were important. Then we had each participant share their processes with the group and discuss.

At the end of the individual brainstorming session, one participant wanted to clarify that he interpreted “painful process” as time consuming, burdensome, or challenging. I immediately learned how important it is to avoid ambiguous and hazy terminology in the prompt. Even if you clarify the terminology in the set-up, it is best to have visual definitions or prompts for participants to refer to.

Additionally, researchers need to moderate the session and make sure participants have equal opportunities to talk and share their ideas, instead of allowing one or two outspoken participants to dominate the session. I found that it can be helpful to include both individual and group activities. Having participants work individually at first helps promote participation since everyone has already generated some ideas, and it keeps them honest and prevents groupthink from taking over. Users can share their ideas and build off of each other instead of simply agreeing with everything others might say in order to avoid formulating their own thoughts.

Finally, it is really important to keep your research questions in mind when writing your report. Focus groups can cover a lot of ground and capture a variety of viewpoints, so it can be overwhelming to review and analyze all of the data. Keeping sight of the research questions and goals will help narrow your focus and create a targeted and useful deliverable.

Onsite visits are quite different from other research methods. They involve going into participants’ natural settings and letting them show you around. They are great opportunities to gain empathy for your end users. By going into their offices and seeing them work, you can gather additional insight that might be missed over the phone. For example, on the phone one participant said she looks at her calendar, however when we visited her office we saw that she had multiple calendars, each with a distinct and exclusive purpose. Onsites are a great opportunity to see users’ workarounds, artifacts, daily documents, work routines, and flows.

I conducted an onsite visit at a popular Las Vegas hotel and casino’s office headquarters to observe onboarding sessions for new hires. I learned to be flexible and work with our client contacts. We communicated our goals and expectations for the visit early on to ensure we would be able to interview a variety of people and observe the onboarding sessions over two days. Right before the trip though, we learned we would not be able to talk to managers nor new hires, two of three user groups we had planned on talking to. We adapted to our circumstances though and were able to interview new hires more informally than initially planned. The new hires were only paid for a few hours in the morning to complete their new hire paperwork and onboarding materials, however they had some downtime after they onboarded and were waiting to have their I-9 documents cleared. I was able to informally ask them about the process they just completed, while it was still fresh in their minds.

I also learned to not be afraid to jump in and get involved. The clients introduced us to the new hires early on and explained why we were there, which helped ease any uncertainty or anxiety our presence caused and encouraged new hires to be as candid and natural as possible. You get the most out of your visit if you immerse yourself in the environment and tasks, taking extensive notes and a flurry of pictures. But remember to not be disruptive, as your hosts are opening up their office and time to you. When synthesizing and analyzing your plethora of data, you must keep in mind that your insights are only from one client though. In most studies you get to talk to participants from eight different companies, whereas in onsite visits you get to talk to a variety of people within one organization. You can dive deeper into their particular process and pain points, but you must take your findings with a grain of salt and remember that this is only one company in one industry and it is not necessarily representative of the larger picture.

Thus there are a variety of research methods at your fingertips to talk to users and gain valuable feedback about your ideas, designs, and products. Finding the best methods for you based on your goals, resources, and timeframe will serve you well. Over time you will get experience with each method and learn how to select the best method for your circumstance. Conducting a variety of research and reflecting on what did and did not work will help you grow as a researcher and refine your skills and professional value.