Elizabeth Allen
Establishing Confidence in UX Research
In this episode of Brave UX, Elizabeth Allen shares her perspective on advanced degrees for UX researchers, how she helps product managers to make the most of her work and how to improve the chances of your research having impact.
Highlights include:
- Top tips for running effective usability tests.
- How do you get stakeholders on the same page?
- How to avoid some common UX research mistakes?
Who is Elizabeth Allen, PhD?
Elizabeth Allen is the Founder & Principal of Brazen, a UX research and training consultancy based in Toronto, Canada.
She holds a PhD in cognitive psychology from The University of Chicago and has over 13 years of experience running UX research at companies such as Prosper Marketplace and Shopify.
Elizabeth is also an active contributor to the global UX community, speaking at conferences such as UXNZ, Web Directions and Interaction.
Transcript
- Brendan Jarvis:
- Hello and welcome to another episode of Brave UX. This is Brendan Jarvis, Managing Founder of The Space InBetween and it's my job to help you put the pieces of the product puzzle together, unpacking the stories, learnings and expert advice of world class UX research and product management professionals. My guest today is Dr. Elizabeth Allen. Elizabeth is a Toronto based researcher, consultant, and educator with over 13 years of experience helping organizations to uncover insights that fuel smarter design decisions. She holds a PhD in cognitive psychology from the University of Chicago and she has also lectured there in psychological and philosophical ideas of consciousness. Elizabeth is an active contributor to the global UX community speaking at conferences such as UXNZ, Web Directions and Interaction. She is the co-chair of the User Experience Professionals Association conference, a content advisor for the UX research conference, and also a mentor for Hexagon UX. Before starting Brazen her UX research consultancy, Elizabeth worked as a UX researcher at Shopify, the e-commerce platform that powers over 1 million businesses in 175 countries working across chatbot and point of sale products. Now she helps clients from around the world, including Mozilla, Instacart, and Shopify to challenge assumptions and shape better products through research. You can find Elizabeth brazen.io and on Twitter @ElizAllen_. Elizabeth, welcome to the show.
- Elizabeth Allen:
- Thank you so much Brendan. It's thanks to be here and also I am so rarely introduced as Dr. Elizabeth Allen. I appreciate that. It makes me feel very special today. Thank you.
- Brendan Jarvis:
- Oh, no worries. My wife's actually a doctor and I always feel like it's appropriate to give respect to the qualification. Look, before we get into the serious stuff, I read somewhere that you play the banjo and I figured that there's gotta be a story there. How did that all come about
- Elizabeth Allen:
- [laugh] that I actually, my banjos right behind me, but I, I'm not, I promise I'm not going to play it. Yeah, so this must have been maybe 10 years or so. I had a friend whose uncle was a banjo player and who had a band and was kind of exposed to it that way and I just really thought it sounded great and I was really interested in it and I would play with his banjo occasionally. And then I think for maybe some birthday or something, I treated myself to a banjo and been playing sporadically ever since. I live in an apartment so I can only play play so much without feeling like I'm bothering other people. So I'm not quite at where you'd expect for someone who's had a banjo for 10 years, but it's a nice very non-screen using nice very analog non-technology thing to do in my spare time.
- Brendan Jarvis:
- [laugh]. Yeah, it's so important to get a bit of time away from the screen, particularly when you work in tech like we do. Yeah. Obviously aside from aside from playing some banjo, you've got your own consultancy and you're also very active as I mentioned in your introduction in the UX community. And when I was researching for this interview, I obviously came across the Hexagon UX website and there was a quote on that website that really flawed me. I actually posted about that on LinkedIn recently and the quote was in the United States, which neither of us are from fewer large companies are run by women than by men named John. And like I said that my God. Yeah, it really flawed me. And I just wanted to ask you a little bit about what is Hexagon UX?
- Elizabeth Allen:
- Oh yeah hexagon UX is an organization operating in Toronto and I'm, I think a bunch of other cities around North America or if not the world which is basically a kind of mentorship sort of program. They might do other things as well, but I was involved in mentoring there and essentially pairing up women and people of marginalized genders with other individuals who can mentor them. And I think generally different aspects of tech. So in my case I was paired with a young woman who was getting started as a UX researcher and just needed a bit of coaching and advice along the way. And it was really great. It's a really kind of laid back organization, a nice way to make connections. I had a really good time mentoring. They kind of provide you with a bit of a framework for every week or every other week, what sorts of things you might wanna do together and that kind of thing.
- And there's a little party at the end. So yeah, it's just a really nice locally run organization that I had a great time being a part of. Cause I find that as a woman in tech it's really important to me to be at least somewhat of a role model for other people. And so that was a nice easy way to give back a little bit. One of my goals for the year was to try to do a bit more kind of volunteering or mentoring or something like that. And someone, it's hard to set that up on your own. So it was nice to have a kind of bridge to that sort of experience
- Brendan Jarvis:
- To have an organization that you believe in to plug into [affirmative]. What sort of things in your experience can people in tech do to better support diversity? Whether that's women or people from marginalized genders or different ethnicities? What sort of things can we do?
- Elizabeth Allen:
- Yeah, absolutely. I mean there's a lot. I think one is being the sort of place that those type of people would want to work at. So for example, having really strong codes of conduct in place. Having an HR team that's obviously committed to addressing issues with that. I find that [laugh] as a woman who has looked for jobs in the past, if I go on a company's website and I see that the leadership team is like 90% male or 90% white or whatever, that I'm probably not gonna be as interested as if there were a more diverse team. And so I think it's hard for organizations sometimes because they've gotten to a point where they have that kind of problem and they want to fix it, which is cool. But that doesn't mean that I as the kind of employee should have to sacrifice whatever environment I would rather be in for that cause if you know what I mean.
- So I think that that's important. And also another thing that I think a lot of companies do is talk about the pipeline problem. They're just, there aren't enough of those sorts of people out there that we want to hire. And the problem is that there aren't enough of you. Not that we are being exclusive with the way we hire or something. And that's almost never the case. There are certainly biases people have in terms of who they choose to interview, what kinds of decisions they make based on resumes, things like that. And a lot of barriers that people of different genders or racial backgrounds or ethnicities or whatever have to cross in order to just even be seen by these organizations. And I think unfortunately there's still a lot of education and work to be done around helping these companies understand that it really isn't our problem, it's your problem and it's your job to fix it. And we have a lot of benefits we can bring you as a company if you just take a second to consider us as you would. You're kind of straight white male individually typically tend to hire. So yeah, I probably have a lot more thoughts on that about all I could share at the moment.
- Brendan Jarvis:
- [laugh]. Yeah, no, fair enough mean, it just strikes me when you see quotes like that and I know that they're speaking in the broader context that people seem to be ignoring the research out there that proves that diversity is so much better for creative process, whether that's product or design and it, it's just how do we actually make that a reality? Like you say, visibility is a bit of an issue and one of the objectives I have for this show is actually to make sure that it's representative not of what we have now, but what we need to have in terms of the people that are getting the platform and getting some visibility. Yeah,
- Elizabeth Allen:
- Absolutely.
- Brendan Jarvis:
- So I wanted to ask you, I also was looking back at your UX NZ 2019 talk, which was a really great talk by the way. And you were describing your PhD experience as something that you quote wouldn't recommend to most people in quote, but you didn't elaborate on that at all. You sort just rolled into the rest of your talk [laugh]. And I just wanted to ask you tell me about that. What was it about that you wouldn't recommend to most people?
- Elizabeth Allen:
- [laugh], first of all, it's funny cuz usually when I'm giving a talk at a conference, I'm pretty much not retaining anything. I'm just talking. I can't remember what I'm saying. So it's good to know that I just kind of breezed past that and moved on to something else. Yeah, so I did a PhD in cognitive psychology. It took me about five and a half years, which is a bit faster than I think a lot of programs are. And that included a master's kind of along the way. And I personally had a great experience, I had a fantastic advisor. He allowed me to do whatever I wanted to do. I had a lot of friends in the program. For me, it all went quite well, but almost the whole time I was doing the PhD, I was like 99% sure that the outcome of this I wanted was to have a lab, be a professor, do academic research.
- And so that was the way I would get to that goal. Then about maybe six months before I finished my PhD, I kind of had a bit of a freak out where I realized this is not what I want. The academic job market's very difficult. I'd probably have to move somewhere that I didn't really wanna live. I wasn't necessarily so in love with the research problem I was focused on and it would be hard to change. There's other things I was thinking about and I managed to find UX research and that's obviously been a good career for me. But I would definitely say that you should only do a PhD if you really are so very sure that you want to have an academic career and that you're pretty sure you'll be able to get the job that will get you there. Otherwise I would suggest maybe doing a master's or something like that instead. And you could always maybe extend that if you really feel like you do wanna do a PhD. But in my case, I have no regrets. But if I had left after maybe two years with my masters, I probably would be kind of better off in my career than I am today. Cause I would've had those extra few years in the job market. So that might've helped a little bit more.
- Brendan Jarvis:
- Oh that's really interesting. That's really interesting that you say that. Just extra time that you would've had. Was there any particular person or moment that you recall where you had that freak out that prompted that
- Elizabeth Allen:
- [laugh]? Honestly, I don't recall, but I just remember waking up one day and being like, I don't know what I'm gonna do. I think one, certainly one factor at the time was I had a partner and I was like, well what am I gonna do? Am I gonna move to be with them or how are we gonna work that out? And that was something that was on my mind of course. And I think also just you know, spend five and a half years researching the same sort of idea and it just might not be as exciting as it was at the beginning. And so I think I just did some Googling around to be like, Hey, what can you do with the psychology PhD? And luckily UX research is a really good fit for a lot of people and I'm really glad that I found it.
- Brendan Jarvis:
- Yeah, so I mean obviously you bring a lot to it in terms of your academic background and the sort of rigor that goes into that level of research that you produce for your PhD. How important do you believe it is to have an academic background as a UX researcher?
- Elizabeth Allen:
- I would definitely say not so important that you need a PhD. That part is that's too much. You don't need that. Having, having some background is useful. I wouldn't necessarily say it's important. I think what it helped me with personally was getting comfortable with writing. So I had to write a lot. And being a good writer is important for being a good researcher cause you have to convince other people to take your research seriously getting comfortable with presenting stuff. So as part of any sort of advanced academic degree, you probably have to give talks and stuff to your lab or to at a conference or whatever. And that helped a lot. And also just mean honestly, just developing my kind of critical thinking skills, understanding how to frame a problem and what sorts of ways to approach solving the problem. Doing that in a formal way over and over again certainly helped a lot.
- But that being said, you can get those sorts of experiences in a regular job as well. It's not just coming from academia. I think that one other thing I guess that I feel like academia gave me was a kind of broader context to embed myself in terms of human behavior and psychology and whatever. I studied my particular problem, but I was also taking courses on social psychology and developmental psychology and philosophy and things like that. And that kind of all somehow fed into who I am today. And I feel like those were all beneficial experiences but at the same time I could have spent those three or four years after my master's at a job that could have taught me a whole bunch of stuff. I don't even know that I don't know now. So it's hard to say.
- Brendan Jarvis:
- Yeah, yeah, yeah. So now you are a UX researcher. How do you think of your purpose as a researcher?
- Elizabeth Allen:
- Ooh, that's a good question. I remember my first UX research job. I worked at a consulting firm called Centralis, which is in the Chicago area and I believe it was Kathy Kaiser, who's one of the partners who leads Centralis, I believe it was either her or Lyman Casey, her partner who said something. I feel like one aspect of my job at least is kind of eliminating those tiny little frustrations that occur in someone's day. It's really satisfying when you can, maybe you're doing some usability testing or something and you uncover a particular issue and the client or the company has no idea about this and you're like, oh, they can solve this problem relatively easily and it's gonna eliminate some little frustration from thousands of people's days. And that's really cool. That's one kind of aspect of that. It's more I mean it's a little bit more patting myself in the back I guess, but it's kind of a nice feel good [laugh], good aspect of what I do.
- But honestly I feel like mean instead, now that I've been doing this for longer and doing a lot more generative research and foundational research I find that what gives me a lot of the good feelings these days is this idea of a company coming to me and being like, we don't know anything about our users. We have this cool idea, or we have a start of a product, but we don't actually know who we're designing for and what they're like. And being able to show them those people holistically, almost like lots of aspects of their lives. Doing a really nice deep user interview project where I can kind of be like, Hey, here's all the things you should be considering when you're designing for this group of people. I find that is really rewarding and also usually really rewarding for the client, which is great because I find that feeling of giving a client their first experience with research is really awesome. So that's the kind of work I like to do a lot of.
- Brendan Jarvis:
- And I think it is important to have that clarity of purpose. And you mentioned usability testing, that sort of immediacy of the impact that you're having, even if it is effectively just removing problems and that if there were no problems, you wouldn't get that feedback, which would be success, which is quite a weird thing to consider that your job is basically to remove that friction at that level. Yeah. What other common misconceptions that you've come across that people have about what it is that you do as a UX researcher?
- Elizabeth Allen:
- That's a good question. People still talk about focus groups a lot and they assume that I do a lot of focus groups. I just talked about this today with somebody, and it's funny cuz not that focus groups are not valuable in certain specific contexts. I'm not saying never do a focus group, but that's very little. I think I've done one focus group project in the past three years. It's a very small amount of what I do. I think that when people who have never maybe worked with a researcher before hear about UX research, they think about the evaluative stuff, the usability testing and that kind of thing. Like AB testing, like, oh, I have a prototype and I need you to test it. Those kind of smaller projects that are in the solution phase, you already decided this is my solution now I wanna make it tweak it to make it better.
- Whereas really where I prefer to operate is more in the problem phase. Even understanding what the problem is and who we're trying to solve it for, that's like where I'm kind of more comfortable. And so I feel like people have maybe a more narrow view of what UX research is and what's involved. Let's see. Another thing that's been coming up a lot lately is this idea of throwing a survey at most of your problems, like surveys or something. Everyone knows what a survey is. Everyone's experienced a survey. It's easy for 'em to grab onto that as something that they should be doing. Which surveys are great. I'm not bad mouthing surveys at all, but they're not great for everything. And I as researchers know that surveys are great for getting a little bit of information from a lot of people, asking relatively simple questions, getting relatively simple answers.
- And it's not until you start to have a conversation with someone that you can really dig deep and kind of understand the things you would never have thought to ask in a survey. And so I think that a lot of people are like, well, with a survey I can get responses from thousands of people that's better than doing five interviews or 10 interviews or whatever. And in some ways it is to answer certain questions, but you're gonna miss a lot if you just jump into doing a survey. And so that's something I've kind of butted my head against a lot over the past couple of years probably.
- Brendan Jarvis:
- Yeah. And how successful have you been in shaping that conversation?
- Elizabeth Allen:
- I think pretty successful. I think one of the good things about being a consultant versus working in a house somewhere is I think I'm viewed as an expert and people are used to this idea of paying me for my expertise or my thoughts. And so they're probably more willing to trust me. Whereas if I was in house just working as a researcher somewhere, there's probably a little bit more arguing and pushback that's gonna happen, I think. And so it's worked out pretty well for me. Not every client goes for it, cuz usually interview projects are gonna take longer time and they're gonna cost more money than running a survey. And also, you of course run into those people who just are never gonna be convinced by something that's not numbers, they won't wanna know. 73% of people chose option C, right? They don't wanna hear the reasons why option C was important to them. They just wanna know that's what people say. And so that does come up sometimes. I think that over time you get better at having those conversations. And I think also over time, people, non researcher people get more exposed to research and they kind of start to understand. So they still might have a more narrow view than what I would want, but they're starting to kind of get better, I guess.
- Brendan Jarvis:
- [laugh]. Yeah. Yeah. So if you imagine that you're talking to an internal research team at the moment that's having some of these issues that you've just about, what would be some advice that you would offer them?
- Elizabeth Allen:
- Yeah, I mean, I think the number one thing that I do with kind of stakeholders that's most convincing is try to show them the risk that's involved in not doing the thing that I want them to do. So let's say we decide not to do this research or not to do research at all. What are the risks? What could go wrong if we make an assumption about this group of users and we're really wrong? Is it just a small thing? Is it just a couple people will be upset or is it like we could kill somebody? There's kind of a lot in that spectrum. And so having companies think through risks and realizing, Hey, wait a minute, there's a lot we don't know about and [laugh], if we don't learn this stuff, we're gonna be in big trouble. That's important. And there's an activity that I like to do with clients sometimes called the assumption slam, which is something I picked up when I worked at Shopify.
- And the idea is that you have a two by two grid and one axis is riskiness low to high, and the other one is knowledge or confidence that we have about this low to high. And the idea is you start to brainstorm all the kind of assumptions or hunches or hypotheses you have, and you stick those and post it notes on that grid and you kind of argue about where they should be placed and that kind of thing. And what happens is you have this quadrant that's high risk and low knowledge, and that's where you should really be researching. We don't know anything about this and if we get it wrong, we're screwed. So that's kind of a good way to, if you do that activity with a company they might be like, oh, okay, I get it now. That's kind of a good visual way to lead them through that. So yeah, an exercise that would be something I could recommend to an internal user research team
- Brendan Jarvis:
- That sounds really valuable. Was that called the Assumption slam?
- Elizabeth Allen:
- Yeah, it was called the Assumption Slam, and I'm pretty sure that the first time I heard about it was from JB Booth who works at Shopify. So I don't know if she got that from somewhere else, but I think that's the first time that I heard about it. And it's super useful and it generates a lot of arguments, [laugh], [laugh], which are usually good conversations,
- Brendan Jarvis:
- Hopefully in a cathartic way that everyone feels better afterwards.
- Elizabeth Allen:
- Yeah, yeah, exactly. Actually, sorry. Oh, sorry, sorry, go
- Brendan Jarvis:
- Ahead. No, you go.
- Elizabeth Allen:
- Just to add on, another thing that I find is really valuable for internal teams is if you're working with stakeholders who don't believe that there's this problem that needs solving, then of course exposing those stakeholders to the end users in a kind of easy and safe way can be great. Include them, let them watch the interviews grab quotes to put in your presentations, let people really see these are the real people and they're really having this problem. Definitely in the past, I've had situations where I've done some research and the team is really on board, but then I go to present and the CEO is there, and they've been totally not involved in the research at all. And now they're there and they're like, like, oh, well, you must have just recruited stupid people. Essentially [laugh] like us just recruited the wrong people. These are people that don't know how to use a phone or whatever. But then when I show them the videos, they're like, oh, okay, these seem to be typical regular people and they're having these issues, we've gotta fix that right now. So yeah, that's another technique I would suggest in that situation.
- Brendan Jarvis:
- So that's observing a usability test or something similar?
- Elizabeth Allen:
- Yeah,
- Brendan Jarvis:
- Yeah. We talk about it as this notion of making the user real as a way of I suppose, removing the bias or the assumption that goes behind some of those perspectives and getting people in the room is really valuable. So we've been talking a little bit about stakeholders, and I know that you've worked both internally and as a consultant. What are some of the key people in a product based organization that you need to connect with and get across the research? Who do you really need to build great relationships with if you're in a product based organization?
- Elizabeth Allen:
- Yeah, I mean, I would say number one is probably the product manager or whoever's making product decisions. That's the kind of closest or easiest for you to influence, which is often the product manager. I find when I worked at Shopify, I was kind of spoiled because we worked on these really great teams that were comprised of a UX researcher a data person, a designer, and a product manager, and also a content strategist sometimes, which is the perfect kind of UX bundle. And we would make a lot of decisions as a team, and it was super, super great. Not every organization has that, of course. I find that I feel most successful on projects when I have a good relationship with a product manager and if there are designers on the team, it's also really important for me to be involved with them too, especially if it's an evaluative project where they're creating prototypes or something.
- Of course, I want to be able to communicate my findings to them in a good way, but also sometimes a higher level than that. I find that especially as a consultant, usually whoever's hired me directly, the product manager or whoever has someone they're reporting to, or maybe even a higher level above that who's kind of like, oh, we're spending money on this consultant. I wanna keep an eye on that as well. And that can be scary sometimes. Not gonna lie, but it's also great because I feel like the chances of the results of my research actually making an impact and actually resulting in a new thing they go up a whole bunch if I have those higher level stakeholders involved. So it's usually hard. It's harder to get them involved, especially with the kind of day to day aspects. Usually they just show up for the final presentation, which isn't always the greatest. But if I can have that conversation with them and get them interested, then I'm usually much more confident that something's actually going to come of this. As opposed to other projects I've done either in house or as a consultant where a couple of years passed, then Oh, did they ever, what happened to that thing? And they just never did it. [laugh].
- Brendan Jarvis:
- Yeah,
- Elizabeth Allen:
- I think there's still some things from, I think there's still some things from my very first UX research job that never, you know, just check the website every so often. You're like, Nope, they just didn't. But that's okay.
- Brendan Jarvis:
- And that can be quite disheartening as a researcher, because a lot of effort and energy goes into running the research and then figuring out what the insight is behind it. It's something that I hear time and time again, and I think what you're saying about making sure that you are connecting with product people, but also with any sort more senior stakeholders is really important. And as a consultant, possibly part of survival as well in terms of it moving I to continue working for that company when you're seen in, yeah, yeah. You mentioned some of your previous places that you've worked, and I also discovered that you worked at Prosper, which is what I read anyway, was the first peer tope lending platform in the usa. You know, what advice, if you were to give advice to someone that was in a similar position of establishing the research practice like you did would you offer Yeah,
- Elizabeth Allen:
- Yeah, absolutely. So yeah, just for context for anyone who's listening so I was the first research hire at Prosper, which was a kind of, I guess medium size startup in San Francisco. So they did peer to peer lending or marketplace lending. So basically I want to borrow money and instead of borrowing it from a bank, I borrow it through Prosper. And my loan is funded by in investors all over North America, or at least least USA in kind of these little bit of little chunks. So they're kind of investing little bits in my loan and I get the money in the end anyway. And they hired me as their first researcher, which was the reason why I wanted that job. I love the idea of building a research team from the ground up and getting all these processes in place, and it was really fantastic, but it was also really hard and some things that were difficult. Number one was everyone wanted a piece of me, everyone wanted to do research, they never had research before, and they kind of knew that, oh, Elizabeth is here, we should be doing research, so I'm gonna contact her, but research and this. So I had to,
- Brendan Jarvis:
- Sorry to interrupt, this was 10 years in that existed for 10 years before you joined, and they'd never run any research,
- Elizabeth Allen:
- I guess. Yeah, honestly, it was a few years ago now, so I don't remember much about the kind of origins of Prosper, but yeah, they might have been around for 10 years, I'm not sure. But [laugh] certainly when I came around, however, there were probably at least a few hundred hundred people. They had definitely never done any research as far as I know. I'm sure they might have hired out for some consulting, but they never had an internal researcher. So everyone wanted to do some research, and I really had to set boundaries very quickly. And in my case, it really helped that I had a manager who helped me do that. He pretty much totally empowered me to say no to everything that I wanted to say no to, and saying no was the most important thing I did there. I basically said no to whole sections of the business, mean my manager was kind of like, Hey, this part of the business is what's highest priority right now and use that information as you will. And so there are these poor product managers who would be asking me for research for a whole year, and I just would say no every single time because they just weren't important enough to me, sadly, [laugh].
- Brendan Jarvis:
- So you were really popular,
- Elizabeth Allen:
- Maintain I was really popular, but I also had to maintain those relationships because I worked with them and I liked them. And luckily it was all fine. No big hard feelings there, I don't think. But just deciding, doing that work to prioritize which teams or what research questions are really important and saying no to everything else was very useful. I think also, almost like paradoxically, I also had the issue of everyone wanted research, but not everyone trusted research or thought research was a good idea. So I was dealing with both those things at the same time. And so it was also important for me to get some quick wins basically. And so I did that in two ways. One was running a really, really fast a evaluative research study on, I think it was just their website, I think maybe just the marketing site, just really quick.
- Within a few weeks of me starting there, I was like, okay, bam, here are some research results. And we learned from them and we made changes really quickly. So that was part of it. And the other part was making my research visible across the company in terms of, this is of course, before the pandemic, bringing users into the company, doing research in the office. We had a nice glass walled kind of box to work in, and I would bring people into these really testing or do interviews and everyone could watch. Everyone could see, I'm bringing the participant in Elizabeth's doing research today. And I think that was really helpful as well, to show really quickly, we are getting research done at Prosper. And so that was really cool. So I think that those things were really important. And also [laugh] documentation. So I had the opportunity to start from scratch, which is a really exciting opportunity.
- And so the company was already using Confluence, and because they were already using Confluence, I figured let's use their tool because I didn't want to make everyone use a new tool, so I created a Confluence area or whatever you call it, that was a repository for all the research and highlighted when this research was done, what the key insights were, and then here's all the information if you need it and kind of tried to organize it really well. So that was really a really cool experience. And then I think I only was there for about a year, and by the time I left, I'd hired one person and then ideally we were gonna hire more, but Shopify just reeled me in too quickly and I left. But [laugh], it was really cool. I mean, I think that I would not take that position if it's your very first research job. You should have some experience already and then you will hopefully be successful in a role like that.
- Brendan Jarvis:
- Yeah, yeah. So I mean, saying what you've seen, having been the first researcher at a company, setting up the practice, having worked at Shopify internally for a couple of years and now externally as a consultant for a range of organizations, what capacity have you seen that's most effective for the organization? Or is there a mix of capacities that can work?
- Elizabeth Allen:
- Sorry, when you say capacity, what do you mean?
- Brendan Jarvis:
- What model is most effective for an organization that's trying to do UX research, whether it's bringing in external consultants or an internal team or some sort of mix in between.
- Elizabeth Allen:
- I get it. Okay. Yes, definitely. So I would say that it seems like a lot of companies start with the consulting thing, which makes a lot of sense because you don't always have the money or the time to hire, hire an in-house researcher right away, or maybe you just don't have the need to just yet. And so that's a really good way for companies to start off. They can hire someone like me to come in and very quickly execute a research project, and that's usually enough to get people excited. And then maybe it builds momentum to hire a researcher down the line. I think that having one researcher in house is a great first step, but I think that ideally you would pretty quickly hire at least one other person to back that person up. I have friends who have been in the same situation I was in where they were the only researcher, and if there's a lot of demand at the company, it can get overwhelming pretty quickly, especially if they don't have the authority to say no to projects and they end up working crazy hours once the company gets bigger.
- Shopify for example, obviously a big company, it's nice if are, there's a researcher or a team of researchers who kind of focuses on one product area. So at Shopify we had what are called product lines, and I spent a lot of my time working in the retail section basically. So anything related to selling in person was Shopify. And it was just nice to be able to own the research for that particular area. There's a kind of pride in that. And also you just get to really deeply know the team. I think a lot of other companies operate where researchers are more internal consultants where they go off and do projects for different teams. And that's okay if you don't have, not every company has enough researchers, but if you're a bigger company, I love that I was dedicated to solving this one kind of universe of problems. I didn't have to worry about everything at Shopify, and that was really useful. So I think once the company's big enough to scale that way, then that's a really good place to be in.
- Brendan Jarvis:
- That does seem to be the two models that operate out there. It of like the research team is internal, but they operate as their independent squad and then they solve research briefs for product and when they come in or when they pitch ideas to research or there's the embedded model. And I can definitely see the benefits of both. I just wanted to turn our attention to your UX NZ 2019 talk, which I've mentioned I think a couple of times now, but I really did find it fascinating cause in that talk, you were speaking about some of the hop psychology myths that exist around color, which is a very, it's a very specific thing, but in that you gave an example of an AB test and early AB test that had been done within an interface where there was a change from a green button to a red button, and you were using that to highlight context as being really important in interpretation of results. And in this case, I think it was a 70% increase in clicks had happened because they changed to a red button but largely because the interface was all green, lots of elements on the interface were green previously, so the button sort of blended in and now contrasted and stood out. And so it seemed like you were worried that there was a tendency or at least a reality out there, that there was a bit of skin deep analysis going on and people were ignoring context or oversimplifying the results of research.
- Elizabeth Allen:
- Yeah, I think that when people recognize that they have a problem, they want to get to that solution as quickly as possible. So I'm not surprised that people will grab onto what they think is the simplest and easiest way to get to the bottom of something. And in that particular example, this was a situation where someone did this AB test compared a green button to a red button, found the red button was better and was like, okay, red is the best color to use for these buttons without, as you mentioned, without considering the broader context in which those colors were displayed. And there was all this green stuff in the background, so the green button kind of blended in and that was a very likely alternative explanation for this. And if they tried other colors, maybe they would've found that orange is just as good as red, or blue is just as good as red or whatever.
- So I, I think people are drawn to that kind of simple AB testing or really simple evaluative research because they think they what the problem is and that's the quickest way to solve the problem. So yeah, I, and I don't blame these people. If I was a product manager or someone in that kind of situation, I might be inclined to run the same sort of test. They might not have thought of the surrounding context. I think that's why it can be helpful to have teams where maybe one person is not going off and doing this stuff on their own, but you actually get collaboration and bring this up in a meeting or something and talk about, Hey, if we were to run red versus green, what could we actually conclude from this? Could we conclude that red is the best color? Or do we just know that not green is the best color, for example, and where should we go from there?
- So yeah, [laugh], that talk was really fun because I got to talk about all this cool color stuff, which I don't think about very often because my degree was such a long time ago. But it's just kind of interesting to take that those little bits of academic knowledge and try to dive into how they can apply to real problems that product teams are facing every day. And that was just like when I saw that example, it's in a whole bunch of blog posts and stuff and I'm just like, oh, come on, this is, there's such an easy alternative explanation for this. Come on, think harder.
- Brendan Jarvis:
- Yeah, I had a nightmare that I was gonna wake up one day and fire up the internet and find that all interfaces had red buttons and [laugh]. That was just how design was going to go. But thankfully that hasn't happened. But I think it's really insightful that question of, I think you said, do we really know that it's red buttons perform better than green or is it that any other color could perform better than green? And that sort of level of rigor that you put into the questioning that goes around these decisions. And it seems to me that there's a huge pressure that particularly product managers face around the roadmap and the backlog. And I wondered if you had any more insight into that having worked at product organizations as to what other sort of behaviors do you suspect might drive that kind of rush to implement solution without necessarily thinking too much about the reasons why?
- Elizabeth Allen:
- Yeah, I mean find that, I don't know every product manager, but I find that many of the product managers I've worked with are just trying to do too much in a year or a quarter or whatever. And that's not their fault. They're, they're obviously being told to do certain things and so often when I look at a roadmap, I'm like, can we cut this in half? There's just no way to do all these things very well, especially if you want research to inform them. I often end up in these situations where people want research immediately because next week they're supposed to make a change or whatever. And I'm like, no, no, if you wanna actually do the right research, this is research that needs to happen. If it's generative research or more deep foundational research, it has to happen before to inform the roadmap like that.
- We don't even know what the roadmap should be. You have to do the research first [laugh]. And that can be really hard to get across in a maybe less mature product organization, really the product manager and or the product director or whatever. And the researcher should be paired together because the research should inform what the product direction is and kind of vice versa. And so I think I have a lot of sympathy for people who are working under those kind of roadmaps and deadlines because yes, it is very tempting to just do the bare minimum research you need to do to tick the box of, okay, boss, I did some research and it just sucks because doing bad research is doing worse than doing no research. If you do poor quality research or research the wrong thing, then who cares? I think it was Erica Hall, I might be getting this wrong, but I think it was Erica Hall that said something like if you set up your shop in the wrong problem space, then who cares what your solution is? It's a solution to the wrong problem. And so I think that what a lot of companies don't do is kind of spend more time in that problem phase and noodle on that for a longer time before jumping into solutions at the same time. I understand because companies want you to do things and make money and it's hard to be like, let's just sit and think about this for a few months. [laugh]
- Brendan Jarvis:
- Mean this is a real area of tension that exists between product and research. And I think from what you were saying anyway, it sounds like having a bit of empathy for the product manager and understanding the pressure that exists around that need to deliver, particularly when you've gotta generate revenue or stick to a roadmap is really key. But it also sounds like maybe the conditions that create this tension between research and delivery effectively outside of both the product manager and the researcher's role of responsibility, and perhaps they exist somewhere else in the organization and there's a job to be done there.
- Elizabeth Allen:
- Yeah, absolutely. I think mean a lot of companies have grown without any sort of research culture baked into them and trying to tack on research later involves changing a lot of mental models and changing a lot of behaviors that are deeply ingrained. And I mean, the product manager, him or herself might be totally on board with doing the research, but if they're getting pressure from someone above them to move more quickly and they feel like they can't push back on that, then that's gonna happen. They can't really do too much about that. And it's unfortunate.
- Brendan Jarvis:
- Yeah. So on that note, as researchers, what could we do to help the product managers more effectively manage that and I suppose enhance the maturity of the organization over time?
- Elizabeth Allen:
- Yeah, I, when I started a new company, I like to do stakeholder interviews, so I would hopefully have a decent relationship with the person above the product manager. They at least know who I am and why I'm here and why they should listen to me. So I think building those relationships will be important. And I think also just developing the relationship with the product manager themselves, so you kind of make sure to back each other up or inviting each other to meetings where if the PM knows that he's gonna have to make an argument that's gonna be difficult for them to make a loan, then they should bring me along. And I can kind of speak to the research aspect because probably the best way to convince the stakeholder to do the research is to talk about risk and what happens if we don't do the research and maybe I'm the best person to speak to that and we should have that relationship going so I can be involved in that part as
- Brendan Jarvis:
- Well. Yeah, I really like that. And I think that's a really important point because both roles are effectively and everyone involved in a product is trying to deliver a world class result. And if you can back each other up and see, rather than the tension existing between yourself and the other party, and rather it's something else in the organization to address together, I think that's quite a powerful way of framing it. So we were just talking a little bit about the maturity of different organizations and their response to research. There's something that I've observed a lot here in New Zealand at least, is that it seems that there are a lot of UX designers who have possibly come from UI at least initially, that are running research themselves. How do you feel about that
- Elizabeth Allen:
- In general? I feel great about it because the more people who are familiar with research and getting their hands dirty with research, the better. But there's some caveats to that, right? I think a lot of designers are pressured to do this sort of research without really having the kind of training or background and that's obviously not so great. I think also companies sometimes think they can solve their research problems by just having their designers do usability testing or ad testing on the stuff they're creating, but they're not taking the time to have someone come in and do the more generative or foundational work that kind of feeds down. So in general, I think it's great if a designer is willing and able to do their own evaluative research, that's kind of a nice piece that they can take and do on their own but the idea of them also doing foundational, longer term, deeper research, that's probably gonna take too much time and effort they have their design job to do, that's probably gonna be pretty difficult.
- So yeah, I mean, I'm all about people doing research as long as they know what they're doing. To some extent, [laugh], like for example, right now I have a client who my client's, an agency who has a large corporate client who they're running a program right now to teach people in the corporate client how to do design thinking sort of activities, how to run certain types of research, how to do design thinking stuff. And they're actually working on their own problems to solve. They kind of have a project to work on and it's great. I'm teaching this, I spent this morning teaching journey mapping to these people who are like everything from IT people to working in the stores or the branches of this company, people who are not involved in research at all, but they're all getting a chance to do interviews and do journey mapping and whatever. And I think that's great cause they're gonna take that and bring that to their role in some sort of way or bring that to another company they work for. And it doesn't mean they're gonna be research experts, but they're gonna have a lot better background than someone just kind of coming in off the street and trying to run a usability test. Yeah.
- Brendan Jarvis:
- And teaching something that seems to be quite a big part of your career. I mean, obviously in academia you are a lecturer and I know you've got your own training that you deliver. I think you also work through O'Reilly Media, so that's obviously quite a big name in terms of training and other sort of insight that they provide. And you've got your UX NZ usability testing workshop, I believe it's called mastering usability testing. Is this something that's designed for a broad audience or for designers? I mean, who really are you trying to reach through this training?
- Elizabeth Allen:
- Yeah, that's a great question. I mean, I think of the audience for this research to be people who have not necessarily done any of their own usability testing before, maybe have, but really have a strong grounding in it, but either need to do it for their job or kind of imagine that in the future they would want to do it for their job. So the idea is that it's a half day, three hour workshop and we go through everything from defining your research questions, understanding why you should be doing this sort of research in the first place, to coming up with how we're gonna structure our tasks, how we're gonna recruit participants when we actually run the sessions. How do you get someone to open up and talk to you? How do you not be biased in the questions you're asking [affirmative] when you get the data, what do you do with it?
- How do you make sense of all that data? And then how do you turn that kind of synthesized data into, okay, here are the five or 10 top insights and recommendations that kind of come out of this. So basically everything from getting the first idea to maybe do some research to, okay, stakeholder here is what we should do as a result of the research. And so I think I've taught this sort of thing a bunch of times before. And typically I find designers, product managers, sometimes content strategists, sometimes developers and definitely newer UX researchers or people who want to transition into UX research as a career.
- Brendan Jarvis:
- Yeah. That's really cool. So that's next month, I think it's on the 17th or the 18th of November. And you're delivering it on online, am I right?
- Elizabeth Allen:
- Yeah, so I mean, in an ideal world, I'd be in New Zealand. I was, last year, I actually taught the same workshop at UX New Zealand last year and it was wonderful. And this year of course not doing that. So it's gonna be on Zoom. Yeah, it's at a reasonable time in both New Zealand and in North America. So hopefully everyone in Europe will be able to [laugh] attend that too. Hopefully it'll be a decent time for most people [laugh].
- Brendan Jarvis:
- And if you were gonna give people some sort of top tips or insights that they might get from that workshop, what would say your top three tips be as a result?
- Elizabeth Allen:
- I think writing good questions is really gonna be useful, and that I find that that's probably the biggest benefit. Cause you can use that for not just usability testing, but also interviews or surveys and other kinds of methods. How to ask a good question is actually harder than it sounds and getting people to open up in the session. So building that rapport with a participant, making them feel comfortable listening actively and responding to their body language, that kind of stuff. So that kind of in the session, how do I think and respond to what people are doing, that's gonna be a big chunk of what we talk about. And I mean, I guess just in general, the importance of planning. I think most of what I talk about in the session is here's how to plan for X, Y, and Z, which is not the most exciting thing to talk about all the time, but if you have a solid plan going into it, you're gonna have a good result.
- The worst thing to do is to do a usability test or whatever type of research. And then you do all that work and then you're like, oh, wait a minute, why didn't we ask about this? Or why didn't we think of this? And that's cuz you didn't have the time to plan or you didn't take the time to plan. So making sure you come up with a research plan that has your goals in mind, your research questions in mind, and then figuring out how you're gonna get the answers to those research questions through the testing is a really important part of this.
- Brendan Jarvis:
- So what sort of mistakes have you seen when people are planning or running research?
- Elizabeth Allen:
- Yeah, I mean not involving stakeholders. So coming up with a discussion guide for an interview, for example, but they haven't gotten the feedback from the people who are gonna actually be consuming the results of the research. So they go and they give their presentation at the end and they're like, some CEO or whatever is like, wait, but this weren't, why didn't you ask about the thing? And you're like, oh, I didn't even know I was supposed to ask about the thing. And that's obviously a big deal I think not knowing how to deal with people's reactions in testing. So this hasn't come up that often and certainly less with usability testing than interviews, for example. But if someone gets emotional or upset during the interview for whatever reason, how do you handle something like that? That's something that people certainly make mistakes with. If someone starts crying, not even acknowledging it and just pushing forward.
- Or if someone is not giving you the responses that you want, ending the session and not giving them an incentive or something like that. These are kind of things that hopefully most people realize they shouldn't do, but people do it. And also just a big mistake people make is doing all the work to do the research to get all the data and then just not doing anything with it. We did the research, we have a hundred pages of notes, but oh, we don't have the time to actually piece through this and get the insights that are there. So that is definitely a pitfall.
- Brendan Jarvis:
- [laugh]. Yeah, I mean it also sounds like a massive waste of time and energy of everybody involved. Yeah. So we were talking earlier, you mentioned how it was important to use the right methods for a study and that sometimes people come to you and they think, oh, let's run this survey, the survey's gonna solve all of our problems. And it sounded like you were able to walk them back from that solution before you get going. So when someone comes to you with a brief or with an idea that they need to run research, what is the first thing that you do?
- Elizabeth Allen:
- Yeah, I mean, the first thing I wanna find out is what are the goals of the research, regardless of what method you wanna use, what are you actually trying to learn? And maybe that does not match up with whatever method they have chosen, which is totally fine. A lot of people haven't done research before. They know maybe a couple of things they're familiar with, and so maybe to them it's totally reasonable for them to, I don't know, do a survey to find out what kind of workflows people use to open a Word document or something like that. But that's not really a great survey question. Probably [affirmative] what a weird example I just picked off the top of my head. I love it. Yeah, [laugh] clearly staring at Word right now, my [laugh]. So I I find that, yes, that part's important.
- I think that trying to help them think through not just what the goals of the research are, but what the outcome they're gonna do with the outcome of the research. What's the point of doing, for example a nice really broad foundational research project? If you already know that you are being gonna be forced to design this very specific solution, maybe there's no point in spending the time doing that really great research if you're gonna not be allowed to use the results. So that's something to keep in mind. Luckily I find that most people are not in that kind of situation, but I mean, I think that some people just don't know what kind of methods are out there. And luckily, as someone who's done a lot of this, I can use my expertise to kind of be like, Hey, a survey is great. Let's do a survey later because of these reasons. But I think let's start by talking to some people, cuz there's gonna be some things we'll learn through these conversations that will inform our research, our survey questions, and make that survey much stronger if we do that later. So just kind of talking through my rationale I think is important
- Brendan Jarvis:
- And this is really valuable and you're obviously someone who has some depth of experience in both qualitative and quantitative methods.
- Elizabeth Allen:
- [affirmative]
- Brendan Jarvis:
- Is qualitative research not taken seriously enough?
- Elizabeth Allen:
- Sometimes it's getting better. I think it's getting better. It also depends a lot on the company. Of course. I think that a lot of people bulk at qualitative research at first cuz of the small sample sizes. I mean, people are like, what? We're gonna do eight interviews? Why don't we survey 10,000 people instead? And so you have to help 'em understand. Both methods are great for answering different questions. Yeah, I mean definitely even if I have a company or a client totally all in qualitative research, they still want to quibble sometimes about exactly how many interviews we're gonna do or exactly how many sessions we're gonna do. And the way I tend to pitch it as is we should keep doing these sessions until we stop learning new things or stop learning enough new things. And so maybe we start with six or eight or whatever and we can always reevaluate and be like, you know what? This is still really interesting. Let's keep going. So yeah, I think it's always gonna be a challenge for some people to accept qualitative research, but I think it's getting easier the more people that are exposed to it.
- Brendan Jarvis:
- So Elizabeth, what is it that you enjoy most about being a US UX researcher?
- Elizabeth Allen:
- I think I mentioned somewhere earlier in the interview this idea of showing clients their users or their perspective users for the first time. That's probably the most exciting thing I love when I do a project and a client is I feel smarter now. I feel like I've learned something that I wouldn't have learned before. That's the number one thing. I, I mean, love my sessions with the participants themselves. Tomorrow I have four user interviews scheduled, which is probably plenty given that I have other meetings during the day as well. But I'm gonna enjoy those four conversations. I get to talk to people who I wouldn't normally encounter necessarily. A lot of the projects I do are with people that I just don't meet on a daily basis. I did a project last year where I spoke to people who coordinate clinical trials for hospitals. For example, I would never get to learn anything about being a clinical trial coordinator, but I got to talk to probably six people who did it and it was really cool and a very complicated field to be in.
- So I enjoy that. I enjoy, in my case, because I'm a consultant, I enjoy working in different industries. So I get to become an expert in peer-to-peer finance for a little while and then I move on to healthcare and then I move on to e-commerce or whatever. And that's really cool. I find that I've amassed breadth of knowledge that's been really useful. And in particular because I've happened to work on a lot of e-commerce projects and worked with Shopify a bunch. That kind of gets me a reputation, which gets me more work in and I kind of get to further refine that expertise, which I enjoy.
- Brendan Jarvis:
- Yeah, absolutely. And if you had to label one thing as something that you enjoy the least, what would that be?
- Elizabeth Allen:
- [laugh] recruiting. I don't really enjoy. I mean, yeah, recruiting I don't enjoy because there's this kind of waiting period where I have people I want to recruit and I have to wait for the responses and I have all these slots to put on my calendar and I'm waiting for people to sign up and I hate blocking myself that way. Or maybe I have to get a third party recruiting vendor to help me out. And then I have to get a big chunk of money for my client to help with that. And that's kind of annoying. I find that recruiting is just like the, I'd rather not deal with it, but I have to
- Brendan Jarvis:
- [laugh]. Yeah, that comes up so often actually. Yeah, I'm think, I'm not sure anyone enjoys recruiting even the recruiters.
- Elizabeth Allen:
- No, [laugh].
- Brendan Jarvis:
- So what I thought we'd do is now that we're sort of winding down our chat, I thought play a fun game. Well, hopefully it's fun. Okay, well you, you'd be the judge of that. And it's called what is the first word that comes to mind? And so I'm gonna say a word and I want you to reply with the first word that comes to mind for you.
- Elizabeth Allen:
- Okay.
- Brendan Jarvis:
- Are you ready?
- Elizabeth Allen:
- I think so.
- Brendan Jarvis:
- Okay. UX research,
- Elizabeth Allen:
- Happy
- Brendan Jarvis:
- Product management.
- Elizabeth Allen:
- Confusing.
- Brendan Jarvis:
- Nickelback,
- Elizabeth Allen:
- [laugh] Canadian, unfortunately.
- Brendan Jarvis:
- [laugh] [laugh]. So you've run a lot of research, been with a lot of participants, I think you mentioned, and I'm not sure if it was an example that a participant may have cried before. That's obviously not funny. But what is the funniest thing that you've observed as a user researcher?
- Elizabeth Allen:
- The first thing that comes to my mind, which honestly I shouldn't even say is funny. I feel kind of bad saying it's funny, but I did an interview with someone who presumably ran a small business of some store, but I didn't know much about it. And during the interview, he basically confessed to me, I think he was telling the truth that he was some sort of hired assassin for the government and that he had had a very dark past life where he had killed people. And I mean, again, I shouldn't say this is funny because I don't know if it's true. It's not funny. And also it's not true. It's also not funny. So take with that when you will. But it was certainly the most memorable, I guess session that I had.
- Brendan Jarvis:
- [laugh]. Yeah, it sounds really weird. And I suppose you gotta be pretty prepared for that sort of stuff to happen when you're dealing with people that you don't, don't know.
- Elizabeth Allen:
- Yeah, exactly.
- Brendan Jarvis:
- All right. So if you could give yourself some career advice. So think of the first day that you started as a UX researcher. If you were able to tweet yourself one tweet of advice to you on that day, what would you say?
- Elizabeth Allen:
- I would say be open to learning. Speak slower, stop interrupting people. [laugh]. I love it. Cause I definitely, I, I've gotten better over time, but I definitely had more trouble at the beginning doing the kind of active listening and not trying to jump in with my next question. Cause I was just nervous basically. And I still struggle with it sometimes, but I've gotten better.
- Brendan Jarvis:
- Yeah. We all, well look, Elizabeth, it's been so great having you on the show. I've really enjoyed the conversation. Thank you for being so generous with your experience and your expertise. There's so much great advice here. I really can't wait to go back through this video and prepare the show notes for everyone. So to everyone that's tuned in, thank you as well. Everything we've covered today, including Elizabeth's workshop at UXNZ next month in November will be linked to and all the other fantastic resources that we've mentioned. If you have enjoyed the show, please remember to subscribe to the channel and to comment on this video and we'll keep you up to date with all the other amazing interviews we have lined up. Elizabeth, is there anything you wanna say before we finish up?
- Elizabeth Allen:
- No, I don't think so. Just thank you so much, Brendan, for the opportunity. It's just nice to meet someone who actually remembers a talk that I did and wanted to speak to me because of it. This is very rare and it's just nice to talk to you. So thank you so much.
- Brendan Jarvis:
- Wonderful. Thanks Elizabeth.