Steve Bromley
Building Effective User Research Teams
In this episode of Brave UX, find out how Steve Bromley became one of the UK’s leading user researchers by building trust in UX research and dispelling common myths about it.
Highlights include:
- How do you build trust in user research?
- What are the challenges of running collaborative studies?
- What do you tell a product team when a tester falls asleep?
Who is Steve Bromley?
Steve Bromley is a UX researcher, consultant, author and educator. He was a lead user researcher at Playstation, where he worked on numerous AAA titles including Horizon: Zero Dawn, No Man's Sky, and LittleBigPlanet 3.
Since leaving Playstation, Steve has established user research teams for the UK Parliament and Reach plc, the UK’s largest commercial publisher. Steve holds a Masters of Human Centred Computing from the University of Sussex.
Transcript
- Brendan Jarvis:
- We all wanna create amazing products, but how this show helps you to put the pieces of the product puzzle together, unpacking the stories, learnings, and expert advice of world class UX research and product management practitioners. I'm your host, Managing Founder of The Space InBetween, Brendan Jarvis, and you are tuned into the very first episode of Brave UX. I hope you find today's conversation useful in your quest to create great products. My guest today is Steve Bromley. Steve is a Brighton based UX researcher, consultant, author, and educator who helps organizations to start user research teams and run effective research. Steve holds a masters of human centered computing from the University of Sussex where he has since returned as a guest lecturer. He's also guest lectured at the University of Ontario Institute of Technology, presented at numerous conferences including games ur, the research thing, UX Brighton, CHI play, and he has also had his work featured at GDC and the developed conference for most of the past decade.
- Steve invested his time working at PlayStation where he was the lead user researcher on numerous AAA titles including Horizon Zero Dawn, No Man's Sky, and Little Big Planet Three. While at PlayStation, Steve founded the IGDA games User research mentoring program, which is connected over 150 students with more than 50 industry professionals from an impressive array of top gaming companies such as Sony, EA, Valve, Ubisoft and Microsoft. After leaving PlayStation, Steve went on to set up user research teams for the UK Parliament and most recently for the UK's largest commercial publisher reach plc. Steve is also the author of the book Building User Research Teams: How to Create UX Research Teams that Deliver Impactful Insights, which is available on Amazon and Book Depository and is an e-book for Kindle. You can find [email protected]. That's B r Om, B R O M L E Y, and on Twitter at Steve Bromley. Steve, welcome to the show.
- Steve Bromley:
- Thank you very much for having me Brendan, and thank you for a lovely introduction.
- Brendan Jarvis:
- Oh, you're most welcome. Now Steve really curious, did you always want to be a user researcher because it doesn't strike me as something that's top of the list for most kids.
- Steve Bromley:
- Yes the answer is no. I didn't know existed for a long time. I think I was lucky and stumbled upon it. So I originally did my undergraduate degree in history which is obviously not particularly related to user research or software design or games design. I quickly found after graduating that actually it's very hard to get a job doing something directly related to history. It's one of those jobs, one of those degrees where they chuck you out into the real world and say, cool, you can do anything. Off you go [laugh]. And I ended up on a graduate scheme for American Express, the credit card company for their tech graduate scheme. As part of that, they asked you to pick a master's to fund and there was a choice of one that was very codey and very developer and then human centered computing, which I hadn't heard of and wasn't familiar with and I thought although I'd done a little bit of web design and things like that before, perhaps being a developer wasn't the right route for me.
- And so I picked the other one mainly because it wasn't going to be quite so heavy on the development. And then I was just very lucky at the University of Sussex, there was a lecture at the time Dr. Graham McAllister who was also spinning up his own startup at the same time called Vertical Slice. They were the, as far as I know, the first company that does play testing for video games as an agency. And I, yeah, that was revolution to me. I didn't know that, oh, you can get a job doing usability research, which I didn't at the time know what it was on video games. That's incredibly exciting and I think that was the wake up call for me. So I was very lucky that I did my dissertation with Graham McAllister. He hooks me up with a local games company inside Brighton and Relentless, I dunno if you recall, and the place two days there was the game called Buzz, which was the close game and everyone had a buzz in, you have to buzz in with your answers. They were making that game and they were thinking about future games and I was lucky enough to do a project with them as part of my masters. And then after that, yes I was like, okay, the world is my oyster. This is a field which is super exciting. You get to work on games that people love. You get to make a difference about games by doing usability and user research and I think that set my path ever since.
- Brendan Jarvis:
- Tell me, you said you get to make a difference by doing the usability research and gaming. What sort of difference is that? What have you seen?
- Steve Bromley:
- Yes, so a thing that surprised me all ever since I've been a user researcher, not just for games but since is even when the developers think, oh I've made this thing and it's fine and I can't anticipate there being any issues, you always do find usability issues. You find there are bits where players haven't understood where they're meant to go in the game. They haven't understood the objective, they don't understand how to use the features in it. Just the same as on websites where you see people don't understand which button to click or they're not understanding what information this field is asking for. And you see as when you are running user research, you see the impact that has on players. They are confused, they get bored. Particularly when we did a lot of tests with kids when I was at PlayStation, [affirmative] kids do have no reservations telling you, oh I'm bored, don't want to do this anymore. Can I go home? Which is sad, want to have been brought to play computer games that they want and go home. And by being able to find these issues throughout development, you can fix them obviously before real players in the real world see them and have a real impact on that player experience. Making sure that players do understand what they're meant to do, that they aren't interpreting the fun in the way that designers designed and there are no barriers to creating the right user experience that the developers want either from their game or from their software.
- Brendan Jarvis:
- You mentioned that chil children are fairly open and honest and somewhat brutal in the feedback that they provide. What of did you have to give any of the observers that may have been involved in the creation of the games before they heard that?
- Steve Bromley:
- Yeah, that's a great point. So perhaps we didn't warn the developers beforehand about how brutal kids would be. And a problem is a lot of developers, well lots of people have kids of their own, but if you are a developer you might not encounter real kids that often really playing games in a real context. And so it can be quite surprising, I think I've kept this somewhere. We were playing a platform game with some kits and halfway through the session a kits wrote a tiny notes on a piece of paper and then put their hand up very shyly and handed it over to me and I was like, okay, what is it what we're going to see? And I went up and the note said, can I play Mario please? Which is very cute, but again, not what a developer would want to hear, but probably what a developer needs to hear, they would much rather know before they finish the game, oh we need to change something because this isn't engaging kids, rather than finding that out in the reviews or after the game launches. But yes, they can be quite brutal.
- Brendan Jarvis:
- Not the place for the fragile ego.
- Steve Bromley:
- Exactly. Perhaps any user research session you have our risk of having your ego damage if you feel particularly attached to the thing you've made.
- Brendan Jarvis:
- Yeah, I think I'm gonna have to improve our guidelines and briefing [laugh] to make sure that those are properly expectations are properly managed. Steve, you obviously you've invested a lot of energy into education and investment of professional, your professional career in user research and it seems to me looking through your bio and the amount of work that you've done speaking at conferences, obviously your masters and all the other associated things you've done such as the book you seem have a very strong passion for knowledge and for sharing that knowledge. Is that something that's always been the case for you? Where does that come from?
- Steve Bromley:
- Yeah, I think that's something that struck me about the gamed user research community. When I joined myself as a junior researcher who knew nothing about any of this perhaps related to how secret games are about the actual game being made and the contents of the thing we're working on, the community is in contrast that very open about okay, here's the methods we use and here's some best practice and here's some ways that you can get better insight from your sessions. Which I think is a great ethos to have. Obviously user research is still somewhat in its early days where a lot of teams, both games teams and also software development teams don't yet consider it a core part of their development process. And some that do things like usability testing aren't taking advantage of. You can do research early in development and that has some benefits as well.
- And so a lot of places have reasonably low research maturity and I think the impact of that is that the community recognizes that we should be stronger as a whole and we should share our approaches to try and raise the bar of research across the whole industry. There's a lot of good work being done in the game user research community by volunteers to create conferences, create community events, create training things. And I think I've just wanted to be part of that. I've seen the benefit of let's all work together to help raise the bar of where user research adoption is in the world. And it's been nice to have the opportunity to do that through some of these different mediums like presenting at universities you also for a little bit used to present at schools to try and for younger children give them an idea that this is a career that you could do. So it's not such a surprise that it was for me and things like writing the book or other initiatives.
- Brendan Jarvis:
- And what was it in particular that inspired you to set up the games user research mentoring program? Cause that seems to me, I mean that just reading about that and having a look through that website, I mean it seems to have had quite an impact in the lives of quite a few young people took some quite serious studios.
- Steve Bromley:
- Yes. Again, we've been very lucky that people from these triple A studios from games that everyone's heard of have volunteered their time to do this. I think it was that I recognized there are a few gaps in that journey from how do you have no job or just leaving school to how do you start working in a company like Microsoft or Valve or ea? Some of the problems are, there's just not enough knowledge that what games user research is or what good user research practice is. And a lot of the concepts that user research covers are similar in, as you say, sounds similar to things like market research, which has been around for a lot longer as and because of that it can be a bit unclear for someone joining the industry, what actually does a user researcher do? And also what can I do to develop my own skills to be the right person to get these kind of user research jobs.
- One of the challenges that I think people leaving university have is getting that first job or getting a portfolio of projects together so they can't show, oh I can do usability testing or I can run these studies and I would be a suitable hire for some of these jobs. And that's particularly difficult for things like user research because you need some users to do it. Usually you have to pay users to turn up and someone who's just left university has no fund or ability to pay five people to turn up for usability session so that they can have a practice piece of their portfolio. And so a lot of the work that our mentors do have been running through projects, setting tasks related to each bit of the research process for our mentees and then giving them feedback on the work they're doing and just giving them the opportunity to create some portfolio pieces that they can then use for job interviews, which I think has been very valuable. Lots of our mentees have gone on to work in the game industry, lots of others haven't and have gone to work in user research more generally, but there is a degree of crossover between those skills. So again, that seems like a win as well.
- Brendan Jarvis:
- You see that was valuable and I think it's probably invaluable, particularly in the current climate with the world largely in recession or if not heading into it and younger people are gonna find it more and more difficult to enter into the industry. So I mean look, it's really fantastic initiative that you've put together there and I'm sure it's made a big difference and in the lives of a lot of young people and will continue to do so.
- Steve Bromley:
- Yeah, hopefully. So
- Brendan Jarvis:
- Look gaming you worked at PlayStation for I believe five years mean that sounds like a dream job if you're a gamer, which I know that you are cuz we were chatting before the show. Is it even a place you can work if you are not a gamer?
- Steve Bromley:
- Yeah, I think there are some challenges for people who are not familiar with games to work in a game setting like PlayStation. I guess the biggest one is you have to know some of the references that your participants are gonna talk about. If you are playing a game and the participant is you're interviewing a participant and they say, oh the inventory system in this game is like Halo's infantry system. I think you as a researcher need to know enough about the field to be able to recognize what they're saying and what that means and then communicate that to a gains developer. And I guess that's another one of the challenges that to have a degree of credibility in front of a game development team, you have to know enough about what you're talking about to be able to have a conversation about games or games development. Luckily a lot of people like games very much and lots of people very enthusiastic about it. So I think there are people out there who care enough about games to want to do it and PlayStation is a very exciting place to work. They have lots of parties, you get free games, you get to go to launch events and things that are really exciting things but perhaps you do have to be a gamer to enjoy that as much as possible.
- Brendan Jarvis:
- Yeah, the credibility is code word for a misspent youth depending on how you look at it. You mentioned that
- Steve Bromley:
- You go
- Brendan Jarvis:
- On. No, please continue.
- Steve Bromley:
- I remember if you're familiar with Gary Larson, the far Side comics and there's one where there's a kid playing on Mario Gaskets from the eighties and his parents are imagining the jobs he's gonna get paid 300 pounds to play games a day and all these things. And the joke is meant to be that obviously it's unrealistic for play for their parents to think, oh kid can get paid for playing video games. But actually it can happen for things like working at PlayStation or being a game Jud researcher, that kind of background of spending all your youth playing video games is super valuable.
- Brendan Jarvis:
- Or even eSports, which I mean some of the prize pools are rivaling some of the real world sports.
- Steve Bromley:
- Yeah, great point. Yes, that as well.
- Brendan Jarvis:
- So I mean PlayStation sounds like an amazing place to work. Was it hard place to leave?
- Steve Bromley:
- Yes, definitely. So especially because everyone was so friendly and I worked with a great team of there of people who knew what they were doing. Were really generous with their time and expertise and also the opportunity to work on some of these games as well as those big AAA games that you mentioned earlier PlayStation or either owns or finances, a lot of small studios where you'd work with small teams who are perhaps only five people working on a game and those were particularly fun to work on cause they're such a small team and can have make such a big difference to run usability studies and see what real players impact is. And that was very difficult to leave particularly because of the parties I mentioned earlier and the free games. Definitely the free games.
- Brendan Jarvis:
- How much [affirmative], I was just gonna say how much did sing Star feature at the parties? Cuz you mentioned that was one of your favorite games when we were chatting before.
- Steve Bromley:
- Oh yes yes. Sing Star sos definitely a lot of fun, especially to test because you're just getting groups of friends in to come and sing Bohemian rap in front of you which never gets old.
- Brendan Jarvis:
- I've eventually got the lyrics of that here Steve, if you wanted to do a bit of karaoke
- Steve Bromley:
- [laugh] got Cas keyboard somewhere, let's give it a go.
- Brendan Jarvis:
- Excellent [laugh].
- Steve Bromley:
- Yes. But yeah, especially the party games like Sing Star and we also worked on some party games which are just great for casual, getting on the TV and playing together. I think those were particularly fun with the people who work at the PlayStation, a good thing for lunch breaks or after work. Nice to have a group of people who have the same interest as you and you can enjoy these things together. You asked about leaving PlayStation. So one of the things that I felt I needed to do for my own personal development is play was great and very mature at its usability testing where they would evaluate games as they're being made and test the prototypes and get some feedback about okay, players didn't understand what they're meant to do or where they're meant to go, but perhaps because games are an art form. We didn't do so many studies earlier in the development process.
- So thinking outside of games before you design something, like if you were designing Uber before you start designing that you'd want to understand, okay, how do people get taxis currently and what are the problems with getting taxis currently and where's the opportunity for a new product to come or a new future to come to make that better? And I didn't have much chance to run those kind of studies at PlayStation, which is one of the things that made me look outside the games industry for my next roles just to fill out that gap of experience and make sure that I'd was able to do each part of user research process, not just test things that had been made.
- Brendan Jarvis:
- Yeah, that's an interesting topic there. So it sounded like at PlayStation it was mainly focused on the evaluative track of user research. So we have a thing, we've made this thing, it's relatively finished in its form. Let's see how it performs in the hands of the users versus what it sounded like you were exploring there when you left, which was this explorative, generative even pre-pro definition type of research. So how did you navigate that transition? How were you able to leverage the experience you had with PlayStation into your move into the more mainstream user research?
- Steve Bromley:
- Yeah, there were some bits of my experience at PlayStation that had touched on doing earlier more generative work. One of the last projects that I worked on before leaving was they were launching the PlayStation VR headset, the virtual reality system And virtual reality was a lot more explorative in the studies we were running because at that point the Oculus wasn't out. There were no other VR headsets at least for open for consumers to buy at that time. A lot of the questions that games teams should have or the people working on the headset were very fundamental. How do people interact with vr? What's a safe space for people to use VR in? What are the challenges that people have using in vr? All of these earlier questions that you'd want to answer before you start to design some software or websites. And I think that that project in particular starts to expose me to, okay, what are some approaches that are very helpful early in development? Things where you can be more contextual, understanding people in their real space, some people's real life. And that was some of the experience that I took forward with me into new roles to demonstrate that I could do these kind of studies as well as testing things as well.
- Brendan Jarvis:
- It sounds like you spend a lot of time in people's lounges.
- Steve Bromley:
- Yes, there's a lot of traipsing all over London and seeing different people's lounges and setting up VR headsets in there so that we could see if it'd work. I dunno if you remember the Microsoft Connect their augmented reality system. Yeah. Where had lots of games where you wave your hand around. I think one of the things that had been noticed was, although that was very successful in America where they had very big lounges in other countries such as Japan and here in the UK where the average living room is reasonably small some of the interactions require and some of the games wouldn't work in these setups. You'd have to move your sofas up away and push the TV back just to have enough play space to make the connect work. And I think that's one of the things we are thinking about is how do we make sure it is compatible with this international audience and not just work in America.
- Brendan Jarvis:
- Yeah, yeah, I can empathize with that. Having bought a vibe actually and then tried to make that work and one of our flats and quickly found out that the room was about 30 centimeters too small to actually work for room scale vr. So it was quickly listed on listed for sale after that.
- Steve Bromley:
- Yes, perhaps your usability lab could be adapted to just save up the whole space with a VR setup.
- Brendan Jarvis:
- It can be, that was actually one of the very important measurements should that develop as a local opportunity for the gaming industry here. So yes, definitely. So very serious question, just to finish off here on games used research God of War or Uncharted.
- Steve Bromley:
- Oh definitely God of war. I was never that into the earlier I played the earlier God of wars and you just walked forward and you hit square as much as you can and be everyone. Whereas the story and the narrative driving the new God of War I thought was fantastic. I really liked it. Although Uncharted is obviously also a lovely game. I think the last of us is much more thematically closer to this type of stuff I like from Naughty Dog and so God of War was my one of last generation and I'm really looking forward to seeing the next one in the next generation.
- Brendan Jarvis:
- Most definitely. Hey, so Steve, you left PlayStation and you went to the UK Parliament and I can't think of actually on the face of it to different kind of environments to enter a lot of assumption in there. What were the major differences and how did you adjust to such a different environment?
- Steve Bromley:
- Yeah, great question. So the context at Parliament when I joined is that they didn't have an in-house user research team at the time and didn't have any history of running user research. As you can imagine, it's an institution that's been around for almost a thousand years and they're very set in their ways and their current ways weren't particularly around let's look at user-centered design and let's learn from users as part of our development process.
- Brendan Jarvis:
- Interesting perspective for a democracy, not very true.
- Steve Bromley:
- So yeah, it was obviously a very different context and I think the types of studies that we've touched upon perhaps were significantly different going from building software which are entertainment, medium and artistic, and then moving to building, trying to build functional websites that allow citizens to do demo democracy related tasks. I guess the first thing that felt very significantly different was just how much fun the participants were having. When someone comes to PlayStation to take play an unreleased game, that's probably the best thing they're doing this month. Probably the best thing they're doing this year from their perspective and they're having a great time. Whereas if you are coming onto coming into a Userability session and you have to read some long reports from a committee is published as for your tests, again, that's probably not the best thing you've done that day. So I guess
- Brendan Jarvis:
- It's also a good reason not to disclose too much to participants as to what they'll be doing before they get there.
- Steve Bromley:
- Exactly. You'd have to incentivize 'em a lot more if you told them they would be reading very long reports when you got there. But from their development processes, actually again that was somewhat different. And an education for me, which I very much enjoyed I talked about earlier about Playsations focus on usability testing because Parliament recognizes that it needs to be four systems and for everyone a lot of that foundational work to understand what are people's context and what are they trying to do when they contact their MP for example, is a lot more important. And so a lot more of that discovery work needs to happen. Understanding people in the real world and the real problems that they have will directly inform how do you make parliament's websites. Another thing that struck me as different was the ethos. And this is something that I think is really important and lesson I'll try to take forward from Parliament.
- How we worked in PlayStation where we were essentially an internal agency where we would take a brief and would run a study and would do some analysis and then present it back to the team at the end. And that's great and works really well for those kind of usability studies where you find problems, ethos and parliament instead was a lot more about cross-functional teams and multidisciplinary work. So instead of being a research team, you would embed your user researcher inside a product team and they would work very closely without a designer, a product manager, a content designer. And instead of going away and running a study and bringing it back, a lot of the effort that a user researcher would do would be thinking about, okay, how can we expose every part of the research process to the team we're working with and make sure that they understand it as well.
- The obvious one that I think is very common is bringing in teams to observe studies. At least they can see the users firsthand and see the experience and they'll trust, oh I've learned these, that these problems exist and I know that these are real users so I believe it. But there are some ways you can go beyond that such as doing analysis together as a team where everyone's analyzing the findings and affinity mapping together or thinking about how to share those findings as a team as well. So rather than report finding something like a workshop where you can discuss the findings together and come to conclusion based on that, I think that kind of collaborative research approach has a lot of advantages. It reduces that communications gap between the researcher who knows everything that you need to know about users and the designer who has to make design decisions. It brings that those people closer together and reduces that gap. But I also see it's a lot harder for a research team and a researcher because their role becomes a lot more about facilitation than running research studies.
- Brendan Jarvis:
- Yeah, I was gonna say that's the approach that we take here of this space in between. But I was going to say it sounds like that could go horribly wrong. Have you got any stories to share the dangers of facilitation or bringing people into the research process maybe too soon?
- Steve Bromley:
- Yeah, perhaps not a specific incident, but I think that was one of the mistakes that I personally made at Parliament was I looked at other organizations who were working in this model and it's been very successful. A very prominent example here in the UK is government digital service, gds, who do a lot of work and have done a lot to advance a field of user research. And I looked at that and said, okay, that looks great. They're having great time with it, let's dive straight in with that. And I think the challenge was I was trying to go straight in at a very high research maturity level for an organization that didn't have any history of doing user research. And so a lot of the conflicts you might have at that generates a lot of conflict. Obviously that collaborative research practice requires the other people who are not researchers such as the designers and product managers to understand enough about research to play an active role in it and make sensible decisions and do things like analysis, which can be very difficult for your first exposure to research.
- Also, it takes a lot of people's time if you want people to come and watch research sessions and analyze findings together and come up with a report together. If you're a developer, that's time you're not writing code and feels quite like you're being taken away from your job. And because of that there can be challenges getting people to take part or see the value of it. And I attribute that to trying to jump into a high maturity model straight away. The approach I've tried to take since is, okay, let's build up to that model, let's start an agency style. Let's gradually open up opportunities to get teams who are particularly interested, more involved and eventually end up in that same place. But the ramping up will be easier for everyone involved.
- Brendan Jarvis:
- It sounds like a bit of almost a cultural learning or journey that you have to take people on.
- Steve Bromley:
- Yeah, I can see that. And I see that a lot of the work that researchers have to do, especially as they get more senior, is that kind of advocacy and evangelism to explain this is what user research is, here's the benefits of doing it, this is why it's worth your time. Try and dispel some of those myths that people might have had around user research.
- Brendan Jarvis:
- What are some of those myths, Steve?
- Steve Bromley:
- Yes. So there there's some that you hear again and again, I'm sure you've heard in your own experience as well. I guess one of the first ones is people's concern that research costs money and they think, oh this study's too expensive and instead of paying the money to run this research study, I'm just going to dive in and we'll launch it and we'll see how it does earlier.
- Brendan Jarvis:
- Yeah, and if I said that to you, so imagine I'm your stake your senior stakeholder, I'm a product manager for example, and I say to you, Steve, we can't do this. It's gonna take too much time and it's too expensive. What are you gonna say to me?
- Steve Bromley:
- Yeah, great point. And I hear your feedback. So it is true that research is expensive. There are some costs that I highly recommend you don't cut back on. The most prominent one is recruiting participants. If you're not spending money to get the right type of people in, you are not learning relevant things. And if you're not incentivizing them to turn up and paying them to actually bother to turn up to the session, again, you're gonna waste everyone's time cuz they won't turn up. And then everyone just sat around doing nothing. And it's true that has a cost and sometimes that can be a reasonably high upfront cost. As user researchers, we believe that that cost is ultimately saved back during development time and later on in the process by running these studies earlier on, you identify all that what features do we need to make or that feature doesn't work for our users and we need to make some changes or that button's in the wrong place and we need to move it so that people understand what they're meant to do and address some of these issues where people might not understand what the software is or how it works.
- And by finding out those issues earlier in the development process, you're not wasting that development time on something that's ultimately not going to work and you can fix that much earlier than you would fix it otherwise this has some direct financial benefits. You will obviously not spend that time that developer would've spent working on the wrong thing and you'll get to the finished product quicker. It also means will have an impact on some of the metrics like customer retention. If when you launched you got a hundred customers in and they all had a bad experience and they bounce out, that costs financial costs obviously. And by preventing that happening, by fixing those issues before your a hundred customers come in, you might save 50 of those customers and again see a direct return on investment for the research study that you've done. And so because of that, although research is expensive, we think it's costs less overall when you look at the whole project than not running research studies.
- Brendan Jarvis:
- Yeah, it's something that I've heard a bit as well and it seems to be a difficult one to demonstrate in advance. Unlike something where you are testing in production and you can AB test it and or multivariate and see exactly what the performance is with a significant volume of users coming through. It almost sounds like the organization has to take a leap of faith, at least initially that research will deliver the benefits that you're speaking of.
- Steve Bromley:
- Yes. And you do see this on teams that work on multiple projects. Again returning to games, if you look at the development of Halo, I think on their earlier Halo games, they ran very few studies at the end of development and then as the titles went on, they were running more studies earlier and by the last one they were running them right from the start. There are ways that you can try and demonstrate that benefit quicker. As a use researcher. One that I've been thinking about is if you do have an existing product, looking at some of the metrics that are important to teams a very common one is the number of support tickets that customers raise or the support calls that you get. And by running something simple like usability study, you can hopefully see an impact on the number of support requests that teams are required to deal with. That can be attributed to a financial cost. You can work out how much time people are spending answering support tickets and then say, okay, this study has saved us this much money. And by getting some buy-in on some of those simpler tests like usability studies over the life cycle of more projects, you can start to move earlier in that process whereas you get evidence that this is having impact and convince people.
- Brendan Jarvis:
- And I suppose this illustrates what seems to be a bias in a lot of organizations towards the quantitative [affirmative] and in your book and as we are talking about the usability studies you've been running at PlayStation and since you've left as well that's the qualitative end of user research, isn't it?
- Steve Bromley:
- That's correct. And I agree with you entirely that because there are some nice numbers and people can see the difference between two numbers. The truth of quantitative research is often easier to convince people of or easier to demonstrate through things like AB testing. I think one of the challenges that user research, qualitative user research has is a lot of the benefits of understanding your audience better are very hard to quantify. And also people forget that they didn't know this before. You'll watch a user do something and then have learned, oh that's what the user's behavior is or that's a problem that the user has and by tomorrow you'll forget that you didn't always know that and just think, oh, I always knew that users had difficulty ordering a taxi in the room without recognizing that how the study has influenced your thinking which makes it somewhat difficult, especially with those generative studies to quantify the benefits. It's easier of usability testing because you can count the number of issues you see, you can rate the severity of the issues and there's lots of ways that you can quantify that. But you do need a degree of buy in and research maturity from an organization before they can see the benefits of understanding your users better through a
- Brendan Jarvis:
- Study. So get stuck in, find something that you can quantify as a result of the research that you've done a small thing and use that to build confidence and trust in the research process. And that sounds really sensible. What is it do you think, in your experience with organizations that does bias them towards feeling more comfortable with the hard data, like the quantitative side of things?
- Steve Bromley:
- I guess as we talked about the fact that they are numbers and people understand numbers, I guess numbers especially speaks to people on more senior levels as you go higher up the chain and graphs and charts are the type of things that they deal with a lot. Whereas the actual granularity that you get from a good user research study is extremely valuable. Invaluable, I'd say to designers who are actually making those implementation decisions or product managers who are actually making those feature decisions but doesn't scale up in the same way where it's very hard to communicate that up to an executive level. Whereas a number, everyone can look at a number on a graph and see the difference, but that same number, that quant study also doesn't have that same impact for the teams you work with. It's much less valuable for a designer or product manager to know that this one is 56% better than this one than it is to understand okay, users looked at the screen and they understood this aspect of it and the reason they understood this aspect is because of their history with this. That gives lots of inspiration, which is really useful on the ground
- Brendan Jarvis:
- [affirmative]. So you're saying that it's difficult for senior management to buy empathy?
- Steve Bromley:
- Yes, that would be it.
- Brendan Jarvis:
- Yeah. Steve, you also warned your book about user researchers in smaller qualitative studies including metrics including measurements. Can you tell us a little bit more about why that is problematic?
- Steve Bromley:
- Yes, of course. So a very common thing that user researchers might be tempted to do is to ask the participants to rate things at the end of a session. They've used your website, can you rate this website out of 10? And they'll say, oh, it's eight out of 10 and the next one will say it's three out of 10. And again, there's a temptation because as we've heard people like quantum like numbers to take those ratings and sit them in the report and say, okay, the website was seven out of 10 overall, which sounds fine, [laugh] the issue being a lot of these qualitative studies aren't seeing enough users to make reliable conclusions from that quant data. So there are statistical tests that you can run and people who are better at stats and me can explain it better. But you can anticipate by seeing those, that number of users you've seen what the range of answers a wider audience would give.
- And from a qualitative study that range is very large. You would say if people were rating words out of 10, if you were only seeing five or six users once you did that statistical statistical test, it would only give you an indication that the average person will rate it somewhere between two and eight. And that's such a large gap that it's not particularly useful. However, you need a degree of exposure and understanding statistics to know why you shouldn't just take the average, why you shouldn't just go for, okay, on average people said seven and so I'm gonna put seven. And there's a high risk if you aren't being careful with how you're communicating this, that teams are going to look at the numbers you've counted, notice the average, and then compare that between different rounds. They say, oh, last round we were six, now we're seven so we're getting better. Statistically we know that's not true from these types of studies, or at least we know we can't tell that from these types of studies. And so you're giving teams misleading conclusions and they're gonna have bad information and make poor decisions because you haven't been careful with the information that is safe to communicate from tests.
- Brendan Jarvis:
- And ultimately I [crosstalk] [affirmative] but undermine the confidence people have in the research if you end up making decisions that are not helpful to the organization.
- Steve Bromley:
- Yeah, that's exactly it. A thing that I think is very valuable is building trust with a research team. And one of the tools I think is very helpful for building trust is using caveats and saying what you don't know. So from a research study we're gonna learn 10 reliable things and then some things which are not perhaps so reliable or give an indication there might be something here but we don't know exactly what it is. And being very clear to your team about we believe that these aren't are safe and we think these are interesting, but we know they're not safe and don't do anything based on them yet we just explore further. I think that builds trust that hey, the ones that they think are safe must be good because they've told us the ones that aren't safe. And I think that's a really powerful tool to caveat your findings in that way.
- Brendan Jarvis:
- How would you determine whether a piece of feedback Beck was safe or unsafe?
- Steve Bromley:
- One of the obvious safe ones or ones that we consider very safe are usability findings. We know that when we've seen an issue, it's true that the issue exists and we know that that issue isn't the intended experience and at that point it's very sensible to do something to fix that problem rather than spending time to work out how many people encounter that problem. That's potentially just wasted effort. We know the problem exists, let's do something about it instead. Other types of finding are the ones where there's more risk and should be more careful with numbers like we talked about taking ratings. You can do that statistical test and you can expose the confidence intervals and say, we know it's somewhere between here and here but we dunno where it is in that bracket. And so again, that can be you are confident with how much you do know and how much you don't know until you can communicate that The riskiest ones and the ones we use most heavy caveats around is opinions. If someone says, oh, I liked this website or I liked that it was green or I really like the font, we know that that opinion exists, but we don't know how many people hold that opinion and how many people hold the contrary opinion. And it's those kind of findings where most careful to say, if you want us to explore this, we should combine it with a quantum method. Like designing a survey to explore how many people had that opinion and how many people had a different opinion before you take action.
- Brendan Jarvis:
- Have you got any stories from the trenches that you can share of when something like that is originated either at PlayStation or since leaving where you've then gone on to run a study this, does anything come to mind? Like a survey off the back of a usability study? Yeah.
- Steve Bromley:
- Yes. So it's reasonably common in game user research and one of the reasons it is reasonably common in game user research cause the opportunity is there. So there's a couple of aspects of this is as you're familiar with from games review scores are very important. And so team is very much like the idea of having a score. And as we've talked about, to get reliable scores and to do things like surveys reliably, you need a lot of people with games. There is an opportunity to get a lot of people because games are so exceptionally long. God of war we talked about earlier is probably 20 hours long. If you wanted to even do a usability study on that, that's 20 hours of gameplay, which is what's that? That's a week to watch one player And a consequence in games, user research is very common to run many players simultaneously where you'll have 10 or 20 people playing at the same time so that you can get them just through that quantity of game.
- So in that week you'll see 20 people, even if you can't observe them more closely, you at least have 20 people who have played through the game and can start to do those kind of surveys. And because even as simple usability test needs that number of players cuz of the play time, you can then do a lot more of these interesting survey questions where you can ask opinions about what do people play, think about the characters, what do people think about the stories? And ask people to give ratings as well. And you, you've got enough people coming through that you can start to quantify this in a way that wouldn't be reliable if you were doing five users for an hour long usability tests on a website.
- Brendan Jarvis:
- Yeah, because of that variance and the interpretation of that small sample size.
- Steve Bromley:
- Yeah, exactly.
- Brendan Jarvis:
- Yeah. If you were dealing with a team a product team that was feeling a bit reluctant about putting forward a product that they considered not to be ready, which is often something that comes up, it's not ready yet, what sort of things would you tell them to encourage them to test?
- Steve Bromley:
- Great question. And I also hear that quite a lot to where teams think you have to be in a certain states before it's sensible to test things. There's a couple of reasons why this is isn't a great idea to wait until you think it's ready. The first one is it's going to be a lot more expensive you to fix problems later on in development as we talked about earlier. Then if you found those problem when you just had the idea or you just made a wire frame, it's reasonably cheap to change your wire frame and it's free to change your ideas. But if you wait until we've built it and it's ready, then it's reasonably expensive to change something in production. There'll be a lot more development time. You'll need to go through a UI artist again or a UX designer will have to go through it again.
- And that becomes very expensive. And not just in cost, it also a political momentum here as well. So once you've spent a week or a month developing a feature, people are gonna be very reluctant to change that feature. And so they'll just keep on with a poor experience because they've spent so long making it then if they'd learn earlier and weren't so invested in the idea because of that sunk cost idea of, oh, I've invested all my time, I have to keep this thing at this point that really encourage people to stick with something even if it's not perfect. And another reason why this occurs is people perhaps don't understand the ways in which user research work and particularly how we define research objectives. So we can say, okay, we want to learn these things from this study and we could ignore all the bits that aren't finished or aren't representative.
- We can still find very valuable information about the bits that aren't are in a good enough state's test, and we can combine it with prototypes or we can mock something up or the researcher can do something to fill the gap for the bits that aren't working. So we can work around the fact that it's not ready yet. And once a team start to understand, oh, actually user researcher has skills that allow them to overcome the bumps in what we've made or overcome bugs and we learned useful things earlier and again it'll save us time and development costs, that becomes a very valuable tool for convincing people to run research earlier.
- Brendan Jarvis:
- Yeah, it almost sounds like as researchers you need to develop a keen appreciation for other people's feelings and almost like a therapist.
- Steve Bromley:
- Yeah, I think that's very fair. One, again, as I've become more senior, one of the lessons I've taken is the importance of that cross-disciplinary learning. It's easy enough to sit there as a researcher and think, oh, why is no one doing user research studies and they're not listening to me and my studies have no impact. And it's usually because people have different incentives. If you spoke to people they have pressures either on their time or they've got deadlines or they've got deliverables that they have to meet. And as a researcher, you need to understand what are the incentives acting on the people we work with, the product managers and the designers, what are their goals and what are the problems that they have? And then we can come to them with framing our studies in the right way so that if they do care about costs, we can come and frame our research about how it's gonna reduce costs. If they want development time to be shorter, we can come and say, oh, actually our studies will reduce your development time. If they just want better customer attention, we can come and say, okay, here's some studies we can run to help customer attention. And by understanding what people are trying to get done in their own job, it allows us to run the right study and ultimately make better things for our users, which is our goal.
- Brendan Jarvis:
- It almost sounds like that is for a low research maturity organization for a loan researcher or a new team. Sounds like a really great place to start.
- Steve Bromley:
- I agree with that. I think I recommend starting by interviewing the people that you work with as your first thing as you join as a researcher. So go and find who's making decisions in the company, what are the current decisions they're making and what are the challenges they have? And then once you've done that, you can work out, okay, what's the most impactful study that I can run early on to show the value of this kind of working? Often that can be usability testing cause it's easy to understand what's the benefit of usability test and we did this test and we learn all these problems until it's something we can fix. And that can be a really great place for a new researcher to start when they're in a new place.
- Brendan Jarvis:
- So what has been the most fun part of being a user researcher?
- Steve Bromley:
- Yes. So I think it is that direct interaction with participants. It's often you can be stuck in our tech bubble where everyone we work with is very likers and they have the same interests and the same situation and you forget how different people in the real world are. And by having to interact directly with participants, you can see how different you are as someone who works in tech from a normal person. That ranges from seeing how kids interact with games. As we talked about earlier when we're running studies on sing star and party games, just that interaction between groups of friends can be tremendously fun to see and be involved with. Or on some of those studies where we are looking at bigger issues like local news or politics. Again, seeing the diversity of viewpoints out there, the diversity of behaviors that people have is always surprising and keeps the job very fresh for me.
- Brendan Jarvis:
- Yeah, never a dull day.
- Steve Bromley:
- Exactly.
- Brendan Jarvis:
- But I can't imagine, Steve, that it's been all roses. What's been the sort of the least fun or the most challenging part?
- Steve Bromley:
- Yeah, I guess some of the things I've done wrong in the past is not taking that time to understand the other people I work with and what they're trying to get done. And just diving in saying, okay, we're a user research team, we'll go and understand your users and we'll make a nice report box and because of the nice report you'll have learnt some things that will be really useful to you. And then what we've got from that is running low impact studies, which teams, they look at the report and they nod and they think, oh, this is very interesting, very nice. And then go on and do the thing that they're doing anyway. And for researchers that can be extremely frustrating, but you think, look at all this great work and look at all the inspiration I've found for you and you are just not using it. And I've started to recognize that that is a failing of the research team to understand what are people trying to get done and how can we frame our research studies so that are useful to people rather than just doing what we think we should be doing.
- Brendan Jarvis:
- Yeah, a hundred percent. Now hope you don't mind, I'm gonna play a fun little very simple game with you called complete the sentence.
- Steve Bromley:
- Okay.
- Brendan Jarvis:
- Are you ready?
- Steve Bromley:
- I'm as ready as I'll ever be.
- Brendan Jarvis:
- Okay. Product managers are
- Steve Bromley:
- Time pressured, but lovely people
- Brendan Jarvis:
- [laugh]. Very safe answer. I like it. I like it. Given all the sessions that you've observed or conducted and moderated or been involved with, what's the funniest thing that you've seen someone do? You mentioned the child handing that note over, which was pretty up there, but has it been anything else?
- Steve Bromley:
- I, I've seen a lot in research sessions. I've seen people fall asleep in research sessions, which you'd be surprised of, but often we would have group sessions with two people playing together and we on more than one occasion, actually the one of the players has fallen asleep during that session which isn't a great reflection o of the we're working on if someone can't make just fall asleep on our sofa during it.
- Brendan Jarvis:
- Hopefully the design team wasn't there.
- Steve Bromley:
- Yes. Perhaps we lost the video for that one. Maybe
- Brendan Jarvis:
- [laugh].
- Steve Bromley:
- Yes. It's all been all kind of things. We've had colleagues, again working with kids where the kids are start to strip off and again, that's extremely inappropriate and hard to deal with when you are not parent. Luckily the parents are usually there in this case. Yeah. But yeah, the children's behavior has been the strangest thing to deal
- Brendan Jarvis:
- With. Yeah, we don't have time today, but perhaps a conversation for another day could be ethics. And I know that you've spoken a little bit about that in the past, but it obviously plays a role in research on a more serious side. And in particular when you're dealing with different groups like children or people that are disabled, that plays an important role.
- Steve Bromley:
- That balance of consent, getting people to understand what they're going to be doing from the research study and being able to meaningfully say, I'm okay with this to happen, is super important for researchers to be aware of and do.
- Brendan Jarvis:
- Yeah, a hundred percent. So if you were able to go back in time or send yourself a tweet back to your pre user research days as you were starting your job the first day at your job as a user researcher that's when Twitter was 140 characters too. So not this long, long winded Twitter that we have now. What would you say to yourself in one tweet would be the advice you'd give?
- Steve Bromley:
- Yeah, it's quite hard. Oh, advice. It would have to be something about the community. Let me try and get into 140 characters [laugh], but
- Brendan Jarvis:
- I'm not counting [laugh].
- Steve Bromley:
- I guess research the field of research advances through sharing knowledge and building a strong research community. And so the advice would have to be look out for other researchers doing good work. Be open about what you're working on and listen and learn from the people around you.
- Brendan Jarvis:
- Well, you've certainly delivered on that, Steve, through what you've been doing. And that brings me to one of the final questions I have for you tonight, which is what's next? I think what you mentioned that you might be working on something special.
- Steve Bromley:
- Yes. So as we talked about earlier, I've been doing a lot of work for games, user research, mentoring. So for the next book that I want to work on, I want to produce a book that explains how to be a game user researcher, which aren't a lot of those questions that mentees have come to our mentors about. It should explain how does games development work, how do you run a study in games, user research. And then lots of advice about getting that first job in user research, which could be tremendously difficult to do. So there's a gonna be a book and also a whole bunch of free community resources because as we've talked about, the importance of making resources free to people. The website for this will be gamesuserresearch.com, but there's not much at the moment. But again, check back soon and there there'll be something there.
- Brendan Jarvis:
- Gamesuserresearch.com.
- Steve Bromley:
- Yes, it should. The book should be out in early 2021, so worth having a look at later.
- Brendan Jarvis:
- Oh, that's wonderful Steve. And in the meantime, how else can people keep in touch?
- Steve Bromley:
- Yes, so I'm reasonably active on Twitter. As you mentioned earlier. My Twitter hand is at Steves bromley, and I think everything I do is referenced on my website, stevebromley.com. I currently am running a newsletter for people who are starting news research teams, keeping up with blog post about that experience of growing research teams and the work that's gonna be happening in games user research. The best place to keep in touch would be or to follow that would be on the websites, stevebromley.com.
- Brendan Jarvis:
- Stevebromley.com. Wonderful. Well, Steve, look, it's been an absolute pleasure having you on this the first show for Brave UX. Thanks for being so generous with your time and sharing your knowledge. It's been such a fabulous conversation and I really think that our viewers will have a lot of insight to unpack here, and I really can't wait to read your next book. And I just wanted to bring everyone's attention again to building user research teams. It's definitely worth its weight and gold and there'll be something coming out on social soon as to how you might be able to get yourself a copy of that from me. And I just wanted to thank everybody who's tuned in. Thank you very much. I hope you enjoyed the show. And one of the things that we're working on is bringing more world class experts like Steve to have great conversations like this. So don't forget to comment and subscribe if you found it valuable, so you can keep in touch and get those direct to your feeds. So everything we'll covered today will be available in the show notes, including where to find Steve as we've just talked about his books and all the other resources that we've mentioned. So until next time, stay safe. Cheers everybody.