Story navigation: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
If there's one thing I learned from "Maze of Death," my contribution to the first Make a Good Mega Man Level Contest (MaGMML), it's that levels need to be playtested by other people—people who think and play differently than you do. People who don't already know what's coming. People who aren't predisposed to thinking that everything is perfect and wonderful. It's important to playtest the heck out of your own creation, but you can't stop there.
In playtesting as in life, there is strength in diversity. Gaming skill varies greatly from one person to the next, and we all have our own ideas about what is and isn't fun. Our expectations are informed by the games we've played and the lives we've led. We differ in how we deliver feedback, and in what we deem important to point out. Some of us are attentive to detail; some of us are oblivious to the obvious. Some of us like to take our time, explore, and think things through; some of us like to barge ahead and rush through challenges. I could go on. The broader and more eclectic the playtesting group, the more bugs you'll catch, and the better your final product is likely to be.
From the beginning, my plan for OH JOES! was to playtest in phases. Whenever I had a substantial amount of new content, I'd send the game for playtesting. Not every playtester was involved in every round of playtesting, and the size of the group depended on the size of the development milestone I'd reached. My rationale was that I'd get better feedback if I had a mix of some playtesters who'd seen several iterations of the game, and some playtesters who came in fresh at various points in development. Rotating playtesters also helped people from getting burned out, and it ensured that the finished product didn't just cater to the tastes and opinions of the same few playtesters.
By the time OH JOES! was released, almost 30 playtesters had taken the game for a spin, and I extend my thanks to all of them. At first, the group was mostly comprised of the people actively helping me with the project—CosmicGem, Entity1037, Jasper Valentine, and Renhoek. Along the way, I roped people I knew (either personally or through years of Internet interactions) into playtesting, including Alice "Mother" Kojiro, B.B. Sting, Dash Jump, Desertskunk, Mr_Cobb, ProcneDevi, Super Adapter, and Zapetroid. A number of folks specifically from the Mega Man fan community either volunteered or took me up on a request to playtest, such as happygreenfrog, IcyTower, Mark in Austin, NaOH, PKWeegee, Quack Man, and Raine.
I thought it was important to get feedback from the people with whom I'd be judging MaGMML3, so I invited Mick Galbani, Pachy, and Shinryu (and ACESpark, but he was extremely busy at the time). For good measure, I recruited Cheez, the MaGMML judge who seemed most likely to appreciate my game, as well as Dolphin, a regular judge of informal Mega Man Maker contests. I wanted input from fellow fangame developers, so I asked for help from Blyka, Lars Luron, SnoruntPyro, and WreckingPrograms. I was also planning on reaching out to the likes of MegaPhilX, MrWeirdGuy, GoldWaterDLS, and YouTuber RoahmMythril, but by then I had a near-excessive number of playtesters for a game that was only four stages long.
If there's one thing I learned from "Maze of Death," my contribution to the first Make a Good Mega Man Level Contest (MaGMML), it's that levels need to be playtested by other people—people who think and play differently than you do. People who don't already know what's coming. People who aren't predisposed to thinking that everything is perfect and wonderful. It's important to playtest the heck out of your own creation, but you can't stop there.
In playtesting as in life, there is strength in diversity. Gaming skill varies greatly from one person to the next, and we all have our own ideas about what is and isn't fun. Our expectations are informed by the games we've played and the lives we've led. We differ in how we deliver feedback, and in what we deem important to point out. Some of us are attentive to detail; some of us are oblivious to the obvious. Some of us like to take our time, explore, and think things through; some of us like to barge ahead and rush through challenges. I could go on. The broader and more eclectic the playtesting group, the more bugs you'll catch, and the better your final product is likely to be.
From the beginning, my plan for OH JOES! was to playtest in phases. Whenever I had a substantial amount of new content, I'd send the game for playtesting. Not every playtester was involved in every round of playtesting, and the size of the group depended on the size of the development milestone I'd reached. My rationale was that I'd get better feedback if I had a mix of some playtesters who'd seen several iterations of the game, and some playtesters who came in fresh at various points in development. Rotating playtesters also helped people from getting burned out, and it ensured that the finished product didn't just cater to the tastes and opinions of the same few playtesters.
By the time OH JOES! was released, almost 30 playtesters had taken the game for a spin, and I extend my thanks to all of them. At first, the group was mostly comprised of the people actively helping me with the project—CosmicGem, Entity1037, Jasper Valentine, and Renhoek. Along the way, I roped people I knew (either personally or through years of Internet interactions) into playtesting, including Alice "Mother" Kojiro, B.B. Sting, Dash Jump, Desertskunk, Mr_Cobb, ProcneDevi, Super Adapter, and Zapetroid. A number of folks specifically from the Mega Man fan community either volunteered or took me up on a request to playtest, such as happygreenfrog, IcyTower, Mark in Austin, NaOH, PKWeegee, Quack Man, and Raine.
I thought it was important to get feedback from the people with whom I'd be judging MaGMML3, so I invited Mick Galbani, Pachy, and Shinryu (and ACESpark, but he was extremely busy at the time). For good measure, I recruited Cheez, the MaGMML judge who seemed most likely to appreciate my game, as well as Dolphin, a regular judge of informal Mega Man Maker contests. I wanted input from fellow fangame developers, so I asked for help from Blyka, Lars Luron, SnoruntPyro, and WreckingPrograms. I was also planning on reaching out to the likes of MegaPhilX, MrWeirdGuy, GoldWaterDLS, and YouTuber RoahmMythril, but by then I had a near-excessive number of playtesters for a game that was only four stages long.
As often as possible, I pestered people to playtest my game in person; otherwise, my preference was a playthrough video with audio commentary. It's easier for me to gauge difficulty, fun, and fairness when I know exactly how the player reacts to each situation, both visually and verbally. Glitches are easier to replicate when I can study a recording for possible causes, too. Some people preferred text feedback via e-mail, Facebook, Discord, or even a PDF, sometimes with screenshots or short video clips attached. This allowed people an opportunity to articulate their thoughts in a way that might be difficult in the middle of a playthrough, plus it cut straight to the heart of what they felt did and didn't work. Getting feedback in a variety of formats kept the playtesting process fresh for me, and I got a well-rounded sense of my game that way.
Whenever I watched someone play through the game, I kept a notepad handy and scribbled a bulleted list of things I intended to change. A playtesting session could yield up to three or four pages of notes, with about two dozen bullets per side. I used plenty of shorthand, so Hammer Joe became just "ham"; I've since discarded all these notes, but I'm sure you can imagine what it was like returning to a list a few days later and trying to remember what things like "ham stuck on ice" meant. I crossed items off the list as I addressed them (in whatever order I felt like); any items I didn't really feel like dealing with immediately got transferred to a Word document to be sorted out later. Many of these, such as native controller support, sat on the list so long that they turned into "nice to have" items to be addressed after the game's release, if there was enough interest in the game to justify the time and effort.
For the most part, I contacted my playtesters individually for their videos and written comments. I've been a playtester for other games where feedback was provided in a group chat, and though there are benefits to that approach, I find it lends itself to derailed trains of thought, people jumping on a bandwagon, and people criticizing your criticism. The individual approach also allowed me to address feedback at my own pace, without feeling pressured to keep up with an ongoing conversation.
Most of the playtesting feedback for OH JOES! was very freeform; I welcomed any and all comments, usually with minimal guidance about what I wanted people to test. To some degree, this was a mistake. Hardly anybody tried the final iteration of Easy Mode, I don't think anybody played the game in French, the glitch-prone Item-2 was largely ignored, and there were a couple paths in the final stage that few people bothered with on Difficult Mode.
Whenever I watched someone play through the game, I kept a notepad handy and scribbled a bulleted list of things I intended to change. A playtesting session could yield up to three or four pages of notes, with about two dozen bullets per side. I used plenty of shorthand, so Hammer Joe became just "ham"; I've since discarded all these notes, but I'm sure you can imagine what it was like returning to a list a few days later and trying to remember what things like "ham stuck on ice" meant. I crossed items off the list as I addressed them (in whatever order I felt like); any items I didn't really feel like dealing with immediately got transferred to a Word document to be sorted out later. Many of these, such as native controller support, sat on the list so long that they turned into "nice to have" items to be addressed after the game's release, if there was enough interest in the game to justify the time and effort.
For the most part, I contacted my playtesters individually for their videos and written comments. I've been a playtester for other games where feedback was provided in a group chat, and though there are benefits to that approach, I find it lends itself to derailed trains of thought, people jumping on a bandwagon, and people criticizing your criticism. The individual approach also allowed me to address feedback at my own pace, without feeling pressured to keep up with an ongoing conversation.
Most of the playtesting feedback for OH JOES! was very freeform; I welcomed any and all comments, usually with minimal guidance about what I wanted people to test. To some degree, this was a mistake. Hardly anybody tried the final iteration of Easy Mode, I don't think anybody played the game in French, the glitch-prone Item-2 was largely ignored, and there were a couple paths in the final stage that few people bothered with on Difficult Mode.
My original plan for the last round of testing was to send my playtesters a questionnaire about their experience. I used to do this after every Dungeons & Dragons campaign I ran for my friends, asking what specific moments and general characteristics of the quest they liked most and least. This questionnaire also would have assessed which game options were selected and which paths were taken, to give me a better idea of the gaps in testing. When it came time for final playtesting, however, I was burned out on being a game developer. I wanted the game out the door. I asked my playtesters to look only for glaring problems and tiny things that were easy to change. So much for the questionnaire.
Keep in mind that, despite the sheer number of contributors to the project, I was the only one incorporating playtester feedback. Revamping challenges, fixing glitches, adding new features, revising dialogue, correcting graphical mishaps—that was all me. I tried not to outsource changes unless I was utterly stuck. This proved to be exhausting, because the changes required way more programming than I ever wanted to do for this game. Programming was something I tried to get out of the way early so that I could focus on the fun parts, but it kept coming back to haunt me. It was not uncommon for me to spend an entire evening trying to address a single item off my bulleted list. There's a reason why this game took 2 years to make.
It was fascinating to see what kinds of issues each of my playtesters focused on. I could tell who was a programmer, who was a level designer, who had an artistic eye, and who was more accustomed to playing fangames and ROM hacks than the official games. Comments ranged from extremely general (eg, stage 3 is too hard) to extremely specific (eg, here are the particular notes I'd like to change in the music). Some issues were no-brainers to fix (eg, the game crashes when I do this thing); other issues were annoyingly on the cusp between my fault and the player's fault (eg, magnets pulling more strongly when stacked on top of each other, which one playtester misinterpreted as individual magnets having arbitrarily inconsistent pull strengths). Sometimes, I got conflicting feedback (eg, one person said the Chill block sections were brutal and always avoided them; another person said those sections were the best).
Indeed, the hardest and most frustrating part was figuring out when and how to compromise. Incorporating feedback is a balancing act between your vision for the project and what other people want out of it, a matter of filtering out personal preference and bias (both yours and the playtester's) to get to the core of what actually needs to change. Accepting a suggestion doesn't automatically make the game better—in fact, it usually necessitates more playtesting to determine how well the suggestion works.
There are plenty of reasons why you might choose to ignore feedback, but if you start to see a theme in the ideas you've rejected, you can be sure that issue will become a sticking point once the game is released. My MaGMML2 level, "Guts Man's Asteroid," fell short of the top tier because I didn't fully address my playtesters' complaints about the boulder droppers. Despite practically every official Mega Man game having items that are off limits if you come into the stage without the right character, weapon, or upgrade, OH JOES! has received nothing but grief for some of the JOES letters being inaccessible to certain characters. Do yourself a favor and find a way to make a concession, however small or indirect, even if you completely disagree with your playtesters. Especially if your playtesters are unanimous.
Keep in mind that, despite the sheer number of contributors to the project, I was the only one incorporating playtester feedback. Revamping challenges, fixing glitches, adding new features, revising dialogue, correcting graphical mishaps—that was all me. I tried not to outsource changes unless I was utterly stuck. This proved to be exhausting, because the changes required way more programming than I ever wanted to do for this game. Programming was something I tried to get out of the way early so that I could focus on the fun parts, but it kept coming back to haunt me. It was not uncommon for me to spend an entire evening trying to address a single item off my bulleted list. There's a reason why this game took 2 years to make.
It was fascinating to see what kinds of issues each of my playtesters focused on. I could tell who was a programmer, who was a level designer, who had an artistic eye, and who was more accustomed to playing fangames and ROM hacks than the official games. Comments ranged from extremely general (eg, stage 3 is too hard) to extremely specific (eg, here are the particular notes I'd like to change in the music). Some issues were no-brainers to fix (eg, the game crashes when I do this thing); other issues were annoyingly on the cusp between my fault and the player's fault (eg, magnets pulling more strongly when stacked on top of each other, which one playtester misinterpreted as individual magnets having arbitrarily inconsistent pull strengths). Sometimes, I got conflicting feedback (eg, one person said the Chill block sections were brutal and always avoided them; another person said those sections were the best).
Indeed, the hardest and most frustrating part was figuring out when and how to compromise. Incorporating feedback is a balancing act between your vision for the project and what other people want out of it, a matter of filtering out personal preference and bias (both yours and the playtester's) to get to the core of what actually needs to change. Accepting a suggestion doesn't automatically make the game better—in fact, it usually necessitates more playtesting to determine how well the suggestion works.
There are plenty of reasons why you might choose to ignore feedback, but if you start to see a theme in the ideas you've rejected, you can be sure that issue will become a sticking point once the game is released. My MaGMML2 level, "Guts Man's Asteroid," fell short of the top tier because I didn't fully address my playtesters' complaints about the boulder droppers. Despite practically every official Mega Man game having items that are off limits if you come into the stage without the right character, weapon, or upgrade, OH JOES! has received nothing but grief for some of the JOES letters being inaccessible to certain characters. Do yourself a favor and find a way to make a concession, however small or indirect, even if you completely disagree with your playtesters. Especially if your playtesters are unanimous.
For the most part, though, my playtesters seemed to appreciate and enjoy the game. It was gratifying to hear them laugh and smile (yes, you can hear a smile) in their playthrough videos. One playtester described me as a "potent designer." Another said OH JOES! might be the most fun he's ever had playing Mega Man, which strikes me as something that can't possibly be true, but I'll take it anyhow. There were plenty of critiques about the premise being stretched too thin, the stages being too long, and so forth. At the same time, there was also a lot of praise for the soundtrack, the graphics (especially the intro cutscene), the game's sense of humor, and the quality of individual challenges. The general consensus was that OH JOES!, while certainly not the next Rock Force, was fun enough to be worth a look. That's all I really wanted to accomplish.
A couple playtesters didn't seem to enjoy the game as much as the others, but they all had one thing in common: they had never heard of OH JOES! before I asked them to playtest. My other playtesters had been following the project to some extent, and they had a sense of what kind of game this was. This, in combination with some of the specific criticisms I received, led me to believe that false expectations might be influencing some playtesters' opinions. In fact, after I recommended using special weapons more often and approaching each challenge as a puzzle, one playtester who had given up on the game gave it a second chance and ended up having a blast.
It's like when my friends took me to see Hot Fuzz. "What's Hot Fuzz?" I asked. "It's Hot Fuzz! With Simon Pegg!" my friends responded. And then they stuffed me in the car and drove off to the theater. I had never heard of Simon Pegg. I had no clue what this movie was about. I spent a very long, unpleasant time in the theater suffering through a series of uncomfortably awkward character interactions. Then Simon Pegg kicked an old woman in the face, and I realized, "OH! IT'S A COMEDY!" It was a completely different movie from there on out, and this epiphany allowed me to enjoy the entire thing on a rewatch.
Expectations can radically affect the perceived quality of a thing. Viewed as a serious drama, Hot Fuzz is terrible. Viewed as a traditional Mega Man game, OH JOES! is a poorly designed disappointment. In lieu of overhauling the game to address the broadest complaints, I resolved to set clear expectations on the download page and wherever else I advertised the game.
There's plenty more I could write about the playtesting process, but I'd like to conclude by showing you the process. Culled, with permission, from over 19 hours of footage, I present to you a tiny sample OH JOES! playtesting from March 2017 through April 2018. This casually edited video is by no means comprehensive, balanced, or polished; moreover, it's comprised of footage never intended for anyone's eyes and ears but mine. Still, I hope you find the video enlightening and entertaining.
A couple playtesters didn't seem to enjoy the game as much as the others, but they all had one thing in common: they had never heard of OH JOES! before I asked them to playtest. My other playtesters had been following the project to some extent, and they had a sense of what kind of game this was. This, in combination with some of the specific criticisms I received, led me to believe that false expectations might be influencing some playtesters' opinions. In fact, after I recommended using special weapons more often and approaching each challenge as a puzzle, one playtester who had given up on the game gave it a second chance and ended up having a blast.
It's like when my friends took me to see Hot Fuzz. "What's Hot Fuzz?" I asked. "It's Hot Fuzz! With Simon Pegg!" my friends responded. And then they stuffed me in the car and drove off to the theater. I had never heard of Simon Pegg. I had no clue what this movie was about. I spent a very long, unpleasant time in the theater suffering through a series of uncomfortably awkward character interactions. Then Simon Pegg kicked an old woman in the face, and I realized, "OH! IT'S A COMEDY!" It was a completely different movie from there on out, and this epiphany allowed me to enjoy the entire thing on a rewatch.
Expectations can radically affect the perceived quality of a thing. Viewed as a serious drama, Hot Fuzz is terrible. Viewed as a traditional Mega Man game, OH JOES! is a poorly designed disappointment. In lieu of overhauling the game to address the broadest complaints, I resolved to set clear expectations on the download page and wherever else I advertised the game.
There's plenty more I could write about the playtesting process, but I'd like to conclude by showing you the process. Culled, with permission, from over 19 hours of footage, I present to you a tiny sample OH JOES! playtesting from March 2017 through April 2018. This casually edited video is by no means comprehensive, balanced, or polished; moreover, it's comprised of footage never intended for anyone's eyes and ears but mine. Still, I hope you find the video enlightening and entertaining.
RSS Feed
