Maze Tussle

Pinball Deluxe kept crashing in SameBoy. Lior Halphon checked the timing, simplified the game’s behavior, and tried the same experiments on a real Game Boy. The handheld carried on playing. The emulator did not. The answer, as Lior recounts it, was an interrupt behavior documented by fellow researcher Gekkio. With the game’s timing already so tight, one extra cycle could be enough to bring it down. The real hardware in that story has a personal history. Lior first got to know Nintendo’s handheld through emulation and Pokémon, then bought a Game Boy Color in 2004. Years later, a flash cartridge let him bring his own hacks and demos to the same machine. It is still his favorite handheld and the one he reaches for when he wants to see what the hardware actually does. SameBoy began as a small personal project. Lior saw room for a better Game Boy emulator on the Mac, and soon set himself the much harder task of matching real hardware closely enough that software could not tell the difference. His experiments with audio uncovered a problem with the accepted explanation for frequency sweeps. His work on the display meant accounting for the moments when a game writes to a register while the screen is being drawn. The audio processing unit, or APU, and the picture processing unit, or PPU, come up often in the conversation below. There is another side to SameBoy. Open-source boot ROMs let players get started without supplying their own startup firmware. Recreating accessories led Lior to the unreleased WorkBoy, whose software gradually revealed its features as he worked out how it talked to the handheld. And when Digital Eclipse used SameBoy for commercial releases, Lior got to see his own name in the credits alongside games he remembers playing as a kid. This interview is part of Retro Replay’s History of Emulation series. Our SameBoy profile covers supported systems and features. Interview by David Becker for Retro Replay. SameBoy’s replacement boot ROM in a desktop emulator window. SameBoy project screenshots. Explore the interview The Game Boy Color he still tests on An audio test passes for the wrong reason When two Game Boy Colors disagree What the display and sound hardware really do Writing a new boot ROM Super Game Boy, WorkBoy, and unusual cartridges One fix, thousands of games The games Lior plays for fun From a Mac project to commercial releases The Game Boy Color he still tests on Retro Replay: Let’s start with a traditional one. What was your first experience with the Game Boy, and was it a system you had a strong personal attachment to before you ever thought about writing an emulator for it? Lior Halphon: Nintendo consoles weren't very popular in my country until the Wii era, so I was mostly experiencing Nintendo games through emulation. I got into Game Boy emulation specifically when Pokémon started airing, and I loved the console ever since. I eventually got a Game Boy Color as my first console in mid-2004, but that was after the console was already discontinued, so I didn't play it that much. Almost a decade later, I bought a Flash Cart for the same Game Boy Color, allowing me to play my own ROM hacks and demos on it, giving it a new life. That same Game Boy is still my favorite and go-to test device when researching. Retro Replay: Before I drag you back to the beginning of SameBoy, what part of the project is most interesting to you technically right now? Lior Halphon: SameBoy's development always goes in two fronts – the accuracy front and the user-experience front; and honestly, I enjoy both equally. In the accuracy front, it's always fascinating to discover new things about hardware that is almost 40 years old. Even before the development of SameBoy even started, I discovered two APU debug registers which would eventually allow Game Boy Color audio emulation to reach the same accuracy. In the UX front, I love coming up with new ways to experience Game Boy games. This is why SameBoy has so many unique features such as Apple Watch support, audio-based controller rumble, and file browser ROM previews. Retro Replay: When you started SameBoy, Game Boy emulation had already existed for decades. What made you look at the existing landscape and decide there was still room, or perhaps a need, for another emulator? Lior Halphon: SameBoy originally started as just another toy project of mine, and I didn't have actual concrete goals when I started developing it. However, during that time, the state of Game Boy emulation on Mac was quite bad. Windows users had BGB with its high accuracy and features, and various forks of VBA that were quite feature-complete. However, Mac users only had a few obscure and inaccurate emulators that were missing a lot of basic features, and the SDL version of VBA which was very buggy and was frequently freezing and crashing. This eventually helped me shape SameBoy's goals and develop it into something more than a toy project. Retro Replay: Was extreme accuracy one of the goals from the very beginning, or did the project become increasingly accuracy-focused as you discovered how many hardware details were still poorly understood? Lior Halphon: Well, yes and no. As mentioned, SameBoy originally started as a toy project without very concrete goals, but I did have a vague goal of making it "as accurate as I can". Eventually, but not long into its development, being indistinguishable from real hardware became one of SameBoy's primary goals. This is, by the way, why it's called SameBoy! It behaves just the same! An audio test passes for the wrong reason Retro Replay: Was there an early moment when SameBoy behaved correctly in a situation where another emulator did not, and you realized that your approach might actually be uncovering something important? Lior Halphon: One of the most popular test suites for Game Boy emulators is Blargg's. The APU tests in Blargg's suite were famously hard to pass, but they were very well documented, and each edge case they tested was nicely explained. Back in 2017, SameBoy was still failing these tests. Since I was aiming for sample-perfect accuracy, and Blargg's tests were written before I discovered the APU debug registers, I decided it's time for me to write my own test suite based on these registers while completely rewriting audio emulation in SameBoy as I do. I slowly progressed, passing both my and Blargg's tests, until I reached frequency sweeps. Blargg's explanation of how sweep behaves was regularly contradicting the behavior I was seeing on real hardware. If I implemented Blargg's behaviors, I would pass his tests, but fail mine. Meanwhile, real hardware passed all tests. Eventually, I figured out that Blargg's explanation of the behavior is simply wrong; it assumes some "locking" mechanism that simply never existed on real hardware! After a few adjustments, SameBoy passed all tests without implementing this non-existent locking mechanism. Until then, every emulator that passed these tests implemented a hardware feature that simply never existed in reality! Retro Replay: People often describe the Game Boy as relatively simple hardware. After working on SameBoy, how misleading is that description? Lior Halphon: At its core, this claim isn't actually wrong. The Game Boy, in a nutshell, is quite simple if you understand how it works. Take the APU for example – it is, in its essence, just a bunch of counters and shift registers connected to each other in ways that create different kinds of waveforms. The catch, however, is that this simplicity is a double-edged sword. The hardware itself is so simple that it's simply lacking any means of handling edge cases, such as race conditions and signal propagation delays. There are so many hardware quirks that are caused by the CPU writing to a register while the rest of the hardware is simply not expecting it. So yes, the Game Boy is a relatively simple hardware, but paradoxically, its simplicity is what makes it complex. Retro Replay: What part of the Game Boy turned out to be far more complicated than you expected when you first started? Lior Halphon: The sweep envelope. For the most part, the APU is quite well designed and implemented in hardware, with two exceptions. One is the volume envelope, which could benefit from a simpler design that would have prevented a lot of hardware quirks while saving die space, and the second is the sweep envelope. This feature is absurdly buggy in hardware, has so many edge cases, heavily relies on signal propagation delay, suffers from register data corruptions, and has instance-specific behaviors which are often non-deterministic. All of this just for a simple sweep feature that only exists in a single audio channel! When two Game Boy Colors disagree Retro Replay: SameBoy models differences between actual hardware revisions rather than treating every Game Boy or Game Boy Color as one identical machine. How significant can those revision differences actually be to software? Lior Halphon: For the most part, they're extremely minor. Early revisions of the Game Boy Color have quite a few bugs in the APU. Later revisions of the Game Boy Color introduce a few timing changes to the PPU, which can break some visual effects or change RNG behavior. For example, “Baby Felix - Halloween”'s language selection screen displays each letter in a random color. If you boot the game on different Game Boy revisions, you will get different colors! Retro Replay: Is there a particular hardware revision difference that sounds almost insignificant on paper but can completely change the behavior of a test ROM or game? Lior Halphon: The first Game Boy Color SoC revision introduced quite a few APU bugs and differences compared to the APU found in the original Game Boy. If you play certain games on these revisions, some audio channels can be completely missing. The most notable game that is glitched audio on older revisions is Prehistorik Man. These bugs were gradually fixed in later revisions. Retro Replay: SameBoy is known for extremely precise LCD timing. Why is the Game Boy LCD controller such a difficult component to reproduce accurately? Lior Halphon: It's a combination of multiple things, but in my opinion, it mostly stems from the "double-edged simplicity" mentioned earlier. The Game Boy won't stop you from changing PPU registers at pretty much any time, but it doesn't have any intentional handling of any edge case. Combine this freedom with the fact that an LCD-based console doesn't have to output pixels in a constant rate (like home consoles outputting analog signals do), and you've got some really tough emulation edge cases. What the display and sound hardware really do Retro Replay: Do projects like GBVideoPlayer, Demotronic, or unusual commercial games sometimes function almost like hardware research tools because they push the system so far beyond ordinary use? Lior Halphon: Not particularly. Yes, sometimes demos and graphically impressive games reveal emulation inaccuracies, but even in these cases, they almost always exploit just a single edge case in one specific configuration, which normally isn't enough to draw any meaningful conclusions on how the hardware actually behaves. When I encounter such ROMs, I usually try to reduce their behavior to a simple test ROM that fails on SameBoy and passes on hardware, then I expand it to cover more edge cases. Game Boy Color demos in SameBoy. Lior describes using small test ROMs to isolate the behaviors that larger demos expose. SameBoy project screenshots. Retro Replay: Have you ever implemented behavior that looked completely nonsensical, only to test real hardware and discover that yes, the actual Game Boy really does behave that way? Lior Halphon: In recent years, my development process almost always includes running test ROMs on real hardware and reading the schematics before I even consider implementing a behavior, so these situations don't actually happen. Retro Replay: What is the strangest Game Boy hardware quirk you have uncovered during SameBoy development? Lior Halphon: That's a tough one, but there are a few good contenders. One of my favorites is an APU bug. The wave channel normally reads its wave data linearly, one byte after the other. But if certain conditions are met, the APU will suddenly use the lowest nibble of the CPU's address bus as the address to wave RAM instead of the expected value. Retro Replay: SameBoy also puts a great deal of emphasis on accurate audio. What makes reproducing the Game Boy APU correctly more complicated than the relatively simple four-channel specification makes it appear? Lior Halphon: One important detail to remember about the Game Boy audio system is that it isn't fully digital. Each channel has its own Digital-to-Analog converter, which can be disabled and enabled dynamically, and their analog characteristics need to be emulated for the sound to sound right. Mixing is also done completely in an analog manner. You also have to accurately apply high-pass and low-pass filters. But even inside the digital realm, there are a lot of things to consider. The APU has many, many bugs. Internal registers can be corrupted by mostly normal use. Certain register writes have unintended side effects, which games can intentionally and unintentionally rely on. Poorly tested features such as the sweep envelope can unexpectedly halt their channel too early. Retro Replay: Is there an audio behavior that most players would probably never consciously notice, but once you knew it was wrong, you simply could not leave it alone? Lior Halphon: It's probably the noise register. The noise register's Linear Feedback Shift Register (LFSR) is what generates the actual noise used in the channel. Due to how LFSRs work, changing the noise characteristics while the channel is active has a chance to mute the channel. However, on the Game Boy, these also cause weird corruptions of the LFSR register itself, which are revision-specific, and sometimes even instance-specific and non-deterministic. Incorrect emulation of these can cause the channel to mute when it shouldn't, and vice versa. It was an extremely tedious behavior to research, but it's almost 100% accurate. (The involvement of non-determinism here makes it essentially impossible to fully emulate accurately.) Writing a new boot ROM Retro Replay: SameBoy includes open-source replacements for the original boot ROMs. What originally motivated you to recreate those rather than simply requiring users to obtain Nintendo’s ROMs themselves? Lior Halphon: SameBoy has a huge focus on ease of use and requiring zero-configuration. Requiring a user to obtain ROMs manually is not user-friendly at all. It's possible for an emulator to work without boot ROMs at all, but there's a lot of functionality you'd be missing without it. The solution was to create my own boot ROMs, replicating the original boot ROMs' features without copyright infringement. Since there's a limit to how accurate you can make the boot ROMs without infringing copyright (getting the timing to exactly match in every single case basically requires using the exact original code), there's still the option to use official ROMs. Retro Replay: Reverse engineering the Game Boy Color boot ROM also meant reproducing its game-specific palette behavior. How difficult was figuring out exactly how Nintendo was identifying and colorizing older Game Boy games? Lior Halphon: It wasn't something that wasn't done before; the guys over at The Cutting Room Floor did it before SameBoy's development was even started. However, the Game Boy Color boot ROM is tiny and mostly simple, so it wasn't very hard to reverse. I'm not a stranger to reverse engineering embedded code. Retro Replay: Was there a moment during that work when you realized the boot ROM itself contained significantly more interesting behavior than most emulator users probably assumed? Lior Halphon: Most of the weird quirks of the boot ROM were discovered before I started my own work on it, but there's one behavior that I was the first to notice. A Game Boy ROM's header has a bit that tells the boot ROM that the ROM needs to run in GBC mode. If that bit is not set, a value is written to a hardware register that enables DMG compatibility mode. But the interesting part is that if that bit is set, the value that is written to that hardware register is almost fully controlled by the ROM. That seemed to be by design. I was experimenting with these values and found that certain bits put the Game Boy in weird half-DMG half-CGB modes that behaved very weirdly. When the 2019 Gigaleak happened, it was discovered that these modes were part of a scrapped hardware feature that would allow specialized hardware in cartridges to directly drive the LCD of the Game Boy through the cartridge connector! Super Game Boy, WorkBoy, and unusual cartridges Retro Replay: Super Game Boy support creates an unusual problem because you are reproducing Game Boy behavior while also accounting for something originally running through SNES hardware. What makes accurate Super Game Boy emulation particularly challenging? Lior Halphon: You basically have to create two cycle-accurate emulators and sync them perfectly – a Game Boy emulator and an SNES emulator. This is why it took me some time to add SGB emulation, as I wasn't sure how to even tackle it, since an SNES emulator was completely out of scope. I ended up opting for high-level emulation, but it turned out so convincing that some people believed I actually used low-level emulation! Not long after I added SGB support, Near approached me and offered SameBoy integration into bsnes. After a few days of work, we got SameBoy running as the Game Boy side of bsnes' SGB emulation. By combining our two highly-accurate emulators, we managed to create the most accurate SGB emulator ever created! It manages to pass quite a few tests that no other emulator can, including standalone SameBoy! Retro Replay: SameBoy also emulates things like the Game Boy Camera, Printer, infrared communication, WorkBoy, motion controls, and obscure cartridge hardware. Which peripheral has been the most entertaining to implement? Lior Halphon: It's probably the Workboy. There was basically nothing known about this piece of hardware other than a few old photos, and suddenly the Workboy ROM drops. I was slowly reversing the ROM to understand the protocol, implementing it bit-by-bit until the ROM slowly starts to expose more of its features in SameBoy. It was really exciting to be the first person to see these features in decades! SameBoy’s Printer Feed window shows an emulated Game Boy Printer output. SameBoy project screenshots. Retro Replay: Which peripheral has been the biggest pain in the neck? Lior Halphon: There's an obscure Japanese Tamagotchi game that uses a specialized cartridge running some custom and non-trivial firmware. Unfortunately, nobody has ever dumped the cartridge firmware, and I don't have the cart myself (or the skills and equipment to dump it). Since this isn't just a simple mapper like most cartridges, I'd like to low-level emulate it. However, without the firmware dump, it remains one of the only official cartridges to not be supported at all in SameBoy. One fix, thousands of games Retro Replay: Automated testing is now a huge part of emulator development. SameBoy regularly tests thousands of games in addition to dedicated hardware test ROMs. How much has automated testing changed the way you develop the emulator compared with manually launching games and looking for problems? Lior Halphon: During the time I was swiftly making big discoveries that had great implications on emulation, the automation was extremely useful. Sometimes I'd run an automation after emulating a new quirk, only to discover it introduced graphical glitches to 20 random games. Without the automation, I wouldn't be able to make any big accuracy improvements without having to spend months of regression fixes after every release. Retro Replay: Have you ever made a tiny accuracy improvement only to have the automated tests reveal that it unexpectedly fixed a large number of games? Lior Halphon: Yes! Originally that was a major goal of the automation, as it was also designed to detect common game crashes. I'd pick a random crashing game, figure the bug out, fix it, and run another automation. It was actually quite common for me to check the automation status after such a fix and find out that a few dozen games were also fixed! Retro Replay: Flip that around: have you ever made what looked like an obviously correct change and watched the test suite suddenly explode with regressions? Lior Halphon: So many times! There are so many tiny PPU quirks that only trigger on extremely specific conditions. Sometimes my own implementation had a too broad condition, and the automation revealed that I broke half of the Game Boy library! Since these broken fixes were never pushed, these automation runs were never uploaded, but there were many of these. Retro Replay: What Game Boy or Game Boy Color game has personally caused you the most trouble as an emulator developer? Lior Halphon: Probably Pinball Deluxe. It was my first GitHub issue (I even opened it for myself!), and I battled it for years. I failed so many times when I tried to fix it. I verified my timings so many times, and they were perfect. I modified the ROM to simplify its behavior and see if it has any effect on real hardware – but nothing. The game always randomly crashed on SameBoy, but never on hardware. Turned out the game was relying on an edge case in how interrupt dispatching works on the Game Boy, which was found by Gekkio. Once I emulated it, the game finally stopped crashing! The annoying thing about this game is that its timing is so tight. If an inaccurate emulator's timing is completely off, it might actually work just fine. But if your timing is accurate and you miss this quirk, an interrupt handler might take just one cycle too long to execute, and you're going to crash. So much trouble for a game that isn't even good! The games Lior plays for fun Retro Replay: Fun one: when you are not examining it as a test case, what is your favorite Game Boy or Game Boy Color game purely as a player? Lior Halphon: This is going to be the most unoriginal answer, but Pokémon Crystal. I'm a Johto boy, Gen 2 is my favorite Pokémon generation, and I absolutely love these games. The modern ROM hacking scene for Gen 2 also allows these games to be so replayable. I often make my own ROM hacks of Pokémon Crystal for myself to play; I never grow tired of it. Pokémon Crystal Version on Game Boy Color. Lior calls it his favorite and still makes his own ROM hacks for it. Retro Replay: And if you could only keep ten Game Boy games for the rest of your life, which one is absolutely guaranteed a place? Lior Halphon: Definitely one of each Pokémon generation, together with Pokémon Pinball and Puzzle Challenge (I love Panel-de-Pon!). I'd also add a few other classics such as Link's Awakening DX, Super Mario Land, and Kirby Tilt 'n' Tumble. Mole Mania is definitely getting a place too; I think it's quite an underrated game. From a Mac project to commercial releases Retro Replay: SameBoy is now used as a core or component in other software in addition to being a standalone emulator. Is it strange seeing code you originally wrote for your own project become infrastructure inside applications you had nothing to do with creating? Lior Halphon: It's strange, but it's really satisfying to see SameBoy being used in cool and unexpected ways. Like inside Windfish, an interactive Game Boy disassembler, or RetroPlug, that makes a VST out of SameBoy. Retro Replay: Have any commercial developers or publishers approached you about using SameBoy for official releases or preservation work? If so, does seeing an emulator become part of an official product change the way you think about emulation’s relationship with the games industry? Lior Halphon: Yes, Digital Eclipse, for example, is regularly using SameBoy for their commercial Game Boy re-releases. Seeing my name in the credits of a commercial console game release for the first time felt so weird. It's so nice when these are also ports of games I really liked playing as a kid, such as Lion King, Teenage Mutant Ninja Turtles, and Yu-Gi-Oh!. Several other commercial and indie game developers have used SameBoy to release their games on consoles and PC too, and it's sure nice seeing emulation getting its spot in the industry. Retro Replay: SameBoy is now available through Apple’s App Store, something that would have sounded almost impossible during much of emulator history. What was it like watching emulation move from something platform holders often resisted to software that can now exist openly in mainstream app stores? Lior Halphon: I was genuinely shocked about Apple's change at first. Apple was always very against having emulators in the App Store in order to preserve good connections with Nintendo and other console manufacturers. On the technical level, Apple also used to not allow applications that could execute user-provided code, even non-native code, which is exactly what an emulator does. So seeing Apple suddenly allowing emulators almost explicitly was so unexpected! Being on the App Store greatly expanded SameBoy's reach, and it also allowed me to add a few features that couldn't be reasonably added in the sideloaded/jailbreak version of SameBoy, such as Apple Watch support. Retro Replay: If you could sit down with one original Nintendo Game Boy hardware engineer for an afternoon and ask them anything you wanted, what unresolved question would you ask first? Lior Halphon: I'd ask about the Game Boy Color prototype boot ROMs. Prototype Game Boy Color models had a feature that allowed them to dynamically change the palette while playing older, non-Color games. The hardware is still there, but since the Game Boy Color boot ROM won't use it, it can't be used. The development of this feature is so interesting to me. How did they plan to implement such features without breaking games? What eventually led them to drop it? What other implementations were considered? I'm also very curious about other unused/internal hardware features. I'd also love to know the official names of the undocumented PCM registers I found in 2014! Retro Replay: Finally, when this interview is published, are there any SameBoy resources, technical documentation, test ROM projects, repositories, current projects, homebrew efforts, or anything else you would like me to direct readers toward? Lior Halphon: SameBoy can be downloaded from its official website or the iOS App Store. It's being actively developed on GitHub. It's worth mentioning that although accurate Game Boy emulation is a "highly competitive" field, Game Boy research is still a group project. I often “share notes” with other developers and researchers, such as BGB developer beware, GSE developer CasualPokePlayer, mooneye-gb developer Gekkio, Emulicious developer Calindro, and Beaten Dying Moon developer Matt Currie. The latter two also helped me find and obtain a rare Game Boy Color revision. Other extremely useful resources I've been using are Furrtek's DMG-CPU schematics, which I've used to find countless previously undiscovered bugs, as well as the more recent schematics by Michael Singer. SameBoy downloads and research Desktop downloads and feature documentation. SameBoy official site SameBoy for iPhone, iPad, and Apple Watch. App Store listing Source code, issues, and release notes. SameBoy on GitHub Hardware research and emulator tests. SameSuite Thank you, Lior Our thanks to Lior for taking the time to talk through the games, the tests, and the strange things that happen between one clock cycle and the next. The featured cover incorporates official SameBoy screenshots. SameBoy project screenshots.