Bugs, bugs, bugs

Weekly post assuring you guys the corgi is still alive. 🙂

I fixed a problem with the ADC/SBC instructions (add/subtract with carry) relating to how the carry and overflow flags were updated – this allows Harvest Moon DS to boot, but it soon crashes afterwards due to unimplemented graphics registers. I’m trying to test other games as well, but many of them get stuck on a “save data error” screen. Which makes sense, as I’ve yet to implement that.

The way CorgiDS handles drawing to the screen is also funky:funwithsprites

As you can see, the Digimon aren’t looking too well. What’s happening is that the sprites are being overwritten by other sprite tiles with a “transparent” color, or more specifically, a color with 0 alpha. In the process of trying to fix this issue, I think it’s a good idea to rewrite the 2D graphics engines to take into account background and sprite priority. Currently CorgiDS uses hardcoded priorities based upon background/sprite number – which is normally accurate – but both can have custom priorities that supersede this rule. Even if it’s not super important now, the rewrite will allow for more flexibility with future graphical effects.

After the rewrite, my next task will be to handle rotscaling as well as windows (which should function like their Game Boy analog). Maybe engage in some bughunting as well. Plenty of work to do on the corgi!

Less bark, more bite

Two months of work have certainly paid off. While I haven’t gotten the firmware completely under control, I have accomplished something equally exciting: getting to the title screen of my first commercial game.

better

Digimon World Dusk: a childhood game of mine. Perhaps not the highest quality title for the DS, but it’s pretty decent. I chose this game (as well as a couple of others) due to its relative simplicity, in that it doesn’t use 3D graphics or any stupid tricks in order to play correctly; it just so happened that this was the first to boot successfully.

Make no mistake, however: CorgiDS is in no condition to be released in the foreseeable future. The emulation speed is horrible, only a small subset of the graphical features are in place, and certain bugs are still lingering about. For instance, Harvest Moon DS refuses to boot at all, and the firmware immediately skips to the boot menu without displaying the health and safety screen.

So, where to go from here? First I want to fix the remaining bugs so that my sizable game/homebrew library will be completely functional. Next I’ll work on implementing rotation and scaling for the 2D engine, and maybe along the way adding some much needed optimization. There’s some other 2D features I’ll need to add, but I’ll just implement those as needed.

After that, however? It’ll be time to start work on the 3D GPU, probably the most difficult part to get working. If you follow melonDS, you already know that this is also the weirdest part of the DS. Even now no emulator handles all of its edge cases correctly, and some of its more obscure features still need to be reverse-engineered. It should be a lot of fun. 🙂

Love, PSI

Slow week

Unfortunately I don’t have any visual results. The firmware almost reaches the starting screen (giving you health and safety information), and a single frame of it can be seen before the graphics become garbage. I implemented some very basic stubs for the sound and WiFi that don’t do anything, and I also fixed a pretty bad DMA bug that resulted in only one half or one fourth of the operation being completed. Most of the homebrew, as well as the few commercial games I’ve tested, fail to display anything at all – the ROMs affected seem to get stuck in an infinite loop before they reach the main game loop.

Most of my work now consists of comparing the results of no$gba to my emulator and trying to fix any differences. Tedious stuff, but I’m close to a breakthrough I feel…

Back in business!

After working on the GUI and fixing several mysterious bugs that only appeared after the migration (in particular a nasty buffer overflow), I can say that CorgiDS is growing at a nice rate. Check it out:

mandelbrot

Before you ask, no, CorgiDS isn’t magical enough to render this in 0 ms. The “time taken” works on DeSmuME, so I have no idea why it doesn’t on CorgiDS. Issues with the Real Time Clock might be a possibility, but it’s not exactly concerning me right now.

While the bottom screen in this demo only uses the simple “frame buffer” mode, the top screen makes use of one of the tiled background modes based upon the GBA hardware. If you’re aware of how the Game Boy draws things, the way the DS/GBA handles this is surprisingly similar. Unfortunately this is one of the simpler modes that commercial games and more sophisticated homebrew don’t make use of; I’ve yet to get into affine backgrounds or sprites, and all sorts of crazy things like rotation and scaling can be applied to those.

The GUI is much more advanced now and despite not having many options, feels almost complete. You can load a ROM, choose where the emulator looks for the BIOS and firmware images, and even save a screenshot; in fact I used that last feature for the above image. The best part is thanks to Qt, everything should not only be cross-platform, but also have a native appearance. I’ve kept my GUI code as separate as possible from the emulator code so adding new frontends, if needed, will be pretty painless. This may all seem like small stuff to be writing about, but a large number of emulators – not necessarily in the DS scene – have obtuse and complicated interfaces that scare away beginners, and it’s one of my goals to have an intuitive, easy to use GUI for CorgiDS.

My next big goal, assuming no major setbacks, will be to load the firmware and get the graphics from that displaying. This makes use of the aforementioned affine backgrounds and sprites, so this ought to keep me occupied for a while. Until next time!

Homebrew location: http://www.gamebrew.org/wiki/Mandelbrot_Fractal

Booting the Nintendo DS – a technical summary

The Nintendo DS boot process involves three parts – two BIOS ROMs for the ARM CPUs and a firmware image. While it is feasible to high-level emulate all three components and directly load a game ROM on start-up, such a task will seem daunting in the beginning of a DS emulator’s life, as it was for mine. Thus, in order to gain a more comprehensive understanding of the DS, I decided that my first goal would be to low-level emulate booting from the BIOS. Contrary to my expectations, this turned out to be far harder than I thought it would be. It was definitely an interesting experience, and I learned a lot from it, so I’d like to share some of the grisly details of the DS BIOS (here I will refer to both BIOS ROMs collectively unless stated otherwise).

The basics

First of all, what does the DS BIOS do? It is not simply a boot ROM like that present in the Game Boy. Rather, the BIOS also provides many useful services to games through the use of SWIs (software interrupts). These functions range from waiting for a specific interrupt to occur to providing decompression methods. Some of these services can be difficult to high-level emulate well, so many emulators require their users to dump the BIOS files from their DS. This provides more impetus to low-level emulate the boot process, at least in the beginning.

The BIOS also contains an exception vector table – simply put, a list of branch instructions leading to specific regions in the BIOS. The ARM architecture provides a form of exception handling – whenever an exception is fired, the CPU jumps to a specific instruction in the exception vector table depending on the type of exception. However, the DS mostly handles a subset of the supported exceptions – reset, IRQ (interrupt request), and SWI. Other exceptions are either not possible due to a lack of hardware or simply lead to a debugging function.

Next we’ll examine exactly how booting works, as well as the difficulties that arise from the process.

Booting the ARM9

Execution begins at memory address 0xFFFF0000, where the reset exception vector is located. This address holds a branch instruction that goes to this code:

Screen Shot 2017-07-27 at 9.29.11 PM

Some explanation is required. For those unaware with ARM architecture, the ARM9 and ARM7 each have sixteen 32-bit registers. Most of these are general-purpose, but some are usually only used for specific tasks, such as PC (program counter), SP (stack pointer), and LR (link register). The first two are self-explanatory, but the LR is interesting – when a function is called in ARM assembly, the return address is stored in LR. The first two lines of code store the value 4 into LR if it was equal to zero when the reset vector was called. I’m not sure why exactly this is necessary, but perhaps it has to do something with debugging – a value of zero might indicate problems with null pointers.

The rest of the code has to do with the POSTFLG I/O register. This register is zero when the DS first powers on and is set to one after the booting process is complete. Thus, if the reset vector is called in the middle of game execution (likely due to stray pointer bugs), the BIOS prepares to enter the exception debugging function. This “safeguarding” code is also present in the ARM7 BIOS, and it seems that accidentally calling the reset vector was a common enough issue that Nintendo believed it was necessary to provide some protections. What follows next is simply hardware initialization – setting stack pointers and the like.

The CP15

The only interesting part of the boot process here is the CP15, a coprocessor included within the ARM9 that handles setting up memory controls as well as other functions not too related to the DS. The important parts here are TCM (tightly-coupled memory) and the cache. Despite the ARM9 having a 66 MHz clock speed compared to the ARM7’s 33 MHz, the ARM9 has a severe hardware bug that makes accessing most memory much slower than it already is. Depending on the memory type, the ARM9 can end up being around two to four times slower than the ARM7. The TCM and cache are the only ways to alleviate this issue, as they can be fully used by the 66 MHz clock. The BIOS has to configure both by interacting with the CP15, which mainly consists of data transfers. As TCM is used in part to configure how interrupts are handled, it’s necessary to at least emulate this part; the cache can be ignored for the purposes of booting.

There isn’t much else to say in regards to the ARM9. Throughout the initialization procedure, the two CPUs will use a process called IPCSYNC to synchronize with each other. This works by sending a four-bit value (ranging from 0 to 15) to the other processor and waiting in a busy loop until the other CPU sends a certain value back. At a certain point, the ARM9 sticks itself in a busy loop until the ARM7 has finished all of its tasks, which brings us to our next topic…

Booting the ARM7

The ARM7 mostly has the same hardware initialization procedures, save for the fact that it doesn’t have a CP15 (and by extension, TCM and cache). The other little difference is that the ARM7 begins execution at 0x00000000 instead of 0xFFFF0000. However, the ARM7 has additional responsibilities, such as setting up the RTC (Real Time Clock) and loading the firmware and game cartridge into memory. In fact, the ARM7 is the only one with access to the first two components, as well as other details such as WiFi and the touchscreen.

The RTC isn’t strictly necessary to emulate, but I chose to do so anyway. The DS interacts with this by bitbanging an I/O register – only a single bit can be transferred at a time. The actual functionality of the RTC isn’t that complicated, only figuring out how to put these bits together. It’s worth noting that the time function on the RTC is used as a PRNG for the game cart code – more on that later.

The firmware lies outside of the ARM7’s memory range and is instead accessible by the Serial Peripheral Interface bus (this is also how the ARM7 pulls data from the touchscreen). The ARM7 will first configure the SPI by setting a control register, and then it begins data transfers by writing to a strobe register. One byte is sent or received at a time – a seemingly slow process, but the bulk of the data is only received during start-up. Once again, not a difficult process.

This pales in comparison to what happens next.

Loading the secure area

The game cartridge also lies outside accessible memory and is accessed through I/O registers. To interact with the cart, the DS sends an eight-byte command, and several cycles later, it reads one word (four bytes) of data. Once the DS has read a word, the cartridge will automatically send more data until a sufficient amount has been transferred. This process by itself is similar to using the SPI.

The start-up process works as follows: The DS sends a “dummy” command that probably acts as an activation signal on real hardware. Next, it retrieves the cartridge header – the first 512 bytes, containing information such as the locations in memory where the ARM9/ARM7 should begin execution of the game. After that, the DS retrieves the chip ID, which indicates certain things like the chip manufacturer and the size of the chip. Although the chip ID is not stored anywhere in the ROM, a fake but realistic one can be used instead in emulation. So far so good.

Then the DS sends a command indicating that all further commands will be encrypted, which is where the problems begin.

Nintendo saw it fit to encrypt most game ROMs using the Blowfish algorithm. However, this only happens within the first 2 KB of an 8 KB region called the “secure area.” The first eight bytes of the secure area consist of the double encrypted string “encryObj” – if decrypted successfully, this string is overwritten in memory with the word “0xE7FFDEFF”, but if not, the entire 2 KB is overwritten (as you can imagine, this doesn’t bode well for running the game). ROM dumps, however, tend to have this region already decrypted, so the emulator has to manually re-encrypt the secure area if one wishes to boot from the BIOS. Furthermore, as indicated above, the commands sent to the cartridge are also encrypted, so the emulator must decrypt them as well.

After encrypting all commands, the DS will then retrieve the chip ID either once or twice (which must match with the ID fetched earlier) depending on the value of the RTC. It will then load the 8 KB secure area in a random order using four 2 KB blocks. The BIOS then deactivates command encryption, synchronizes with the ARM9, and prepares to execute firmware code.

To say there is room for error here is an understatement. My initial implementations of CorgiDS were marred with bugs on both the CPU and cartridge side. For example, a particularly silly bug was my misunderstanding of the behavior of the carry flag. On the Game Boy Z80, the carry flag is set whenever there is a borrow from a subtraction operation x – y; in simpler terms, the carry flag is set when x < y. I had assumed this to also be the case for the DS, and only after vigorous debugging did I find out the ARM architecture uses the opposite behavior: the carry flag is set when there is NO borrow, or when x >= y. This wasn’t the only issue: many stupid bugs that only surfaced when the BIOS got to this part resulted in me having to spend a couple of weeks figuring out why nothing was working. Many times I wanted to completely give up, to shove this project aside as a failed experiment.

But finally, after completing this last test of will, CorgiDS began to execute the firmware. The elation I felt from seeing this happen was simply unmatched.

Modern consoles may have more complicated boot-up processes, up to sporting a custom operating system. However, the DS is not to be taken lightly either. Its innocuous exterior hides quirky and downright buggy hardware that games are more than happy to take advantage of, and the BIOS only touches a small portion of it. Nevertheless, booting successfully represents a major step in any emulator, as it indicates a large chunk of the hardware has already been emulated. Once this step has been completed, the project is finally worthy of being called an emulator, even if there’s far more work ahead.

(The header image for this post is a screenshot of CorgiDS running the “hello world” example that comes with libnds)

Migrating to Qt and bugs

My original code used SDL, which would be okay except it’s quite difficult to create useful GUIs with it. Thus, I installed Qt so that I could create a proper intuitive interface from the start. Unfortunately, this means all the SDL related code is nonfunctional for the moment; furthermore I’m not entirely familiar with Qt, so progress will be slow for now as I read tutorials and watch some videos to get me up to speed. This is meant to be a long-term project however, so by spending extra time now, CorgiDS will benefit all the more.

As for the bug: I noticed that certain homebrew that I tested would jump to uninitialized or zeroed-out memory; in either case, nothing with valid code to execute. I suspected this to be a problem with the CPU for a while, but as it turns out, this was a red herring. I pinpointed the exact location where the emulator messed up, and I fired up the nocash debugger. The code that I tested (dslinux) would jump to invalid memory if a certain part of memory was set to zero – nocash gave a non-zero value and worked fine. I looked up the memory address on GBAtek and found it had something to do with the firmware user settings.

There it was!

I had overlooked the firmware when writing my direct boot function. The firmware writes to memory the user settings, which contains useful data such as touchscreen calibration points and the user’s language. Without including this, the memory supposed to hold these values would always be zero, and as GBAtek specifies, some of these values are never supposed to be zero. Why the ROMs decided to shit themselves because of this I’m not entirely sure, but regardless, now that I’ve added this functionality, everything seems to work as it should.

My next goal will be to get CorgiDS to its former state before the Qt migration. This will take some time, and of course I’ll be adding more GUI specifics beyond that. Stay tuned!

What lies ahead?

Welcome to the CorgiDS development blog! I’m sure many of you have seen the reddit announcement of my new emulator, and I was pleased to see how much positive reception it received. It’s encouraged me to work even harder than before.

Now, an important topic is the road map I have planned for CorgiDS’s first release. What are all the features that need to be added before I see it fit to release my project to the public?

My feature list is split into two sections: essential and optional. Essential represents all the features I will add for v0.1, and optional represents those features that would be nice but aren’t guaranteed to be added for v0.1.

Essential:

  • A robust 2D graphics engine, obviously. It should be good enough to run several commercial games that don’t require the 3D portion of the GPU.
  • Two separate interfaces: a Qt frontend with full support, and a minimalist SDL frontend.
  • A powerful GUI debugger included within the Qt frontend. The debugger will support automatic disassembly, memory viewing, peek and poke functions, and program counter breakpoints. Not only will this help with development immensely, but also it ought to be quite useful for homebrew programmers as well as my fellow emudevs.
  • Optimization. CorgiDS is, at the moment, VERY slow. Even a simple “hello world” console test barely makes it above 30 FPS, so in that regard, CorgiDS is half as fast as a regular DS. I’ll need to rewrite the core loop to make things faster without sacrificing any accuracy.

Optional:

  • Basic 3D support. Arguably this should be in the essential section, but my goal for v0.1 is to be able to run some commercial games, and not all of them necessarily need 3D graphics. Nevertheless, I will more than likely put work into this section before the release. I will also need further optimization if I work on this.
  • Sound. It’s a pretty lovely feature for games, but its inclusion will complicate matters. This is not the most unlikely thing to add, however, and sound will be guaranteed in v0.2 if nothing else.
  • Hi-res support. If it’s not particularly difficult to implement, CorgiDS will come included with this.
  • A nice logo for the project, ideally including a corgi or two. I don’t have the skills necessary to create this, but if someone is willing to pitch in, feel free to contact me and we can work something out.
  • “DS View.” To my knowledge, the other emulators consist of the two DS screens stacked directly on each other in one window. At the moment, CorgiDS does the same as well, but my idea is to include a mode where the two screens are separated and placed on a background all within one window to reflect how an actual DS looks.
  • Savestates. In all honesty, this likely won’t be added for v0.1. The option is there, however.
  • A VRAM viewer for the debugger. This likely won’t be included either, but it is something to consider for future releases.
  • Other things I might be forgetting.

I can’t make any guarantees as to when you can expect the release, but likely something will be ready before the end of this year. Expect to see plenty more posts on here as time goes on, so be sure to follow the ongoing development of CorgiDS!