citra-web/site/content/entry/citra-progress-report-2017-august.md

+++ date = "2017-09-16T00:28:00-04:00" title = "Citra Progress Report - 2017 August" tags = [ "progress-report" ] author = "anodium" coauthor = "saphiresurf" forum = 3566 +++

2017 has been an amazing year, with more work having been put into the project than ever before, but it's not over yet! Last we met was June, and just two months later the Citra issue tracker is brimming with lots of changes once more. I am extremely excited for this month (and what's coming up the next few months) but we're getting ahead of ourselves! On this progress report, let's check out the big fish in the July and August pond of patches!

Updating The Software Renderer (This, that, here, there, and those) by wwylele and Subv

Citra has two main rendering backends, software and OpenGL, but until very recently, no one gave much attention to the software backend. The OpenGL backend is faster, makes better use of the GPU, and allows things such as texture forwarding for higher resolution rendering. But there is one thing on which the OpenGL backend falls flat on its face---accuracy.

However, wwylele has just revived the software renderer (inspired by an attempt Subv made prior), adding almost all of the features the hardware renderer had received over the years. In fact, every addition to the hardware renderer has been given an equivalent in software:

{{% table %}}

Feature	Hardware	Software
Fragment Lighting	#1264	#2766, #2822
Spot Lighting	#2727	#2871
Geometric Factor	#2776	#2872
Bump Mapping	#2762	#2891
{{% /table %}}

Despite the software backend being incredibly slow, it is important to have a complete software implementation of the 3DS' GPU so that Citra can be used as a stable, working, and complete reference implementation in the future, when obtaining a working real console may be much more difficult.

Handle Invalid Filenames When Renaming by j-selby

Citra emulates the 3DS system services at a high level of emulation, or HLE for short. What this means is that every time a 3DS application or game running in Citra makes a request to the 3DS System Software, Citra captures the request and tries to translate it into its PC equivalent, rather then running the 3DS System Software directly.

For example, if a game makes a call to FS:OpenFile, Citra in turn calls the operating system's file opening function (_wfopen_s() on Windows, or open() on macOS and Linux), with the path to the virtual SD card added to the beginning.

Now, on top of the usual names for files and folders, there's two special folders inside every single folder on your computer, . and ... These aren't actual folders in the sense that you can place files and other folders inside of them. Instead, they each symbolize the current folder, and the folder one level above it, respectively. For example, C:/Windows/System32/.. actually means C:/Windows/.

With this in mind, a few Citra developers believed a game could, in theory, chain multiple ..s together to get to a file they weren't supposed to know even existed, like /../../../../Documents/IMPORTANT.docx. Citra would then ask the operating system to open the file %AppData%/Citra/sdmc/../../../../Documents/IMPORTANT.docx, which actually means that it would open C:/Users/Anodium/Documents/IMPORTANT.docx!

Before you suggest that Citra simply ignore .., a game can use it for legitimate purposes. And if it were to use it, it would most likely crash, as the resulting file path wouldn't exist.

This was already handled for most file functions in Citra, opening, reading, writing, etc. except for a few things, like renaming a file. In this case, the malicious game could just ask Citra to change /../../../../Documents/IMPORTANT.docx's name to /delicious_secrets.docx, effectively moving IMPORTANT.docx into Citra's virtual SD card! From there, the game could just ask Citra to open /delicious_secrets.docx to read the file it was originally forbidden from accessing.

This patch now fixes this, such that if any 3DS game or application tries to do exploit the rename file function from inside Citra, rather than doing what it asks, Citra gives them ERROR_INVALID_PATH, which most games interpret by crashing. So far, we haven't found any real 3DS software that tries to do this, but at least now future attempts to do so are blocked.

Do note though, that most of us are not security experts, and even for those who are, harderning the software is much more difficult and much less worthwhile than actually having an accurate 3DS emulation. The surface area of an emulation project is huge, and so writing malicious code intending to exploit one is not too difficult in any case.

But at the same time, malware authors generally target either the largest populations, or the most lucrative populations, and emulation is neither. And for the effort required to do so, most will only toy with the idea in their head, or maybe write a proof-of-concept for fun, rather than genuine malicious intent.

Fix Edge Cases for TextureCopy by wwylele

The 3DS GPU has a data transfer mode called TextureCopy, which as the name says, is for copying textures but with a configurable gap in case the texture is going to be copied into a smaller resolution area. This mode is enabled by setting the 3rd flag in the GPU and causes it to ignore every other flag except the 2nd, which is used to tell it whether or not to crop the texture.

This is nice because it can be used as a quick and easy way for 3DS developers to duplicate textures, and can be used in situations such as the one pictured running on hardware in Pokémon Super Mystery Dungeon below. But when it came to running something that took advantage of this feature in Citra, it didn't always work the same as it would on console.

{{< figure src="/images/entry/citra-progress-report-2017-august/texturecopy-before.png" title="How jagged" alt="Pokémon Super Mystery Dungeon During Deoxy's and Rayquaza's Face Off I N S P A C E" >}}

Fortunately (and to much rejoicing!) wwylele stepped into the ring to wrestle with this issue. They prepared a test program to help gain an understanding of how the hardware handles the TextureCopy operation in comparison to Citra. After the hard work of doing the research was out of the way, wwylele implemented it in Citra.

{{< figure src="/images/entry/citra-progress-report-2017-august/texturecopy-after.png" title="Deoxys is having a bit of a hard time, no?" alt="Pokémon Super Mystery Dungeon During Deoxy's and Rayquaza's Face Off I N S P A C E" >}}

Use Docker For Linux Builds by j-selby

Ubuntu Linux 14.04 is the de-facto standard desktop Linux distribution. It's also old. Very old. So old, in fact, that the compiler it ships with can't compile Citra. And our buildbot, Travis CI, that automatically compiles and builds Citra from source, just so happens to use Ubuntu 14.04 VMs. :(

Formerly, we would update the compiler from a third-party repository before compiling Citra itself. This also had the side-effect of updating the standard library that comes with the compiler, as each compiler version is inextricably tied to the same version of library by design. Unfortunately, a recent update to the library was incompatible with a large majority of systems because it's too new for Ubuntu 14.04, breaking the Linux build once again.

Now, rather than building Citra directly inside Travis, a Docker container is started that's running Ubuntu 16.04 instead, which is much more well supported (and yes, it can compile Citra out of the box!).

UI Themes by Kloen

Kloen has put the time and work into Citra's Qt frontend to make it themeable. Now users can enjoy a dark mode and other custom colour schemes, just by editing a CSS stylesheet!

{{< figure src="/images/entry/citra-progress-report-2017-august/theme-comparison.png" title="CHOOSE YOUR CHARACTER" alt="Comparison of Dark Theme and Light Theme" >}}

Load Shared Font From System Archive by wwylele

This does not eliminate the need for dumping a shared font from a legitimate system.

This in itself isn't an extremely visible or perceivable change from a user perspective, but it is something that helps us take another stride towards accurately recreating the way the 3DS actually operates. All system data is uniformly stored in the system archive now that the system font can be included with it. Citra does still fall back to the deprecated shared_font.bin file if it was dumped before, for compatibility's sake. 3dsutil has been updated to be able to dump everything as a system archive, so that Citra may be able to work with it in a way that's more accurate to the actual hardware!

And Everyone Else

Just because your work wasn't written on here doesn't mean it's not as important! Every little bit counts, every pull request inches the project one step closer down the long road of accurately emulating the Nintendo 3DS. So I would like to give my thanks to everyone who's contributed these past two months, for giving us those extra metres down this road.