Progress Report 2017 Fourth Quarter (#73)

* Add 2017octnov Markdown boilerplate

* Write into

* Add TODOs

* Remove citra-emu/citra#3141

* Tweak 20180104

* test

* Write #3001

* Write #2993 #3048 #2975 #3029 #3113 #3144

* Write #2966

* Write #2982

* Write #2982

* Write outtro

* Add fixes proposed by @Subv

* Add boilerplate

* Add nits suggested by @Leo626

* Add candidate PRs

* Merge 2017octnov and 2017dec into 2017q4

* Add nit suggested by @bunnei

* Add banner by @bunnei

* Okay

* Mawp

* Write #3091

* Write #3184

* Write #3159

* Write #3116

* Write #3145

* Nit suggested by @jroweboy

* Reword #3091

* Add fsUSER-threads.png

* Add oot3dcia.png menucia.png

* Add fullscreen.png

* Fix metadata for release

* Apply some suggestions by @jroweboy

* Apply suggestion by @wwylele

* Add `figure`s as suggested by @jroweboy

* Add acnl-after.png

* Add acnl-before.png

* Replace acnl-after.png

* Add controller.png

* Retake controller.png

* Add promo.png

* Add shaderjit.png

* Quickfix bug

* Fix metadata for release

site/content/entry/citra-progress-report-2017-q4.md

@@ -0,0 +1,219 @@
++++
+date = "2018-02-04T17:00:00-04:00"
+title = "Citra Progress Report - 2017 Q4"
+tags = [ "progress-report" ]
+author = "anodium"
+forum = 9777
++++
+
+New year, new changes in Citra. Specifically, lots of technical changes under
+the hood, from applets to IPC, have made Citra even more accurate and laid a
+foundation for even more goodies the next year. With many great changes all
+around, this article is going to be packed to the brim with all the new goodies
+that have come to Citra during those crimson months.
+
+But, enough faffing about! Let's get right into it:
+
+## [citra-qt : Adding fullscreen mode](https://github.com/citra-emu/citra/pull/3001) by [Styleoshin](https://github.com/Styleoshin)
+
+One of the most requested features for Citra has been enlarging the window to
+cover the entirety of the screen. After almost a year of requests on both the
+Citra Discourse and Discord servers, [Styleoshin](https://github.com/Styleoshin)
+has finally delivered!
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/fullscreen.png" 
+    title="No more pesky window décor!" >}}
+
+Simply go into `View → Fullscreen`, or just strike the Alt + Return keycombo, and
+enjoy Citra to its fullest on your largest monitor.
+
+## Rewrite AM Service and Add CIA Installation ([Here](https://github.com/citra-emu/citra/pull/2993), [here](https://github.com/citra-emu/citra/pull/3048), [here](https://github.com/citra-emu/citra/pull/2975), [there](https://github.com/citra-emu/citra/pull/3029), [this](https://github.com/citra-emu/citra/pull/3113), and [that](https://github.com/citra-emu/citra/pull/3144)) by [shinyquagsire23](https://github.com/shinyquagsire23) and [BreadFish64](https://github.com/BreadFish64)
+
+The `am` service on a Nintendo 3DS handles the installation, tracking, and removal
+of applications and games installed on the console and SD card(s) via a centralized
+database in the NAND, with some parts of the database for SD card-installed titles
+on the SD card itself. It has a handful of nice features such as installing packages
+as they're streamed in, along with DLC support being cleanly implemented as a
+side-effect of the CIA format's design. CIAs themselves are also designed to be
+streamed in, which is used extensively across 3DS apps, such as the eShop streaming
+downloads directly into `am` so that it installs at the same time, or `dlp` using
+it to stream Download Play CIAs from one 3DS to another directly.
+
+The CIA format starts off with a header that describes where each component of
+itself starts within the stream and a bitfield. This is then followed by a
+certificate signed by Nintendo, the CIA's ticket, and the title metadata (or TMD
+for short). The TMD then contains a list of every CXI file the CIA could possibly
+have, along with their name, size, etc.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/oot3dcia.png" 
+    title="A diagram of *The Legend of Zelda: Ocarina of Time 3D*'s CIA" >}}
+
+The astute reader might've noticed I said that the TMD contains a list of every
+*possible* CXI. This is because a CIA might not contain every CXI for the game,
+such as in the case of DLC. The bitfield I mentioned in the header is used to list
+off which of the entries in the TMD actually exists within that CIA. This is
+(ab)used in some cases, such as the Home Menu Themes, in that every single theme
+that exists is actually a different CXI within the same CIA. Just tick the themes
+the user owns in the bitfield and then attach those CXIs at the end of it. Dead
+simple way for it to work perfectly with both streaming of the file, and making
+personalized CIAs for each eShop user.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/menucia.png" 
+    title="A tiny portion of the Home Menu Themes CIA" >}}
+
+For the `am` side of streaming installations, the way it handles a request to
+install an application is by creating and then giving the requester a handle to
+a virtual file. This leaves most of the busy work to `am`, while the app only needs
+to worry about writing the CIA file into that handle.
+
+[shinyquagsire23](https://github.com/shinyquagsire23) reimplemented the entirety
+of how Citra handles NCCH files, which quickly paved the way to implementing `am`
+in Citra. This allowed users to use FBI to install  applications from its virtual
+SD card. He and [BreadFish64](https://github.com/BreadFish64) also went a step
+further and added support for installing CIAs via the SDL and Qt frontends
+directly, removing the need to use FBI. Do note that you cannot yet run installed
+applications and games directly, but CXI executables can access the virtual SD
+card for DLC, updates, etc.
+
+## [Qtifw build installer](https://github.com/citra-emu/citra/pull/2966) by [j-selby](https://github.com/j-selby)
+
+Citra now has a fancy new installer and updater, thanks to [j-selby](https://github.com/j-selby)
+and [jroweboy](https://github.com/jroweboy)'s efforts! If you haven't already, we
+strongly recommend you download and install Citra through it, since it will make
+updating as easy as re-running the installer. You can check it out over [here](/download/),
+
+## [macOS: Build x86_64h slice](https://github.com/citra-emu/citra/pull/2982) by [MerryMage](https://github.com/MerryMage)
+
+A little known feature in macOS is fat binaries, the ability to have the same executable
+contain binaries for multiple architectures. This was used extensively around 2007
+to support Mac OS X's transition from PowerPC to x86_64, allowing developers to
+have one binary work on all Macs effortlessly.
+
+After poking around Apple's LLVM fork, [MerryMage](https://github.com/MerryMage)
+found that they had a specific flag to build binaries for Intel Haswell and later.
+Enabling it made a fat binary that contained both pre-Haswell and post-Haswell code,
+allowing Citra to take advantage of the newer CPU's instructions, without dropping
+support for older ones.
+
+## [Kernel/IPC: Add a small delay after each SyncRequest to prevent thread starvation.](https://github.com/citra-emu/citra/pull/3091) by [Subv](https://github.com/Subv)
+
+When communicating with services, titles on a real Nintendo 3DS expect to wait some
+amount of time before the service replies or responds. Until now, Citra didn't implement
+this, it responded without incrementing the virtual clock. From the emulated system's
+perspective, it appeared as though services replied and reponded literally instantly
+because of this, leading to strange side-effects.
+
+For example, some buildings in *Animal Crossing: New Leaf* fail to render and
+cannot be interacted with.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/acnl-before.png" 
+    title="Animal Crossing: New Leaf's acting up" >}}
+
+Or worse, some games such as *Star Fox 3D* can't even
+reach the title screen.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/fsUSER-threads.png" 
+    title="A demonstration of how instant service replies can break a title" >}}
+
+The example in this figure is taken directly from *Star Fox 3D*. When the game boots,
+it checks if any save data exists on the console's SD card, and tries to recreate it
+if it doesn't. It does this by sending a request to the service responsible for
+file access, and then waiting for a reply. Because the request is synchronous,
+the thread that made the request is put to sleep and its priority lowered. The
+game actually uses this time that would otherwise be spent waiting by having other
+threads run while the original thread was still asleep, so that by the time it
+woke up with the response, it would have other resources it needed ready to go.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/acnl-after.png" 
+    title="Animal Crossing: New Leaf, among other games, are now working perfectly with this change!" >}}
+
+As Citra's responses are instantaneous from the point-of-view of the game,
+the secondary thread doesn't have nearly enough time to finish its job. And, the
+first thread's priority comes back up above the secondary's once it wakes up,
+this leads to the first thread waiting on the secondary thread to finish, but it
+never gets a chance to, due to its now much lower priority. Essentially, the
+first thread waits on the secondary forever, because the secondary never gets a
+chance to actually finish what it was doing.
+
+[ds84182](https://github.com/ds84182) and [Subv](https://github.com/Subv) each
+wrote homebrew software which found and measured the issue, respectively, which
+[B3n30](https://github.com/B3n30) ran on a real Nintendo 3DS. With which, an
+average delay for every type of service reply was found. Then, [Subv](https://github.com/Subv)
+made Citra's virtual clock increment by this amount before fulfilling any service
+request, solving many of the issues this brought.
+
+## [core/arm: Improve timing accuracy before service calls in JIT](https://github.com/citra-emu/citra/pull/3184) by [MerryMage](https://github.com/MerryMage)
+
+As our previous section demonstrates, Citra's virtual clock should be
+incremented before any time code that was supposed to be on a real
+Nintendo 3DS runs within Citra, including services. This change by [MerryMage](https://github.com/MerryMage)
+makes services increment the clock *before* they are called, instead of after.
+
+<!-- NOTE: Possibly hard to understand/too verbose? Reword if so. -->
+One of the reasons this is important is that services can schedule calls to
+other services to run in the future. If the current time in the virtual clock is
+incorrect when the service schedule an event, it would run too early since
+the time for the scheduled event would effectively be shifted back by the virtual
+time it takes for the service that scheduled it would run.
+
+Although this doesn't fix any reported bugs, it does make the timing emulation
+of Citra much more accurate.
+
+## [citra-qt : Fix a bug in our fullscreen implementation](https://github.com/citra-emu/citra/pull/3159) by [FearlessTobi](https://github.com/FearlessTobi)
+
+In Citra's GUI, a strange bug exists, where if you have fullscreen enabled, and
+you start a game while the window is maximized, the window instead unmaximizes
+before starting the game. This was due to Citra not keeping track of the window's
+state (be it normal, maximized, minimized, or fullscreen), and instead simply
+defaulting to normal. [FearlessTobi](https://github.com/FearlessTobi) found,
+[reported](https://github.com/citra-emu/citra/pull/3001#issuecomment-338401048),
+and fixed the bug. Citra now keeps track of the window's state, position, and
+size, every time it goes fullscreen and restores them every time it leaves
+fullscreen.
+
+## [Allow input configuration with SDL joysticks](https://github.com/citra-emu/citra/pull/3116) by [muemart](https://github.com/muemart)
+
+In the input configuration menu, there has been the ability to change the key
+bindings on the keyboard for years now. Unfortunately, this menu allowed rebinding
+to keyboard keys only, not gamepads or any other input device one could use. This
+led to guides such as [this one](https://community.citra-emu.org/t/temporary-controller-configurations-for-citra/1061)
+being written, to help users manually change the configuration files if they
+wanted to use gamepads with Citra.
+
+This was extremely inconvenient and user-unfriendly to most, and so [muemart](https://github.com/muemart)
+took it upon themselves to finally add support for configuring gamepads within the
+configuration menu. Now, it's as simple as clicking a button, and pressing the
+corresponsing button on your controller, to set it up.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/controller.png" 
+    title="Citra's input configuration menu" >}}
+
+## [shader_jit_x64_compiler: Remove ABI overhead of LG2 and EX2](https://github.com/citra-emu/citra/pull/3145) by [MerryMage](https://github.com/MerryMage)
+
+The shader JIT in Citra is a component of the video core responsible for
+recompiling GPU shaders for the 3DS to x86 code, so that they can be run on the
+user's CPU directly.
+
+Shader intructions like LG2 (calculate binary logarithm) and EX2 (calculate binary
+exponential) can be run potentially thousands of times per second in a typical
+Nintendo 3DS title, so it's usually very worthwhile to try to optimize these as
+best as possible.
+
+In doing so, [MerryMage](https://github.com/MerryMage) actually rewrote these two
+instructions to pure x86 assembly, and runs them inline with the rest of the instructions.
+A faster algorithm, combined with not having to deal with the overhead of calling
+an external library for math functions, led to this change almost halving the
+amount of time it takes to calculate these!
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/shaderjit.png" 
+    title="Graph comparing times before and after this change" >}}
+
+## Conclusion
+
+As always, [changes big or small](https://github.com/citra-emu/citra/graphs/contributors?from=2017-10-01&to=2017-12-31&type=c)
+are vital to the project. Brick by brick, Citra will eventually be an accurate enough
+emulation of the Nintendo 3DS for most, if not all, uses. Thank you everyone for
+the hard work you've poured in.
+
+{{< figure src="/images/entry/citra-progress-report-2017-q4/promo.png" 
+    title="And more to come." >}}