docs/templates/pages/transport.html

{% extends "konami.html" %}
{% block title %}Network format{% endblock %}
{% block body %}
<h1>Network format</h1>

<p>eAmuse packets are sent and received over HTTP (no S), with requests being in the body of <code>POST</code> requests,
    and replies being in the, well, reply.</p>
<p>The packets are typically both encrypted and compressed. The compression format used is indicated by the
    <code>X-Compress</code> header, and valid values are
</p>
<ul>
    <li><code>none</code></li>
    <li><code>lz77</code></li>
</ul>
<details>
    <summary>Source code details</summary>
    <figure>
        <img src="./images/lz77.png">
        <figcaption><code>libavs-win32-ea3.dll:0x1000fa29</code></figcaption>
    </figure>
</details>
<p>Encryption is performed <b>after</b> compression, and uses RC4. RC4 is symmetric, so decryption is performed the same
    as encryption. That is, <code>packet = encrypt(compress(data))</code> and
    <code>data = decompress(decrypt(data))</code>.
</p>

<h2 id="keys">Encryption keys</h2>
<p>Encryption is not performed using a single static key. Instead, each request and response has its own key that is
    generated.</p>
<p>These keys are generated baesd on the <code>X-Eamuse-Info</code> header.</p>
<p>Keys follow the format <code>1-[0-9a-f]{8}-[0-9a-f]{4}</code>. This corresponds to
    <code>[version]-[seconds]-[salt]</code>. The salt is generated by a simple PRNG.
</p>
<details>
    <summary>The PRNG</summary>
    <pre>{% highlight "c" %}
uint32_t PRNG_STATE = 0x41c64e6d;

uint32_t prng() {
    int upper = (PRNG_STATE * 0x838c9cda) + 0x6072;
    PRNG_STATE = (PRNG_STATE * 0x41c64e6d + 0x3039) * 0x41c64e6d + 0x3039;
    return upper & 0x7fff0000 | PRNG_STATE >> 0xf & 0xffff;
}
{% endhighlight %}</pre>
    <p>This is a simple linear conguential pseudorandom number generator (what a mouthful) being stepped twice every
        call. The constants being used for the main LCPRNG are taken from glibc, along with the dropping of the "least
        random" bit. Where this implementation differs however is that the LCPRNG is stepped twice every call, and that
        a second LCPRNG is used for the upper 2 bytes.</p>
    <p>One interesting observation is that the "least random" bit, that is typically discarded by the
        <code>0x7ff...</code> mask is not actually being discarded here, as the upper two bytes of
        <code>PRNG_STATE</code>, not the lower two, are used unmasked.
    </p>
    <p>Another interseting observation is that due to the nature of LCGs, stepping it twice is no more secure than once.
        The implementation presented here is actually just a single step of
        <code>PRNG_STATE = (PRNG_STATE * 0xc2a29a69) + 0xd3dc167e</code>. Make sure to understand cryptography before
        trying to roll your own!
    </p>
</details>
<p>Our per-packet key is then generated using <code>md5(seconds | salt | ENC_KEY)</code>. <code>ENC_KEY</code> is
    currently <code>69d74627d985ee2187161570d08d93b12455035b6df0d8205df5</code> for all games.
</p>
<details>
    <summary>Source code details</summary>
    <p>The interesting stuff can be found at <code>libavs-win32-ea3.dll:0x1002a800</code>. Rather than screenshots, I've
        gone and tidied up the code somewhat to make it easier to follow. <code>eamuse_info</code> is pre-populated with
        <code>"X-Eamuse-Info: 1-"</code> by the calling function (<code>0x1000eeed</code>) after which the pointer is
        incremented to leave it right after that <code>-</code>, ready for us to <code>vsnprintf</code> into it.
    </p>
    <pre>{% highlight "c" %}
static const char *ENC_KEY[26] = "Wait you didn't think I'd put this here, did you?";

int xrpc_crypt(char *packet, char *xeamuse_info) {
    char md5_key[16];
    char key[32];

    // Copy the {8}-{4} hex char pairs into key as 6 bytes
    if (copy_from_hex(key, 4, xeamuse_info) == -1)
        return -1;
    if (xeamuse_info[8] != '-')
        return -1;
    if (copy_from_hex(key + 4, 2, xeamuse_info + 9) == -1)
        return -1;

    // Add our constant key after the two variable parts...
    strncpy(key + 6, ENC_KEY, 26);
    // ...MD5 it all...
    mdigest_create_local(0, key, 32, md5_key, 16);
    // ...and use that digest as the key for RC4
    arc4(packet, md5_key, 16);
    return 0;
}

int xrpc_key_and_crypt(char *packet, char *eamuse_info, size_t size) {
    uint64_t miliseconds = read_timer(0);
    uint32_t seconds = __aulldiv(miliseconds, 1000, 0);
    uint16_t salt = prng() & 0xffff;

    int bytes_formatted = vsnprintf(eamuse_info, size, "%08x-%04x", seconds, salt);
    if (bytes_formatted < size) {
        xrpc_crypt(packet, eamuse_info);
    }
    return bytes_formatted;
}
{% endhighlight %}</pre>

    <p></p>
</details>

<h2 id="lz77">LZ77</h2>
<p>Packets are compressed using lzss. The compressed data structure is a repeating cycle of an 8 bit flags byte,
    followed by 8 values. Each value is either a single literal byte, if the corresponding bit in the preceeding flag is
    high, or is a two byte lookup into the window.</p>
<p>The lookup bytes are structured as <code>pppppppp ppppllll</code> where <code>p</code> is a 12 bit index in the
    window, and <code>l</code> is a 4 bit integer that determines how many times to repeat the value located at that
    index in the window.</p>

<p>The exact algorithm used for compression is not especially important, as long as it follows this format. One can
    feasibly perform no compression at all, and instead insert <code>0xFF</code> every 8 bytes (starting at index 0), to
    indicate that all values are literals. While obviously poor for compression, this is an easy way to test without
    first implementing a compressor.</p>
{% endblock %}