mirror of
https://gitea.tendokyu.moe/eamuse/docs.git
synced 2024-11-23 22:40:57 +01:00
Initial
This commit is contained in:
Bottersnike
commit
546df244b0
BIN
images/encoding_table.png
Normal file
BIN
images/encoding_table.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
BIN
images/no_array.png
Normal file
BIN
images/no_array.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
BIN
images/no_array_2.png
Normal file
BIN
images/no_array_2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.5 KiB |
BIN
images/types_table.png
Normal file
BIN
images/types_table.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
BIN
images/unsorted/javaw_DSoqceZKFz.png
Normal file
BIN
images/unsorted/javaw_DSoqceZKFz.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
BIN
images/xml_encoding_table.png
Normal file
BIN
images/xml_encoding_table.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
BIN
images/yes_array.png
Normal file
BIN
images/yes_array.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
58
index.html
Normal file
58
index.html
Normal file
@ -0,0 +1,58 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>eAmuse API</title>
|
||||
|
||||
<link rel="stylesheet" href="styles.css">
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<table>
|
||||
<tr>
|
||||
<td><a href=".">Contents</a></td>
|
||||
<td><a href="./transport.html">Transport layer</a></td>
|
||||
<td><a href="./packet.html">Packet format</a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<h1>Benami/Konami eAmuse API</h1>
|
||||
<p>Why?</p>
|
||||
<p>I was curious how these APIs work, yet could find little to nothing on Google. There are a number of
|
||||
closed-source projects, with presumably similarly closed-source internal documentation, and a scattering of
|
||||
implementations of things, yet I couldn't find a site that actually just documents how the API works. If I'm
|
||||
going to have to reverse engineer an open source project (or a closed source one, for that matter), I might as
|
||||
well just go reverse engineer an actual game (or it's stdlib, as most of my time has been spent currently).</p>
|
||||
<p>These pages are very much a work in progress, and are being written <i>as</i> I reverse engineer parts of the
|
||||
protocol. I've been asserting all my assumptions by writing my own implementation as I go, however it currently
|
||||
isn't sharable quality code and, more importantly, the purpose of these pages is to make implementation of one's
|
||||
own code hopefully trivial.</p>
|
||||
<p>Sharing annotated sources for all of the games' stdlibs would be both impractical and unwise. Where relevant
|
||||
however I try to include snippets to illustrate concepts, and have included their locations in the source for if
|
||||
you feel like taking a dive too.</p>
|
||||
<p>If you're here because you work on one of those aforementioned closed source projects, hello! Feel free to share
|
||||
knowledge with the rest of the world, or point out corrections. Or don't; you do you.</p>
|
||||
|
||||
<h1>Contents</h1>
|
||||
<ol>
|
||||
<li><a href="./transport.html">Transport layer</a></li>
|
||||
<ol>
|
||||
<li><a href="./transport.html#packet">Packet structure</a></li>
|
||||
<li><a href="./transport.html#type">Types</a></li>
|
||||
</ol>
|
||||
<li><a href="./packet.html">The inner packet structure</a></li>
|
||||
<ol>
|
||||
<li><a href="./packet.html#xml">XML packets</a></li>
|
||||
<li><a href="./packet.html#binary">Binary packed packets</a></li>
|
||||
<li><a href="./packet.html#schema">Binary schemas</a></li>
|
||||
<li><a href="./packet.html#data">Binary data</a></li>
|
||||
</ol>
|
||||
</ol>
|
||||
|
||||
<p><small>This site intentionally looks not-great. I don't feel like changing that, and honestly quite like the aesthetic.</small></p>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
893
packet.html
Normal file
893
packet.html
Normal file
@ -0,0 +1,893 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Packet format | eAmuse API</title>
|
||||
|
||||
<link rel="stylesheet" href="styles.css">
|
||||
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<table>
|
||||
<tr>
|
||||
<td><a href=".">Contents</a></td>
|
||||
<td><a href="./transport.html">Transport layer</a></td>
|
||||
<td><a href="./packet.html">Packet format</a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<h1>Packet format</h1>
|
||||
|
||||
<p>eAmuse uses XML for its application layer payloads*. This XML is either verbatim, or in a custom packed binary
|
||||
format.<br /><small>*Newer games use JSON, but this page is about XML.</small></p>
|
||||
|
||||
|
||||
<h2 id="xml">The XML format</h2>
|
||||
|
||||
<p>Each tag that contains a value has a <code>__type</code> attribute that identifies what type it is. Array types
|
||||
have a <code>__count</code> attribute indicating how many items are in the array. Binary blobs additionally have
|
||||
a <code>__size</code> attribute indicating their length (this is notably not present on strings, however).</p>
|
||||
<p>It is perhaps simpler to illustrate with an example, so:</p>
|
||||
<pre><code><?xml version='1.0' encoding='UTF-8'?>
|
||||
<call model="KFC:J:A:A:2019020600" srcid="1000" tag="b0312077">
|
||||
<eventlog method="write">
|
||||
<retrycnt __type="u32" />
|
||||
<data>
|
||||
<eventid __type="str">G_CARDED</eventid>
|
||||
<eventorder __type="s32">5</eventorder>
|
||||
<pcbtime __type="u64">1639669516779</pcbtime>
|
||||
<gamesession __type="s64">1</gamesession>
|
||||
<strdata1 __type="str" />
|
||||
<strdata2 __type="str" />
|
||||
<numdata1 __type="s64">1</numdata1>
|
||||
<numdata2 __type="s64" />
|
||||
<locationid __type="str">ea</locationid>
|
||||
</data>
|
||||
</eventlog>
|
||||
</call></code></pre>
|
||||
<p>Arrays are encoded by concatenating every value together, with spaces between them. Data types that have multiple
|
||||
values, are serialized similarly.</p>
|
||||
<p>Therefore, an element storing an array of <code>3u8</code> (<code>[(1, 2, 3), (4, 5, 6)]</code>) would look like
|
||||
this</p>
|
||||
<pre><code><demo __type="3u8" __count="2">1 2 3 4 5 6</demo></code></pre>
|
||||
<p>Besides this, this is otherwise a rather standard XML.</p>
|
||||
|
||||
<h2 id="binary">Packed binary overview</h2>
|
||||
|
||||
<p>Many packets, rather than using a string-based XML format, use a custom binary packed format instead. While it
|
||||
can be a little confusing, remembering that this is encoding an XML tree can make it easier to parse.</p>
|
||||
<p>To start with, let's take a look at the overall structure of the packets.</p>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<td>0</td>
|
||||
<td>1</td>
|
||||
<td>2</td>
|
||||
<td>3</td>
|
||||
<td>4</td>
|
||||
<td>5</td>
|
||||
<td>6</td>
|
||||
<td>7</td>
|
||||
<td>8</td>
|
||||
<td>9</td>
|
||||
<td>10</td>
|
||||
<td>11</td>
|
||||
<td>12</td>
|
||||
<td>13</td>
|
||||
<td>14</td>
|
||||
<td>15</td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tr>
|
||||
<td><i>A0</i></td>
|
||||
<td>C</td>
|
||||
<td>E</td>
|
||||
<td>~E</td>
|
||||
<td colspan="4">Head length</td>
|
||||
<td style="border-bottom: none" colspan="8"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-top: none; border-bottom: none;" colspan="16">Schema definition</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-top: none;" colspan="12"></td>
|
||||
<td colspan="1"><i>FF</i></td>
|
||||
<td colspan="3">Align</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="4">Data length</td>
|
||||
<td style="border-bottom: none" colspan="12"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-top: none; border-bottom: none;" colspan="16">Payload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-top: none;" colspan="13"></td>
|
||||
<td colspan="3">Align</td>
|
||||
</tr>
|
||||
</table>
|
||||
<p>Every packet starts with the magic byte <code>0xA0</code>. Following this is the content byte, the encoding byte,
|
||||
and then the 2's compliment of the encoding byte.</p>
|
||||
<p>Currently known possible values for the content byte are:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<td>C</td>
|
||||
<td>Content</td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tr>
|
||||
<td>0x42</td>
|
||||
<td>Compressed data</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x43</td>
|
||||
<td>Compressed, no data</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x45</td>
|
||||
<td>Decompressed data</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x46</td>
|
||||
<td>Decompressed, no data</td>
|
||||
</tr>
|
||||
</table>
|
||||
<p>Decompressed packets contain an XML string. Compressed packets are what we're interested in here.</p>
|
||||
<p>The encoding flag indicates the encoding for all string types in the packet (more on those later). Possible
|
||||
values are:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<td>E</td>
|
||||
<td>~E</td>
|
||||
<td colspan="3">Encoding name</td>
|
||||
</tr>
|
||||
</thead>
|
||||
|
||||
<tr>
|
||||
<td>0x20</td>
|
||||
<td>0xDF</td>
|
||||
<td>ASCII</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x40</td>
|
||||
<td>0xBF</td>
|
||||
<td>ISO-8859-1</td>
|
||||
<td>ISO_8859-1</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x60</td>
|
||||
<td>0x9F</td>
|
||||
<td>EUC-JP</td>
|
||||
<td>EUCJP</td>
|
||||
<td>EUC_JP</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x80</td>
|
||||
<td>0x7F</td>
|
||||
<td>SHIFT-JIS</td>
|
||||
<td>SHIFT_JIS</td>
|
||||
<td>SJIS</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0xA0</td>
|
||||
<td>0x5F</td>
|
||||
<td>UTF-8</td>
|
||||
<td>UTF8</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
</table>
|
||||
<details>
|
||||
<summary>Source code details</summary>
|
||||
<p>The full table for these values can be found in libavs.</p>
|
||||
<figure>
|
||||
<img src="./images/encoding_table.png">
|
||||
<figcaption><code>libavs-win32.dll:0x1006b960</code></figcaption>
|
||||
</figure>
|
||||
<p>A second table exists just before this on in the source, responsible for the
|
||||
<code><?xml version='1.0' encoding='??'?></code> line in XML files.
|
||||
</p>
|
||||
<figure>
|
||||
<img src="./images/xml_encoding_table.png">
|
||||
<figcaption><code>libavs-win32.dll:0x1006b940</code></figcaption>
|
||||
</figure>
|
||||
<p>This is indexed using the following function, which maps the above encoding IDs to 1, 2, 3, 4 and 5
|
||||
respectively.</p>
|
||||
<pre><code>char* xml_get_encoding_name(uint encoding_id) {
|
||||
return ENCODING_NAME_TABLE[((encoding_id & 0xe0) >> 5) * 4];
|
||||
}</code></pre>
|
||||
</details>
|
||||
<p>While validating <code>~E</code> isn't technically required, it acts as a useful assertion that the packet being
|
||||
parsed is valid.</p>
|
||||
|
||||
<h2 id="schema">The packet schema header</h2>
|
||||
<p>Following the 4 byte header, is a 4 byte integer containing the length of the next part of the header (this is
|
||||
technically made redundant as this structure is also terminated).</p>
|
||||
<p>This part of the header defines the schema that the main payload uses.</p>
|
||||
|
||||
<p>A tag definition looks like:</p>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<td>0</td>
|
||||
<td>1</td>
|
||||
<td>2</td>
|
||||
<td>3</td>
|
||||
<td>4</td>
|
||||
<td>5</td>
|
||||
<td>6</td>
|
||||
<td>7</td>
|
||||
<td>8</td>
|
||||
<td>9</td>
|
||||
<td>10</td>
|
||||
<td>11</td>
|
||||
<td>12</td>
|
||||
<td>13</td>
|
||||
<td>14</td>
|
||||
<td>15</td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tr>
|
||||
<td>Type</td>
|
||||
<td>nlen</td>
|
||||
<td colspan="7">Tag name</td>
|
||||
<td style="border-bottom: none" colspan="8"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-top: none;" colspan="15">Attributes and children</td>
|
||||
<td colspan="1"><i>FE</i></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<p>Structure names are encoded as densely packed 6 bit values, length prefixed (<code>nlen</code>). The acceptable
|
||||
alphabet is <code>0123456789:ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz</code>, and the packed values
|
||||
are indecies within this alphabet.</p>
|
||||
|
||||
<p>The children can be a combination of either attribute names, or child tags. Attribute names are represented by
|
||||
the byte <code>0x2E</code> followed by a length prefixed name as defined above. Child tags follow the above
|
||||
format. Type <code>0x2E</code> must therefore be considered reserved as a possible structure type.</p>
|
||||
|
||||
<p>Attributes (type <code>0x2E</code>) represent a string attribute. Any other attribute must be defined as a child
|
||||
tag. Is it notable that 0 children is allowable, which is how the majority of values are encoded.</p>
|
||||
<p>All valid IDs, and their respective type, are listed in the following table. The bucket column here will be
|
||||
used later when unpacking the main data, so we need not worry about it for now, but be warned it exists and is
|
||||
possibly the least fun part of this format.</p>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<td>ID</td>
|
||||
<td>Bytes</td>
|
||||
<td>C type</td>
|
||||
<td>Bucket</td>
|
||||
<td colspan="2">XML names</td>
|
||||
<td></td>
|
||||
<td>ID</td>
|
||||
<td>Bytes</td>
|
||||
<td>C type</td>
|
||||
<td>Bucket</td>
|
||||
<td colspan="2">XML names</td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tr>
|
||||
<td>0x01</td>
|
||||
<td>0</td>
|
||||
<td>void</td>
|
||||
<td>-</td>
|
||||
<td>void</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x21</td>
|
||||
<td>24</td>
|
||||
<td>uint64[3]</td>
|
||||
<td>int</td>
|
||||
<td>3u64</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x02</td>
|
||||
<td>1</td>
|
||||
<td>int8</td>
|
||||
<td>byte</td>
|
||||
<td>s8</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x22</td>
|
||||
<td>12</td>
|
||||
<td>float[3]</td>
|
||||
<td>int</td>
|
||||
<td>3f</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x03</td>
|
||||
<td>1</td>
|
||||
<td>uint8</td>
|
||||
<td>byte</td>
|
||||
<td>u8</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x23</td>
|
||||
<td>24</td>
|
||||
<td>double[3]</td>
|
||||
<td>int</td>
|
||||
<td>3d</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x04</td>
|
||||
<td>2</td>
|
||||
<td>int16</td>
|
||||
<td>short</td>
|
||||
<td>s16</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x24</td>
|
||||
<td>4</td>
|
||||
<td>int8[4]</td>
|
||||
<td>int</td>
|
||||
<td>4s8</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x05</td>
|
||||
<td>2</td>
|
||||
<td>uint16</td>
|
||||
<td>short</td>
|
||||
<td>s16</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x25</td>
|
||||
<td>4</td>
|
||||
<td>uint8[4]</td>
|
||||
<td>int</td>
|
||||
<td>4u8</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x06</td>
|
||||
<td>4</td>
|
||||
<td>int32</td>
|
||||
<td>int</td>
|
||||
<td>s32</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x26</td>
|
||||
<td>8</td>
|
||||
<td>int16[4]</td>
|
||||
<td>int</td>
|
||||
<td>4s16</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x07</td>
|
||||
<td>4</td>
|
||||
<td>uint32</td>
|
||||
<td>int</td>
|
||||
<td>u32</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x27</td>
|
||||
<td>8</td>
|
||||
<td>uint8[4]</td>
|
||||
<td>int</td>
|
||||
<td>4s16</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x08</td>
|
||||
<td>8</td>
|
||||
<td>int64</td>
|
||||
<td>int</td>
|
||||
<td>s64</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x28</td>
|
||||
<td>16</td>
|
||||
<td>int32[4]</td>
|
||||
<td>int</td>
|
||||
<td>4s32</td>
|
||||
<td>vs32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x09</td>
|
||||
<td>8</td>
|
||||
<td>uint64</td>
|
||||
<td>int</td>
|
||||
<td>u64</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x29</td>
|
||||
<td>16</td>
|
||||
<td>uint32[4]</td>
|
||||
<td>int</td>
|
||||
<td>4u32</td>
|
||||
<td>vs32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x0a</td>
|
||||
<td><i>prefix</i></td>
|
||||
<td>char[]</td>
|
||||
<td>int</td>
|
||||
<td>bin</td>
|
||||
<td>binary</td>
|
||||
<td></td>
|
||||
<td>0x2a</td>
|
||||
<td>32</td>
|
||||
<td>int64[4]</td>
|
||||
<td>int</td>
|
||||
<td>4s64</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x0b</td>
|
||||
<td><i>prefix</i></td>
|
||||
<td>char[]</td>
|
||||
<td>int</td>
|
||||
<td>str</td>
|
||||
<td>string</td>
|
||||
<td></td>
|
||||
<td>0x2b</td>
|
||||
<td>32</td>
|
||||
<td>uint64[4]</td>
|
||||
<td>int</td>
|
||||
<td>4u64</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x0c</td>
|
||||
<td>4</td>
|
||||
<td>uint8[4]</td>
|
||||
<td>int</td>
|
||||
<td>ip4</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x2c</td>
|
||||
<td>16</td>
|
||||
<td>float[4]</td>
|
||||
<td>int</td>
|
||||
<td>4f</td>
|
||||
<td>vf</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x0d</td>
|
||||
<td>4</td>
|
||||
<td>uint32</td>
|
||||
<td>int</td>
|
||||
<td>time</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x2d</td>
|
||||
<td>32</td>
|
||||
<td>double[4]</td>
|
||||
<td>int</td>
|
||||
<td>4d</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x0e</td>
|
||||
<td>4</td>
|
||||
<td>float</td>
|
||||
<td>int</td>
|
||||
<td>float</td>
|
||||
<td>f</td>
|
||||
<td></td>
|
||||
<td>0x2e</td>
|
||||
<td><i>prefix</i></td>
|
||||
<td>char[]</td>
|
||||
<td>int</td>
|
||||
<td>attr</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x0f</td>
|
||||
<td>8</td>
|
||||
<td>double</td>
|
||||
<td>int</td>
|
||||
<td>double</td>
|
||||
<td>d</td>
|
||||
<td></td>
|
||||
<td>0x2f</td>
|
||||
<td>0</td>
|
||||
<td></td>
|
||||
<td>-</td>
|
||||
<td>array</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x10</td>
|
||||
<td>2</td>
|
||||
<td>int8[2]</td>
|
||||
<td>short</td>
|
||||
<td>2s8</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x30</td>
|
||||
<td>16</td>
|
||||
<td>int8[16]</td>
|
||||
<td>int</td>
|
||||
<td>vs8</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x11</td>
|
||||
<td>2</td>
|
||||
<td>uint8[2]</td>
|
||||
<td>short</td>
|
||||
<td>2u8</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x31</td>
|
||||
<td>16</td>
|
||||
<td>uint8[16]</td>
|
||||
<td>int</td>
|
||||
<td>vu8</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x12</td>
|
||||
<td>4</td>
|
||||
<td>int16[2]</td>
|
||||
<td>int</td>
|
||||
<td>2s16</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x32</td>
|
||||
<td>16</td>
|
||||
<td>int8[8]</td>
|
||||
<td>int</td>
|
||||
<td>vs16</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x13</td>
|
||||
<td>4</td>
|
||||
<td>uint16[2]</td>
|
||||
<td>int</td>
|
||||
<td>2s16</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x33</td>
|
||||
<td>16</td>
|
||||
<td>uint8[8]</td>
|
||||
<td>int</td>
|
||||
<td>vu16</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x14</td>
|
||||
<td>8</td>
|
||||
<td>int32[2]</td>
|
||||
<td>int</td>
|
||||
<td>2s32</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x34</td>
|
||||
<td>1</td>
|
||||
<td>bool</td>
|
||||
<td>byte</td>
|
||||
<td>bool</td>
|
||||
<td>b</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x15</td>
|
||||
<td>8</td>
|
||||
<td>uint32[2]</td>
|
||||
<td>int</td>
|
||||
<td>2u32</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x35</td>
|
||||
<td>2</td>
|
||||
<td>bool[2]</td>
|
||||
<td>short</td>
|
||||
<td>2b</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x16</td>
|
||||
<td>16</td>
|
||||
<td>int16[2]</td>
|
||||
<td>int</td>
|
||||
<td>2s64</td>
|
||||
<td>vs64</td>
|
||||
<td></td>
|
||||
<td>0x36</td>
|
||||
<td>3</td>
|
||||
<td>bool[3]</td>
|
||||
<td>int</td>
|
||||
<td>3b</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x17</td>
|
||||
<td>16</td>
|
||||
<td>uint16[2]</td>
|
||||
<td>int</td>
|
||||
<td>2u64</td>
|
||||
<td>vu64</td>
|
||||
<td></td>
|
||||
<td>0x37</td>
|
||||
<td>4</td>
|
||||
<td>bool[4]</td>
|
||||
<td>int</td>
|
||||
<td>4b</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x18</td>
|
||||
<td>8</td>
|
||||
<td>float[2]</td>
|
||||
<td>int</td>
|
||||
<td>2f</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x38</td>
|
||||
<td>16</td>
|
||||
<td>bool[16]</td>
|
||||
<td>int</td>
|
||||
<td>vb</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x19</td>
|
||||
<td>16</td>
|
||||
<td>double[2]</td>
|
||||
<td>int</td>
|
||||
<td>2d</td>
|
||||
<td>vd</td>
|
||||
<td></td>
|
||||
<td>0x38</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x1a</td>
|
||||
<td>3</td>
|
||||
<td>int8[3]</td>
|
||||
<td>int</td>
|
||||
<td>3s8</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x39</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x1b</td>
|
||||
<td>3</td>
|
||||
<td>uint8[3]</td>
|
||||
<td>int</td>
|
||||
<td>3u8</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x3a</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x1c</td>
|
||||
<td>6</td>
|
||||
<td>int16[3]</td>
|
||||
<td>int</td>
|
||||
<td>3s16</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x3b</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x1d</td>
|
||||
<td>6</td>
|
||||
<td>uint16[3]</td>
|
||||
<td>int</td>
|
||||
<td>3s16</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x3c</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x1e</td>
|
||||
<td>12</td>
|
||||
<td>int32[3]</td>
|
||||
<td>int</td>
|
||||
<td>3s32</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x3d</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x1f</td>
|
||||
<td>12</td>
|
||||
<td>uint32[3]</td>
|
||||
<td>int</td>
|
||||
<td>3u32</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x3e</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>0x20</td>
|
||||
<td>24</td>
|
||||
<td>int64[3]</td>
|
||||
<td>int</td>
|
||||
<td>3s64</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td>0x3f</td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<p>Strings should be encoded and decoded according to the encoding specified in the packet header. Null termination is optional, however should be stripped during decoding.</p>
|
||||
<p>All of these IDs are <code>& 0x3F</code>. Any value can be turned into an array by setting the 7<sup>th</sup> bit
|
||||
high (<code>| 0x40</code>). Arrays of this form, in the data section, will be an aligned <code>size: u32</code>
|
||||
immediately followed by <code>size</code> bytes' worth of (unaligned!) values of the unmasked type.</p>
|
||||
|
||||
<details>
|
||||
<summary>Source code details</summary>
|
||||
<p>The full table for these values can be found in libavs. This table contains the names of every tag, along
|
||||
with additional information such as how many bytes that data type requires, and which parsing function
|
||||
should be used.</p>
|
||||
<figure>
|
||||
<img src="./images/types_table.png">
|
||||
<figcaption><code>libavs-win32.dll:0x100782a8</code></figcaption>
|
||||
</figure>
|
||||
</details>
|
||||
<details>
|
||||
<summary>Note about the <code>array</code> type:</summary>
|
||||
<p>While I'm not totally sure, I have a suspicion this type is used internally as a pseudo-type. Trying to
|
||||
identify its function as a parsable type has some obvious blockers:</p>
|
||||
|
||||
<p>All of the types have convenient <code>printf</code>-using helper functions that are used to emit them when
|
||||
serializing XML. All except one.</p>
|
||||
<img src="./images/no_array.png">
|
||||
<p>If we have a look inside the function that populates node sizes (<code>libavs-win32.dll:0x1000cf00</code>),
|
||||
it has an explicit case, however is the same fallback as the default case.</p>
|
||||
<img src="./images/no_array_2.png">
|
||||
|
||||
<p>In the same function, however, we can find a second (technically first) check for the array type.</p>
|
||||
<img src="./images/yes_array.png">
|
||||
<p>This seems to suggest that internally arrays are represented as a normal node, with the <code>array</code>
|
||||
type, however when serializing it's converted into the array types we're used to (well, will be after the
|
||||
next sections) by masking 0x40 onto the contained type.</p>
|
||||
<p>Also of interest from this snippet is the fact that <code>void</code>, <code>bin</code>, <code>str</code>,
|
||||
and <code>attr</code> cannot be arrays. <code>void</code> and <code>attr</code> make sense, however
|
||||
<code>str</code> and <code>bin</code> are more interesting. I suspect this is because konami want to be able
|
||||
to preallocate the memory, which wouldn't be possible with these variable length structures.
|
||||
</p>
|
||||
</details>
|
||||
|
||||
<h2 id="data">The data section</h2>
|
||||
|
||||
<p>This is where all the actual packet data is. For the most part, parsing this is the easy part. We traverse our
|
||||
schema, and read values out of the packet according to the value indicated in the schema. Unfortunately, konami
|
||||
decided all data should be aligned very specifically, and that gaps left during alignment should be backfilled
|
||||
later. This makes both reading and writing somewhat more complicated, however the system can be fairly easily
|
||||
understood.</p>
|
||||
<p>Firstly, we divide the payload up into 4 byte chunks. Each chunk can be allocated to either store individual
|
||||
bytes, shorts, or ints (these are the buckets in the table above). When reading or writing a value, we first
|
||||
check if a chunk allocated to the desired type's bucket is available and has free/as-yet-unread space within it.
|
||||
If so, we will store/read our data to/from there. If there is no such chunk, we claim the next unclaimed chunk
|
||||
for our bucket.</p>
|
||||
<p>For example, imagine we write the sequence <code>byte, int, byte, short, byte, int, short</code>. The final output should look like:</p>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<td>0</td>
|
||||
<td>1</td>
|
||||
<td>2</td>
|
||||
<td>3</td>
|
||||
<td>4</td>
|
||||
<td>5</td>
|
||||
<td>6</td>
|
||||
<td>7</td>
|
||||
<td>8</td>
|
||||
<td>9</td>
|
||||
<td>10</td>
|
||||
<td>11</td>
|
||||
<td>12</td>
|
||||
<td>13</td>
|
||||
<td>14</td>
|
||||
<td>15</td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tr>
|
||||
<td>byte</td>
|
||||
<td>byte</td>
|
||||
<td>byte</td>
|
||||
<td></td>
|
||||
<td colspan="4">int</td>
|
||||
<td colspan="2">short</td>
|
||||
<td colspan="2">short</td>
|
||||
<td colspan="4">int</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<p>While this might seem a silly system compared to just not aligning values, it is at least possible to intuit that it helps reduce wasted space. It should be noted that any variable-length structure, such as a string or an array, claims all chunks it encroaches on for the <code>int</code> bucket, disallowing the storage of bytes or shorts within them.</p>
|
||||
|
||||
<details>
|
||||
<summary>Implementing a packer</summary>
|
||||
<p>While the intuitive way to understand the packing algorithm is via chunks and buckets, a far more efficient implementation can be made that uses three pointers. Rather than try to explain in words, hopefully this python implementation should suffice as explanation:<pre><code>class Packer:
|
||||
def __init__(self, offset=0):
|
||||
self._word_cursor = offset
|
||||
self._short_cursor = offset
|
||||
self._byte_cursor = offset
|
||||
self._boundary = offset % 4
|
||||
|
||||
def _next_block(self):
|
||||
self._word_cursor += 4
|
||||
return self._word_cursor - 4
|
||||
|
||||
def request_allocation(self, size):
|
||||
if size == 0:
|
||||
return self._word_cursor
|
||||
elif size == 1:
|
||||
if self._byte_cursor % 4 == self._boundary:
|
||||
self._byte_cursor = self._next_block() + 1
|
||||
else:
|
||||
self._byte_cursor += 1
|
||||
return self._byte_cursor - 1
|
||||
elif size == 2:
|
||||
if self._short_cursor % 4 == self._boundary:
|
||||
self._short_cursor = self._next_block() + 2
|
||||
else:
|
||||
self._short_cursor += 2
|
||||
return self._short_cursor - 2
|
||||
else:
|
||||
old_cursor = self._word_cursor
|
||||
for _ in range(math.ceil(size / 4)):
|
||||
self._word_cursor += 4
|
||||
return old_cursor
|
||||
|
||||
def notify_skipped(self, no_bytes):
|
||||
for _ in range(math.ceil(no_bytes / 4)):
|
||||
self.request_allocation(4)</code></pre></p>
|
||||
</details>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
57
styles.css
Normal file
57
styles.css
Normal file
@ -0,0 +1,57 @@
|
||||
body {
|
||||
/* font-family: sans-serif; */
|
||||
}
|
||||
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
font-family: monospace;
|
||||
letter-spacing: .02em;
|
||||
}
|
||||
|
||||
thead {
|
||||
font-weight: bold;
|
||||
border-bottom: 2px solid #000;
|
||||
}
|
||||
|
||||
td {
|
||||
border: 1px solid #111;
|
||||
padding: 2px;
|
||||
text-align: center;
|
||||
min-width: 32px;
|
||||
}
|
||||
|
||||
td a {
|
||||
display: block;
|
||||
padding: 4px 8px;
|
||||
}
|
||||
|
||||
code {
|
||||
display: inline-block;
|
||||
letter-spacing: .02em;
|
||||
padding: 2px 4px;
|
||||
font-size: 90%;
|
||||
color: #c7254e;
|
||||
background-color: #f9f2f4;
|
||||
border-radius: 4px;
|
||||
}
|
||||
pre > code {
|
||||
border-radius: 4px;
|
||||
background: #f8f8f8;
|
||||
border: 1px solid #ccc;
|
||||
padding: 4px;
|
||||
color: #333;
|
||||
padding: 9.5px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
|
||||
summary {
|
||||
user-select: none;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
details {
|
||||
background: lightblue;
|
||||
border: 1px solid cornflowerblue;
|
||||
padding: 4px;
|
||||
margin: 4px 0;
|
||||
}
|
||||
43
transport.html
Normal file
43
transport.html
Normal file
@ -0,0 +1,43 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Transport | eAmuse API</title>
|
||||
|
||||
<link rel="stylesheet" href="styles.css">
|
||||
</head>
|
||||
<body>
|
||||
<table>
|
||||
<tr>
|
||||
<td><a href=".">Contents</a></td>
|
||||
<td><a href="./transport.html">Transport layer</a></td>
|
||||
<td><a href="./packet.html">Packet format</a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<h1>Network format</h1>
|
||||
|
||||
<p>eAmuse packets are sent and received over HTTP (no S), with requests being in the body of <code>POST</code> requests, and replies being in the, well, reply.</p>
|
||||
<p>The packets are typically both encrypted and compressed. The compression format used is indicated by the <code>X-Compress</code> header, and valid values are</p>
|
||||
<ul>
|
||||
<li><code>none</code></li>
|
||||
<li><code>lz77</code></li>
|
||||
</ul>
|
||||
<p>Encryption is performed <b>after</b> compression, and uses RC4. RC4 is symmetric, so decryption is performed the same as encryption. That is, <code>packet = encrypt(compress(data))</code> and <code>data = decompress(decrypt(data))</code>.</p>
|
||||
|
||||
<h2 id="keys">Encryption keys</h2>
|
||||
<p>Encryption is not performed using a single static key. Instead, each request and response has its own key that is generated.</p>
|
||||
<p>These keys are generated baesd on the <code>X-Eamuse-Info</code> header.</p>
|
||||
<p>This header loosely follows the format <code>1-[0-9a-f]{8}-[0-9a-f]{4}</code>. This corresponds to <code>[version]-[serial]-[salt]</code>. <b>TODO: Confirm this</b></p>
|
||||
<p>Our per-packet key is then generated using <code>md5(serial | salt | KEY)</code>. Identifying <code>KEY</code> is left as an exercise for the reader, however should not be especially challenging.</p>
|
||||
|
||||
<h2 id="lz77">LZ77</h2>
|
||||
<p>Packets are compressed using lzss. The compressed data structure is a repeating cycle of an 8 bit flags byte, followed by 8 values. Each value is either a single literal byte, if the corresponding bit in the preceeding flag is high, or is a two byte lookup into the window.</p>
|
||||
<p>The lookup bytes are structured as <code>pppppppp ppppllll</code> where <code>p</code> is a 12 bit index in the window, and <code>l</code> is a 4 bit integer that determines how many times to repeat the value located at that index in the window.</p>
|
||||
|
||||
<p>The exact algorithm used for compression is not especially important, as long as it follows this format. One can feasibly perform no compression at all, and instead insert <code>0xFF</code> every 8 bytes (starting at index 0), to indicate that all values are literals. While obviously poor for compression, this is an easy way to test without first implementing a compressor.</p>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
Loading…
Reference in New Issue
Block a user