Add a readme for AFP.

bemani/format/afp/README.md

@@ -0,0 +1,210 @@
+# AFP Animation Format
+
+As far back as I can see, Konami has used an animation format that I refer to as
+AFP in virtually all of their games. Basically, if it runs on a PC, it probably
+has a variant of AFP running on it. This also goes for several of their commercial
+PC titles, some console titles and mobile titles as well. The name "AFP" comes
+from the library that implements most of the format but I have no idea what this
+actually stands for. In very old PC-based arcade games such as The\*BishiBashi the
+library code makes references to Flash. If it encounters a plain SWF file it returns
+an error stating that old data is not supported and encouraging the artist to
+convert the data to the "new" format. Many other similarities between SWF and AFP
+are present including a legacy bytecode table that is identical to Flash, a tag ID
+to string function that matches Flash exactly up to the point where Konami started
+adding their own, and the fact that the core engine works identically down to the
+affine transformations, additive and multiplicative colors, tag system, blending
+modes, masking and a whole host of other things.
+
+At some point, most of the old SWF code was removed from the project. A modern
+AFP library is almost unrecognizeable from a classic one, save for some basic
+code that parses the still-supported tags. The legacy SWF bytecode parser has
+been removed completely, support for recognizing vanilla SWF tags has been removed,
+and support for bytecode that is interactive in nature has been removed leaving
+only the shell of a bytecode engine that can update properties for effects such
+as applying masks, setting objects visible or invisible and rewinding/advancing
+individual sprites on the main canvas. While Konami appears to have put a decent
+effort into making all of their format changes purely additive, if one was to
+try to render an animation from an older game inside a newer game's engine it
+would not be a surprise if the resulting graphics were subtly wrong.
+
+The AFP file format itself is not an all-encompassing container. It superficially
+resembles a SWF file but has an entirely different header and different ways of
+encoding the tags and optional features of the format. Curiously, one cannot decode
+a chunk of AFP data without the corresponding BSI data which is simply a list of
+locations to byteswap as well as the length to byteswap. This appears to be a
+rudimentary way of hiding plain strings in the AFP data itself so that they can't
+be found in a hex editor. The byteswap instructions are trivial as well as reversible.
+Along with AFP data, one would need shape structures as well as textures. These are
+not found inside the AFP data but are instead inside a parent container. For older
+games this is a TXP2 file which contains special sections for texture sheets,
+individual textures, shapes, fonts, AFP data and BSI data. For newer games this is
+a standard Konami IFS file where the AFP data is found in the `afp/` folder, the
+BSI data is found in the `afp/bsi/` folder, the shapes are found in the `geo/`
+folder and the textures are found in the `tex/` folder. Aside from container format
+differences the underlying files are identically parsed and have equivalent
+meanings regardless of where one unpacks them from.
+
+The basic concept of how an animation is stored and represented is fairly simple.
+Each animation has a specified number of frames and there is a lookup table in
+the AFP header specifying which tags should be acted upon for each frame. To play
+an animation, one must act upon each tag for a frame and then display or save the
+resulting image. Tags have actions such as defining a shape or sprite for later
+use, requests to place a previously defined shape or sprite on the canvas,
+requests to update the properties of a previously placed object on the canvas,
+requests to delete a previously placed object from the canvas and arbitrary
+bytecode to execute as part of the frame's processing. Placed objects on the
+canvas each have an associated additive and multiplicative color to apply to them
+before blending, a blending mode such as normal, additive or subtractive, an
+affine or perspective transform matrix used to determine how to position the
+object and finally either a bounding box for a solid color rectangle, a reference
+to a texture or a list of child objects to treat as a single sprite. Objects are
+placed onto their own depth planes and rendered from lowest plane to highest plane
+one by one until the final frame has been constructed.
+
+# Tags
+
+Since tags are the heart of the format (as they are in SWF files), I go into a bit
+more detail about the crucial ones here. In the older versions of the format
+vanilla SWF tags could also be used but they are documented elsewhere online and
+in practice have never been seen inside an AFP animation.
+
+## AP2 Shape Tag
+
+The shape tag defines a shape (either a reference to a texture quad or a solid
+color rectangle) and assigns it an internal ID for later use. Both texture quads
+and solid rectangles are represented by the same shape structure which contains
+the shape's bounding rectangle, UV vertex points and draw parameters (such as whether
+this shape is a rectangle or a quad in the first place). If the shape is a texture
+quad the structure will also include a string reference to the name of the texture
+itself. The UV coordinates and bounding rectangle are always identical to the size
+of the referenced texture in the case of texture quad shapes and appear to be in
+the format simply to pass forward to a graphics card. Acting upon a shape tag means
+adding the shape to an internal library of objects that can be placed on the canvas.
+
+## AP2 Image Tag
+
+This appears to be functionally identical to a shape, save for missing all of the
+graphics card coordinates. Instead of pointing at a shape structure, image tags point
+directly at a texture itself. Acting upon an image tag means adding the image to an
+internal library of objects that can be placed on the canvas in a similar manner to
+a shape.
+
+## AP2 Sprite Tag
+
+The sprite tag defines a sprite, which in SWF and AFP nomenclature just means an
+embedded animation complete with its own list of frames and tags. In reality, the root
+animation itself can be seen as a sprite tag, and in fact the format allows importing
+another animation as a sprite to be placed on the canvas just like an image or shape.
+Sprites are handy because they allow an animator to define things like characters with
+their own shapes, transform and move the objects that make up those characters, and then
+later place the character on the canvas just like a shape or image. As you might expect,
+since a sprite is just a set of tags to act upon each frame, sprites can include their
+own child sprites indefinitely. Acting upon a sprite tag means adding the sprite to
+an internal library of objects that can be placed on the canvas.
+
+## AP2 Place Object Tag
+
+The place object tag is the meat of the format. Place object tags specify a placed
+object ID to refer later to the placed object as well as a depth to place the object
+on. They refer to sprites, images and shapes by their registered ID. They can come in
+one of two flavors: create or update. In create mode, a new object is placed onto the
+canvas at the specified depth. That object is looked up in the internal library of
+objects that was previously added to in a shape, image or sprite tag. In update mode,
+a previously placed object is looked up by object ID and depth and its properties are
+updated to match the update tag's specifications. In both cases, the place object tag
+can include a blend mode, an affine or projection transform, an additive and
+multiplicative color and a few other properties. The blend mode and colors specify to
+the renderer how to combine the pixels of the placed object with other objects that are
+placed on the canvas. The transform specifies how the object is positioned, rotated and
+stretched. Using a series of updates, an animator could rotate an object, move its
+position on the canvas, change its transparency to fade it in or out and basically any
+other operation.
+
+Acting upon a place object tag means placing a new object on the canvas to be
+rendered this frame or updating an existing object with new properties which will be
+rendered this frame. Objects that are already on the canvas and not updated by a
+place object tag will keep their properties and be rendered identically on subsequent
+frames. The only exception to this is placed sprites, which automatically advance to
+the next frame of their own embedded animation in order to render correctly.
+
+## AP2 Remove Object Tag
+
+The remove object tag looks up an object previously placed by a place object tag and
+removes it from the canvas. It can either remove an object by object ID and depth or
+it can remove the last placed object on a particular depth. Acting upon a remove object
+tag means removing an existing object from the canvas so that it is not rendered on
+this frame or subsequent frames.
+
+## AP2 Do Action Tag
+
+The do action tag executes AFP bytecode in order to perform some action. This can mean
+requesting a mask to be applied to a placed object, requesting a placed sprite be
+advanced or rewound to a specific frame, requesting a placed object be made invisible
+or a host of other useful effects. AFP animations use this to provide looping effects
+on sprites since the format does not natively support loops. They also use this to
+apply masks to placed objects as the format does not natively support defining a mask.
+Originally do action tags could also provide interactivity such as in The\*BishiBashi
+where the levels themselves were implemented entirely in AFP files and played using
+bytecode to read inputs, check game scores and the like. Modern games have stripped
+this functionality out and only use it to provide a few useful hooks such as the above
+documented effects and things such as triggering sounds to play or informing other
+systems what frame of the animation is being rendered.
+
+Acting upon a do action tag means using a bytecode interpreter to execute the bytecode
+and apply the desired property changes to the currently placed objects on the canvas.
+Note also that the place object tag supports bytecode triggers including an "on load"
+trigger that requires bytecode to be executed when the object is placed. So, bytecode
+can be found in two different locations but is executed identically either way.
+
+## AP2 Place Camera Tag
+
+The place camera tag registers a camera by specifying where in 3D the camera is and
+the focal length from the camera to the render plane. The camera always looks straight
+down at the canvas and the focal length in practice alwys matches the Z offset of the
+camera. This is only used in conjunction with objects that have a perspective transform
+to correctly render the object based on its depth in the Z plane. Acting upon a place
+camera tag means storing where the camera is for future projection perspective renders
+when the engine goes to display all of the placed objects for a frame.
+
+# Quirks
+
+No format that is this complex will be without its quirks. At some point, Konami
+hacked in 3D perspective support to the format. This is extremely rudimentary
+and only allows for a camera looking straight down at the canvas and for 2D
+quads to be transformed using a 3D perspective transform. Its my best guess that
+artists were tired of having to render out individual frames and specify them
+in order to achieve correct 3D rotation of arbitrary objects so this system was
+put in place. There is no culling, no normals, no depth buffer, FOV, nothing. It
+exists simply as a way to specify what depth to project a 2D quad that has been
+rotated in 3D space. This is used in newer games mostly for perspective-correct
+rotation of objects such as records.
+
+Now, unlike the 2D portion which was based on Flash and has had some 15 or so
+years of testing to mature, the 3D portion is much more hacked in and has a host
+of quriks. For instance, it appears that they did not do perspective-based masking
+correctly and as such some animations which should have masking animations applied
+to them appear to do so incorrectly. They also appear to have a bug where specifying
+3D perspective projection on an object in a sprite and then placing that sprite
+down on the main canvas results in a 2D affine projection instead of a 3D one.
+These are all subtle enough that its quite possible QA never caught the problem or
+the artist complained and was told it was not a big enough deal to fix.
+
+I briefly considered having a compatibility quirks flag system in order to attempt
+to fix these issues and present a renderer that exactly matches what games do.
+The problem with that is that this renderer already attempts to be compatible
+with about a decade and a half of format changes. Given how much of the format
+has been ripped out for newer games, this is already an impossible task. I would
+also have to spend a great deal more time trying to figure out exactly HOW they
+got these things subtly wrong to replicate them versus understanding how the format
+was intended to work. I've made the decision that it is not worth my time to do so.
+
+# References
+
+Much of the understanding of this format came from the various online SWF file format
+references as so much of the format mirrors Flash even today. Detailed documentation
+on the format itself is entirely missing but the code in this directory provides a
+working reference implementation of the format. Wherever this document is vague or
+confusing, please refer to the code in `swf.py` for how to parse AFP, `render.py` for
+how to render AFP, `container.py` for how to parse TXP2 files and `geo.py` for how to
+parse shape structures. Please also refer to `decompile.py` for a ton of details on
+how the bytecode itself works.