bemaniutils/bemani/format/afp/README.md

# AFP Animation Format

As far back as I can see, Konami has used an animation format that I refer to as
AFP in virtually all of their games. Basically, if it runs on a PC, it probably
has a variant of AFP running on it. This also goes for several of their commercial
PC titles, some console titles and mobile titles as well. The name "AFP" comes
from the library that implements most of the format but I have no idea what this
actually stands for. In very old PC-based arcade games such as The\*BishiBashi the
library code makes references to Flash. If it encounters a plain SWF file it returns
an error stating that old data is not supported and encouraging the artist to
convert the data to the "new" format. Many other similarities between SWF and AFP
are present including a legacy bytecode table that is identical to Flash, a tag ID
to string function that matches Flash exactly up to the point where Konami started
adding their own, and the fact that the core engine works identically down to the
affine transformations, additive and multiplicative colors, tag system, blending
modes, masking and a whole host of other things.

At some point, most of the old SWF code was removed from the project. A modern
AFP library is almost unrecognizeable from a classic one, save for some basic
code that parses the still-supported tags. The legacy SWF bytecode parser has
been removed completely, support for recognizing vanilla SWF tags has been removed,
and support for bytecode that is interactive in nature has been removed leaving
only the shell of a bytecode engine that can update properties for effects such
as applying masks, setting objects visible or invisible and rewinding/advancing
individual sprites on the main canvas. While Konami appears to have put a decent
effort into making all of their format changes purely additive, if one was to
try to render an animation from an older game inside a newer game's engine it
would not be a surprise if the resulting graphics were subtly wrong.

The AFP file format itself is not an all-encompassing container. It superficially
resembles a SWF file but has an entirely different header and different ways of
encoding the tags and optional features of the format. Curiously, one cannot decode
a chunk of AFP data without the corresponding BSI data which is simply a list of
locations to byteswap as well as the length to byteswap. This appears to be a
rudimentary way of hiding plain strings in the AFP data itself so that they can't
be found in a hex editor. The byteswap instructions are trivial as well as reversible.
Along with AFP data, one would need shape structures as well as textures. These are
not found inside the AFP data but are instead inside a parent container. For older
games this is a TXP2 file which contains special sections for texture sheets,
individual textures, shapes, fonts, AFP data and BSI data. For newer games this is
a standard Konami IFS file where the AFP data is found in the `afp/` folder, the
BSI data is found in the `afp/bsi/` folder, the shapes are found in the `geo/`
folder and the textures are found in the `tex/` folder. Aside from container format
differences the underlying files are identically parsed and have equivalent
meanings regardless of where one unpacks them from.

The basic concept of how an animation is stored and represented is fairly simple.
Each animation has a specified number of frames and there is a lookup table in
the AFP header specifying which tags should be acted upon for each frame. To play
an animation, one must act upon each tag for a frame and then display or save the
resulting image. Tags have actions such as defining a shape or sprite for later
use, requests to place a previously defined shape or sprite on the canvas,
requests to update the properties of a previously placed object on the canvas,
requests to delete a previously placed object from the canvas and arbitrary
bytecode to execute as part of the frame's processing. Placed objects on the
canvas each have an associated additive and multiplicative color to apply to them
before blending, a blending mode such as normal, additive or subtractive, an
affine or perspective transform matrix used to determine how to position the
object and finally either a bounding box for a solid color rectangle, a reference
to a texture or a list of child objects to treat as a single sprite. Objects are
placed onto their own depth planes and rendered from lowest plane to highest plane
one by one until the final frame has been constructed.

# Tags

Since tags are the heart of the format (as they are in SWF files), I go into a bit
more detail about the crucial ones here. In the older versions of the format
vanilla SWF tags could also be used but they are documented elsewhere online and
in practice have never been seen inside an AFP animation.

## AP2 Shape Tag

The shape tag defines a shape (either a reference to a texture quad or a solid
color rectangle) and assigns it an internal ID for later use. Both texture quads
and solid rectangles are represented by the same shape structure which contains
the shape's bounding rectangle, UV vertex points and draw parameters (such as whether
this shape is a rectangle or a quad in the first place). If the shape is a texture
quad the structure will also include a string reference to the name of the texture
itself. The UV coordinates and bounding rectangle are always identical to the size
of the referenced texture in the case of texture quad shapes and appear to be in
the format simply to pass forward to a graphics card. Acting upon a shape tag means
adding the shape to an internal library of objects that can be placed on the canvas.

## AP2 Image Tag

This appears to be functionally identical to a shape, save for missing all of the
graphics card coordinates. Instead of pointing at a shape structure, image tags point
directly at a texture itself. Acting upon an image tag means adding the image to an
internal library of objects that can be placed on the canvas in a similar manner to
a shape.

## AP2 Sprite Tag

The sprite tag defines a sprite, which in SWF and AFP nomenclature just means an
embedded animation complete with its own list of frames and tags. In reality, the root
animation itself can be seen as a sprite tag, and in fact the format allows importing
another animation as a sprite to be placed on the canvas just like an image or shape.
Sprites are handy because they allow an animator to define things like characters with
their own shapes, transform and move the objects that make up those characters, and then
later place the character on the canvas just like a shape or image. As you might expect,
since a sprite is just a set of tags to act upon each frame, sprites can include their
own child sprites indefinitely. Acting upon a sprite tag means adding the sprite to
an internal library of objects that can be placed on the canvas.

## AP2 Place Object Tag

The place object tag is the meat of the format. Place object tags specify a placed
object ID to refer later to the placed object as well as a depth to place the object
on. They refer to sprites, images and shapes by their registered ID. They can come in
one of two flavors: create or update. In create mode, a new object is placed onto the
canvas at the specified depth. That object is looked up in the internal library of
objects that was previously added to in a shape, image or sprite tag. In update mode,
a previously placed object is looked up by object ID and depth and its properties are
updated to match the update tag's specifications. In both cases, the place object tag
can include a blend mode, an affine or projection transform, an additive and
multiplicative color and a few other properties. The blend mode and colors specify to
the renderer how to combine the pixels of the placed object with other objects that are
placed on the canvas. The transform specifies how the object is positioned, rotated and
stretched. Using a series of updates, an animator could rotate an object, move its
position on the canvas, change its transparency to fade it in or out and basically any
other operation.

Acting upon a place object tag means placing a new object on the canvas to be
rendered this frame or updating an existing object with new properties which will be
rendered this frame. Objects that are already on the canvas and not updated by a
place object tag will keep their properties and be rendered identically on subsequent
frames. The only exception to this is placed sprites, which automatically advance to
the next frame of their own embedded animation in order to render correctly.

## AP2 Remove Object Tag

The remove object tag looks up an object previously placed by a place object tag and
removes it from the canvas. It can either remove an object by object ID and depth or
it can remove the last placed object on a particular depth. Acting upon a remove object
tag means removing an existing object from the canvas so that it is not rendered on
this frame or subsequent frames.

## AP2 Do Action Tag

The do action tag executes AFP bytecode in order to perform some action. This can mean
requesting a mask to be applied to a placed object, requesting a placed sprite be
advanced or rewound to a specific frame, requesting a placed object be made invisible
or a host of other useful effects. AFP animations use this to provide looping effects
on sprites since the format does not natively support loops. They also use this to
apply masks to placed objects as the format does not natively support defining a mask.
Originally do action tags could also provide interactivity such as in The\*BishiBashi
where the levels themselves were implemented entirely in AFP files and played using
bytecode to read inputs, check game scores and the like. Modern games have stripped
this functionality out and only use it to provide a few useful hooks such as the above
documented effects and things such as triggering sounds to play or informing other
systems what frame of the animation is being rendered.

Acting upon a do action tag means using a bytecode interpreter to execute the bytecode
and apply the desired property changes to the currently placed objects on the canvas.
Note also that the place object tag supports bytecode triggers including an "on load"
trigger that requires bytecode to be executed when the object is placed. So, bytecode
can be found in two different locations but is executed identically either way.

## AP2 Place Camera Tag

The place camera tag registers a camera by specifying where in 3D the camera is and
the focal length from the camera to the render plane. The camera always looks straight
down at the canvas and the focal length in practice alwys matches the Z offset of the
camera. This is only used in conjunction with objects that have a perspective transform
to correctly render the object based on its depth in the Z plane. Acting upon a place
camera tag means storing where the camera is for future projection perspective renders
when the engine goes to display all of the placed objects for a frame.

# Quirks

No format that is this complex will be without its quirks. At some point, Konami
hacked in 3D perspective support to the format. This is extremely rudimentary
and only allows for a camera looking straight down at the canvas and for 2D
quads to be transformed using a 3D perspective transform. Its my best guess that
artists were tired of having to render out individual frames and specify them
in order to achieve correct 3D rotation of arbitrary objects so this system was
put in place. There is no culling, no normals, no depth buffer, FOV, nothing. It
exists simply as a way to specify what depth to project a 2D quad that has been
rotated in 3D space. This is used in newer games mostly for perspective-correct
rotation of objects such as records.

Now, unlike the 2D portion which was based on Flash and has had some 15 or so
years of testing to mature, the 3D portion is much more hacked in and has a host
of quriks. For instance, it appears that they did not do perspective-based masking
correctly and as such some animations which should have masking animations applied
to them appear to do so incorrectly. They also appear to have a bug where specifying
3D perspective projection on an object in a sprite and then placing that sprite
down on the main canvas results in a 2D affine projection instead of a 3D one.
These are all subtle enough that its quite possible QA never caught the problem or
the artist complained and was told it was not a big enough deal to fix.

I briefly considered having a compatibility quirks flag system in order to attempt
to fix these issues and present a renderer that exactly matches what games do.
The problem with that is that this renderer already attempts to be compatible
with about a decade and a half of format changes. Given how much of the format
has been ripped out for newer games, this is already an impossible task. I would
also have to spend a great deal more time trying to figure out exactly HOW they
got these things subtly wrong to replicate them versus understanding how the format
was intended to work. I've made the decision that it is not worth my time to do so.

# References

Much of the understanding of this format came from the various online SWF file format
references as so much of the format mirrors Flash even today. Detailed documentation
on the format itself is entirely missing but the code in this directory provides a
working reference implementation of the format. Wherever this document is vague or
confusing, please refer to the code in `swf.py` for how to parse AFP, `render.py` for
how to render AFP, `container.py` for how to parse TXP2 files and `geo.py` for how to
parse shape structures. Please also refer to `decompile.py` for a ton of details on
how the bytecode itself works.