Cool_Tools/bemaniutils
1
0
Fork 0
mirror of synced 2025-01-31 12:13:49 +01:00
Code Releases Activity

Do a pass over the README as its been awhile since I last checked it.

Browse Source
This commit is contained in:
Jennifer Taylor 2021-05-11 18:17:59 +00:00
parent 26c2a99a6e
commit bbcb3b86b5
1 changed files with 77 additions and 51 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

128
README.md
View File

@ -31,11 +31,13 @@ they all lack source as far as I could tell, so I developed this. Run it like
Utilities for working with several animation formats found across a vast range Utilities for working with several animation formats found across a vast range
of games. This includes a TXP2 container parser, a GE2D shape parser and an of games. This includes a TXP2 container parser, a GE2D shape parser and an
AFP/BSI parser. Together, they make a set of utilities that attempts to render AFP/BSI parser. Together, they make a set of utilities that attempts to work
animations out of IFS and TXP2 files as found in various games. Note that this with AFP, the fork of SWF that handles animations in various games. This utility
format is similar to SWF and thus very complicated. Therefore, it is unlikely is capable of rendering animations out of IFS and TXP2 files as well as providing
that these tools will correctly render all animations from all games that it decompiled pseudocode for the flash-like bytecode found in many animation files.
encounters. Run it like `./afputils --help` to see help output and determine Note that this format is similar to SWF and thus very complicated. Therefore, it
is unlikely that these tools will correctly handle all animations from all games
that it encounters. Run it like `./afputils --help` to see help output and determine
how to use it. how to use it.
## api ## api
@ -48,8 +50,8 @@ this should connect to as well as what game series are supported. See
`config/server.yaml` for an example file that you can modify. `config/server.yaml` for an example file that you can modify.
Do not use this utility to serve production traffic. Instead, see Do not use this utility to serve production traffic. Instead, see
`bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with uWSGI `bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python
and nginx. virtualenv containing this project and its dependencies, uWSGI and nginx.
## arcutils ## arcutils
@ -79,7 +81,7 @@ a standalone sniffing utility, but I don't have the time.
## binutils ## binutils
A utility for unpacking raw binxml files (files that use the same encoding scheme A utility for unpacking raw binxml data (files that use the same encoding scheme
as the binary network protocol) to their XML representation. This is useful for as the binary network protocol) to their XML representation. This is useful for
examining raw binary blobs or digging into unknown file formats that contain binxml. examining raw binary blobs or digging into unknown file formats that contain binxml.
Run it like `./binutils --help` to see help and learn how to use this. Run it like `./binutils --help` to see help and learn how to use this.
@ -87,7 +89,7 @@ Run it like `./binutils --help` to see help and learn how to use this.
## bootstrap ## bootstrap
A utility for quickly bootstrapping a local setup's music database from an already A utility for quickly bootstrapping a local setup's music database from an already
running BEMAPI-compatible server. This is more documented in the below "Database running BEMAPI-compatible server. This is better documented in the below "Database
Initialization" section. Initialization" section.
## cardconvert ## cardconvert
@ -96,7 +98,9 @@ A command-line utility for converting between card numbers written on the back o
card and the card ID stored in the RFID of the card. Run it like `./cardconvert --help` card and the card ID stored in the RFID of the card. Run it like `./cardconvert --help`
to see how to use this. This will sanitize input, so you can feed it card numbers to see how to use this. This will sanitize input, so you can feed it card numbers
with or without spaces, and you can mix up 1 and I as well as 0 and O, and it will with or without spaces, and you can mix up 1 and I as well as 0 and O, and it will
properly handle decoding. This supports both new and old style cards. properly handle decoding. This supports both new and old style e-Amusement cards but
does not support the cross-play network cards with five groups of digits on the back
of the card.
## dbutils ## dbutils
@ -122,15 +126,15 @@ information about the MySQL database that this should connect to as well as what
series are supported. See `config/server.yaml` for an example file that you can modify. series are supported. See `config/server.yaml` for an example file that you can modify.
Do not use this utility to serve production traffic. Instead, see Do not use this utility to serve production traffic. Instead, see
`bemani/wsgi/frontend.wsgi` for a ready-to-go WSGI file that can be used with uWSGI `bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python
and nginx. virtualenv containing this project and its dependencies, uWSGI and nginx.
## ifsutils ## ifsutils
A mediocre utility that can extract `.ifs` files. This has a lot of baked in A mediocre utility that can extract `.ifs` files. This has a lot of baked in
assumptions and is not nearly as good as other open-source utilities for extracting assumptions and is not nearly as good as other open-source utilities for extracting
files. It also cannot repack files. This is included for posterity, and because some files. It also cannot repack files yet. This is included for posterity, and because
bootstrapping code requires it in order to fully start a production server. some bootstrapping code requires it in order to fully start a production server.
Run it like `./ifsutils --help` to see help output and learn how to use it. Run it like `./ifsutils --help` to see help output and learn how to use it.
## iidxutils ## iidxutils
@ -149,7 +153,8 @@ Its possible that this could be used to on-the-fly patch packets coming back fro
network which you don't control to do things such as enable paseli and adjust other network which you don't control to do things such as enable paseli and adjust other
settings that you cannot normally access. Logs in an identical format to bemanishark. settings that you cannot normally access. Logs in an identical format to bemanishark.
Useful for black-box RE of other networks. Note that this does not have the ability Useful for black-box RE of other networks. Note that this does not have the ability
to MITM SSL-encrypted traffic, so don't bother trying to use this on an official network. to MITM SSL-encrypted traffic, so don't bother trying to use this on an official
network.
This also has the ability to route a packet to one of several known networks based on This also has the ability to route a packet to one of several known networks based on
the PCBID, so this can also be used as a proxy for switching networks on the fly. the PCBID, so this can also be used as a proxy for switching networks on the fly.
@ -163,9 +168,9 @@ Run it like `./proxy --help` to see how to use this utility.
## psmap ## psmap
A utility to take an offset from a DLL file and produce python code that would generate A utility to take an offset from a DLL/EXE file and produce python code that would generate
a suitable response that said DLL will properly parse. Essentially, if you are a suitable response that said DLL/EXE will properly parse. Essentially, if you are
reversing a new game and they use the `psmap` utility to decode all or part of a reversing a new game and they use the `psmap` function to decode all or part of a
packet, you can grab either the physical offset into the DLL or the virtual address of packet, you can grab either the physical offset into the DLL or the virtual address of
the data and use this utility to generate the code necessary to service that request. the data and use this utility to generate the code necessary to service that request.
Note that this doesn't currently work on 64bit games, but it should be trivial to Note that this doesn't currently work on 64bit games, but it should be trivial to
@ -185,9 +190,11 @@ below in the "Installation" section.
A utility to take a packet as logged by proxy, services, trafficgen or bemanishark, A utility to take a packet as logged by proxy, services, trafficgen or bemanishark,
and replay that packet against a particular server. Useful for quickly grabbing and replay that packet against a particular server. Useful for quickly grabbing
packets that caused a crash and debugging the crash (and verifying the fix). It also packets that caused a crash and debugging the crash (and verifying the fix). It is
lets you replay that packet against your production instance once you fixed the issue also compatible with the packet logs found on exception and unsupported packet
in case that packet was a score or profile update that you care about. messages in the Admin Event Logs page on the frontend. It also lets you replay that
packet against your production instance once you fix the issue in case that packet
was a score or profile update that you care about.
## responsegen ## responsegen
@ -200,8 +207,8 @@ see all information and usage.
## scheduler ## scheduler
A command-line utility for kicking off scheduled work that must be performed against a A command-line utility for kicking off scheduled work that must be performed against the
DB. This includes picking new dailies/weeklies, new courses, etc... depending on the DB. This includes picking new dailies/weeklies, new courses, and others depending on the
game and any requirements that the server perform some actual calculation based on game and any requirements that the server perform some actual calculation based on
time. Essentially, any game backend that includes a `run_scheduled_work` override will time. Essentially, any game backend that includes a `run_scheduled_work` override will
be acted on by this utility. Note that this takes care of scheduling cadence and be acted on by this utility. Note that this takes care of scheduling cadence and
@ -216,12 +223,12 @@ libraries also used in "bemanishark" and "trafficgen". Currently it lets most mo
BEMANI games boot and supports full profile and events for Beatmania IIDX 20-24, BEMANI games boot and supports full profile and events for Beatmania IIDX 20-24,
Pop'n Music 19-24, Jubeat Saucer, Saucer Fulfill, Prop, Qubell and Clan, Sound Voltex Pop'n Music 19-24, Jubeat Saucer, Saucer Fulfill, Prop, Qubell and Clan, Sound Voltex
1, 2, 3 Season 1/2 and 4, Dance Dance Revolution X2, X3, 2013, 2014 and Ace, MÚSECA 1, 1, 2, 3 Season 1/2 and 4, Dance Dance Revolution X2, X3, 2013, 2014 and Ace, MÚSECA 1,
MÚSECA 1+1/2, Reflec Beat, Limelight, Colette, groovin'!! Upper, Volzza 1 and Volzza 2, MÚSECA 1+1/2, MÚSECA Plus, Reflec Beat, Limelight, Colette, groovin'!! Upper, Volzza
and finally The\*BishiBashi. 1 and Volzza 2, and finally The\*BishiBashi.
Do not use this utility to serve production traffic. Instead, see Do not use this utility to serve production traffic. Instead, see
`bemani/wsgi/services.wsgi` for a ready-to-go WSGI file that can be used with uWSGI `bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python
and nginx. virtualenv containing this project and its dependencies, uWSGI and nginx.
## shell ## shell
@ -231,12 +238,13 @@ probably the easiest way to do so.
## struct ## struct
A convenience utility for helping reverse-engineer structures out of game DLLs. You A convenience utility for helping reverse-engineer structures out of game DLLs/EXEs.
can give this a physical DLL offset or a virtual memory address for the start and You can give this a physical DLL offset or a virtual memory address for the start and
end of the data as well as a python struct format (documentation at end of the data as well as a python struct format (documentation at
https://docs.python.org/3.6/library/struct.html) and this will print the decoded https://docs.python.org/3.6/library/struct.html) and this will print the decoded
data to the screen as a series of tuples. Run it like `./struct --help` to see how data to the screen one entry per line. It includes several enhancements for decoding
to use this. pointers to sub-structures and pointers to C strings. Run it like `./struct --help`
to see how to use this.
## trafficgen ## trafficgen
@ -257,12 +265,16 @@ Unit test frontend utility. This will invoke nosetests on the embarrasingly smal
collection of unit tests for this repository. If you are making modifications, it can collection of unit tests for this repository. If you are making modifications, it can
be useful to write a test first (placed in the `bemani/tests/` directory) and code be useful to write a test first (placed in the `bemani/tests/` directory) and code
from there. It is also useful when optimizing or profiling, and also to verify that from there. It is also useful when optimizing or profiling, and also to verify that
you haven't regressed anything. you haven't regressed anything. Supports all options that nosetests does including
filtering, verbose printing and such. Run it like `./verifylibs --help` to see how
to do these things. When submitting pull requests make sure to run this so you know
that all tests pass.
## verifylint ## verifylint
Lint invocation utility. This simply invokes flake8 with various options so that you Lint invocation utility. This simply invokes flake8 with various options so that you
can see you haven't introduced any lint errors. can see you haven't introduced any lint errors. When submitting pull requests make sure
to run this so you know you aren't introducing any lint errors into the codebase.
## verifytraffic ## verifytraffic
@ -272,13 +284,17 @@ game and verify that network services are actually working. This assumes that yo
running "services". Do not point this at a production instance since it **will** running "services". Do not point this at a production instance since it **will**
submit bogus cards, scores, names and the like and mess up your network. This takes submit bogus cards, scores, names and the like and mess up your network. This takes
a config file which sets up how the client should behave. See `config/trafficgen.yaml` a config file which sets up how the client should behave. See `config/trafficgen.yaml`
for a sample file that can be used. for a sample file that can be used. When submitting pull requests make sure to run
this against a development version of your server so you know you haven't broken any
existing game implementations.
## verifytyping ## verifytyping
Typing invocation utility. Since this repository is fully typed, this verifies that you Typing invocation utility. Since this repository is fully typed, this verifies that you
haven't introduced any type errors and often catches bugs far faster than attemping to haven't introduced any type errors and often catches bugs far faster than attemping to
play a round only to see that you misused a class or misspelled a variable. play a round only to see that you misused a class or misspelled a variable. When
submitting pull requests make sure to run this so you know you aren't introducing any
type errors into the codebase.
# Installation # Installation
@ -288,10 +304,13 @@ The code contained here assumes Python 3.6 as the base. If you don't have or don
want to install Python 3.6 as your system python, it is recommended to use want to install Python 3.6 as your system python, it is recommended to use
virtualenv to create a virtual environment. The rest of the installation will assume virtualenv to create a virtual environment. The rest of the installation will assume
you have Python 3.6 working properly (and are in an activated virtual environment if you have Python 3.6 working properly (and are in an activated virtual environment if
this is the route you've chosen to go). This code is designed to run on Linux. this is the route you've chosen to go). If you have a newer version of python available
this code should be compatible with that as well. This code is designed to run on Linux.
However, it has been tested successfully on Windows and OSX as it doesn't use any However, it has been tested successfully on Windows and OSX as it doesn't use any
system libraries and sticks to pure python. YMMV in this regard, however, since the system libraries and contains pure Python implementations of all necessary pieces.
whole suite is built and tested using a Debian-based derivative. YMMV in this regard, however, since the whole suite is built and tested using a
Debian-based derivative and several critical pieces of code have much faster Cython
implementations.
To install the required libraries, run the following command out of the root of the To install the required libraries, run the following command out of the root of the
repository. This should allow all of the programs to at least start, but it still repository. This should allow all of the programs to at least start, but it still
@ -313,7 +332,7 @@ pip install -r requirements.txt
``` ```
Installing MySQL is outside the scope of this readme, so it is assumed that you have Installing MySQL is outside the scope of this readme, so it is assumed that you have
a MySQL database with access to create a new DB and tables within it. Note that this a MySQL database with permission to create a new DB and tables within it. Note that this
software requires MySQL version 5.7 or greater. This is due to the extensive use of software requires MySQL version 5.7 or greater. This is due to the extensive use of
the "json" column type added in 5.7. Create a database (the default database with the "json" column type added in 5.7. Create a database (the default database with
this code is 'bemani') accessed by some user and password (the default user/pass for this code is 'bemani') accessed by some user and password (the default user/pass for
@ -347,7 +366,7 @@ At this point, games will boot when pointed at the network, but you won't be abl
to save scores. This is due to the missing song/chart -> score mapping. You will find to save scores. This is due to the missing song/chart -> score mapping. You will find
default configuration files for the traffic generator and the services backend in default configuration files for the traffic generator and the services backend in
the config/ directory. If you've customized your database setup, you will want to the config/ directory. If you've customized your database setup, you will want to
update the hostname/username/password/database here. You will also want to update update the hostname/username/password/database in the configs. You will also want to update
the server address and frontend URL to customize your instance. the server address and frontend URL to customize your instance.
To create the song/chart -> score mapping, you will want to run through the following To create the song/chart -> score mapping, you will want to run through the following
@ -355,16 +374,16 @@ section to import data from each game series. Be sure to substitute your own ser
config in place of the default if you've customized it. Note that if there have been updates config in place of the default if you've customized it. Note that if there have been updates
to the files since you initially imported, you can run with the `--update` flag which to the files since you initially imported, you can run with the `--update` flag which
forces the metadata to be overwritten in the DB instead of skipped. This won't normally forces the metadata to be overwritten in the DB instead of skipped. This won't normally
happen, but if you make improvements to music DB parsing, you will want this to update happen, but if you make improvements to music DB parsing, you will want to do this to update
your network. your database.
Note that you'll see a lot of re-used entries. That will happen when the import script Note that you'll see a lot of re-used song entries. That will happen when the import script
finds an existing set of charts for the same song in a different game version and links finds an existing set of charts for the same song in a different game version and links
the two game versions together. This is how scores can be shared across different versions the two game versions together. This is how scores can be shared across different versions
of the same game. of the same game.
If you happen to already be an authorized client of a BEMAPI-compatible server, you can If you happen to already be an authorized client of a BEMAPI-compatible server, you can
fast-track initializing your server by pointing it at the remote and using its existing fast-track initializing your server by pointing it at the remote server and using its existing
database to seed your own. If this is the case, run the following command to perform database to seed your own. If this is the case, run the following command to perform
a complete initialization. If you wish to update your initial setup with newer data, a complete initialization. If you wish to update your initial setup with newer data,
perhaps because a new supported game is available, you can run the following script and perhaps because a new supported game is available, you can run the following script and
@ -591,9 +610,9 @@ An example invocation of the tool is as follows:
Once your network is up and running, if you pull new code down, the DB schema may have Once your network is up and running, if you pull new code down, the DB schema may have
changed. For that, use the same DB util script detailed above in the following manner. changed. For that, use the same DB util script detailed above in the following manner.
This will walk through all migration scripts that you haven't applied and bring your DB This will walk through all migration scripts that you haven't applied and bring your DB
up to spec. It is recommended to create a deploy script that knows how to restart uWSGI, up to spec. It is recommended to create a deploy script that knows how to install dependencies
install a new version of these utilities to your production virtualenv and then runs this and install a new version of these utilities to your production virtualenv and then runs the
script to ensure that your production DB is kept in sync: following script to ensure that your production DB is kept in sync with upstream changes:
``` ```
./dbutils --config config/server.yaml upgrade ./dbutils --config config/server.yaml upgrade
@ -608,6 +627,12 @@ script to promote yourself to admin, similar to this command:
./dbutils --config config/server.yaml add-admin --username <your-name-here> ./dbutils --config config/server.yaml add-admin --username <your-name-here>
``` ```
Once you have create an admin account, you can use tools on the frontend to establish
arcades and their owners. Any administrator can check system settings including event
logs, configure news entries, help users recover passwords and change cards and the like.
Arcade owners can choose how paseli is supported on the machines in their arcade, grant
users credits and configure game options such as which events are active.
## Troubleshooting ## Troubleshooting
If you followed the above instructions, the network should "Just Work" for you. However If you followed the above instructions, the network should "Just Work" for you. However
@ -643,14 +668,15 @@ As alluded to several times in this README, the recommended way to run a product
instance of this code is to set up uWSGI fronted by nginx. You should SSL-encrypt instance of this code is to set up uWSGI fronted by nginx. You should SSL-encrypt
the frontend and the API services, and its recommended to use LetsEncrypt for a the frontend and the API services, and its recommended to use LetsEncrypt for a
free certificate that you can manage easily. There are other ways to run this software free certificate that you can manage easily. There are other ways to run this software
but I have no experience with or advice on them. and provide SSL credentials but I have no experience with or advice on them.
The easiest way to get up and running is to install MySQL 5.7, nginx and uWSGI along The easiest way to get up and running is to install MySQL 5.7, nginx and uWSGI along
with Python 3.6 or higher. Create a directory where the services will live and place with Python 3.6 or higher. Create a directory where the services will live and place
a virtualenv inside it (outside the scope of this document). Then, the wsgi files found a virtualenv inside it (outside the scope of this document). Then, the wsgi files found
in `bemani/wsgi/` can be placed in here, uWSGI pointed at them and nginx set up. The in `bemani/wsgi/` can be placed in the directory, uWSGI pointed at them and nginx set up.
setup for the top-level package will include all of the frontend templates, so you can The setup for the top-level package will include all of the frontend templates, so you
set up a nginx directory to serve the static resources directly. can set up a nginx directory to serve the static resources directly by pointing at the
static directory inside your virtualenv.
For example configurations, an example install script, and an example script to back For example configurations, an example install script, and an example script to back
up your MySQL instance, see the `examples/` directory. up your MySQL instance, see the `examples/` directory.