Minor adjustments to the readme formatting, add some todos.
This commit is contained in:
Jennifer Taylor
parent
74c0407173
commit
90fd8ad5d8
157
README.md
157
README.md
@ -1,5 +1,5 @@
|
||||
Introduction
|
||||
------------
|
||||
# Introduction
|
||||
|
||||
A collection of programs for working with various games in the BEMANI series. This
|
||||
could be untangled quite a bit into various modules that provide simpler pieces.
|
||||
However, this is how it ended up evolving over time. This repository includes
|
||||
@ -20,15 +20,15 @@ for being good RE partners for several games, and sharing good finds. Thanks to
|
||||
helvetica for helping with game RE and retweeting cute anime ladies onto my feed
|
||||
every goddamn night.
|
||||
|
||||
2dxutils
|
||||
========
|
||||
## 2dxutils
|
||||
|
||||
A utility for unpacking and repacking `.2dx` files. This isn't the best utility and
|
||||
I think there are more complete and more accurate programs out there. However,
|
||||
they all lack source as far as I could tell, so I developed this. Run it like
|
||||
`./2dxutils --help` to see help output and determine how to use this.
|
||||
|
||||
api
|
||||
===
|
||||
## api
|
||||
|
||||
Development version of this repository's BEMAPI implementation. Run it like
|
||||
`./api --help` to see help output and determine how to use this. Much like
|
||||
"services" and "frontend", this should be pointed at the development version of
|
||||
@ -40,14 +40,14 @@ Do not use this utility to serve production traffic. Instead, see
|
||||
`bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with uWSGI
|
||||
and nginx.
|
||||
|
||||
arcutils
|
||||
========
|
||||
## arcutils
|
||||
|
||||
A utility for unpacking `.arc` files. This does not currently repack files. However,
|
||||
the format is so trivial that adding such a feature would be fairly easy. Run it
|
||||
like `./arcutils --help` to see help output and determine how to use this.
|
||||
|
||||
bemanishark
|
||||
===========
|
||||
## bemanishark
|
||||
|
||||
A wire sniffer that can decode eAmuse packets and print them. Run it on a computer
|
||||
that can sniff traffic between an eAmusement server and a supported game and it will
|
||||
spit out the requests and responses XML-formatted identically to the legacy easerver
|
||||
@ -66,23 +66,29 @@ TCP stream.
|
||||
This utility might be better if rewritten to be a plugin for Wireshark instead of
|
||||
a standalone sniffing utility, but I don't have the time.
|
||||
|
||||
binutils
|
||||
========
|
||||
## binutils
|
||||
|
||||
A utility for unpacking raw binxml files (files that use the same encoding scheme
|
||||
as the binary network protocol) to their XML representation. This is useful for
|
||||
examining raw binary blobs or digging into unknown file formats that contain binxml.
|
||||
Run it like `./binutils --help` to see help and learn how to use this.
|
||||
|
||||
cardconvert
|
||||
===========
|
||||
## bootstrap
|
||||
|
||||
A utility for quickly bootstrapping a local setup's music database from an already
|
||||
running BEMAPI-compatible server. This is more documented in the below "Database
|
||||
Initialization" section.
|
||||
|
||||
## cardconvert
|
||||
|
||||
A command-line utility for converting between card numbers written on the back of a
|
||||
card and the card ID stored in the RFID of the card. Run it like `./cardconvert --help`
|
||||
to see how to use this. This will sanitize input, so you can feed it card numbers
|
||||
with or without spaces, and you can mix up 1 and I as well as 0 and O, and it will
|
||||
properly handle decoding. This supports both new and old style cards.
|
||||
|
||||
dbutils
|
||||
=======
|
||||
## dbutils
|
||||
|
||||
A command-line utility for working with the DB used by "api", "services" and "frontend".
|
||||
This utility includes options for creating tables in a newly-created DB, granting and
|
||||
revoking admin rights to the frontend, generating migration scripts for live DBs, and
|
||||
@ -95,8 +101,8 @@ bring your production DB up to sync with the code you are deploying. Run it like
|
||||
`./dbutils --help` to see all options. The config file that this works on is the same
|
||||
that is given to "api", "services" and "frontend".
|
||||
|
||||
frontend
|
||||
========
|
||||
## frontend
|
||||
|
||||
Development version of a frontend server allowing for account and server administration
|
||||
as well as score viewing and profile editing. Run it like `./frontend --help` to see
|
||||
help output and determine how to use this. Much like "services" and "api", this should
|
||||
@ -108,22 +114,22 @@ Do not use this utility to serve production traffic. Instead, see
|
||||
`bemani/wsgi/frontend.wsgi` for a ready-to-go WSGI file that can be used with uWSGI
|
||||
and nginx.
|
||||
|
||||
ifsutils
|
||||
========
|
||||
## ifsutils
|
||||
|
||||
A mediocre utility that can extract `.ifs` files. This has a lot of baked in
|
||||
assumptions and is not nearly as good as other open-source utilities for extracting
|
||||
files. It also cannot repack files. This is included for posterity, and because some
|
||||
bootstrapping code requires it in order to fully start a production server.
|
||||
Run it like `./ifsutils --help` to see help output and learn how to use it.
|
||||
|
||||
iidxutils
|
||||
=========
|
||||
## iidxutils
|
||||
|
||||
A utility for patching IIDX music database files. Note that this currently can only
|
||||
apply a "hide leggendarias from normal folders" patch, although its probable that it
|
||||
can be extended for other uses.
|
||||
|
||||
proxy
|
||||
=====
|
||||
## proxy
|
||||
|
||||
A utility to MITM an eAmuse session. Point a game at the port this listens on, and
|
||||
point it at another network to see the packets flowing between the two. Takes care
|
||||
of rewriting the facility message to MITM all messages. Has the ability to rewrite
|
||||
@ -144,8 +150,8 @@ of this utility located at `bemani/wsgi/proxy.wsgi` along with uWSGI and nginx.
|
||||
|
||||
Run it like `./proxy --help` to see how to use this utility.
|
||||
|
||||
psmap
|
||||
=====
|
||||
## psmap
|
||||
|
||||
A utility to take an offset from a DLL file and produce python code that would generate
|
||||
a suitable response that said DLL will properly parse. Essentially, if you are
|
||||
reversing a new game and they use the `psmap` utility to decode all or part of a
|
||||
@ -155,8 +161,8 @@ Note that this doesn't currently work on 64bit games, but it should be trivial t
|
||||
figure out the differences in the 64-bit psmap implementation. Run it like
|
||||
`./psmap --help` to see how to use this utility.
|
||||
|
||||
read
|
||||
====
|
||||
## read
|
||||
|
||||
A utility to read music DB information out of game files and populate a database.
|
||||
This should be given the same config file as "api", "services" or "frontend" and
|
||||
assumes that "dbutils" has already been used to instantiate a valid MySQL DB. It
|
||||
@ -164,16 +170,16 @@ also assumes you have the correct game files to read out of. Run it like
|
||||
`./read --help` to see how to use it. This utility's uses are extensively documented
|
||||
below in the "Installation" section.
|
||||
|
||||
replay
|
||||
======
|
||||
## replay
|
||||
|
||||
A utility to take a packet as logged by proxy, services, trafficgen or bemanishark,
|
||||
and replay that packet against a particular server. Useful for quickly grabbing
|
||||
packets that caused a crash and debugging the crash (and verifying the fix). It also
|
||||
lets you replay that packet against your production instance once you fixed the issue
|
||||
in case that packet was a score or profile update that you care about.
|
||||
|
||||
responsegen
|
||||
===========
|
||||
## responsegen
|
||||
|
||||
A utility to take a packet as logged by proxy, services, trafficgen or bemanishark,
|
||||
and generate python code that would have generated that exact packet. Useful for
|
||||
quickly grabbing packets sniffed from another network and prototyping new game support.
|
||||
@ -181,8 +187,8 @@ Think of this as a combination of "replay" and "psmap". This is also extremely u
|
||||
when building new integration test clients. Run it like `./responsegen --help` to
|
||||
see all information and usage.
|
||||
|
||||
scheduler
|
||||
=========
|
||||
## scheduler
|
||||
|
||||
A command-line utility for kicking off scheduled work that must be performed against a
|
||||
DB. This includes picking new dailies/weeklies, new courses, etc... depending on the
|
||||
game and any requirements that the server perform some actual calculation based on
|
||||
@ -192,8 +198,8 @@ should be seen as a utility-specific cron handler. You can safely run this repea
|
||||
and as frequently as desired. Run like `./scheduler --help` to see how to ues this.
|
||||
This should be given the same config file as "api", "frontend" and "services".
|
||||
|
||||
services
|
||||
========
|
||||
## services
|
||||
|
||||
Development version of an eAmusement protocol server using flask and the protocol
|
||||
libraries also used in "bemanishark" and "trafficgen". Currently it lets most modern
|
||||
BEMANI games boot and supports full profile and events for Beatmania IIDX 20-24,
|
||||
@ -206,14 +212,14 @@ Do not use this utility to serve production traffic. Instead, see
|
||||
`bemani/wsgi/services.wsgi` for a ready-to-go WSGI file that can be used with uWSGI
|
||||
and nginx.
|
||||
|
||||
shell
|
||||
=====
|
||||
## shell
|
||||
|
||||
A convenience wrapper to invoke a Python 3 shell that has paths set up to import the
|
||||
modules in this repository. If you want to tinker or write a quick one-off, this is
|
||||
probably the easiest way to do so.
|
||||
|
||||
struct
|
||||
======
|
||||
## struct
|
||||
|
||||
A convenience utility for helping reverse-engineer structures out of game DLLs. You
|
||||
can give this a physical DLL offset or a virtual memory address for the start and
|
||||
end of the data as well as a python struct format (documentation at
|
||||
@ -221,8 +227,8 @@ https://docs.python.org/3.6/library/struct.html) and this will print the decoded
|
||||
data to the screen as a series of tuples. Run it like `./struct --help` to see how
|
||||
to use this.
|
||||
|
||||
trafficgen
|
||||
==========
|
||||
## trafficgen
|
||||
|
||||
A utility for simulating traffic to an eAmusement service. Given a particular game,
|
||||
this will run through and attempt to verify simple operation of that service. No
|
||||
guarantees are made on the accuracy of the emulation though I've strived to be
|
||||
@ -234,21 +240,21 @@ Dance Revolution X2, X3, 2013, 2014 and Ace, The\*BishiBashi, MÚSECA 1 and MÚS
|
||||
Reflec Beat, Reflec Beat Limelight, Reflec Beat Colette, groovin'!! Upper, Volzza 1 and
|
||||
Volzza 2 and can verify card events and score events, as well as PASELI transactions.
|
||||
|
||||
verifylibs
|
||||
==========
|
||||
## verifylibs
|
||||
|
||||
Unit test frontend utility. This will invoke nosetests on the embarrasingly small
|
||||
collection of unit tests for this repository. If you are making modifications, it can
|
||||
be useful to write a test first (placed in the `bemani/tests/` directory) and code
|
||||
from there. It is also useful when optimizing or profiling, and also to verify that
|
||||
you haven't regressed anything.
|
||||
|
||||
verifylint
|
||||
==========
|
||||
## verifylint
|
||||
|
||||
Lint invocation utility. This simply invokes flake8 with various options so that you
|
||||
can see you haven't introduced any lint errors.
|
||||
|
||||
verifytraffic
|
||||
=============
|
||||
## verifytraffic
|
||||
|
||||
A utility which attempts to call "trafficgen" for each supported game on the network.
|
||||
Think of this as a full integration test suite, as it will sweep through each supported
|
||||
game and verify that network services are actually working. This assumes that you are
|
||||
@ -257,17 +263,16 @@ submit bogus cards, scores, names and the like and mess up your network. This ta
|
||||
a config file which sets up how the client should behave. See `config/trafficgen.yaml`
|
||||
for a sample file that can be used.
|
||||
|
||||
verifytyping
|
||||
============
|
||||
## verifytyping
|
||||
|
||||
Typing invocation utility. Since this repository is fully typed, this verifies that you
|
||||
haven't introduced any type errors and often catches bugs far faster than attemping to
|
||||
play a round only to see that you misused a class or misspelled a variable.
|
||||
|
||||
Installation
|
||||
------------
|
||||
# Installation
|
||||
|
||||
## Dependency Setup
|
||||
|
||||
Dependency Setup
|
||||
================
|
||||
The code contained here assumes Python 3.6 as the base. If you don't have or don't
|
||||
want to install Python 3.6 as your system python, it is recommended to use
|
||||
virtualenv to create a virtual environment. The rest of the installation will assume
|
||||
@ -319,8 +324,8 @@ This is due to the way `/tmp` on Linux restricts file access to the creator only
|
||||
if you share your cache with multiple utilities running under different users, it will
|
||||
fail to reuse the cache and drastically slow down the frontend.
|
||||
|
||||
Database Initialization
|
||||
=======================
|
||||
## Database Initialization
|
||||
|
||||
At this point, games will boot when pointed at the network, but you won't be able
|
||||
to save scores. This is due to the missing song/chart -> score mapping. You will find
|
||||
default configuration files for the traffic generator and the services backend in
|
||||
@ -355,16 +360,16 @@ If you do not have a BEMAPI-compatible server, you can initialize the server fro
|
||||
game files of the games you wish to run. See the following sections for how exactly to
|
||||
do that.
|
||||
|
||||
Pop'n Music
|
||||
~~~~~~~~~~~
|
||||
### Pop'n Music
|
||||
|
||||
For Pop'n Music, get the game DLL from the version of the game you want to import and
|
||||
run a command like so. This network supports versions 19-24 so you will want to run this
|
||||
command once for every version, giving the correct DLL file:
|
||||
|
||||
./read --config config/server.yaml --series pnm --version 22 --bin popn22.dll
|
||||
|
||||
Jubeat
|
||||
~~~~~~
|
||||
### Jubeat
|
||||
|
||||
For Jubeat, get the music XML out of the data directory of the mix you are importing,
|
||||
and then use "read" with `--series jubeat` and `--version` corresponding to the following
|
||||
table:
|
||||
@ -386,8 +391,8 @@ after importing all mixes:
|
||||
./read --config config/server.yaml --series jubeat --version all --tsv \
|
||||
data/jubeat.tsv
|
||||
|
||||
IIDX
|
||||
~~~~
|
||||
### IIDX
|
||||
|
||||
For IIDX, you will need the data directory of the mix you wish to support. The import
|
||||
script automatically scrapes the music DB as well as the song charts to determine
|
||||
difficulty, notecounts and BPM. For a normal mix, you will want to run the command like
|
||||
@ -410,8 +415,8 @@ after importing all mixes (this fixes some inconsistencies in names):
|
||||
./read --config config/server.yaml --series iidx --version all --tsv \
|
||||
data/iidx.tsv
|
||||
|
||||
DDR
|
||||
~~~
|
||||
### DDR
|
||||
|
||||
For DDR, you will need the game DLL and `musicdb.xml` from the game you wish to import,
|
||||
and then run a command similar to the following. You will want to use the version
|
||||
corresponding to version in the following table:
|
||||
@ -431,8 +436,8 @@ For DDR Ace, there is no `musicdb.xml` or game DLL needed. Instead, you will nee
|
||||
./read --config config/server.yaml --series ddr --version 16 \
|
||||
--bin data/arc/startup.arc
|
||||
|
||||
SDVX
|
||||
~~~~
|
||||
### SDVX
|
||||
|
||||
For SDVX, you will need the game DLL and `music_db.xml` from the game you wish to import,
|
||||
and then run the following command, modifying the version parameter as required.
|
||||
Note that for SDVX 1, you want the `music_db.xml` file in `data/others/music_db/` directory,
|
||||
@ -460,24 +465,24 @@ work properly. To do so, run the following command.
|
||||
./read --config config/server.yaml --series sdvx --version 4 \
|
||||
--xml data/others/appeal_card.xml
|
||||
|
||||
MÚSECA
|
||||
~~~~~~
|
||||
### MÚSECA
|
||||
|
||||
For MÚSECA, you will need the `music-info.xml` file from the game you wish to import.
|
||||
Then, run the following command, modifying the version parameter as required.
|
||||
|
||||
./read --config config/server.yaml --series museca --version 1 \
|
||||
--xml data/museca/xml/music-info.xml
|
||||
|
||||
Reflec Beat
|
||||
~~~~~~~~~~~
|
||||
### Reflec Beat
|
||||
|
||||
For Reflec Beat, get the game DLL from the version of the game you want to import and
|
||||
run a command like so. This network supports Reflec Beat up through Volzza 2, so you
|
||||
will want to run this with versions 1-6 to completely initialize:
|
||||
|
||||
./read --config config/server.yaml --series reflec --version 1 --bin reflecbeat.dll
|
||||
|
||||
Running Locally
|
||||
===============
|
||||
## Running Locally
|
||||
|
||||
Once you've set all of this up, you can start the network in debug mode using a command
|
||||
similar to:
|
||||
|
||||
@ -537,8 +542,8 @@ script to promote yourself to admin, similar to this command:
|
||||
|
||||
./dbutils --config config/server.yaml add-admin --username <your-name-here>
|
||||
|
||||
Production Setup
|
||||
================
|
||||
## Production Setup
|
||||
|
||||
As alluded to several times in this README, the recommended way to run a production
|
||||
instance of this code is to set up uWSGI fronted by nginx. You should SSL-encrypt
|
||||
the frontend and the API services, and its recommended to use LetsEncrypt for a
|
||||
@ -555,8 +560,8 @@ set up a nginx directory to serve the static resources directly.
|
||||
For example configurations, an example install script, and an example script to back
|
||||
up your MySQL instance, see the `examples/` directory.
|
||||
|
||||
Contributing
|
||||
------------
|
||||
# Contributing
|
||||
|
||||
Contributions are welcome! Before submitting a pull request, ensure that your code
|
||||
is type-hint clean by running `./verifytyping` and ensure that it hasn't broken basic
|
||||
libraries with `./verifylibs`. Make sure that it is also lint-clean with `./verifylint`.
|
||||
|
||||
5
TODO.md
5
TODO.md
@ -8,3 +8,8 @@ Things that I have not gotten around to doing.
|
||||
- Make the frontend work better on mobile. It works well enough, but it could be a much better experience.
|
||||
- Support for DanEvo. I meant to do this but my DanEvo ended up in storage before I could tackle it, so the only thing that exists at the moment is a rudimentary music DB parser.
|
||||
- Figure out phase/unlock/etc bits for some older IIDX and Pop'n Music versions and hook them up to the Arcade panel to allow switching events.
|
||||
|
||||
General maintenance that I have not yet performed.
|
||||
|
||||
- Unpin flake8 and address new lint errors from newer version of flake8.
|
||||
- Upgrade legacy uses of .format() to use f-strings instead.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user