Cool_Tools/bemaniutils
1
0
Fork 0
mirror of synced 2024-11-13 17:40:47 +01:00
Code Releases Activity

Update README with a fresh set of eyes (I forgot most of what I wrote).

Browse Source
This commit is contained in:
Jennifer Taylor 2023-08-20 19:35:06 +00:00
parent 387ec1a272
commit 04c3cd1f23
3 changed files with 195 additions and 112 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

295
README.md
View File

@ -22,36 +22,42 @@ every goddamn night.
## 2dxutils ## 2dxutils
A utility for unpacking and repacking `.2dx` files. This isn't the best utility and A utility for unpacking and repacking `.2dx` audio container files. Can extract
I think there are more complete and more accurate programs out there. However, RIFF WAV audio out of an existing `.2dx` file, update an existing file or create
they all lack source as far as I could tell, so I developed this. Run it like a new file from scratch given some WAV files. This isn't the best utility and I
think there are more complete and more accurate programs out there. However, they
all lack source as far as I could tell, so I developed this. Run it like
`./2dxutils --help` to see help output and determine how to use this. `./2dxutils --help` to see help output and determine how to use this.
## afputils ## afputils
Utilities for working with several animation formats found across a vast range Utilities for working with several animation formats found across a vast range
of games. This includes a TXP2 container parser, a GE2D shape parser and an of games. This includes a TXP2 container parser and repacker, a GE2D shape parser
AFP/BSI parser. Together, they make a set of utilities that attempts to work and an AFP/BSI parser. Together, they make a set of utilities that attempts to work
with AFP, the fork of SWF that handles animations in various games. This utility with AFP, the fork of SWF that handles animations in various games. This utility
is capable of rendering animations out of IFS and TXP2 files as well as providing is capable of rendering animations out of IFS and TXP2 files as well as providing
decompiled pseudocode for the flash-like bytecode found in many animation files. decompiled pseudocode for the flash-like bytecode found in many animation files. It
Note that this format is similar to SWF and thus very complicated. Therefore, it is also capable of unpacking and repacking TXP2 containers with new texture files.
Note that this format is based on SWF and thus very complicated. Therefore, it
is unlikely that these tools will correctly handle all animations from all games is unlikely that these tools will correctly handle all animations from all games
that it encounters. Run it like `./afputils --help` to see help output and determine that it encounters. Run it like `./afputils --help` to see help output and determine
how to use it. how to use it.
## api ## api
Development version of this repository's BEMAPI implementation. Run it like Development version of this repository's BEMAPI implementation. This serves as the
`./api --help` to see help output and determine how to use this. Much like REST-like API layer for inter-network federation of scores, profiles and rivals.
"services" and "frontend", this should be pointed at the development version of Run it like `./api --help` to see help output and determine how to use this. Much
your services config file, which holds information about the MySQL database that like "services" and "frontend", this should be pointed at the development version
of your services config file which holds information about the MySQL database that
this should connect to as well as what game series are supported. See this should connect to as well as what game series are supported. See
`config/server.yaml` for an example file that you can modify. `config/server.yaml` for an example file that you can modify.
Do not use this utility to serve production traffic. Instead, see Do not use this utility to serve production traffic. Instead, see
`bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python `bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python
virtualenv containing this project and its dependencies, uWSGI and nginx. virtualenv containing this project and its dependencies, uWSGI and nginx. Note that
if you do not wish to make your network available for federation then this entire
service can be omitted.
## arcutils ## arcutils
@ -65,19 +71,22 @@ A utility which takes a particular game's asset directory and converts files for
on the frontend. This optionally enables things such as customization previews on the on the frontend. This optionally enables things such as customization previews on the
frontend. Much like "read", this requires a properly setup config file which points frontend. Much like "read", this requires a properly setup config file which points
the frontend as well as this utility at the correct location to store converted the frontend as well as this utility at the correct location to store converted
assets. See `config/server.yaml` for an example file that you can modify. assets. See `config/server.yaml` for an example file that you can modify. Unlike "read",
this utility is entirely optional. However, if you do not convert assets for games that
you are running, you will miss out on preview graphics on the frontend. Run it like
`./assetparse --help` to see help output and determine how to use this.
## bemanishark ## bemanishark
A wire sniffer that can decode eAmuse packets and print them. Run it on a computer A wire sniffer that can decode eAmuse packets and print them. Run it on a computer
that can sniff traffic between an eAmusement server and a supported game and it will that can sniff traffic between an eAmusement server and a supported game and it will
spit out the requests and responses XML-formatted identically to the legacy easerver spit out the requests and responses XML-formatted identically to the XML output in
XML output. This works on both binary and XML traffic. Note that it does not have the the logs of "services". This works on both binary and XML traffic. Note that it does
capability to sniff SSL-encrypted traffic, so don't even bother trying to run this not have the capability to sniff SSL-encrypted traffic, so don't even bother trying
at an arcade with official support. to run this at an arcade with official support.
Run it like `sudo ./bemanishark` to invoke. Will run indefinitely until killed Run it like `sudo ./bemanishark` to invoke. Will run indefinitely until killed
(Ctrl-C will suffice). Run like `./bemanishark --help for options. Without options, (Ctrl-C will suffice). Run like `./bemanishark --help` for options. Without options,
it assumes you want to sniff port 80 for all addresses. Note that it doesn't support it assumes you want to sniff port 80 for all addresses. Note that it doesn't support
the Base64 binary blob formats found in SN1 and 2. Note also that over time it will the Base64 binary blob formats found in SN1 and 2. Note also that over time it will
start to lose packets. This is a bug that I never figured out, and it appears to be start to lose packets. This is a bug that I never figured out, and it appears to be
@ -97,8 +106,12 @@ Run it like `./binutils --help` to see help and learn how to use this.
## bootstrap ## bootstrap
A utility for quickly bootstrapping a local setup's music database from an already A utility for quickly bootstrapping a local setup's music database from an already
running BEMAPI-compatible server. This is better documented in the below "Database running BEMAPI-compatible server that has been set up for federation. This is better
Initialization" section. documented in the below "Database Initialization" section. Note that this utility assumes
no omnimix support and will bootstrap only normal game databases. The BEMAPI federation
protocol does support omnimix, so if you are bootstrapping against a running instance
that has omnimix databases and you wish to support omnimixes as well, you can look at
the source to this and manually run commands for the games in question.
## cardconvert ## cardconvert
@ -114,11 +127,11 @@ of the card.
A command-line utility for working with the DB used by "api", "services" and "frontend". A command-line utility for working with the DB used by "api", "services" and "frontend".
This utility includes options for creating tables in a newly-created DB, granting and This utility includes options for creating tables in a newly-created DB, granting and
revoking admin rights to the frontend, generating migration scripts for live DBs, and revoking admin rights to the frontend, generating migration scripts for production DBs,
upgrading live DBs based on previously created migration scripts. Its driven by alembic and upgrading production DBs based on previously created migration scripts. Its driven
under the hood. You will use `create` on initial setup to generate a working MySQL by alembic under the hood. You will use `create` on initial setup to generate a working
database. If you change the schema in code, you can use this again with the `generate` MySQL database. If you change the schema in code, you can use this again with the `generate`
option to generate a migration sript. Whenever you run an upgrade to your production option to generate a migration script. Whenever you run an upgrade to your production
instance, you should run this against your production DB with the `upgrade` option to instance, you should run this against your production DB with the `upgrade` option to
bring your production DB up to sync with the code you are deploying. Run it like bring your production DB up to sync with the code you are deploying. Run it like
`./dbutils --help` to see all options. The config file that this works on is the same `./dbutils --help` to see all options. The config file that this works on is the same
@ -129,20 +142,22 @@ that is given to "api", "services" and "frontend".
A simple wrapper frontend to black, the formatter used on this project. Running this will A simple wrapper frontend to black, the formatter used on this project. Running this will
auto-format all of the python code that might need formatting, leaving the rest out. When auto-format all of the python code that might need formatting, leaving the rest out. When
submitting pull requests make sure to run this so that your code conforms to the style submitting pull requests make sure to run this so that your code conforms to the style
used by this project! used by this project! Run this like `./formatfiles` to fix up all files in the repository.
## frontend ## frontend
Development version of a frontend server allowing for account and server administration Development version of a frontend server allowing for account and server administration
as well as score viewing and profile editing. Run it like `./frontend --help` to see as well as score viewing and profile editing. Run it like `./frontend --help` to see
help output and determine how to use this. Much like "services" and "api", this should help output and determine how to use this. Much like "services" and "api", this should
be pointed at the development version of your services config file, which holds be pointed at the development version of your services config file which holds
information about the MySQL database that this should connect to as well as what game information about the MySQL database that this should connect to as well as what game
series are supported. See `config/server.yaml` for an example file that you can modify. series are supported. See `config/server.yaml` for an example file that you can modify.
Do not use this utility to serve production traffic. Instead, see Do not use this utility to serve production traffic. Instead, see
`bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python `bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python
virtualenv containing this project and its dependencies, uWSGI and nginx. virtualenv containing this project and its dependencies, uWSGI and nginx. Note thati
this shares a config file with "services" and "api" but is independent, sharing state
with them using the production DB only.
## ifsutils ## ifsutils
@ -155,34 +170,45 @@ Run it like `./ifsutils --help` to see help output and learn how to use it.
## iidxutils ## iidxutils
A utility for patching IIDX music database files. Note that this currently can only A utility for patching IIDX music database files. Note that this currently can only
apply a "hide leggendarias from normal folders" patch, although its probable that it apply a "hide leggendarias from normal folders" patch, although it would be trivial
can be extended for other uses. to extend for other uses such as song renames, difficulty patches and other fixups.
Run it like `./iidxutils --help` to see help output and learn how to use it.
## jsx ## jsx
A utility which takes the existing JSX files in the repository and builds them for A utility which takes the existing JSX files in the repository and compiles them to
you. This is offered purely as a way to serve JSX files in a production setup from raw JS files for you. This is offered purely as a way to serve JSX files in a
nginx or similar instead of compiling them on-the-fly when they are requested. You production setup from nginx or similar instead of compiling them on-the-fly when
can use this to lower cold-start load times of your frontend. they are requested. You can use this to lower cold-start load times of your frontend.
If you do not make use of this or you are running the developent version of "frontend"
then JSX files are compiled on-the-fly when they are requested by the browser. This
behavior makes fast iteration possible by treating JSX files the same way that the
"frontend" debug utility treats python source files but removes the ability for a
production webserver such as nginx to serve static files. Run it like `./jsx --help`
to see help output and learn how to use it.
## proxy ## proxy
A utility to MITM an eAmuse session. Point a game at the port this listens on, and A utility to MITM a network session. Point a game at the port this listens on, and
point it at another network to see the packets flowing between the two. Takes care point this at another network to see the packets flowing between the two. Takes care
of rewriting the facility message to MITM all messages. Has the ability to rewrite of rewriting the facility message to MITM all messages. Has the ability to rewrite
a request/response on the fly which is not currently used except for facility rewriting. a request/response on the fly which is not currently used except for facility rewriting.
Its possible that this could be used to on-the-fly patch packets coming back from a Its possible that this could be used to on-the-fly patch packets coming back from a
network which you don't control to do things such as enable paseli and adjust other network which you don't control to do things such as enable paseli and adjust other
settings that you cannot normally access. Logs in an identical format to bemanishark. settings that you cannot normally access. Logs in an identical format to "bemanishark".
Useful for black-box RE of other networks. Note that this does not have the ability Useful for black-box RE of other networks. Note that this does not have the ability
to MITM SSL-encrypted traffic, so don't bother trying to use this on an official to MITM SSL-encrypted traffic, so don't bother trying to use this on an official
network. network.
This also has the ability to route a packet to one of several known networks based on This also has the ability to route a packet to one of several known networks based on
the PCBID, so this can also be used as a proxy for switching networks on the fly. the PCBID, so this can also be used as a proxy for switching networks on the fly.
With a config file, this can be used as a VIP of sorts, allowing you to point all of With a config file, this can be used as a HTTP VIP of sorts, allowing you to point all of
your games at a single server that runs this proxy, and forward games on a per-PCBID your games at a single server that runs this proxy and forward games on a per-PCBID
basis to various networks behind the scenes. For an example config file to use "proxy" basis to various networks behind the scenes. This can come in especially handy if you
are serving traffic from the same network your games are on. The network will auto-detect
the public-facing IP of games when they connect and use that info for matching support.
This breaks for local connections, so you might want to set up an offsite proxy instance
so that the correct public-facing IP is detected. For an example config file to use "proxy"
as a VIP, see `config/proxy.yaml`. For a more reliable proxy, use the wsgi version as a VIP, see `config/proxy.yaml`. For a more reliable proxy, use the wsgi version
of this utility located at `bemani/wsgi/proxy.wsgi` along with uWSGI and nginx. of this utility located at `bemani/wsgi/proxy.wsgi` along with uWSGI and nginx.
@ -190,11 +216,14 @@ Run it like `./proxy --help` to see how to use this utility.
## psmap ## psmap
A utility to take an offset from a DLL/EXE file and produce python code that would generate A utility to take an offset from a DLL/EXE file pointing at a psmap structure and
a suitable response that said DLL/EXE will properly parse. Essentially, if you are produce python code that would generate a suitable response that said DLL/EXE will
reversing a new game and they use the `psmap` function to decode all or part of a properly parse. Essentially, if you are reversing a new game and they use the `psmap`
packet, you can grab either the physical offset into the DLL or the virtual address of function to decode all or part of a packet, you can grab either the physical offset
the data and use this utility to generate the code necessary to service that request. into the DLL or the virtual address of the data and use this utility to generate
the code necessary to service that request. Note that some psmap structures are
dynamically generated at runtime. "psmap" supports this by emulating x86 and x64 code
to reconstruct the final structure. This feature can be optionally enabled if needed.
Run it like `./psmap --help` to see how to use this utility. Run it like `./psmap --help` to see how to use this utility.
## read ## read
@ -208,22 +237,23 @@ below in the "Installation" section.
## replay ## replay
A utility to take a packet as logged by proxy, services, trafficgen or bemanishark, A utility to take a packet as logged by "proxy", "services", "trafficgen" or
and replay that packet against a particular server. Useful for quickly grabbing "bemanishark", and replay that packet against a particular server. Useful for
packets that caused a crash and debugging the crash (and verifying the fix). It is quickly grabbing packets that caused a crash and debugging the crash (and verifying
also compatible with the packet logs found on exception and unsupported packet the fix). It is also compatible with the packet logs found on exception and unsupported
messages in the Admin Event Logs page on the frontend. It also lets you replay that packet messages in the Admin Event Logs page on the frontend. It also lets you
packet against your production instance once you fix the issue in case that packet replay that packet against your production instance once you fix the issue in case
was a score or profile update that you care about. that packet was a score or profile update that you care about. Run it like
`./replay --help` to see all information and usage.
## responsegen ## responsegen
A utility to take a packet as logged by proxy, services, trafficgen or bemanishark, A utility to take a packet as logged by "proxy", "services", "trafficgen" or
and generate python code that would have generated that exact packet. Useful for "bemanishark", and generate python code that would have generated that exact packet.
quickly grabbing packets sniffed from another network and prototyping new game support. Useful for quickly grabbing packets sniffed from another network and prototyping new
Think of this as a combination of "replay" and "psmap". This is also extremely useful game support. Think of this as a combination of "replay" and "psmap". This is also
when building new integration test clients. Run it like `./responsegen --help` to extremely useful when building new integration test clients. Run it like
see all information and usage. `./responsegen --help` to see all information and usage.
## sampleclient ## sampleclient
@ -231,8 +261,8 @@ A very barebones sample client for the BEMAPI implementation contained in this r
Run it like `./sampleclient --help` to see help output and determine how to use this. Run it like `./sampleclient --help` to see help output and determine how to use this.
Essentially, this is provided as a barebones client that does nothing other than Essentially, this is provided as a barebones client that does nothing other than
print fetched info to the screen. You can use this as a starting point for an print fetched info to the screen. You can use this as a starting point for an
application that uses BEMAPI to fetch info from an api instance or to test your application that uses BEMAPI to fetch info from an "api" instance or to test your
production installation. production installation to make sure it is ready for federation.
## scheduler ## scheduler
@ -249,11 +279,16 @@ This should be given the same config file as "api", "frontend" and "services".
Development version of an eAmusement protocol server using flask and the protocol Development version of an eAmusement protocol server using flask and the protocol
libraries also used in "bemanishark" and "trafficgen". Currently it lets most modern libraries also used in "bemanishark" and "trafficgen". Currently it lets most modern
BEMANI games boot and supports full profile and events for Beatmania IIDX 20-26, BEMANI games boot and supports full scores, profile and events for Beatmania IIDX 20-26,
Pop'n Music 19-26, Jubeat Saucer, Saucer Fulfill, Prop, Qubell, Clan and Festo, Sound Pop'n Music 19-26, Jubeat Saucer, Saucer Fulfill, Prop, Qubell, Clan and Festo, Sound
Voltex 1, 2, 3 Season 1/2 and 4, Dance Dance Revolution X2, X3, 2013, 2014 and Ace, Voltex 1, 2, 3 Season 1/2 and 4, Dance Dance Revolution X2, X3, 2013, 2014 and Ace,
MÚSECA 1, MÚSECA 1+1/2, MÚSECA Plus, Reflec Beat, Limelight, Colette, groovin'!! Upper, MÚSECA 1, MÚSECA 1+1/2, MÚSECA Plus, Reflec Beat, Limelight, Colette, groovin'!! Upper,
Volzza 1 and Volzza 2, Metal Gear Arcade, and finally The\*BishiBashi. Volzza 1 and Volzza 2, Metal Gear Arcade, and finally The\*BishiBashi. Note that it also
has matching support for all Reflec Beat versions as well as MGA. By default, this serves
traffic based solely on the database it is configured against. If you federate with
other networks using the "Data API" admin page, it will upgrade to serving traffic
based on the profiles, scores and statistics of all connected networks as well as the
local database. Run like `./services --help` to see how to use this.
Do not use this utility to serve production traffic. Instead, see Do not use this utility to serve production traffic. Instead, see
`bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python `bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with a Python
@ -263,7 +298,8 @@ virtualenv containing this project and its dependencies, uWSGI and nginx.
A convenience wrapper to invoke a Python 3 shell that has paths set up to import the A convenience wrapper to invoke a Python 3 shell that has paths set up to import the
modules in this repository. If you want to tinker or write a quick one-off, this is modules in this repository. If you want to tinker or write a quick one-off, this is
probably the easiest way to do so. probably the easiest way to do so. Run this like `./shell` to drop into a Python REPL
which has the paths set up for correct imports.
## struct ## struct
@ -272,8 +308,9 @@ You can give this a physical DLL offset or a virtual memory address for the star
end of the data as well as a python struct format (documentation at end of the data as well as a python struct format (documentation at
https://docs.python.org/3.6/library/struct.html) and this will print the decoded https://docs.python.org/3.6/library/struct.html) and this will print the decoded
data to the screen one entry per line. It includes several enhancements for decoding data to the screen one entry per line. It includes several enhancements for decoding
pointers to sub-structures and pointers to C strings. Run it like `./struct --help` pointers to sub-structures and pointers to C strings. Note that much like "psmap", this
to see how to use this. has the ability to print out structures that are dynamically constructed at runtime by
emulating x86 and x64 instructions. Run it like `./struct --help` to see how to use this.
## trafficgen ## trafficgen
@ -286,8 +323,10 @@ currently generates traffic emulating Beatmania IIDX 20-26, Pop'n Music 19-26, J
Saucer, Fulfill, Prop, Qubell, Clan and Festo, Sound Voltex 1, 2, 3 Season 1/2 and 4, Saucer, Fulfill, Prop, Qubell, Clan and Festo, Sound Voltex 1, 2, 3 Season 1/2 and 4,
Dance Dance Revolution X2, X3, 2013, 2014 and Ace, The\*BishiBashi, MÚSECA 1 and MÚSECA Dance Dance Revolution X2, X3, 2013, 2014 and Ace, The\*BishiBashi, MÚSECA 1 and MÚSECA
1+1/2, Reflec Beat, Reflec Beat Limelight, Reflec Beat Colette, groovin'!! Upper, 1+1/2, Reflec Beat, Reflec Beat Limelight, Reflec Beat Colette, groovin'!! Upper,
Volzza 1 and Volzza 2 and can verify card events and score events, as well as PASELI Volzza 1 and Volzza 2 ad Metal Gear Arcade and can verify card events and score events
transactions. as well as PASELI transactions. Run it like `./trafficgen --help` to see how to use this.
Note tha this takes a config file which sets up how the clients behave. See
`config/trafficgen.yaml` for a sample file that can be used.
## verifylibs ## verifylibs
@ -297,14 +336,15 @@ be useful to write a test first (placed in the `bemani/tests/` directory) and co
from there. It is also useful when optimizing or profiling, and also to verify that from there. It is also useful when optimizing or profiling, and also to verify that
you haven't regressed anything. Supports all options that nosetests does including you haven't regressed anything. Supports all options that nosetests does including
filtering, verbose printing and such. Run it like `./verifylibs --help` to see how filtering, verbose printing and such. Run it like `./verifylibs --help` to see how
to do these things. When submitting pull requests make sure to run this so you know to do these things. When submitting pull requests make sure to run this across all
that all tests pass. tests by running `./verifylibs` so you know that all tests pass.
## verifylint ## verifylint
Lint invocation utility. This simply invokes flake8 with various options so that you Lint invocation utility. This simply invokes flake8 with various options so that you
can see you haven't introduced any lint errors. When submitting pull requests make sure can see you haven't introduced any lint errors. When submitting pull requests make sure
to run this so you know you aren't introducing any lint errors into the codebase. to run this so you know you aren't introducing any lint errors into the codebase. Run
it like `./verifylint` to print out any lint warnings your modifications have caused.
## verifytraffic ## verifytraffic
@ -320,33 +360,33 @@ existing game implementations.
## verifytyping ## verifytyping
Typing invocation utility. Since this repository is fully typed, this verifies that you Type-checking invocation utility. Since this repository is fully typed, this verifies
haven't introduced any type errors and often catches bugs far faster than attemping to that you haven't introduced any type errors and often catches bugs far faster than
play a round only to see that you misused a class or misspelled a variable. When attemping to play a round only to see that you misused a class or misspelled a variable.
submitting pull requests make sure to run this so you know you aren't introducing any When submitting pull requests make sure to run this like `./verifytyping` so you know
type errors into the codebase. you aren't introducing any type errors into the codebase.
# Installation # Installation
## Dependency Setup ## Dependency Setup
The code contained here assumes Python 3.6 as the base. If you don't have or don't The code contained here assumes Python 3.6 as the base although it should work with
want to install Python 3.6 as your system python, it is recommended to use any newer version of python as well. If you don't have or don't want to install Python
virtualenv to create a virtual environment. The rest of the installation will assume 3.6 as your system python, it is recommended to use virtualenv to create a virtual
you have Python 3.6 working properly (and are in an activated virtual environment if environment. The rest of the installation will assume you have Python 3.6 working
this is the route you've chosen to go). If you have a newer version of python available properly (and are in an activated virtual environment if this is the route you've
this code should be compatible with that as well. This code is designed to run on Linux. chosen to go). If you have a newer version of python available this code should be
However, it has been tested successfully on Windows and OSX as it doesn't use any compatible with that as well. This code is designed to run on Linux. However, it has
system libraries and contains pure Python implementations of all necessary pieces. been tested successfully on Windows and OSX as it doesn't use any system-specific
YMMV in this regard, however, since the whole suite is built and tested using a libraries and contains pure Python implementations of all necessary pieces. YMMV in
Debian-based derivative and several critical pieces of code have much faster Cython this regard, however, since the whole suite is built and tested using a Debian-based
implementations. derivative and several critical pieces of code have much faster Cython implementations.
To install the required libraries, run the following command out of the root of the To install the required libraries, run the following command out of the root of the
repository. This should allow all of the programs to at least start, but it still repository. This should allow all of the programs to at least start, but it still
requires a MySQL database for many of them to be useful. This step has a dependency requires a MySQL database for many of them to be useful. This step has a dependency
on an isntalled MySQL server and client as well as MySQL client development libraries. on an isntalled MySQL server and client as well as MySQL client development libraries.
It also assumes that you've installed the 'wheel' package already. In order to It also assumes that you've installed the 'wheel' python package already. In order to
compile the mysql client libraries, you will need to have libssl and libcrypto on your compile the mysql client libraries, you will need to have libssl and libcrypto on your
system as well. To satisfy these requirements on a Debian-based install, run the system as well. To satisfy these requirements on a Debian-based install, run the
following command: following command:
@ -378,17 +418,20 @@ above.
In order to run the frontend, Python will need to find a javascript runtime. This In order to run the frontend, Python will need to find a javascript runtime. This
is so it can precompile react components at render time so there doesn't need to be is so it can precompile react components at render time so there doesn't need to be
a compile step when developing. I found it absolutely bonkers that the backend could a compile step when developing the front-end. I found it absolutely bonkers that the
be on-the-fly reloaded but I had to go through an entire build process to produce backend could be on-the-fly reloaded but I had to go through an entire build process
interpreted JS code, so I went the route of self-contained services instead. Installing to produce interpreted JS code, so I went the route of self-contained services
a JS runtime is also outside the scope of this document, but a quick way to get started instead. Installing a JS runtime is also outside the scope of this document, but a
is to install node.js. quick way to get started is to install node.js.
The default configuration points the frontend/backend cache at `/tmp`. It is recommended The default configuration points the frontend/backend cache at `/tmp`. If you are going
to change to a different directory, as using `/tmp` can cause some items not to be cached. to run with a filesystem cache in production then it is recommended to change to a
This is due to the way `/tmp` on Linux restricts file access to the creator only, so different directory, as using `/tmp` can cause some items not to be cached. This is
if you share your cache with multiple utilities running under different users, it will due to the way `/tmp` on Linux restricts file access to the creator only, so if you share
fail to reuse the cache and drastically slow down the frontend. your cache with multiple utilities running under different users, it will fail to reuse
the cache and drastically slow down the frontend. Alternatively, you can set up a
memcached server and point your production instance at that instead of using a filesystem
cache.
## Database Initialization ## Database Initialization
@ -415,29 +458,36 @@ of the same game.
If you happen to already be an authorized client of a BEMAPI-compatible server, you can If you happen to already be an authorized client of a BEMAPI-compatible server, you can
fast-track initializing your server by pointing it at the remote server and using its existing fast-track initializing your server by pointing it at the remote server and using its existing
database to seed your own. If this is the case, run the following command to perform database to seed your own. If this is the case, run the following command to perform
a complete initialization. If you wish to update your initial setup with newer data, a complete initialization. Note that the "bootstrap" script has entries for non-omnimix
perhaps because a new supported game is available, you can run the following script and versions only. You can edit it to add omnimix versions as well if you wish to provide
append the `--update` flag to it. Otherwise, run the following command like so: omnimix support and are pointing at another BEMAPI-compatible instance which also has
support. If you wish to update your initial setup with newer data, perhaps because a
new supported game is available, you can run the following script and append the
`--update` flag to it. Otherwise, run the following command like so:
``` ```
./bootstrap --config config/server.yaml --server http://some-server.here/ --token some-token-here ./bootstrap --config config/server.yaml --server http://some-server.here/ --token some-token-here
``` ```
If you do not have a BEMAPI-compatible server, you can initialize the server from the If you are not federating with an existing BEMAPI-compatible server, you can initialize
game files of the games you wish to run. See the following sections for how exactly to the server from the game files of the games you wish to run. See the following sections
do that. for how exactly to do that.
### Pop'n Music ### Pop'n Music
For Pop'n Music, get the game DLL from the version of the game you want to import and For Pop'n Music, get the game DLL from the version of the game you want to import and
run a command like so. This network supports versions 19-26 so you will want to run this run a command like so. This network supports versions 19-26 so you will want to run this
command once for every version, giving the correct DLL file: command once for every version, giving the correct DLL file. Note that there are several
versions of each game floating around and the "read" script attempts to support as many
as it can but you might encounter a version of the game which hasn't been mapped yet.
An example is as follows:
``` ```
./read --config config/server.yaml --series pnm --version 22 --bin popn22.dll ./read --config config/server.yaml --series pnm --version 22 --bin popn22.dll
``` ```
For add songs of a XML from omnimix v2, run a command like this: If you are looking to support omnimix v2, you can add songs out of an XML like this:
``` ```
./read --config config/server.yaml --series pnm --version omni-24 --bin popn24.dll --xml your_songs_db.xml ./read --config config/server.yaml --series pnm --version omni-24 --bin popn24.dll --xml your_songs_db.xml
@ -446,7 +496,7 @@ For add songs of a XML from omnimix v2, run a command like this:
If you have more than one XML you want to add, you can run this command with a folder with all your XML files: If you have more than one XML you want to add, you can run this command with a folder with all your XML files:
``` ```
./read --config config/server.yaml --series pnm --version omni-24 --bin popn24.dll --folder my_xmls_path ./read --config config/server.yaml --series pnm --version omni-24 --bin popn24.dll --folder my/xmls/path
``` ```
### Jubeat ### Jubeat
@ -642,6 +692,12 @@ You can start the BEMAPI REST server in debug mode using a command similar to:
./api --port 18573 --config config/server.yaml ./api --port 18573 --config config/server.yaml
``` ```
You can run scheduled work to generate dailies and other such things using a command like so:
```
./scheduler --config config/server.yaml
```
The network config for any particular game should look similar to the following, with The network config for any particular game should look similar to the following, with
the correct hostname or IP filled in for the services URL. No path is necessary. Note the correct hostname or IP filled in for the services URL. No path is necessary. Note
that if you wish to switch between an existing network and one you serve using the that if you wish to switch between an existing network and one you serve using the
@ -672,7 +728,9 @@ You will want to set up a cron job or similar scheduling agent to call "schedule
on a regular basis. It is recommended to call it every five minutes since there are cache on a regular basis. It is recommended to call it every five minutes since there are cache
warming portions for the front-end that expire every 10 minutes. Game code will register warming portions for the front-end that expire every 10 minutes. Game code will register
with internal handlers to perform daily/weekly actions which are kicked off by this script. with internal handlers to perform daily/weekly actions which are kicked off by this script.
An example invocation of the tool is as follows: Note that if you don't want to have this done automatically in your development environment,
you can simply invoke it before testing with a game. An example invocation of the tool is
as follows:
``` ```
./scheduler --config config/server.yaml ./scheduler --config config/server.yaml
@ -756,6 +814,19 @@ export PYTHONIOENCODING=utf-8
export PYTHONLEGACYWINDOWSSTDIO=utf-8 export PYTHONLEGACYWINDOWSSTDIO=utf-8
``` ```
### None of my own games pointing at my self-hosted network can be matched with
This is due to the way routers handle internal connections to their public-facing IP.
Even if you tell your games to connect to the DNS entry or public-facing IP of your
network, the router will handle the request internally. Therefore, all of the games
on your own local network will have their public-facing IP detected wrong. You can
get around this by either hosting your network services off-site, or paying for cheap
colocation somewhere and running a "proxy" instance there. The proxy utility (and its
production wsgi counterpart) know how to forward the detected IP through any number of
proxy hops. Once you have set up an external proxy relay or moved your network services
off-site, your own games will get their public-facing IP detected correctly. Remember
that you should still forward the port assigned to each game on the admin interface.
## Production Setup ## Production Setup
As alluded to several times in this README, the recommended way to run a production As alluded to several times in this README, the recommended way to run a production
@ -770,7 +841,9 @@ a virtualenv inside it (outside the scope of this document). Then, the wsgi file
in `bemani/wsgi/` can be placed in the directory, uWSGI pointed at them and nginx set up. in `bemani/wsgi/` can be placed in the directory, uWSGI pointed at them and nginx set up.
The setup for the top-level package will include all of the frontend templates, so you The setup for the top-level package will include all of the frontend templates, so you
can set up a nginx directory to serve the static resources directly by pointing at the can set up a nginx directory to serve the static resources directly by pointing at the
static directory inside your virtualenv. static directory inside your virtualenv. If you use "assetparse" to extract assets or
"jsx" to compile static JS, you'll want to add entries in your nginx config to serve
those as well.
For example configurations, an example install script, and an example script to back For example configurations, an example install script, and an example script to back
up your MySQL instance, see the `examples/` directory. Note that several files have up your MySQL instance, see the `examples/` directory. Note that several files have
@ -798,7 +871,7 @@ For documentation on how the protocol layer works, see "PROTOCOL". For documenta
on how the eAmusement server is intended to work, see "BACKEND". Inside `bemani/data/` on how the eAmusement server is intended to work, see "BACKEND". Inside `bemani/data/`
the various DB model files have comments detailing the intended usage of each of the the various DB model files have comments detailing the intended usage of each of the
tables. For documentation on how the BEMAPI REST API should respond, please see the tables. For documentation on how the BEMAPI REST API should respond, please see the
BEMAPI specification repository at https://github.com/DragonMinded/bemapi. BEMAPI specification in `bemani/api/`.
When updating DB schema in the various `bemani/data/` python files, you will most-likely When updating DB schema in the various `bemani/data/` python files, you will most-likely
want to generate a migration for others to use. For that, we've integrated with alembic want to generate a migration for others to use. For that, we've integrated with alembic
@ -840,7 +913,7 @@ compiled `.so` files, otherwise your changes will not show up when you test.
# License and Usage # License and Usage
All of the code in this repository is released under public domain. No attribution or All of the code in this repository is released under the public domain. No attribution or
releasing of source code is required. No warranty of the code or functionality is implied. releasing of source code is required. No warranty of the code or functionality is implied.
However, open source would not flourish without contributions from users the world across. However, open source would not flourish without contributions from users the world across.
Pull requests are therefore appreciated! Please note, however, that the code contained in this Pull requests are therefore appreciated! Please note, however, that the code contained in this
@ -853,6 +926,6 @@ by the original manufacturer or games which have not reached their support end-o
Similarly, while I cannot control what you decide to do with this software, it would be Similarly, while I cannot control what you decide to do with this software, it would be
very, very stupid to attempt to run this in a public arcade or convention, or to attempt very, very stupid to attempt to run this in a public arcade or convention, or to attempt
to use this with games that you or another person or business are charging money for. If to use this with games that you or another person or business are charging money for. If
you do decide to do this anyway, do not advertise association with any of this software in you decide to do this anyway, do not advertise association with any of this software in
any form whatsoever. Attempting to use this software for commercial gain or to compete any form whatsoever. Attempting to use this software for commercial gain or to compete
publicly with official game support goes directly against the stated goals of this software. publicly with official game support goes directly against the stated goals of this software.

9
examples/install
View File

@ -8,10 +8,19 @@ set -e
pushd /path/to/git/checkout pushd /path/to/git/checkout
source /path/to/your/virtualenv/bin/activate source /path/to/your/virtualenv/bin/activate
# Install dependencies, install this library.
pip install --upgrade pip pip install --upgrade pip
pip install . -U --force-reinstall pip install . -U --force-reinstall
# Copy the WSGI files over the old ones, recompile JSX.
cp bemani/wsgi/*.wsgi /path/to/your/wsgi/files
./jsx --output-directory /path/to/static/jsx/directory
# Perform DB schema migrations
./dbutils --config /path/to/your/root/server.yaml upgrade ./dbutils --config /path/to/your/root/server.yaml upgrade
deactivate deactivate
popd popd
sudo service uwsgi restart && ./preload sudo service uwsgi restart && ./preload

3
examples/preload
View File

@ -2,7 +2,8 @@
# A utility that hits your web frontend, causing the cache for JSX # A utility that hits your web frontend, causing the cache for JSX
# translations to be rendered. Call this after/during your installation # translations to be rendered. Call this after/during your installation
# so that the first web request after an installation doesn't need to # so that the first web request after an installation doesn't need to
# warm the cache. # warm the cache. Note that this isn't necessary if you are compiling
# JSX files for static serving via nginx.
cd /path/to/your/virtualenv/lib/python3.6/site-packages/bemani/frontend/static cd /path/to/your/virtualenv/lib/python3.6/site-packages/bemani/frontend/static
for url in $(find -name "*.react.js" | sed 's,^\.,https://your-domain.com/jsx,'); do for url in $(find -name "*.react.js" | sed 's,^\.,https://your-domain.com/jsx,'); do